No OneTemporary
Actions

Size

5 MB

Referenced Files

None

Subscribers

None

View Options

This file is larger than 256 KB, so syntax highlighting was skipped.

	Index: vendor-sys/openzfs/dist/CODE_OF_CONDUCT.md
	===================================================================
	--- vendor-sys/openzfs/dist/CODE_OF_CONDUCT.md (revision 366773)
	+++ vendor-sys/openzfs/dist/CODE_OF_CONDUCT.md (revision 366774)
	@@ -1,2 +1,2 @@
	The [OpenZFS Code of Conduct](http://www.open-zfs.org/wiki/Code_of_Conduct)
	-applies to spaces associated with the ZFS on Linux project, including GitHub.
	+applies to spaces associated with the OpenZFS project, including GitHub.
	Index: vendor-sys/openzfs/dist/META
	===================================================================
	--- vendor-sys/openzfs/dist/META (revision 366773)
	+++ vendor-sys/openzfs/dist/META (revision 366774)
	@@ -1,10 +1,10 @@
	Meta: 1
	Name: zfs
	Branch: 1.0
	Version: 2.0.0
	-Release: rc2
	+Release: rc3
	Release-Tags: relext
	License: CDDL
	Author: OpenZFS
	-Linux-Maximum: 5.8
	+Linux-Maximum: 5.9
	Linux-Minimum: 3.10
	Index: vendor-sys/openzfs/dist/README.md
	===================================================================
	--- vendor-sys/openzfs/dist/README.md (revision 366773)
	+++ vendor-sys/openzfs/dist/README.md (revision 366774)
	@@ -1,35 +1,35 @@
	![img](https://openzfs.github.io/openzfs-docs/_static/img/logo/480px-Open-ZFS-Secondary-Logo-Colour-halfsize.png)

	OpenZFS is an advanced file system and volume manager which was originally
	developed for Solaris and is now maintained by the OpenZFS community.
	This repository contains the code for running OpenZFS on Linux and FreeBSD.

	[![codecov](https://codecov.io/gh/openzfs/zfs/branch/master/graph/badge.svg)](https://codecov.io/gh/openzfs/zfs)
	[![coverity](https://scan.coverity.com/projects/1973/badge.svg)](https://scan.coverity.com/projects/openzfs-zfs)

	# Official Resources

	* [Documentation](https://openzfs.github.io/openzfs-docs/) - for using and developing this repo
	* [ZoL Site](https://zfsonlinux.org) - Linux release info & links
	* [Mailing lists](https://openzfs.github.io/openzfs-docs/Project%20and%20Community/Mailing%20Lists.html)
	* [OpenZFS site](http://open-zfs.org/) - for conference videos and info on other platforms (illumos, OSX, Windows, etc)

	# Installation

	-Full documentation for installing OpenZFS on your favorite Linux distribution can
	-be found at the [ZoL Site](https://zfsonlinux.org/).
	+Full documentation for installing OpenZFS on your favorite operating system can
	+be found at the [Getting Started Page](https://openzfs.github.io/openzfs-docs/Getting%20Started/index.html).

	# Contribute & Develop

	We have a separate document with [contribution guidelines](./.github/CONTRIBUTING.md).

	We have a [Code of Conduct](./CODE_OF_CONDUCT.md).

	# Release

	OpenZFS is released under a CDDL license.
	For more details see the NOTICE, LICENSE and COPYRIGHT files; `UCRL-CODE-235197`

	# Supported Kernels
	* The `META` file contains the officially recognized supported Linux kernel versions.
	* Supported FreeBSD versions are 12-STABLE and 13-CURRENT.
	Index: vendor-sys/openzfs/dist/cmd/arc_summary/arc_summary2
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/arc_summary/arc_summary2 (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/arc_summary/arc_summary2 (revision 366774)
	@@ -1,1093 +1,1093 @@
	#!/usr/bin/env python2
	#
	# $Id: arc_summary.pl,v 388:e27800740aa2 2011-07-08 02:53:29Z jhell $
	#
	# Copyright (c) 2008 Ben Rockwood <benr@cuddletech.com>,
	# Copyright (c) 2010 Martin Matuska <mm@FreeBSD.org>,
	# Copyright (c) 2010-2011 Jason J. Hellenthal <jhell@DataIX.net>,
	# All rights reserved.
	#
	# Redistribution and use in source and binary forms, with or without
	# modification, are permitted provided that the following conditions
	# are met:
	#
	# 1. Redistributions of source code must retain the above copyright
	# notice, this list of conditions and the following disclaimer.
	# 2. Redistributions in binary form must reproduce the above copyright
	# notice, this list of conditions and the following disclaimer in the
	# documentation and/or other materials provided with the distribution.
	#
	# THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS ``AS IS'' AND
	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
	# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
	# ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
	# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
	# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
	# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
	# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
	# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
	# SUCH DAMAGE.
	#
	# If you are having troubles when using this script from cron(8) please try
	# adjusting your PATH before reporting problems.
	#
	# Note some of this code uses older code (eg getopt instead of argparse,
	# subprocess.Popen() instead of subprocess.run()) because we need to support
	# some very old versions of Python.
	#

	"""Print statistics on the ZFS Adjustable Replacement Cache (ARC)

	Provides basic information on the ARC, its efficiency, the L2ARC (if present),
	the Data Management Unit (DMU), Virtual Devices (VDEVs), and tunables. See the
	in-source documentation and code at
	-https://github.com/zfsonlinux/zfs/blob/master/module/zfs/arc.c for details.
	+https://github.com/openzfs/zfs/blob/master/module/zfs/arc.c for details.
	"""

	import getopt
	import os
	import sys
	import time
	import errno

	from subprocess import Popen, PIPE
	from decimal import Decimal as D


	if sys.platform.startswith('freebsd'):
	# Requires py27-sysctl on FreeBSD
	import sysctl

	def load_kstats(namespace):
	"""Collect information on a specific subsystem of the ARC"""

	base = 'kstat.zfs.misc.%s.' % namespace
	return [(kstat.name, D(kstat.value)) for kstat in sysctl.filter(base)]

	def load_tunables():
	return dict((ctl.name, ctl.value) for ctl in sysctl.filter('vfs.zfs'))

	elif sys.platform.startswith('linux'):

	def load_kstats(namespace):
	"""Collect information on a specific subsystem of the ARC"""

	kstat = 'kstat.zfs.misc.%s.%%s' % namespace
	path = '/proc/spl/kstat/zfs/%s' % namespace
	with open(path) as f:
	entries = [line.strip().split() for line in f][2:] # Skip header
	return [(kstat % name, D(value)) for name, _, value in entries]

	def load_tunables():
	basepath = '/sys/module/zfs/parameters'
	tunables = {}
	for name in os.listdir(basepath):
	if not name:
	continue
	path = '%s/%s' % (basepath, name)
	with open(path) as f:
	value = f.read()
	tunables[name] = value.strip()
	return tunables


	show_tunable_descriptions = False
	alternate_tunable_layout = False


	def handle_Exception(ex_cls, ex, tb):
	if ex is IOError:
	if ex.errno == errno.EPIPE:
	sys.exit()

	if ex is KeyboardInterrupt:
	sys.exit()


	sys.excepthook = handle_Exception


	def get_Kstat():
	"""Collect information on the ZFS subsystem from the /proc virtual
	file system. The name "kstat" is a holdover from the Solaris utility
	of the same name.
	"""

	Kstat = {}
	Kstat.update(load_kstats('arcstats'))
	Kstat.update(load_kstats('zfetchstats'))
	Kstat.update(load_kstats('vdev_cache_stats'))
	return Kstat


	def fBytes(b=0):
	"""Return human-readable representation of a byte value in
	powers of 2 (eg "KiB" for "kibibytes", etc) to two decimal
	points. Values smaller than one KiB are returned without
	decimal points.
	"""

	prefixes = [
	[2**80, "YiB"], # yobibytes (yotta)
	[2**70, "ZiB"], # zebibytes (zetta)
	[2**60, "EiB"], # exbibytes (exa)
	[2**50, "PiB"], # pebibytes (peta)
	[2**40, "TiB"], # tebibytes (tera)
	[2**30, "GiB"], # gibibytes (giga)
	[2**20, "MiB"], # mebibytes (mega)
	[2**10, "KiB"]] # kibibytes (kilo)

	if b >= 2**10:

	for limit, unit in prefixes:

	if b >= limit:
	value = b / limit
	break

	result = "%0.2f\t%s" % (value, unit)

	else:

	result = "%d\tBytes" % b

	return result


	def fHits(hits=0):
	"""Create a human-readable representation of the number of hits.
	The single-letter symbols used are SI to avoid the confusion caused
	by the different "short scale" and "long scale" representations in
	English, which use the same words for different values. See
	https://en.wikipedia.org/wiki/Names_of_large_numbers and
	https://physics.nist.gov/cuu/Units/prefixes.html
	"""

	numbers = [
	[10**24, 'Y'], # yotta (septillion)
	[10**21, 'Z'], # zetta (sextillion)
	[10**18, 'E'], # exa (quintrillion)
	[10**15, 'P'], # peta (quadrillion)
	[10**12, 'T'], # tera (trillion)
	[10**9, 'G'], # giga (billion)
	[10**6, 'M'], # mega (million)
	[10**3, 'k']] # kilo (thousand)

	if hits >= 1000:

	for limit, symbol in numbers:

	if hits >= limit:
	value = hits/limit
	break

	result = "%0.2f%s" % (value, symbol)

	else:

	result = "%d" % hits

	return result


	def fPerc(lVal=0, rVal=0, Decimal=2):
	"""Calculate percentage value and return in human-readable format"""

	if rVal > 0:
	return str("%0." + str(Decimal) + "f") % (100 * (lVal / rVal)) + "%"
	else:
	return str("%0." + str(Decimal) + "f") % 100 + "%"


	def get_arc_summary(Kstat):
	"""Collect general data on the ARC"""

	output = {}
	memory_throttle_count = Kstat[
	"kstat.zfs.misc.arcstats.memory_throttle_count"
	]

	if memory_throttle_count > 0:
	output['health'] = 'THROTTLED'
	else:
	output['health'] = 'HEALTHY'

	output['memory_throttle_count'] = fHits(memory_throttle_count)

	# ARC Misc.
	deleted = Kstat["kstat.zfs.misc.arcstats.deleted"]
	mutex_miss = Kstat["kstat.zfs.misc.arcstats.mutex_miss"]
	evict_skip = Kstat["kstat.zfs.misc.arcstats.evict_skip"]

	# ARC Misc.
	output["arc_misc"] = {}
	output["arc_misc"]["deleted"] = fHits(deleted)
	output["arc_misc"]['mutex_miss'] = fHits(mutex_miss)
	output["arc_misc"]['evict_skips'] = fHits(evict_skip)

	# ARC Sizing
	arc_size = Kstat["kstat.zfs.misc.arcstats.size"]
	mru_size = Kstat["kstat.zfs.misc.arcstats.mru_size"]
	mfu_size = Kstat["kstat.zfs.misc.arcstats.mfu_size"]
	meta_limit = Kstat["kstat.zfs.misc.arcstats.arc_meta_limit"]
	meta_size = Kstat["kstat.zfs.misc.arcstats.arc_meta_used"]
	dnode_limit = Kstat["kstat.zfs.misc.arcstats.arc_dnode_limit"]
	dnode_size = Kstat["kstat.zfs.misc.arcstats.dnode_size"]
	target_max_size = Kstat["kstat.zfs.misc.arcstats.c_max"]
	target_min_size = Kstat["kstat.zfs.misc.arcstats.c_min"]
	target_size = Kstat["kstat.zfs.misc.arcstats.c"]

	target_size_ratio = (target_max_size / target_min_size)

	# ARC Sizing
	output['arc_sizing'] = {}
	output['arc_sizing']['arc_size'] = {
	'per': fPerc(arc_size, target_max_size),
	'num': fBytes(arc_size),
	}
	output['arc_sizing']['target_max_size'] = {
	'ratio': target_size_ratio,
	'num': fBytes(target_max_size),
	}
	output['arc_sizing']['target_min_size'] = {
	'per': fPerc(target_min_size, target_max_size),
	'num': fBytes(target_min_size),
	}
	output['arc_sizing']['target_size'] = {
	'per': fPerc(target_size, target_max_size),
	'num': fBytes(target_size),
	}
	output['arc_sizing']['meta_limit'] = {
	'per': fPerc(meta_limit, target_max_size),
	'num': fBytes(meta_limit),
	}
	output['arc_sizing']['meta_size'] = {
	'per': fPerc(meta_size, meta_limit),
	'num': fBytes(meta_size),
	}
	output['arc_sizing']['dnode_limit'] = {
	'per': fPerc(dnode_limit, meta_limit),
	'num': fBytes(dnode_limit),
	}
	output['arc_sizing']['dnode_size'] = {
	'per': fPerc(dnode_size, dnode_limit),
	'num': fBytes(dnode_size),
	}

	# ARC Hash Breakdown
	output['arc_hash_break'] = {}
	output['arc_hash_break']['hash_chain_max'] = Kstat[
	"kstat.zfs.misc.arcstats.hash_chain_max"
	]
	output['arc_hash_break']['hash_chains'] = Kstat[
	"kstat.zfs.misc.arcstats.hash_chains"
	]
	output['arc_hash_break']['hash_collisions'] = Kstat[
	"kstat.zfs.misc.arcstats.hash_collisions"
	]
	output['arc_hash_break']['hash_elements'] = Kstat[
	"kstat.zfs.misc.arcstats.hash_elements"
	]
	output['arc_hash_break']['hash_elements_max'] = Kstat[
	"kstat.zfs.misc.arcstats.hash_elements_max"
	]

	output['arc_size_break'] = {}
	output['arc_size_break']['recently_used_cache_size'] = {
	'per': fPerc(mru_size, mru_size + mfu_size),
	'num': fBytes(mru_size),
	}
	output['arc_size_break']['frequently_used_cache_size'] = {
	'per': fPerc(mfu_size, mru_size + mfu_size),
	'num': fBytes(mfu_size),
	}

	# ARC Hash Breakdown
	hash_chain_max = Kstat["kstat.zfs.misc.arcstats.hash_chain_max"]
	hash_chains = Kstat["kstat.zfs.misc.arcstats.hash_chains"]
	hash_collisions = Kstat["kstat.zfs.misc.arcstats.hash_collisions"]
	hash_elements = Kstat["kstat.zfs.misc.arcstats.hash_elements"]
	hash_elements_max = Kstat["kstat.zfs.misc.arcstats.hash_elements_max"]

	output['arc_hash_break'] = {}
	output['arc_hash_break']['elements_max'] = fHits(hash_elements_max)
	output['arc_hash_break']['elements_current'] = {
	'per': fPerc(hash_elements, hash_elements_max),
	'num': fHits(hash_elements),
	}
	output['arc_hash_break']['collisions'] = fHits(hash_collisions)
	output['arc_hash_break']['chain_max'] = fHits(hash_chain_max)
	output['arc_hash_break']['chains'] = fHits(hash_chains)

	return output


	def _arc_summary(Kstat):
	"""Print information on the ARC"""

	# ARC Sizing
	arc = get_arc_summary(Kstat)

	sys.stdout.write("ARC Summary: (%s)\n" % arc['health'])

	sys.stdout.write("\tMemory Throttle Count:\t\t\t%s\n" %
	arc['memory_throttle_count'])
	sys.stdout.write("\n")

	# ARC Misc.
	sys.stdout.write("ARC Misc:\n")
	sys.stdout.write("\tDeleted:\t\t\t\t%s\n" % arc['arc_misc']['deleted'])
	sys.stdout.write("\tMutex Misses:\t\t\t\t%s\n" %
	arc['arc_misc']['mutex_miss'])
	sys.stdout.write("\tEvict Skips:\t\t\t\t%s\n" %
	arc['arc_misc']['evict_skips'])
	sys.stdout.write("\n")

	# ARC Sizing
	sys.stdout.write("ARC Size:\t\t\t\t%s\t%s\n" % (
	arc['arc_sizing']['arc_size']['per'],
	arc['arc_sizing']['arc_size']['num']
	)
	)
	sys.stdout.write("\tTarget Size: (Adaptive)\t\t%s\t%s\n" % (
	arc['arc_sizing']['target_size']['per'],
	arc['arc_sizing']['target_size']['num'],
	)
	)

	sys.stdout.write("\tMin Size (Hard Limit):\t\t%s\t%s\n" % (
	arc['arc_sizing']['target_min_size']['per'],
	arc['arc_sizing']['target_min_size']['num'],
	)
	)

	sys.stdout.write("\tMax Size (High Water):\t\t%d:1\t%s\n" % (
	arc['arc_sizing']['target_max_size']['ratio'],
	arc['arc_sizing']['target_max_size']['num'],
	)
	)

	sys.stdout.write("\nARC Size Breakdown:\n")
	sys.stdout.write("\tRecently Used Cache Size:\t%s\t%s\n" % (
	arc['arc_size_break']['recently_used_cache_size']['per'],
	arc['arc_size_break']['recently_used_cache_size']['num'],
	)
	)
	sys.stdout.write("\tFrequently Used Cache Size:\t%s\t%s\n" % (
	arc['arc_size_break']['frequently_used_cache_size']['per'],
	arc['arc_size_break']['frequently_used_cache_size']['num'],
	)
	)
	sys.stdout.write("\tMetadata Size (Hard Limit):\t%s\t%s\n" % (
	arc['arc_sizing']['meta_limit']['per'],
	arc['arc_sizing']['meta_limit']['num'],
	)
	)
	sys.stdout.write("\tMetadata Size:\t\t\t%s\t%s\n" % (
	arc['arc_sizing']['meta_size']['per'],
	arc['arc_sizing']['meta_size']['num'],
	)
	)
	sys.stdout.write("\tDnode Size (Hard Limit):\t%s\t%s\n" % (
	arc['arc_sizing']['dnode_limit']['per'],
	arc['arc_sizing']['dnode_limit']['num'],
	)
	)
	sys.stdout.write("\tDnode Size:\t\t\t%s\t%s\n" % (
	arc['arc_sizing']['dnode_size']['per'],
	arc['arc_sizing']['dnode_size']['num'],
	)
	)

	sys.stdout.write("\n")

	# ARC Hash Breakdown
	sys.stdout.write("ARC Hash Breakdown:\n")
	sys.stdout.write("\tElements Max:\t\t\t\t%s\n" %
	arc['arc_hash_break']['elements_max'])
	sys.stdout.write("\tElements Current:\t\t%s\t%s\n" % (
	arc['arc_hash_break']['elements_current']['per'],
	arc['arc_hash_break']['elements_current']['num'],
	)
	)
	sys.stdout.write("\tCollisions:\t\t\t\t%s\n" %
	arc['arc_hash_break']['collisions'])
	sys.stdout.write("\tChain Max:\t\t\t\t%s\n" %
	arc['arc_hash_break']['chain_max'])
	sys.stdout.write("\tChains:\t\t\t\t\t%s\n" %
	arc['arc_hash_break']['chains'])


	def get_arc_efficiency(Kstat):
	"""Collect information on the efficiency of the ARC"""

	output = {}

	arc_hits = Kstat["kstat.zfs.misc.arcstats.hits"]
	arc_misses = Kstat["kstat.zfs.misc.arcstats.misses"]
	demand_data_hits = Kstat["kstat.zfs.misc.arcstats.demand_data_hits"]
	demand_data_misses = Kstat["kstat.zfs.misc.arcstats.demand_data_misses"]
	demand_metadata_hits = Kstat[
	"kstat.zfs.misc.arcstats.demand_metadata_hits"
	]
	demand_metadata_misses = Kstat[
	"kstat.zfs.misc.arcstats.demand_metadata_misses"
	]
	mfu_ghost_hits = Kstat["kstat.zfs.misc.arcstats.mfu_ghost_hits"]
	mfu_hits = Kstat["kstat.zfs.misc.arcstats.mfu_hits"]
	mru_ghost_hits = Kstat["kstat.zfs.misc.arcstats.mru_ghost_hits"]
	mru_hits = Kstat["kstat.zfs.misc.arcstats.mru_hits"]
	prefetch_data_hits = Kstat["kstat.zfs.misc.arcstats.prefetch_data_hits"]
	prefetch_data_misses = Kstat[
	"kstat.zfs.misc.arcstats.prefetch_data_misses"
	]
	prefetch_metadata_hits = Kstat[
	"kstat.zfs.misc.arcstats.prefetch_metadata_hits"
	]
	prefetch_metadata_misses = Kstat[
	"kstat.zfs.misc.arcstats.prefetch_metadata_misses"
	]

	anon_hits = arc_hits - (
	mfu_hits + mru_hits + mfu_ghost_hits + mru_ghost_hits
	)
	arc_accesses_total = (arc_hits + arc_misses)
	demand_data_total = (demand_data_hits + demand_data_misses)
	prefetch_data_total = (prefetch_data_hits + prefetch_data_misses)
	real_hits = (mfu_hits + mru_hits)

	output["total_accesses"] = fHits(arc_accesses_total)
	output["cache_hit_ratio"] = {
	'per': fPerc(arc_hits, arc_accesses_total),
	'num': fHits(arc_hits),
	}
	output["cache_miss_ratio"] = {
	'per': fPerc(arc_misses, arc_accesses_total),
	'num': fHits(arc_misses),
	}
	output["actual_hit_ratio"] = {
	'per': fPerc(real_hits, arc_accesses_total),
	'num': fHits(real_hits),
	}
	output["data_demand_efficiency"] = {
	'per': fPerc(demand_data_hits, demand_data_total),
	'num': fHits(demand_data_total),
	}

	if prefetch_data_total > 0:
	output["data_prefetch_efficiency"] = {
	'per': fPerc(prefetch_data_hits, prefetch_data_total),
	'num': fHits(prefetch_data_total),
	}

	if anon_hits > 0:
	output["cache_hits_by_cache_list"] = {}
	output["cache_hits_by_cache_list"]["anonymously_used"] = {
	'per': fPerc(anon_hits, arc_hits),
	'num': fHits(anon_hits),
	}

	output["most_recently_used"] = {
	'per': fPerc(mru_hits, arc_hits),
	'num': fHits(mru_hits),
	}
	output["most_frequently_used"] = {
	'per': fPerc(mfu_hits, arc_hits),
	'num': fHits(mfu_hits),
	}
	output["most_recently_used_ghost"] = {
	'per': fPerc(mru_ghost_hits, arc_hits),
	'num': fHits(mru_ghost_hits),
	}
	output["most_frequently_used_ghost"] = {
	'per': fPerc(mfu_ghost_hits, arc_hits),
	'num': fHits(mfu_ghost_hits),
	}

	output["cache_hits_by_data_type"] = {}
	output["cache_hits_by_data_type"]["demand_data"] = {
	'per': fPerc(demand_data_hits, arc_hits),
	'num': fHits(demand_data_hits),
	}
	output["cache_hits_by_data_type"]["prefetch_data"] = {
	'per': fPerc(prefetch_data_hits, arc_hits),
	'num': fHits(prefetch_data_hits),
	}
	output["cache_hits_by_data_type"]["demand_metadata"] = {
	'per': fPerc(demand_metadata_hits, arc_hits),
	'num': fHits(demand_metadata_hits),
	}
	output["cache_hits_by_data_type"]["prefetch_metadata"] = {
	'per': fPerc(prefetch_metadata_hits, arc_hits),
	'num': fHits(prefetch_metadata_hits),
	}

	output["cache_misses_by_data_type"] = {}
	output["cache_misses_by_data_type"]["demand_data"] = {
	'per': fPerc(demand_data_misses, arc_misses),
	'num': fHits(demand_data_misses),
	}
	output["cache_misses_by_data_type"]["prefetch_data"] = {
	'per': fPerc(prefetch_data_misses, arc_misses),
	'num': fHits(prefetch_data_misses),
	}
	output["cache_misses_by_data_type"]["demand_metadata"] = {
	'per': fPerc(demand_metadata_misses, arc_misses),
	'num': fHits(demand_metadata_misses),
	}
	output["cache_misses_by_data_type"]["prefetch_metadata"] = {
	'per': fPerc(prefetch_metadata_misses, arc_misses),
	'num': fHits(prefetch_metadata_misses),
	}

	return output


	def _arc_efficiency(Kstat):
	"""Print information on the efficiency of the ARC"""

	arc = get_arc_efficiency(Kstat)

	sys.stdout.write("ARC Total accesses:\t\t\t\t\t%s\n" %
	arc['total_accesses'])
	sys.stdout.write("\tCache Hit Ratio:\t\t%s\t%s\n" % (
	arc['cache_hit_ratio']['per'],
	arc['cache_hit_ratio']['num'],
	)
	)
	sys.stdout.write("\tCache Miss Ratio:\t\t%s\t%s\n" % (
	arc['cache_miss_ratio']['per'],
	arc['cache_miss_ratio']['num'],
	)
	)

	sys.stdout.write("\tActual Hit Ratio:\t\t%s\t%s\n" % (
	arc['actual_hit_ratio']['per'],
	arc['actual_hit_ratio']['num'],
	)
	)

	sys.stdout.write("\n")
	sys.stdout.write("\tData Demand Efficiency:\t\t%s\t%s\n" % (
	arc['data_demand_efficiency']['per'],
	arc['data_demand_efficiency']['num'],
	)
	)

	if 'data_prefetch_efficiency' in arc:
	sys.stdout.write("\tData Prefetch Efficiency:\t%s\t%s\n" % (
	arc['data_prefetch_efficiency']['per'],
	arc['data_prefetch_efficiency']['num'],
	)
	)
	sys.stdout.write("\n")

	sys.stdout.write("\tCACHE HITS BY CACHE LIST:\n")
	if 'cache_hits_by_cache_list' in arc:
	sys.stdout.write("\t Anonymously Used:\t\t%s\t%s\n" % (
	arc['cache_hits_by_cache_list']['anonymously_used']['per'],
	arc['cache_hits_by_cache_list']['anonymously_used']['num'],
	)
	)
	sys.stdout.write("\t Most Recently Used:\t\t%s\t%s\n" % (
	arc['most_recently_used']['per'],
	arc['most_recently_used']['num'],
	)
	)
	sys.stdout.write("\t Most Frequently Used:\t\t%s\t%s\n" % (
	arc['most_frequently_used']['per'],
	arc['most_frequently_used']['num'],
	)
	)
	sys.stdout.write("\t Most Recently Used Ghost:\t%s\t%s\n" % (
	arc['most_recently_used_ghost']['per'],
	arc['most_recently_used_ghost']['num'],
	)
	)
	sys.stdout.write("\t Most Frequently Used Ghost:\t%s\t%s\n" % (
	arc['most_frequently_used_ghost']['per'],
	arc['most_frequently_used_ghost']['num'],
	)
	)

	sys.stdout.write("\n\tCACHE HITS BY DATA TYPE:\n")
	sys.stdout.write("\t Demand Data:\t\t\t%s\t%s\n" % (
	arc["cache_hits_by_data_type"]['demand_data']['per'],
	arc["cache_hits_by_data_type"]['demand_data']['num'],
	)
	)
	sys.stdout.write("\t Prefetch Data:\t\t%s\t%s\n" % (
	arc["cache_hits_by_data_type"]['prefetch_data']['per'],
	arc["cache_hits_by_data_type"]['prefetch_data']['num'],
	)
	)
	sys.stdout.write("\t Demand Metadata:\t\t%s\t%s\n" % (
	arc["cache_hits_by_data_type"]['demand_metadata']['per'],
	arc["cache_hits_by_data_type"]['demand_metadata']['num'],
	)
	)
	sys.stdout.write("\t Prefetch Metadata:\t\t%s\t%s\n" % (
	arc["cache_hits_by_data_type"]['prefetch_metadata']['per'],
	arc["cache_hits_by_data_type"]['prefetch_metadata']['num'],
	)
	)

	sys.stdout.write("\n\tCACHE MISSES BY DATA TYPE:\n")
	sys.stdout.write("\t Demand Data:\t\t\t%s\t%s\n" % (
	arc["cache_misses_by_data_type"]['demand_data']['per'],
	arc["cache_misses_by_data_type"]['demand_data']['num'],
	)
	)
	sys.stdout.write("\t Prefetch Data:\t\t%s\t%s\n" % (
	arc["cache_misses_by_data_type"]['prefetch_data']['per'],
	arc["cache_misses_by_data_type"]['prefetch_data']['num'],
	)
	)
	sys.stdout.write("\t Demand Metadata:\t\t%s\t%s\n" % (
	arc["cache_misses_by_data_type"]['demand_metadata']['per'],
	arc["cache_misses_by_data_type"]['demand_metadata']['num'],
	)
	)
	sys.stdout.write("\t Prefetch Metadata:\t\t%s\t%s\n" % (
	arc["cache_misses_by_data_type"]['prefetch_metadata']['per'],
	arc["cache_misses_by_data_type"]['prefetch_metadata']['num'],
	)
	)


	def get_l2arc_summary(Kstat):
	"""Collection information on the L2ARC"""

	output = {}

	l2_abort_lowmem = Kstat["kstat.zfs.misc.arcstats.l2_abort_lowmem"]
	l2_cksum_bad = Kstat["kstat.zfs.misc.arcstats.l2_cksum_bad"]
	l2_evict_lock_retry = Kstat["kstat.zfs.misc.arcstats.l2_evict_lock_retry"]
	l2_evict_reading = Kstat["kstat.zfs.misc.arcstats.l2_evict_reading"]
	l2_feeds = Kstat["kstat.zfs.misc.arcstats.l2_feeds"]
	l2_free_on_write = Kstat["kstat.zfs.misc.arcstats.l2_free_on_write"]
	l2_hdr_size = Kstat["kstat.zfs.misc.arcstats.l2_hdr_size"]
	l2_hits = Kstat["kstat.zfs.misc.arcstats.l2_hits"]
	l2_io_error = Kstat["kstat.zfs.misc.arcstats.l2_io_error"]
	l2_misses = Kstat["kstat.zfs.misc.arcstats.l2_misses"]
	l2_rw_clash = Kstat["kstat.zfs.misc.arcstats.l2_rw_clash"]
	l2_size = Kstat["kstat.zfs.misc.arcstats.l2_size"]
	l2_asize = Kstat["kstat.zfs.misc.arcstats.l2_asize"]
	l2_writes_done = Kstat["kstat.zfs.misc.arcstats.l2_writes_done"]
	l2_writes_error = Kstat["kstat.zfs.misc.arcstats.l2_writes_error"]
	l2_writes_sent = Kstat["kstat.zfs.misc.arcstats.l2_writes_sent"]

	l2_access_total = (l2_hits + l2_misses)
	output['l2_health_count'] = (l2_writes_error + l2_cksum_bad + l2_io_error)

	output['l2_access_total'] = l2_access_total
	output['l2_size'] = l2_size
	output['l2_asize'] = l2_asize

	if l2_size > 0 and l2_access_total > 0:

	if output['l2_health_count'] > 0:
	output["health"] = "DEGRADED"
	else:
	output["health"] = "HEALTHY"

	output["low_memory_aborts"] = fHits(l2_abort_lowmem)
	output["free_on_write"] = fHits(l2_free_on_write)
	output["rw_clashes"] = fHits(l2_rw_clash)
	output["bad_checksums"] = fHits(l2_cksum_bad)
	output["io_errors"] = fHits(l2_io_error)

	output["l2_arc_size"] = {}
	output["l2_arc_size"]["adative"] = fBytes(l2_size)
	output["l2_arc_size"]["actual"] = {
	'per': fPerc(l2_asize, l2_size),
	'num': fBytes(l2_asize)
	}
	output["l2_arc_size"]["head_size"] = {
	'per': fPerc(l2_hdr_size, l2_size),
	'num': fBytes(l2_hdr_size),
	}

	output["l2_arc_evicts"] = {}
	output["l2_arc_evicts"]['lock_retries'] = fHits(l2_evict_lock_retry)
	output["l2_arc_evicts"]['reading'] = fHits(l2_evict_reading)

	output['l2_arc_breakdown'] = {}
	output['l2_arc_breakdown']['value'] = fHits(l2_access_total)
	output['l2_arc_breakdown']['hit_ratio'] = {
	'per': fPerc(l2_hits, l2_access_total),
	'num': fHits(l2_hits),
	}
	output['l2_arc_breakdown']['miss_ratio'] = {
	'per': fPerc(l2_misses, l2_access_total),
	'num': fHits(l2_misses),
	}
	output['l2_arc_breakdown']['feeds'] = fHits(l2_feeds)

	output['l2_arc_buffer'] = {}

	output['l2_arc_writes'] = {}
	output['l2_writes_done'] = l2_writes_done
	output['l2_writes_sent'] = l2_writes_sent
	if l2_writes_done != l2_writes_sent:
	output['l2_arc_writes']['writes_sent'] = {
	'value': "FAULTED",
	'num': fHits(l2_writes_sent),
	}
	output['l2_arc_writes']['done_ratio'] = {
	'per': fPerc(l2_writes_done, l2_writes_sent),
	'num': fHits(l2_writes_done),
	}
	output['l2_arc_writes']['error_ratio'] = {
	'per': fPerc(l2_writes_error, l2_writes_sent),
	'num': fHits(l2_writes_error),
	}
	else:
	output['l2_arc_writes']['writes_sent'] = {
	'per': fPerc(100),
	'num': fHits(l2_writes_sent),
	}

	return output


	def _l2arc_summary(Kstat):
	"""Print information on the L2ARC"""

	arc = get_l2arc_summary(Kstat)

	if arc['l2_size'] > 0 and arc['l2_access_total'] > 0:
	sys.stdout.write("L2 ARC Summary: ")
	if arc['l2_health_count'] > 0:
	sys.stdout.write("(DEGRADED)\n")
	else:
	sys.stdout.write("(HEALTHY)\n")
	sys.stdout.write("\tLow Memory Aborts:\t\t\t%s\n" %
	arc['low_memory_aborts'])
	sys.stdout.write("\tFree on Write:\t\t\t\t%s\n" % arc['free_on_write'])
	sys.stdout.write("\tR/W Clashes:\t\t\t\t%s\n" % arc['rw_clashes'])
	sys.stdout.write("\tBad Checksums:\t\t\t\t%s\n" % arc['bad_checksums'])
	sys.stdout.write("\tIO Errors:\t\t\t\t%s\n" % arc['io_errors'])
	sys.stdout.write("\n")

	sys.stdout.write("L2 ARC Size: (Adaptive)\t\t\t\t%s\n" %
	arc["l2_arc_size"]["adative"])
	sys.stdout.write("\tCompressed:\t\t\t%s\t%s\n" % (
	arc["l2_arc_size"]["actual"]["per"],
	arc["l2_arc_size"]["actual"]["num"],
	)
	)
	sys.stdout.write("\tHeader Size:\t\t\t%s\t%s\n" % (
	arc["l2_arc_size"]["head_size"]["per"],
	arc["l2_arc_size"]["head_size"]["num"],
	)
	)
	sys.stdout.write("\n")

	if arc["l2_arc_evicts"]['lock_retries'] != '0' or \
	arc["l2_arc_evicts"]["reading"] != '0':
	sys.stdout.write("L2 ARC Evicts:\n")
	sys.stdout.write("\tLock Retries:\t\t\t\t%s\n" %
	arc["l2_arc_evicts"]['lock_retries'])
	sys.stdout.write("\tUpon Reading:\t\t\t\t%s\n" %
	arc["l2_arc_evicts"]["reading"])
	sys.stdout.write("\n")

	sys.stdout.write("L2 ARC Breakdown:\t\t\t\t%s\n" %
	arc['l2_arc_breakdown']['value'])
	sys.stdout.write("\tHit Ratio:\t\t\t%s\t%s\n" % (
	arc['l2_arc_breakdown']['hit_ratio']['per'],
	arc['l2_arc_breakdown']['hit_ratio']['num'],
	)
	)

	sys.stdout.write("\tMiss Ratio:\t\t\t%s\t%s\n" % (
	arc['l2_arc_breakdown']['miss_ratio']['per'],
	arc['l2_arc_breakdown']['miss_ratio']['num'],
	)
	)

	sys.stdout.write("\tFeeds:\t\t\t\t\t%s\n" %
	arc['l2_arc_breakdown']['feeds'])
	sys.stdout.write("\n")

	sys.stdout.write("L2 ARC Writes:\n")
	if arc['l2_writes_done'] != arc['l2_writes_sent']:
	sys.stdout.write("\tWrites Sent: (%s)\t\t\t\t%s\n" % (
	arc['l2_arc_writes']['writes_sent']['value'],
	arc['l2_arc_writes']['writes_sent']['num'],
	)
	)
	sys.stdout.write("\t Done Ratio:\t\t\t%s\t%s\n" % (
	arc['l2_arc_writes']['done_ratio']['per'],
	arc['l2_arc_writes']['done_ratio']['num'],
	)
	)
	sys.stdout.write("\t Error Ratio:\t\t\t%s\t%s\n" % (
	arc['l2_arc_writes']['error_ratio']['per'],
	arc['l2_arc_writes']['error_ratio']['num'],
	)
	)
	else:
	sys.stdout.write("\tWrites Sent:\t\t\t%s\t%s\n" % (
	arc['l2_arc_writes']['writes_sent']['per'],
	arc['l2_arc_writes']['writes_sent']['num'],
	)
	)


	def get_dmu_summary(Kstat):
	"""Collect information on the DMU"""

	output = {}

	zfetch_hits = Kstat["kstat.zfs.misc.zfetchstats.hits"]
	zfetch_misses = Kstat["kstat.zfs.misc.zfetchstats.misses"]

	zfetch_access_total = (zfetch_hits + zfetch_misses)
	output['zfetch_access_total'] = zfetch_access_total

	if zfetch_access_total > 0:
	output['dmu'] = {}
	output['dmu']['efficiency'] = {}
	output['dmu']['efficiency']['value'] = fHits(zfetch_access_total)
	output['dmu']['efficiency']['hit_ratio'] = {
	'per': fPerc(zfetch_hits, zfetch_access_total),
	'num': fHits(zfetch_hits),
	}
	output['dmu']['efficiency']['miss_ratio'] = {
	'per': fPerc(zfetch_misses, zfetch_access_total),
	'num': fHits(zfetch_misses),
	}

	return output


	def _dmu_summary(Kstat):
	"""Print information on the DMU"""

	arc = get_dmu_summary(Kstat)

	if arc['zfetch_access_total'] > 0:
	sys.stdout.write("DMU Prefetch Efficiency:\t\t\t\t\t%s\n" %
	arc['dmu']['efficiency']['value'])
	sys.stdout.write("\tHit Ratio:\t\t\t%s\t%s\n" % (
	arc['dmu']['efficiency']['hit_ratio']['per'],
	arc['dmu']['efficiency']['hit_ratio']['num'],
	)
	)
	sys.stdout.write("\tMiss Ratio:\t\t\t%s\t%s\n" % (
	arc['dmu']['efficiency']['miss_ratio']['per'],
	arc['dmu']['efficiency']['miss_ratio']['num'],
	)
	)

	sys.stdout.write("\n")


	def get_vdev_summary(Kstat):
	"""Collect information on the VDEVs"""

	output = {}

	vdev_cache_delegations = \
	Kstat["kstat.zfs.misc.vdev_cache_stats.delegations"]
	vdev_cache_misses = Kstat["kstat.zfs.misc.vdev_cache_stats.misses"]
	vdev_cache_hits = Kstat["kstat.zfs.misc.vdev_cache_stats.hits"]
	vdev_cache_total = (vdev_cache_misses + vdev_cache_hits +
	vdev_cache_delegations)

	output['vdev_cache_total'] = vdev_cache_total

	if vdev_cache_total > 0:
	output['summary'] = fHits(vdev_cache_total)
	output['hit_ratio'] = {
	'per': fPerc(vdev_cache_hits, vdev_cache_total),
	'num': fHits(vdev_cache_hits),
	}
	output['miss_ratio'] = {
	'per': fPerc(vdev_cache_misses, vdev_cache_total),
	'num': fHits(vdev_cache_misses),
	}
	output['delegations'] = {
	'per': fPerc(vdev_cache_delegations, vdev_cache_total),
	'num': fHits(vdev_cache_delegations),
	}

	return output


	def _vdev_summary(Kstat):
	"""Print information on the VDEVs"""

	arc = get_vdev_summary(Kstat)

	if arc['vdev_cache_total'] > 0:
	sys.stdout.write("VDEV Cache Summary:\t\t\t\t%s\n" % arc['summary'])
	sys.stdout.write("\tHit Ratio:\t\t\t%s\t%s\n" % (
	arc['hit_ratio']['per'],
	arc['hit_ratio']['num'],
	))
	sys.stdout.write("\tMiss Ratio:\t\t\t%s\t%s\n" % (
	arc['miss_ratio']['per'],
	arc['miss_ratio']['num'],
	))
	sys.stdout.write("\tDelegations:\t\t\t%s\t%s\n" % (
	arc['delegations']['per'],
	arc['delegations']['num'],
	))


	def _tunable_summary(Kstat):
	"""Print information on tunables, including descriptions if requested"""

	global show_tunable_descriptions
	global alternate_tunable_layout

	tunables = load_tunables()
	descriptions = {}

	if show_tunable_descriptions:

	command = ["/sbin/modinfo", "zfs", "-0"]

	try:
	p = Popen(command, stdin=PIPE, stdout=PIPE,
	stderr=PIPE, shell=False, close_fds=True)
	p.wait()

	# By default, Python 2 returns a string as the first element of the
	# tuple from p.communicate(), while Python 3 returns bytes which
	# must be decoded first. The better way to do this would be with
	# subprocess.run() or at least .check_output(), but this fails on
	# CentOS 6 because of its old version of Python 2
	desc = bytes.decode(p.communicate()[0])
	description_list = desc.strip().split('\0')

	if p.returncode == 0:
	for tunable in description_list:
	if tunable[0:5] == 'parm:':
	tunable = tunable[5:].strip()
	name, description = tunable.split(':', 1)
	if not description:
	description = "Description unavailable"
	descriptions[name] = description
	else:
	sys.stderr.write("%s: '%s' exited with code %i\n" %
	(sys.argv[0], command[0], p.returncode))
	sys.stderr.write("Tunable descriptions will be disabled.\n")
	except OSError as e:
	sys.stderr.write("%s: Cannot run '%s': %s\n" %
	(sys.argv[0], command[0], e.strerror))
	sys.stderr.write("Tunable descriptions will be disabled.\n")

	sys.stdout.write("ZFS Tunables:\n")

	if alternate_tunable_layout:
	fmt = "\t%s=%s\n"
	else:
	fmt = "\t%-50s%s\n"

	for name in sorted(tunables.keys()):
	if show_tunable_descriptions and name in descriptions:
	sys.stdout.write("\t# %s\n" % descriptions[name])

	sys.stdout.write(fmt % (name, tunables[name]))


	unSub = [
	_arc_summary,
	_arc_efficiency,
	_l2arc_summary,
	_dmu_summary,
	_vdev_summary,
	_tunable_summary
	]


	def zfs_header():
	"""Print title string with date"""

	daydate = time.strftime('%a %b %d %H:%M:%S %Y')

	sys.stdout.write('\n'+'-'*72+'\n')
	sys.stdout.write('ZFS Subsystem Report\t\t\t\t%s' % daydate)
	sys.stdout.write('\n')


	def usage():
	"""Print usage information"""

	sys.stdout.write("Usage: arc_summary [-h] [-a] [-d] [-p PAGE]\n\n")
	sys.stdout.write("\t -h, --help : "
	"Print this help message and exit\n")
	sys.stdout.write("\t -a, --alternate : "
	"Show an alternate sysctl layout\n")
	sys.stdout.write("\t -d, --description : "
	"Show the sysctl descriptions\n")
	sys.stdout.write("\t -p PAGE, --page=PAGE : "
	"Select a single output page to display,\n")
	sys.stdout.write("\t "
	"should be an integer between 1 and " +
	str(len(unSub)) + "\n\n")
	sys.stdout.write("Examples:\n")
	sys.stdout.write("\tarc_summary -a\n")
	sys.stdout.write("\tarc_summary -p 4\n")
	sys.stdout.write("\tarc_summary -ad\n")
	sys.stdout.write("\tarc_summary --page=2\n")


	def main():
	"""Main function"""

	global show_tunable_descriptions
	global alternate_tunable_layout

	try:
	opts, args = getopt.getopt(
	sys.argv[1:],
	"adp:h", ["alternate", "description", "page=", "help"]
	)
	except getopt.error as e:
	sys.stderr.write("Error: %s\n" % e.msg)
	usage()
	sys.exit(1)

	args = {}
	for opt, arg in opts:
	if opt in ('-a', '--alternate'):
	args['a'] = True
	if opt in ('-d', '--description'):
	args['d'] = True
	if opt in ('-p', '--page'):
	args['p'] = arg
	if opt in ('-h', '--help'):
	usage()
	sys.exit(0)

	Kstat = get_Kstat()

	alternate_tunable_layout = 'a' in args
	show_tunable_descriptions = 'd' in args

	pages = []

	if 'p' in args:
	try:
	pages.append(unSub[int(args['p']) - 1])
	except IndexError:
	sys.stderr.write('the argument to -p must be between 1 and ' +
	str(len(unSub)) + '\n')
	sys.exit(1)
	else:
	pages = unSub

	zfs_header()
	for page in pages:
	page(Kstat)
	sys.stdout.write("\n")


	if __name__ == '__main__':
	main()
	Index: vendor-sys/openzfs/dist/cmd/arc_summary/arc_summary3
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/arc_summary/arc_summary3 (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/arc_summary/arc_summary3 (revision 366774)
	@@ -1,943 +1,943 @@
	#!/usr/bin/env python3
	#
	# Copyright (c) 2008 Ben Rockwood <benr@cuddletech.com>,
	# Copyright (c) 2010 Martin Matuska <mm@FreeBSD.org>,
	# Copyright (c) 2010-2011 Jason J. Hellenthal <jhell@DataIX.net>,
	# Copyright (c) 2017 Scot W. Stevenson <scot.stevenson@gmail.com>
	# All rights reserved.
	#
	# Redistribution and use in source and binary forms, with or without
	# modification, are permitted provided that the following conditions
	# are met:
	#
	# 1. Redistributions of source code must retain the above copyright
	# notice, this list of conditions and the following disclaimer.
	# 2. Redistributions in binary form must reproduce the above copyright
	# notice, this list of conditions and the following disclaimer in the
	# documentation and/or other materials provided with the distribution.
	#
	# THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS ``AS IS'' AND
	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
	# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
	# ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
	# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
	# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
	# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
	# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
	# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
	# SUCH DAMAGE.
	"""Print statistics on the ZFS ARC Cache and other information

	Provides basic information on the ARC, its efficiency, the L2ARC (if present),
	the Data Management Unit (DMU), Virtual Devices (VDEVs), and tunables. See
	the in-source documentation and code at
	-https://github.com/zfsonlinux/zfs/blob/master/module/zfs/arc.c for details.
	+https://github.com/openzfs/zfs/blob/master/module/zfs/arc.c for details.
	The original introduction to arc_summary can be found at
	http://cuddletech.com/?p=454
	"""

	import argparse
	import os
	import subprocess
	import sys
	import time

	-DESCRIPTION = 'Print ARC and other statistics for ZFS on Linux'
	+DESCRIPTION = 'Print ARC and other statistics for OpenZFS'
	INDENT = ' '*8
	LINE_LENGTH = 72
	DATE_FORMAT = '%a %b %d %H:%M:%S %Y'
	TITLE = 'ZFS Subsystem Report'

	SECTIONS = 'arc archits dmu l2arc spl tunables vdev zil'.split()
	SECTION_HELP = 'print info from one section ('+' '.join(SECTIONS)+')'

	# Tunables and SPL are handled separately because they come from
	# different sources
	SECTION_PATHS = {'arc': 'arcstats',
	'dmu': 'dmu_tx',
	'l2arc': 'arcstats', # L2ARC stuff lives in arcstats
	'vdev': 'vdev_cache_stats',
	'xuio': 'xuio_stats',
	'zfetch': 'zfetchstats',
	'zil': 'zil'}

	parser = argparse.ArgumentParser(description=DESCRIPTION)
	parser.add_argument('-a', '--alternate', action='store_true', default=False,
	help='use alternate formatting for tunables and SPL',
	dest='alt')
	parser.add_argument('-d', '--description', action='store_true', default=False,
	help='print descriptions with tunables and SPL',
	dest='desc')
	parser.add_argument('-g', '--graph', action='store_true', default=False,
	help='print graph on ARC use and exit', dest='graph')
	parser.add_argument('-p', '--page', type=int, dest='page',
	help='print page by number (DEPRECATED, use "-s")')
	parser.add_argument('-r', '--raw', action='store_true', default=False,
	help='dump all available data with minimal formatting',
	dest='raw')
	parser.add_argument('-s', '--section', dest='section', help=SECTION_HELP)
	ARGS = parser.parse_args()


	if sys.platform.startswith('freebsd'):
	# Requires py36-sysctl on FreeBSD
	import sysctl

	VDEV_CACHE_SIZE = 'vdev.cache_size'

	def load_kstats(section):
	base = 'kstat.zfs.misc.{section}.'.format(section=section)
	# base is removed from the name
	fmt = lambda kstat: '{name} : {value}'.format(name=kstat.name[len(base):],
	value=kstat.value)
	return [fmt(kstat) for kstat in sysctl.filter(base)]

	def get_params(base):
	cut = 8 # = len('vfs.zfs.')
	return {ctl.name[cut:]: str(ctl.value) for ctl in sysctl.filter(base)}

	def get_tunable_params():
	return get_params('vfs.zfs')

	def get_vdev_params():
	return get_params('vfs.zfs.vdev')

	def get_version_impl(request):
	# FreeBSD reports versions for zpl and spa instead of zfs and spl.
	name = {'zfs': 'zpl',
	'spl': 'spa'}[request]
	mib = 'vfs.zfs.version.{}'.format(name)
	version = sysctl.filter(mib)[0].value
	return '{} version {}'.format(name, version)

	def get_descriptions(_request):
	# py-sysctl doesn't give descriptions, so we have to shell out.
	command = ['sysctl', '-d', 'vfs.zfs']

	# The recommended way to do this is with subprocess.run(). However,
	# some installed versions of Python are < 3.5, so we offer them
	# the option of doing it the old way (for now)
	if 'run' in dir(subprocess):
	info = subprocess.run(command, stdout=subprocess.PIPE,
	universal_newlines=True)
	lines = info.stdout.split('\n')
	else:
	info = subprocess.check_output(command, universal_newlines=True)
	lines = info.split('\n')

	def fmt(line):
	name, desc = line.split(':', 1)
	return (name.strip(), desc.strip())

	return dict([fmt(line) for line in lines if len(line) > 0])


	elif sys.platform.startswith('linux'):
	KSTAT_PATH = '/proc/spl/kstat/zfs'
	SPL_PATH = '/sys/module/spl/parameters'
	TUNABLES_PATH = '/sys/module/zfs/parameters'

	VDEV_CACHE_SIZE = 'zfs_vdev_cache_size'

	def load_kstats(section):
	path = os.path.join(KSTAT_PATH, section)
	with open(path) as f:
	return list(f)[2:] # Get rid of header

	def get_params(basepath):
	"""Collect information on the Solaris Porting Layer (SPL) or the
	tunables, depending on the PATH given. Does not check if PATH is
	legal.
	"""
	result = {}
	for name in os.listdir(basepath):
	path = os.path.join(basepath, name)
	with open(path) as f:
	value = f.read()
	result[name] = value.strip()
	return result

	def get_spl_params():
	return get_params(SPL_PATH)

	def get_tunable_params():
	return get_params(TUNABLES_PATH)

	def get_vdev_params():
	return get_params(TUNABLES_PATH)

	def get_version_impl(request):
	# The original arc_summary called /sbin/modinfo/{spl,zfs} to get
	# the version information. We switch to /sys/module/{spl,zfs}/version
	# to make sure we get what is really loaded in the kernel
	command = ["cat", "/sys/module/{0}/version".format(request)]
	req = request.upper()

	# The recommended way to do this is with subprocess.run(). However,
	# some installed versions of Python are < 3.5, so we offer them
	# the option of doing it the old way (for now)
	if 'run' in dir(subprocess):
	info = subprocess.run(command, stdout=subprocess.PIPE,
	universal_newlines=True)
	version = info.stdout.strip()
	else:
	info = subprocess.check_output(command, universal_newlines=True)
	version = info.strip()

	return version

	def get_descriptions(request):
	"""Get the descriptions of the Solaris Porting Layer (SPL) or the
	tunables, return with minimal formatting.
	"""

	if request not in ('spl', 'zfs'):
	print('ERROR: description of "{0}" requested)'.format(request))
	sys.exit(1)

	descs = {}
	target_prefix = 'parm:'

	# We would prefer to do this with /sys/modules -- see the discussion at
	# get_version() -- but there isn't a way to get the descriptions from
	# there, so we fall back on modinfo
	command = ["/sbin/modinfo", request, "-0"]

	# The recommended way to do this is with subprocess.run(). However,
	# some installed versions of Python are < 3.5, so we offer them
	# the option of doing it the old way (for now)
	info = ''

	try:

	if 'run' in dir(subprocess):
	info = subprocess.run(command, stdout=subprocess.PIPE,
	universal_newlines=True)
	raw_output = info.stdout.split('\0')
	else:
	info = subprocess.check_output(command,
	universal_newlines=True)
	raw_output = info.split('\0')

	except subprocess.CalledProcessError:
	print("Error: Descriptions not available",
	"(can't access kernel module)")
	sys.exit(1)

	for line in raw_output:

	if not line.startswith(target_prefix):
	continue

	line = line[len(target_prefix):].strip()
	name, raw_desc = line.split(':', 1)
	desc = raw_desc.rsplit('(', 1)[0]

	if desc == '':
	desc = '(No description found)'

	descs[name.strip()] = desc.strip()

	return descs


	def cleanup_line(single_line):
	"""Format a raw line of data from /proc and isolate the name value
	part, returning a tuple with each. Currently, this gets rid of the
	middle '4'. For example "arc_no_grow 4 0" returns the tuple
	("arc_no_grow", "0").
	"""
	name, _, value = single_line.split()

	return name, value


	def draw_graph(kstats_dict):
	"""Draw a primitive graph representing the basic information on the
	ARC -- its size and the proportion used by MFU and MRU -- and quit.
	We use max size of the ARC to calculate how full it is. This is a
	very rough representation.
	"""

	arc_stats = isolate_section('arcstats', kstats_dict)

	GRAPH_INDENT = ' '*4
	GRAPH_WIDTH = 60
	arc_size = f_bytes(arc_stats['size'])
	arc_perc = f_perc(arc_stats['size'], arc_stats['c_max'])
	mfu_size = f_bytes(arc_stats['mfu_size'])
	mru_size = f_bytes(arc_stats['mru_size'])
	meta_limit = f_bytes(arc_stats['arc_meta_limit'])
	meta_size = f_bytes(arc_stats['arc_meta_used'])
	dnode_limit = f_bytes(arc_stats['arc_dnode_limit'])
	dnode_size = f_bytes(arc_stats['dnode_size'])

	info_form = ('ARC: {0} ({1}) MFU: {2} MRU: {3} META: {4} ({5}) '
	'DNODE {6} ({7})')
	info_line = info_form.format(arc_size, arc_perc, mfu_size, mru_size,
	meta_size, meta_limit, dnode_size,
	dnode_limit)
	info_spc = ' '*int((GRAPH_WIDTH-len(info_line))/2)
	info_line = GRAPH_INDENT+info_spc+info_line

	graph_line = GRAPH_INDENT+'+'+('-'*(GRAPH_WIDTH-2))+'+'

	mfu_perc = float(int(arc_stats['mfu_size'])/int(arc_stats['c_max']))
	mru_perc = float(int(arc_stats['mru_size'])/int(arc_stats['c_max']))
	arc_perc = float(int(arc_stats['size'])/int(arc_stats['c_max']))
	total_ticks = float(arc_perc)*GRAPH_WIDTH
	mfu_ticks = mfu_perc*GRAPH_WIDTH
	mru_ticks = mru_perc*GRAPH_WIDTH
	other_ticks = total_ticks-(mfu_ticks+mru_ticks)

	core_form = 'F'int(mfu_ticks)+'R'int(mru_ticks)+'O'*int(other_ticks)
	core_spc = ' '*(GRAPH_WIDTH-(2+len(core_form)))
	core_line = GRAPH_INDENT+'\|'+core_form+core_spc+'\|'

	for line in ('', info_line, graph_line, core_line, graph_line, ''):
	print(line)


	def f_bytes(byte_string):
	"""Return human-readable representation of a byte value in
	powers of 2 (eg "KiB" for "kibibytes", etc) to two decimal
	points. Values smaller than one KiB are returned without
	decimal points. Note "bytes" is a reserved keyword.
	"""

	prefixes = ([2**80, "YiB"], # yobibytes (yotta)
	[2**70, "ZiB"], # zebibytes (zetta)
	[2**60, "EiB"], # exbibytes (exa)
	[2**50, "PiB"], # pebibytes (peta)
	[2**40, "TiB"], # tebibytes (tera)
	[2**30, "GiB"], # gibibytes (giga)
	[2**20, "MiB"], # mebibytes (mega)
	[2**10, "KiB"]) # kibibytes (kilo)

	bites = int(byte_string)

	if bites >= 2**10:
	for limit, unit in prefixes:

	if bites >= limit:
	value = bites / limit
	break

	result = '{0:.1f} {1}'.format(value, unit)
	else:
	result = '{0} Bytes'.format(bites)

	return result


	def f_hits(hits_string):
	"""Create a human-readable representation of the number of hits.
	The single-letter symbols used are SI to avoid the confusion caused
	by the different "short scale" and "long scale" representations in
	English, which use the same words for different values. See
	https://en.wikipedia.org/wiki/Names_of_large_numbers and:
	https://physics.nist.gov/cuu/Units/prefixes.html
	"""

	numbers = ([10**24, 'Y'], # yotta (septillion)
	[10**21, 'Z'], # zetta (sextillion)
	[10**18, 'E'], # exa (quintrillion)
	[10**15, 'P'], # peta (quadrillion)
	[10**12, 'T'], # tera (trillion)
	[10**9, 'G'], # giga (billion)
	[10**6, 'M'], # mega (million)
	[10**3, 'k']) # kilo (thousand)

	hits = int(hits_string)

	if hits >= 1000:
	for limit, symbol in numbers:

	if hits >= limit:
	value = hits/limit
	break

	result = "%0.1f%s" % (value, symbol)
	else:
	result = "%d" % hits

	return result


	def f_perc(value1, value2):
	"""Calculate percentage and return in human-readable form. If
	rounding produces the result '0.0' though the first number is
	not zero, include a 'less-than' symbol to avoid confusion.
	Division by zero is handled by returning 'n/a'; no error
	is called.
	"""

	v1 = float(value1)
	v2 = float(value2)

	try:
	perc = 100 * v1/v2
	except ZeroDivisionError:
	result = 'n/a'
	else:
	result = '{0:0.1f} %'.format(perc)

	if result == '0.0 %' and v1 > 0:
	result = '< 0.1 %'

	return result


	def format_raw_line(name, value):
	"""For the --raw option for the tunable and SPL outputs, decide on the
	correct formatting based on the --alternate flag.
	"""

	if ARGS.alt:
	result = '{0}{1}={2}'.format(INDENT, name, value)
	else:
	spc = LINE_LENGTH-(len(INDENT)+len(value))
	result = '{0}{1:<{spc}}{2}'.format(INDENT, name, value, spc=spc)

	return result


	def get_kstats():
	"""Collect information on the ZFS subsystem. The step does not perform any
	further processing, giving us the option to only work on what is actually
	needed. The name "kstat" is a holdover from the Solaris utility of the same
	name.
	"""

	result = {}

	for section in SECTION_PATHS.values():
	if section not in result:
	result[section] = load_kstats(section)

	return result


	def get_version(request):
	"""Get the version number of ZFS or SPL on this machine for header.
	Returns an error string, but does not raise an error, if we can't
	get the ZFS/SPL version.
	"""

	if request not in ('spl', 'zfs'):
	error_msg = '(ERROR: "{0}" requested)'.format(request)
	return error_msg

	return get_version_impl(request)


	def print_header():
	"""Print the initial heading with date and time as well as info on the
	kernel and ZFS versions. This is not called for the graph.
	"""

	# datetime is now recommended over time but we keep the exact formatting
	# from the older version of arc_summary in case there are scripts
	# that expect it in this way
	daydate = time.strftime(DATE_FORMAT)
	spc_date = LINE_LENGTH-len(daydate)
	sys_version = os.uname()

	sys_msg = sys_version.sysname+' '+sys_version.release
	zfs = get_version('zfs')
	spc_zfs = LINE_LENGTH-len(zfs)

	machine_msg = 'Machine: '+sys_version.nodename+' ('+sys_version.machine+')'
	spl = get_version('spl')
	spc_spl = LINE_LENGTH-len(spl)

	print('\n'+('-'*LINE_LENGTH))
	print('{0:<{spc}}{1}'.format(TITLE, daydate, spc=spc_date))
	print('{0:<{spc}}{1}'.format(sys_msg, zfs, spc=spc_zfs))
	print('{0:<{spc}}{1}\n'.format(machine_msg, spl, spc=spc_spl))


	def print_raw(kstats_dict):
	"""Print all available data from the system in a minimally sorted format.
	This can be used as a source to be piped through 'grep'.
	"""

	sections = sorted(kstats_dict.keys())

	for section in sections:

	print('\n{0}:'.format(section.upper()))
	lines = sorted(kstats_dict[section])

	for line in lines:
	name, value = cleanup_line(line)
	print(format_raw_line(name, value))

	# Tunables and SPL must be handled separately because they come from a
	# different source and have descriptions the user might request
	print()
	section_spl()
	section_tunables()


	def isolate_section(section_name, kstats_dict):
	"""From the complete information on all sections, retrieve only those
	for one section.
	"""

	try:
	section_data = kstats_dict[section_name]
	except KeyError:
	print('ERROR: Data on {0} not available'.format(section_data))
	sys.exit(1)

	section_dict = dict(cleanup_line(l) for l in section_data)

	return section_dict


	# Formatted output helper functions


	def prt_1(text, value):
	"""Print text and one value, no indent"""
	spc = ' '*(LINE_LENGTH-(len(text)+len(value)))
	print('{0}{spc}{1}'.format(text, value, spc=spc))


	def prt_i1(text, value):
	"""Print text and one value, with indent"""
	spc = ' '*(LINE_LENGTH-(len(INDENT)+len(text)+len(value)))
	print(INDENT+'{0}{spc}{1}'.format(text, value, spc=spc))


	def prt_2(text, value1, value2):
	"""Print text and two values, no indent"""
	values = '{0:>9} {1:>9}'.format(value1, value2)
	spc = ' '*(LINE_LENGTH-(len(text)+len(values)+2))
	print('{0}{spc} {1}'.format(text, values, spc=spc))


	def prt_i2(text, value1, value2):
	"""Print text and two values, with indent"""
	values = '{0:>9} {1:>9}'.format(value1, value2)
	spc = ' '*(LINE_LENGTH-(len(INDENT)+len(text)+len(values)+2))
	print(INDENT+'{0}{spc} {1}'.format(text, values, spc=spc))


	# The section output concentrates on important parameters instead of
	# being exhaustive (that is what the --raw parameter is for)


	def section_arc(kstats_dict):
	"""Give basic information on the ARC, MRU and MFU. This is the first
	and most used section.
	"""

	arc_stats = isolate_section('arcstats', kstats_dict)

	throttle = arc_stats['memory_throttle_count']

	if throttle == '0':
	health = 'HEALTHY'
	else:
	health = 'THROTTLED'

	prt_1('ARC status:', health)
	prt_i1('Memory throttle count:', throttle)
	print()

	arc_size = arc_stats['size']
	arc_target_size = arc_stats['c']
	arc_max = arc_stats['c_max']
	arc_min = arc_stats['c_min']
	mfu_size = arc_stats['mfu_size']
	mru_size = arc_stats['mru_size']
	meta_limit = arc_stats['arc_meta_limit']
	meta_size = arc_stats['arc_meta_used']
	dnode_limit = arc_stats['arc_dnode_limit']
	dnode_size = arc_stats['dnode_size']
	target_size_ratio = '{0}:1'.format(int(arc_max) // int(arc_min))

	prt_2('ARC size (current):',
	f_perc(arc_size, arc_max), f_bytes(arc_size))
	prt_i2('Target size (adaptive):',
	f_perc(arc_target_size, arc_max), f_bytes(arc_target_size))
	prt_i2('Min size (hard limit):',
	f_perc(arc_min, arc_max), f_bytes(arc_min))
	prt_i2('Max size (high water):',
	target_size_ratio, f_bytes(arc_max))
	caches_size = int(mfu_size)+int(mru_size)
	prt_i2('Most Frequently Used (MFU) cache size:',
	f_perc(mfu_size, caches_size), f_bytes(mfu_size))
	prt_i2('Most Recently Used (MRU) cache size:',
	f_perc(mru_size, caches_size), f_bytes(mru_size))
	prt_i2('Metadata cache size (hard limit):',
	f_perc(meta_limit, arc_max), f_bytes(meta_limit))
	prt_i2('Metadata cache size (current):',
	f_perc(meta_size, meta_limit), f_bytes(meta_size))
	prt_i2('Dnode cache size (hard limit):',
	f_perc(dnode_limit, meta_limit), f_bytes(dnode_limit))
	prt_i2('Dnode cache size (current):',
	f_perc(dnode_size, dnode_limit), f_bytes(dnode_size))
	print()

	print('ARC hash breakdown:')
	prt_i1('Elements max:', f_hits(arc_stats['hash_elements_max']))
	prt_i2('Elements current:',
	f_perc(arc_stats['hash_elements'], arc_stats['hash_elements_max']),
	f_hits(arc_stats['hash_elements']))
	prt_i1('Collisions:', f_hits(arc_stats['hash_collisions']))

	prt_i1('Chain max:', f_hits(arc_stats['hash_chain_max']))
	prt_i1('Chains:', f_hits(arc_stats['hash_chains']))
	print()

	print('ARC misc:')
	prt_i1('Deleted:', f_hits(arc_stats['deleted']))
	prt_i1('Mutex misses:', f_hits(arc_stats['mutex_miss']))
	prt_i1('Eviction skips:', f_hits(arc_stats['evict_skip']))
	print()


	def section_archits(kstats_dict):
	"""Print information on how the caches are accessed ("arc hits").
	"""

	arc_stats = isolate_section('arcstats', kstats_dict)
	all_accesses = int(arc_stats['hits'])+int(arc_stats['misses'])
	actual_hits = int(arc_stats['mfu_hits'])+int(arc_stats['mru_hits'])

	prt_1('ARC total accesses (hits + misses):', f_hits(all_accesses))
	ta_todo = (('Cache hit ratio:', arc_stats['hits']),
	('Cache miss ratio:', arc_stats['misses']),
	('Actual hit ratio (MFU + MRU hits):', actual_hits))

	for title, value in ta_todo:
	prt_i2(title, f_perc(value, all_accesses), f_hits(value))

	dd_total = int(arc_stats['demand_data_hits']) +\
	int(arc_stats['demand_data_misses'])
	prt_i2('Data demand efficiency:',
	f_perc(arc_stats['demand_data_hits'], dd_total),
	f_hits(dd_total))

	dp_total = int(arc_stats['prefetch_data_hits']) +\
	int(arc_stats['prefetch_data_misses'])
	prt_i2('Data prefetch efficiency:',
	f_perc(arc_stats['prefetch_data_hits'], dp_total),
	f_hits(dp_total))

	known_hits = int(arc_stats['mfu_hits']) +\
	int(arc_stats['mru_hits']) +\
	int(arc_stats['mfu_ghost_hits']) +\
	int(arc_stats['mru_ghost_hits'])

	anon_hits = int(arc_stats['hits'])-known_hits

	print()
	print('Cache hits by cache type:')
	cl_todo = (('Most frequently used (MFU):', arc_stats['mfu_hits']),
	('Most recently used (MRU):', arc_stats['mru_hits']),
	('Most frequently used (MFU) ghost:',
	arc_stats['mfu_ghost_hits']),
	('Most recently used (MRU) ghost:',
	arc_stats['mru_ghost_hits']))

	for title, value in cl_todo:
	prt_i2(title, f_perc(value, arc_stats['hits']), f_hits(value))

	# For some reason, anon_hits can turn negative, which is weird. Until we
	# have figured out why this happens, we just hide the problem, following
	# the behavior of the original arc_summary.
	if anon_hits >= 0:
	prt_i2('Anonymously used:',
	f_perc(anon_hits, arc_stats['hits']), f_hits(anon_hits))

	print()
	print('Cache hits by data type:')
	dt_todo = (('Demand data:', arc_stats['demand_data_hits']),
	('Demand prefetch data:', arc_stats['prefetch_data_hits']),
	('Demand metadata:', arc_stats['demand_metadata_hits']),
	('Demand prefetch metadata:',
	arc_stats['prefetch_metadata_hits']))

	for title, value in dt_todo:
	prt_i2(title, f_perc(value, arc_stats['hits']), f_hits(value))

	print()
	print('Cache misses by data type:')
	dm_todo = (('Demand data:', arc_stats['demand_data_misses']),
	('Demand prefetch data:',
	arc_stats['prefetch_data_misses']),
	('Demand metadata:', arc_stats['demand_metadata_misses']),
	('Demand prefetch metadata:',
	arc_stats['prefetch_metadata_misses']))

	for title, value in dm_todo:
	prt_i2(title, f_perc(value, arc_stats['misses']), f_hits(value))

	print()


	def section_dmu(kstats_dict):
	"""Collect information on the DMU"""

	zfetch_stats = isolate_section('zfetchstats', kstats_dict)

	zfetch_access_total = int(zfetch_stats['hits'])+int(zfetch_stats['misses'])

	prt_1('DMU prefetch efficiency:', f_hits(zfetch_access_total))
	prt_i2('Hit ratio:', f_perc(zfetch_stats['hits'], zfetch_access_total),
	f_hits(zfetch_stats['hits']))
	prt_i2('Miss ratio:', f_perc(zfetch_stats['misses'], zfetch_access_total),
	f_hits(zfetch_stats['misses']))
	print()


	def section_l2arc(kstats_dict):
	"""Collect information on L2ARC device if present. If not, tell user
	that we're skipping the section.
	"""

	# The L2ARC statistics live in the same section as the normal ARC stuff
	arc_stats = isolate_section('arcstats', kstats_dict)

	if arc_stats['l2_size'] == '0':
	print('L2ARC not detected, skipping section\n')
	return

	l2_errors = int(arc_stats['l2_writes_error']) +\
	int(arc_stats['l2_cksum_bad']) +\
	int(arc_stats['l2_io_error'])

	l2_access_total = int(arc_stats['l2_hits'])+int(arc_stats['l2_misses'])
	health = 'HEALTHY'

	if l2_errors > 0:
	health = 'DEGRADED'

	prt_1('L2ARC status:', health)

	l2_todo = (('Low memory aborts:', 'l2_abort_lowmem'),
	('Free on write:', 'l2_free_on_write'),
	('R/W clashes:', 'l2_rw_clash'),
	('Bad checksums:', 'l2_cksum_bad'),
	('I/O errors:', 'l2_io_error'))

	for title, value in l2_todo:
	prt_i1(title, f_hits(arc_stats[value]))

	print()
	prt_1('L2ARC size (adaptive):', f_bytes(arc_stats['l2_size']))
	prt_i2('Compressed:', f_perc(arc_stats['l2_asize'], arc_stats['l2_size']),
	f_bytes(arc_stats['l2_asize']))
	prt_i2('Header size:',
	f_perc(arc_stats['l2_hdr_size'], arc_stats['l2_size']),
	f_bytes(arc_stats['l2_hdr_size']))

	print()
	prt_1('L2ARC breakdown:', f_hits(l2_access_total))
	prt_i2('Hit ratio:',
	f_perc(arc_stats['l2_hits'], l2_access_total),
	f_hits(arc_stats['l2_hits']))
	prt_i2('Miss ratio:',
	f_perc(arc_stats['l2_misses'], l2_access_total),
	f_hits(arc_stats['l2_misses']))
	prt_i1('Feeds:', f_hits(arc_stats['l2_feeds']))

	print()
	print('L2ARC writes:')

	if arc_stats['l2_writes_done'] != arc_stats['l2_writes_sent']:
	prt_i2('Writes sent:', 'FAULTED', f_hits(arc_stats['l2_writes_sent']))
	prt_i2('Done ratio:',
	f_perc(arc_stats['l2_writes_done'],
	arc_stats['l2_writes_sent']),
	f_hits(arc_stats['l2_writes_done']))
	prt_i2('Error ratio:',
	f_perc(arc_stats['l2_writes_error'],
	arc_stats['l2_writes_sent']),
	f_hits(arc_stats['l2_writes_error']))
	else:
	prt_i2('Writes sent:', '100 %', f_hits(arc_stats['l2_writes_sent']))

	print()
	print('L2ARC evicts:')
	prt_i1('Lock retries:', f_hits(arc_stats['l2_evict_lock_retry']))
	prt_i1('Upon reading:', f_hits(arc_stats['l2_evict_reading']))
	print()


	def section_spl(*_):
	"""Print the SPL parameters, if requested with alternative format
	and/or descriptions. This does not use kstats.
	"""

	if sys.platform.startswith('freebsd'):
	# No SPL support in FreeBSD
	return

	spls = get_spl_params()
	keylist = sorted(spls.keys())
	print('Solaris Porting Layer (SPL):')

	if ARGS.desc:
	descriptions = get_descriptions('spl')

	for key in keylist:
	value = spls[key]

	if ARGS.desc:
	try:
	print(INDENT+'#', descriptions[key])
	except KeyError:
	print(INDENT+'# (No description found)') # paranoid

	print(format_raw_line(key, value))

	print()


	def section_tunables(*_):
	"""Print the tunables, if requested with alternative format and/or
	descriptions. This does not use kstasts.
	"""

	tunables = get_tunable_params()
	keylist = sorted(tunables.keys())
	print('Tunables:')

	if ARGS.desc:
	descriptions = get_descriptions('zfs')

	for key in keylist:
	value = tunables[key]

	if ARGS.desc:
	try:
	print(INDENT+'#', descriptions[key])
	except KeyError:
	print(INDENT+'# (No description found)') # paranoid

	print(format_raw_line(key, value))

	print()


	def section_vdev(kstats_dict):
	"""Collect information on VDEV caches"""

	# Currently [Nov 2017] the VDEV cache is disabled, because it is actually
	# harmful. When this is the case, we just skip the whole entry. See
	- # https://github.com/zfsonlinux/zfs/blob/master/module/zfs/vdev_cache.c
	+ # https://github.com/openzfs/zfs/blob/master/module/zfs/vdev_cache.c
	# for details
	tunables = get_vdev_params()

	if tunables[VDEV_CACHE_SIZE] == '0':
	print('VDEV cache disabled, skipping section\n')
	return

	vdev_stats = isolate_section('vdev_cache_stats', kstats_dict)

	vdev_cache_total = int(vdev_stats['hits']) +\
	int(vdev_stats['misses']) +\
	int(vdev_stats['delegations'])

	prt_1('VDEV cache summary:', f_hits(vdev_cache_total))
	prt_i2('Hit ratio:', f_perc(vdev_stats['hits'], vdev_cache_total),
	f_hits(vdev_stats['hits']))
	prt_i2('Miss ratio:', f_perc(vdev_stats['misses'], vdev_cache_total),
	f_hits(vdev_stats['misses']))
	prt_i2('Delegations:', f_perc(vdev_stats['delegations'], vdev_cache_total),
	f_hits(vdev_stats['delegations']))
	print()


	def section_zil(kstats_dict):
	"""Collect information on the ZFS Intent Log. Some of the information
	- taken from https://github.com/zfsonlinux/zfs/blob/master/include/sys/zil.h
	+ taken from https://github.com/openzfs/zfs/blob/master/include/sys/zil.h
	"""

	zil_stats = isolate_section('zil', kstats_dict)

	prt_1('ZIL committed transactions:',
	f_hits(zil_stats['zil_itx_count']))
	prt_i1('Commit requests:', f_hits(zil_stats['zil_commit_count']))
	prt_i1('Flushes to stable storage:',
	f_hits(zil_stats['zil_commit_writer_count']))
	prt_i2('Transactions to SLOG storage pool:',
	f_bytes(zil_stats['zil_itx_metaslab_slog_bytes']),
	f_hits(zil_stats['zil_itx_metaslab_slog_count']))
	prt_i2('Transactions to non-SLOG storage pool:',
	f_bytes(zil_stats['zil_itx_metaslab_normal_bytes']),
	f_hits(zil_stats['zil_itx_metaslab_normal_count']))
	print()


	section_calls = {'arc': section_arc,
	'archits': section_archits,
	'dmu': section_dmu,
	'l2arc': section_l2arc,
	'spl': section_spl,
	'tunables': section_tunables,
	'vdev': section_vdev,
	'zil': section_zil}


	def main():
	"""Run program. The options to draw a graph and to print all data raw are
	treated separately because they come with their own call.
	"""

	kstats = get_kstats()

	if ARGS.graph:
	draw_graph(kstats)
	sys.exit(0)

	print_header()

	if ARGS.raw:
	print_raw(kstats)

	elif ARGS.section:

	try:
	section_calls[ARGS.section](kstats)
	except KeyError:
	print('Error: Section "{0}" unknown'.format(ARGS.section))
	sys.exit(1)

	elif ARGS.page:
	print('WARNING: Pages are deprecated, please use "--section"\n')

	pages_to_calls = {1: 'arc',
	2: 'archits',
	3: 'l2arc',
	4: 'dmu',
	5: 'vdev',
	6: 'tunables'}

	try:
	call = pages_to_calls[ARGS.page]
	except KeyError:
	print('Error: Page "{0}" not supported'.format(ARGS.page))
	sys.exit(1)
	else:
	section_calls[call](kstats)

	else:
	# If no parameters were given, we print all sections. We might want to
	# change the sequence by hand
	calls = sorted(section_calls.keys())

	for section in calls:
	section_calls[section](kstats)

	sys.exit(0)


	if __name__ == '__main__':
	main()
	Index: vendor-sys/openzfs/dist/cmd/dbufstat/dbufstat.in
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/dbufstat/dbufstat.in (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/dbufstat/dbufstat.in (revision 366774)
	@@ -1,669 +1,684 @@
	#!/usr/bin/env @PYTHON_SHEBANG@
	#
	# Print out statistics for all cached dmu buffers. This information
	# is available through the dbufs kstat and may be post-processed as
	# needed by the script.
	#
	# CDDL HEADER START
	#
	# The contents of this file are subject to the terms of the
	# Common Development and Distribution License, Version 1.0 only
	# (the "License"). You may not use this file except in compliance
	# with the License.
	#
	# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	# or http://www.opensolaris.org/os/licensing.
	# See the License for the specific language governing permissions
	# and limitations under the License.
	#
	# When distributing Covered Code, include this CDDL HEADER in each
	# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	# If applicable, add the following below this CDDL HEADER, with the
	# fields enclosed by brackets "[]" replaced with your own identifying
	# information: Portions Copyright [yyyy] [name of copyright owner]
	#
	# CDDL HEADER END
	#
	# Copyright (C) 2013 Lawrence Livermore National Security, LLC.
	# Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	#
	# This script must remain compatible with Python 2.6+ and Python 3.4+.
	#

	import sys
	import getopt
	import errno
	import re

	bhdr = ["pool", "objset", "object", "level", "blkid", "offset", "dbsize"]
	bxhdr = ["pool", "objset", "object", "level", "blkid", "offset", "dbsize",
	"meta", "state", "dbholds", "dbc", "list", "atype", "flags",
	"count", "asize", "access", "mru", "gmru", "mfu", "gmfu", "l2",
	"l2_dattr", "l2_asize", "l2_comp", "aholds", "dtype", "btype",
	"data_bs", "meta_bs", "bsize", "lvls", "dholds", "blocks", "dsize"]
	bincompat = ["cached", "direct", "indirect", "bonus", "spill"]

	dhdr = ["pool", "objset", "object", "dtype", "cached"]
	dxhdr = ["pool", "objset", "object", "dtype", "btype", "data_bs", "meta_bs",
	"bsize", "lvls", "dholds", "blocks", "dsize", "cached", "direct",
	"indirect", "bonus", "spill"]
	dincompat = ["level", "blkid", "offset", "dbsize", "meta", "state", "dbholds",
	"dbc", "list", "atype", "flags", "count", "asize", "access",
	"mru", "gmru", "mfu", "gmfu", "l2", "l2_dattr", "l2_asize",
	"l2_comp", "aholds"]

	thdr = ["pool", "objset", "dtype", "cached"]
	txhdr = ["pool", "objset", "dtype", "cached", "direct", "indirect",
	"bonus", "spill"]
	tincompat = ["object", "level", "blkid", "offset", "dbsize", "meta", "state",
	"dbc", "dbholds", "list", "atype", "flags", "count", "asize",
	"access", "mru", "gmru", "mfu", "gmfu", "l2", "l2_dattr",
	"l2_asize", "l2_comp", "aholds", "btype", "data_bs", "meta_bs",
	"bsize", "lvls", "dholds", "blocks", "dsize"]

	cols = {
	# hdr: [size, scale, description]
	"pool": [15, -1, "pool name"],
	"objset": [6, -1, "dataset identification number"],
	"object": [10, -1, "object number"],
	"level": [5, -1, "indirection level of buffer"],
	"blkid": [8, -1, "block number of buffer"],
	"offset": [12, 1024, "offset in object of buffer"],
	"dbsize": [7, 1024, "size of buffer"],
	"meta": [4, -1, "is this buffer metadata?"],
	"state": [5, -1, "state of buffer (read, cached, etc)"],
	"dbholds": [7, 1000, "number of holds on buffer"],
	"dbc": [3, -1, "in dbuf cache"],
	"list": [4, -1, "which ARC list contains this buffer"],
	"atype": [7, -1, "ARC header type (data or metadata)"],
	"flags": [9, -1, "ARC read flags"],
	"count": [5, -1, "ARC data count"],
	"asize": [7, 1024, "size of this ARC buffer"],
	"access": [10, -1, "time this ARC buffer was last accessed"],
	"mru": [5, 1000, "hits while on the ARC's MRU list"],
	"gmru": [5, 1000, "hits while on the ARC's MRU ghost list"],
	"mfu": [5, 1000, "hits while on the ARC's MFU list"],
	"gmfu": [5, 1000, "hits while on the ARC's MFU ghost list"],
	"l2": [5, 1000, "hits while on the L2ARC"],
	"l2_dattr": [8, -1, "L2ARC disk address/offset"],
	"l2_asize": [8, 1024, "L2ARC alloc'd size (depending on compression)"],
	"l2_comp": [21, -1, "L2ARC compression algorithm for buffer"],
	"aholds": [6, 1000, "number of holds on this ARC buffer"],
	"dtype": [27, -1, "dnode type"],
	"btype": [27, -1, "bonus buffer type"],
	"data_bs": [7, 1024, "data block size"],
	"meta_bs": [7, 1024, "metadata block size"],
	"bsize": [6, 1024, "bonus buffer size"],
	"lvls": [6, -1, "number of indirection levels"],
	"dholds": [6, 1000, "number of holds on dnode"],
	"blocks": [8, 1000, "number of allocated blocks"],
	"dsize": [12, 1024, "size of dnode"],
	"cached": [6, 1024, "bytes cached for all blocks"],
	"direct": [6, 1024, "bytes cached for direct blocks"],
	"indirect": [8, 1024, "bytes cached for indirect blocks"],
	"bonus": [5, 1024, "bytes cached for bonus buffer"],
	"spill": [5, 1024, "bytes cached for spill block"],
	}

	hdr = None
	xhdr = None
	sep = " " # Default separator is 2 spaces
	cmd = ("Usage: dbufstat [-bdhnrtvx] [-i file] [-f fields] [-o file] "
	"[-s string] [-F filter]\n")
	raw = 0


	+if sys.platform.startswith("freebsd"):
	+ import io
	+ # Requires py-sysctl on FreeBSD
	+ import sysctl
	+
	+ def default_ifile():
	+ dbufs = sysctl.filter("kstat.zfs.misc.dbufs")[0].value
	+ sys.stdin = io.StringIO(dbufs)
	+ return "-"
	+
	+elif sys.platform.startswith("linux"):
	+ def default_ifile():
	+ return "/proc/spl/kstat/zfs/dbufs"
	+
	+
	def print_incompat_helper(incompat):
	cnt = 0
	for key in sorted(incompat):
	if cnt is 0:
	sys.stderr.write("\t")
	elif cnt > 8:
	sys.stderr.write(",\n\t")
	cnt = 0
	else:
	sys.stderr.write(", ")

	sys.stderr.write("%s" % key)
	cnt += 1

	sys.stderr.write("\n\n")


	def detailed_usage():
	sys.stderr.write("%s\n" % cmd)

	sys.stderr.write("Field definitions incompatible with '-b' option:\n")
	print_incompat_helper(bincompat)

	sys.stderr.write("Field definitions incompatible with '-d' option:\n")
	print_incompat_helper(dincompat)

	sys.stderr.write("Field definitions incompatible with '-t' option:\n")
	print_incompat_helper(tincompat)

	sys.stderr.write("Field definitions are as follows:\n")
	for key in sorted(cols.keys()):
	sys.stderr.write("%11s : %s\n" % (key, cols[key][2]))
	sys.stderr.write("\n")

	sys.exit(0)


	def usage():
	sys.stderr.write("%s\n" % cmd)
	sys.stderr.write("\t -b : Print table of information for each dbuf\n")
	sys.stderr.write("\t -d : Print table of information for each dnode\n")
	sys.stderr.write("\t -h : Print this help message\n")
	sys.stderr.write("\t -n : Exclude header from output\n")
	sys.stderr.write("\t -r : Print raw values\n")
	sys.stderr.write("\t -t : Print table of information for each dnode type"
	"\n")
	sys.stderr.write("\t -v : List all possible field headers and definitions"
	"\n")
	sys.stderr.write("\t -x : Print extended stats\n")
	sys.stderr.write("\t -i : Redirect input from the specified file\n")
	sys.stderr.write("\t -f : Specify specific fields to print (see -v)\n")
	sys.stderr.write("\t -o : Redirect output to the specified file\n")
	sys.stderr.write("\t -s : Override default field separator with custom "
	"character or string\n")
	sys.stderr.write("\t -F : Filter output by value or regex\n")
	sys.stderr.write("\nExamples:\n")
	sys.stderr.write("\tdbufstat -d -o /tmp/d.log\n")
	sys.stderr.write("\tdbufstat -t -s \",\" -o /tmp/t.log\n")
	sys.stderr.write("\tdbufstat -v\n")
	sys.stderr.write("\tdbufstat -d -f pool,object,objset,dsize,cached\n")
	sys.stderr.write("\tdbufstat -bx -F dbc=1,objset=54,pool=testpool\n")
	sys.stderr.write("\n")

	sys.exit(1)


	def prettynum(sz, scale, num=0):
	global raw

	suffix = [' ', 'K', 'M', 'G', 'T', 'P', 'E', 'Z']
	index = 0
	save = 0

	if raw or scale == -1:
	return "%*s" % (sz, num)

	# Rounding error, return 0
	elif 0 < num < 1:
	num = 0

	while num > scale and index < 5:
	save = num
	num = num / scale
	index += 1

	if index == 0:
	return "%*d" % (sz, num)

	if (save / scale) < 10:
	return "%*.1f%s" % (sz - 1, num, suffix[index])
	else:
	return "%*d%s" % (sz - 1, num, suffix[index])


	def print_values(v):
	global hdr
	global sep

	try:
	for col in hdr:
	sys.stdout.write("%s%s" % (
	prettynum(cols[col][0], cols[col][1], v[col]), sep))
	sys.stdout.write("\n")
	except IOError as e:
	if e.errno == errno.EPIPE:
	sys.exit(1)


	def print_header():
	global hdr
	global sep

	try:
	for col in hdr:
	sys.stdout.write("%*s%s" % (cols[col][0], col, sep))
	sys.stdout.write("\n")
	except IOError as e:
	if e.errno == errno.EPIPE:
	sys.exit(1)


	def get_typestring(t):
	ot_strings = [
	"DMU_OT_NONE",
	# general:
	"DMU_OT_OBJECT_DIRECTORY",
	"DMU_OT_OBJECT_ARRAY",
	"DMU_OT_PACKED_NVLIST",
	"DMU_OT_PACKED_NVLIST_SIZE",
	"DMU_OT_BPOBJ",
	"DMU_OT_BPOBJ_HDR",
	# spa:
	"DMU_OT_SPACE_MAP_HEADER",
	"DMU_OT_SPACE_MAP",
	# zil:
	"DMU_OT_INTENT_LOG",
	# dmu:
	"DMU_OT_DNODE",
	"DMU_OT_OBJSET",
	# dsl:
	"DMU_OT_DSL_DIR",
	"DMU_OT_DSL_DIR_CHILD_MAP",
	"DMU_OT_DSL_DS_SNAP_MAP",
	"DMU_OT_DSL_PROPS",
	"DMU_OT_DSL_DATASET",
	# zpl:
	"DMU_OT_ZNODE",
	"DMU_OT_OLDACL",
	"DMU_OT_PLAIN_FILE_CONTENTS",
	"DMU_OT_DIRECTORY_CONTENTS",
	"DMU_OT_MASTER_NODE",
	"DMU_OT_UNLINKED_SET",
	# zvol:
	"DMU_OT_ZVOL",
	"DMU_OT_ZVOL_PROP",
	# other; for testing only!
	"DMU_OT_PLAIN_OTHER",
	"DMU_OT_UINT64_OTHER",
	"DMU_OT_ZAP_OTHER",
	# new object types:
	"DMU_OT_ERROR_LOG",
	"DMU_OT_SPA_HISTORY",
	"DMU_OT_SPA_HISTORY_OFFSETS",
	"DMU_OT_POOL_PROPS",
	"DMU_OT_DSL_PERMS",
	"DMU_OT_ACL",
	"DMU_OT_SYSACL",
	"DMU_OT_FUID",
	"DMU_OT_FUID_SIZE",
	"DMU_OT_NEXT_CLONES",
	"DMU_OT_SCAN_QUEUE",
	"DMU_OT_USERGROUP_USED",
	"DMU_OT_USERGROUP_QUOTA",
	"DMU_OT_USERREFS",
	"DMU_OT_DDT_ZAP",
	"DMU_OT_DDT_STATS",
	"DMU_OT_SA",
	"DMU_OT_SA_MASTER_NODE",
	"DMU_OT_SA_ATTR_REGISTRATION",
	"DMU_OT_SA_ATTR_LAYOUTS",
	"DMU_OT_SCAN_XLATE",
	"DMU_OT_DEDUP",
	"DMU_OT_DEADLIST",
	"DMU_OT_DEADLIST_HDR",
	"DMU_OT_DSL_CLONES",
	"DMU_OT_BPOBJ_SUBOBJ"]
	otn_strings = {
	0x80: "DMU_OTN_UINT8_DATA",
	0xc0: "DMU_OTN_UINT8_METADATA",
	0x81: "DMU_OTN_UINT16_DATA",
	0xc1: "DMU_OTN_UINT16_METADATA",
	0x82: "DMU_OTN_UINT32_DATA",
	0xc2: "DMU_OTN_UINT32_METADATA",
	0x83: "DMU_OTN_UINT64_DATA",
	0xc3: "DMU_OTN_UINT64_METADATA",
	0x84: "DMU_OTN_ZAP_DATA",
	0xc4: "DMU_OTN_ZAP_METADATA",
	0xa0: "DMU_OTN_UINT8_ENC_DATA",
	0xe0: "DMU_OTN_UINT8_ENC_METADATA",
	0xa1: "DMU_OTN_UINT16_ENC_DATA",
	0xe1: "DMU_OTN_UINT16_ENC_METADATA",
	0xa2: "DMU_OTN_UINT32_ENC_DATA",
	0xe2: "DMU_OTN_UINT32_ENC_METADATA",
	0xa3: "DMU_OTN_UINT64_ENC_DATA",
	0xe3: "DMU_OTN_UINT64_ENC_METADATA",
	0xa4: "DMU_OTN_ZAP_ENC_DATA",
	0xe4: "DMU_OTN_ZAP_ENC_METADATA"}

	# If "-rr" option is used, don't convert to string representation
	if raw > 1:
	return "%i" % t

	try:
	if t < len(ot_strings):
	return ot_strings[t]
	else:
	return otn_strings[t]
	except (IndexError, KeyError):
	return "(UNKNOWN)"


	def get_compstring(c):
	comp_strings = ["ZIO_COMPRESS_INHERIT", "ZIO_COMPRESS_ON",
	"ZIO_COMPRESS_OFF", "ZIO_COMPRESS_LZJB",
	"ZIO_COMPRESS_EMPTY", "ZIO_COMPRESS_GZIP_1",
	"ZIO_COMPRESS_GZIP_2", "ZIO_COMPRESS_GZIP_3",
	"ZIO_COMPRESS_GZIP_4", "ZIO_COMPRESS_GZIP_5",
	"ZIO_COMPRESS_GZIP_6", "ZIO_COMPRESS_GZIP_7",
	"ZIO_COMPRESS_GZIP_8", "ZIO_COMPRESS_GZIP_9",
	"ZIO_COMPRESS_ZLE", "ZIO_COMPRESS_LZ4",
	"ZIO_COMPRESS_ZSTD", "ZIO_COMPRESS_FUNCTION"]

	# If "-rr" option is used, don't convert to string representation
	if raw > 1:
	return "%i" % c

	try:
	return comp_strings[c]
	except IndexError:
	return "%i" % c


	def parse_line(line, labels):
	global hdr

	new = dict()
	val = None
	for col in hdr:
	# These are "special" fields computed in the update_dict
	# function, prevent KeyError exception on labels[col] for these.
	if col not in ['bonus', 'cached', 'direct', 'indirect', 'spill']:
	val = line[labels[col]]

	if col in ['pool', 'flags']:
	new[col] = str(val)
	elif col in ['dtype', 'btype']:
	new[col] = get_typestring(int(val))
	elif col in ['l2_comp']:
	new[col] = get_compstring(int(val))
	else:
	new[col] = int(val)

	return new


	def update_dict(d, k, line, labels):
	pool = line[labels['pool']]
	objset = line[labels['objset']]
	key = line[labels[k]]

	dbsize = int(line[labels['dbsize']])
	blkid = int(line[labels['blkid']])
	level = int(line[labels['level']])

	if pool not in d:
	d[pool] = dict()

	if objset not in d[pool]:
	d[pool][objset] = dict()

	if key not in d[pool][objset]:
	d[pool][objset][key] = parse_line(line, labels)
	d[pool][objset][key]['bonus'] = 0
	d[pool][objset][key]['cached'] = 0
	d[pool][objset][key]['direct'] = 0
	d[pool][objset][key]['indirect'] = 0
	d[pool][objset][key]['spill'] = 0

	d[pool][objset][key]['cached'] += dbsize

	if blkid == -1:
	d[pool][objset][key]['bonus'] += dbsize
	elif blkid == -2:
	d[pool][objset][key]['spill'] += dbsize
	else:
	if level == 0:
	d[pool][objset][key]['direct'] += dbsize
	else:
	d[pool][objset][key]['indirect'] += dbsize

	return d


	def skip_line(vals, filters):
	'''
	Determines if a line should be skipped during printing
	based on a set of filters
	'''
	if len(filters) == 0:
	return False

	for key in vals:
	if key in filters:
	val = prettynum(cols[key][0], cols[key][1], vals[key]).strip()
	# we want a full match here
	if re.match("(?:" + filters[key] + r")\Z", val) is None:
	return True

	return False


	def print_dict(d, filters, noheader):
	if not noheader:
	print_header()
	for pool in list(d.keys()):
	for objset in list(d[pool].keys()):
	for v in list(d[pool][objset].values()):
	if not skip_line(v, filters):
	print_values(v)


	def dnodes_build_dict(filehandle):
	labels = dict()
	dnodes = dict()

	# First 3 lines are header information, skip the first two
	for i in range(2):
	next(filehandle)

	# The third line contains the labels and index locations
	for i, v in enumerate(next(filehandle).split()):
	labels[v] = i

	# The rest of the file is buffer information
	for line in filehandle:
	update_dict(dnodes, 'object', line.split(), labels)

	return dnodes


	def types_build_dict(filehandle):
	labels = dict()
	types = dict()

	# First 3 lines are header information, skip the first two
	for i in range(2):
	next(filehandle)

	# The third line contains the labels and index locations
	for i, v in enumerate(next(filehandle).split()):
	labels[v] = i

	# The rest of the file is buffer information
	for line in filehandle:
	update_dict(types, 'dtype', line.split(), labels)

	return types


	def buffers_print_all(filehandle, filters, noheader):
	labels = dict()

	# First 3 lines are header information, skip the first two
	for i in range(2):
	next(filehandle)

	# The third line contains the labels and index locations
	for i, v in enumerate(next(filehandle).split()):
	labels[v] = i

	if not noheader:
	print_header()

	# The rest of the file is buffer information
	for line in filehandle:
	vals = parse_line(line.split(), labels)
	if not skip_line(vals, filters):
	print_values(vals)


	def main():
	global hdr
	global sep
	global raw

	desired_cols = None
	bflag = False
	dflag = False
	hflag = False
	ifile = None
	ofile = None
	tflag = False
	vflag = False
	xflag = False
	nflag = False
	filters = dict()

	try:
	opts, args = getopt.getopt(
	sys.argv[1:],
	"bdf:hi:o:rs:tvxF:n",
	[
	"buffers",
	"dnodes",
	"columns",
	"help",
	"infile",
	"outfile",
	"separator",
	"types",
	"verbose",
	"extended",
	"filter"
	]
	)
	except getopt.error:
	usage()
	opts = None

	for opt, arg in opts:
	if opt in ('-b', '--buffers'):
	bflag = True
	if opt in ('-d', '--dnodes'):
	dflag = True
	if opt in ('-f', '--columns'):
	desired_cols = arg
	if opt in ('-h', '--help'):
	hflag = True
	if opt in ('-i', '--infile'):
	ifile = arg
	if opt in ('-o', '--outfile'):
	ofile = arg
	if opt in ('-r', '--raw'):
	raw += 1
	if opt in ('-s', '--separator'):
	sep = arg
	if opt in ('-t', '--types'):
	tflag = True
	if opt in ('-v', '--verbose'):
	vflag = True
	if opt in ('-x', '--extended'):
	xflag = True
	if opt in ('-n', '--noheader'):
	nflag = True
	if opt in ('-F', '--filter'):
	fils = [x.strip() for x in arg.split(",")]

	for fil in fils:
	f = [x.strip() for x in fil.split("=")]

	if len(f) != 2:
	sys.stderr.write("Invalid filter '%s'.\n" % fil)
	sys.exit(1)

	if f[0] not in cols:
	sys.stderr.write("Invalid field '%s' in filter.\n" % f[0])
	sys.exit(1)

	if f[0] in filters:
	sys.stderr.write("Field '%s' specified multiple times in "
	"filter.\n" % f[0])
	sys.exit(1)

	try:
	re.compile("(?:" + f[1] + r")\Z")
	except re.error:
	sys.stderr.write("Invalid regex for field '%s' in "
	"filter.\n" % f[0])
	sys.exit(1)

	filters[f[0]] = f[1]

	if hflag or (xflag and desired_cols):
	usage()

	if vflag:
	detailed_usage()

	# Ensure at most only one of b, d, or t flags are set
	if (bflag and dflag) or (bflag and tflag) or (dflag and tflag):
	usage()

	if bflag:
	hdr = bxhdr if xflag else bhdr
	elif tflag:
	hdr = txhdr if xflag else thdr
	else: # Even if dflag is False, it's the default if none set
	dflag = True
	hdr = dxhdr if xflag else dhdr

	if desired_cols:
	hdr = desired_cols.split(",")

	invalid = []
	incompat = []
	for ele in hdr:
	if ele not in cols:
	invalid.append(ele)
	elif ((bflag and bincompat and ele in bincompat) or
	(dflag and dincompat and ele in dincompat) or
	(tflag and tincompat and ele in tincompat)):
	incompat.append(ele)

	if len(invalid) > 0:
	sys.stderr.write("Invalid column definition! -- %s\n" % invalid)
	usage()

	if len(incompat) > 0:
	sys.stderr.write("Incompatible field specified! -- %s\n" %
	incompat)
	usage()

	if ofile:
	try:
	tmp = open(ofile, "w")
	sys.stdout = tmp

	except IOError:
	sys.stderr.write("Cannot open %s for writing\n" % ofile)
	sys.exit(1)

	if not ifile:
	- ifile = '/proc/spl/kstat/zfs/dbufs'
	+ ifile = default_ifile()

	if ifile is not "-":
	try:
	tmp = open(ifile, "r")
	sys.stdin = tmp
	except IOError:
	sys.stderr.write("Cannot open %s for reading\n" % ifile)
	sys.exit(1)

	if bflag:
	buffers_print_all(sys.stdin, filters, nflag)

	if dflag:
	print_dict(dnodes_build_dict(sys.stdin), filters, nflag)

	if tflag:
	print_dict(types_build_dict(sys.stdin), filters, nflag)


	if __name__ == '__main__':
	main()
	Index: vendor-sys/openzfs/dist/cmd/zdb/zdb.c
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zdb/zdb.c (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zdb/zdb.c (revision 366774)
	@@ -1,8598 +1,8612 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/

	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2011, 2019 by Delphix. All rights reserved.
	* Copyright (c) 2014 Integros [integros.com]
	* Copyright 2016 Nexenta Systems, Inc.
	* Copyright (c) 2017, 2018 Lawrence Livermore National Security, LLC.
	* Copyright (c) 2015, 2017, Intel Corporation.
	* Copyright (c) 2020 Datto Inc.
	* Copyright (c) 2020, The FreeBSD Foundation [1]
	*
	* [1] Portions of this software were developed by Allan Jude
	* under sponsorship from the FreeBSD Foundation.
	*/

	#include <stdio.h>
	#include <unistd.h>
	#include <stdlib.h>
	#include <ctype.h>
	#include <sys/zfs_context.h>
	#include <sys/spa.h>
	#include <sys/spa_impl.h>
	#include <sys/dmu.h>
	#include <sys/zap.h>
	#include <sys/fs/zfs.h>
	#include <sys/zfs_znode.h>
	#include <sys/zfs_sa.h>
	#include <sys/sa.h>
	#include <sys/sa_impl.h>
	#include <sys/vdev.h>
	#include <sys/vdev_impl.h>
	#include <sys/metaslab_impl.h>
	#include <sys/dmu_objset.h>
	#include <sys/dsl_dir.h>
	#include <sys/dsl_dataset.h>
	#include <sys/dsl_pool.h>
	#include <sys/dsl_bookmark.h>
	#include <sys/dbuf.h>
	#include <sys/zil.h>
	#include <sys/zil_impl.h>
	#include <sys/stat.h>
	#include <sys/resource.h>
	#include <sys/dmu_send.h>
	#include <sys/dmu_traverse.h>
	#include <sys/zio_checksum.h>
	#include <sys/zio_compress.h>
	#include <sys/zfs_fuid.h>
	#include <sys/arc.h>
	#include <sys/arc_impl.h>
	#include <sys/ddt.h>
	#include <sys/zfeature.h>
	#include <sys/abd.h>
	#include <sys/blkptr.h>
	#include <sys/dsl_crypt.h>
	#include <sys/dsl_scan.h>
	#include <sys/btree.h>
	#include <zfs_comutil.h>
	#include <sys/zstd/zstd.h>

	#include <libnvpair.h>
	#include <libzutil.h>

	#include "zdb.h"

	#define ZDB_COMPRESS_NAME(idx) ((idx) < ZIO_COMPRESS_FUNCTIONS ? \
	zio_compress_table[(idx)].ci_name : "UNKNOWN")
	#define ZDB_CHECKSUM_NAME(idx) ((idx) < ZIO_CHECKSUM_FUNCTIONS ? \
	zio_checksum_table[(idx)].ci_name : "UNKNOWN")
	#define ZDB_OT_TYPE(idx) ((idx) < DMU_OT_NUMTYPES ? (idx) : \
	(idx) == DMU_OTN_ZAP_DATA \|\| (idx) == DMU_OTN_ZAP_METADATA ? \
	DMU_OT_ZAP_OTHER : \
	(idx) == DMU_OTN_UINT64_DATA \|\| (idx) == DMU_OTN_UINT64_METADATA ? \
	DMU_OT_UINT64_OTHER : DMU_OT_NUMTYPES)

	static char *
	zdb_ot_name(dmu_object_type_t type)
	{
	if (type < DMU_OT_NUMTYPES)
	return (dmu_ot[type].ot_name);
	else if ((type & DMU_OT_NEWTYPE) &&
	((type & DMU_OT_BYTESWAP_MASK) < DMU_BSWAP_NUMFUNCS))
	return (dmu_ot_byteswap[type & DMU_OT_BYTESWAP_MASK].ob_name);
	else
	return ("UNKNOWN");
	}

	extern int reference_tracking_enable;
	extern int zfs_recover;
	extern unsigned long zfs_arc_meta_min, zfs_arc_meta_limit;
	extern int zfs_vdev_async_read_max_active;
	extern boolean_t spa_load_verify_dryrun;
	extern int zfs_reconstruct_indirect_combinations_max;
	extern int zfs_btree_verify_intensity;

	static const char cmdname[] = "zdb";
	uint8_t dump_opt[256];

	typedef void object_viewer_t(objset_t , uint64_t, void data, size_t size);

	uint64_t *zopt_metaslab = NULL;
	static unsigned zopt_metaslab_args = 0;

	typedef struct zopt_object_range {
	uint64_t zor_obj_start;
	uint64_t zor_obj_end;
	uint64_t zor_flags;
	} zopt_object_range_t;
	zopt_object_range_t *zopt_object_ranges = NULL;
	static unsigned zopt_object_args = 0;

	static int flagbits[256];

	#define ZOR_FLAG_PLAIN_FILE 0x0001
	#define ZOR_FLAG_DIRECTORY 0x0002
	#define ZOR_FLAG_SPACE_MAP 0x0004
	#define ZOR_FLAG_ZAP 0x0008
	#define ZOR_FLAG_ALL_TYPES -1
	#define ZOR_SUPPORTED_FLAGS (ZOR_FLAG_PLAIN_FILE \| \
	ZOR_FLAG_DIRECTORY \| \
	ZOR_FLAG_SPACE_MAP \| \
	ZOR_FLAG_ZAP)

	#define ZDB_FLAG_CHECKSUM 0x0001
	#define ZDB_FLAG_DECOMPRESS 0x0002
	#define ZDB_FLAG_BSWAP 0x0004
	#define ZDB_FLAG_GBH 0x0008
	#define ZDB_FLAG_INDIRECT 0x0010
	#define ZDB_FLAG_RAW 0x0020
	#define ZDB_FLAG_PRINT_BLKPTR 0x0040
	#define ZDB_FLAG_VERBOSE 0x0080

	uint64_t max_inflight_bytes = 256 * 1024 * 1024; /* 256MB */
	static int leaked_objects = 0;
	static range_tree_t *mos_refd_objs;

	static void snprintf_blkptr_compact(char , size_t, const blkptr_t ,
	boolean_t);
	static void mos_obj_refd(uint64_t);
	static void mos_obj_refd_multiple(uint64_t);
	static int dump_bpobj_cb(void arg, const blkptr_t bp, boolean_t free,
	dmu_tx_t *tx);

	typedef struct sublivelist_verify {
	/* all ALLOC'd blkptr_t in one sub-livelist */
	zfs_btree_t sv_all_allocs;

	/* all FREE'd blkptr_t in one sub-livelist */
	zfs_btree_t sv_all_frees;

	/* FREE's that haven't yet matched to an ALLOC, in one sub-livelist */
	zfs_btree_t sv_pair;

	/* ALLOC's without a matching FREE, accumulates across sub-livelists */
	zfs_btree_t sv_leftover;
	} sublivelist_verify_t;

	static int
	livelist_compare(const void larg, const void rarg)
	{
	const blkptr_t *l = larg;
	const blkptr_t *r = rarg;

	/* Sort them according to dva[0] */
	uint64_t l_dva0_vdev, r_dva0_vdev;
	l_dva0_vdev = DVA_GET_VDEV(&l->blk_dva[0]);
	r_dva0_vdev = DVA_GET_VDEV(&r->blk_dva[0]);
	if (l_dva0_vdev < r_dva0_vdev)
	return (-1);
	else if (l_dva0_vdev > r_dva0_vdev)
	return (+1);

	/* if vdevs are equal, sort by offsets. */
	uint64_t l_dva0_offset;
	uint64_t r_dva0_offset;
	l_dva0_offset = DVA_GET_OFFSET(&l->blk_dva[0]);
	r_dva0_offset = DVA_GET_OFFSET(&r->blk_dva[0]);
	if (l_dva0_offset < r_dva0_offset) {
	return (-1);
	} else if (l_dva0_offset > r_dva0_offset) {
	return (+1);
	}

	/*
	* Since we're storing blkptrs without cancelling FREE/ALLOC pairs,
	* it's possible the offsets are equal. In that case, sort by txg
	*/
	if (l->blk_birth < r->blk_birth) {
	return (-1);
	} else if (l->blk_birth > r->blk_birth) {
	return (+1);
	}
	return (0);
	}

	typedef struct sublivelist_verify_block {
	dva_t svb_dva;

	/*
	* We need this to check if the block marked as allocated
	* in the livelist was freed (and potentially reallocated)
	* in the metaslab spacemaps at a later TXG.
	*/
	uint64_t svb_allocated_txg;
	} sublivelist_verify_block_t;

	static void zdb_print_blkptr(const blkptr_t *bp, int flags);

	static int
	sublivelist_verify_blkptr(void arg, const blkptr_t bp, boolean_t free,
	dmu_tx_t *tx)
	{
	ASSERT3P(tx, ==, NULL);
	struct sublivelist_verify *sv = arg;
	char blkbuf[BP_SPRINTF_LEN];
	zfs_btree_index_t where;
	if (free) {
	zfs_btree_add(&sv->sv_pair, bp);
	/* Check if the FREE is a duplicate */
	if (zfs_btree_find(&sv->sv_all_frees, bp, &where) != NULL) {
	snprintf_blkptr_compact(blkbuf, sizeof (blkbuf), bp,
	free);
	(void) printf("\tERROR: Duplicate FREE: %s\n", blkbuf);
	} else {
	zfs_btree_add_idx(&sv->sv_all_frees, bp, &where);
	}
	} else {
	/* Check if the ALLOC has been freed */
	if (zfs_btree_find(&sv->sv_pair, bp, &where) != NULL) {
	zfs_btree_remove_idx(&sv->sv_pair, &where);
	} else {
	for (int i = 0; i < SPA_DVAS_PER_BP; i++) {
	if (DVA_IS_EMPTY(&bp->blk_dva[i]))
	break;
	sublivelist_verify_block_t svb = {
	.svb_dva = bp->blk_dva[i],
	.svb_allocated_txg = bp->blk_birth
	};

	if (zfs_btree_find(&sv->sv_leftover, &svb,
	&where) == NULL) {
	zfs_btree_add_idx(&sv->sv_leftover,
	&svb, &where);
	}
	}
	}
	/* Check if the ALLOC is a duplicate */
	if (zfs_btree_find(&sv->sv_all_allocs, bp, &where) != NULL) {
	snprintf_blkptr_compact(blkbuf, sizeof (blkbuf), bp,
	free);
	(void) printf("\tERROR: Duplicate ALLOC: %s\n", blkbuf);
	} else {
	zfs_btree_add_idx(&sv->sv_all_allocs, bp, &where);
	}
	}
	return (0);
	}

	static int
	sublivelist_verify_func(void args, dsl_deadlist_entry_t dle)
	{
	int err;
	char blkbuf[BP_SPRINTF_LEN];
	struct sublivelist_verify *sv = args;

	zfs_btree_create(&sv->sv_all_allocs, livelist_compare,
	sizeof (blkptr_t));

	zfs_btree_create(&sv->sv_all_frees, livelist_compare,
	sizeof (blkptr_t));

	zfs_btree_create(&sv->sv_pair, livelist_compare,
	sizeof (blkptr_t));

	err = bpobj_iterate_nofree(&dle->dle_bpobj, sublivelist_verify_blkptr,
	sv, NULL);

	zfs_btree_clear(&sv->sv_all_allocs);
	zfs_btree_destroy(&sv->sv_all_allocs);

	zfs_btree_clear(&sv->sv_all_frees);
	zfs_btree_destroy(&sv->sv_all_frees);

	blkptr_t *e;
	zfs_btree_index_t *cookie = NULL;
	while ((e = zfs_btree_destroy_nodes(&sv->sv_pair, &cookie)) != NULL) {
	snprintf_blkptr_compact(blkbuf, sizeof (blkbuf), e, B_TRUE);
	(void) printf("\tERROR: Unmatched FREE: %s\n", blkbuf);
	}
	zfs_btree_destroy(&sv->sv_pair);

	return (err);
	}

	static int
	livelist_block_compare(const void larg, const void rarg)
	{
	const sublivelist_verify_block_t *l = larg;
	const sublivelist_verify_block_t *r = rarg;

	if (DVA_GET_VDEV(&l->svb_dva) < DVA_GET_VDEV(&r->svb_dva))
	return (-1);
	else if (DVA_GET_VDEV(&l->svb_dva) > DVA_GET_VDEV(&r->svb_dva))
	return (+1);

	if (DVA_GET_OFFSET(&l->svb_dva) < DVA_GET_OFFSET(&r->svb_dva))
	return (-1);
	else if (DVA_GET_OFFSET(&l->svb_dva) > DVA_GET_OFFSET(&r->svb_dva))
	return (+1);

	if (DVA_GET_ASIZE(&l->svb_dva) < DVA_GET_ASIZE(&r->svb_dva))
	return (-1);
	else if (DVA_GET_ASIZE(&l->svb_dva) > DVA_GET_ASIZE(&r->svb_dva))
	return (+1);

	return (0);
	}

	/*
	* Check for errors in a livelist while tracking all unfreed ALLOCs in the
	* sublivelist_verify_t: sv->sv_leftover
	*/
	static void
	livelist_verify(dsl_deadlist_t dl, void arg)
	{
	sublivelist_verify_t *sv = arg;
	dsl_deadlist_iterate(dl, sublivelist_verify_func, sv);
	}

	/*
	* Check for errors in the livelist entry and discard the intermediary
	* data structures
	*/
	/* ARGSUSED */
	static int
	sublivelist_verify_lightweight(void args, dsl_deadlist_entry_t dle)
	{
	sublivelist_verify_t sv;
	zfs_btree_create(&sv.sv_leftover, livelist_block_compare,
	sizeof (sublivelist_verify_block_t));
	int err = sublivelist_verify_func(&sv, dle);
	zfs_btree_clear(&sv.sv_leftover);
	zfs_btree_destroy(&sv.sv_leftover);
	return (err);
	}

	typedef struct metaslab_verify {
	/*
	* Tree containing all the leftover ALLOCs from the livelists
	* that are part of this metaslab.
	*/
	zfs_btree_t mv_livelist_allocs;

	/*
	* Metaslab information.
	*/
	uint64_t mv_vdid;
	uint64_t mv_msid;
	uint64_t mv_start;
	uint64_t mv_end;

	/*
	* What's currently allocated for this metaslab.
	*/
	range_tree_t *mv_allocated;
	} metaslab_verify_t;

	typedef void ll_iter_t(dsl_deadlist_t ll, void arg);

	typedef int (zdb_log_sm_cb_t)(spa_t spa, space_map_entry_t *sme, uint64_t txg,
	void *arg);

	typedef struct unflushed_iter_cb_arg {
	spa_t *uic_spa;
	uint64_t uic_txg;
	void *uic_arg;
	zdb_log_sm_cb_t uic_cb;
	} unflushed_iter_cb_arg_t;

	static int
	iterate_through_spacemap_logs_cb(space_map_entry_t sme, void arg)
	{
	unflushed_iter_cb_arg_t *uic = arg;
	return (uic->uic_cb(uic->uic_spa, sme, uic->uic_txg, uic->uic_arg));
	}

	static void
	iterate_through_spacemap_logs(spa_t spa, zdb_log_sm_cb_t cb, void arg)
	{
	if (!spa_feature_is_active(spa, SPA_FEATURE_LOG_SPACEMAP))
	return;

	spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER);
	for (spa_log_sm_t *sls = avl_first(&spa->spa_sm_logs_by_txg);
	sls; sls = AVL_NEXT(&spa->spa_sm_logs_by_txg, sls)) {
	space_map_t *sm = NULL;
	VERIFY0(space_map_open(&sm, spa_meta_objset(spa),
	sls->sls_sm_obj, 0, UINT64_MAX, SPA_MINBLOCKSHIFT));

	unflushed_iter_cb_arg_t uic = {
	.uic_spa = spa,
	.uic_txg = sls->sls_txg,
	.uic_arg = arg,
	.uic_cb = cb
	};
	VERIFY0(space_map_iterate(sm, space_map_length(sm),
	iterate_through_spacemap_logs_cb, &uic));
	space_map_close(sm);
	}
	spa_config_exit(spa, SCL_CONFIG, FTAG);
	}

	static void
	verify_livelist_allocs(metaslab_verify_t *mv, uint64_t txg,
	uint64_t offset, uint64_t size)
	{
	sublivelist_verify_block_t svb;
	DVA_SET_VDEV(&svb.svb_dva, mv->mv_vdid);
	DVA_SET_OFFSET(&svb.svb_dva, offset);
	DVA_SET_ASIZE(&svb.svb_dva, size);
	zfs_btree_index_t where;
	uint64_t end_offset = offset + size;

	/*
	* Look for an exact match for spacemap entry in the livelist entries.
	* Then, look for other livelist entries that fall within the range
	* of the spacemap entry as it may have been condensed
	*/
	sublivelist_verify_block_t *found =
	zfs_btree_find(&mv->mv_livelist_allocs, &svb, &where);
	if (found == NULL) {
	found = zfs_btree_next(&mv->mv_livelist_allocs, &where, &where);
	}
	for (; found != NULL && DVA_GET_VDEV(&found->svb_dva) == mv->mv_vdid &&
	DVA_GET_OFFSET(&found->svb_dva) < end_offset;
	found = zfs_btree_next(&mv->mv_livelist_allocs, &where, &where)) {
	if (found->svb_allocated_txg <= txg) {
	(void) printf("ERROR: Livelist ALLOC [%llx:%llx] "
	"from TXG %llx FREED at TXG %llx\n",
	(u_longlong_t)DVA_GET_OFFSET(&found->svb_dva),
	(u_longlong_t)DVA_GET_ASIZE(&found->svb_dva),
	(u_longlong_t)found->svb_allocated_txg,
	(u_longlong_t)txg);
	}
	}
	}

	static int
	metaslab_spacemap_validation_cb(space_map_entry_t sme, void arg)
	{
	metaslab_verify_t *mv = arg;
	uint64_t offset = sme->sme_offset;
	uint64_t size = sme->sme_run;
	uint64_t txg = sme->sme_txg;

	if (sme->sme_type == SM_ALLOC) {
	if (range_tree_contains(mv->mv_allocated,
	offset, size)) {
	(void) printf("ERROR: DOUBLE ALLOC: "
	"%llu [%llx:%llx] "
	"%llu:%llu LOG_SM\n",
	(u_longlong_t)txg, (u_longlong_t)offset,
	(u_longlong_t)size, (u_longlong_t)mv->mv_vdid,
	(u_longlong_t)mv->mv_msid);
	} else {
	range_tree_add(mv->mv_allocated,
	offset, size);
	}
	} else {
	if (!range_tree_contains(mv->mv_allocated,
	offset, size)) {
	(void) printf("ERROR: DOUBLE FREE: "
	"%llu [%llx:%llx] "
	"%llu:%llu LOG_SM\n",
	(u_longlong_t)txg, (u_longlong_t)offset,
	(u_longlong_t)size, (u_longlong_t)mv->mv_vdid,
	(u_longlong_t)mv->mv_msid);
	} else {
	range_tree_remove(mv->mv_allocated,
	offset, size);
	}
	}

	if (sme->sme_type != SM_ALLOC) {
	/*
	* If something is freed in the spacemap, verify that
	* it is not listed as allocated in the livelist.
	*/
	verify_livelist_allocs(mv, txg, offset, size);
	}
	return (0);
	}

	static int
	spacemap_check_sm_log_cb(spa_t spa, space_map_entry_t sme,
	uint64_t txg, void *arg)
	{
	metaslab_verify_t *mv = arg;
	uint64_t offset = sme->sme_offset;
	uint64_t vdev_id = sme->sme_vdev;

	vdev_t *vd = vdev_lookup_top(spa, vdev_id);

	/* skip indirect vdevs */
	if (!vdev_is_concrete(vd))
	return (0);

	if (vdev_id != mv->mv_vdid)
	return (0);

	metaslab_t *ms = vd->vdev_ms[offset >> vd->vdev_ms_shift];
	if (ms->ms_id != mv->mv_msid)
	return (0);

	if (txg < metaslab_unflushed_txg(ms))
	return (0);


	ASSERT3U(txg, ==, sme->sme_txg);
	return (metaslab_spacemap_validation_cb(sme, mv));
	}

	static void
	spacemap_check_sm_log(spa_t spa, metaslab_verify_t mv)
	{
	iterate_through_spacemap_logs(spa, spacemap_check_sm_log_cb, mv);
	}

	static void
	spacemap_check_ms_sm(space_map_t sm, metaslab_verify_t mv)
	{
	if (sm == NULL)
	return;

	VERIFY0(space_map_iterate(sm, space_map_length(sm),
	metaslab_spacemap_validation_cb, mv));
	}

	static void iterate_deleted_livelists(spa_t spa, ll_iter_t func, void arg);

	/*
	* Transfer blocks from sv_leftover tree to the mv_livelist_allocs if
	* they are part of that metaslab (mv_msid).
	*/
	static void
	mv_populate_livelist_allocs(metaslab_verify_t mv, sublivelist_verify_t sv)
	{
	zfs_btree_index_t where;
	sublivelist_verify_block_t *svb;
	ASSERT3U(zfs_btree_numnodes(&mv->mv_livelist_allocs), ==, 0);
	for (svb = zfs_btree_first(&sv->sv_leftover, &where);
	svb != NULL;
	svb = zfs_btree_next(&sv->sv_leftover, &where, &where)) {
	if (DVA_GET_VDEV(&svb->svb_dva) != mv->mv_vdid)
	continue;

	if (DVA_GET_OFFSET(&svb->svb_dva) < mv->mv_start &&
	(DVA_GET_OFFSET(&svb->svb_dva) +
	DVA_GET_ASIZE(&svb->svb_dva)) > mv->mv_start) {
	(void) printf("ERROR: Found block that crosses "
	"metaslab boundary: <%llu:%llx:%llx>\n",
	(u_longlong_t)DVA_GET_VDEV(&svb->svb_dva),
	(u_longlong_t)DVA_GET_OFFSET(&svb->svb_dva),
	(u_longlong_t)DVA_GET_ASIZE(&svb->svb_dva));
	continue;
	}

	if (DVA_GET_OFFSET(&svb->svb_dva) < mv->mv_start)
	continue;

	if (DVA_GET_OFFSET(&svb->svb_dva) >= mv->mv_end)
	continue;

	if ((DVA_GET_OFFSET(&svb->svb_dva) +
	DVA_GET_ASIZE(&svb->svb_dva)) > mv->mv_end) {
	(void) printf("ERROR: Found block that crosses "
	"metaslab boundary: <%llu:%llx:%llx>\n",
	(u_longlong_t)DVA_GET_VDEV(&svb->svb_dva),
	(u_longlong_t)DVA_GET_OFFSET(&svb->svb_dva),
	(u_longlong_t)DVA_GET_ASIZE(&svb->svb_dva));
	continue;
	}

	zfs_btree_add(&mv->mv_livelist_allocs, svb);
	}

	for (svb = zfs_btree_first(&mv->mv_livelist_allocs, &where);
	svb != NULL;
	svb = zfs_btree_next(&mv->mv_livelist_allocs, &where, &where)) {
	zfs_btree_remove(&sv->sv_leftover, svb);
	}
	}

	/*
	* [Livelist Check]
	* Iterate through all the sublivelists and:
	* - report leftover frees
	* - report double ALLOCs/FREEs
	* - record leftover ALLOCs together with their TXG [see Cross Check]
	*
	* [Spacemap Check]
	* for each metaslab:
	* - iterate over spacemap and then the metaslab's entries in the
	* spacemap log, then report any double FREEs and ALLOCs (do not
	* blow up).
	*
	* [Cross Check]
	* After finishing the Livelist Check phase and while being in the
	* Spacemap Check phase, we find all the recorded leftover ALLOCs
	* of the livelist check that are part of the metaslab that we are
	* currently looking at in the Spacemap Check. We report any entries
	* that are marked as ALLOCs in the livelists but have been actually
	* freed (and potentially allocated again) after their TXG stamp in
	* the spacemaps. Also report any ALLOCs from the livelists that
	* belong to indirect vdevs (e.g. their vdev completed removal).
	*
	* Note that this will miss Log Spacemap entries that cancelled each other
	* out before being flushed to the metaslab, so we are not guaranteed
	* to match all erroneous ALLOCs.
	*/
	static void
	livelist_metaslab_validate(spa_t *spa)
	{
	(void) printf("Verifying deleted livelist entries\n");

	sublivelist_verify_t sv;
	zfs_btree_create(&sv.sv_leftover, livelist_block_compare,
	sizeof (sublivelist_verify_block_t));
	iterate_deleted_livelists(spa, livelist_verify, &sv);

	(void) printf("Verifying metaslab entries\n");
	vdev_t *rvd = spa->spa_root_vdev;
	for (uint64_t c = 0; c < rvd->vdev_children; c++) {
	vdev_t *vd = rvd->vdev_child[c];

	if (!vdev_is_concrete(vd))
	continue;

	for (uint64_t mid = 0; mid < vd->vdev_ms_count; mid++) {
	metaslab_t *m = vd->vdev_ms[mid];

	(void) fprintf(stderr,
	"\rverifying concrete vdev %llu, "
	"metaslab %llu of %llu ...",
	(longlong_t)vd->vdev_id,
	(longlong_t)mid,
	(longlong_t)vd->vdev_ms_count);

	uint64_t shift, start;
	range_seg_type_t type =
	metaslab_calculate_range_tree_type(vd, m,
	&start, &shift);
	metaslab_verify_t mv;
	mv.mv_allocated = range_tree_create(NULL,
	type, NULL, start, shift);
	mv.mv_vdid = vd->vdev_id;
	mv.mv_msid = m->ms_id;
	mv.mv_start = m->ms_start;
	mv.mv_end = m->ms_start + m->ms_size;
	zfs_btree_create(&mv.mv_livelist_allocs,
	livelist_block_compare,
	sizeof (sublivelist_verify_block_t));

	mv_populate_livelist_allocs(&mv, &sv);

	spacemap_check_ms_sm(m->ms_sm, &mv);
	spacemap_check_sm_log(spa, &mv);

	range_tree_vacate(mv.mv_allocated, NULL, NULL);
	range_tree_destroy(mv.mv_allocated);
	zfs_btree_clear(&mv.mv_livelist_allocs);
	zfs_btree_destroy(&mv.mv_livelist_allocs);
	}
	}
	(void) fprintf(stderr, "\n");

	/*
	* If there are any segments in the leftover tree after we walked
	* through all the metaslabs in the concrete vdevs then this means
	* that we have segments in the livelists that belong to indirect
	* vdevs and are marked as allocated.
	*/
	if (zfs_btree_numnodes(&sv.sv_leftover) == 0) {
	zfs_btree_destroy(&sv.sv_leftover);
	return;
	}
	(void) printf("ERROR: Found livelist blocks marked as allocated "
	"for indirect vdevs:\n");

	zfs_btree_index_t *where = NULL;
	sublivelist_verify_block_t *svb;
	while ((svb = zfs_btree_destroy_nodes(&sv.sv_leftover, &where)) !=
	NULL) {
	int vdev_id = DVA_GET_VDEV(&svb->svb_dva);
	ASSERT3U(vdev_id, <, rvd->vdev_children);
	vdev_t *vd = rvd->vdev_child[vdev_id];
	ASSERT(!vdev_is_concrete(vd));
	(void) printf("<%d:%llx:%llx> TXG %llx\n",
	vdev_id, (u_longlong_t)DVA_GET_OFFSET(&svb->svb_dva),
	(u_longlong_t)DVA_GET_ASIZE(&svb->svb_dva),
	(u_longlong_t)svb->svb_allocated_txg);
	}
	(void) printf("\n");
	zfs_btree_destroy(&sv.sv_leftover);
	}

	/*
	* These libumem hooks provide a reasonable set of defaults for the allocator's
	* debugging facilities.
	*/
	const char *
	_umem_debug_init(void)
	{
	return ("default,verbose"); /* $UMEM_DEBUG setting */
	}

	const char *
	_umem_logging_init(void)
	{
	return ("fail,contents"); /* $UMEM_LOGGING setting */
	}

	static void
	usage(void)
	{
	(void) fprintf(stderr,
	"Usage:\t%s [-AbcdDFGhikLMPsvXy] [-e [-V] [-p <path> ...]] "
	"[-I <inflight I/Os>]\n"
	"\t\t[-o <var>=<value>]... [-t <txg>] [-U <cache>] [-x <dumpdir>]\n"
	"\t\t[<poolname>[/<dataset \| objset id>] [<object \| range> ...]]\n"
	"\t%s [-AdiPv] [-e [-V] [-p <path> ...]] [-U <cache>]\n"
	"\t\t[<poolname>[/<dataset \| objset id>] [<object \| range> ...]\n"
	"\t%s [-v] <bookmark>\n"
	"\t%s -C [-A] [-U <cache>]\n"
	"\t%s -l [-Aqu] <device>\n"
	"\t%s -m [-AFLPX] [-e [-V] [-p <path> ...]] [-t <txg>] "
	"[-U <cache>]\n\t\t<poolname> [<vdev> [<metaslab> ...]]\n"
	"\t%s -O <dataset> <path>\n"
	"\t%s -R [-A] [-e [-V] [-p <path> ...]] [-U <cache>]\n"
	"\t\t<poolname> <vdev>:<offset>:<size>[:<flags>]\n"
	"\t%s -E [-A] word0:word1:...:word15\n"
	"\t%s -S [-AP] [-e [-V] [-p <path> ...]] [-U <cache>] "
	"<poolname>\n\n",
	cmdname, cmdname, cmdname, cmdname, cmdname, cmdname, cmdname,
	cmdname, cmdname, cmdname);

	(void) fprintf(stderr, " Dataset name must include at least one "
	"separator character '/' or '@'\n");
	(void) fprintf(stderr, " If dataset name is specified, only that "
	"dataset is dumped\n");
	(void) fprintf(stderr, " If object numbers or object number "
	"ranges are specified, only those\n"
	" objects or ranges are dumped.\n\n");
	(void) fprintf(stderr,
	" Object ranges take the form <start>:<end>[:<flags>]\n"
	" start Starting object number\n"
	" end Ending object number, or -1 for no upper bound\n"
	" flags Optional flags to select object types:\n"
	" A All objects (this is the default)\n"
	" d ZFS directories\n"
	" f ZFS files \n"
	" m SPA space maps\n"
	" z ZAPs\n"
	" - Negate effect of next flag\n\n");
	(void) fprintf(stderr, " Options to control amount of output:\n");
	(void) fprintf(stderr, " -b block statistics\n");
	(void) fprintf(stderr, " -c checksum all metadata (twice for "
	"all data) blocks\n");
	(void) fprintf(stderr, " -C config (or cachefile if alone)\n");
	(void) fprintf(stderr, " -d dataset(s)\n");
	(void) fprintf(stderr, " -D dedup statistics\n");
	(void) fprintf(stderr, " -E decode and display block from an "
	"embedded block pointer\n");
	(void) fprintf(stderr, " -h pool history\n");
	(void) fprintf(stderr, " -i intent logs\n");
	(void) fprintf(stderr, " -l read label contents\n");
	(void) fprintf(stderr, " -k examine the checkpointed state "
	"of the pool\n");
	(void) fprintf(stderr, " -L disable leak tracking (do not "
	"load spacemaps)\n");
	(void) fprintf(stderr, " -m metaslabs\n");
	(void) fprintf(stderr, " -M metaslab groups\n");
	(void) fprintf(stderr, " -O perform object lookups by path\n");
	(void) fprintf(stderr, " -R read and display block from a "
	"device\n");
	(void) fprintf(stderr, " -s report stats on zdb's I/O\n");
	(void) fprintf(stderr, " -S simulate dedup to measure effect\n");
	(void) fprintf(stderr, " -v verbose (applies to all "
	"others)\n");
	(void) fprintf(stderr, " -y perform livelist and metaslab "
	"validation on any livelists being deleted\n\n");
	(void) fprintf(stderr, " Below options are intended for use "
	"with other options:\n");
	(void) fprintf(stderr, " -A ignore assertions (-A), enable "
	"panic recovery (-AA) or both (-AAA)\n");
	(void) fprintf(stderr, " -e pool is exported/destroyed/"
	"has altroot/not in a cachefile\n");
	(void) fprintf(stderr, " -F attempt automatic rewind within "
	"safe range of transaction groups\n");
	(void) fprintf(stderr, " -G dump zfs_dbgmsg buffer before "
	"exiting\n");
	(void) fprintf(stderr, " -I <number of inflight I/Os> -- "
	"specify the maximum number of\n "
	"checksumming I/Os [default is 200]\n");
	(void) fprintf(stderr, " -o <variable>=<value> set global "
	"variable to an unsigned 32-bit integer\n");
	(void) fprintf(stderr, " -p <path> -- use one or more with "
	"-e to specify path to vdev dir\n");
	(void) fprintf(stderr, " -P print numbers in parseable form\n");
	(void) fprintf(stderr, " -q don't print label contents\n");
	(void) fprintf(stderr, " -t <txg> -- highest txg to use when "
	"searching for uberblocks\n");
	(void) fprintf(stderr, " -u uberblock\n");
	(void) fprintf(stderr, " -U <cachefile_path> -- use alternate "
	"cachefile\n");
	(void) fprintf(stderr, " -V do verbatim import\n");
	(void) fprintf(stderr, " -x <dumpdir> -- "
	"dump all read blocks into specified directory\n");
	(void) fprintf(stderr, " -X attempt extreme rewind (does not "
	"work with dataset)\n");
	(void) fprintf(stderr, " -Y attempt all reconstruction "
	"combinations for split blocks\n");
	(void) fprintf(stderr, " -Z show ZSTD headers \n");
	(void) fprintf(stderr, "Specify an option more than once (e.g. -bb) "
	"to make only that option verbose\n");
	(void) fprintf(stderr, "Default is to dump everything non-verbosely\n");
	exit(1);
	}

	static void
	dump_debug_buffer(void)
	{
	if (dump_opt['G']) {
	(void) printf("\n");
	(void) fflush(stdout);
	zfs_dbgmsg_print("zdb");
	}
	}

	/*
	* Called for usage errors that are discovered after a call to spa_open(),
	* dmu_bonus_hold(), or pool_match(). abort() is called for other errors.
	*/

	static void
	fatal(const char *fmt, ...)
	{
	va_list ap;

	va_start(ap, fmt);
	(void) fprintf(stderr, "%s: ", cmdname);
	(void) vfprintf(stderr, fmt, ap);
	va_end(ap);
	(void) fprintf(stderr, "\n");

	dump_debug_buffer();

	exit(1);
	}

	/* ARGSUSED */
	static void
	dump_packed_nvlist(objset_t os, uint64_t object, void data, size_t size)
	{
	nvlist_t *nv;
	size_t nvsize = (uint64_t )data;
	char *packed = umem_alloc(nvsize, UMEM_NOFAIL);

	VERIFY(0 == dmu_read(os, object, 0, nvsize, packed, DMU_READ_PREFETCH));

	VERIFY(nvlist_unpack(packed, nvsize, &nv, 0) == 0);

	umem_free(packed, nvsize);

	dump_nvlist(nv, 8);

	nvlist_free(nv);
	}

	/* ARGSUSED */
	static void
	dump_history_offsets(objset_t os, uint64_t object, void data, size_t size)
	{
	spa_history_phys_t *shp = data;

	if (shp == NULL)
	return;

	(void) printf("\t\tpool_create_len = %llu\n",
	(u_longlong_t)shp->sh_pool_create_len);
	(void) printf("\t\tphys_max_off = %llu\n",
	(u_longlong_t)shp->sh_phys_max_off);
	(void) printf("\t\tbof = %llu\n",
	(u_longlong_t)shp->sh_bof);
	(void) printf("\t\teof = %llu\n",
	(u_longlong_t)shp->sh_eof);
	(void) printf("\t\trecords_lost = %llu\n",
	(u_longlong_t)shp->sh_records_lost);
	}

	static void
	zdb_nicenum(uint64_t num, char *buf, size_t buflen)
	{
	if (dump_opt['P'])
	(void) snprintf(buf, buflen, "%llu", (longlong_t)num);
	else
	nicenum(num, buf, sizeof (buf));
	}

	static const char histo_stars[] = "****************************************";
	static const uint64_t histo_width = sizeof (histo_stars) - 1;

	static void
	dump_histogram(const uint64_t *histo, int size, int offset)
	{
	int i;
	int minidx = size - 1;
	int maxidx = 0;
	uint64_t max = 0;

	for (i = 0; i < size; i++) {
	if (histo[i] > max)
	max = histo[i];
	if (histo[i] > 0 && i > maxidx)
	maxidx = i;
	if (histo[i] > 0 && i < minidx)
	minidx = i;
	}

	if (max < histo_width)
	max = histo_width;

	for (i = minidx; i <= maxidx; i++) {
	(void) printf("\t\t\t%3u: %6llu %s\n",
	i + offset, (u_longlong_t)histo[i],
	&histo_stars[(max - histo[i]) * histo_width / max]);
	}
	}

	static void
	dump_zap_stats(objset_t *os, uint64_t object)
	{
	int error;
	zap_stats_t zs;

	error = zap_get_stats(os, object, &zs);
	if (error)
	return;

	if (zs.zs_ptrtbl_len == 0) {
	ASSERT(zs.zs_num_blocks == 1);
	(void) printf("\tmicrozap: %llu bytes, %llu entries\n",
	(u_longlong_t)zs.zs_blocksize,
	(u_longlong_t)zs.zs_num_entries);
	return;
	}

	(void) printf("\tFat ZAP stats:\n");

	(void) printf("\t\tPointer table:\n");
	(void) printf("\t\t\t%llu elements\n",
	(u_longlong_t)zs.zs_ptrtbl_len);
	(void) printf("\t\t\tzt_blk: %llu\n",
	(u_longlong_t)zs.zs_ptrtbl_zt_blk);
	(void) printf("\t\t\tzt_numblks: %llu\n",
	(u_longlong_t)zs.zs_ptrtbl_zt_numblks);
	(void) printf("\t\t\tzt_shift: %llu\n",
	(u_longlong_t)zs.zs_ptrtbl_zt_shift);
	(void) printf("\t\t\tzt_blks_copied: %llu\n",
	(u_longlong_t)zs.zs_ptrtbl_blks_copied);
	(void) printf("\t\t\tzt_nextblk: %llu\n",
	(u_longlong_t)zs.zs_ptrtbl_nextblk);

	(void) printf("\t\tZAP entries: %llu\n",
	(u_longlong_t)zs.zs_num_entries);
	(void) printf("\t\tLeaf blocks: %llu\n",
	(u_longlong_t)zs.zs_num_leafs);
	(void) printf("\t\tTotal blocks: %llu\n",
	(u_longlong_t)zs.zs_num_blocks);
	(void) printf("\t\tzap_block_type: 0x%llx\n",
	(u_longlong_t)zs.zs_block_type);
	(void) printf("\t\tzap_magic: 0x%llx\n",
	(u_longlong_t)zs.zs_magic);
	(void) printf("\t\tzap_salt: 0x%llx\n",
	(u_longlong_t)zs.zs_salt);

	(void) printf("\t\tLeafs with 2^n pointers:\n");
	dump_histogram(zs.zs_leafs_with_2n_pointers, ZAP_HISTOGRAM_SIZE, 0);

	(void) printf("\t\tBlocks with n*5 entries:\n");
	dump_histogram(zs.zs_blocks_with_n5_entries, ZAP_HISTOGRAM_SIZE, 0);

	(void) printf("\t\tBlocks n/10 full:\n");
	dump_histogram(zs.zs_blocks_n_tenths_full, ZAP_HISTOGRAM_SIZE, 0);

	(void) printf("\t\tEntries with n chunks:\n");
	dump_histogram(zs.zs_entries_using_n_chunks, ZAP_HISTOGRAM_SIZE, 0);

	(void) printf("\t\tBuckets with n entries:\n");
	dump_histogram(zs.zs_buckets_with_n_entries, ZAP_HISTOGRAM_SIZE, 0);
	}

	/ARGSUSED/
	static void
	dump_none(objset_t os, uint64_t object, void data, size_t size)
	{
	}

	/ARGSUSED/
	static void
	dump_unknown(objset_t os, uint64_t object, void data, size_t size)
	{
	(void) printf("\tUNKNOWN OBJECT TYPE\n");
	}

	/ARGSUSED/
	static void
	dump_uint8(objset_t os, uint64_t object, void data, size_t size)
	{
	}

	/ARGSUSED/
	static void
	dump_uint64(objset_t os, uint64_t object, void data, size_t size)
	{
	uint64_t *arr;
	uint64_t oursize;
	if (dump_opt['d'] < 6)
	return;

	if (data == NULL) {
	dmu_object_info_t doi;

	VERIFY0(dmu_object_info(os, object, &doi));
	size = doi.doi_max_offset;
	/*
	* We cap the size at 1 mebibyte here to prevent
	* allocation failures and nigh-infinite printing if the
	* object is extremely large.
	*/
	oursize = MIN(size, 1 << 20);
	arr = kmem_alloc(oursize, KM_SLEEP);

	int err = dmu_read(os, object, 0, oursize, arr, 0);
	if (err != 0) {
	(void) printf("got error %u from dmu_read\n", err);
	kmem_free(arr, oursize);
	return;
	}
	} else {
	/*
	* Even though the allocation is already done in this code path,
	* we still cap the size to prevent excessive printing.
	*/
	oursize = MIN(size, 1 << 20);
	arr = data;
	}

	if (size == 0) {
	(void) printf("\t\t[]\n");
	return;
	}

	(void) printf("\t\t[%0llx", (u_longlong_t)arr[0]);
	for (size_t i = 1; i * sizeof (uint64_t) < oursize; i++) {
	if (i % 4 != 0)
	(void) printf(", %0llx", (u_longlong_t)arr[i]);
	else
	(void) printf(",\n\t\t%0llx", (u_longlong_t)arr[i]);
	}
	if (oursize != size)
	(void) printf(", ... ");
	(void) printf("]\n");

	if (data == NULL)
	kmem_free(arr, oursize);
	}

	/ARGSUSED/
	static void
	dump_zap(objset_t os, uint64_t object, void data, size_t size)
	{
	zap_cursor_t zc;
	zap_attribute_t attr;
	void *prop;
	unsigned i;

	dump_zap_stats(os, object);
	(void) printf("\n");

	for (zap_cursor_init(&zc, os, object);
	zap_cursor_retrieve(&zc, &attr) == 0;
	zap_cursor_advance(&zc)) {
	(void) printf("\t\t%s = ", attr.za_name);
	if (attr.za_num_integers == 0) {
	(void) printf("\n");
	continue;
	}
	prop = umem_zalloc(attr.za_num_integers *
	attr.za_integer_length, UMEM_NOFAIL);
	(void) zap_lookup(os, object, attr.za_name,
	attr.za_integer_length, attr.za_num_integers, prop);
	if (attr.za_integer_length == 1) {
	- (void) printf("%s", (char *)prop);
	+ if (strcmp(attr.za_name,
	+ DSL_CRYPTO_KEY_MASTER_KEY) == 0 \|\|
	+ strcmp(attr.za_name,
	+ DSL_CRYPTO_KEY_HMAC_KEY) == 0 \|\|
	+ strcmp(attr.za_name, DSL_CRYPTO_KEY_IV) == 0 \|\|
	+ strcmp(attr.za_name, DSL_CRYPTO_KEY_MAC) == 0 \|\|
	+ strcmp(attr.za_name, DMU_POOL_CHECKSUM_SALT) == 0) {
	+ uint8_t *u8 = prop;
	+
	+ for (i = 0; i < attr.za_num_integers; i++) {
	+ (void) printf("%02x", u8[i]);
	+ }
	+ } else {
	+ (void) printf("%s", (char *)prop);
	+ }
	} else {
	for (i = 0; i < attr.za_num_integers; i++) {
	switch (attr.za_integer_length) {
	case 2:
	(void) printf("%u ",
	((uint16_t *)prop)[i]);
	break;
	case 4:
	(void) printf("%u ",
	((uint32_t *)prop)[i]);
	break;
	case 8:
	(void) printf("%lld ",
	(u_longlong_t)((int64_t *)prop)[i]);
	break;
	}
	}
	}
	(void) printf("\n");
	umem_free(prop, attr.za_num_integers * attr.za_integer_length);
	}
	zap_cursor_fini(&zc);
	}

	static void
	dump_bpobj(objset_t os, uint64_t object, void data, size_t size)
	{
	bpobj_phys_t *bpop = data;
	uint64_t i;
	char bytes[32], comp[32], uncomp[32];

	/* make sure the output won't get truncated */
	CTASSERT(sizeof (bytes) >= NN_NUMBUF_SZ);
	CTASSERT(sizeof (comp) >= NN_NUMBUF_SZ);
	CTASSERT(sizeof (uncomp) >= NN_NUMBUF_SZ);

	if (bpop == NULL)
	return;

	zdb_nicenum(bpop->bpo_bytes, bytes, sizeof (bytes));
	zdb_nicenum(bpop->bpo_comp, comp, sizeof (comp));
	zdb_nicenum(bpop->bpo_uncomp, uncomp, sizeof (uncomp));

	(void) printf("\t\tnum_blkptrs = %llu\n",
	(u_longlong_t)bpop->bpo_num_blkptrs);
	(void) printf("\t\tbytes = %s\n", bytes);
	if (size >= BPOBJ_SIZE_V1) {
	(void) printf("\t\tcomp = %s\n", comp);
	(void) printf("\t\tuncomp = %s\n", uncomp);
	}
	if (size >= BPOBJ_SIZE_V2) {
	(void) printf("\t\tsubobjs = %llu\n",
	(u_longlong_t)bpop->bpo_subobjs);
	(void) printf("\t\tnum_subobjs = %llu\n",
	(u_longlong_t)bpop->bpo_num_subobjs);
	}
	if (size >= sizeof (*bpop)) {
	(void) printf("\t\tnum_freed = %llu\n",
	(u_longlong_t)bpop->bpo_num_freed);
	}

	if (dump_opt['d'] < 5)
	return;

	for (i = 0; i < bpop->bpo_num_blkptrs; i++) {
	char blkbuf[BP_SPRINTF_LEN];
	blkptr_t bp;

	int err = dmu_read(os, object,
	i * sizeof (bp), sizeof (bp), &bp, 0);
	if (err != 0) {
	(void) printf("got error %u from dmu_read\n", err);
	break;
	}
	snprintf_blkptr_compact(blkbuf, sizeof (blkbuf), &bp,
	BP_GET_FREE(&bp));
	(void) printf("\t%s\n", blkbuf);
	}
	}

	/* ARGSUSED */
	static void
	dump_bpobj_subobjs(objset_t os, uint64_t object, void data, size_t size)
	{
	dmu_object_info_t doi;
	int64_t i;

	VERIFY0(dmu_object_info(os, object, &doi));
	uint64_t *subobjs = kmem_alloc(doi.doi_max_offset, KM_SLEEP);

	int err = dmu_read(os, object, 0, doi.doi_max_offset, subobjs, 0);
	if (err != 0) {
	(void) printf("got error %u from dmu_read\n", err);
	kmem_free(subobjs, doi.doi_max_offset);
	return;
	}

	int64_t last_nonzero = -1;
	for (i = 0; i < doi.doi_max_offset / 8; i++) {
	if (subobjs[i] != 0)
	last_nonzero = i;
	}

	for (i = 0; i <= last_nonzero; i++) {
	(void) printf("\t%llu\n", (u_longlong_t)subobjs[i]);
	}
	kmem_free(subobjs, doi.doi_max_offset);
	}

	/ARGSUSED/
	static void
	dump_ddt_zap(objset_t os, uint64_t object, void data, size_t size)
	{
	dump_zap_stats(os, object);
	/* contents are printed elsewhere, properly decoded */
	}

	/ARGSUSED/
	static void
	dump_sa_attrs(objset_t os, uint64_t object, void data, size_t size)
	{
	zap_cursor_t zc;
	zap_attribute_t attr;

	dump_zap_stats(os, object);
	(void) printf("\n");

	for (zap_cursor_init(&zc, os, object);
	zap_cursor_retrieve(&zc, &attr) == 0;
	zap_cursor_advance(&zc)) {
	(void) printf("\t\t%s = ", attr.za_name);
	if (attr.za_num_integers == 0) {
	(void) printf("\n");
	continue;
	}
	(void) printf(" %llx : [%d:%d:%d]\n",
	(u_longlong_t)attr.za_first_integer,
	(int)ATTR_LENGTH(attr.za_first_integer),
	(int)ATTR_BSWAP(attr.za_first_integer),
	(int)ATTR_NUM(attr.za_first_integer));
	}
	zap_cursor_fini(&zc);
	}

	/ARGSUSED/
	static void
	dump_sa_layouts(objset_t os, uint64_t object, void data, size_t size)
	{
	zap_cursor_t zc;
	zap_attribute_t attr;
	uint16_t *layout_attrs;
	unsigned i;

	dump_zap_stats(os, object);
	(void) printf("\n");

	for (zap_cursor_init(&zc, os, object);
	zap_cursor_retrieve(&zc, &attr) == 0;
	zap_cursor_advance(&zc)) {
	(void) printf("\t\t%s = [", attr.za_name);
	if (attr.za_num_integers == 0) {
	(void) printf("\n");
	continue;
	}

	VERIFY(attr.za_integer_length == 2);
	layout_attrs = umem_zalloc(attr.za_num_integers *
	attr.za_integer_length, UMEM_NOFAIL);

	VERIFY(zap_lookup(os, object, attr.za_name,
	attr.za_integer_length,
	attr.za_num_integers, layout_attrs) == 0);

	for (i = 0; i != attr.za_num_integers; i++)
	(void) printf(" %d ", (int)layout_attrs[i]);
	(void) printf("]\n");
	umem_free(layout_attrs,
	attr.za_num_integers * attr.za_integer_length);
	}
	zap_cursor_fini(&zc);
	}

	/ARGSUSED/
	static void
	dump_zpldir(objset_t os, uint64_t object, void data, size_t size)
	{
	zap_cursor_t zc;
	zap_attribute_t attr;
	const char *typenames[] = {
	/* 0 */ "not specified",
	/* 1 */ "FIFO",
	/* 2 */ "Character Device",
	/* 3 */ "3 (invalid)",
	/* 4 */ "Directory",
	/* 5 */ "5 (invalid)",
	/* 6 */ "Block Device",
	/* 7 */ "7 (invalid)",
	/* 8 */ "Regular File",
	/* 9 */ "9 (invalid)",
	/* 10 */ "Symbolic Link",
	/* 11 */ "11 (invalid)",
	/* 12 */ "Socket",
	/* 13 */ "Door",
	/* 14 */ "Event Port",
	/* 15 */ "15 (invalid)",
	};

	dump_zap_stats(os, object);
	(void) printf("\n");

	for (zap_cursor_init(&zc, os, object);
	zap_cursor_retrieve(&zc, &attr) == 0;
	zap_cursor_advance(&zc)) {
	(void) printf("\t\t%s = %lld (type: %s)\n",
	attr.za_name, ZFS_DIRENT_OBJ(attr.za_first_integer),
	typenames[ZFS_DIRENT_TYPE(attr.za_first_integer)]);
	}
	zap_cursor_fini(&zc);
	}

	static int
	get_dtl_refcount(vdev_t *vd)
	{
	int refcount = 0;

	if (vd->vdev_ops->vdev_op_leaf) {
	space_map_t *sm = vd->vdev_dtl_sm;

	if (sm != NULL &&
	sm->sm_dbuf->db_size == sizeof (space_map_phys_t))
	return (1);
	return (0);
	}

	for (unsigned c = 0; c < vd->vdev_children; c++)
	refcount += get_dtl_refcount(vd->vdev_child[c]);
	return (refcount);
	}

	static int
	get_metaslab_refcount(vdev_t *vd)
	{
	int refcount = 0;

	if (vd->vdev_top == vd) {
	for (uint64_t m = 0; m < vd->vdev_ms_count; m++) {
	space_map_t *sm = vd->vdev_ms[m]->ms_sm;

	if (sm != NULL &&
	sm->sm_dbuf->db_size == sizeof (space_map_phys_t))
	refcount++;
	}
	}
	for (unsigned c = 0; c < vd->vdev_children; c++)
	refcount += get_metaslab_refcount(vd->vdev_child[c]);

	return (refcount);
	}

	static int
	get_obsolete_refcount(vdev_t *vd)
	{
	uint64_t obsolete_sm_object;
	int refcount = 0;

	VERIFY0(vdev_obsolete_sm_object(vd, &obsolete_sm_object));
	if (vd->vdev_top == vd && obsolete_sm_object != 0) {
	dmu_object_info_t doi;
	VERIFY0(dmu_object_info(vd->vdev_spa->spa_meta_objset,
	obsolete_sm_object, &doi));
	if (doi.doi_bonus_size == sizeof (space_map_phys_t)) {
	refcount++;
	}
	} else {
	ASSERT3P(vd->vdev_obsolete_sm, ==, NULL);
	ASSERT3U(obsolete_sm_object, ==, 0);
	}
	for (unsigned c = 0; c < vd->vdev_children; c++) {
	refcount += get_obsolete_refcount(vd->vdev_child[c]);
	}

	return (refcount);
	}

	static int
	get_prev_obsolete_spacemap_refcount(spa_t *spa)
	{
	uint64_t prev_obj =
	spa->spa_condensing_indirect_phys.scip_prev_obsolete_sm_object;
	if (prev_obj != 0) {
	dmu_object_info_t doi;
	VERIFY0(dmu_object_info(spa->spa_meta_objset, prev_obj, &doi));
	if (doi.doi_bonus_size == sizeof (space_map_phys_t)) {
	return (1);
	}
	}
	return (0);
	}

	static int
	get_checkpoint_refcount(vdev_t *vd)
	{
	int refcount = 0;

	if (vd->vdev_top == vd && vd->vdev_top_zap != 0 &&
	zap_contains(spa_meta_objset(vd->vdev_spa),
	vd->vdev_top_zap, VDEV_TOP_ZAP_POOL_CHECKPOINT_SM) == 0)
	refcount++;

	for (uint64_t c = 0; c < vd->vdev_children; c++)
	refcount += get_checkpoint_refcount(vd->vdev_child[c]);

	return (refcount);
	}

	static int
	get_log_spacemap_refcount(spa_t *spa)
	{
	return (avl_numnodes(&spa->spa_sm_logs_by_txg));
	}

	static int
	verify_spacemap_refcounts(spa_t *spa)
	{
	uint64_t expected_refcount = 0;
	uint64_t actual_refcount;

	(void) feature_get_refcount(spa,
	&spa_feature_table[SPA_FEATURE_SPACEMAP_HISTOGRAM],
	&expected_refcount);
	actual_refcount = get_dtl_refcount(spa->spa_root_vdev);
	actual_refcount += get_metaslab_refcount(spa->spa_root_vdev);
	actual_refcount += get_obsolete_refcount(spa->spa_root_vdev);
	actual_refcount += get_prev_obsolete_spacemap_refcount(spa);
	actual_refcount += get_checkpoint_refcount(spa->spa_root_vdev);
	actual_refcount += get_log_spacemap_refcount(spa);

	if (expected_refcount != actual_refcount) {
	(void) printf("space map refcount mismatch: expected %lld != "
	"actual %lld\n",
	(longlong_t)expected_refcount,
	(longlong_t)actual_refcount);
	return (2);
	}
	return (0);
	}

	static void
	dump_spacemap(objset_t os, space_map_t sm)
	{
	const char *ddata[] = { "ALLOC", "FREE", "CONDENSE", "INVALID",
	"INVALID", "INVALID", "INVALID", "INVALID" };

	if (sm == NULL)
	return;

	(void) printf("space map object %llu:\n",
	(longlong_t)sm->sm_object);
	(void) printf(" smp_length = 0x%llx\n",
	(longlong_t)sm->sm_phys->smp_length);
	(void) printf(" smp_alloc = 0x%llx\n",
	(longlong_t)sm->sm_phys->smp_alloc);

	if (dump_opt['d'] < 6 && dump_opt['m'] < 4)
	return;

	/*
	* Print out the freelist entries in both encoded and decoded form.
	*/
	uint8_t mapshift = sm->sm_shift;
	int64_t alloc = 0;
	uint64_t word, entry_id = 0;
	for (uint64_t offset = 0; offset < space_map_length(sm);
	offset += sizeof (word)) {

	VERIFY0(dmu_read(os, space_map_object(sm), offset,
	sizeof (word), &word, DMU_READ_PREFETCH));

	if (sm_entry_is_debug(word)) {
	uint64_t de_txg = SM_DEBUG_TXG_DECODE(word);
	uint64_t de_sync_pass = SM_DEBUG_SYNCPASS_DECODE(word);
	if (de_txg == 0) {
	(void) printf(
	"\t [%6llu] PADDING\n",
	(u_longlong_t)entry_id);
	} else {
	(void) printf(
	"\t [%6llu] %s: txg %llu pass %llu\n",
	(u_longlong_t)entry_id,
	ddata[SM_DEBUG_ACTION_DECODE(word)],
	(u_longlong_t)de_txg,
	(u_longlong_t)de_sync_pass);
	}
	entry_id++;
	continue;
	}

	uint8_t words;
	char entry_type;
	uint64_t entry_off, entry_run, entry_vdev = SM_NO_VDEVID;

	if (sm_entry_is_single_word(word)) {
	entry_type = (SM_TYPE_DECODE(word) == SM_ALLOC) ?
	'A' : 'F';
	entry_off = (SM_OFFSET_DECODE(word) << mapshift) +
	sm->sm_start;
	entry_run = SM_RUN_DECODE(word) << mapshift;
	words = 1;
	} else {
	/* it is a two-word entry so we read another word */
	ASSERT(sm_entry_is_double_word(word));

	uint64_t extra_word;
	offset += sizeof (extra_word);
	VERIFY0(dmu_read(os, space_map_object(sm), offset,
	sizeof (extra_word), &extra_word,
	DMU_READ_PREFETCH));

	ASSERT3U(offset, <=, space_map_length(sm));

	entry_run = SM2_RUN_DECODE(word) << mapshift;
	entry_vdev = SM2_VDEV_DECODE(word);
	entry_type = (SM2_TYPE_DECODE(extra_word) == SM_ALLOC) ?
	'A' : 'F';
	entry_off = (SM2_OFFSET_DECODE(extra_word) <<
	mapshift) + sm->sm_start;
	words = 2;
	}

	(void) printf("\t [%6llu] %c range:"
	" %010llx-%010llx size: %06llx vdev: %06llu words: %u\n",
	(u_longlong_t)entry_id,
	entry_type, (u_longlong_t)entry_off,
	(u_longlong_t)(entry_off + entry_run),
	(u_longlong_t)entry_run,
	(u_longlong_t)entry_vdev, words);

	if (entry_type == 'A')
	alloc += entry_run;
	else
	alloc -= entry_run;
	entry_id++;
	}
	if (alloc != space_map_allocated(sm)) {
	(void) printf("space_map_object alloc (%lld) INCONSISTENT "
	"with space map summary (%lld)\n",
	(longlong_t)space_map_allocated(sm), (longlong_t)alloc);
	}
	}

	static void
	dump_metaslab_stats(metaslab_t *msp)
	{
	char maxbuf[32];
	range_tree_t *rt = msp->ms_allocatable;
	zfs_btree_t *t = &msp->ms_allocatable_by_size;
	int free_pct = range_tree_space(rt) * 100 / msp->ms_size;

	/* max sure nicenum has enough space */
	CTASSERT(sizeof (maxbuf) >= NN_NUMBUF_SZ);

	zdb_nicenum(metaslab_largest_allocatable(msp), maxbuf, sizeof (maxbuf));

	(void) printf("\t %25s %10lu %7s %6s %4s %4d%%\n",
	"segments", zfs_btree_numnodes(t), "maxsize", maxbuf,
	"freepct", free_pct);
	(void) printf("\tIn-memory histogram:\n");
	dump_histogram(rt->rt_histogram, RANGE_TREE_HISTOGRAM_SIZE, 0);
	}

	static void
	dump_metaslab(metaslab_t *msp)
	{
	vdev_t *vd = msp->ms_group->mg_vd;
	spa_t *spa = vd->vdev_spa;
	space_map_t *sm = msp->ms_sm;
	char freebuf[32];

	zdb_nicenum(msp->ms_size - space_map_allocated(sm), freebuf,
	sizeof (freebuf));

	(void) printf(
	"\tmetaslab %6llu offset %12llx spacemap %6llu free %5s\n",
	(u_longlong_t)msp->ms_id, (u_longlong_t)msp->ms_start,
	(u_longlong_t)space_map_object(sm), freebuf);

	if (dump_opt['m'] > 2 && !dump_opt['L']) {
	mutex_enter(&msp->ms_lock);
	VERIFY0(metaslab_load(msp));
	range_tree_stat_verify(msp->ms_allocatable);
	dump_metaslab_stats(msp);
	metaslab_unload(msp);
	mutex_exit(&msp->ms_lock);
	}

	if (dump_opt['m'] > 1 && sm != NULL &&
	spa_feature_is_active(spa, SPA_FEATURE_SPACEMAP_HISTOGRAM)) {
	/*
	* The space map histogram represents free space in chunks
	* of sm_shift (i.e. bucket 0 refers to 2^sm_shift).
	*/
	(void) printf("\tOn-disk histogram:\t\tfragmentation %llu\n",
	(u_longlong_t)msp->ms_fragmentation);
	dump_histogram(sm->sm_phys->smp_histogram,
	SPACE_MAP_HISTOGRAM_SIZE, sm->sm_shift);
	}

	ASSERT(msp->ms_size == (1ULL << vd->vdev_ms_shift));
	dump_spacemap(spa->spa_meta_objset, msp->ms_sm);

	if (spa_feature_is_active(spa, SPA_FEATURE_LOG_SPACEMAP)) {
	(void) printf("\tFlush data:\n\tunflushed txg=%llu\n\n",
	(u_longlong_t)metaslab_unflushed_txg(msp));
	}
	}

	static void
	print_vdev_metaslab_header(vdev_t *vd)
	{
	vdev_alloc_bias_t alloc_bias = vd->vdev_alloc_bias;
	const char *bias_str = "";
	if (alloc_bias == VDEV_BIAS_LOG \|\| vd->vdev_islog) {
	bias_str = VDEV_ALLOC_BIAS_LOG;
	} else if (alloc_bias == VDEV_BIAS_SPECIAL) {
	bias_str = VDEV_ALLOC_BIAS_SPECIAL;
	} else if (alloc_bias == VDEV_BIAS_DEDUP) {
	bias_str = VDEV_ALLOC_BIAS_DEDUP;
	}

	uint64_t ms_flush_data_obj = 0;
	if (vd->vdev_top_zap != 0) {
	int error = zap_lookup(spa_meta_objset(vd->vdev_spa),
	vd->vdev_top_zap, VDEV_TOP_ZAP_MS_UNFLUSHED_PHYS_TXGS,
	sizeof (uint64_t), 1, &ms_flush_data_obj);
	if (error != ENOENT) {
	ASSERT0(error);
	}
	}

	(void) printf("\tvdev %10llu %s",
	(u_longlong_t)vd->vdev_id, bias_str);

	if (ms_flush_data_obj != 0) {
	(void) printf(" ms_unflushed_phys object %llu",
	(u_longlong_t)ms_flush_data_obj);
	}

	(void) printf("\n\t%-10s%5llu %-19s %-15s %-12s\n",
	"metaslabs", (u_longlong_t)vd->vdev_ms_count,
	"offset", "spacemap", "free");
	(void) printf("\t%15s %19s %15s %12s\n",
	"---------------", "-------------------",
	"---------------", "------------");
	}

	static void
	dump_metaslab_groups(spa_t *spa)
	{
	vdev_t *rvd = spa->spa_root_vdev;
	metaslab_class_t *mc = spa_normal_class(spa);
	uint64_t fragmentation;

	metaslab_class_histogram_verify(mc);

	for (unsigned c = 0; c < rvd->vdev_children; c++) {
	vdev_t *tvd = rvd->vdev_child[c];
	metaslab_group_t *mg = tvd->vdev_mg;

	if (mg == NULL \|\| mg->mg_class != mc)
	continue;

	metaslab_group_histogram_verify(mg);
	mg->mg_fragmentation = metaslab_group_fragmentation(mg);

	(void) printf("\tvdev %10llu\t\tmetaslabs%5llu\t\t"
	"fragmentation",
	(u_longlong_t)tvd->vdev_id,
	(u_longlong_t)tvd->vdev_ms_count);
	if (mg->mg_fragmentation == ZFS_FRAG_INVALID) {
	(void) printf("%3s\n", "-");
	} else {
	(void) printf("%3llu%%\n",
	(u_longlong_t)mg->mg_fragmentation);
	}
	dump_histogram(mg->mg_histogram, RANGE_TREE_HISTOGRAM_SIZE, 0);
	}

	(void) printf("\tpool %s\tfragmentation", spa_name(spa));
	fragmentation = metaslab_class_fragmentation(mc);
	if (fragmentation == ZFS_FRAG_INVALID)
	(void) printf("\t%3s\n", "-");
	else
	(void) printf("\t%3llu%%\n", (u_longlong_t)fragmentation);
	dump_histogram(mc->mc_histogram, RANGE_TREE_HISTOGRAM_SIZE, 0);
	}

	static void
	print_vdev_indirect(vdev_t *vd)
	{
	vdev_indirect_config_t *vic = &vd->vdev_indirect_config;
	vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping;
	vdev_indirect_births_t *vib = vd->vdev_indirect_births;

	if (vim == NULL) {
	ASSERT3P(vib, ==, NULL);
	return;
	}

	ASSERT3U(vdev_indirect_mapping_object(vim), ==,
	vic->vic_mapping_object);
	ASSERT3U(vdev_indirect_births_object(vib), ==,
	vic->vic_births_object);

	(void) printf("indirect births obj %llu:\n",
	(longlong_t)vic->vic_births_object);
	(void) printf(" vib_count = %llu\n",
	(longlong_t)vdev_indirect_births_count(vib));
	for (uint64_t i = 0; i < vdev_indirect_births_count(vib); i++) {
	vdev_indirect_birth_entry_phys_t *cur_vibe =
	&vib->vib_entries[i];
	(void) printf("\toffset %llx -> txg %llu\n",
	(longlong_t)cur_vibe->vibe_offset,
	(longlong_t)cur_vibe->vibe_phys_birth_txg);
	}
	(void) printf("\n");

	(void) printf("indirect mapping obj %llu:\n",
	(longlong_t)vic->vic_mapping_object);
	(void) printf(" vim_max_offset = 0x%llx\n",
	(longlong_t)vdev_indirect_mapping_max_offset(vim));
	(void) printf(" vim_bytes_mapped = 0x%llx\n",
	(longlong_t)vdev_indirect_mapping_bytes_mapped(vim));
	(void) printf(" vim_count = %llu\n",
	(longlong_t)vdev_indirect_mapping_num_entries(vim));

	if (dump_opt['d'] <= 5 && dump_opt['m'] <= 3)
	return;

	uint32_t *counts = vdev_indirect_mapping_load_obsolete_counts(vim);

	for (uint64_t i = 0; i < vdev_indirect_mapping_num_entries(vim); i++) {
	vdev_indirect_mapping_entry_phys_t *vimep =
	&vim->vim_entries[i];
	(void) printf("\t<%llx:%llx:%llx> -> "
	"<%llx:%llx:%llx> (%x obsolete)\n",
	(longlong_t)vd->vdev_id,
	(longlong_t)DVA_MAPPING_GET_SRC_OFFSET(vimep),
	(longlong_t)DVA_GET_ASIZE(&vimep->vimep_dst),
	(longlong_t)DVA_GET_VDEV(&vimep->vimep_dst),
	(longlong_t)DVA_GET_OFFSET(&vimep->vimep_dst),
	(longlong_t)DVA_GET_ASIZE(&vimep->vimep_dst),
	counts[i]);
	}
	(void) printf("\n");

	uint64_t obsolete_sm_object;
	VERIFY0(vdev_obsolete_sm_object(vd, &obsolete_sm_object));
	if (obsolete_sm_object != 0) {
	objset_t *mos = vd->vdev_spa->spa_meta_objset;
	(void) printf("obsolete space map object %llu:\n",
	(u_longlong_t)obsolete_sm_object);
	ASSERT(vd->vdev_obsolete_sm != NULL);
	ASSERT3U(space_map_object(vd->vdev_obsolete_sm), ==,
	obsolete_sm_object);
	dump_spacemap(mos, vd->vdev_obsolete_sm);
	(void) printf("\n");
	}
	}

	static void
	dump_metaslabs(spa_t *spa)
	{
	vdev_t vd, rvd = spa->spa_root_vdev;
	uint64_t m, c = 0, children = rvd->vdev_children;

	(void) printf("\nMetaslabs:\n");

	if (!dump_opt['d'] && zopt_metaslab_args > 0) {
	c = zopt_metaslab[0];

	if (c >= children)
	(void) fatal("bad vdev id: %llu", (u_longlong_t)c);

	if (zopt_metaslab_args > 1) {
	vd = rvd->vdev_child[c];
	print_vdev_metaslab_header(vd);

	for (m = 1; m < zopt_metaslab_args; m++) {
	if (zopt_metaslab[m] < vd->vdev_ms_count)
	dump_metaslab(
	vd->vdev_ms[zopt_metaslab[m]]);
	else
	(void) fprintf(stderr, "bad metaslab "
	"number %llu\n",
	(u_longlong_t)zopt_metaslab[m]);
	}
	(void) printf("\n");
	return;
	}
	children = c + 1;
	}
	for (; c < children; c++) {
	vd = rvd->vdev_child[c];
	print_vdev_metaslab_header(vd);

	print_vdev_indirect(vd);

	for (m = 0; m < vd->vdev_ms_count; m++)
	dump_metaslab(vd->vdev_ms[m]);
	(void) printf("\n");
	}
	}

	static void
	dump_log_spacemaps(spa_t *spa)
	{
	if (!spa_feature_is_active(spa, SPA_FEATURE_LOG_SPACEMAP))
	return;

	(void) printf("\nLog Space Maps in Pool:\n");
	for (spa_log_sm_t *sls = avl_first(&spa->spa_sm_logs_by_txg);
	sls; sls = AVL_NEXT(&spa->spa_sm_logs_by_txg, sls)) {
	space_map_t *sm = NULL;
	VERIFY0(space_map_open(&sm, spa_meta_objset(spa),
	sls->sls_sm_obj, 0, UINT64_MAX, SPA_MINBLOCKSHIFT));

	(void) printf("Log Spacemap object %llu txg %llu\n",
	(u_longlong_t)sls->sls_sm_obj, (u_longlong_t)sls->sls_txg);
	dump_spacemap(spa->spa_meta_objset, sm);
	space_map_close(sm);
	}
	(void) printf("\n");
	}

	static void
	dump_dde(const ddt_t ddt, const ddt_entry_t dde, uint64_t index)
	{
	const ddt_phys_t *ddp = dde->dde_phys;
	const ddt_key_t *ddk = &dde->dde_key;
	const char *types[4] = { "ditto", "single", "double", "triple" };
	char blkbuf[BP_SPRINTF_LEN];
	blkptr_t blk;
	int p;

	for (p = 0; p < DDT_PHYS_TYPES; p++, ddp++) {
	if (ddp->ddp_phys_birth == 0)
	continue;
	ddt_bp_create(ddt->ddt_checksum, ddk, ddp, &blk);
	snprintf_blkptr(blkbuf, sizeof (blkbuf), &blk);
	(void) printf("index %llx refcnt %llu %s %s\n",
	(u_longlong_t)index, (u_longlong_t)ddp->ddp_refcnt,
	types[p], blkbuf);
	}
	}

	static void
	dump_dedup_ratio(const ddt_stat_t *dds)
	{
	double rL, rP, rD, D, dedup, compress, copies;

	if (dds->dds_blocks == 0)
	return;

	rL = (double)dds->dds_ref_lsize;
	rP = (double)dds->dds_ref_psize;
	rD = (double)dds->dds_ref_dsize;
	D = (double)dds->dds_dsize;

	dedup = rD / D;
	compress = rL / rP;
	copies = rD / rP;

	(void) printf("dedup = %.2f, compress = %.2f, copies = %.2f, "
	"dedup * compress / copies = %.2f\n\n",
	dedup, compress, copies, dedup * compress / copies);
	}

	static void
	dump_ddt(ddt_t *ddt, enum ddt_type type, enum ddt_class class)
	{
	char name[DDT_NAMELEN];
	ddt_entry_t dde;
	uint64_t walk = 0;
	dmu_object_info_t doi;
	uint64_t count, dspace, mspace;
	int error;

	error = ddt_object_info(ddt, type, class, &doi);

	if (error == ENOENT)
	return;
	ASSERT(error == 0);

	error = ddt_object_count(ddt, type, class, &count);
	ASSERT(error == 0);
	if (count == 0)
	return;

	dspace = doi.doi_physical_blocks_512 << 9;
	mspace = doi.doi_fill_count * doi.doi_data_block_size;

	ddt_object_name(ddt, type, class, name);

	(void) printf("%s: %llu entries, size %llu on disk, %llu in core\n",
	name,
	(u_longlong_t)count,
	(u_longlong_t)(dspace / count),
	(u_longlong_t)(mspace / count));

	if (dump_opt['D'] < 3)
	return;

	zpool_dump_ddt(NULL, &ddt->ddt_histogram[type][class]);

	if (dump_opt['D'] < 4)
	return;

	if (dump_opt['D'] < 5 && class == DDT_CLASS_UNIQUE)
	return;

	(void) printf("%s contents:\n\n", name);

	while ((error = ddt_object_walk(ddt, type, class, &walk, &dde)) == 0)
	dump_dde(ddt, &dde, walk);

	ASSERT3U(error, ==, ENOENT);

	(void) printf("\n");
	}

	static void
	dump_all_ddts(spa_t *spa)
	{
	ddt_histogram_t ddh_total;
	ddt_stat_t dds_total;

	bzero(&ddh_total, sizeof (ddh_total));
	bzero(&dds_total, sizeof (dds_total));

	for (enum zio_checksum c = 0; c < ZIO_CHECKSUM_FUNCTIONS; c++) {
	ddt_t *ddt = spa->spa_ddt[c];
	for (enum ddt_type type = 0; type < DDT_TYPES; type++) {
	for (enum ddt_class class = 0; class < DDT_CLASSES;
	class++) {
	dump_ddt(ddt, type, class);
	}
	}
	}

	ddt_get_dedup_stats(spa, &dds_total);

	if (dds_total.dds_blocks == 0) {
	(void) printf("All DDTs are empty\n");
	return;
	}

	(void) printf("\n");

	if (dump_opt['D'] > 1) {
	(void) printf("DDT histogram (aggregated over all DDTs):\n");
	ddt_get_dedup_histogram(spa, &ddh_total);
	zpool_dump_ddt(&dds_total, &ddh_total);
	}

	dump_dedup_ratio(&dds_total);
	}

	static void
	dump_dtl_seg(void *arg, uint64_t start, uint64_t size)
	{
	char *prefix = arg;

	(void) printf("%s [%llu,%llu) length %llu\n",
	prefix,
	(u_longlong_t)start,
	(u_longlong_t)(start + size),
	(u_longlong_t)(size));
	}

	static void
	dump_dtl(vdev_t *vd, int indent)
	{
	spa_t *spa = vd->vdev_spa;
	boolean_t required;
	const char *name[DTL_TYPES] = { "missing", "partial", "scrub",
	"outage" };
	char prefix[256];

	spa_vdev_state_enter(spa, SCL_NONE);
	required = vdev_dtl_required(vd);
	(void) spa_vdev_state_exit(spa, NULL, 0);

	if (indent == 0)
	(void) printf("\nDirty time logs:\n\n");

	(void) printf("\t%*s%s [%s]\n", indent, "",
	vd->vdev_path ? vd->vdev_path :
	vd->vdev_parent ? vd->vdev_ops->vdev_op_type : spa_name(spa),
	required ? "DTL-required" : "DTL-expendable");

	for (int t = 0; t < DTL_TYPES; t++) {
	range_tree_t *rt = vd->vdev_dtl[t];
	if (range_tree_space(rt) == 0)
	continue;
	(void) snprintf(prefix, sizeof (prefix), "\t%*s%s",
	indent + 2, "", name[t]);
	range_tree_walk(rt, dump_dtl_seg, prefix);
	if (dump_opt['d'] > 5 && vd->vdev_children == 0)
	dump_spacemap(spa->spa_meta_objset,
	vd->vdev_dtl_sm);
	}

	for (unsigned c = 0; c < vd->vdev_children; c++)
	dump_dtl(vd->vdev_child[c], indent + 4);
	}

	static void
	dump_history(spa_t *spa)
	{
	nvlist_t **events = NULL;
	char *buf;
	uint64_t resid, len, off = 0;
	uint_t num = 0;
	int error;
	time_t tsec;
	struct tm t;
	char tbuf[30];
	char internalstr[MAXPATHLEN];

	if ((buf = malloc(SPA_OLD_MAXBLOCKSIZE)) == NULL) {
	(void) fprintf(stderr, "%s: unable to allocate I/O buffer\n",
	__func__);
	return;
	}

	do {
	len = SPA_OLD_MAXBLOCKSIZE;

	if ((error = spa_history_get(spa, &off, &len, buf)) != 0) {
	(void) fprintf(stderr, "Unable to read history: "
	"error %d\n", error);
	free(buf);
	return;
	}

	if (zpool_history_unpack(buf, len, &resid, &events, &num) != 0)
	break;

	off -= resid;
	} while (len != 0);

	(void) printf("\nHistory:\n");
	for (unsigned i = 0; i < num; i++) {
	uint64_t time, txg, ievent;
	char cmd, intstr;
	boolean_t printed = B_FALSE;

	if (nvlist_lookup_uint64(events[i], ZPOOL_HIST_TIME,
	&time) != 0)
	goto next;
	if (nvlist_lookup_string(events[i], ZPOOL_HIST_CMD,
	&cmd) != 0) {
	if (nvlist_lookup_uint64(events[i],
	ZPOOL_HIST_INT_EVENT, &ievent) != 0)
	goto next;
	verify(nvlist_lookup_uint64(events[i],
	ZPOOL_HIST_TXG, &txg) == 0);
	verify(nvlist_lookup_string(events[i],
	ZPOOL_HIST_INT_STR, &intstr) == 0);
	if (ievent >= ZFS_NUM_LEGACY_HISTORY_EVENTS)
	goto next;

	(void) snprintf(internalstr,
	sizeof (internalstr),
	"[internal %s txg:%lld] %s",
	zfs_history_event_names[ievent],
	(longlong_t)txg, intstr);
	cmd = internalstr;
	}
	tsec = time;
	(void) localtime_r(&tsec, &t);
	(void) strftime(tbuf, sizeof (tbuf), "%F.%T", &t);
	(void) printf("%s %s\n", tbuf, cmd);
	printed = B_TRUE;

	next:
	if (dump_opt['h'] > 1) {
	if (!printed)
	(void) printf("unrecognized record:\n");
	dump_nvlist(events[i], 2);
	}
	}
	free(buf);
	}

	/ARGSUSED/
	static void
	dump_dnode(objset_t os, uint64_t object, void data, size_t size)
	{
	}

	static uint64_t
	blkid2offset(const dnode_phys_t dnp, const blkptr_t bp,
	const zbookmark_phys_t *zb)
	{
	if (dnp == NULL) {
	ASSERT(zb->zb_level < 0);
	if (zb->zb_object == 0)
	return (zb->zb_blkid);
	return (zb->zb_blkid * BP_GET_LSIZE(bp));
	}

	ASSERT(zb->zb_level >= 0);

	return ((zb->zb_blkid <<
	(zb->zb_level * (dnp->dn_indblkshift - SPA_BLKPTRSHIFT))) *
	dnp->dn_datablkszsec << SPA_MINBLOCKSHIFT);
	}

	static void
	snprintf_zstd_header(spa_t spa, char blkbuf, size_t buflen,
	const blkptr_t *bp)
	{
	abd_t *pabd;
	void *buf;
	zio_t *zio;
	zfs_zstdhdr_t zstd_hdr;
	int error;

	if (BP_GET_COMPRESS(bp) != ZIO_COMPRESS_ZSTD)
	return;

	if (BP_IS_HOLE(bp))
	return;

	if (BP_IS_EMBEDDED(bp)) {
	buf = malloc(SPA_MAXBLOCKSIZE);
	if (buf == NULL) {
	(void) fprintf(stderr, "out of memory\n");
	exit(1);
	}
	decode_embedded_bp_compressed(bp, buf);
	memcpy(&zstd_hdr, buf, sizeof (zstd_hdr));
	free(buf);
	zstd_hdr.c_len = BE_32(zstd_hdr.c_len);
	zstd_hdr.raw_version_level = BE_32(zstd_hdr.raw_version_level);
	(void) snprintf(blkbuf + strlen(blkbuf),
	buflen - strlen(blkbuf),
	" ZSTD:size=%u:version=%u:level=%u:EMBEDDED",
	zstd_hdr.c_len, zstd_hdr.version, zstd_hdr.level);
	return;
	}

	pabd = abd_alloc_for_io(SPA_MAXBLOCKSIZE, B_FALSE);
	zio = zio_root(spa, NULL, NULL, 0);

	/* Decrypt but don't decompress so we can read the compression header */
	zio_nowait(zio_read(zio, spa, bp, pabd, BP_GET_PSIZE(bp), NULL, NULL,
	ZIO_PRIORITY_SYNC_READ, ZIO_FLAG_CANFAIL \| ZIO_FLAG_RAW_COMPRESS,
	NULL));
	error = zio_wait(zio);
	if (error) {
	(void) fprintf(stderr, "read failed: %d\n", error);
	return;
	}
	buf = abd_borrow_buf_copy(pabd, BP_GET_LSIZE(bp));
	memcpy(&zstd_hdr, buf, sizeof (zstd_hdr));
	zstd_hdr.c_len = BE_32(zstd_hdr.c_len);
	zstd_hdr.raw_version_level = BE_32(zstd_hdr.raw_version_level);

	(void) snprintf(blkbuf + strlen(blkbuf),
	buflen - strlen(blkbuf),
	" ZSTD:size=%u:version=%u:level=%u:NORMAL",
	zstd_hdr.c_len, zstd_hdr.version, zstd_hdr.level);

	abd_return_buf_copy(pabd, buf, BP_GET_LSIZE(bp));
	}

	static void
	snprintf_blkptr_compact(char blkbuf, size_t buflen, const blkptr_t bp,
	boolean_t bp_freed)
	{
	const dva_t *dva = bp->blk_dva;
	int ndvas = dump_opt['d'] > 5 ? BP_GET_NDVAS(bp) : 1;
	int i;

	if (dump_opt['b'] >= 6) {
	snprintf_blkptr(blkbuf, buflen, bp);
	if (bp_freed) {
	(void) snprintf(blkbuf + strlen(blkbuf),
	buflen - strlen(blkbuf), " %s", "FREE");
	}
	return;
	}

	if (BP_IS_EMBEDDED(bp)) {
	(void) sprintf(blkbuf,
	"EMBEDDED et=%u %llxL/%llxP B=%llu",
	(int)BPE_GET_ETYPE(bp),
	(u_longlong_t)BPE_GET_LSIZE(bp),
	(u_longlong_t)BPE_GET_PSIZE(bp),
	(u_longlong_t)bp->blk_birth);
	return;
	}

	blkbuf[0] = '\0';

	for (i = 0; i < ndvas; i++)
	(void) snprintf(blkbuf + strlen(blkbuf),
	buflen - strlen(blkbuf), "%llu:%llx:%llx ",
	(u_longlong_t)DVA_GET_VDEV(&dva[i]),
	(u_longlong_t)DVA_GET_OFFSET(&dva[i]),
	(u_longlong_t)DVA_GET_ASIZE(&dva[i]));

	if (BP_IS_HOLE(bp)) {
	(void) snprintf(blkbuf + strlen(blkbuf),
	buflen - strlen(blkbuf),
	"%llxL B=%llu",
	(u_longlong_t)BP_GET_LSIZE(bp),
	(u_longlong_t)bp->blk_birth);
	} else {
	(void) snprintf(blkbuf + strlen(blkbuf),
	buflen - strlen(blkbuf),
	"%llxL/%llxP F=%llu B=%llu/%llu",
	(u_longlong_t)BP_GET_LSIZE(bp),
	(u_longlong_t)BP_GET_PSIZE(bp),
	(u_longlong_t)BP_GET_FILL(bp),
	(u_longlong_t)bp->blk_birth,
	(u_longlong_t)BP_PHYSICAL_BIRTH(bp));
	if (bp_freed)
	(void) snprintf(blkbuf + strlen(blkbuf),
	buflen - strlen(blkbuf), " %s", "FREE");
	(void) snprintf(blkbuf + strlen(blkbuf),
	buflen - strlen(blkbuf), " cksum=%llx:%llx:%llx:%llx",
	(u_longlong_t)bp->blk_cksum.zc_word[0],
	(u_longlong_t)bp->blk_cksum.zc_word[1],
	(u_longlong_t)bp->blk_cksum.zc_word[2],
	(u_longlong_t)bp->blk_cksum.zc_word[3]);
	}
	}

	static void
	print_indirect(spa_t spa, blkptr_t bp, const zbookmark_phys_t *zb,
	const dnode_phys_t *dnp)
	{
	char blkbuf[BP_SPRINTF_LEN];
	int l;

	if (!BP_IS_EMBEDDED(bp)) {
	ASSERT3U(BP_GET_TYPE(bp), ==, dnp->dn_type);
	ASSERT3U(BP_GET_LEVEL(bp), ==, zb->zb_level);
	}

	(void) printf("%16llx ", (u_longlong_t)blkid2offset(dnp, bp, zb));

	ASSERT(zb->zb_level >= 0);

	for (l = dnp->dn_nlevels - 1; l >= -1; l--) {
	if (l == zb->zb_level) {
	(void) printf("L%llx", (u_longlong_t)zb->zb_level);
	} else {
	(void) printf(" ");
	}
	}

	snprintf_blkptr_compact(blkbuf, sizeof (blkbuf), bp, B_FALSE);
	if (dump_opt['Z'] && BP_GET_COMPRESS(bp) == ZIO_COMPRESS_ZSTD)
	snprintf_zstd_header(spa, blkbuf, sizeof (blkbuf), bp);
	(void) printf("%s\n", blkbuf);
	}

	static int
	visit_indirect(spa_t spa, const dnode_phys_t dnp,
	blkptr_t bp, const zbookmark_phys_t zb)
	{
	int err = 0;

	if (bp->blk_birth == 0)
	return (0);

	print_indirect(spa, bp, zb, dnp);

	if (BP_GET_LEVEL(bp) > 0 && !BP_IS_HOLE(bp)) {
	arc_flags_t flags = ARC_FLAG_WAIT;
	int i;
	blkptr_t *cbp;
	int epb = BP_GET_LSIZE(bp) >> SPA_BLKPTRSHIFT;
	arc_buf_t *buf;
	uint64_t fill = 0;
	ASSERT(!BP_IS_REDACTED(bp));

	err = arc_read(NULL, spa, bp, arc_getbuf_func, &buf,
	ZIO_PRIORITY_ASYNC_READ, ZIO_FLAG_CANFAIL, &flags, zb);
	if (err)
	return (err);
	ASSERT(buf->b_data);

	/* recursively visit blocks below this */
	cbp = buf->b_data;
	for (i = 0; i < epb; i++, cbp++) {
	zbookmark_phys_t czb;

	SET_BOOKMARK(&czb, zb->zb_objset, zb->zb_object,
	zb->zb_level - 1,
	zb->zb_blkid * epb + i);
	err = visit_indirect(spa, dnp, cbp, &czb);
	if (err)
	break;
	fill += BP_GET_FILL(cbp);
	}
	if (!err)
	ASSERT3U(fill, ==, BP_GET_FILL(bp));
	arc_buf_destroy(buf, &buf);
	}

	return (err);
	}

	/ARGSUSED/
	static void
	dump_indirect(dnode_t *dn)
	{
	dnode_phys_t *dnp = dn->dn_phys;
	int j;
	zbookmark_phys_t czb;

	(void) printf("Indirect blocks:\n");

	SET_BOOKMARK(&czb, dmu_objset_id(dn->dn_objset),
	dn->dn_object, dnp->dn_nlevels - 1, 0);
	for (j = 0; j < dnp->dn_nblkptr; j++) {
	czb.zb_blkid = j;
	(void) visit_indirect(dmu_objset_spa(dn->dn_objset), dnp,
	&dnp->dn_blkptr[j], &czb);
	}

	(void) printf("\n");
	}

	/ARGSUSED/
	static void
	dump_dsl_dir(objset_t os, uint64_t object, void data, size_t size)
	{
	dsl_dir_phys_t *dd = data;
	time_t crtime;
	char nice[32];

	/* make sure nicenum has enough space */
	CTASSERT(sizeof (nice) >= NN_NUMBUF_SZ);

	if (dd == NULL)
	return;

	ASSERT3U(size, >=, sizeof (dsl_dir_phys_t));

	crtime = dd->dd_creation_time;
	(void) printf("\t\tcreation_time = %s", ctime(&crtime));
	(void) printf("\t\thead_dataset_obj = %llu\n",
	(u_longlong_t)dd->dd_head_dataset_obj);
	(void) printf("\t\tparent_dir_obj = %llu\n",
	(u_longlong_t)dd->dd_parent_obj);
	(void) printf("\t\torigin_obj = %llu\n",
	(u_longlong_t)dd->dd_origin_obj);
	(void) printf("\t\tchild_dir_zapobj = %llu\n",
	(u_longlong_t)dd->dd_child_dir_zapobj);
	zdb_nicenum(dd->dd_used_bytes, nice, sizeof (nice));
	(void) printf("\t\tused_bytes = %s\n", nice);
	zdb_nicenum(dd->dd_compressed_bytes, nice, sizeof (nice));
	(void) printf("\t\tcompressed_bytes = %s\n", nice);
	zdb_nicenum(dd->dd_uncompressed_bytes, nice, sizeof (nice));
	(void) printf("\t\tuncompressed_bytes = %s\n", nice);
	zdb_nicenum(dd->dd_quota, nice, sizeof (nice));
	(void) printf("\t\tquota = %s\n", nice);
	zdb_nicenum(dd->dd_reserved, nice, sizeof (nice));
	(void) printf("\t\treserved = %s\n", nice);
	(void) printf("\t\tprops_zapobj = %llu\n",
	(u_longlong_t)dd->dd_props_zapobj);
	(void) printf("\t\tdeleg_zapobj = %llu\n",
	(u_longlong_t)dd->dd_deleg_zapobj);
	(void) printf("\t\tflags = %llx\n",
	(u_longlong_t)dd->dd_flags);

	#define DO(which) \
	zdb_nicenum(dd->dd_used_breakdown[DD_USED_ ## which], nice, \
	sizeof (nice)); \
	(void) printf("\t\tused_breakdown[" #which "] = %s\n", nice)
	DO(HEAD);
	DO(SNAP);
	DO(CHILD);
	DO(CHILD_RSRV);
	DO(REFRSRV);
	#undef DO
	(void) printf("\t\tclones = %llu\n",
	(u_longlong_t)dd->dd_clones);
	}

	/ARGSUSED/
	static void
	dump_dsl_dataset(objset_t os, uint64_t object, void data, size_t size)
	{
	dsl_dataset_phys_t *ds = data;
	time_t crtime;
	char used[32], compressed[32], uncompressed[32], unique[32];
	char blkbuf[BP_SPRINTF_LEN];

	/* make sure nicenum has enough space */
	CTASSERT(sizeof (used) >= NN_NUMBUF_SZ);
	CTASSERT(sizeof (compressed) >= NN_NUMBUF_SZ);
	CTASSERT(sizeof (uncompressed) >= NN_NUMBUF_SZ);
	CTASSERT(sizeof (unique) >= NN_NUMBUF_SZ);

	if (ds == NULL)
	return;

	ASSERT(size == sizeof (*ds));
	crtime = ds->ds_creation_time;
	zdb_nicenum(ds->ds_referenced_bytes, used, sizeof (used));
	zdb_nicenum(ds->ds_compressed_bytes, compressed, sizeof (compressed));
	zdb_nicenum(ds->ds_uncompressed_bytes, uncompressed,
	sizeof (uncompressed));
	zdb_nicenum(ds->ds_unique_bytes, unique, sizeof (unique));
	snprintf_blkptr(blkbuf, sizeof (blkbuf), &ds->ds_bp);

	(void) printf("\t\tdir_obj = %llu\n",
	(u_longlong_t)ds->ds_dir_obj);
	(void) printf("\t\tprev_snap_obj = %llu\n",
	(u_longlong_t)ds->ds_prev_snap_obj);
	(void) printf("\t\tprev_snap_txg = %llu\n",
	(u_longlong_t)ds->ds_prev_snap_txg);
	(void) printf("\t\tnext_snap_obj = %llu\n",
	(u_longlong_t)ds->ds_next_snap_obj);
	(void) printf("\t\tsnapnames_zapobj = %llu\n",
	(u_longlong_t)ds->ds_snapnames_zapobj);
	(void) printf("\t\tnum_children = %llu\n",
	(u_longlong_t)ds->ds_num_children);
	(void) printf("\t\tuserrefs_obj = %llu\n",
	(u_longlong_t)ds->ds_userrefs_obj);
	(void) printf("\t\tcreation_time = %s", ctime(&crtime));
	(void) printf("\t\tcreation_txg = %llu\n",
	(u_longlong_t)ds->ds_creation_txg);
	(void) printf("\t\tdeadlist_obj = %llu\n",
	(u_longlong_t)ds->ds_deadlist_obj);
	(void) printf("\t\tused_bytes = %s\n", used);
	(void) printf("\t\tcompressed_bytes = %s\n", compressed);
	(void) printf("\t\tuncompressed_bytes = %s\n", uncompressed);
	(void) printf("\t\tunique = %s\n", unique);
	(void) printf("\t\tfsid_guid = %llu\n",
	(u_longlong_t)ds->ds_fsid_guid);
	(void) printf("\t\tguid = %llu\n",
	(u_longlong_t)ds->ds_guid);
	(void) printf("\t\tflags = %llx\n",
	(u_longlong_t)ds->ds_flags);
	(void) printf("\t\tnext_clones_obj = %llu\n",
	(u_longlong_t)ds->ds_next_clones_obj);
	(void) printf("\t\tprops_obj = %llu\n",
	(u_longlong_t)ds->ds_props_obj);
	(void) printf("\t\tbp = %s\n", blkbuf);
	}

	/* ARGSUSED */
	static int
	dump_bptree_cb(void arg, const blkptr_t bp, dmu_tx_t *tx)
	{
	char blkbuf[BP_SPRINTF_LEN];

	if (bp->blk_birth != 0) {
	snprintf_blkptr(blkbuf, sizeof (blkbuf), bp);
	(void) printf("\t%s\n", blkbuf);
	}
	return (0);
	}

	static void
	dump_bptree(objset_t os, uint64_t obj, const char name)
	{
	char bytes[32];
	bptree_phys_t *bt;
	dmu_buf_t *db;

	/* make sure nicenum has enough space */
	CTASSERT(sizeof (bytes) >= NN_NUMBUF_SZ);

	if (dump_opt['d'] < 3)
	return;

	VERIFY3U(0, ==, dmu_bonus_hold(os, obj, FTAG, &db));
	bt = db->db_data;
	zdb_nicenum(bt->bt_bytes, bytes, sizeof (bytes));
	(void) printf("\n %s: %llu datasets, %s\n",
	name, (unsigned long long)(bt->bt_end - bt->bt_begin), bytes);
	dmu_buf_rele(db, FTAG);

	if (dump_opt['d'] < 5)
	return;

	(void) printf("\n");

	(void) bptree_iterate(os, obj, B_FALSE, dump_bptree_cb, NULL, NULL);
	}

	/* ARGSUSED */
	static int
	dump_bpobj_cb(void arg, const blkptr_t bp, boolean_t bp_freed, dmu_tx_t *tx)
	{
	char blkbuf[BP_SPRINTF_LEN];

	ASSERT(bp->blk_birth != 0);
	snprintf_blkptr_compact(blkbuf, sizeof (blkbuf), bp, bp_freed);
	(void) printf("\t%s\n", blkbuf);
	return (0);
	}

	static void
	dump_full_bpobj(bpobj_t bpo, const char name, int indent)
	{
	char bytes[32];
	char comp[32];
	char uncomp[32];
	uint64_t i;

	/* make sure nicenum has enough space */
	CTASSERT(sizeof (bytes) >= NN_NUMBUF_SZ);
	CTASSERT(sizeof (comp) >= NN_NUMBUF_SZ);
	CTASSERT(sizeof (uncomp) >= NN_NUMBUF_SZ);

	if (dump_opt['d'] < 3)
	return;

	zdb_nicenum(bpo->bpo_phys->bpo_bytes, bytes, sizeof (bytes));
	if (bpo->bpo_havesubobj && bpo->bpo_phys->bpo_subobjs != 0) {
	zdb_nicenum(bpo->bpo_phys->bpo_comp, comp, sizeof (comp));
	zdb_nicenum(bpo->bpo_phys->bpo_uncomp, uncomp, sizeof (uncomp));
	if (bpo->bpo_havefreed) {
	(void) printf(" %*s: object %llu, %llu local "
	"blkptrs, %llu freed, %llu subobjs in object %llu, "
	"%s (%s/%s comp)\n",
	indent * 8, name,
	(u_longlong_t)bpo->bpo_object,
	(u_longlong_t)bpo->bpo_phys->bpo_num_blkptrs,
	(u_longlong_t)bpo->bpo_phys->bpo_num_freed,
	(u_longlong_t)bpo->bpo_phys->bpo_num_subobjs,
	(u_longlong_t)bpo->bpo_phys->bpo_subobjs,
	bytes, comp, uncomp);
	} else {
	(void) printf(" %*s: object %llu, %llu local "
	"blkptrs, %llu subobjs in object %llu, "
	"%s (%s/%s comp)\n",
	indent * 8, name,
	(u_longlong_t)bpo->bpo_object,
	(u_longlong_t)bpo->bpo_phys->bpo_num_blkptrs,
	(u_longlong_t)bpo->bpo_phys->bpo_num_subobjs,
	(u_longlong_t)bpo->bpo_phys->bpo_subobjs,
	bytes, comp, uncomp);
	}

	for (i = 0; i < bpo->bpo_phys->bpo_num_subobjs; i++) {
	uint64_t subobj;
	bpobj_t subbpo;
	int error;
	VERIFY0(dmu_read(bpo->bpo_os,
	bpo->bpo_phys->bpo_subobjs,
	i * sizeof (subobj), sizeof (subobj), &subobj, 0));
	error = bpobj_open(&subbpo, bpo->bpo_os, subobj);
	if (error != 0) {
	(void) printf("ERROR %u while trying to open "
	"subobj id %llu\n",
	error, (u_longlong_t)subobj);
	continue;
	}
	dump_full_bpobj(&subbpo, "subobj", indent + 1);
	bpobj_close(&subbpo);
	}
	} else {
	if (bpo->bpo_havefreed) {
	(void) printf(" %*s: object %llu, %llu blkptrs, "
	"%llu freed, %s\n",
	indent * 8, name,
	(u_longlong_t)bpo->bpo_object,
	(u_longlong_t)bpo->bpo_phys->bpo_num_blkptrs,
	(u_longlong_t)bpo->bpo_phys->bpo_num_freed,
	bytes);
	} else {
	(void) printf(" %*s: object %llu, %llu blkptrs, "
	"%s\n",
	indent * 8, name,
	(u_longlong_t)bpo->bpo_object,
	(u_longlong_t)bpo->bpo_phys->bpo_num_blkptrs,
	bytes);
	}
	}

	if (dump_opt['d'] < 5)
	return;


	if (indent == 0) {
	(void) bpobj_iterate_nofree(bpo, dump_bpobj_cb, NULL, NULL);
	(void) printf("\n");
	}
	}

	static int
	dump_bookmark(dsl_pool_t dp, char name, boolean_t print_redact,
	boolean_t print_list)
	{
	int err = 0;
	zfs_bookmark_phys_t prop;
	objset_t *mos = dp->dp_spa->spa_meta_objset;
	err = dsl_bookmark_lookup(dp, name, NULL, &prop);

	if (err != 0) {
	return (err);
	}

	(void) printf("\t#%s: ", strchr(name, '#') + 1);
	(void) printf("{guid: %llx creation_txg: %llu creation_time: "
	"%llu redaction_obj: %llu}\n", (u_longlong_t)prop.zbm_guid,
	(u_longlong_t)prop.zbm_creation_txg,
	(u_longlong_t)prop.zbm_creation_time,
	(u_longlong_t)prop.zbm_redaction_obj);

	IMPLY(print_list, print_redact);
	if (!print_redact \|\| prop.zbm_redaction_obj == 0)
	return (0);

	redaction_list_t *rl;
	VERIFY0(dsl_redaction_list_hold_obj(dp,
	prop.zbm_redaction_obj, FTAG, &rl));

	redaction_list_phys_t *rlp = rl->rl_phys;
	(void) printf("\tRedacted:\n\t\tProgress: ");
	if (rlp->rlp_last_object != UINT64_MAX \|\|
	rlp->rlp_last_blkid != UINT64_MAX) {
	(void) printf("%llu %llu (incomplete)\n",
	(u_longlong_t)rlp->rlp_last_object,
	(u_longlong_t)rlp->rlp_last_blkid);
	} else {
	(void) printf("complete\n");
	}
	(void) printf("\t\tSnapshots: [");
	for (unsigned int i = 0; i < rlp->rlp_num_snaps; i++) {
	if (i > 0)
	(void) printf(", ");
	(void) printf("%0llu",
	(u_longlong_t)rlp->rlp_snaps[i]);
	}
	(void) printf("]\n\t\tLength: %llu\n",
	(u_longlong_t)rlp->rlp_num_entries);

	if (!print_list) {
	dsl_redaction_list_rele(rl, FTAG);
	return (0);
	}

	if (rlp->rlp_num_entries == 0) {
	dsl_redaction_list_rele(rl, FTAG);
	(void) printf("\t\tRedaction List: []\n\n");
	return (0);
	}

	redact_block_phys_t *rbp_buf;
	uint64_t size;
	dmu_object_info_t doi;

	VERIFY0(dmu_object_info(mos, prop.zbm_redaction_obj, &doi));
	size = doi.doi_max_offset;
	rbp_buf = kmem_alloc(size, KM_SLEEP);

	err = dmu_read(mos, prop.zbm_redaction_obj, 0, size,
	rbp_buf, 0);
	if (err != 0) {
	dsl_redaction_list_rele(rl, FTAG);
	kmem_free(rbp_buf, size);
	return (err);
	}

	(void) printf("\t\tRedaction List: [{object: %llx, offset: "
	"%llx, blksz: %x, count: %llx}",
	(u_longlong_t)rbp_buf[0].rbp_object,
	(u_longlong_t)rbp_buf[0].rbp_blkid,
	(uint_t)(redact_block_get_size(&rbp_buf[0])),
	(u_longlong_t)redact_block_get_count(&rbp_buf[0]));

	for (size_t i = 1; i < rlp->rlp_num_entries; i++) {
	(void) printf(",\n\t\t{object: %llx, offset: %llx, "
	"blksz: %x, count: %llx}",
	(u_longlong_t)rbp_buf[i].rbp_object,
	(u_longlong_t)rbp_buf[i].rbp_blkid,
	(uint_t)(redact_block_get_size(&rbp_buf[i])),
	(u_longlong_t)redact_block_get_count(&rbp_buf[i]));
	}
	dsl_redaction_list_rele(rl, FTAG);
	kmem_free(rbp_buf, size);
	(void) printf("]\n\n");
	return (0);
	}

	static void
	dump_bookmarks(objset_t *os, int verbosity)
	{
	zap_cursor_t zc;
	zap_attribute_t attr;
	dsl_dataset_t *ds = dmu_objset_ds(os);
	dsl_pool_t *dp = spa_get_dsl(os->os_spa);
	objset_t *mos = os->os_spa->spa_meta_objset;
	if (verbosity < 4)
	return;
	dsl_pool_config_enter(dp, FTAG);

	for (zap_cursor_init(&zc, mos, ds->ds_bookmarks_obj);
	zap_cursor_retrieve(&zc, &attr) == 0;
	zap_cursor_advance(&zc)) {
	char osname[ZFS_MAX_DATASET_NAME_LEN];
	char buf[ZFS_MAX_DATASET_NAME_LEN];
	dmu_objset_name(os, osname);
	VERIFY3S(0, <=, snprintf(buf, sizeof (buf), "%s#%s", osname,
	attr.za_name));
	(void) dump_bookmark(dp, buf, verbosity >= 5, verbosity >= 6);
	}
	zap_cursor_fini(&zc);
	dsl_pool_config_exit(dp, FTAG);
	}

	static void
	bpobj_count_refd(bpobj_t *bpo)
	{
	mos_obj_refd(bpo->bpo_object);

	if (bpo->bpo_havesubobj && bpo->bpo_phys->bpo_subobjs != 0) {
	mos_obj_refd(bpo->bpo_phys->bpo_subobjs);
	for (uint64_t i = 0; i < bpo->bpo_phys->bpo_num_subobjs; i++) {
	uint64_t subobj;
	bpobj_t subbpo;
	int error;
	VERIFY0(dmu_read(bpo->bpo_os,
	bpo->bpo_phys->bpo_subobjs,
	i * sizeof (subobj), sizeof (subobj), &subobj, 0));
	error = bpobj_open(&subbpo, bpo->bpo_os, subobj);
	if (error != 0) {
	(void) printf("ERROR %u while trying to open "
	"subobj id %llu\n",
	error, (u_longlong_t)subobj);
	continue;
	}
	bpobj_count_refd(&subbpo);
	bpobj_close(&subbpo);
	}
	}
	}

	static int
	dsl_deadlist_entry_count_refd(void arg, dsl_deadlist_entry_t dle)
	{
	spa_t *spa = arg;
	uint64_t empty_bpobj = spa->spa_dsl_pool->dp_empty_bpobj;
	if (dle->dle_bpobj.bpo_object != empty_bpobj)
	bpobj_count_refd(&dle->dle_bpobj);
	return (0);
	}

	static int
	dsl_deadlist_entry_dump(void arg, dsl_deadlist_entry_t dle)
	{
	ASSERT(arg == NULL);
	if (dump_opt['d'] >= 5) {
	char buf[128];
	(void) snprintf(buf, sizeof (buf),
	"mintxg %llu -> obj %llu",
	(longlong_t)dle->dle_mintxg,
	(longlong_t)dle->dle_bpobj.bpo_object);

	dump_full_bpobj(&dle->dle_bpobj, buf, 0);
	} else {
	(void) printf("mintxg %llu -> obj %llu\n",
	(longlong_t)dle->dle_mintxg,
	(longlong_t)dle->dle_bpobj.bpo_object);
	}
	return (0);
	}

	static void
	dump_blkptr_list(dsl_deadlist_t dl, char name)
	{
	char bytes[32];
	char comp[32];
	char uncomp[32];
	char entries[32];
	spa_t *spa = dmu_objset_spa(dl->dl_os);
	uint64_t empty_bpobj = spa->spa_dsl_pool->dp_empty_bpobj;

	if (dl->dl_oldfmt) {
	if (dl->dl_bpobj.bpo_object != empty_bpobj)
	bpobj_count_refd(&dl->dl_bpobj);
	} else {
	mos_obj_refd(dl->dl_object);
	dsl_deadlist_iterate(dl, dsl_deadlist_entry_count_refd, spa);
	}

	/* make sure nicenum has enough space */
	CTASSERT(sizeof (bytes) >= NN_NUMBUF_SZ);
	CTASSERT(sizeof (comp) >= NN_NUMBUF_SZ);
	CTASSERT(sizeof (uncomp) >= NN_NUMBUF_SZ);
	CTASSERT(sizeof (entries) >= NN_NUMBUF_SZ);

	if (dump_opt['d'] < 3)
	return;

	if (dl->dl_oldfmt) {
	dump_full_bpobj(&dl->dl_bpobj, "old-format deadlist", 0);
	return;
	}

	zdb_nicenum(dl->dl_phys->dl_used, bytes, sizeof (bytes));
	zdb_nicenum(dl->dl_phys->dl_comp, comp, sizeof (comp));
	zdb_nicenum(dl->dl_phys->dl_uncomp, uncomp, sizeof (uncomp));
	zdb_nicenum(avl_numnodes(&dl->dl_tree), entries, sizeof (entries));
	(void) printf("\n %s: %s (%s/%s comp), %s entries\n",
	name, bytes, comp, uncomp, entries);

	if (dump_opt['d'] < 4)
	return;

	(void) printf("\n");

	dsl_deadlist_iterate(dl, dsl_deadlist_entry_dump, NULL);
	}

	static int
	verify_dd_livelist(objset_t *os)
	{
	uint64_t ll_used, used, ll_comp, comp, ll_uncomp, uncomp;
	dsl_pool_t *dp = spa_get_dsl(os->os_spa);
	dsl_dir_t *dd = os->os_dsl_dataset->ds_dir;

	ASSERT(!dmu_objset_is_snapshot(os));
	if (!dsl_deadlist_is_open(&dd->dd_livelist))
	return (0);

	/* Iterate through the livelist to check for duplicates */
	dsl_deadlist_iterate(&dd->dd_livelist, sublivelist_verify_lightweight,
	NULL);

	dsl_pool_config_enter(dp, FTAG);
	dsl_deadlist_space(&dd->dd_livelist, &ll_used,
	&ll_comp, &ll_uncomp);

	dsl_dataset_t *origin_ds;
	ASSERT(dsl_pool_config_held(dp));
	VERIFY0(dsl_dataset_hold_obj(dp,
	dsl_dir_phys(dd)->dd_origin_obj, FTAG, &origin_ds));
	VERIFY0(dsl_dataset_space_written(origin_ds, os->os_dsl_dataset,
	&used, &comp, &uncomp));
	dsl_dataset_rele(origin_ds, FTAG);
	dsl_pool_config_exit(dp, FTAG);
	/*
	* It's possible that the dataset's uncomp space is larger than the
	* livelist's because livelists do not track embedded block pointers
	*/
	if (used != ll_used \|\| comp != ll_comp \|\| uncomp < ll_uncomp) {
	char nice_used[32], nice_comp[32], nice_uncomp[32];
	(void) printf("Discrepancy in space accounting:\n");
	zdb_nicenum(used, nice_used, sizeof (nice_used));
	zdb_nicenum(comp, nice_comp, sizeof (nice_comp));
	zdb_nicenum(uncomp, nice_uncomp, sizeof (nice_uncomp));
	(void) printf("dir: used %s, comp %s, uncomp %s\n",
	nice_used, nice_comp, nice_uncomp);
	zdb_nicenum(ll_used, nice_used, sizeof (nice_used));
	zdb_nicenum(ll_comp, nice_comp, sizeof (nice_comp));
	zdb_nicenum(ll_uncomp, nice_uncomp, sizeof (nice_uncomp));
	(void) printf("livelist: used %s, comp %s, uncomp %s\n",
	nice_used, nice_comp, nice_uncomp);
	return (1);
	}
	return (0);
	}

	static avl_tree_t idx_tree;
	static avl_tree_t domain_tree;
	static boolean_t fuid_table_loaded;
	static objset_t *sa_os = NULL;
	static sa_attr_type_t *sa_attr_table = NULL;

	static int
	open_objset(const char path, void tag, objset_t **osp)
	{
	int err;
	uint64_t sa_attrs = 0;
	uint64_t version = 0;

	VERIFY3P(sa_os, ==, NULL);
	/*
	* We can't own an objset if it's redacted. Therefore, we do this
	* dance: hold the objset, then acquire a long hold on its dataset, then
	* release the pool (which is held as part of holding the objset).
	*/
	err = dmu_objset_hold(path, tag, osp);
	if (err != 0) {
	(void) fprintf(stderr, "failed to hold dataset '%s': %s\n",
	path, strerror(err));
	return (err);
	}
	dsl_dataset_long_hold(dmu_objset_ds(*osp), tag);
	dsl_pool_rele(dmu_objset_pool(*osp), tag);

	if (dmu_objset_type(osp) == DMU_OST_ZFS && !(osp)->os_encrypted) {
	(void) zap_lookup(*osp, MASTER_NODE_OBJ, ZPL_VERSION_STR,
	8, 1, &version);
	if (version >= ZPL_VERSION_SA) {
	(void) zap_lookup(*osp, MASTER_NODE_OBJ, ZFS_SA_ATTRS,
	8, 1, &sa_attrs);
	}
	err = sa_setup(*osp, sa_attrs, zfs_attr_table, ZPL_END,
	&sa_attr_table);
	if (err != 0) {
	(void) fprintf(stderr, "sa_setup failed: %s\n",
	strerror(err));
	dsl_dataset_long_rele(dmu_objset_ds(*osp), tag);
	dsl_dataset_rele(dmu_objset_ds(*osp), tag);
	*osp = NULL;
	}
	}
	sa_os = *osp;

	return (0);
	}

	static void
	close_objset(objset_t os, void tag)
	{
	VERIFY3P(os, ==, sa_os);
	if (os->os_sa != NULL)
	sa_tear_down(os);
	dsl_dataset_long_rele(dmu_objset_ds(os), tag);
	dsl_dataset_rele(dmu_objset_ds(os), tag);
	sa_attr_table = NULL;
	sa_os = NULL;
	}

	static void
	fuid_table_destroy(void)
	{
	if (fuid_table_loaded) {
	zfs_fuid_table_destroy(&idx_tree, &domain_tree);
	fuid_table_loaded = B_FALSE;
	}
	}

	/*
	* print uid or gid information.
	* For normal POSIX id just the id is printed in decimal format.
	* For CIFS files with FUID the fuid is printed in hex followed by
	* the domain-rid string.
	*/
	static void
	print_idstr(uint64_t id, const char *id_type)
	{
	if (FUID_INDEX(id)) {
	char *domain;

	domain = zfs_fuid_idx_domain(&idx_tree, FUID_INDEX(id));
	(void) printf("\t%s %llx [%s-%d]\n", id_type,
	(u_longlong_t)id, domain, (int)FUID_RID(id));
	} else {
	(void) printf("\t%s %llu\n", id_type, (u_longlong_t)id);
	}

	}

	static void
	dump_uidgid(objset_t *os, uint64_t uid, uint64_t gid)
	{
	uint32_t uid_idx, gid_idx;

	uid_idx = FUID_INDEX(uid);
	gid_idx = FUID_INDEX(gid);

	/* Load domain table, if not already loaded */
	if (!fuid_table_loaded && (uid_idx \|\| gid_idx)) {
	uint64_t fuid_obj;

	/* first find the fuid object. It lives in the master node */
	VERIFY(zap_lookup(os, MASTER_NODE_OBJ, ZFS_FUID_TABLES,
	8, 1, &fuid_obj) == 0);
	zfs_fuid_avl_tree_create(&idx_tree, &domain_tree);
	(void) zfs_fuid_table_load(os, fuid_obj,
	&idx_tree, &domain_tree);
	fuid_table_loaded = B_TRUE;
	}

	print_idstr(uid, "uid");
	print_idstr(gid, "gid");
	}

	static void
	dump_znode_sa_xattr(sa_handle_t *hdl)
	{
	nvlist_t *sa_xattr;
	nvpair_t *elem = NULL;
	int sa_xattr_size = 0;
	int sa_xattr_entries = 0;
	int error;
	char *sa_xattr_packed;

	error = sa_size(hdl, sa_attr_table[ZPL_DXATTR], &sa_xattr_size);
	if (error \|\| sa_xattr_size == 0)
	return;

	sa_xattr_packed = malloc(sa_xattr_size);
	if (sa_xattr_packed == NULL)
	return;

	error = sa_lookup(hdl, sa_attr_table[ZPL_DXATTR],
	sa_xattr_packed, sa_xattr_size);
	if (error) {
	free(sa_xattr_packed);
	return;
	}

	error = nvlist_unpack(sa_xattr_packed, sa_xattr_size, &sa_xattr, 0);
	if (error) {
	free(sa_xattr_packed);
	return;
	}

	while ((elem = nvlist_next_nvpair(sa_xattr, elem)) != NULL)
	sa_xattr_entries++;

	(void) printf("\tSA xattrs: %d bytes, %d entries\n\n",
	sa_xattr_size, sa_xattr_entries);
	while ((elem = nvlist_next_nvpair(sa_xattr, elem)) != NULL) {
	uchar_t *value;
	uint_t cnt, idx;

	(void) printf("\t\t%s = ", nvpair_name(elem));
	nvpair_value_byte_array(elem, &value, &cnt);
	for (idx = 0; idx < cnt; ++idx) {
	if (isprint(value[idx]))
	(void) putchar(value[idx]);
	else
	(void) printf("\\%3.3o", value[idx]);
	}
	(void) putchar('\n');
	}

	nvlist_free(sa_xattr);
	free(sa_xattr_packed);
	}

	static void
	dump_znode_symlink(sa_handle_t *hdl)
	{
	int sa_symlink_size = 0;
	char linktarget[MAXPATHLEN];
	linktarget[0] = '\0';
	int error;

	error = sa_size(hdl, sa_attr_table[ZPL_SYMLINK], &sa_symlink_size);
	if (error \|\| sa_symlink_size == 0) {
	return;
	}
	if (sa_lookup(hdl, sa_attr_table[ZPL_SYMLINK],
	&linktarget, sa_symlink_size) == 0)
	(void) printf("\ttarget %s\n", linktarget);
	}

	/ARGSUSED/
	static void
	dump_znode(objset_t os, uint64_t object, void data, size_t size)
	{
	char path[MAXPATHLEN * 2]; /* allow for xattr and failure prefix */
	sa_handle_t *hdl;
	uint64_t xattr, rdev, gen;
	uint64_t uid, gid, mode, fsize, parent, links;
	uint64_t pflags;
	uint64_t acctm[2], modtm[2], chgtm[2], crtm[2];
	time_t z_crtime, z_atime, z_mtime, z_ctime;
	sa_bulk_attr_t bulk[12];
	int idx = 0;
	int error;

	VERIFY3P(os, ==, sa_os);
	if (sa_handle_get(os, object, NULL, SA_HDL_PRIVATE, &hdl)) {
	(void) printf("Failed to get handle for SA znode\n");
	return;
	}

	SA_ADD_BULK_ATTR(bulk, idx, sa_attr_table[ZPL_UID], NULL, &uid, 8);
	SA_ADD_BULK_ATTR(bulk, idx, sa_attr_table[ZPL_GID], NULL, &gid, 8);
	SA_ADD_BULK_ATTR(bulk, idx, sa_attr_table[ZPL_LINKS], NULL,
	&links, 8);
	SA_ADD_BULK_ATTR(bulk, idx, sa_attr_table[ZPL_GEN], NULL, &gen, 8);
	SA_ADD_BULK_ATTR(bulk, idx, sa_attr_table[ZPL_MODE], NULL,
	&mode, 8);
	SA_ADD_BULK_ATTR(bulk, idx, sa_attr_table[ZPL_PARENT],
	NULL, &parent, 8);
	SA_ADD_BULK_ATTR(bulk, idx, sa_attr_table[ZPL_SIZE], NULL,
	&fsize, 8);
	SA_ADD_BULK_ATTR(bulk, idx, sa_attr_table[ZPL_ATIME], NULL,
	acctm, 16);
	SA_ADD_BULK_ATTR(bulk, idx, sa_attr_table[ZPL_MTIME], NULL,
	modtm, 16);
	SA_ADD_BULK_ATTR(bulk, idx, sa_attr_table[ZPL_CRTIME], NULL,
	crtm, 16);
	SA_ADD_BULK_ATTR(bulk, idx, sa_attr_table[ZPL_CTIME], NULL,
	chgtm, 16);
	SA_ADD_BULK_ATTR(bulk, idx, sa_attr_table[ZPL_FLAGS], NULL,
	&pflags, 8);

	if (sa_bulk_lookup(hdl, bulk, idx)) {
	(void) sa_handle_destroy(hdl);
	return;
	}

	z_crtime = (time_t)crtm[0];
	z_atime = (time_t)acctm[0];
	z_mtime = (time_t)modtm[0];
	z_ctime = (time_t)chgtm[0];

	if (dump_opt['d'] > 4) {
	error = zfs_obj_to_path(os, object, path, sizeof (path));
	if (error == ESTALE) {
	(void) snprintf(path, sizeof (path), "on delete queue");
	} else if (error != 0) {
	leaked_objects++;
	(void) snprintf(path, sizeof (path),
	"path not found, possibly leaked");
	}
	(void) printf("\tpath %s\n", path);
	}

	if (S_ISLNK(mode))
	dump_znode_symlink(hdl);
	dump_uidgid(os, uid, gid);
	(void) printf("\tatime %s", ctime(&z_atime));
	(void) printf("\tmtime %s", ctime(&z_mtime));
	(void) printf("\tctime %s", ctime(&z_ctime));
	(void) printf("\tcrtime %s", ctime(&z_crtime));
	(void) printf("\tgen %llu\n", (u_longlong_t)gen);
	(void) printf("\tmode %llo\n", (u_longlong_t)mode);
	(void) printf("\tsize %llu\n", (u_longlong_t)fsize);
	(void) printf("\tparent %llu\n", (u_longlong_t)parent);
	(void) printf("\tlinks %llu\n", (u_longlong_t)links);
	(void) printf("\tpflags %llx\n", (u_longlong_t)pflags);
	if (dmu_objset_projectquota_enabled(os) && (pflags & ZFS_PROJID)) {
	uint64_t projid;

	if (sa_lookup(hdl, sa_attr_table[ZPL_PROJID], &projid,
	sizeof (uint64_t)) == 0)
	(void) printf("\tprojid %llu\n", (u_longlong_t)projid);
	}
	if (sa_lookup(hdl, sa_attr_table[ZPL_XATTR], &xattr,
	sizeof (uint64_t)) == 0)
	(void) printf("\txattr %llu\n", (u_longlong_t)xattr);
	if (sa_lookup(hdl, sa_attr_table[ZPL_RDEV], &rdev,
	sizeof (uint64_t)) == 0)
	(void) printf("\trdev 0x%016llx\n", (u_longlong_t)rdev);
	dump_znode_sa_xattr(hdl);
	sa_handle_destroy(hdl);
	}

	/ARGSUSED/
	static void
	dump_acl(objset_t os, uint64_t object, void data, size_t size)
	{
	}

	/ARGSUSED/
	static void
	dump_dmu_objset(objset_t os, uint64_t object, void data, size_t size)
	{
	}

	static object_viewer_t *object_viewer[DMU_OT_NUMTYPES + 1] = {
	dump_none, /* unallocated */
	dump_zap, /* object directory */
	dump_uint64, /* object array */
	dump_none, /* packed nvlist */
	dump_packed_nvlist, /* packed nvlist size */
	dump_none, /* bpobj */
	dump_bpobj, /* bpobj header */
	dump_none, /* SPA space map header */
	dump_none, /* SPA space map */
	dump_none, /* ZIL intent log */
	dump_dnode, /* DMU dnode */
	dump_dmu_objset, /* DMU objset */
	dump_dsl_dir, /* DSL directory */
	dump_zap, /* DSL directory child map */
	dump_zap, /* DSL dataset snap map */
	dump_zap, /* DSL props */
	dump_dsl_dataset, /* DSL dataset */
	dump_znode, /* ZFS znode */
	dump_acl, /* ZFS V0 ACL */
	dump_uint8, /* ZFS plain file */
	dump_zpldir, /* ZFS directory */
	dump_zap, /* ZFS master node */
	dump_zap, /* ZFS delete queue */
	dump_uint8, /* zvol object */
	dump_zap, /* zvol prop */
	dump_uint8, /* other uint8[] */
	dump_uint64, /* other uint64[] */
	dump_zap, /* other ZAP */
	dump_zap, /* persistent error log */
	dump_uint8, /* SPA history */
	dump_history_offsets, /* SPA history offsets */
	dump_zap, /* Pool properties */
	dump_zap, /* DSL permissions */
	dump_acl, /* ZFS ACL */
	dump_uint8, /* ZFS SYSACL */
	dump_none, /* FUID nvlist */
	dump_packed_nvlist, /* FUID nvlist size */
	dump_zap, /* DSL dataset next clones */
	dump_zap, /* DSL scrub queue */
	dump_zap, /* ZFS user/group/project used */
	dump_zap, /* ZFS user/group/project quota */
	dump_zap, /* snapshot refcount tags */
	dump_ddt_zap, /* DDT ZAP object */
	dump_zap, /* DDT statistics */
	dump_znode, /* SA object */
	dump_zap, /* SA Master Node */
	dump_sa_attrs, /* SA attribute registration */
	dump_sa_layouts, /* SA attribute layouts */
	dump_zap, /* DSL scrub translations */
	dump_none, /* fake dedup BP */
	dump_zap, /* deadlist */
	dump_none, /* deadlist hdr */
	dump_zap, /* dsl clones */
	dump_bpobj_subobjs, /* bpobj subobjs */
	dump_unknown, /* Unknown type, must be last */
	};

	static boolean_t
	match_object_type(dmu_object_type_t obj_type, uint64_t flags)
	{
	boolean_t match = B_TRUE;

	switch (obj_type) {
	case DMU_OT_DIRECTORY_CONTENTS:
	if (!(flags & ZOR_FLAG_DIRECTORY))
	match = B_FALSE;
	break;
	case DMU_OT_PLAIN_FILE_CONTENTS:
	if (!(flags & ZOR_FLAG_PLAIN_FILE))
	match = B_FALSE;
	break;
	case DMU_OT_SPACE_MAP:
	if (!(flags & ZOR_FLAG_SPACE_MAP))
	match = B_FALSE;
	break;
	default:
	if (strcmp(zdb_ot_name(obj_type), "zap") == 0) {
	if (!(flags & ZOR_FLAG_ZAP))
	match = B_FALSE;
	break;
	}

	/*
	* If all bits except some of the supported flags are
	* set, the user combined the all-types flag (A) with
	* a negated flag to exclude some types (e.g. A-f to
	* show all object types except plain files).
	*/
	if ((flags \| ZOR_SUPPORTED_FLAGS) != ZOR_FLAG_ALL_TYPES)
	match = B_FALSE;

	break;
	}

	return (match);
	}

	static void
	dump_object(objset_t *os, uint64_t object, int verbosity,
	boolean_t print_header, uint64_t dnode_slots_used, uint64_t flags)
	{
	dmu_buf_t *db = NULL;
	dmu_object_info_t doi;
	dnode_t *dn;
	boolean_t dnode_held = B_FALSE;
	void *bonus = NULL;
	size_t bsize = 0;
	char iblk[32], dblk[32], lsize[32], asize[32], fill[32], dnsize[32];
	char bonus_size[32];
	char aux[50];
	int error;

	/* make sure nicenum has enough space */
	CTASSERT(sizeof (iblk) >= NN_NUMBUF_SZ);
	CTASSERT(sizeof (dblk) >= NN_NUMBUF_SZ);
	CTASSERT(sizeof (lsize) >= NN_NUMBUF_SZ);
	CTASSERT(sizeof (asize) >= NN_NUMBUF_SZ);
	CTASSERT(sizeof (bonus_size) >= NN_NUMBUF_SZ);

	if (*print_header) {
	(void) printf("\n%10s %3s %5s %5s %5s %6s %5s %6s %s\n",
	"Object", "lvl", "iblk", "dblk", "dsize", "dnsize",
	"lsize", "%full", "type");
	*print_header = 0;
	}

	if (object == 0) {
	dn = DMU_META_DNODE(os);
	dmu_object_info_from_dnode(dn, &doi);
	} else {
	/*
	* Encrypted datasets will have sensitive bonus buffers
	* encrypted. Therefore we cannot hold the bonus buffer and
	* must hold the dnode itself instead.
	*/
	error = dmu_object_info(os, object, &doi);
	if (error)
	fatal("dmu_object_info() failed, errno %u", error);

	if (os->os_encrypted &&
	DMU_OT_IS_ENCRYPTED(doi.doi_bonus_type)) {
	error = dnode_hold(os, object, FTAG, &dn);
	if (error)
	fatal("dnode_hold() failed, errno %u", error);
	dnode_held = B_TRUE;
	} else {
	error = dmu_bonus_hold(os, object, FTAG, &db);
	if (error)
	fatal("dmu_bonus_hold(%llu) failed, errno %u",
	object, error);
	bonus = db->db_data;
	bsize = db->db_size;
	dn = DB_DNODE((dmu_buf_impl_t *)db);
	}
	}

	/*
	* Default to showing all object types if no flags were specified.
	*/
	if (flags != 0 && flags != ZOR_FLAG_ALL_TYPES &&
	!match_object_type(doi.doi_type, flags))
	goto out;

	if (dnode_slots_used)
	*dnode_slots_used = doi.doi_dnodesize / DNODE_MIN_SIZE;

	zdb_nicenum(doi.doi_metadata_block_size, iblk, sizeof (iblk));
	zdb_nicenum(doi.doi_data_block_size, dblk, sizeof (dblk));
	zdb_nicenum(doi.doi_max_offset, lsize, sizeof (lsize));
	zdb_nicenum(doi.doi_physical_blocks_512 << 9, asize, sizeof (asize));
	zdb_nicenum(doi.doi_bonus_size, bonus_size, sizeof (bonus_size));
	zdb_nicenum(doi.doi_dnodesize, dnsize, sizeof (dnsize));
	(void) sprintf(fill, "%6.2f", 100.0 * doi.doi_fill_count *
	doi.doi_data_block_size / (object == 0 ? DNODES_PER_BLOCK : 1) /
	doi.doi_max_offset);

	aux[0] = '\0';

	if (doi.doi_checksum != ZIO_CHECKSUM_INHERIT \|\| verbosity >= 6) {
	(void) snprintf(aux + strlen(aux), sizeof (aux) - strlen(aux),
	" (K=%s)", ZDB_CHECKSUM_NAME(doi.doi_checksum));
	}

	if (doi.doi_compress == ZIO_COMPRESS_INHERIT &&
	ZIO_COMPRESS_HASLEVEL(os->os_compress) && verbosity >= 6) {
	const char *compname = NULL;
	if (zfs_prop_index_to_string(ZFS_PROP_COMPRESSION,
	ZIO_COMPRESS_RAW(os->os_compress, os->os_complevel),
	&compname) == 0) {
	(void) snprintf(aux + strlen(aux),
	sizeof (aux) - strlen(aux), " (Z=inherit=%s)",
	compname);
	} else {
	(void) snprintf(aux + strlen(aux),
	sizeof (aux) - strlen(aux),
	" (Z=inherit=%s-unknown)",
	ZDB_COMPRESS_NAME(os->os_compress));
	}
	} else if (doi.doi_compress == ZIO_COMPRESS_INHERIT && verbosity >= 6) {
	(void) snprintf(aux + strlen(aux), sizeof (aux) - strlen(aux),
	" (Z=inherit=%s)", ZDB_COMPRESS_NAME(os->os_compress));
	} else if (doi.doi_compress != ZIO_COMPRESS_INHERIT \|\| verbosity >= 6) {
	(void) snprintf(aux + strlen(aux), sizeof (aux) - strlen(aux),
	" (Z=%s)", ZDB_COMPRESS_NAME(doi.doi_compress));
	}

	(void) printf("%10lld %3u %5s %5s %5s %6s %5s %6s %s%s\n",
	(u_longlong_t)object, doi.doi_indirection, iblk, dblk,
	asize, dnsize, lsize, fill, zdb_ot_name(doi.doi_type), aux);

	if (doi.doi_bonus_type != DMU_OT_NONE && verbosity > 3) {
	(void) printf("%10s %3s %5s %5s %5s %5s %5s %6s %s\n",
	"", "", "", "", "", "", bonus_size, "bonus",
	zdb_ot_name(doi.doi_bonus_type));
	}

	if (verbosity >= 4) {
	(void) printf("\tdnode flags: %s%s%s%s\n",
	(dn->dn_phys->dn_flags & DNODE_FLAG_USED_BYTES) ?
	"USED_BYTES " : "",
	(dn->dn_phys->dn_flags & DNODE_FLAG_USERUSED_ACCOUNTED) ?
	"USERUSED_ACCOUNTED " : "",
	(dn->dn_phys->dn_flags & DNODE_FLAG_USEROBJUSED_ACCOUNTED) ?
	"USEROBJUSED_ACCOUNTED " : "",
	(dn->dn_phys->dn_flags & DNODE_FLAG_SPILL_BLKPTR) ?
	"SPILL_BLKPTR" : "");
	(void) printf("\tdnode maxblkid: %llu\n",
	(longlong_t)dn->dn_phys->dn_maxblkid);

	if (!dnode_held) {
	object_viewer[ZDB_OT_TYPE(doi.doi_bonus_type)](os,
	object, bonus, bsize);
	} else {
	(void) printf("\t\t(bonus encrypted)\n");
	}

	if (!os->os_encrypted \|\| !DMU_OT_IS_ENCRYPTED(doi.doi_type)) {
	object_viewer[ZDB_OT_TYPE(doi.doi_type)](os, object,
	NULL, 0);
	} else {
	(void) printf("\t\t(object encrypted)\n");
	}

	*print_header = B_TRUE;
	}

	if (verbosity >= 5)
	dump_indirect(dn);

	if (verbosity >= 5) {
	/*
	* Report the list of segments that comprise the object.
	*/
	uint64_t start = 0;
	uint64_t end;
	uint64_t blkfill = 1;
	int minlvl = 1;

	if (dn->dn_type == DMU_OT_DNODE) {
	minlvl = 0;
	blkfill = DNODES_PER_BLOCK;
	}

	for (;;) {
	char segsize[32];
	/* make sure nicenum has enough space */
	CTASSERT(sizeof (segsize) >= NN_NUMBUF_SZ);
	error = dnode_next_offset(dn,
	0, &start, minlvl, blkfill, 0);
	if (error)
	break;
	end = start;
	error = dnode_next_offset(dn,
	DNODE_FIND_HOLE, &end, minlvl, blkfill, 0);
	zdb_nicenum(end - start, segsize, sizeof (segsize));
	(void) printf("\t\tsegment [%016llx, %016llx)"
	" size %5s\n", (u_longlong_t)start,
	(u_longlong_t)end, segsize);
	if (error)
	break;
	start = end;
	}
	}

	out:
	if (db != NULL)
	dmu_buf_rele(db, FTAG);
	if (dnode_held)
	dnode_rele(dn, FTAG);
	}

	static void
	count_dir_mos_objects(dsl_dir_t *dd)
	{
	mos_obj_refd(dd->dd_object);
	mos_obj_refd(dsl_dir_phys(dd)->dd_child_dir_zapobj);
	mos_obj_refd(dsl_dir_phys(dd)->dd_deleg_zapobj);
	mos_obj_refd(dsl_dir_phys(dd)->dd_props_zapobj);
	mos_obj_refd(dsl_dir_phys(dd)->dd_clones);

	/*
	* The dd_crypto_obj can be referenced by multiple dsl_dir's.
	* Ignore the references after the first one.
	*/
	mos_obj_refd_multiple(dd->dd_crypto_obj);
	}

	static void
	count_ds_mos_objects(dsl_dataset_t *ds)
	{
	mos_obj_refd(ds->ds_object);
	mos_obj_refd(dsl_dataset_phys(ds)->ds_next_clones_obj);
	mos_obj_refd(dsl_dataset_phys(ds)->ds_props_obj);
	mos_obj_refd(dsl_dataset_phys(ds)->ds_userrefs_obj);
	mos_obj_refd(dsl_dataset_phys(ds)->ds_snapnames_zapobj);
	mos_obj_refd(ds->ds_bookmarks_obj);

	if (!dsl_dataset_is_snapshot(ds)) {
	count_dir_mos_objects(ds->ds_dir);
	}
	}

	static const char *objset_types[DMU_OST_NUMTYPES] = {
	"NONE", "META", "ZPL", "ZVOL", "OTHER", "ANY" };

	/*
	* Parse a string denoting a range of object IDs of the form
	* <start>[:<end>[:flags]], and store the results in zor.
	* Return 0 on success. On error, return 1 and update the msg
	* pointer to point to a descriptive error message.
	*/
	static int
	parse_object_range(char range, zopt_object_range_t zor, char **msg)
	{
	uint64_t flags = 0;
	char p, s, dup, flagstr;
	size_t len;
	int i;
	int rc = 0;

	if (strchr(range, ':') == NULL) {
	zor->zor_obj_start = strtoull(range, &p, 0);
	if (*p != '\0') {
	*msg = "Invalid characters in object ID";
	rc = 1;
	}
	zor->zor_obj_end = zor->zor_obj_start;
	return (rc);
	}

	if (strchr(range, ':') == range) {
	*msg = "Invalid leading colon";
	rc = 1;
	return (rc);
	}

	len = strlen(range);
	if (range[len - 1] == ':') {
	*msg = "Invalid trailing colon";
	rc = 1;
	return (rc);
	}

	dup = strdup(range);
	s = strtok(dup, ":");
	zor->zor_obj_start = strtoull(s, &p, 0);

	if (*p != '\0') {
	*msg = "Invalid characters in start object ID";
	rc = 1;
	goto out;
	}

	s = strtok(NULL, ":");
	zor->zor_obj_end = strtoull(s, &p, 0);

	if (*p != '\0') {
	*msg = "Invalid characters in end object ID";
	rc = 1;
	goto out;
	}

	if (zor->zor_obj_start > zor->zor_obj_end) {
	*msg = "Start object ID may not exceed end object ID";
	rc = 1;
	goto out;
	}

	s = strtok(NULL, ":");
	if (s == NULL) {
	zor->zor_flags = ZOR_FLAG_ALL_TYPES;
	goto out;
	} else if (strtok(NULL, ":") != NULL) {
	*msg = "Invalid colon-delimited field after flags";
	rc = 1;
	goto out;
	}

	flagstr = s;
	for (i = 0; flagstr[i]; i++) {
	int bit;
	boolean_t negation = (flagstr[i] == '-');

	if (negation) {
	i++;
	if (flagstr[i] == '\0') {
	*msg = "Invalid trailing negation operator";
	rc = 1;
	goto out;
	}
	}
	bit = flagbits[(uchar_t)flagstr[i]];
	if (bit == 0) {
	*msg = "Invalid flag";
	rc = 1;
	goto out;
	}
	if (negation)
	flags &= ~bit;
	else
	flags \|= bit;
	}
	zor->zor_flags = flags;

	out:
	free(dup);
	return (rc);
	}

	static void
	dump_objset(objset_t *os)
	{
	dmu_objset_stats_t dds = { 0 };
	uint64_t object, object_count;
	uint64_t refdbytes, usedobjs, scratch;
	char numbuf[32];
	char blkbuf[BP_SPRINTF_LEN + 20];
	char osname[ZFS_MAX_DATASET_NAME_LEN];
	const char *type = "UNKNOWN";
	int verbosity = dump_opt['d'];
	boolean_t print_header;
	unsigned i;
	int error;
	uint64_t total_slots_used = 0;
	uint64_t max_slot_used = 0;
	uint64_t dnode_slots;
	uint64_t obj_start;
	uint64_t obj_end;
	uint64_t flags;

	/* make sure nicenum has enough space */
	CTASSERT(sizeof (numbuf) >= NN_NUMBUF_SZ);

	dsl_pool_config_enter(dmu_objset_pool(os), FTAG);
	dmu_objset_fast_stat(os, &dds);
	dsl_pool_config_exit(dmu_objset_pool(os), FTAG);

	print_header = B_TRUE;

	if (dds.dds_type < DMU_OST_NUMTYPES)
	type = objset_types[dds.dds_type];

	if (dds.dds_type == DMU_OST_META) {
	dds.dds_creation_txg = TXG_INITIAL;
	usedobjs = BP_GET_FILL(os->os_rootbp);
	refdbytes = dsl_dir_phys(os->os_spa->spa_dsl_pool->dp_mos_dir)->
	dd_used_bytes;
	} else {
	dmu_objset_space(os, &refdbytes, &scratch, &usedobjs, &scratch);
	}

	ASSERT3U(usedobjs, ==, BP_GET_FILL(os->os_rootbp));

	zdb_nicenum(refdbytes, numbuf, sizeof (numbuf));

	if (verbosity >= 4) {
	(void) snprintf(blkbuf, sizeof (blkbuf), ", rootbp ");
	(void) snprintf_blkptr(blkbuf + strlen(blkbuf),
	sizeof (blkbuf) - strlen(blkbuf), os->os_rootbp);
	} else {
	blkbuf[0] = '\0';
	}

	dmu_objset_name(os, osname);

	(void) printf("Dataset %s [%s], ID %llu, cr_txg %llu, "
	"%s, %llu objects%s%s\n",
	osname, type, (u_longlong_t)dmu_objset_id(os),
	(u_longlong_t)dds.dds_creation_txg,
	numbuf, (u_longlong_t)usedobjs, blkbuf,
	(dds.dds_inconsistent) ? " (inconsistent)" : "");

	for (i = 0; i < zopt_object_args; i++) {
	obj_start = zopt_object_ranges[i].zor_obj_start;
	obj_end = zopt_object_ranges[i].zor_obj_end;
	flags = zopt_object_ranges[i].zor_flags;

	object = obj_start;
	if (object == 0 \|\| obj_start == obj_end)
	dump_object(os, object, verbosity, &print_header, NULL,
	flags);
	else
	object--;

	while ((dmu_object_next(os, &object, B_FALSE, 0) == 0) &&
	object <= obj_end) {
	dump_object(os, object, verbosity, &print_header, NULL,
	flags);
	}
	}

	if (zopt_object_args > 0) {
	(void) printf("\n");
	return;
	}

	if (dump_opt['i'] != 0 \|\| verbosity >= 2)
	dump_intent_log(dmu_objset_zil(os));

	if (dmu_objset_ds(os) != NULL) {
	dsl_dataset_t *ds = dmu_objset_ds(os);
	dump_blkptr_list(&ds->ds_deadlist, "Deadlist");
	if (dsl_deadlist_is_open(&ds->ds_dir->dd_livelist) &&
	!dmu_objset_is_snapshot(os)) {
	dump_blkptr_list(&ds->ds_dir->dd_livelist, "Livelist");
	if (verify_dd_livelist(os) != 0)
	fatal("livelist is incorrect");
	}

	if (dsl_dataset_remap_deadlist_exists(ds)) {
	(void) printf("ds_remap_deadlist:\n");
	dump_blkptr_list(&ds->ds_remap_deadlist, "Deadlist");
	}
	count_ds_mos_objects(ds);
	}

	if (dmu_objset_ds(os) != NULL)
	dump_bookmarks(os, verbosity);

	if (verbosity < 2)
	return;

	if (BP_IS_HOLE(os->os_rootbp))
	return;

	dump_object(os, 0, verbosity, &print_header, NULL, 0);
	object_count = 0;
	if (DMU_USERUSED_DNODE(os) != NULL &&
	DMU_USERUSED_DNODE(os)->dn_type != 0) {
	dump_object(os, DMU_USERUSED_OBJECT, verbosity, &print_header,
	NULL, 0);
	dump_object(os, DMU_GROUPUSED_OBJECT, verbosity, &print_header,
	NULL, 0);
	}

	if (DMU_PROJECTUSED_DNODE(os) != NULL &&
	DMU_PROJECTUSED_DNODE(os)->dn_type != 0)
	dump_object(os, DMU_PROJECTUSED_OBJECT, verbosity,
	&print_header, NULL, 0);

	object = 0;
	while ((error = dmu_object_next(os, &object, B_FALSE, 0)) == 0) {
	dump_object(os, object, verbosity, &print_header, &dnode_slots,
	0);
	object_count++;
	total_slots_used += dnode_slots;
	max_slot_used = object + dnode_slots - 1;
	}

	(void) printf("\n");

	(void) printf(" Dnode slots:\n");
	(void) printf("\tTotal used: %10llu\n",
	(u_longlong_t)total_slots_used);
	(void) printf("\tMax used: %10llu\n",
	(u_longlong_t)max_slot_used);
	(void) printf("\tPercent empty: %10lf\n",
	(double)(max_slot_used - total_slots_used)*100 /
	(double)max_slot_used);
	(void) printf("\n");

	if (error != ESRCH) {
	(void) fprintf(stderr, "dmu_object_next() = %d\n", error);
	abort();
	}

	ASSERT3U(object_count, ==, usedobjs);

	if (leaked_objects != 0) {
	(void) printf("%d potentially leaked objects detected\n",
	leaked_objects);
	leaked_objects = 0;
	}
	}

	static void
	dump_uberblock(uberblock_t ub, const char header, const char *footer)
	{
	time_t timestamp = ub->ub_timestamp;

	(void) printf("%s", header ? header : "");
	(void) printf("\tmagic = %016llx\n", (u_longlong_t)ub->ub_magic);
	(void) printf("\tversion = %llu\n", (u_longlong_t)ub->ub_version);
	(void) printf("\ttxg = %llu\n", (u_longlong_t)ub->ub_txg);
	(void) printf("\tguid_sum = %llu\n", (u_longlong_t)ub->ub_guid_sum);
	(void) printf("\ttimestamp = %llu UTC = %s",
	(u_longlong_t)ub->ub_timestamp, asctime(localtime(&timestamp)));

	(void) printf("\tmmp_magic = %016llx\n",
	(u_longlong_t)ub->ub_mmp_magic);
	if (MMP_VALID(ub)) {
	(void) printf("\tmmp_delay = %0llu\n",
	(u_longlong_t)ub->ub_mmp_delay);
	if (MMP_SEQ_VALID(ub))
	(void) printf("\tmmp_seq = %u\n",
	(unsigned int) MMP_SEQ(ub));
	if (MMP_FAIL_INT_VALID(ub))
	(void) printf("\tmmp_fail = %u\n",
	(unsigned int) MMP_FAIL_INT(ub));
	if (MMP_INTERVAL_VALID(ub))
	(void) printf("\tmmp_write = %u\n",
	(unsigned int) MMP_INTERVAL(ub));
	/* After MMP_* to make summarize_uberblock_mmp cleaner */
	(void) printf("\tmmp_valid = %x\n",
	(unsigned int) ub->ub_mmp_config & 0xFF);
	}

	if (dump_opt['u'] >= 4) {
	char blkbuf[BP_SPRINTF_LEN];
	snprintf_blkptr(blkbuf, sizeof (blkbuf), &ub->ub_rootbp);
	(void) printf("\trootbp = %s\n", blkbuf);
	}
	(void) printf("\tcheckpoint_txg = %llu\n",
	(u_longlong_t)ub->ub_checkpoint_txg);
	(void) printf("%s", footer ? footer : "");
	}

	static void
	dump_config(spa_t *spa)
	{
	dmu_buf_t *db;
	size_t nvsize = 0;
	int error = 0;


	error = dmu_bonus_hold(spa->spa_meta_objset,
	spa->spa_config_object, FTAG, &db);

	if (error == 0) {
	nvsize = (uint64_t )db->db_data;
	dmu_buf_rele(db, FTAG);

	(void) printf("\nMOS Configuration:\n");
	dump_packed_nvlist(spa->spa_meta_objset,
	spa->spa_config_object, (void *)&nvsize, 1);
	} else {
	(void) fprintf(stderr, "dmu_bonus_hold(%llu) failed, errno %d",
	(u_longlong_t)spa->spa_config_object, error);
	}
	}

	static void
	dump_cachefile(const char *cachefile)
	{
	int fd;
	struct stat64 statbuf;
	char *buf;
	nvlist_t *config;

	if ((fd = open64(cachefile, O_RDONLY)) < 0) {
	(void) printf("cannot open '%s': %s\n", cachefile,
	strerror(errno));
	exit(1);
	}

	if (fstat64(fd, &statbuf) != 0) {
	(void) printf("failed to stat '%s': %s\n", cachefile,
	strerror(errno));
	exit(1);
	}

	if ((buf = malloc(statbuf.st_size)) == NULL) {
	(void) fprintf(stderr, "failed to allocate %llu bytes\n",
	(u_longlong_t)statbuf.st_size);
	exit(1);
	}

	if (read(fd, buf, statbuf.st_size) != statbuf.st_size) {
	(void) fprintf(stderr, "failed to read %llu bytes\n",
	(u_longlong_t)statbuf.st_size);
	exit(1);
	}

	(void) close(fd);

	if (nvlist_unpack(buf, statbuf.st_size, &config, 0) != 0) {
	(void) fprintf(stderr, "failed to unpack nvlist\n");
	exit(1);
	}

	free(buf);

	dump_nvlist(config, 0);

	nvlist_free(config);
	}

	/*
	* ZFS label nvlist stats
	*/
	typedef struct zdb_nvl_stats {
	int zns_list_count;
	int zns_leaf_count;
	size_t zns_leaf_largest;
	size_t zns_leaf_total;
	nvlist_t *zns_string;
	nvlist_t *zns_uint64;
	nvlist_t *zns_boolean;
	} zdb_nvl_stats_t;

	static void
	collect_nvlist_stats(nvlist_t nvl, zdb_nvl_stats_t stats)
	{
	nvlist_t list, *array;
	nvpair_t *nvp = NULL;
	char *name;
	uint_t i, items;

	stats->zns_list_count++;

	while ((nvp = nvlist_next_nvpair(nvl, nvp)) != NULL) {
	name = nvpair_name(nvp);

	switch (nvpair_type(nvp)) {
	case DATA_TYPE_STRING:
	fnvlist_add_string(stats->zns_string, name,
	fnvpair_value_string(nvp));
	break;
	case DATA_TYPE_UINT64:
	fnvlist_add_uint64(stats->zns_uint64, name,
	fnvpair_value_uint64(nvp));
	break;
	case DATA_TYPE_BOOLEAN:
	fnvlist_add_boolean(stats->zns_boolean, name);
	break;
	case DATA_TYPE_NVLIST:
	if (nvpair_value_nvlist(nvp, &list) == 0)
	collect_nvlist_stats(list, stats);
	break;
	case DATA_TYPE_NVLIST_ARRAY:
	if (nvpair_value_nvlist_array(nvp, &array, &items) != 0)
	break;

	for (i = 0; i < items; i++) {
	collect_nvlist_stats(array[i], stats);

	/* collect stats on leaf vdev */
	if (strcmp(name, "children") == 0) {
	size_t size;

	(void) nvlist_size(array[i], &size,
	NV_ENCODE_XDR);
	stats->zns_leaf_total += size;
	if (size > stats->zns_leaf_largest)
	stats->zns_leaf_largest = size;
	stats->zns_leaf_count++;
	}
	}
	break;
	default:
	(void) printf("skip type %d!\n", (int)nvpair_type(nvp));
	}
	}
	}

	static void
	dump_nvlist_stats(nvlist_t *nvl, size_t cap)
	{
	zdb_nvl_stats_t stats = { 0 };
	size_t size, sum = 0, total;
	size_t noise;

	/* requires nvlist with non-unique names for stat collection */
	VERIFY0(nvlist_alloc(&stats.zns_string, 0, 0));
	VERIFY0(nvlist_alloc(&stats.zns_uint64, 0, 0));
	VERIFY0(nvlist_alloc(&stats.zns_boolean, 0, 0));
	VERIFY0(nvlist_size(stats.zns_boolean, &noise, NV_ENCODE_XDR));

	(void) printf("\n\nZFS Label NVList Config Stats:\n");

	VERIFY0(nvlist_size(nvl, &total, NV_ENCODE_XDR));
	(void) printf(" %d bytes used, %d bytes free (using %4.1f%%)\n\n",
	(int)total, (int)(cap - total), 100.0 * total / cap);

	collect_nvlist_stats(nvl, &stats);

	VERIFY0(nvlist_size(stats.zns_uint64, &size, NV_ENCODE_XDR));
	size -= noise;
	sum += size;
	(void) printf("%12s %4d %6d bytes (%5.2f%%)\n", "integers:",
	(int)fnvlist_num_pairs(stats.zns_uint64),
	(int)size, 100.0 * size / total);

	VERIFY0(nvlist_size(stats.zns_string, &size, NV_ENCODE_XDR));
	size -= noise;
	sum += size;
	(void) printf("%12s %4d %6d bytes (%5.2f%%)\n", "strings:",
	(int)fnvlist_num_pairs(stats.zns_string),
	(int)size, 100.0 * size / total);

	VERIFY0(nvlist_size(stats.zns_boolean, &size, NV_ENCODE_XDR));
	size -= noise;
	sum += size;
	(void) printf("%12s %4d %6d bytes (%5.2f%%)\n", "booleans:",
	(int)fnvlist_num_pairs(stats.zns_boolean),
	(int)size, 100.0 * size / total);

	size = total - sum; /* treat remainder as nvlist overhead */
	(void) printf("%12s %4d %6d bytes (%5.2f%%)\n\n", "nvlists:",
	stats.zns_list_count, (int)size, 100.0 * size / total);

	if (stats.zns_leaf_count > 0) {
	size_t average = stats.zns_leaf_total / stats.zns_leaf_count;

	(void) printf("%12s %4d %6d bytes average\n", "leaf vdevs:",
	stats.zns_leaf_count, (int)average);
	(void) printf("%24d bytes largest\n",
	(int)stats.zns_leaf_largest);

	if (dump_opt['l'] >= 3 && average > 0)
	(void) printf(" space for %d additional leaf vdevs\n",
	(int)((cap - total) / average));
	}
	(void) printf("\n");

	nvlist_free(stats.zns_string);
	nvlist_free(stats.zns_uint64);
	nvlist_free(stats.zns_boolean);
	}

	typedef struct cksum_record {
	zio_cksum_t cksum;
	boolean_t labels[VDEV_LABELS];
	avl_node_t link;
	} cksum_record_t;

	static int
	cksum_record_compare(const void x1, const void x2)
	{
	const cksum_record_t l = (cksum_record_t )x1;
	const cksum_record_t r = (cksum_record_t )x2;
	int arraysize = ARRAY_SIZE(l->cksum.zc_word);
	int difference;

	for (int i = 0; i < arraysize; i++) {
	difference = TREE_CMP(l->cksum.zc_word[i], r->cksum.zc_word[i]);
	if (difference)
	break;
	}

	return (difference);
	}

	static cksum_record_t *
	cksum_record_alloc(zio_cksum_t *cksum, int l)
	{
	cksum_record_t *rec;

	rec = umem_zalloc(sizeof (*rec), UMEM_NOFAIL);
	rec->cksum = *cksum;
	rec->labels[l] = B_TRUE;

	return (rec);
	}

	static cksum_record_t *
	cksum_record_lookup(avl_tree_t tree, zio_cksum_t cksum)
	{
	cksum_record_t lookup = { .cksum = *cksum };
	avl_index_t where;

	return (avl_find(tree, &lookup, &where));
	}

	static cksum_record_t *
	cksum_record_insert(avl_tree_t tree, zio_cksum_t cksum, int l)
	{
	cksum_record_t *rec;

	rec = cksum_record_lookup(tree, cksum);
	if (rec) {
	rec->labels[l] = B_TRUE;
	} else {
	rec = cksum_record_alloc(cksum, l);
	avl_add(tree, rec);
	}

	return (rec);
	}

	static int
	first_label(cksum_record_t *rec)
	{
	for (int i = 0; i < VDEV_LABELS; i++)
	if (rec->labels[i])
	return (i);

	return (-1);
	}

	static void
	print_label_numbers(char prefix, cksum_record_t rec)
	{
	printf("%s", prefix);
	for (int i = 0; i < VDEV_LABELS; i++)
	if (rec->labels[i] == B_TRUE)
	printf("%d ", i);
	printf("\n");
	}

	#define MAX_UBERBLOCK_COUNT (VDEV_UBERBLOCK_RING >> UBERBLOCK_SHIFT)

	typedef struct zdb_label {
	vdev_label_t label;
	nvlist_t *config_nv;
	cksum_record_t *config;
	cksum_record_t *uberblocks[MAX_UBERBLOCK_COUNT];
	boolean_t header_printed;
	boolean_t read_failed;
	} zdb_label_t;

	static void
	print_label_header(zdb_label_t *label, int l)
	{

	if (dump_opt['q'])
	return;

	if (label->header_printed == B_TRUE)
	return;

	(void) printf("------------------------------------\n");
	(void) printf("LABEL %d\n", l);
	(void) printf("------------------------------------\n");

	label->header_printed = B_TRUE;
	}

	static void
	print_l2arc_header(void)
	{
	(void) printf("------------------------------------\n");
	(void) printf("L2ARC device header\n");
	(void) printf("------------------------------------\n");
	}

	static void
	print_l2arc_log_blocks(void)
	{
	(void) printf("------------------------------------\n");
	(void) printf("L2ARC device log blocks\n");
	(void) printf("------------------------------------\n");
	}

	static void
	dump_l2arc_log_entries(uint64_t log_entries,
	l2arc_log_ent_phys_t *le, uint64_t i)
	{
	for (int j = 0; j < log_entries; j++) {
	dva_t dva = le[j].le_dva;
	(void) printf("lb[%4llu]\tle[%4d]\tDVA asize: %llu, "
	"vdev: %llu, offset: %llu\n",
	(u_longlong_t)i, j + 1,
	(u_longlong_t)DVA_GET_ASIZE(&dva),
	(u_longlong_t)DVA_GET_VDEV(&dva),
	(u_longlong_t)DVA_GET_OFFSET(&dva));
	(void) printf("\|\t\t\t\tbirth: %llu\n",
	(u_longlong_t)le[j].le_birth);
	(void) printf("\|\t\t\t\tlsize: %llu\n",
	(u_longlong_t)L2BLK_GET_LSIZE((&le[j])->le_prop));
	(void) printf("\|\t\t\t\tpsize: %llu\n",
	(u_longlong_t)L2BLK_GET_PSIZE((&le[j])->le_prop));
	(void) printf("\|\t\t\t\tcompr: %llu\n",
	(u_longlong_t)L2BLK_GET_COMPRESS((&le[j])->le_prop));
	(void) printf("\|\t\t\t\tcomplevel: %llu\n",
	(u_longlong_t)(&le[j])->le_complevel);
	(void) printf("\|\t\t\t\ttype: %llu\n",
	(u_longlong_t)L2BLK_GET_TYPE((&le[j])->le_prop));
	(void) printf("\|\t\t\t\tprotected: %llu\n",
	(u_longlong_t)L2BLK_GET_PROTECTED((&le[j])->le_prop));
	(void) printf("\|\t\t\t\tprefetch: %llu\n",
	(u_longlong_t)L2BLK_GET_PREFETCH((&le[j])->le_prop));
	(void) printf("\|\t\t\t\taddress: %llu\n",
	(u_longlong_t)le[j].le_daddr);
	(void) printf("\|\n");
	}
	(void) printf("\n");
	}

	static void
	dump_l2arc_log_blkptr(l2arc_log_blkptr_t lbps)
	{
	(void) printf("\|\t\tdaddr: %llu\n", (u_longlong_t)lbps.lbp_daddr);
	(void) printf("\|\t\tpayload_asize: %llu\n",
	(u_longlong_t)lbps.lbp_payload_asize);
	(void) printf("\|\t\tpayload_start: %llu\n",
	(u_longlong_t)lbps.lbp_payload_start);
	(void) printf("\|\t\tlsize: %llu\n",
	(u_longlong_t)L2BLK_GET_LSIZE((&lbps)->lbp_prop));
	(void) printf("\|\t\tasize: %llu\n",
	(u_longlong_t)L2BLK_GET_PSIZE((&lbps)->lbp_prop));
	(void) printf("\|\t\tcompralgo: %llu\n",
	(u_longlong_t)L2BLK_GET_COMPRESS((&lbps)->lbp_prop));
	(void) printf("\|\t\tcksumalgo: %llu\n",
	(u_longlong_t)L2BLK_GET_CHECKSUM((&lbps)->lbp_prop));
	(void) printf("\|\n\n");
	}

	static void
	dump_l2arc_log_blocks(int fd, l2arc_dev_hdr_phys_t l2dhdr,
	l2arc_dev_hdr_phys_t *rebuild)
	{
	l2arc_log_blk_phys_t this_lb;
	uint64_t asize;
	l2arc_log_blkptr_t lbps[2];
	abd_t *abd;
	zio_cksum_t cksum;
	int failed = 0;
	l2arc_dev_t dev;

	if (!dump_opt['q'])
	print_l2arc_log_blocks();
	bcopy((&l2dhdr)->dh_start_lbps, lbps, sizeof (lbps));

	dev.l2ad_evict = l2dhdr.dh_evict;
	dev.l2ad_start = l2dhdr.dh_start;
	dev.l2ad_end = l2dhdr.dh_end;

	if (l2dhdr.dh_start_lbps[0].lbp_daddr == 0) {
	/* no log blocks to read */
	if (!dump_opt['q']) {
	(void) printf("No log blocks to read\n");
	(void) printf("\n");
	}
	return;
	} else {
	dev.l2ad_hand = lbps[0].lbp_daddr +
	L2BLK_GET_PSIZE((&lbps[0])->lbp_prop);
	}

	dev.l2ad_first = !!(l2dhdr.dh_flags & L2ARC_DEV_HDR_EVICT_FIRST);

	for (;;) {
	if (!l2arc_log_blkptr_valid(&dev, &lbps[0]))
	break;

	/* L2BLK_GET_PSIZE returns aligned size for log blocks */
	asize = L2BLK_GET_PSIZE((&lbps[0])->lbp_prop);
	if (pread64(fd, &this_lb, asize, lbps[0].lbp_daddr) != asize) {
	if (!dump_opt['q']) {
	(void) printf("Error while reading next log "
	"block\n\n");
	}
	break;
	}

	fletcher_4_native_varsize(&this_lb, asize, &cksum);
	if (!ZIO_CHECKSUM_EQUAL(cksum, lbps[0].lbp_cksum)) {
	failed++;
	if (!dump_opt['q']) {
	(void) printf("Invalid cksum\n");
	dump_l2arc_log_blkptr(lbps[0]);
	}
	break;
	}

	switch (L2BLK_GET_COMPRESS((&lbps[0])->lbp_prop)) {
	case ZIO_COMPRESS_OFF:
	break;
	default:
	abd = abd_alloc_for_io(asize, B_TRUE);
	abd_copy_from_buf_off(abd, &this_lb, 0, asize);
	zio_decompress_data(L2BLK_GET_COMPRESS(
	(&lbps[0])->lbp_prop), abd, &this_lb,
	asize, sizeof (this_lb), NULL);
	abd_free(abd);
	break;
	}

	if (this_lb.lb_magic == BSWAP_64(L2ARC_LOG_BLK_MAGIC))
	byteswap_uint64_array(&this_lb, sizeof (this_lb));
	if (this_lb.lb_magic != L2ARC_LOG_BLK_MAGIC) {
	if (!dump_opt['q'])
	(void) printf("Invalid log block magic\n\n");
	break;
	}

	rebuild->dh_lb_count++;
	rebuild->dh_lb_asize += asize;
	if (dump_opt['l'] > 1 && !dump_opt['q']) {
	(void) printf("lb[%4llu]\tmagic: %llu\n",
	(u_longlong_t)rebuild->dh_lb_count,
	(u_longlong_t)this_lb.lb_magic);
	dump_l2arc_log_blkptr(lbps[0]);
	}

	if (dump_opt['l'] > 2 && !dump_opt['q'])
	dump_l2arc_log_entries(l2dhdr.dh_log_entries,
	this_lb.lb_entries,
	rebuild->dh_lb_count);

	if (l2arc_range_check_overlap(lbps[1].lbp_payload_start,
	lbps[0].lbp_payload_start, dev.l2ad_evict) &&
	!dev.l2ad_first)
	break;

	lbps[0] = lbps[1];
	lbps[1] = this_lb.lb_prev_lbp;
	}

	if (!dump_opt['q']) {
	(void) printf("log_blk_count:\t %llu with valid cksum\n",
	(u_longlong_t)rebuild->dh_lb_count);
	(void) printf("\t\t %d with invalid cksum\n", failed);
	(void) printf("log_blk_asize:\t %llu\n\n",
	(u_longlong_t)rebuild->dh_lb_asize);
	}
	}

	static int
	dump_l2arc_header(int fd)
	{
	l2arc_dev_hdr_phys_t l2dhdr, rebuild;
	int error = B_FALSE;

	bzero(&l2dhdr, sizeof (l2dhdr));
	bzero(&rebuild, sizeof (rebuild));

	if (pread64(fd, &l2dhdr, sizeof (l2dhdr),
	VDEV_LABEL_START_SIZE) != sizeof (l2dhdr)) {
	error = B_TRUE;
	} else {
	if (l2dhdr.dh_magic == BSWAP_64(L2ARC_DEV_HDR_MAGIC))
	byteswap_uint64_array(&l2dhdr, sizeof (l2dhdr));

	if (l2dhdr.dh_magic != L2ARC_DEV_HDR_MAGIC)
	error = B_TRUE;
	}

	if (error) {
	(void) printf("L2ARC device header not found\n\n");
	/* Do not return an error here for backward compatibility */
	return (0);
	} else if (!dump_opt['q']) {
	print_l2arc_header();

	(void) printf(" magic: %llu\n",
	(u_longlong_t)l2dhdr.dh_magic);
	(void) printf(" version: %llu\n",
	(u_longlong_t)l2dhdr.dh_version);
	(void) printf(" pool_guid: %llu\n",
	(u_longlong_t)l2dhdr.dh_spa_guid);
	(void) printf(" flags: %llu\n",
	(u_longlong_t)l2dhdr.dh_flags);
	(void) printf(" start_lbps[0]: %llu\n",
	(u_longlong_t)
	l2dhdr.dh_start_lbps[0].lbp_daddr);
	(void) printf(" start_lbps[1]: %llu\n",
	(u_longlong_t)
	l2dhdr.dh_start_lbps[1].lbp_daddr);
	(void) printf(" log_blk_ent: %llu\n",
	(u_longlong_t)l2dhdr.dh_log_entries);
	(void) printf(" start: %llu\n",
	(u_longlong_t)l2dhdr.dh_start);
	(void) printf(" end: %llu\n",
	(u_longlong_t)l2dhdr.dh_end);
	(void) printf(" evict: %llu\n",
	(u_longlong_t)l2dhdr.dh_evict);
	(void) printf(" lb_asize_refcount: %llu\n",
	(u_longlong_t)l2dhdr.dh_lb_asize);
	(void) printf(" lb_count_refcount: %llu\n",
	(u_longlong_t)l2dhdr.dh_lb_count);
	(void) printf(" trim_action_time: %llu\n",
	(u_longlong_t)l2dhdr.dh_trim_action_time);
	(void) printf(" trim_state: %llu\n\n",
	(u_longlong_t)l2dhdr.dh_trim_state);
	}

	dump_l2arc_log_blocks(fd, l2dhdr, &rebuild);
	/*
	* The total aligned size of log blocks and the number of log blocks
	* reported in the header of the device may be less than what zdb
	* reports by dump_l2arc_log_blocks() which emulates l2arc_rebuild().
	* This happens because dump_l2arc_log_blocks() lacks the memory
	* pressure valve that l2arc_rebuild() has. Thus, if we are on a system
	* with low memory, l2arc_rebuild will exit prematurely and dh_lb_asize
	* and dh_lb_count will be lower to begin with than what exists on the
	* device. This is normal and zdb should not exit with an error. The
	* opposite case should never happen though, the values reported in the
	* header should never be higher than what dump_l2arc_log_blocks() and
	* l2arc_rebuild() report. If this happens there is a leak in the
	* accounting of log blocks.
	*/
	if (l2dhdr.dh_lb_asize > rebuild.dh_lb_asize \|\|
	l2dhdr.dh_lb_count > rebuild.dh_lb_count)
	return (1);

	return (0);
	}

	static void
	dump_config_from_label(zdb_label_t *label, size_t buflen, int l)
	{
	if (dump_opt['q'])
	return;

	if ((dump_opt['l'] < 3) && (first_label(label->config) != l))
	return;

	print_label_header(label, l);
	dump_nvlist(label->config_nv, 4);
	print_label_numbers(" labels = ", label->config);

	if (dump_opt['l'] >= 2)
	dump_nvlist_stats(label->config_nv, buflen);
	}

	#define ZDB_MAX_UB_HEADER_SIZE 32

	static void
	dump_label_uberblocks(zdb_label_t *label, uint64_t ashift, int label_num)
	{

	vdev_t vd;
	char header[ZDB_MAX_UB_HEADER_SIZE];

	vd.vdev_ashift = ashift;
	vd.vdev_top = &vd;

	for (int i = 0; i < VDEV_UBERBLOCK_COUNT(&vd); i++) {
	uint64_t uoff = VDEV_UBERBLOCK_OFFSET(&vd, i);
	uberblock_t ub = (void )((char *)&label->label + uoff);
	cksum_record_t *rec = label->uberblocks[i];

	if (rec == NULL) {
	if (dump_opt['u'] >= 2) {
	print_label_header(label, label_num);
	(void) printf(" Uberblock[%d] invalid\n", i);
	}
	continue;
	}

	if ((dump_opt['u'] < 3) && (first_label(rec) != label_num))
	continue;

	if ((dump_opt['u'] < 4) &&
	(ub->ub_mmp_magic == MMP_MAGIC) && ub->ub_mmp_delay &&
	(i >= VDEV_UBERBLOCK_COUNT(&vd) - MMP_BLOCKS_PER_LABEL))
	continue;

	print_label_header(label, label_num);
	(void) snprintf(header, ZDB_MAX_UB_HEADER_SIZE,
	" Uberblock[%d]\n", i);
	dump_uberblock(ub, header, "");
	print_label_numbers(" labels = ", rec);
	}
	}

	static char curpath[PATH_MAX];

	/*
	* Iterate through the path components, recursively passing
	* current one's obj and remaining path until we find the obj
	* for the last one.
	*/
	static int
	dump_path_impl(objset_t os, uint64_t obj, char name)
	{
	int err;
	boolean_t header = B_TRUE;
	uint64_t child_obj;
	char *s;
	dmu_buf_t *db;
	dmu_object_info_t doi;

	if ((s = strchr(name, '/')) != NULL)
	*s = '\0';
	err = zap_lookup(os, obj, name, 8, 1, &child_obj);

	(void) strlcat(curpath, name, sizeof (curpath));

	if (err != 0) {
	(void) fprintf(stderr, "failed to lookup %s: %s\n",
	curpath, strerror(err));
	return (err);
	}

	child_obj = ZFS_DIRENT_OBJ(child_obj);
	err = sa_buf_hold(os, child_obj, FTAG, &db);
	if (err != 0) {
	(void) fprintf(stderr,
	"failed to get SA dbuf for obj %llu: %s\n",
	(u_longlong_t)child_obj, strerror(err));
	return (EINVAL);
	}
	dmu_object_info_from_db(db, &doi);
	sa_buf_rele(db, FTAG);

	if (doi.doi_bonus_type != DMU_OT_SA &&
	doi.doi_bonus_type != DMU_OT_ZNODE) {
	(void) fprintf(stderr, "invalid bonus type %d for obj %llu\n",
	doi.doi_bonus_type, (u_longlong_t)child_obj);
	return (EINVAL);
	}

	if (dump_opt['v'] > 6) {
	(void) printf("obj=%llu %s type=%d bonustype=%d\n",
	(u_longlong_t)child_obj, curpath, doi.doi_type,
	doi.doi_bonus_type);
	}

	(void) strlcat(curpath, "/", sizeof (curpath));

	switch (doi.doi_type) {
	case DMU_OT_DIRECTORY_CONTENTS:
	if (s != NULL && *(s + 1) != '\0')
	return (dump_path_impl(os, child_obj, s + 1));
	/FALLTHROUGH/
	case DMU_OT_PLAIN_FILE_CONTENTS:
	dump_object(os, child_obj, dump_opt['v'], &header, NULL, 0);
	return (0);
	default:
	(void) fprintf(stderr, "object %llu has non-file/directory "
	"type %d\n", (u_longlong_t)obj, doi.doi_type);
	break;
	}

	return (EINVAL);
	}

	/*
	* Dump the blocks for the object specified by path inside the dataset.
	*/
	static int
	dump_path(char ds, char path)
	{
	int err;
	objset_t *os;
	uint64_t root_obj;

	err = open_objset(ds, FTAG, &os);
	if (err != 0)
	return (err);

	err = zap_lookup(os, MASTER_NODE_OBJ, ZFS_ROOT_OBJ, 8, 1, &root_obj);
	if (err != 0) {
	(void) fprintf(stderr, "can't lookup root znode: %s\n",
	strerror(err));
	close_objset(os, FTAG);
	return (EINVAL);
	}

	(void) snprintf(curpath, sizeof (curpath), "dataset=%s path=/", ds);

	err = dump_path_impl(os, root_obj, path);

	close_objset(os, FTAG);
	return (err);
	}

	static int
	dump_label(const char *dev)
	{
	char path[MAXPATHLEN];
	zdb_label_t labels[VDEV_LABELS];
	uint64_t psize, ashift, l2cache;
	struct stat64 statbuf;
	boolean_t config_found = B_FALSE;
	boolean_t error = B_FALSE;
	boolean_t read_l2arc_header = B_FALSE;
	avl_tree_t config_tree;
	avl_tree_t uberblock_tree;
	void node, cookie;
	int fd;

	bzero(labels, sizeof (labels));

	/*
	* Check if we were given absolute path and use it as is.
	* Otherwise if the provided vdev name doesn't point to a file,
	* try prepending expected disk paths and partition numbers.
	*/
	(void) strlcpy(path, dev, sizeof (path));
	if (dev[0] != '/' && stat64(path, &statbuf) != 0) {
	int error;

	error = zfs_resolve_shortname(dev, path, MAXPATHLEN);
	if (error == 0 && zfs_dev_is_whole_disk(path)) {
	if (zfs_append_partition(path, MAXPATHLEN) == -1)
	error = ENOENT;
	}

	if (error \|\| (stat64(path, &statbuf) != 0)) {
	(void) printf("failed to find device %s, try "
	"specifying absolute path instead\n", dev);
	return (1);
	}
	}

	if ((fd = open64(path, O_RDONLY)) < 0) {
	(void) printf("cannot open '%s': %s\n", path, strerror(errno));
	exit(1);
	}

	if (fstat64_blk(fd, &statbuf) != 0) {
	(void) printf("failed to stat '%s': %s\n", path,
	strerror(errno));
	(void) close(fd);
	exit(1);
	}

	if (S_ISBLK(statbuf.st_mode) && zfs_dev_flush(fd) != 0)
	(void) printf("failed to invalidate cache '%s' : %s\n", path,
	strerror(errno));

	avl_create(&config_tree, cksum_record_compare,
	sizeof (cksum_record_t), offsetof(cksum_record_t, link));
	avl_create(&uberblock_tree, cksum_record_compare,
	sizeof (cksum_record_t), offsetof(cksum_record_t, link));

	psize = statbuf.st_size;
	psize = P2ALIGN(psize, (uint64_t)sizeof (vdev_label_t));
	ashift = SPA_MINBLOCKSHIFT;

	/*
	* 1. Read the label from disk
	* 2. Unpack the configuration and insert in config tree.
	* 3. Traverse all uberblocks and insert in uberblock tree.
	*/
	for (int l = 0; l < VDEV_LABELS; l++) {
	zdb_label_t *label = &labels[l];
	char *buf = label->label.vl_vdev_phys.vp_nvlist;
	size_t buflen = sizeof (label->label.vl_vdev_phys.vp_nvlist);
	nvlist_t *config;
	cksum_record_t *rec;
	zio_cksum_t cksum;
	vdev_t vd;

	if (pread64(fd, &label->label, sizeof (label->label),
	vdev_label_offset(psize, l, 0)) != sizeof (label->label)) {
	if (!dump_opt['q'])
	(void) printf("failed to read label %d\n", l);
	label->read_failed = B_TRUE;
	error = B_TRUE;
	continue;
	}

	label->read_failed = B_FALSE;

	if (nvlist_unpack(buf, buflen, &config, 0) == 0) {
	nvlist_t *vdev_tree = NULL;
	size_t size;

	if ((nvlist_lookup_nvlist(config,
	ZPOOL_CONFIG_VDEV_TREE, &vdev_tree) != 0) \|\|
	(nvlist_lookup_uint64(vdev_tree,
	ZPOOL_CONFIG_ASHIFT, &ashift) != 0))
	ashift = SPA_MINBLOCKSHIFT;

	if (nvlist_size(config, &size, NV_ENCODE_XDR) != 0)
	size = buflen;

	/* If the device is a cache device clear the header. */
	if (!read_l2arc_header) {
	if (nvlist_lookup_uint64(config,
	ZPOOL_CONFIG_POOL_STATE, &l2cache) == 0 &&
	l2cache == POOL_STATE_L2CACHE) {
	read_l2arc_header = B_TRUE;
	}
	}

	fletcher_4_native_varsize(buf, size, &cksum);
	rec = cksum_record_insert(&config_tree, &cksum, l);

	label->config = rec;
	label->config_nv = config;
	config_found = B_TRUE;
	} else {
	error = B_TRUE;
	}

	vd.vdev_ashift = ashift;
	vd.vdev_top = &vd;

	for (int i = 0; i < VDEV_UBERBLOCK_COUNT(&vd); i++) {
	uint64_t uoff = VDEV_UBERBLOCK_OFFSET(&vd, i);
	uberblock_t ub = (void )((char *)label + uoff);

	if (uberblock_verify(ub))
	continue;

	fletcher_4_native_varsize(ub, sizeof (*ub), &cksum);
	rec = cksum_record_insert(&uberblock_tree, &cksum, l);

	label->uberblocks[i] = rec;
	}
	}

	/*
	* Dump the label and uberblocks.
	*/
	for (int l = 0; l < VDEV_LABELS; l++) {
	zdb_label_t *label = &labels[l];
	size_t buflen = sizeof (label->label.vl_vdev_phys.vp_nvlist);

	if (label->read_failed == B_TRUE)
	continue;

	if (label->config_nv) {
	dump_config_from_label(label, buflen, l);
	} else {
	if (!dump_opt['q'])
	(void) printf("failed to unpack label %d\n", l);
	}

	if (dump_opt['u'])
	dump_label_uberblocks(label, ashift, l);

	nvlist_free(label->config_nv);
	}

	/*
	* Dump the L2ARC header, if existent.
	*/
	if (read_l2arc_header)
	error \|= dump_l2arc_header(fd);

	cookie = NULL;
	while ((node = avl_destroy_nodes(&config_tree, &cookie)) != NULL)
	umem_free(node, sizeof (cksum_record_t));

	cookie = NULL;
	while ((node = avl_destroy_nodes(&uberblock_tree, &cookie)) != NULL)
	umem_free(node, sizeof (cksum_record_t));

	avl_destroy(&config_tree);
	avl_destroy(&uberblock_tree);

	(void) close(fd);

	return (config_found == B_FALSE ? 2 :
	(error == B_TRUE ? 1 : 0));
	}

	static uint64_t dataset_feature_count[SPA_FEATURES];
	static uint64_t global_feature_count[SPA_FEATURES];
	static uint64_t remap_deadlist_count = 0;

	/ARGSUSED/
	static int
	dump_one_objset(const char dsname, void arg)
	{
	int error;
	objset_t *os;
	spa_feature_t f;

	error = open_objset(dsname, FTAG, &os);
	if (error != 0)
	return (0);

	for (f = 0; f < SPA_FEATURES; f++) {
	if (!dsl_dataset_feature_is_active(dmu_objset_ds(os), f))
	continue;
	ASSERT(spa_feature_table[f].fi_flags &
	ZFEATURE_FLAG_PER_DATASET);
	dataset_feature_count[f]++;
	}

	if (dsl_dataset_remap_deadlist_exists(dmu_objset_ds(os))) {
	remap_deadlist_count++;
	}

	for (dsl_bookmark_node_t *dbn =
	avl_first(&dmu_objset_ds(os)->ds_bookmarks); dbn != NULL;
	dbn = AVL_NEXT(&dmu_objset_ds(os)->ds_bookmarks, dbn)) {
	mos_obj_refd(dbn->dbn_phys.zbm_redaction_obj);
	if (dbn->dbn_phys.zbm_redaction_obj != 0)
	global_feature_count[SPA_FEATURE_REDACTION_BOOKMARKS]++;
	if (dbn->dbn_phys.zbm_flags & ZBM_FLAG_HAS_FBN)
	global_feature_count[SPA_FEATURE_BOOKMARK_WRITTEN]++;
	}

	if (dsl_deadlist_is_open(&dmu_objset_ds(os)->ds_dir->dd_livelist) &&
	!dmu_objset_is_snapshot(os)) {
	global_feature_count[SPA_FEATURE_LIVELIST]++;
	}

	dump_objset(os);
	close_objset(os, FTAG);
	fuid_table_destroy();
	return (0);
	}

	/*
	* Block statistics.
	*/
	#define PSIZE_HISTO_SIZE (SPA_OLD_MAXBLOCKSIZE / SPA_MINBLOCKSIZE + 2)
	typedef struct zdb_blkstats {
	uint64_t zb_asize;
	uint64_t zb_lsize;
	uint64_t zb_psize;
	uint64_t zb_count;
	uint64_t zb_gangs;
	uint64_t zb_ditto_samevdev;
	uint64_t zb_ditto_same_ms;
	uint64_t zb_psize_histogram[PSIZE_HISTO_SIZE];
	} zdb_blkstats_t;

	/*
	* Extended object types to report deferred frees and dedup auto-ditto blocks.
	*/
	#define ZDB_OT_DEFERRED (DMU_OT_NUMTYPES + 0)
	#define ZDB_OT_DITTO (DMU_OT_NUMTYPES + 1)
	#define ZDB_OT_OTHER (DMU_OT_NUMTYPES + 2)
	#define ZDB_OT_TOTAL (DMU_OT_NUMTYPES + 3)

	static const char *zdb_ot_extname[] = {
	"deferred free",
	"dedup ditto",
	"other",
	"Total",
	};

	#define ZB_TOTAL DN_MAX_LEVELS
	#define SPA_MAX_FOR_16M (SPA_MAXBLOCKSHIFT+1)

	typedef struct zdb_cb {
	zdb_blkstats_t zcb_type[ZB_TOTAL + 1][ZDB_OT_TOTAL + 1];
	uint64_t zcb_removing_size;
	uint64_t zcb_checkpoint_size;
	uint64_t zcb_dedup_asize;
	uint64_t zcb_dedup_blocks;
	uint64_t zcb_psize_count[SPA_MAX_FOR_16M];
	uint64_t zcb_lsize_count[SPA_MAX_FOR_16M];
	uint64_t zcb_asize_count[SPA_MAX_FOR_16M];
	uint64_t zcb_psize_len[SPA_MAX_FOR_16M];
	uint64_t zcb_lsize_len[SPA_MAX_FOR_16M];
	uint64_t zcb_asize_len[SPA_MAX_FOR_16M];
	uint64_t zcb_psize_total;
	uint64_t zcb_lsize_total;
	uint64_t zcb_asize_total;
	uint64_t zcb_embedded_blocks[NUM_BP_EMBEDDED_TYPES];
	uint64_t zcb_embedded_histogram[NUM_BP_EMBEDDED_TYPES]
	[BPE_PAYLOAD_SIZE + 1];
	uint64_t zcb_start;
	hrtime_t zcb_lastprint;
	uint64_t zcb_totalasize;
	uint64_t zcb_errors[256];
	int zcb_readfails;
	int zcb_haderrors;
	spa_t *zcb_spa;
	uint32_t **zcb_vd_obsolete_counts;
	} zdb_cb_t;

	/* test if two DVA offsets from same vdev are within the same metaslab */
	static boolean_t
	same_metaslab(spa_t *spa, uint64_t vdev, uint64_t off1, uint64_t off2)
	{
	vdev_t *vd = vdev_lookup_top(spa, vdev);
	uint64_t ms_shift = vd->vdev_ms_shift;

	return ((off1 >> ms_shift) == (off2 >> ms_shift));
	}

	/*
	* Used to simplify reporting of the histogram data.
	*/
	typedef struct one_histo {
	char *name;
	uint64_t *count;
	uint64_t *len;
	uint64_t cumulative;
	} one_histo_t;

	/*
	* The number of separate histograms processed for psize, lsize and asize.
	*/
	#define NUM_HISTO 3

	/*
	* This routine will create a fixed column size output of three different
	* histograms showing by blocksize of 512 - 2^ SPA_MAX_FOR_16M
	* the count, length and cumulative length of the psize, lsize and
	* asize blocks.
	*
	* All three types of blocks are listed on a single line
	*
	* By default the table is printed in nicenumber format (e.g. 123K) but
	* if the '-P' parameter is specified then the full raw number (parseable)
	* is printed out.
	*/
	static void
	dump_size_histograms(zdb_cb_t *zcb)
	{
	/*
	* A temporary buffer that allows us to convert a number into
	* a string using zdb_nicenumber to allow either raw or human
	* readable numbers to be output.
	*/
	char numbuf[32];

	/*
	* Define titles which are used in the headers of the tables
	* printed by this routine.
	*/
	const char blocksize_title1[] = "block";
	const char blocksize_title2[] = "size";
	const char count_title[] = "Count";
	const char length_title[] = "Size";
	const char cumulative_title[] = "Cum.";

	/*
	* Setup the histogram arrays (psize, lsize, and asize).
	*/
	one_histo_t parm_histo[NUM_HISTO];

	parm_histo[0].name = "psize";
	parm_histo[0].count = zcb->zcb_psize_count;
	parm_histo[0].len = zcb->zcb_psize_len;
	parm_histo[0].cumulative = 0;

	parm_histo[1].name = "lsize";
	parm_histo[1].count = zcb->zcb_lsize_count;
	parm_histo[1].len = zcb->zcb_lsize_len;
	parm_histo[1].cumulative = 0;

	parm_histo[2].name = "asize";
	parm_histo[2].count = zcb->zcb_asize_count;
	parm_histo[2].len = zcb->zcb_asize_len;
	parm_histo[2].cumulative = 0;


	(void) printf("\nBlock Size Histogram\n");
	/*
	* Print the first line titles
	*/
	if (dump_opt['P'])
	(void) printf("\n%s\t", blocksize_title1);
	else
	(void) printf("\n%7s ", blocksize_title1);

	for (int j = 0; j < NUM_HISTO; j++) {
	if (dump_opt['P']) {
	if (j < NUM_HISTO - 1) {
	(void) printf("%s\t\t\t", parm_histo[j].name);
	} else {
	/* Don't print trailing spaces */
	(void) printf(" %s", parm_histo[j].name);
	}
	} else {
	if (j < NUM_HISTO - 1) {
	/* Left aligned strings in the output */
	(void) printf("%-7s ",
	parm_histo[j].name);
	} else {
	/* Don't print trailing spaces */
	(void) printf("%s", parm_histo[j].name);
	}
	}
	}
	(void) printf("\n");

	/*
	* Print the second line titles
	*/
	if (dump_opt['P']) {
	(void) printf("%s\t", blocksize_title2);
	} else {
	(void) printf("%7s ", blocksize_title2);
	}

	for (int i = 0; i < NUM_HISTO; i++) {
	if (dump_opt['P']) {
	(void) printf("%s\t%s\t%s\t",
	count_title, length_title, cumulative_title);
	} else {
	(void) printf("%7s%7s%7s",
	count_title, length_title, cumulative_title);
	}
	}
	(void) printf("\n");

	/*
	* Print the rows
	*/
	for (int i = SPA_MINBLOCKSHIFT; i < SPA_MAX_FOR_16M; i++) {

	/*
	* Print the first column showing the blocksize
	*/
	zdb_nicenum((1ULL << i), numbuf, sizeof (numbuf));

	if (dump_opt['P']) {
	printf("%s", numbuf);
	} else {
	printf("%7s:", numbuf);
	}

	/*
	* Print the remaining set of 3 columns per size:
	* for psize, lsize and asize
	*/
	for (int j = 0; j < NUM_HISTO; j++) {
	parm_histo[j].cumulative += parm_histo[j].len[i];

	zdb_nicenum(parm_histo[j].count[i],
	numbuf, sizeof (numbuf));
	if (dump_opt['P'])
	(void) printf("\t%s", numbuf);
	else
	(void) printf("%7s", numbuf);

	zdb_nicenum(parm_histo[j].len[i],
	numbuf, sizeof (numbuf));
	if (dump_opt['P'])
	(void) printf("\t%s", numbuf);
	else
	(void) printf("%7s", numbuf);

	zdb_nicenum(parm_histo[j].cumulative,
	numbuf, sizeof (numbuf));
	if (dump_opt['P'])
	(void) printf("\t%s", numbuf);
	else
	(void) printf("%7s", numbuf);
	}
	(void) printf("\n");
	}
	}

	static void
	zdb_count_block(zdb_cb_t zcb, zilog_t zilog, const blkptr_t *bp,
	dmu_object_type_t type)
	{
	uint64_t refcnt = 0;
	int i;

	ASSERT(type < ZDB_OT_TOTAL);

	if (zilog && zil_bp_tree_add(zilog, bp) != 0)
	return;

	spa_config_enter(zcb->zcb_spa, SCL_CONFIG, FTAG, RW_READER);

	for (i = 0; i < 4; i++) {
	int l = (i < 2) ? BP_GET_LEVEL(bp) : ZB_TOTAL;
	int t = (i & 1) ? type : ZDB_OT_TOTAL;
	int equal;
	zdb_blkstats_t *zb = &zcb->zcb_type[l][t];

	zb->zb_asize += BP_GET_ASIZE(bp);
	zb->zb_lsize += BP_GET_LSIZE(bp);
	zb->zb_psize += BP_GET_PSIZE(bp);
	zb->zb_count++;

	/*
	* The histogram is only big enough to record blocks up to
	* SPA_OLD_MAXBLOCKSIZE; larger blocks go into the last,
	* "other", bucket.
	*/
	unsigned idx = BP_GET_PSIZE(bp) >> SPA_MINBLOCKSHIFT;
	idx = MIN(idx, SPA_OLD_MAXBLOCKSIZE / SPA_MINBLOCKSIZE + 1);
	zb->zb_psize_histogram[idx]++;

	zb->zb_gangs += BP_COUNT_GANG(bp);

	switch (BP_GET_NDVAS(bp)) {
	case 2:
	if (DVA_GET_VDEV(&bp->blk_dva[0]) ==
	DVA_GET_VDEV(&bp->blk_dva[1])) {
	zb->zb_ditto_samevdev++;

	if (same_metaslab(zcb->zcb_spa,
	DVA_GET_VDEV(&bp->blk_dva[0]),
	DVA_GET_OFFSET(&bp->blk_dva[0]),
	DVA_GET_OFFSET(&bp->blk_dva[1])))
	zb->zb_ditto_same_ms++;
	}
	break;
	case 3:
	equal = (DVA_GET_VDEV(&bp->blk_dva[0]) ==
	DVA_GET_VDEV(&bp->blk_dva[1])) +
	(DVA_GET_VDEV(&bp->blk_dva[0]) ==
	DVA_GET_VDEV(&bp->blk_dva[2])) +
	(DVA_GET_VDEV(&bp->blk_dva[1]) ==
	DVA_GET_VDEV(&bp->blk_dva[2]));
	if (equal != 0) {
	zb->zb_ditto_samevdev++;

	if (DVA_GET_VDEV(&bp->blk_dva[0]) ==
	DVA_GET_VDEV(&bp->blk_dva[1]) &&
	same_metaslab(zcb->zcb_spa,
	DVA_GET_VDEV(&bp->blk_dva[0]),
	DVA_GET_OFFSET(&bp->blk_dva[0]),
	DVA_GET_OFFSET(&bp->blk_dva[1])))
	zb->zb_ditto_same_ms++;
	else if (DVA_GET_VDEV(&bp->blk_dva[0]) ==
	DVA_GET_VDEV(&bp->blk_dva[2]) &&
	same_metaslab(zcb->zcb_spa,
	DVA_GET_VDEV(&bp->blk_dva[0]),
	DVA_GET_OFFSET(&bp->blk_dva[0]),
	DVA_GET_OFFSET(&bp->blk_dva[2])))
	zb->zb_ditto_same_ms++;
	else if (DVA_GET_VDEV(&bp->blk_dva[1]) ==
	DVA_GET_VDEV(&bp->blk_dva[2]) &&
	same_metaslab(zcb->zcb_spa,
	DVA_GET_VDEV(&bp->blk_dva[1]),
	DVA_GET_OFFSET(&bp->blk_dva[1]),
	DVA_GET_OFFSET(&bp->blk_dva[2])))
	zb->zb_ditto_same_ms++;
	}
	break;
	}
	}

	spa_config_exit(zcb->zcb_spa, SCL_CONFIG, FTAG);

	if (BP_IS_EMBEDDED(bp)) {
	zcb->zcb_embedded_blocks[BPE_GET_ETYPE(bp)]++;
	zcb->zcb_embedded_histogram[BPE_GET_ETYPE(bp)]
	[BPE_GET_PSIZE(bp)]++;
	return;
	}
	/*
	* The binning histogram bins by powers of two up to
	* SPA_MAXBLOCKSIZE rather than creating bins for
	* every possible blocksize found in the pool.
	*/
	int bin = highbit64(BP_GET_PSIZE(bp)) - 1;

	zcb->zcb_psize_count[bin]++;
	zcb->zcb_psize_len[bin] += BP_GET_PSIZE(bp);
	zcb->zcb_psize_total += BP_GET_PSIZE(bp);

	bin = highbit64(BP_GET_LSIZE(bp)) - 1;

	zcb->zcb_lsize_count[bin]++;
	zcb->zcb_lsize_len[bin] += BP_GET_LSIZE(bp);
	zcb->zcb_lsize_total += BP_GET_LSIZE(bp);

	bin = highbit64(BP_GET_ASIZE(bp)) - 1;

	zcb->zcb_asize_count[bin]++;
	zcb->zcb_asize_len[bin] += BP_GET_ASIZE(bp);
	zcb->zcb_asize_total += BP_GET_ASIZE(bp);

	if (dump_opt['L'])
	return;

	if (BP_GET_DEDUP(bp)) {
	ddt_t *ddt;
	ddt_entry_t *dde;

	ddt = ddt_select(zcb->zcb_spa, bp);
	ddt_enter(ddt);
	dde = ddt_lookup(ddt, bp, B_FALSE);

	if (dde == NULL) {
	refcnt = 0;
	} else {
	ddt_phys_t *ddp = ddt_phys_select(dde, bp);
	ddt_phys_decref(ddp);
	refcnt = ddp->ddp_refcnt;
	if (ddt_phys_total_refcnt(dde) == 0)
	ddt_remove(ddt, dde);
	}
	ddt_exit(ddt);
	}

	VERIFY3U(zio_wait(zio_claim(NULL, zcb->zcb_spa,
	refcnt ? 0 : spa_min_claim_txg(zcb->zcb_spa),
	bp, NULL, NULL, ZIO_FLAG_CANFAIL)), ==, 0);
	}

	static void
	zdb_blkptr_done(zio_t *zio)
	{
	spa_t *spa = zio->io_spa;
	blkptr_t *bp = zio->io_bp;
	int ioerr = zio->io_error;
	zdb_cb_t *zcb = zio->io_private;
	zbookmark_phys_t *zb = &zio->io_bookmark;

	abd_free(zio->io_abd);

	mutex_enter(&spa->spa_scrub_lock);
	spa->spa_load_verify_bytes -= BP_GET_PSIZE(bp);
	cv_broadcast(&spa->spa_scrub_io_cv);

	if (ioerr && !(zio->io_flags & ZIO_FLAG_SPECULATIVE)) {
	char blkbuf[BP_SPRINTF_LEN];

	zcb->zcb_haderrors = 1;
	zcb->zcb_errors[ioerr]++;

	if (dump_opt['b'] >= 2)
	snprintf_blkptr(blkbuf, sizeof (blkbuf), bp);
	else
	blkbuf[0] = '\0';

	(void) printf("zdb_blkptr_cb: "
	"Got error %d reading "
	"<%llu, %llu, %lld, %llx> %s -- skipping\n",
	ioerr,
	(u_longlong_t)zb->zb_objset,
	(u_longlong_t)zb->zb_object,
	(u_longlong_t)zb->zb_level,
	(u_longlong_t)zb->zb_blkid,
	blkbuf);
	}
	mutex_exit(&spa->spa_scrub_lock);
	}

	static int
	zdb_blkptr_cb(spa_t spa, zilog_t zilog, const blkptr_t *bp,
	const zbookmark_phys_t zb, const dnode_phys_t dnp, void *arg)
	{
	zdb_cb_t *zcb = arg;
	dmu_object_type_t type;
	boolean_t is_metadata;

	if (zb->zb_level == ZB_DNODE_LEVEL)
	return (0);

	if (dump_opt['b'] >= 5 && bp->blk_birth > 0) {
	char blkbuf[BP_SPRINTF_LEN];
	snprintf_blkptr(blkbuf, sizeof (blkbuf), bp);
	(void) printf("objset %llu object %llu "
	"level %lld offset 0x%llx %s\n",
	(u_longlong_t)zb->zb_objset,
	(u_longlong_t)zb->zb_object,
	(longlong_t)zb->zb_level,
	(u_longlong_t)blkid2offset(dnp, bp, zb),
	blkbuf);
	}

	if (BP_IS_HOLE(bp) \|\| BP_IS_REDACTED(bp))
	return (0);

	type = BP_GET_TYPE(bp);

	zdb_count_block(zcb, zilog, bp,
	(type & DMU_OT_NEWTYPE) ? ZDB_OT_OTHER : type);

	is_metadata = (BP_GET_LEVEL(bp) != 0 \|\| DMU_OT_IS_METADATA(type));

	if (!BP_IS_EMBEDDED(bp) &&
	(dump_opt['c'] > 1 \|\| (dump_opt['c'] && is_metadata))) {
	size_t size = BP_GET_PSIZE(bp);
	abd_t *abd = abd_alloc(size, B_FALSE);
	int flags = ZIO_FLAG_CANFAIL \| ZIO_FLAG_SCRUB \| ZIO_FLAG_RAW;

	/* If it's an intent log block, failure is expected. */
	if (zb->zb_level == ZB_ZIL_LEVEL)
	flags \|= ZIO_FLAG_SPECULATIVE;

	mutex_enter(&spa->spa_scrub_lock);
	while (spa->spa_load_verify_bytes > max_inflight_bytes)
	cv_wait(&spa->spa_scrub_io_cv, &spa->spa_scrub_lock);
	spa->spa_load_verify_bytes += size;
	mutex_exit(&spa->spa_scrub_lock);

	zio_nowait(zio_read(NULL, spa, bp, abd, size,
	zdb_blkptr_done, zcb, ZIO_PRIORITY_ASYNC_READ, flags, zb));
	}

	zcb->zcb_readfails = 0;

	/* only call gethrtime() every 100 blocks */
	static int iters;
	if (++iters > 100)
	iters = 0;
	else
	return (0);

	if (dump_opt['b'] < 5 && gethrtime() > zcb->zcb_lastprint + NANOSEC) {
	uint64_t now = gethrtime();
	char buf[10];
	uint64_t bytes = zcb->zcb_type[ZB_TOTAL][ZDB_OT_TOTAL].zb_asize;
	int kb_per_sec =
	1 + bytes / (1 + ((now - zcb->zcb_start) / 1000 / 1000));
	int sec_remaining =
	(zcb->zcb_totalasize - bytes) / 1024 / kb_per_sec;

	/* make sure nicenum has enough space */
	CTASSERT(sizeof (buf) >= NN_NUMBUF_SZ);

	zfs_nicebytes(bytes, buf, sizeof (buf));
	(void) fprintf(stderr,
	"\r%5s completed (%4dMB/s) "
	"estimated time remaining: %uhr %02umin %02usec ",
	buf, kb_per_sec / 1024,
	sec_remaining / 60 / 60,
	sec_remaining / 60 % 60,
	sec_remaining % 60);

	zcb->zcb_lastprint = now;
	}

	return (0);
	}

	static void
	zdb_leak(void *arg, uint64_t start, uint64_t size)
	{
	vdev_t *vd = arg;

	(void) printf("leaked space: vdev %llu, offset 0x%llx, size %llu\n",
	(u_longlong_t)vd->vdev_id, (u_longlong_t)start, (u_longlong_t)size);
	}

	static metaslab_ops_t zdb_metaslab_ops = {
	NULL /* alloc */
	};

	/* ARGSUSED */
	static int
	load_unflushed_svr_segs_cb(spa_t spa, space_map_entry_t sme,
	uint64_t txg, void *arg)
	{
	spa_vdev_removal_t *svr = arg;

	uint64_t offset = sme->sme_offset;
	uint64_t size = sme->sme_run;

	/* skip vdevs we don't care about */
	if (sme->sme_vdev != svr->svr_vdev_id)
	return (0);

	vdev_t *vd = vdev_lookup_top(spa, sme->sme_vdev);
	metaslab_t *ms = vd->vdev_ms[offset >> vd->vdev_ms_shift];
	ASSERT(sme->sme_type == SM_ALLOC \|\| sme->sme_type == SM_FREE);

	if (txg < metaslab_unflushed_txg(ms))
	return (0);

	if (sme->sme_type == SM_ALLOC)
	range_tree_add(svr->svr_allocd_segs, offset, size);
	else
	range_tree_remove(svr->svr_allocd_segs, offset, size);

	return (0);
	}

	/* ARGSUSED */
	static void
	claim_segment_impl_cb(uint64_t inner_offset, vdev_t *vd, uint64_t offset,
	uint64_t size, void *arg)
	{
	/*
	* This callback was called through a remap from
	* a device being removed. Therefore, the vdev that
	* this callback is applied to is a concrete
	* vdev.
	*/
	ASSERT(vdev_is_concrete(vd));

	VERIFY0(metaslab_claim_impl(vd, offset, size,
	spa_min_claim_txg(vd->vdev_spa)));
	}

	static void
	claim_segment_cb(void *arg, uint64_t offset, uint64_t size)
	{
	vdev_t *vd = arg;

	vdev_indirect_ops.vdev_op_remap(vd, offset, size,
	claim_segment_impl_cb, NULL);
	}

	/*
	* After accounting for all allocated blocks that are directly referenced,
	* we might have missed a reference to a block from a partially complete
	* (and thus unused) indirect mapping object. We perform a secondary pass
	* through the metaslabs we have already mapped and claim the destination
	* blocks.
	*/
	static void
	zdb_claim_removing(spa_t spa, zdb_cb_t zcb)
	{
	if (dump_opt['L'])
	return;

	if (spa->spa_vdev_removal == NULL)
	return;

	spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER);

	spa_vdev_removal_t *svr = spa->spa_vdev_removal;
	vdev_t *vd = vdev_lookup_top(spa, svr->svr_vdev_id);
	vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping;

	ASSERT0(range_tree_space(svr->svr_allocd_segs));

	range_tree_t *allocs = range_tree_create(NULL, RANGE_SEG64, NULL, 0, 0);
	for (uint64_t msi = 0; msi < vd->vdev_ms_count; msi++) {
	metaslab_t *msp = vd->vdev_ms[msi];

	ASSERT0(range_tree_space(allocs));
	if (msp->ms_sm != NULL)
	VERIFY0(space_map_load(msp->ms_sm, allocs, SM_ALLOC));
	range_tree_vacate(allocs, range_tree_add, svr->svr_allocd_segs);
	}
	range_tree_destroy(allocs);

	iterate_through_spacemap_logs(spa, load_unflushed_svr_segs_cb, svr);

	/*
	* Clear everything past what has been synced,
	* because we have not allocated mappings for
	* it yet.
	*/
	range_tree_clear(svr->svr_allocd_segs,
	vdev_indirect_mapping_max_offset(vim),
	vd->vdev_asize - vdev_indirect_mapping_max_offset(vim));

	zcb->zcb_removing_size += range_tree_space(svr->svr_allocd_segs);
	range_tree_vacate(svr->svr_allocd_segs, claim_segment_cb, vd);

	spa_config_exit(spa, SCL_CONFIG, FTAG);
	}

	/* ARGSUSED */
	static int
	increment_indirect_mapping_cb(void arg, const blkptr_t bp, boolean_t bp_freed,
	dmu_tx_t *tx)
	{
	zdb_cb_t *zcb = arg;
	spa_t *spa = zcb->zcb_spa;
	vdev_t *vd;
	const dva_t *dva = &bp->blk_dva[0];

	ASSERT(!bp_freed);
	ASSERT(!dump_opt['L']);
	ASSERT3U(BP_GET_NDVAS(bp), ==, 1);

	spa_config_enter(spa, SCL_VDEV, FTAG, RW_READER);
	vd = vdev_lookup_top(zcb->zcb_spa, DVA_GET_VDEV(dva));
	ASSERT3P(vd, !=, NULL);
	spa_config_exit(spa, SCL_VDEV, FTAG);

	ASSERT(vd->vdev_indirect_config.vic_mapping_object != 0);
	ASSERT3P(zcb->zcb_vd_obsolete_counts[vd->vdev_id], !=, NULL);

	vdev_indirect_mapping_increment_obsolete_count(
	vd->vdev_indirect_mapping,
	DVA_GET_OFFSET(dva), DVA_GET_ASIZE(dva),
	zcb->zcb_vd_obsolete_counts[vd->vdev_id]);

	return (0);
	}

	static uint32_t *
	zdb_load_obsolete_counts(vdev_t *vd)
	{
	vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping;
	spa_t *spa = vd->vdev_spa;
	spa_condensing_indirect_phys_t *scip =
	&spa->spa_condensing_indirect_phys;
	uint64_t obsolete_sm_object;
	uint32_t *counts;

	VERIFY0(vdev_obsolete_sm_object(vd, &obsolete_sm_object));
	EQUIV(obsolete_sm_object != 0, vd->vdev_obsolete_sm != NULL);
	counts = vdev_indirect_mapping_load_obsolete_counts(vim);
	if (vd->vdev_obsolete_sm != NULL) {
	vdev_indirect_mapping_load_obsolete_spacemap(vim, counts,
	vd->vdev_obsolete_sm);
	}
	if (scip->scip_vdev == vd->vdev_id &&
	scip->scip_prev_obsolete_sm_object != 0) {
	space_map_t *prev_obsolete_sm = NULL;
	VERIFY0(space_map_open(&prev_obsolete_sm, spa->spa_meta_objset,
	scip->scip_prev_obsolete_sm_object, 0, vd->vdev_asize, 0));
	vdev_indirect_mapping_load_obsolete_spacemap(vim, counts,
	prev_obsolete_sm);
	space_map_close(prev_obsolete_sm);
	}
	return (counts);
	}

	static void
	zdb_ddt_leak_init(spa_t spa, zdb_cb_t zcb)
	{
	ddt_bookmark_t ddb;
	ddt_entry_t dde;
	int error;
	int p;

	ASSERT(!dump_opt['L']);

	bzero(&ddb, sizeof (ddb));
	while ((error = ddt_walk(spa, &ddb, &dde)) == 0) {
	blkptr_t blk;
	ddt_phys_t *ddp = dde.dde_phys;

	if (ddb.ddb_class == DDT_CLASS_UNIQUE)
	return;

	ASSERT(ddt_phys_total_refcnt(&dde) > 1);

	for (p = 0; p < DDT_PHYS_TYPES; p++, ddp++) {
	if (ddp->ddp_phys_birth == 0)
	continue;
	ddt_bp_create(ddb.ddb_checksum,
	&dde.dde_key, ddp, &blk);
	if (p == DDT_PHYS_DITTO) {
	zdb_count_block(zcb, NULL, &blk, ZDB_OT_DITTO);
	} else {
	zcb->zcb_dedup_asize +=
	BP_GET_ASIZE(&blk) * (ddp->ddp_refcnt - 1);
	zcb->zcb_dedup_blocks++;
	}
	}
	ddt_t *ddt = spa->spa_ddt[ddb.ddb_checksum];
	ddt_enter(ddt);
	VERIFY(ddt_lookup(ddt, &blk, B_TRUE) != NULL);
	ddt_exit(ddt);
	}

	ASSERT(error == ENOENT);
	}

	typedef struct checkpoint_sm_exclude_entry_arg {
	vdev_t *cseea_vd;
	uint64_t cseea_checkpoint_size;
	} checkpoint_sm_exclude_entry_arg_t;

	static int
	checkpoint_sm_exclude_entry_cb(space_map_entry_t sme, void arg)
	{
	checkpoint_sm_exclude_entry_arg_t *cseea = arg;
	vdev_t *vd = cseea->cseea_vd;
	metaslab_t *ms = vd->vdev_ms[sme->sme_offset >> vd->vdev_ms_shift];
	uint64_t end = sme->sme_offset + sme->sme_run;

	ASSERT(sme->sme_type == SM_FREE);

	/*
	* Since the vdev_checkpoint_sm exists in the vdev level
	* and the ms_sm space maps exist in the metaslab level,
	* an entry in the checkpoint space map could theoretically
	* cross the boundaries of the metaslab that it belongs.
	*
	* In reality, because of the way that we populate and
	* manipulate the checkpoint's space maps currently,
	* there shouldn't be any entries that cross metaslabs.
	* Hence the assertion below.
	*
	* That said, there is no fundamental requirement that
	* the checkpoint's space map entries should not cross
	* metaslab boundaries. So if needed we could add code
	* that handles metaslab-crossing segments in the future.
	*/
	VERIFY3U(sme->sme_offset, >=, ms->ms_start);
	VERIFY3U(end, <=, ms->ms_start + ms->ms_size);

	/*
	* By removing the entry from the allocated segments we
	* also verify that the entry is there to begin with.
	*/
	mutex_enter(&ms->ms_lock);
	range_tree_remove(ms->ms_allocatable, sme->sme_offset, sme->sme_run);
	mutex_exit(&ms->ms_lock);

	cseea->cseea_checkpoint_size += sme->sme_run;
	return (0);
	}

	static void
	zdb_leak_init_vdev_exclude_checkpoint(vdev_t vd, zdb_cb_t zcb)
	{
	spa_t *spa = vd->vdev_spa;
	space_map_t *checkpoint_sm = NULL;
	uint64_t checkpoint_sm_obj;

	/*
	* If there is no vdev_top_zap, we are in a pool whose
	* version predates the pool checkpoint feature.
	*/
	if (vd->vdev_top_zap == 0)
	return;

	/*
	* If there is no reference of the vdev_checkpoint_sm in
	* the vdev_top_zap, then one of the following scenarios
	* is true:
	*
	* 1] There is no checkpoint
	* 2] There is a checkpoint, but no checkpointed blocks
	* have been freed yet
	* 3] The current vdev is indirect
	*
	* In these cases we return immediately.
	*/
	if (zap_contains(spa_meta_objset(spa), vd->vdev_top_zap,
	VDEV_TOP_ZAP_POOL_CHECKPOINT_SM) != 0)
	return;

	VERIFY0(zap_lookup(spa_meta_objset(spa), vd->vdev_top_zap,
	VDEV_TOP_ZAP_POOL_CHECKPOINT_SM, sizeof (uint64_t), 1,
	&checkpoint_sm_obj));

	checkpoint_sm_exclude_entry_arg_t cseea;
	cseea.cseea_vd = vd;
	cseea.cseea_checkpoint_size = 0;

	VERIFY0(space_map_open(&checkpoint_sm, spa_meta_objset(spa),
	checkpoint_sm_obj, 0, vd->vdev_asize, vd->vdev_ashift));

	VERIFY0(space_map_iterate(checkpoint_sm,
	space_map_length(checkpoint_sm),
	checkpoint_sm_exclude_entry_cb, &cseea));
	space_map_close(checkpoint_sm);

	zcb->zcb_checkpoint_size += cseea.cseea_checkpoint_size;
	}

	static void
	zdb_leak_init_exclude_checkpoint(spa_t spa, zdb_cb_t zcb)
	{
	ASSERT(!dump_opt['L']);

	vdev_t *rvd = spa->spa_root_vdev;
	for (uint64_t c = 0; c < rvd->vdev_children; c++) {
	ASSERT3U(c, ==, rvd->vdev_child[c]->vdev_id);
	zdb_leak_init_vdev_exclude_checkpoint(rvd->vdev_child[c], zcb);
	}
	}

	static int
	count_unflushed_space_cb(spa_t spa, space_map_entry_t sme,
	uint64_t txg, void *arg)
	{
	int64_t *ualloc_space = arg;

	uint64_t offset = sme->sme_offset;
	uint64_t vdev_id = sme->sme_vdev;

	vdev_t *vd = vdev_lookup_top(spa, vdev_id);
	if (!vdev_is_concrete(vd))
	return (0);

	metaslab_t *ms = vd->vdev_ms[offset >> vd->vdev_ms_shift];
	ASSERT(sme->sme_type == SM_ALLOC \|\| sme->sme_type == SM_FREE);

	if (txg < metaslab_unflushed_txg(ms))
	return (0);

	if (sme->sme_type == SM_ALLOC)
	*ualloc_space += sme->sme_run;
	else
	*ualloc_space -= sme->sme_run;

	return (0);
	}

	static int64_t
	get_unflushed_alloc_space(spa_t *spa)
	{
	if (dump_opt['L'])
	return (0);

	int64_t ualloc_space = 0;
	iterate_through_spacemap_logs(spa, count_unflushed_space_cb,
	&ualloc_space);
	return (ualloc_space);
	}

	static int
	load_unflushed_cb(spa_t spa, space_map_entry_t sme, uint64_t txg, void *arg)
	{
	maptype_t *uic_maptype = arg;

	uint64_t offset = sme->sme_offset;
	uint64_t size = sme->sme_run;
	uint64_t vdev_id = sme->sme_vdev;

	vdev_t *vd = vdev_lookup_top(spa, vdev_id);

	/* skip indirect vdevs */
	if (!vdev_is_concrete(vd))
	return (0);

	metaslab_t *ms = vd->vdev_ms[offset >> vd->vdev_ms_shift];

	ASSERT(sme->sme_type == SM_ALLOC \|\| sme->sme_type == SM_FREE);
	ASSERT(uic_maptype == SM_ALLOC \|\| uic_maptype == SM_FREE);

	if (txg < metaslab_unflushed_txg(ms))
	return (0);

	if (*uic_maptype == sme->sme_type)
	range_tree_add(ms->ms_allocatable, offset, size);
	else
	range_tree_remove(ms->ms_allocatable, offset, size);

	return (0);
	}

	static void
	load_unflushed_to_ms_allocatables(spa_t *spa, maptype_t maptype)
	{
	iterate_through_spacemap_logs(spa, load_unflushed_cb, &maptype);
	}

	static void
	load_concrete_ms_allocatable_trees(spa_t *spa, maptype_t maptype)
	{
	vdev_t *rvd = spa->spa_root_vdev;
	for (uint64_t i = 0; i < rvd->vdev_children; i++) {
	vdev_t *vd = rvd->vdev_child[i];

	ASSERT3U(i, ==, vd->vdev_id);

	if (vd->vdev_ops == &vdev_indirect_ops)
	continue;

	for (uint64_t m = 0; m < vd->vdev_ms_count; m++) {
	metaslab_t *msp = vd->vdev_ms[m];

	(void) fprintf(stderr,
	"\rloading concrete vdev %llu, "
	"metaslab %llu of %llu ...",
	(longlong_t)vd->vdev_id,
	(longlong_t)msp->ms_id,
	(longlong_t)vd->vdev_ms_count);

	mutex_enter(&msp->ms_lock);
	range_tree_vacate(msp->ms_allocatable, NULL, NULL);

	/*
	* We don't want to spend the CPU manipulating the
	* size-ordered tree, so clear the range_tree ops.
	*/
	msp->ms_allocatable->rt_ops = NULL;

	if (msp->ms_sm != NULL) {
	VERIFY0(space_map_load(msp->ms_sm,
	msp->ms_allocatable, maptype));
	}
	if (!msp->ms_loaded)
	msp->ms_loaded = B_TRUE;
	mutex_exit(&msp->ms_lock);
	}
	}

	load_unflushed_to_ms_allocatables(spa, maptype);
	}

	/*
	* vm_idxp is an in-out parameter which (for indirect vdevs) is the
	* index in vim_entries that has the first entry in this metaslab.
	* On return, it will be set to the first entry after this metaslab.
	*/
	static void
	load_indirect_ms_allocatable_tree(vdev_t vd, metaslab_t msp,
	uint64_t *vim_idxp)
	{
	vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping;

	mutex_enter(&msp->ms_lock);
	range_tree_vacate(msp->ms_allocatable, NULL, NULL);

	/*
	* We don't want to spend the CPU manipulating the
	* size-ordered tree, so clear the range_tree ops.
	*/
	msp->ms_allocatable->rt_ops = NULL;

	for (; *vim_idxp < vdev_indirect_mapping_num_entries(vim);
	(*vim_idxp)++) {
	vdev_indirect_mapping_entry_phys_t *vimep =
	&vim->vim_entries[*vim_idxp];
	uint64_t ent_offset = DVA_MAPPING_GET_SRC_OFFSET(vimep);
	uint64_t ent_len = DVA_GET_ASIZE(&vimep->vimep_dst);
	ASSERT3U(ent_offset, >=, msp->ms_start);
	if (ent_offset >= msp->ms_start + msp->ms_size)
	break;

	/*
	* Mappings do not cross metaslab boundaries,
	* because we create them by walking the metaslabs.
	*/
	ASSERT3U(ent_offset + ent_len, <=,
	msp->ms_start + msp->ms_size);
	range_tree_add(msp->ms_allocatable, ent_offset, ent_len);
	}

	if (!msp->ms_loaded)
	msp->ms_loaded = B_TRUE;
	mutex_exit(&msp->ms_lock);
	}

	static void
	zdb_leak_init_prepare_indirect_vdevs(spa_t spa, zdb_cb_t zcb)
	{
	ASSERT(!dump_opt['L']);

	vdev_t *rvd = spa->spa_root_vdev;
	for (uint64_t c = 0; c < rvd->vdev_children; c++) {
	vdev_t *vd = rvd->vdev_child[c];

	ASSERT3U(c, ==, vd->vdev_id);

	if (vd->vdev_ops != &vdev_indirect_ops)
	continue;

	/*
	* Note: we don't check for mapping leaks on
	* removing vdevs because their ms_allocatable's
	* are used to look for leaks in allocated space.
	*/
	zcb->zcb_vd_obsolete_counts[c] = zdb_load_obsolete_counts(vd);

	/*
	* Normally, indirect vdevs don't have any
	* metaslabs. We want to set them up for
	* zio_claim().
	*/
	VERIFY0(vdev_metaslab_init(vd, 0));

	vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping;
	uint64_t vim_idx = 0;
	for (uint64_t m = 0; m < vd->vdev_ms_count; m++) {

	(void) fprintf(stderr,
	"\rloading indirect vdev %llu, "
	"metaslab %llu of %llu ...",
	(longlong_t)vd->vdev_id,
	(longlong_t)vd->vdev_ms[m]->ms_id,
	(longlong_t)vd->vdev_ms_count);

	load_indirect_ms_allocatable_tree(vd, vd->vdev_ms[m],
	&vim_idx);
	}
	ASSERT3U(vim_idx, ==, vdev_indirect_mapping_num_entries(vim));
	}
	}

	static void
	zdb_leak_init(spa_t spa, zdb_cb_t zcb)
	{
	zcb->zcb_spa = spa;

	if (dump_opt['L'])
	return;

	dsl_pool_t *dp = spa->spa_dsl_pool;
	vdev_t *rvd = spa->spa_root_vdev;

	/*
	* We are going to be changing the meaning of the metaslab's
	* ms_allocatable. Ensure that the allocator doesn't try to
	* use the tree.
	*/
	spa->spa_normal_class->mc_ops = &zdb_metaslab_ops;
	spa->spa_log_class->mc_ops = &zdb_metaslab_ops;

	zcb->zcb_vd_obsolete_counts =
	umem_zalloc(rvd->vdev_children * sizeof (uint32_t *),
	UMEM_NOFAIL);

	/*
	* For leak detection, we overload the ms_allocatable trees
	* to contain allocated segments instead of free segments.
	* As a result, we can't use the normal metaslab_load/unload
	* interfaces.
	*/
	zdb_leak_init_prepare_indirect_vdevs(spa, zcb);
	load_concrete_ms_allocatable_trees(spa, SM_ALLOC);

	/*
	* On load_concrete_ms_allocatable_trees() we loaded all the
	* allocated entries from the ms_sm to the ms_allocatable for
	* each metaslab. If the pool has a checkpoint or is in the
	* middle of discarding a checkpoint, some of these blocks
	* may have been freed but their ms_sm may not have been
	* updated because they are referenced by the checkpoint. In
	* order to avoid false-positives during leak-detection, we
	* go through the vdev's checkpoint space map and exclude all
	* its entries from their relevant ms_allocatable.
	*
	* We also aggregate the space held by the checkpoint and add
	* it to zcb_checkpoint_size.
	*
	* Note that at this point we are also verifying that all the
	* entries on the checkpoint_sm are marked as allocated in
	* the ms_sm of their relevant metaslab.
	* [see comment in checkpoint_sm_exclude_entry_cb()]
	*/
	zdb_leak_init_exclude_checkpoint(spa, zcb);
	ASSERT3U(zcb->zcb_checkpoint_size, ==, spa_get_checkpoint_space(spa));

	/* for cleaner progress output */
	(void) fprintf(stderr, "\n");

	if (bpobj_is_open(&dp->dp_obsolete_bpobj)) {
	ASSERT(spa_feature_is_enabled(spa,
	SPA_FEATURE_DEVICE_REMOVAL));
	(void) bpobj_iterate_nofree(&dp->dp_obsolete_bpobj,
	increment_indirect_mapping_cb, zcb, NULL);
	}

	spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER);
	zdb_ddt_leak_init(spa, zcb);
	spa_config_exit(spa, SCL_CONFIG, FTAG);
	}

	static boolean_t
	zdb_check_for_obsolete_leaks(vdev_t vd, zdb_cb_t zcb)
	{
	boolean_t leaks = B_FALSE;
	vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping;
	uint64_t total_leaked = 0;
	boolean_t are_precise = B_FALSE;

	ASSERT(vim != NULL);

	for (uint64_t i = 0; i < vdev_indirect_mapping_num_entries(vim); i++) {
	vdev_indirect_mapping_entry_phys_t *vimep =
	&vim->vim_entries[i];
	uint64_t obsolete_bytes = 0;
	uint64_t offset = DVA_MAPPING_GET_SRC_OFFSET(vimep);
	metaslab_t *msp = vd->vdev_ms[offset >> vd->vdev_ms_shift];

	/*
	* This is not very efficient but it's easy to
	* verify correctness.
	*/
	for (uint64_t inner_offset = 0;
	inner_offset < DVA_GET_ASIZE(&vimep->vimep_dst);
	inner_offset += 1 << vd->vdev_ashift) {
	if (range_tree_contains(msp->ms_allocatable,
	offset + inner_offset, 1 << vd->vdev_ashift)) {
	obsolete_bytes += 1 << vd->vdev_ashift;
	}
	}

	int64_t bytes_leaked = obsolete_bytes -
	zcb->zcb_vd_obsolete_counts[vd->vdev_id][i];
	ASSERT3U(DVA_GET_ASIZE(&vimep->vimep_dst), >=,
	zcb->zcb_vd_obsolete_counts[vd->vdev_id][i]);

	VERIFY0(vdev_obsolete_counts_are_precise(vd, &are_precise));
	if (bytes_leaked != 0 && (are_precise \|\| dump_opt['d'] >= 5)) {
	(void) printf("obsolete indirect mapping count "
	"mismatch on %llu:%llx:%llx : %llx bytes leaked\n",
	(u_longlong_t)vd->vdev_id,
	(u_longlong_t)DVA_MAPPING_GET_SRC_OFFSET(vimep),
	(u_longlong_t)DVA_GET_ASIZE(&vimep->vimep_dst),
	(u_longlong_t)bytes_leaked);
	}
	total_leaked += ABS(bytes_leaked);
	}

	VERIFY0(vdev_obsolete_counts_are_precise(vd, &are_precise));
	if (!are_precise && total_leaked > 0) {
	int pct_leaked = total_leaked * 100 /
	vdev_indirect_mapping_bytes_mapped(vim);
	(void) printf("cannot verify obsolete indirect mapping "
	"counts of vdev %llu because precise feature was not "
	"enabled when it was removed: %d%% (%llx bytes) of mapping"
	"unreferenced\n",
	(u_longlong_t)vd->vdev_id, pct_leaked,
	(u_longlong_t)total_leaked);
	} else if (total_leaked > 0) {
	(void) printf("obsolete indirect mapping count mismatch "
	"for vdev %llu -- %llx total bytes mismatched\n",
	(u_longlong_t)vd->vdev_id,
	(u_longlong_t)total_leaked);
	leaks \|= B_TRUE;
	}

	vdev_indirect_mapping_free_obsolete_counts(vim,
	zcb->zcb_vd_obsolete_counts[vd->vdev_id]);
	zcb->zcb_vd_obsolete_counts[vd->vdev_id] = NULL;

	return (leaks);
	}

	static boolean_t
	zdb_leak_fini(spa_t spa, zdb_cb_t zcb)
	{
	if (dump_opt['L'])
	return (B_FALSE);

	boolean_t leaks = B_FALSE;
	vdev_t *rvd = spa->spa_root_vdev;
	for (unsigned c = 0; c < rvd->vdev_children; c++) {
	vdev_t *vd = rvd->vdev_child[c];
	metaslab_group_t *mg __maybe_unused = vd->vdev_mg;

	if (zcb->zcb_vd_obsolete_counts[c] != NULL) {
	leaks \|= zdb_check_for_obsolete_leaks(vd, zcb);
	}

	for (uint64_t m = 0; m < vd->vdev_ms_count; m++) {
	metaslab_t *msp = vd->vdev_ms[m];
	ASSERT3P(mg, ==, msp->ms_group);

	/*
	* ms_allocatable has been overloaded
	* to contain allocated segments. Now that
	* we finished traversing all blocks, any
	* block that remains in the ms_allocatable
	* represents an allocated block that we
	* did not claim during the traversal.
	* Claimed blocks would have been removed
	* from the ms_allocatable. For indirect
	* vdevs, space remaining in the tree
	* represents parts of the mapping that are
	* not referenced, which is not a bug.
	*/
	if (vd->vdev_ops == &vdev_indirect_ops) {
	range_tree_vacate(msp->ms_allocatable,
	NULL, NULL);
	} else {
	range_tree_vacate(msp->ms_allocatable,
	zdb_leak, vd);
	}
	if (msp->ms_loaded) {
	msp->ms_loaded = B_FALSE;
	}
	}
	}

	umem_free(zcb->zcb_vd_obsolete_counts,
	rvd->vdev_children * sizeof (uint32_t *));
	zcb->zcb_vd_obsolete_counts = NULL;

	return (leaks);
	}

	/* ARGSUSED */
	static int
	count_block_cb(void arg, const blkptr_t bp, dmu_tx_t *tx)
	{
	zdb_cb_t *zcb = arg;

	if (dump_opt['b'] >= 5) {
	char blkbuf[BP_SPRINTF_LEN];
	snprintf_blkptr(blkbuf, sizeof (blkbuf), bp);
	(void) printf("[%s] %s\n",
	"deferred free", blkbuf);
	}
	zdb_count_block(zcb, NULL, bp, ZDB_OT_DEFERRED);
	return (0);
	}

	/*
	* Iterate over livelists which have been destroyed by the user but
	* are still present in the MOS, waiting to be freed
	*/
	static void
	iterate_deleted_livelists(spa_t spa, ll_iter_t func, void arg)
	{
	objset_t *mos = spa->spa_meta_objset;
	uint64_t zap_obj;
	int err = zap_lookup(mos, DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_DELETED_CLONES, sizeof (uint64_t), 1, &zap_obj);
	if (err == ENOENT)
	return;
	ASSERT0(err);

	zap_cursor_t zc;
	zap_attribute_t attr;
	dsl_deadlist_t ll;
	/* NULL out os prior to dsl_deadlist_open in case it's garbage */
	ll.dl_os = NULL;
	for (zap_cursor_init(&zc, mos, zap_obj);
	zap_cursor_retrieve(&zc, &attr) == 0;
	(void) zap_cursor_advance(&zc)) {
	dsl_deadlist_open(&ll, mos, attr.za_first_integer);
	func(&ll, arg);
	dsl_deadlist_close(&ll);
	}
	zap_cursor_fini(&zc);
	}

	static int
	bpobj_count_block_cb(void arg, const blkptr_t bp, boolean_t bp_freed,
	dmu_tx_t *tx)
	{
	ASSERT(!bp_freed);
	return (count_block_cb(arg, bp, tx));
	}

	static int
	livelist_entry_count_blocks_cb(void args, dsl_deadlist_entry_t dle)
	{
	zdb_cb_t *zbc = args;
	bplist_t blks;
	bplist_create(&blks);
	/* determine which blocks have been alloc'd but not freed */
	VERIFY0(dsl_process_sub_livelist(&dle->dle_bpobj, &blks, NULL, NULL));
	/* count those blocks */
	(void) bplist_iterate(&blks, count_block_cb, zbc, NULL);
	bplist_destroy(&blks);
	return (0);
	}

	static void
	livelist_count_blocks(dsl_deadlist_t ll, void arg)
	{
	dsl_deadlist_iterate(ll, livelist_entry_count_blocks_cb, arg);
	}

	/*
	* Count the blocks in the livelists that have been destroyed by the user
	* but haven't yet been freed.
	*/
	static void
	deleted_livelists_count_blocks(spa_t spa, zdb_cb_t zbc)
	{
	iterate_deleted_livelists(spa, livelist_count_blocks, zbc);
	}

	static void
	dump_livelist_cb(dsl_deadlist_t ll, void arg)
	{
	ASSERT3P(arg, ==, NULL);
	global_feature_count[SPA_FEATURE_LIVELIST]++;
	dump_blkptr_list(ll, "Deleted Livelist");
	dsl_deadlist_iterate(ll, sublivelist_verify_lightweight, NULL);
	}

	/*
	* Print out, register object references to, and increment feature counts for
	* livelists that have been destroyed by the user but haven't yet been freed.
	*/
	static void
	deleted_livelists_dump_mos(spa_t *spa)
	{
	uint64_t zap_obj;
	objset_t *mos = spa->spa_meta_objset;
	int err = zap_lookup(mos, DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_DELETED_CLONES, sizeof (uint64_t), 1, &zap_obj);
	if (err == ENOENT)
	return;
	mos_obj_refd(zap_obj);
	iterate_deleted_livelists(spa, dump_livelist_cb, NULL);
	}

	static int
	dump_block_stats(spa_t *spa)
	{
	zdb_cb_t zcb;
	zdb_blkstats_t zb, tzb;
	uint64_t norm_alloc, norm_space, total_alloc, total_found;
	int flags = TRAVERSE_PRE \| TRAVERSE_PREFETCH_METADATA \|
	TRAVERSE_NO_DECRYPT \| TRAVERSE_HARD;
	boolean_t leaks = B_FALSE;
	int e, c, err;
	bp_embedded_type_t i;

	bzero(&zcb, sizeof (zcb));
	(void) printf("\nTraversing all blocks %s%s%s%s%s...\n\n",
	(dump_opt['c'] \|\| !dump_opt['L']) ? "to verify " : "",
	(dump_opt['c'] == 1) ? "metadata " : "",
	dump_opt['c'] ? "checksums " : "",
	(dump_opt['c'] && !dump_opt['L']) ? "and verify " : "",
	!dump_opt['L'] ? "nothing leaked " : "");

	/*
	* When leak detection is enabled we load all space maps as SM_ALLOC
	* maps, then traverse the pool claiming each block we discover. If
	* the pool is perfectly consistent, the segment trees will be empty
	* when we're done. Anything left over is a leak; any block we can't
	* claim (because it's not part of any space map) is a double
	* allocation, reference to a freed block, or an unclaimed log block.
	*
	* When leak detection is disabled (-L option) we still traverse the
	* pool claiming each block we discover, but we skip opening any space
	* maps.
	*/
	bzero(&zcb, sizeof (zdb_cb_t));
	zdb_leak_init(spa, &zcb);

	/*
	* If there's a deferred-free bplist, process that first.
	*/
	(void) bpobj_iterate_nofree(&spa->spa_deferred_bpobj,
	bpobj_count_block_cb, &zcb, NULL);

	if (spa_version(spa) >= SPA_VERSION_DEADLISTS) {
	(void) bpobj_iterate_nofree(&spa->spa_dsl_pool->dp_free_bpobj,
	bpobj_count_block_cb, &zcb, NULL);
	}

	zdb_claim_removing(spa, &zcb);

	if (spa_feature_is_active(spa, SPA_FEATURE_ASYNC_DESTROY)) {
	VERIFY3U(0, ==, bptree_iterate(spa->spa_meta_objset,
	spa->spa_dsl_pool->dp_bptree_obj, B_FALSE, count_block_cb,
	&zcb, NULL));
	}

	deleted_livelists_count_blocks(spa, &zcb);

	if (dump_opt['c'] > 1)
	flags \|= TRAVERSE_PREFETCH_DATA;

	zcb.zcb_totalasize = metaslab_class_get_alloc(spa_normal_class(spa));
	zcb.zcb_totalasize += metaslab_class_get_alloc(spa_special_class(spa));
	zcb.zcb_totalasize += metaslab_class_get_alloc(spa_dedup_class(spa));
	zcb.zcb_start = zcb.zcb_lastprint = gethrtime();
	err = traverse_pool(spa, 0, flags, zdb_blkptr_cb, &zcb);

	/*
	* If we've traversed the data blocks then we need to wait for those
	* I/Os to complete. We leverage "The Godfather" zio to wait on
	* all async I/Os to complete.
	*/
	if (dump_opt['c']) {
	for (c = 0; c < max_ncpus; c++) {
	(void) zio_wait(spa->spa_async_zio_root[c]);
	spa->spa_async_zio_root[c] = zio_root(spa, NULL, NULL,
	ZIO_FLAG_CANFAIL \| ZIO_FLAG_SPECULATIVE \|
	ZIO_FLAG_GODFATHER);
	}
	}
	ASSERT0(spa->spa_load_verify_bytes);

	/*
	* Done after zio_wait() since zcb_haderrors is modified in
	* zdb_blkptr_done()
	*/
	zcb.zcb_haderrors \|= err;

	if (zcb.zcb_haderrors) {
	(void) printf("\nError counts:\n\n");
	(void) printf("\t%5s %s\n", "errno", "count");
	for (e = 0; e < 256; e++) {
	if (zcb.zcb_errors[e] != 0) {
	(void) printf("\t%5d %llu\n",
	e, (u_longlong_t)zcb.zcb_errors[e]);
	}
	}
	}

	/*
	* Report any leaked segments.
	*/
	leaks \|= zdb_leak_fini(spa, &zcb);

	tzb = &zcb.zcb_type[ZB_TOTAL][ZDB_OT_TOTAL];

	norm_alloc = metaslab_class_get_alloc(spa_normal_class(spa));
	norm_space = metaslab_class_get_space(spa_normal_class(spa));

	total_alloc = norm_alloc +
	metaslab_class_get_alloc(spa_log_class(spa)) +
	metaslab_class_get_alloc(spa_special_class(spa)) +
	metaslab_class_get_alloc(spa_dedup_class(spa)) +
	get_unflushed_alloc_space(spa);
	total_found = tzb->zb_asize - zcb.zcb_dedup_asize +
	zcb.zcb_removing_size + zcb.zcb_checkpoint_size;

	if (total_found == total_alloc && !dump_opt['L']) {
	(void) printf("\n\tNo leaks (block sum matches space"
	" maps exactly)\n");
	} else if (!dump_opt['L']) {
	(void) printf("block traversal size %llu != alloc %llu "
	"(%s %lld)\n",
	(u_longlong_t)total_found,
	(u_longlong_t)total_alloc,
	(dump_opt['L']) ? "unreachable" : "leaked",
	(longlong_t)(total_alloc - total_found));
	leaks = B_TRUE;
	}

	if (tzb->zb_count == 0)
	return (2);

	(void) printf("\n");
	(void) printf("\t%-16s %14llu\n", "bp count:",
	(u_longlong_t)tzb->zb_count);
	(void) printf("\t%-16s %14llu\n", "ganged count:",
	(longlong_t)tzb->zb_gangs);
	(void) printf("\t%-16s %14llu avg: %6llu\n", "bp logical:",
	(u_longlong_t)tzb->zb_lsize,
	(u_longlong_t)(tzb->zb_lsize / tzb->zb_count));
	(void) printf("\t%-16s %14llu avg: %6llu compression: %6.2f\n",
	"bp physical:", (u_longlong_t)tzb->zb_psize,
	(u_longlong_t)(tzb->zb_psize / tzb->zb_count),
	(double)tzb->zb_lsize / tzb->zb_psize);
	(void) printf("\t%-16s %14llu avg: %6llu compression: %6.2f\n",
	"bp allocated:", (u_longlong_t)tzb->zb_asize,
	(u_longlong_t)(tzb->zb_asize / tzb->zb_count),
	(double)tzb->zb_lsize / tzb->zb_asize);
	(void) printf("\t%-16s %14llu ref>1: %6llu deduplication: %6.2f\n",
	"bp deduped:", (u_longlong_t)zcb.zcb_dedup_asize,
	(u_longlong_t)zcb.zcb_dedup_blocks,
	(double)zcb.zcb_dedup_asize / tzb->zb_asize + 1.0);
	(void) printf("\t%-16s %14llu used: %5.2f%%\n", "Normal class:",
	(u_longlong_t)norm_alloc, 100.0 * norm_alloc / norm_space);

	if (spa_special_class(spa)->mc_rotor != NULL) {
	uint64_t alloc = metaslab_class_get_alloc(
	spa_special_class(spa));
	uint64_t space = metaslab_class_get_space(
	spa_special_class(spa));

	(void) printf("\t%-16s %14llu used: %5.2f%%\n",
	"Special class", (u_longlong_t)alloc,
	100.0 * alloc / space);
	}

	if (spa_dedup_class(spa)->mc_rotor != NULL) {
	uint64_t alloc = metaslab_class_get_alloc(
	spa_dedup_class(spa));
	uint64_t space = metaslab_class_get_space(
	spa_dedup_class(spa));

	(void) printf("\t%-16s %14llu used: %5.2f%%\n",
	"Dedup class", (u_longlong_t)alloc,
	100.0 * alloc / space);
	}

	for (i = 0; i < NUM_BP_EMBEDDED_TYPES; i++) {
	if (zcb.zcb_embedded_blocks[i] == 0)
	continue;
	(void) printf("\n");
	(void) printf("\tadditional, non-pointer bps of type %u: "
	"%10llu\n",
	i, (u_longlong_t)zcb.zcb_embedded_blocks[i]);

	if (dump_opt['b'] >= 3) {
	(void) printf("\t number of (compressed) bytes: "
	"number of bps\n");
	dump_histogram(zcb.zcb_embedded_histogram[i],
	sizeof (zcb.zcb_embedded_histogram[i]) /
	sizeof (zcb.zcb_embedded_histogram[i][0]), 0);
	}
	}

	if (tzb->zb_ditto_samevdev != 0) {
	(void) printf("\tDittoed blocks on same vdev: %llu\n",
	(longlong_t)tzb->zb_ditto_samevdev);
	}
	if (tzb->zb_ditto_same_ms != 0) {
	(void) printf("\tDittoed blocks in same metaslab: %llu\n",
	(longlong_t)tzb->zb_ditto_same_ms);
	}

	for (uint64_t v = 0; v < spa->spa_root_vdev->vdev_children; v++) {
	vdev_t *vd = spa->spa_root_vdev->vdev_child[v];
	vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping;

	if (vim == NULL) {
	continue;
	}

	char mem[32];
	zdb_nicenum(vdev_indirect_mapping_num_entries(vim),
	mem, vdev_indirect_mapping_size(vim));

	(void) printf("\tindirect vdev id %llu has %llu segments "
	"(%s in memory)\n",
	(longlong_t)vd->vdev_id,
	(longlong_t)vdev_indirect_mapping_num_entries(vim), mem);
	}

	if (dump_opt['b'] >= 2) {
	int l, t, level;
	(void) printf("\nBlocks\tLSIZE\tPSIZE\tASIZE"
	"\t avg\t comp\t%%Total\tType\n");

	for (t = 0; t <= ZDB_OT_TOTAL; t++) {
	char csize[32], lsize[32], psize[32], asize[32];
	char avg[32], gang[32];
	const char *typename;

	/* make sure nicenum has enough space */
	CTASSERT(sizeof (csize) >= NN_NUMBUF_SZ);
	CTASSERT(sizeof (lsize) >= NN_NUMBUF_SZ);
	CTASSERT(sizeof (psize) >= NN_NUMBUF_SZ);
	CTASSERT(sizeof (asize) >= NN_NUMBUF_SZ);
	CTASSERT(sizeof (avg) >= NN_NUMBUF_SZ);
	CTASSERT(sizeof (gang) >= NN_NUMBUF_SZ);

	if (t < DMU_OT_NUMTYPES)
	typename = dmu_ot[t].ot_name;
	else
	typename = zdb_ot_extname[t - DMU_OT_NUMTYPES];

	if (zcb.zcb_type[ZB_TOTAL][t].zb_asize == 0) {
	(void) printf("%6s\t%5s\t%5s\t%5s"
	"\t%5s\t%5s\t%6s\t%s\n",
	"-",
	"-",
	"-",
	"-",
	"-",
	"-",
	"-",
	typename);
	continue;
	}

	for (l = ZB_TOTAL - 1; l >= -1; l--) {
	level = (l == -1 ? ZB_TOTAL : l);
	zb = &zcb.zcb_type[level][t];

	if (zb->zb_asize == 0)
	continue;

	if (dump_opt['b'] < 3 && level != ZB_TOTAL)
	continue;

	if (level == 0 && zb->zb_asize ==
	zcb.zcb_type[ZB_TOTAL][t].zb_asize)
	continue;

	zdb_nicenum(zb->zb_count, csize,
	sizeof (csize));
	zdb_nicenum(zb->zb_lsize, lsize,
	sizeof (lsize));
	zdb_nicenum(zb->zb_psize, psize,
	sizeof (psize));
	zdb_nicenum(zb->zb_asize, asize,
	sizeof (asize));
	zdb_nicenum(zb->zb_asize / zb->zb_count, avg,
	sizeof (avg));
	zdb_nicenum(zb->zb_gangs, gang, sizeof (gang));

	(void) printf("%6s\t%5s\t%5s\t%5s\t%5s"
	"\t%5.2f\t%6.2f\t",
	csize, lsize, psize, asize, avg,
	(double)zb->zb_lsize / zb->zb_psize,
	100.0 * zb->zb_asize / tzb->zb_asize);

	if (level == ZB_TOTAL)
	(void) printf("%s\n", typename);
	else
	(void) printf(" L%d %s\n",
	level, typename);

	if (dump_opt['b'] >= 3 && zb->zb_gangs > 0) {
	(void) printf("\t number of ganged "
	"blocks: %s\n", gang);
	}

	if (dump_opt['b'] >= 4) {
	(void) printf("psize "
	"(in 512-byte sectors): "
	"number of blocks\n");
	dump_histogram(zb->zb_psize_histogram,
	PSIZE_HISTO_SIZE, 0);
	}
	}
	}

	/* Output a table summarizing block sizes in the pool */
	if (dump_opt['b'] >= 2) {
	dump_size_histograms(&zcb);
	}
	}

	(void) printf("\n");

	if (leaks)
	return (2);

	if (zcb.zcb_haderrors)
	return (3);

	return (0);
	}

	typedef struct zdb_ddt_entry {
	ddt_key_t zdde_key;
	uint64_t zdde_ref_blocks;
	uint64_t zdde_ref_lsize;
	uint64_t zdde_ref_psize;
	uint64_t zdde_ref_dsize;
	avl_node_t zdde_node;
	} zdb_ddt_entry_t;

	/* ARGSUSED */
	static int
	zdb_ddt_add_cb(spa_t spa, zilog_t zilog, const blkptr_t *bp,
	const zbookmark_phys_t zb, const dnode_phys_t dnp, void *arg)
	{
	avl_tree_t *t = arg;
	avl_index_t where;
	zdb_ddt_entry_t *zdde, zdde_search;

	if (zb->zb_level == ZB_DNODE_LEVEL \|\| BP_IS_HOLE(bp) \|\|
	BP_IS_EMBEDDED(bp))
	return (0);

	if (dump_opt['S'] > 1 && zb->zb_level == ZB_ROOT_LEVEL) {
	(void) printf("traversing objset %llu, %llu objects, "
	"%lu blocks so far\n",
	(u_longlong_t)zb->zb_objset,
	(u_longlong_t)BP_GET_FILL(bp),
	avl_numnodes(t));
	}

	if (BP_IS_HOLE(bp) \|\| BP_GET_CHECKSUM(bp) == ZIO_CHECKSUM_OFF \|\|
	BP_GET_LEVEL(bp) > 0 \|\| DMU_OT_IS_METADATA(BP_GET_TYPE(bp)))
	return (0);

	ddt_key_fill(&zdde_search.zdde_key, bp);

	zdde = avl_find(t, &zdde_search, &where);

	if (zdde == NULL) {
	zdde = umem_zalloc(sizeof (*zdde), UMEM_NOFAIL);
	zdde->zdde_key = zdde_search.zdde_key;
	avl_insert(t, zdde, where);
	}

	zdde->zdde_ref_blocks += 1;
	zdde->zdde_ref_lsize += BP_GET_LSIZE(bp);
	zdde->zdde_ref_psize += BP_GET_PSIZE(bp);
	zdde->zdde_ref_dsize += bp_get_dsize_sync(spa, bp);

	return (0);
	}

	static void
	dump_simulated_ddt(spa_t *spa)
	{
	avl_tree_t t;
	void *cookie = NULL;
	zdb_ddt_entry_t *zdde;
	ddt_histogram_t ddh_total;
	ddt_stat_t dds_total;

	bzero(&ddh_total, sizeof (ddh_total));
	bzero(&dds_total, sizeof (dds_total));
	avl_create(&t, ddt_entry_compare,
	sizeof (zdb_ddt_entry_t), offsetof(zdb_ddt_entry_t, zdde_node));

	spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER);

	(void) traverse_pool(spa, 0, TRAVERSE_PRE \| TRAVERSE_PREFETCH_METADATA \|
	TRAVERSE_NO_DECRYPT, zdb_ddt_add_cb, &t);

	spa_config_exit(spa, SCL_CONFIG, FTAG);

	while ((zdde = avl_destroy_nodes(&t, &cookie)) != NULL) {
	ddt_stat_t dds;
	uint64_t refcnt = zdde->zdde_ref_blocks;
	ASSERT(refcnt != 0);

	dds.dds_blocks = zdde->zdde_ref_blocks / refcnt;
	dds.dds_lsize = zdde->zdde_ref_lsize / refcnt;
	dds.dds_psize = zdde->zdde_ref_psize / refcnt;
	dds.dds_dsize = zdde->zdde_ref_dsize / refcnt;

	dds.dds_ref_blocks = zdde->zdde_ref_blocks;
	dds.dds_ref_lsize = zdde->zdde_ref_lsize;
	dds.dds_ref_psize = zdde->zdde_ref_psize;
	dds.dds_ref_dsize = zdde->zdde_ref_dsize;

	ddt_stat_add(&ddh_total.ddh_stat[highbit64(refcnt) - 1],
	&dds, 0);

	umem_free(zdde, sizeof (*zdde));
	}

	avl_destroy(&t);

	ddt_histogram_stat(&dds_total, &ddh_total);

	(void) printf("Simulated DDT histogram:\n");

	zpool_dump_ddt(&dds_total, &ddh_total);

	dump_dedup_ratio(&dds_total);
	}

	static int
	verify_device_removal_feature_counts(spa_t *spa)
	{
	uint64_t dr_feature_refcount = 0;
	uint64_t oc_feature_refcount = 0;
	uint64_t indirect_vdev_count = 0;
	uint64_t precise_vdev_count = 0;
	uint64_t obsolete_counts_object_count = 0;
	uint64_t obsolete_sm_count = 0;
	uint64_t obsolete_counts_count = 0;
	uint64_t scip_count = 0;
	uint64_t obsolete_bpobj_count = 0;
	int ret = 0;

	spa_condensing_indirect_phys_t *scip =
	&spa->spa_condensing_indirect_phys;
	if (scip->scip_next_mapping_object != 0) {
	vdev_t *vd = spa->spa_root_vdev->vdev_child[scip->scip_vdev];
	ASSERT(scip->scip_prev_obsolete_sm_object != 0);
	ASSERT3P(vd->vdev_ops, ==, &vdev_indirect_ops);

	(void) printf("Condensing indirect vdev %llu: new mapping "
	"object %llu, prev obsolete sm %llu\n",
	(u_longlong_t)scip->scip_vdev,
	(u_longlong_t)scip->scip_next_mapping_object,
	(u_longlong_t)scip->scip_prev_obsolete_sm_object);
	if (scip->scip_prev_obsolete_sm_object != 0) {
	space_map_t *prev_obsolete_sm = NULL;
	VERIFY0(space_map_open(&prev_obsolete_sm,
	spa->spa_meta_objset,
	scip->scip_prev_obsolete_sm_object,
	0, vd->vdev_asize, 0));
	dump_spacemap(spa->spa_meta_objset, prev_obsolete_sm);
	(void) printf("\n");
	space_map_close(prev_obsolete_sm);
	}

	scip_count += 2;
	}

	for (uint64_t i = 0; i < spa->spa_root_vdev->vdev_children; i++) {
	vdev_t *vd = spa->spa_root_vdev->vdev_child[i];
	vdev_indirect_config_t *vic = &vd->vdev_indirect_config;

	if (vic->vic_mapping_object != 0) {
	ASSERT(vd->vdev_ops == &vdev_indirect_ops \|\|
	vd->vdev_removing);
	indirect_vdev_count++;

	if (vd->vdev_indirect_mapping->vim_havecounts) {
	obsolete_counts_count++;
	}
	}

	boolean_t are_precise;
	VERIFY0(vdev_obsolete_counts_are_precise(vd, &are_precise));
	if (are_precise) {
	ASSERT(vic->vic_mapping_object != 0);
	precise_vdev_count++;
	}

	uint64_t obsolete_sm_object;
	VERIFY0(vdev_obsolete_sm_object(vd, &obsolete_sm_object));
	if (obsolete_sm_object != 0) {
	ASSERT(vic->vic_mapping_object != 0);
	obsolete_sm_count++;
	}
	}

	(void) feature_get_refcount(spa,
	&spa_feature_table[SPA_FEATURE_DEVICE_REMOVAL],
	&dr_feature_refcount);
	(void) feature_get_refcount(spa,
	&spa_feature_table[SPA_FEATURE_OBSOLETE_COUNTS],
	&oc_feature_refcount);

	if (dr_feature_refcount != indirect_vdev_count) {
	ret = 1;
	(void) printf("Number of indirect vdevs (%llu) " \
	"does not match feature count (%llu)\n",
	(u_longlong_t)indirect_vdev_count,
	(u_longlong_t)dr_feature_refcount);
	} else {
	(void) printf("Verified device_removal feature refcount " \
	"of %llu is correct\n",
	(u_longlong_t)dr_feature_refcount);
	}

	if (zap_contains(spa_meta_objset(spa), DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_OBSOLETE_BPOBJ) == 0) {
	obsolete_bpobj_count++;
	}


	obsolete_counts_object_count = precise_vdev_count;
	obsolete_counts_object_count += obsolete_sm_count;
	obsolete_counts_object_count += obsolete_counts_count;
	obsolete_counts_object_count += scip_count;
	obsolete_counts_object_count += obsolete_bpobj_count;
	obsolete_counts_object_count += remap_deadlist_count;

	if (oc_feature_refcount != obsolete_counts_object_count) {
	ret = 1;
	(void) printf("Number of obsolete counts objects (%llu) " \
	"does not match feature count (%llu)\n",
	(u_longlong_t)obsolete_counts_object_count,
	(u_longlong_t)oc_feature_refcount);
	(void) printf("pv:%llu os:%llu oc:%llu sc:%llu "
	"ob:%llu rd:%llu\n",
	(u_longlong_t)precise_vdev_count,
	(u_longlong_t)obsolete_sm_count,
	(u_longlong_t)obsolete_counts_count,
	(u_longlong_t)scip_count,
	(u_longlong_t)obsolete_bpobj_count,
	(u_longlong_t)remap_deadlist_count);
	} else {
	(void) printf("Verified indirect_refcount feature refcount " \
	"of %llu is correct\n",
	(u_longlong_t)oc_feature_refcount);
	}
	return (ret);
	}

	static void
	zdb_set_skip_mmp(char *target)
	{
	spa_t *spa;

	/*
	* Disable the activity check to allow examination of
	* active pools.
	*/
	mutex_enter(&spa_namespace_lock);
	if ((spa = spa_lookup(target)) != NULL) {
	spa->spa_import_flags \|= ZFS_IMPORT_SKIP_MMP;
	}
	mutex_exit(&spa_namespace_lock);
	}

	#define BOGUS_SUFFIX "_CHECKPOINTED_UNIVERSE"
	/*
	* Import the checkpointed state of the pool specified by the target
	* parameter as readonly. The function also accepts a pool config
	* as an optional parameter, else it attempts to infer the config by
	* the name of the target pool.
	*
	* Note that the checkpointed state's pool name will be the name of
	* the original pool with the above suffix appended to it. In addition,
	* if the target is not a pool name (e.g. a path to a dataset) then
	* the new_path parameter is populated with the updated path to
	* reflect the fact that we are looking into the checkpointed state.
	*
	* The function returns a newly-allocated copy of the name of the
	* pool containing the checkpointed state. When this copy is no
	* longer needed it should be freed with free(3C). Same thing
	* applies to the new_path parameter if allocated.
	*/
	static char *
	import_checkpointed_state(char target, nvlist_t cfg, char **new_path)
	{
	int error = 0;
	char poolname, bogus_name = NULL;

	/* If the target is not a pool, the extract the pool name */
	char *path_start = strchr(target, '/');
	if (path_start != NULL) {
	size_t poolname_len = path_start - target;
	poolname = strndup(target, poolname_len);
	} else {
	poolname = target;
	}

	if (cfg == NULL) {
	zdb_set_skip_mmp(poolname);
	error = spa_get_stats(poolname, &cfg, NULL, 0);
	if (error != 0) {
	fatal("Tried to read config of pool \"%s\" but "
	"spa_get_stats() failed with error %d\n",
	poolname, error);
	}
	}

	if (asprintf(&bogus_name, "%s%s", poolname, BOGUS_SUFFIX) == -1)
	return (NULL);
	fnvlist_add_string(cfg, ZPOOL_CONFIG_POOL_NAME, bogus_name);

	error = spa_import(bogus_name, cfg, NULL,
	ZFS_IMPORT_MISSING_LOG \| ZFS_IMPORT_CHECKPOINT \|
	ZFS_IMPORT_SKIP_MMP);
	if (error != 0) {
	fatal("Tried to import pool \"%s\" but spa_import() failed "
	"with error %d\n", bogus_name, error);
	}

	if (new_path != NULL && path_start != NULL) {
	if (asprintf(new_path, "%s%s", bogus_name, path_start) == -1) {
	if (path_start != NULL)
	free(poolname);
	return (NULL);
	}
	}

	if (target != poolname)
	free(poolname);

	return (bogus_name);
	}

	typedef struct verify_checkpoint_sm_entry_cb_arg {
	vdev_t *vcsec_vd;

	/* the following fields are only used for printing progress */
	uint64_t vcsec_entryid;
	uint64_t vcsec_num_entries;
	} verify_checkpoint_sm_entry_cb_arg_t;

	#define ENTRIES_PER_PROGRESS_UPDATE 10000

	static int
	verify_checkpoint_sm_entry_cb(space_map_entry_t sme, void arg)
	{
	verify_checkpoint_sm_entry_cb_arg_t *vcsec = arg;
	vdev_t *vd = vcsec->vcsec_vd;
	metaslab_t *ms = vd->vdev_ms[sme->sme_offset >> vd->vdev_ms_shift];
	uint64_t end = sme->sme_offset + sme->sme_run;

	ASSERT(sme->sme_type == SM_FREE);

	if ((vcsec->vcsec_entryid % ENTRIES_PER_PROGRESS_UPDATE) == 0) {
	(void) fprintf(stderr,
	"\rverifying vdev %llu, space map entry %llu of %llu ...",
	(longlong_t)vd->vdev_id,
	(longlong_t)vcsec->vcsec_entryid,
	(longlong_t)vcsec->vcsec_num_entries);
	}
	vcsec->vcsec_entryid++;

	/*
	* See comment in checkpoint_sm_exclude_entry_cb()
	*/
	VERIFY3U(sme->sme_offset, >=, ms->ms_start);
	VERIFY3U(end, <=, ms->ms_start + ms->ms_size);

	/*
	* The entries in the vdev_checkpoint_sm should be marked as
	* allocated in the checkpointed state of the pool, therefore
	* their respective ms_allocateable trees should not contain them.
	*/
	mutex_enter(&ms->ms_lock);
	range_tree_verify_not_present(ms->ms_allocatable,
	sme->sme_offset, sme->sme_run);
	mutex_exit(&ms->ms_lock);

	return (0);
	}

	/*
	* Verify that all segments in the vdev_checkpoint_sm are allocated
	* according to the checkpoint's ms_sm (i.e. are not in the checkpoint's
	* ms_allocatable).
	*
	* Do so by comparing the checkpoint space maps (vdev_checkpoint_sm) of
	* each vdev in the current state of the pool to the metaslab space maps
	* (ms_sm) of the checkpointed state of the pool.
	*
	* Note that the function changes the state of the ms_allocatable
	* trees of the current spa_t. The entries of these ms_allocatable
	* trees are cleared out and then repopulated from with the free
	* entries of their respective ms_sm space maps.
	*/
	static void
	verify_checkpoint_vdev_spacemaps(spa_t checkpoint, spa_t current)
	{
	vdev_t *ckpoint_rvd = checkpoint->spa_root_vdev;
	vdev_t *current_rvd = current->spa_root_vdev;

	load_concrete_ms_allocatable_trees(checkpoint, SM_FREE);

	for (uint64_t c = 0; c < ckpoint_rvd->vdev_children; c++) {
	vdev_t *ckpoint_vd = ckpoint_rvd->vdev_child[c];
	vdev_t *current_vd = current_rvd->vdev_child[c];

	space_map_t *checkpoint_sm = NULL;
	uint64_t checkpoint_sm_obj;

	if (ckpoint_vd->vdev_ops == &vdev_indirect_ops) {
	/*
	* Since we don't allow device removal in a pool
	* that has a checkpoint, we expect that all removed
	* vdevs were removed from the pool before the
	* checkpoint.
	*/
	ASSERT3P(current_vd->vdev_ops, ==, &vdev_indirect_ops);
	continue;
	}

	/*
	* If the checkpoint space map doesn't exist, then nothing
	* here is checkpointed so there's nothing to verify.
	*/
	if (current_vd->vdev_top_zap == 0 \|\|
	zap_contains(spa_meta_objset(current),
	current_vd->vdev_top_zap,
	VDEV_TOP_ZAP_POOL_CHECKPOINT_SM) != 0)
	continue;

	VERIFY0(zap_lookup(spa_meta_objset(current),
	current_vd->vdev_top_zap, VDEV_TOP_ZAP_POOL_CHECKPOINT_SM,
	sizeof (uint64_t), 1, &checkpoint_sm_obj));

	VERIFY0(space_map_open(&checkpoint_sm, spa_meta_objset(current),
	checkpoint_sm_obj, 0, current_vd->vdev_asize,
	current_vd->vdev_ashift));

	verify_checkpoint_sm_entry_cb_arg_t vcsec;
	vcsec.vcsec_vd = ckpoint_vd;
	vcsec.vcsec_entryid = 0;
	vcsec.vcsec_num_entries =
	space_map_length(checkpoint_sm) / sizeof (uint64_t);
	VERIFY0(space_map_iterate(checkpoint_sm,
	space_map_length(checkpoint_sm),
	verify_checkpoint_sm_entry_cb, &vcsec));
	if (dump_opt['m'] > 3)
	dump_spacemap(current->spa_meta_objset, checkpoint_sm);
	space_map_close(checkpoint_sm);
	}

	/*
	* If we've added vdevs since we took the checkpoint, ensure
	* that their checkpoint space maps are empty.
	*/
	if (ckpoint_rvd->vdev_children < current_rvd->vdev_children) {
	for (uint64_t c = ckpoint_rvd->vdev_children;
	c < current_rvd->vdev_children; c++) {
	vdev_t *current_vd = current_rvd->vdev_child[c];
	ASSERT3P(current_vd->vdev_checkpoint_sm, ==, NULL);
	}
	}

	/* for cleaner progress output */
	(void) fprintf(stderr, "\n");
	}

	/*
	* Verifies that all space that's allocated in the checkpoint is
	* still allocated in the current version, by checking that everything
	* in checkpoint's ms_allocatable (which is actually allocated, not
	* allocatable/free) is not present in current's ms_allocatable.
	*
	* Note that the function changes the state of the ms_allocatable
	* trees of both spas when called. The entries of all ms_allocatable
	* trees are cleared out and then repopulated from their respective
	* ms_sm space maps. In the checkpointed state we load the allocated
	* entries, and in the current state we load the free entries.
	*/
	static void
	verify_checkpoint_ms_spacemaps(spa_t checkpoint, spa_t current)
	{
	vdev_t *ckpoint_rvd = checkpoint->spa_root_vdev;
	vdev_t *current_rvd = current->spa_root_vdev;

	load_concrete_ms_allocatable_trees(checkpoint, SM_ALLOC);
	load_concrete_ms_allocatable_trees(current, SM_FREE);

	for (uint64_t i = 0; i < ckpoint_rvd->vdev_children; i++) {
	vdev_t *ckpoint_vd = ckpoint_rvd->vdev_child[i];
	vdev_t *current_vd = current_rvd->vdev_child[i];

	if (ckpoint_vd->vdev_ops == &vdev_indirect_ops) {
	/*
	* See comment in verify_checkpoint_vdev_spacemaps()
	*/
	ASSERT3P(current_vd->vdev_ops, ==, &vdev_indirect_ops);
	continue;
	}

	for (uint64_t m = 0; m < ckpoint_vd->vdev_ms_count; m++) {
	metaslab_t *ckpoint_msp = ckpoint_vd->vdev_ms[m];
	metaslab_t *current_msp = current_vd->vdev_ms[m];

	(void) fprintf(stderr,
	"\rverifying vdev %llu of %llu, "
	"metaslab %llu of %llu ...",
	(longlong_t)current_vd->vdev_id,
	(longlong_t)current_rvd->vdev_children,
	(longlong_t)current_vd->vdev_ms[m]->ms_id,
	(longlong_t)current_vd->vdev_ms_count);

	/*
	* We walk through the ms_allocatable trees that
	* are loaded with the allocated blocks from the
	* ms_sm spacemaps of the checkpoint. For each
	* one of these ranges we ensure that none of them
	* exists in the ms_allocatable trees of the
	* current state which are loaded with the ranges
	* that are currently free.
	*
	* This way we ensure that none of the blocks that
	* are part of the checkpoint were freed by mistake.
	*/
	range_tree_walk(ckpoint_msp->ms_allocatable,
	(range_tree_func_t *)range_tree_verify_not_present,
	current_msp->ms_allocatable);
	}
	}

	/* for cleaner progress output */
	(void) fprintf(stderr, "\n");
	}

	static void
	verify_checkpoint_blocks(spa_t *spa)
	{
	ASSERT(!dump_opt['L']);

	spa_t *checkpoint_spa;
	char *checkpoint_pool;
	nvlist_t *config = NULL;
	int error = 0;

	/*
	* We import the checkpointed state of the pool (under a different
	* name) so we can do verification on it against the current state
	* of the pool.
	*/
	checkpoint_pool = import_checkpointed_state(spa->spa_name, config,
	NULL);
	ASSERT(strcmp(spa->spa_name, checkpoint_pool) != 0);

	error = spa_open(checkpoint_pool, &checkpoint_spa, FTAG);
	if (error != 0) {
	fatal("Tried to open pool \"%s\" but spa_open() failed with "
	"error %d\n", checkpoint_pool, error);
	}

	/*
	* Ensure that ranges in the checkpoint space maps of each vdev
	* are allocated according to the checkpointed state's metaslab
	* space maps.
	*/
	verify_checkpoint_vdev_spacemaps(checkpoint_spa, spa);

	/*
	* Ensure that allocated ranges in the checkpoint's metaslab
	* space maps remain allocated in the metaslab space maps of
	* the current state.
	*/
	verify_checkpoint_ms_spacemaps(checkpoint_spa, spa);

	/*
	* Once we are done, we get rid of the checkpointed state.
	*/
	spa_close(checkpoint_spa, FTAG);
	free(checkpoint_pool);
	}

	static void
	dump_leftover_checkpoint_blocks(spa_t *spa)
	{
	vdev_t *rvd = spa->spa_root_vdev;

	for (uint64_t i = 0; i < rvd->vdev_children; i++) {
	vdev_t *vd = rvd->vdev_child[i];

	space_map_t *checkpoint_sm = NULL;
	uint64_t checkpoint_sm_obj;

	if (vd->vdev_top_zap == 0)
	continue;

	if (zap_contains(spa_meta_objset(spa), vd->vdev_top_zap,
	VDEV_TOP_ZAP_POOL_CHECKPOINT_SM) != 0)
	continue;

	VERIFY0(zap_lookup(spa_meta_objset(spa), vd->vdev_top_zap,
	VDEV_TOP_ZAP_POOL_CHECKPOINT_SM,
	sizeof (uint64_t), 1, &checkpoint_sm_obj));

	VERIFY0(space_map_open(&checkpoint_sm, spa_meta_objset(spa),
	checkpoint_sm_obj, 0, vd->vdev_asize, vd->vdev_ashift));
	dump_spacemap(spa->spa_meta_objset, checkpoint_sm);
	space_map_close(checkpoint_sm);
	}
	}

	static int
	verify_checkpoint(spa_t *spa)
	{
	uberblock_t checkpoint;
	int error;

	if (!spa_feature_is_active(spa, SPA_FEATURE_POOL_CHECKPOINT))
	return (0);

	error = zap_lookup(spa->spa_meta_objset, DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_ZPOOL_CHECKPOINT, sizeof (uint64_t),
	sizeof (uberblock_t) / sizeof (uint64_t), &checkpoint);

	if (error == ENOENT && !dump_opt['L']) {
	/*
	* If the feature is active but the uberblock is missing
	* then we must be in the middle of discarding the
	* checkpoint.
	*/
	(void) printf("\nPartially discarded checkpoint "
	"state found:\n");
	if (dump_opt['m'] > 3)
	dump_leftover_checkpoint_blocks(spa);
	return (0);
	} else if (error != 0) {
	(void) printf("lookup error %d when looking for "
	"checkpointed uberblock in MOS\n", error);
	return (error);
	}
	dump_uberblock(&checkpoint, "\nCheckpointed uberblock found:\n", "\n");

	if (checkpoint.ub_checkpoint_txg == 0) {
	(void) printf("\nub_checkpoint_txg not set in checkpointed "
	"uberblock\n");
	error = 3;
	}

	if (error == 0 && !dump_opt['L'])
	verify_checkpoint_blocks(spa);

	return (error);
	}

	/* ARGSUSED */
	static void
	mos_leaks_cb(void *arg, uint64_t start, uint64_t size)
	{
	for (uint64_t i = start; i < size; i++) {
	(void) printf("MOS object %llu referenced but not allocated\n",
	(u_longlong_t)i);
	}
	}

	static void
	mos_obj_refd(uint64_t obj)
	{
	if (obj != 0 && mos_refd_objs != NULL)
	range_tree_add(mos_refd_objs, obj, 1);
	}

	/*
	* Call on a MOS object that may already have been referenced.
	*/
	static void
	mos_obj_refd_multiple(uint64_t obj)
	{
	if (obj != 0 && mos_refd_objs != NULL &&
	!range_tree_contains(mos_refd_objs, obj, 1))
	range_tree_add(mos_refd_objs, obj, 1);
	}

	static void
	mos_leak_vdev_top_zap(vdev_t *vd)
	{
	uint64_t ms_flush_data_obj;
	int error = zap_lookup(spa_meta_objset(vd->vdev_spa),
	vd->vdev_top_zap, VDEV_TOP_ZAP_MS_UNFLUSHED_PHYS_TXGS,
	sizeof (ms_flush_data_obj), 1, &ms_flush_data_obj);
	if (error == ENOENT)
	return;
	ASSERT0(error);

	mos_obj_refd(ms_flush_data_obj);
	}

	static void
	mos_leak_vdev(vdev_t *vd)
	{
	mos_obj_refd(vd->vdev_dtl_object);
	mos_obj_refd(vd->vdev_ms_array);
	mos_obj_refd(vd->vdev_indirect_config.vic_births_object);
	mos_obj_refd(vd->vdev_indirect_config.vic_mapping_object);
	mos_obj_refd(vd->vdev_leaf_zap);
	if (vd->vdev_checkpoint_sm != NULL)
	mos_obj_refd(vd->vdev_checkpoint_sm->sm_object);
	if (vd->vdev_indirect_mapping != NULL) {
	mos_obj_refd(vd->vdev_indirect_mapping->
	vim_phys->vimp_counts_object);
	}
	if (vd->vdev_obsolete_sm != NULL)
	mos_obj_refd(vd->vdev_obsolete_sm->sm_object);

	for (uint64_t m = 0; m < vd->vdev_ms_count; m++) {
	metaslab_t *ms = vd->vdev_ms[m];
	mos_obj_refd(space_map_object(ms->ms_sm));
	}

	if (vd->vdev_top_zap != 0) {
	mos_obj_refd(vd->vdev_top_zap);
	mos_leak_vdev_top_zap(vd);
	}

	for (uint64_t c = 0; c < vd->vdev_children; c++) {
	mos_leak_vdev(vd->vdev_child[c]);
	}
	}

	static void
	mos_leak_log_spacemaps(spa_t *spa)
	{
	uint64_t spacemap_zap;
	int error = zap_lookup(spa_meta_objset(spa),
	DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_LOG_SPACEMAP_ZAP,
	sizeof (spacemap_zap), 1, &spacemap_zap);
	if (error == ENOENT)
	return;
	ASSERT0(error);

	mos_obj_refd(spacemap_zap);
	for (spa_log_sm_t *sls = avl_first(&spa->spa_sm_logs_by_txg);
	sls; sls = AVL_NEXT(&spa->spa_sm_logs_by_txg, sls))
	mos_obj_refd(sls->sls_sm_obj);
	}

	static int
	dump_mos_leaks(spa_t *spa)
	{
	int rv = 0;
	objset_t *mos = spa->spa_meta_objset;
	dsl_pool_t *dp = spa->spa_dsl_pool;

	/* Visit and mark all referenced objects in the MOS */

	mos_obj_refd(DMU_POOL_DIRECTORY_OBJECT);
	mos_obj_refd(spa->spa_pool_props_object);
	mos_obj_refd(spa->spa_config_object);
	mos_obj_refd(spa->spa_ddt_stat_object);
	mos_obj_refd(spa->spa_feat_desc_obj);
	mos_obj_refd(spa->spa_feat_enabled_txg_obj);
	mos_obj_refd(spa->spa_feat_for_read_obj);
	mos_obj_refd(spa->spa_feat_for_write_obj);
	mos_obj_refd(spa->spa_history);
	mos_obj_refd(spa->spa_errlog_last);
	mos_obj_refd(spa->spa_errlog_scrub);
	mos_obj_refd(spa->spa_all_vdev_zaps);
	mos_obj_refd(spa->spa_dsl_pool->dp_bptree_obj);
	mos_obj_refd(spa->spa_dsl_pool->dp_tmp_userrefs_obj);
	mos_obj_refd(spa->spa_dsl_pool->dp_scan->scn_phys.scn_queue_obj);
	bpobj_count_refd(&spa->spa_deferred_bpobj);
	mos_obj_refd(dp->dp_empty_bpobj);
	bpobj_count_refd(&dp->dp_obsolete_bpobj);
	bpobj_count_refd(&dp->dp_free_bpobj);
	mos_obj_refd(spa->spa_l2cache.sav_object);
	mos_obj_refd(spa->spa_spares.sav_object);

	if (spa->spa_syncing_log_sm != NULL)
	mos_obj_refd(spa->spa_syncing_log_sm->sm_object);
	mos_leak_log_spacemaps(spa);

	mos_obj_refd(spa->spa_condensing_indirect_phys.
	scip_next_mapping_object);
	mos_obj_refd(spa->spa_condensing_indirect_phys.
	scip_prev_obsolete_sm_object);
	if (spa->spa_condensing_indirect_phys.scip_next_mapping_object != 0) {
	vdev_indirect_mapping_t *vim =
	vdev_indirect_mapping_open(mos,
	spa->spa_condensing_indirect_phys.scip_next_mapping_object);
	mos_obj_refd(vim->vim_phys->vimp_counts_object);
	vdev_indirect_mapping_close(vim);
	}
	deleted_livelists_dump_mos(spa);

	if (dp->dp_origin_snap != NULL) {
	dsl_dataset_t *ds;

	dsl_pool_config_enter(dp, FTAG);
	VERIFY0(dsl_dataset_hold_obj(dp,
	dsl_dataset_phys(dp->dp_origin_snap)->ds_next_snap_obj,
	FTAG, &ds));
	count_ds_mos_objects(ds);
	dump_blkptr_list(&ds->ds_deadlist, "Deadlist");
	dsl_dataset_rele(ds, FTAG);
	dsl_pool_config_exit(dp, FTAG);

	count_ds_mos_objects(dp->dp_origin_snap);
	dump_blkptr_list(&dp->dp_origin_snap->ds_deadlist, "Deadlist");
	}
	count_dir_mos_objects(dp->dp_mos_dir);
	if (dp->dp_free_dir != NULL)
	count_dir_mos_objects(dp->dp_free_dir);
	if (dp->dp_leak_dir != NULL)
	count_dir_mos_objects(dp->dp_leak_dir);

	mos_leak_vdev(spa->spa_root_vdev);

	for (uint64_t class = 0; class < DDT_CLASSES; class++) {
	for (uint64_t type = 0; type < DDT_TYPES; type++) {
	for (uint64_t cksum = 0;
	cksum < ZIO_CHECKSUM_FUNCTIONS; cksum++) {
	ddt_t *ddt = spa->spa_ddt[cksum];
	mos_obj_refd(ddt->ddt_object[type][class]);
	}
	}
	}

	/*
	* Visit all allocated objects and make sure they are referenced.
	*/
	uint64_t object = 0;
	while (dmu_object_next(mos, &object, B_FALSE, 0) == 0) {
	if (range_tree_contains(mos_refd_objs, object, 1)) {
	range_tree_remove(mos_refd_objs, object, 1);
	} else {
	dmu_object_info_t doi;
	const char *name;
	dmu_object_info(mos, object, &doi);
	if (doi.doi_type & DMU_OT_NEWTYPE) {
	dmu_object_byteswap_t bswap =
	DMU_OT_BYTESWAP(doi.doi_type);
	name = dmu_ot_byteswap[bswap].ob_name;
	} else {
	name = dmu_ot[doi.doi_type].ot_name;
	}

	(void) printf("MOS object %llu (%s) leaked\n",
	(u_longlong_t)object, name);
	rv = 2;
	}
	}
	(void) range_tree_walk(mos_refd_objs, mos_leaks_cb, NULL);
	if (!range_tree_is_empty(mos_refd_objs))
	rv = 2;
	range_tree_vacate(mos_refd_objs, NULL, NULL);
	range_tree_destroy(mos_refd_objs);
	return (rv);
	}

	typedef struct log_sm_obsolete_stats_arg {
	uint64_t lsos_current_txg;

	uint64_t lsos_total_entries;
	uint64_t lsos_valid_entries;

	uint64_t lsos_sm_entries;
	uint64_t lsos_valid_sm_entries;
	} log_sm_obsolete_stats_arg_t;

	static int
	log_spacemap_obsolete_stats_cb(spa_t spa, space_map_entry_t sme,
	uint64_t txg, void *arg)
	{
	log_sm_obsolete_stats_arg_t *lsos = arg;

	uint64_t offset = sme->sme_offset;
	uint64_t vdev_id = sme->sme_vdev;

	if (lsos->lsos_current_txg == 0) {
	/* this is the first log */
	lsos->lsos_current_txg = txg;
	} else if (lsos->lsos_current_txg < txg) {
	/* we just changed log - print stats and reset */
	(void) printf("%-8llu valid entries out of %-8llu - txg %llu\n",
	(u_longlong_t)lsos->lsos_valid_sm_entries,
	(u_longlong_t)lsos->lsos_sm_entries,
	(u_longlong_t)lsos->lsos_current_txg);
	lsos->lsos_valid_sm_entries = 0;
	lsos->lsos_sm_entries = 0;
	lsos->lsos_current_txg = txg;
	}
	ASSERT3U(lsos->lsos_current_txg, ==, txg);

	lsos->lsos_sm_entries++;
	lsos->lsos_total_entries++;

	vdev_t *vd = vdev_lookup_top(spa, vdev_id);
	if (!vdev_is_concrete(vd))
	return (0);

	metaslab_t *ms = vd->vdev_ms[offset >> vd->vdev_ms_shift];
	ASSERT(sme->sme_type == SM_ALLOC \|\| sme->sme_type == SM_FREE);

	if (txg < metaslab_unflushed_txg(ms))
	return (0);
	lsos->lsos_valid_sm_entries++;
	lsos->lsos_valid_entries++;
	return (0);
	}

	static void
	dump_log_spacemap_obsolete_stats(spa_t *spa)
	{
	if (!spa_feature_is_active(spa, SPA_FEATURE_LOG_SPACEMAP))
	return;

	log_sm_obsolete_stats_arg_t lsos;
	bzero(&lsos, sizeof (lsos));

	(void) printf("Log Space Map Obsolete Entry Statistics:\n");

	iterate_through_spacemap_logs(spa,
	log_spacemap_obsolete_stats_cb, &lsos);

	/* print stats for latest log */
	(void) printf("%-8llu valid entries out of %-8llu - txg %llu\n",
	(u_longlong_t)lsos.lsos_valid_sm_entries,
	(u_longlong_t)lsos.lsos_sm_entries,
	(u_longlong_t)lsos.lsos_current_txg);

	(void) printf("%-8llu valid entries out of %-8llu - total\n\n",
	(u_longlong_t)lsos.lsos_valid_entries,
	(u_longlong_t)lsos.lsos_total_entries);
	}

	static void
	dump_zpool(spa_t *spa)
	{
	dsl_pool_t *dp = spa_get_dsl(spa);
	int rc = 0;

	if (dump_opt['y']) {
	livelist_metaslab_validate(spa);
	}

	if (dump_opt['S']) {
	dump_simulated_ddt(spa);
	return;
	}

	if (!dump_opt['e'] && dump_opt['C'] > 1) {
	(void) printf("\nCached configuration:\n");
	dump_nvlist(spa->spa_config, 8);
	}

	if (dump_opt['C'])
	dump_config(spa);

	if (dump_opt['u'])
	dump_uberblock(&spa->spa_uberblock, "\nUberblock:\n", "\n");

	if (dump_opt['D'])
	dump_all_ddts(spa);

	if (dump_opt['d'] > 2 \|\| dump_opt['m'])
	dump_metaslabs(spa);
	if (dump_opt['M'])
	dump_metaslab_groups(spa);
	if (dump_opt['d'] > 2 \|\| dump_opt['m']) {
	dump_log_spacemaps(spa);
	dump_log_spacemap_obsolete_stats(spa);
	}

	if (dump_opt['d'] \|\| dump_opt['i']) {
	spa_feature_t f;
	mos_refd_objs = range_tree_create(NULL, RANGE_SEG64, NULL, 0,
	0);
	dump_objset(dp->dp_meta_objset);

	if (dump_opt['d'] >= 3) {
	dsl_pool_t *dp = spa->spa_dsl_pool;
	dump_full_bpobj(&spa->spa_deferred_bpobj,
	"Deferred frees", 0);
	if (spa_version(spa) >= SPA_VERSION_DEADLISTS) {
	dump_full_bpobj(&dp->dp_free_bpobj,
	"Pool snapshot frees", 0);
	}
	if (bpobj_is_open(&dp->dp_obsolete_bpobj)) {
	ASSERT(spa_feature_is_enabled(spa,
	SPA_FEATURE_DEVICE_REMOVAL));
	dump_full_bpobj(&dp->dp_obsolete_bpobj,
	"Pool obsolete blocks", 0);
	}

	if (spa_feature_is_active(spa,
	SPA_FEATURE_ASYNC_DESTROY)) {
	dump_bptree(spa->spa_meta_objset,
	dp->dp_bptree_obj,
	"Pool dataset frees");
	}
	dump_dtl(spa->spa_root_vdev, 0);
	}

	for (spa_feature_t f = 0; f < SPA_FEATURES; f++)
	global_feature_count[f] = UINT64_MAX;
	global_feature_count[SPA_FEATURE_REDACTION_BOOKMARKS] = 0;
	global_feature_count[SPA_FEATURE_BOOKMARK_WRITTEN] = 0;
	global_feature_count[SPA_FEATURE_LIVELIST] = 0;

	(void) dmu_objset_find(spa_name(spa), dump_one_objset,
	NULL, DS_FIND_SNAPSHOTS \| DS_FIND_CHILDREN);

	if (rc == 0 && !dump_opt['L'])
	rc = dump_mos_leaks(spa);

	for (f = 0; f < SPA_FEATURES; f++) {
	uint64_t refcount;

	uint64_t *arr;
	if (!(spa_feature_table[f].fi_flags &
	ZFEATURE_FLAG_PER_DATASET)) {
	if (global_feature_count[f] == UINT64_MAX)
	continue;
	if (!spa_feature_is_enabled(spa, f)) {
	ASSERT0(global_feature_count[f]);
	continue;
	}
	arr = global_feature_count;
	} else {
	if (!spa_feature_is_enabled(spa, f)) {
	ASSERT0(dataset_feature_count[f]);
	continue;
	}
	arr = dataset_feature_count;
	}
	if (feature_get_refcount(spa, &spa_feature_table[f],
	&refcount) == ENOTSUP)
	continue;
	if (arr[f] != refcount) {
	(void) printf("%s feature refcount mismatch: "
	"%lld consumers != %lld refcount\n",
	spa_feature_table[f].fi_uname,
	(longlong_t)arr[f], (longlong_t)refcount);
	rc = 2;
	} else {
	(void) printf("Verified %s feature refcount "
	"of %llu is correct\n",
	spa_feature_table[f].fi_uname,
	(longlong_t)refcount);
	}
	}

	if (rc == 0)
	rc = verify_device_removal_feature_counts(spa);
	}

	if (rc == 0 && (dump_opt['b'] \|\| dump_opt['c']))
	rc = dump_block_stats(spa);

	if (rc == 0)
	rc = verify_spacemap_refcounts(spa);

	if (dump_opt['s'])
	show_pool_stats(spa);

	if (dump_opt['h'])
	dump_history(spa);

	if (rc == 0)
	rc = verify_checkpoint(spa);

	if (rc != 0) {
	dump_debug_buffer();
	exit(rc);
	}
	}

	#define ZDB_FLAG_CHECKSUM 0x0001
	#define ZDB_FLAG_DECOMPRESS 0x0002
	#define ZDB_FLAG_BSWAP 0x0004
	#define ZDB_FLAG_GBH 0x0008
	#define ZDB_FLAG_INDIRECT 0x0010
	#define ZDB_FLAG_RAW 0x0020
	#define ZDB_FLAG_PRINT_BLKPTR 0x0040
	#define ZDB_FLAG_VERBOSE 0x0080

	static int flagbits[256];
	static char flagbitstr[16];

	static void
	zdb_print_blkptr(const blkptr_t *bp, int flags)
	{
	char blkbuf[BP_SPRINTF_LEN];

	if (flags & ZDB_FLAG_BSWAP)
	byteswap_uint64_array((void *)bp, sizeof (blkptr_t));

	snprintf_blkptr(blkbuf, sizeof (blkbuf), bp);
	(void) printf("%s\n", blkbuf);
	}

	static void
	zdb_dump_indirect(blkptr_t *bp, int nbps, int flags)
	{
	int i;

	for (i = 0; i < nbps; i++)
	zdb_print_blkptr(&bp[i], flags);
	}

	static void
	zdb_dump_gbh(void *buf, int flags)
	{
	zdb_dump_indirect((blkptr_t *)buf, SPA_GBH_NBLKPTRS, flags);
	}

	static void
	zdb_dump_block_raw(void *buf, uint64_t size, int flags)
	{
	if (flags & ZDB_FLAG_BSWAP)
	byteswap_uint64_array(buf, size);
	VERIFY(write(fileno(stdout), buf, size) == size);
	}

	static void
	zdb_dump_block(char label, void buf, uint64_t size, int flags)
	{
	uint64_t d = (uint64_t )buf;
	unsigned nwords = size / sizeof (uint64_t);
	int do_bswap = !!(flags & ZDB_FLAG_BSWAP);
	unsigned i, j;
	const char *hdr;
	char *c;


	if (do_bswap)
	hdr = " 7 6 5 4 3 2 1 0 f e d c b a 9 8";
	else
	hdr = " 0 1 2 3 4 5 6 7 8 9 a b c d e f";

	(void) printf("\n%s\n%6s %s 0123456789abcdef\n", label, "", hdr);

	#ifdef _LITTLE_ENDIAN
	/* correct the endianness */
	do_bswap = !do_bswap;
	#endif
	for (i = 0; i < nwords; i += 2) {
	(void) printf("%06llx: %016llx %016llx ",
	(u_longlong_t)(i * sizeof (uint64_t)),
	(u_longlong_t)(do_bswap ? BSWAP_64(d[i]) : d[i]),
	(u_longlong_t)(do_bswap ? BSWAP_64(d[i + 1]) : d[i + 1]));

	c = (char *)&d[i];
	for (j = 0; j < 2 * sizeof (uint64_t); j++)
	(void) printf("%c", isprint(c[j]) ? c[j] : '.');
	(void) printf("\n");
	}
	}

	/*
	* There are two acceptable formats:
	* leaf_name - For example: c1t0d0 or /tmp/ztest.0a
	* child[.child]* - For example: 0.1.1
	*
	* The second form can be used to specify arbitrary vdevs anywhere
	* in the hierarchy. For example, in a pool with a mirror of
	* RAID-Zs, you can specify either RAID-Z vdev with 0.0 or 0.1 .
	*/
	static vdev_t *
	zdb_vdev_lookup(vdev_t vdev, const char path)
	{
	char s, p, *q;
	unsigned i;

	if (vdev == NULL)
	return (NULL);

	/* First, assume the x.x.x.x format */
	i = strtoul(path, &s, 10);
	if (s == path \|\| (s && s != '.' && s != '\0'))
	goto name;
	if (i >= vdev->vdev_children)
	return (NULL);

	vdev = vdev->vdev_child[i];
	if (s && *s == '\0')
	return (vdev);
	return (zdb_vdev_lookup(vdev, s+1));

	name:
	for (i = 0; i < vdev->vdev_children; i++) {
	vdev_t *vc = vdev->vdev_child[i];

	if (vc->vdev_path == NULL) {
	vc = zdb_vdev_lookup(vc, path);
	if (vc == NULL)
	continue;
	else
	return (vc);
	}

	p = strrchr(vc->vdev_path, '/');
	p = p ? p + 1 : vc->vdev_path;
	q = &vc->vdev_path[strlen(vc->vdev_path) - 2];

	if (strcmp(vc->vdev_path, path) == 0)
	return (vc);
	if (strcmp(p, path) == 0)
	return (vc);
	if (strcmp(q, "s0") == 0 && strncmp(p, path, q - p) == 0)
	return (vc);
	}

	return (NULL);
	}

	static int
	name_from_objset_id(spa_t spa, uint64_t objset_id, char outstr)
	{
	dsl_dataset_t *ds;

	dsl_pool_config_enter(spa->spa_dsl_pool, FTAG);
	int error = dsl_dataset_hold_obj(spa->spa_dsl_pool, objset_id,
	NULL, &ds);
	if (error != 0) {
	(void) fprintf(stderr, "failed to hold objset %llu: %s\n",
	(u_longlong_t)objset_id, strerror(error));
	dsl_pool_config_exit(spa->spa_dsl_pool, FTAG);
	return (error);
	}
	dsl_dataset_name(ds, outstr);
	dsl_dataset_rele(ds, NULL);
	dsl_pool_config_exit(spa->spa_dsl_pool, FTAG);
	return (0);
	}

	static boolean_t
	zdb_parse_block_sizes(char sizes, uint64_t lsize, uint64_t *psize)
	{
	char s0, s1;

	if (sizes == NULL)
	return (B_FALSE);

	s0 = strtok(sizes, "/");
	if (s0 == NULL)
	return (B_FALSE);
	s1 = strtok(NULL, "/");
	*lsize = strtoull(s0, NULL, 16);
	psize = s1 ? strtoull(s1, NULL, 16) : lsize;
	return (lsize >= psize && *psize > 0);
	}

	#define ZIO_COMPRESS_MASK(alg) (1ULL << (ZIO_COMPRESS_##alg))

	static boolean_t
	zdb_decompress_block(abd_t pabd, void buf, void *lbuf, uint64_t lsize,
	uint64_t psize, int flags)
	{
	boolean_t exceeded = B_FALSE;
	/*
	* We don't know how the data was compressed, so just try
	* every decompress function at every inflated blocksize.
	*/
	void *lbuf2 = umem_alloc(SPA_MAXBLOCKSIZE, UMEM_NOFAIL);
	int cfuncs[ZIO_COMPRESS_FUNCTIONS] = { 0 };
	int *cfuncp = cfuncs;
	uint64_t maxlsize = SPA_MAXBLOCKSIZE;
	uint64_t mask = ZIO_COMPRESS_MASK(ON) \| ZIO_COMPRESS_MASK(OFF) \|
	ZIO_COMPRESS_MASK(INHERIT) \| ZIO_COMPRESS_MASK(EMPTY) \|
	(getenv("ZDB_NO_ZLE") ? ZIO_COMPRESS_MASK(ZLE) : 0);
	*cfuncp++ = ZIO_COMPRESS_LZ4;
	*cfuncp++ = ZIO_COMPRESS_LZJB;
	mask \|= ZIO_COMPRESS_MASK(LZ4) \| ZIO_COMPRESS_MASK(LZJB);
	for (int c = 0; c < ZIO_COMPRESS_FUNCTIONS; c++)
	if (((1ULL << c) & mask) == 0)
	*cfuncp++ = c;

	/*
	* On the one hand, with SPA_MAXBLOCKSIZE at 16MB, this
	* could take a while and we should let the user know
	* we are not stuck. On the other hand, printing progress
	* info gets old after a while. User can specify 'v' flag
	* to see the progression.
	*/
	if (lsize == psize)
	lsize += SPA_MINBLOCKSIZE;
	else
	maxlsize = lsize;
	for (; lsize <= maxlsize; lsize += SPA_MINBLOCKSIZE) {
	for (cfuncp = cfuncs; *cfuncp; cfuncp++) {
	if (flags & ZDB_FLAG_VERBOSE) {
	(void) fprintf(stderr,
	"Trying %05llx -> %05llx (%s)\n",
	(u_longlong_t)psize,
	(u_longlong_t)lsize,
	zio_compress_table[*cfuncp].\
	ci_name);
	}

	/*
	* We randomize lbuf2, and decompress to both
	* lbuf and lbuf2. This way, we will know if
	* decompression fill exactly to lsize.
	*/
	VERIFY0(random_get_pseudo_bytes(lbuf2, lsize));

	if (zio_decompress_data(*cfuncp, pabd,
	lbuf, psize, lsize, NULL) == 0 &&
	zio_decompress_data(*cfuncp, pabd,
	lbuf2, psize, lsize, NULL) == 0 &&
	bcmp(lbuf, lbuf2, lsize) == 0)
	break;
	}
	if (*cfuncp != 0)
	break;
	}
	umem_free(lbuf2, SPA_MAXBLOCKSIZE);

	if (lsize > maxlsize) {
	exceeded = B_TRUE;
	}
	buf = lbuf;
	if (*cfuncp == ZIO_COMPRESS_ZLE) {
	printf("\nZLE decompression was selected. If you "
	"suspect the results are wrong,\ntry avoiding ZLE "
	"by setting and exporting ZDB_NO_ZLE=\"true\"\n");
	}

	return (exceeded);
	}

	/*
	* Read a block from a pool and print it out. The syntax of the
	* block descriptor is:
	*
	* pool:vdev_specifier:offset:[lsize/]psize[:flags]
	*
	* pool - The name of the pool you wish to read from
	* vdev_specifier - Which vdev (see comment for zdb_vdev_lookup)
	* offset - offset, in hex, in bytes
	* size - Amount of data to read, in hex, in bytes
	* flags - A string of characters specifying options
	* b: Decode a blkptr at given offset within block
	* c: Calculate and display checksums
	* d: Decompress data before dumping
	* e: Byteswap data before dumping
	* g: Display data as a gang block header
	* i: Display as an indirect block
	* r: Dump raw data to stdout
	* v: Verbose
	*
	*/
	static void
	zdb_read_block(char thing, spa_t spa)
	{
	blkptr_t blk, *bp = &blk;
	dva_t *dva = bp->blk_dva;
	int flags = 0;
	uint64_t offset = 0, psize = 0, lsize = 0, blkptr_offset = 0;
	zio_t *zio;
	vdev_t *vd;
	abd_t *pabd;
	void lbuf, buf;
	char s, p, dup, vdev, flagstr, sizes;
	int i, error;
	boolean_t borrowed = B_FALSE, found = B_FALSE;

	dup = strdup(thing);
	s = strtok(dup, ":");
	vdev = s ? s : "";
	s = strtok(NULL, ":");
	offset = strtoull(s ? s : "", NULL, 16);
	sizes = strtok(NULL, ":");
	s = strtok(NULL, ":");
	flagstr = strdup(s ? s : "");

	s = NULL;
	if (!zdb_parse_block_sizes(sizes, &lsize, &psize))
	s = "invalid size(s)";
	if (!IS_P2ALIGNED(psize, DEV_BSIZE) \|\| !IS_P2ALIGNED(lsize, DEV_BSIZE))
	s = "size must be a multiple of sector size";
	if (!IS_P2ALIGNED(offset, DEV_BSIZE))
	s = "offset must be a multiple of sector size";
	if (s) {
	(void) printf("Invalid block specifier: %s - %s\n", thing, s);
	goto done;
	}

	for (s = strtok(flagstr, ":"); s; s = strtok(NULL, ":")) {
	for (i = 0; i < strlen(flagstr); i++) {
	int bit = flagbits[(uchar_t)flagstr[i]];

	if (bit == 0) {
	(void) printf("***Ignoring flag: %c\n",
	(uchar_t)flagstr[i]);
	continue;
	}
	found = B_TRUE;
	flags \|= bit;

	p = &flagstr[i + 1];
	if (p != ':' && p != '\0') {
	int j = 0, nextbit = flagbits[(uchar_t)*p];
	char *end, offstr[8] = { 0 };
	if ((bit == ZDB_FLAG_PRINT_BLKPTR) &&
	(nextbit == 0)) {
	/* look ahead to isolate the offset */
	while (nextbit == 0 &&
	strchr(flagbitstr, *p) == NULL) {
	offstr[j] = *p;
	j++;
	if (i + j > strlen(flagstr))
	break;
	p++;
	nextbit = flagbits[(uchar_t)*p];
	}
	blkptr_offset = strtoull(offstr, &end,
	16);
	i += j;
	} else if (nextbit == 0) {
	(void) printf("***Ignoring flag arg:"
	" '%c'\n", (uchar_t)*p);
	}
	}
	}
	}
	if (blkptr_offset % sizeof (blkptr_t)) {
	printf("Block pointer offset 0x%llx "
	"must be divisible by 0x%x\n",
	(longlong_t)blkptr_offset, (int)sizeof (blkptr_t));
	goto done;
	}
	if (found == B_FALSE && strlen(flagstr) > 0) {
	printf("Invalid flag arg: '%s'\n", flagstr);
	goto done;
	}

	vd = zdb_vdev_lookup(spa->spa_root_vdev, vdev);
	if (vd == NULL) {
	(void) printf("***Invalid vdev: %s\n", vdev);
	free(dup);
	return;
	} else {
	if (vd->vdev_path)
	(void) fprintf(stderr, "Found vdev: %s\n",
	vd->vdev_path);
	else
	(void) fprintf(stderr, "Found vdev type: %s\n",
	vd->vdev_ops->vdev_op_type);
	}

	pabd = abd_alloc_for_io(SPA_MAXBLOCKSIZE, B_FALSE);
	lbuf = umem_alloc(SPA_MAXBLOCKSIZE, UMEM_NOFAIL);

	BP_ZERO(bp);

	DVA_SET_VDEV(&dva[0], vd->vdev_id);
	DVA_SET_OFFSET(&dva[0], offset);
	DVA_SET_GANG(&dva[0], !!(flags & ZDB_FLAG_GBH));
	DVA_SET_ASIZE(&dva[0], vdev_psize_to_asize(vd, psize));

	BP_SET_BIRTH(bp, TXG_INITIAL, TXG_INITIAL);

	BP_SET_LSIZE(bp, lsize);
	BP_SET_PSIZE(bp, psize);
	BP_SET_COMPRESS(bp, ZIO_COMPRESS_OFF);
	BP_SET_CHECKSUM(bp, ZIO_CHECKSUM_OFF);
	BP_SET_TYPE(bp, DMU_OT_NONE);
	BP_SET_LEVEL(bp, 0);
	BP_SET_DEDUP(bp, 0);
	BP_SET_BYTEORDER(bp, ZFS_HOST_BYTEORDER);

	spa_config_enter(spa, SCL_STATE, FTAG, RW_READER);
	zio = zio_root(spa, NULL, NULL, 0);

	if (vd == vd->vdev_top) {
	/*
	* Treat this as a normal block read.
	*/
	zio_nowait(zio_read(zio, spa, bp, pabd, psize, NULL, NULL,
	ZIO_PRIORITY_SYNC_READ,
	ZIO_FLAG_CANFAIL \| ZIO_FLAG_RAW, NULL));
	} else {
	/*
	* Treat this as a vdev child I/O.
	*/
	zio_nowait(zio_vdev_child_io(zio, bp, vd, offset, pabd,
	psize, ZIO_TYPE_READ, ZIO_PRIORITY_SYNC_READ,
	ZIO_FLAG_DONT_CACHE \| ZIO_FLAG_DONT_PROPAGATE \|
	ZIO_FLAG_DONT_RETRY \| ZIO_FLAG_CANFAIL \| ZIO_FLAG_RAW \|
	ZIO_FLAG_OPTIONAL, NULL, NULL));
	}

	error = zio_wait(zio);
	spa_config_exit(spa, SCL_STATE, FTAG);

	if (error) {
	(void) printf("Read of %s failed, error: %d\n", thing, error);
	goto out;
	}

	uint64_t orig_lsize = lsize;
	buf = lbuf;
	if (flags & ZDB_FLAG_DECOMPRESS) {
	boolean_t failed = zdb_decompress_block(pabd, buf, lbuf,
	lsize, psize, flags);
	if (failed) {
	(void) printf("Decompress of %s failed\n", thing);
	goto out;
	}
	} else {
	buf = abd_borrow_buf_copy(pabd, lsize);
	borrowed = B_TRUE;
	}
	/*
	* Try to detect invalid block pointer. If invalid, try
	* decompressing.
	*/
	if ((flags & ZDB_FLAG_PRINT_BLKPTR \|\| flags & ZDB_FLAG_INDIRECT) &&
	!(flags & ZDB_FLAG_DECOMPRESS)) {
	const blkptr_t b = (const blkptr_t )(void *)
	((uintptr_t)buf + (uintptr_t)blkptr_offset);
	if (zfs_blkptr_verify(spa, b, B_FALSE, BLK_VERIFY_ONLY) ==
	B_FALSE) {
	abd_return_buf_copy(pabd, buf, lsize);
	borrowed = B_FALSE;
	buf = lbuf;
	boolean_t failed = zdb_decompress_block(pabd, buf,
	lbuf, lsize, psize, flags);
	b = (const blkptr_t )(void )
	((uintptr_t)buf + (uintptr_t)blkptr_offset);
	if (failed \|\| zfs_blkptr_verify(spa, b, B_FALSE,
	BLK_VERIFY_LOG) == B_FALSE) {
	printf("invalid block pointer at this DVA\n");
	goto out;
	}
	}
	}

	if (flags & ZDB_FLAG_PRINT_BLKPTR)
	zdb_print_blkptr((blkptr_t )(void )
	((uintptr_t)buf + (uintptr_t)blkptr_offset), flags);
	else if (flags & ZDB_FLAG_RAW)
	zdb_dump_block_raw(buf, lsize, flags);
	else if (flags & ZDB_FLAG_INDIRECT)
	zdb_dump_indirect((blkptr_t *)buf,
	orig_lsize / sizeof (blkptr_t), flags);
	else if (flags & ZDB_FLAG_GBH)
	zdb_dump_gbh(buf, flags);
	else
	zdb_dump_block(thing, buf, lsize, flags);

	/*
	* If :c was specified, iterate through the checksum table to
	* calculate and display each checksum for our specified
	* DVA and length.
	*/
	if ((flags & ZDB_FLAG_CHECKSUM) && !(flags & ZDB_FLAG_RAW) &&
	!(flags & ZDB_FLAG_GBH)) {
	zio_t *czio;
	(void) printf("\n");
	for (enum zio_checksum ck = ZIO_CHECKSUM_LABEL;
	ck < ZIO_CHECKSUM_FUNCTIONS; ck++) {

	if ((zio_checksum_table[ck].ci_flags &
	ZCHECKSUM_FLAG_EMBEDDED) \|\|
	ck == ZIO_CHECKSUM_NOPARITY) {
	continue;
	}
	BP_SET_CHECKSUM(bp, ck);
	spa_config_enter(spa, SCL_STATE, FTAG, RW_READER);
	czio = zio_root(spa, NULL, NULL, ZIO_FLAG_CANFAIL);
	czio->io_bp = bp;

	if (vd == vd->vdev_top) {
	zio_nowait(zio_read(czio, spa, bp, pabd, psize,
	NULL, NULL,
	ZIO_PRIORITY_SYNC_READ,
	ZIO_FLAG_CANFAIL \| ZIO_FLAG_RAW \|
	ZIO_FLAG_DONT_RETRY, NULL));
	} else {
	zio_nowait(zio_vdev_child_io(czio, bp, vd,
	offset, pabd, psize, ZIO_TYPE_READ,
	ZIO_PRIORITY_SYNC_READ,
	ZIO_FLAG_DONT_CACHE \|
	ZIO_FLAG_DONT_PROPAGATE \|
	ZIO_FLAG_DONT_RETRY \|
	ZIO_FLAG_CANFAIL \| ZIO_FLAG_RAW \|
	ZIO_FLAG_SPECULATIVE \|
	ZIO_FLAG_OPTIONAL, NULL, NULL));
	}
	error = zio_wait(czio);
	if (error == 0 \|\| error == ECKSUM) {
	zio_t *ck_zio = zio_root(spa, NULL, NULL, 0);
	ck_zio->io_offset =
	DVA_GET_OFFSET(&bp->blk_dva[0]);
	ck_zio->io_bp = bp;
	zio_checksum_compute(ck_zio, ck, pabd, lsize);
	printf("%12s\tcksum=%llx:%llx:%llx:%llx\n",
	zio_checksum_table[ck].ci_name,
	(u_longlong_t)bp->blk_cksum.zc_word[0],
	(u_longlong_t)bp->blk_cksum.zc_word[1],
	(u_longlong_t)bp->blk_cksum.zc_word[2],
	(u_longlong_t)bp->blk_cksum.zc_word[3]);
	zio_wait(ck_zio);
	} else {
	printf("error %d reading block\n", error);
	}
	spa_config_exit(spa, SCL_STATE, FTAG);
	}
	}

	if (borrowed)
	abd_return_buf_copy(pabd, buf, lsize);

	out:
	abd_free(pabd);
	umem_free(lbuf, SPA_MAXBLOCKSIZE);
	done:
	free(flagstr);
	free(dup);
	}

	static void
	zdb_embedded_block(char *thing)
	{
	blkptr_t bp;
	unsigned long long words = (void )&bp;
	char *buf;
	int err;

	bzero(&bp, sizeof (bp));
	err = sscanf(thing, "%llx:%llx:%llx:%llx:%llx:%llx:%llx:%llx:"
	"%llx:%llx:%llx:%llx:%llx:%llx:%llx:%llx",
	words + 0, words + 1, words + 2, words + 3,
	words + 4, words + 5, words + 6, words + 7,
	words + 8, words + 9, words + 10, words + 11,
	words + 12, words + 13, words + 14, words + 15);
	if (err != 16) {
	(void) fprintf(stderr, "invalid input format\n");
	exit(1);
	}
	ASSERT3U(BPE_GET_LSIZE(&bp), <=, SPA_MAXBLOCKSIZE);
	buf = malloc(SPA_MAXBLOCKSIZE);
	if (buf == NULL) {
	(void) fprintf(stderr, "out of memory\n");
	exit(1);
	}
	err = decode_embedded_bp(&bp, buf, BPE_GET_LSIZE(&bp));
	if (err != 0) {
	(void) fprintf(stderr, "decode failed: %u\n", err);
	exit(1);
	}
	zdb_dump_block_raw(buf, BPE_GET_LSIZE(&bp), 0);
	free(buf);
	}

	int
	main(int argc, char **argv)
	{
	int c;
	struct rlimit rl = { 1024, 1024 };
	spa_t *spa = NULL;
	objset_t *os = NULL;
	int dump_all = 1;
	int verbose = 0;
	int error = 0;
	char **searchdirs = NULL;
	int nsearch = 0;
	char target, target_pool, dsname[ZFS_MAX_DATASET_NAME_LEN];
	nvlist_t *policy = NULL;
	uint64_t max_txg = UINT64_MAX;
	int64_t objset_id = -1;
	int flags = ZFS_IMPORT_MISSING_LOG;
	int rewind = ZPOOL_NEVER_REWIND;
	char spa_config_path_env, objset_str;
	boolean_t target_is_spa = B_TRUE, dataset_lookup = B_FALSE;
	nvlist_t *cfg = NULL;

	(void) setrlimit(RLIMIT_NOFILE, &rl);
	(void) enable_extended_FILE_stdio(-1, -1);

	dprintf_setup(&argc, argv);

	/*
	* If there is an environment variable SPA_CONFIG_PATH it overrides
	* default spa_config_path setting. If -U flag is specified it will
	* override this environment variable settings once again.
	*/
	spa_config_path_env = getenv("SPA_CONFIG_PATH");
	if (spa_config_path_env != NULL)
	spa_config_path = spa_config_path_env;

	/*
	* For performance reasons, we set this tunable down. We do so before
	* the arg parsing section so that the user can override this value if
	* they choose.
	*/
	zfs_btree_verify_intensity = 3;

	while ((c = getopt(argc, argv,
	"AbcCdDeEFGhiI:klLmMo:Op:PqRsSt:uU:vVx:XYyZ")) != -1) {
	switch (c) {
	case 'b':
	case 'c':
	case 'C':
	case 'd':
	case 'D':
	case 'E':
	case 'G':
	case 'h':
	case 'i':
	case 'l':
	case 'm':
	case 'M':
	case 'O':
	case 'R':
	case 's':
	case 'S':
	case 'u':
	case 'y':
	case 'Z':
	dump_opt[c]++;
	dump_all = 0;
	break;
	case 'A':
	case 'e':
	case 'F':
	case 'k':
	case 'L':
	case 'P':
	case 'q':
	case 'X':
	dump_opt[c]++;
	break;
	case 'Y':
	zfs_reconstruct_indirect_combinations_max = INT_MAX;
	zfs_deadman_enabled = 0;
	break;
	/* NB: Sort single match options below. */
	case 'I':
	max_inflight_bytes = strtoull(optarg, NULL, 0);
	if (max_inflight_bytes == 0) {
	(void) fprintf(stderr, "maximum number "
	"of inflight bytes must be greater "
	"than 0\n");
	usage();
	}
	break;
	case 'o':
	error = set_global_var(optarg);
	if (error != 0)
	usage();
	break;
	case 'p':
	if (searchdirs == NULL) {
	searchdirs = umem_alloc(sizeof (char *),
	UMEM_NOFAIL);
	} else {
	char *tmp = umem_alloc((nsearch + 1)
	sizeof (char *), UMEM_NOFAIL);
	bcopy(searchdirs, tmp, nsearch *
	sizeof (char *));
	umem_free(searchdirs,
	nsearch * sizeof (char *));
	searchdirs = tmp;
	}
	searchdirs[nsearch++] = optarg;
	break;
	case 't':
	max_txg = strtoull(optarg, NULL, 0);
	if (max_txg < TXG_INITIAL) {
	(void) fprintf(stderr, "incorrect txg "
	"specified: %s\n", optarg);
	usage();
	}
	break;
	case 'U':
	spa_config_path = optarg;
	if (spa_config_path[0] != '/') {
	(void) fprintf(stderr,
	"cachefile must be an absolute path "
	"(i.e. start with a slash)\n");
	usage();
	}
	break;
	case 'v':
	verbose++;
	break;
	case 'V':
	flags = ZFS_IMPORT_VERBATIM;
	break;
	case 'x':
	vn_dumpdir = optarg;
	break;
	default:
	usage();
	break;
	}
	}

	if (!dump_opt['e'] && searchdirs != NULL) {
	(void) fprintf(stderr, "-p option requires use of -e\n");
	usage();
	}
	if (dump_opt['d']) {
	/* <pool>[/<dataset \| objset id> is accepted */
	if (argv[2] && (objset_str = strchr(argv[2], '/')) != NULL &&
	objset_str++ != NULL) {
	char *endptr;
	errno = 0;
	objset_id = strtoull(objset_str, &endptr, 0);
	/* dataset 0 is the same as opening the pool */
	if (errno == 0 && endptr != objset_str &&
	objset_id != 0) {
	target_is_spa = B_FALSE;
	dataset_lookup = B_TRUE;
	} else if (objset_id != 0) {
	printf("failed to open objset %s "
	"%llu %s", objset_str,
	(u_longlong_t)objset_id,
	strerror(errno));
	exit(1);
	}
	/* normal dataset name not an objset ID */
	if (endptr == objset_str) {
	objset_id = -1;
	}
	}
	}

	#if defined(_LP64)
	/*
	* ZDB does not typically re-read blocks; therefore limit the ARC
	* to 256 MB, which can be used entirely for metadata.
	*/
	zfs_arc_min = zfs_arc_meta_min = 2ULL << SPA_MAXBLOCKSHIFT;
	zfs_arc_max = zfs_arc_meta_limit = 256 * 1024 * 1024;
	#endif

	/*
	* "zdb -c" uses checksum-verifying scrub i/os which are async reads.
	* "zdb -b" uses traversal prefetch which uses async reads.
	* For good performance, let several of them be active at once.
	*/
	zfs_vdev_async_read_max_active = 10;

	/*
	* Disable reference tracking for better performance.
	*/
	reference_tracking_enable = B_FALSE;

	/*
	* Do not fail spa_load when spa_load_verify fails. This is needed
	* to load non-idle pools.
	*/
	spa_load_verify_dryrun = B_TRUE;

	kernel_init(SPA_MODE_READ);

	if (dump_all)
	verbose = MAX(verbose, 1);

	for (c = 0; c < 256; c++) {
	if (dump_all && strchr("AeEFklLOPRSXy", c) == NULL)
	dump_opt[c] = 1;
	if (dump_opt[c])
	dump_opt[c] += verbose;
	}

	aok = (dump_opt['A'] == 1) \|\| (dump_opt['A'] > 2);
	zfs_recover = (dump_opt['A'] > 1);

	argc -= optind;
	argv += optind;
	if (argc < 2 && dump_opt['R'])
	usage();

	if (dump_opt['E']) {
	if (argc != 1)
	usage();
	zdb_embedded_block(argv[0]);
	return (0);
	}

	if (argc < 1) {
	if (!dump_opt['e'] && dump_opt['C']) {
	dump_cachefile(spa_config_path);
	return (0);
	}
	usage();
	}

	if (dump_opt['l'])
	return (dump_label(argv[0]));

	if (dump_opt['O']) {
	if (argc != 2)
	usage();
	dump_opt['v'] = verbose + 3;
	return (dump_path(argv[0], argv[1]));
	}

	if (dump_opt['X'] \|\| dump_opt['F'])
	rewind = ZPOOL_DO_REWIND \|
	(dump_opt['X'] ? ZPOOL_EXTREME_REWIND : 0);

	if (nvlist_alloc(&policy, NV_UNIQUE_NAME_TYPE, 0) != 0 \|\|
	nvlist_add_uint64(policy, ZPOOL_LOAD_REQUEST_TXG, max_txg) != 0 \|\|
	nvlist_add_uint32(policy, ZPOOL_LOAD_REWIND_POLICY, rewind) != 0)
	fatal("internal error: %s", strerror(ENOMEM));

	error = 0;
	target = argv[0];

	if (strpbrk(target, "/@") != NULL) {
	size_t targetlen;

	target_pool = strdup(target);
	*strpbrk(target_pool, "/@") = '\0';

	target_is_spa = B_FALSE;
	targetlen = strlen(target);
	if (targetlen && target[targetlen - 1] == '/')
	target[targetlen - 1] = '\0';
	} else {
	target_pool = target;
	}

	if (dump_opt['e']) {
	importargs_t args = { 0 };

	args.paths = nsearch;
	args.path = searchdirs;
	args.can_be_active = B_TRUE;

	error = zpool_find_config(NULL, target_pool, &cfg, &args,
	&libzpool_config_ops);

	if (error == 0) {

	if (nvlist_add_nvlist(cfg,
	ZPOOL_LOAD_POLICY, policy) != 0) {
	fatal("can't open '%s': %s",
	target, strerror(ENOMEM));
	}

	if (dump_opt['C'] > 1) {
	(void) printf("\nConfiguration for import:\n");
	dump_nvlist(cfg, 8);
	}

	/*
	* Disable the activity check to allow examination of
	* active pools.
	*/
	error = spa_import(target_pool, cfg, NULL,
	flags \| ZFS_IMPORT_SKIP_MMP);
	}
	}

	/*
	* import_checkpointed_state makes the assumption that the
	* target pool that we pass it is already part of the spa
	* namespace. Because of that we need to make sure to call
	* it always after the -e option has been processed, which
	* imports the pool to the namespace if it's not in the
	* cachefile.
	*/
	char *checkpoint_pool = NULL;
	char *checkpoint_target = NULL;
	if (dump_opt['k']) {
	checkpoint_pool = import_checkpointed_state(target, cfg,
	&checkpoint_target);

	if (checkpoint_target != NULL)
	target = checkpoint_target;
	}

	if (target_pool != target)
	free(target_pool);

	if (error == 0) {
	if (dump_opt['k'] && (target_is_spa \|\| dump_opt['R'])) {
	ASSERT(checkpoint_pool != NULL);
	ASSERT(checkpoint_target == NULL);

	error = spa_open(checkpoint_pool, &spa, FTAG);
	if (error != 0) {
	fatal("Tried to open pool \"%s\" but "
	"spa_open() failed with error %d\n",
	checkpoint_pool, error);
	}

	} else if (target_is_spa \|\| dump_opt['R'] \|\| objset_id == 0) {
	zdb_set_skip_mmp(target);
	error = spa_open_rewind(target, &spa, FTAG, policy,
	NULL);
	if (error) {
	/*
	* If we're missing the log device then
	* try opening the pool after clearing the
	* log state.
	*/
	mutex_enter(&spa_namespace_lock);
	if ((spa = spa_lookup(target)) != NULL &&
	spa->spa_log_state == SPA_LOG_MISSING) {
	spa->spa_log_state = SPA_LOG_CLEAR;
	error = 0;
	}
	mutex_exit(&spa_namespace_lock);

	if (!error) {
	error = spa_open_rewind(target, &spa,
	FTAG, policy, NULL);
	}
	}
	} else if (strpbrk(target, "#") != NULL) {
	dsl_pool_t *dp;
	error = dsl_pool_hold(target, FTAG, &dp);
	if (error != 0) {
	fatal("can't dump '%s': %s", target,
	strerror(error));
	}
	error = dump_bookmark(dp, target, B_TRUE, verbose > 1);
	dsl_pool_rele(dp, FTAG);
	if (error != 0) {
	fatal("can't dump '%s': %s", target,
	strerror(error));
	}
	return (error);
	} else {
	zdb_set_skip_mmp(target);
	if (dataset_lookup == B_TRUE) {
	/*
	* Use the supplied id to get the name
	* for open_objset.
	*/
	error = spa_open(target, &spa, FTAG);
	if (error == 0) {
	error = name_from_objset_id(spa,
	objset_id, dsname);
	spa_close(spa, FTAG);
	if (error == 0)
	target = dsname;
	}
	}
	if (error == 0)
	error = open_objset(target, FTAG, &os);
	if (error == 0)
	spa = dmu_objset_spa(os);
	}
	}
	nvlist_free(policy);

	if (error)
	fatal("can't open '%s': %s", target, strerror(error));

	/*
	* Set the pool failure mode to panic in order to prevent the pool
	* from suspending. A suspended I/O will have no way to resume and
	* can prevent the zdb(8) command from terminating as expected.
	*/
	if (spa != NULL)
	spa->spa_failmode = ZIO_FAILURE_MODE_PANIC;

	argv++;
	argc--;
	if (!dump_opt['R']) {
	flagbits['d'] = ZOR_FLAG_DIRECTORY;
	flagbits['f'] = ZOR_FLAG_PLAIN_FILE;
	flagbits['m'] = ZOR_FLAG_SPACE_MAP;
	flagbits['z'] = ZOR_FLAG_ZAP;
	flagbits['A'] = ZOR_FLAG_ALL_TYPES;

	if (argc > 0 && dump_opt['d']) {
	zopt_object_args = argc;
	zopt_object_ranges = calloc(zopt_object_args,
	sizeof (zopt_object_range_t));
	for (unsigned i = 0; i < zopt_object_args; i++) {
	int err;
	char *msg = NULL;

	err = parse_object_range(argv[i],
	&zopt_object_ranges[i], &msg);
	if (err != 0)
	fatal("Bad object or range: '%s': %s\n",
	argv[i], msg ? msg : "");
	}
	} else if (argc > 0 && dump_opt['m']) {
	zopt_metaslab_args = argc;
	zopt_metaslab = calloc(zopt_metaslab_args,
	sizeof (uint64_t));
	for (unsigned i = 0; i < zopt_metaslab_args; i++) {
	errno = 0;
	zopt_metaslab[i] = strtoull(argv[i], NULL, 0);
	if (zopt_metaslab[i] == 0 && errno != 0)
	fatal("bad number %s: %s", argv[i],
	strerror(errno));
	}
	}
	if (os != NULL) {
	dump_objset(os);
	} else if (zopt_object_args > 0 && !dump_opt['m']) {
	dump_objset(spa->spa_meta_objset);
	} else {
	dump_zpool(spa);
	}
	} else {
	flagbits['b'] = ZDB_FLAG_PRINT_BLKPTR;
	flagbits['c'] = ZDB_FLAG_CHECKSUM;
	flagbits['d'] = ZDB_FLAG_DECOMPRESS;
	flagbits['e'] = ZDB_FLAG_BSWAP;
	flagbits['g'] = ZDB_FLAG_GBH;
	flagbits['i'] = ZDB_FLAG_INDIRECT;
	flagbits['r'] = ZDB_FLAG_RAW;
	flagbits['v'] = ZDB_FLAG_VERBOSE;

	for (int i = 0; i < argc; i++)
	zdb_read_block(argv[i], spa);
	}

	if (dump_opt['k']) {
	free(checkpoint_pool);
	if (!target_is_spa)
	free(checkpoint_target);
	}

	if (os != NULL) {
	close_objset(os, FTAG);
	} else {
	spa_close(spa, FTAG);
	}

	fuid_table_destroy();

	dump_debug_buffer();

	kernel_fini();

	return (error);
	}
	Index: vendor-sys/openzfs/dist/cmd/zdb/zdb_il.c
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zdb/zdb_il.c (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zdb/zdb_il.c (revision 366774)
	@@ -1,431 +1,432 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright 2009 Sun Microsystems, Inc. All rights reserved.
	* Copyright (c) 2012 Cyril Plisko. All rights reserved.
	* Use is subject to license terms.
	*/

	/*
	* Copyright (c) 2013, 2017 by Delphix. All rights reserved.
	*/

	/*
	* Print intent log header and statistics.
	*/

	#include <stdio.h>
	#include <stdlib.h>
	#include <ctype.h>
	#include <sys/zfs_context.h>
	#include <sys/spa.h>
	#include <sys/dmu.h>
	#include <sys/stat.h>
	#include <sys/resource.h>
	#include <sys/zil.h>
	#include <sys/zil_impl.h>
	#include <sys/spa_impl.h>
	#include <sys/abd.h>

	#include "zdb.h"

	extern uint8_t dump_opt[256];

	static char tab_prefix[4] = "\t\t\t";

	static void
	print_log_bp(const blkptr_t bp, const char prefix)
	{
	char blkbuf[BP_SPRINTF_LEN];

	snprintf_blkptr(blkbuf, sizeof (blkbuf), bp);
	(void) printf("%s%s\n", prefix, blkbuf);
	}

	/* ARGSUSED */
	static void
	-zil_prt_rec_create(zilog_t zilog, int txtype, void arg)
	+zil_prt_rec_create(zilog_t zilog, int txtype, const void arg)
	{
	- lr_create_t *lr = arg;
	+ const lr_create_t *lr = arg;
	time_t crtime = lr->lr_crtime[0];
	char name, link;
	lr_attr_t *lrattr;

	name = (char *)(lr + 1);

	if (lr->lr_common.lrc_txtype == TX_CREATE_ATTR \|\|
	lr->lr_common.lrc_txtype == TX_MKDIR_ATTR) {
	lrattr = (lr_attr_t *)(lr + 1);
	name += ZIL_XVAT_SIZE(lrattr->lr_attr_masksize);
	}

	if (txtype == TX_SYMLINK) {
	link = name + strlen(name) + 1;
	(void) printf("%s%s -> %s\n", tab_prefix, name, link);
	} else if (txtype != TX_MKXATTR) {
	(void) printf("%s%s\n", tab_prefix, name);
	}

	(void) printf("%s%s", tab_prefix, ctime(&crtime));
	(void) printf("%sdoid %llu, foid %llu, slots %llu, mode %llo\n",
	tab_prefix, (u_longlong_t)lr->lr_doid,
	(u_longlong_t)LR_FOID_GET_OBJ(lr->lr_foid),
	(u_longlong_t)LR_FOID_GET_SLOTS(lr->lr_foid),
	(longlong_t)lr->lr_mode);
	(void) printf("%suid %llu, gid %llu, gen %llu, rdev 0x%llx\n",
	tab_prefix,
	(u_longlong_t)lr->lr_uid, (u_longlong_t)lr->lr_gid,
	(u_longlong_t)lr->lr_gen, (u_longlong_t)lr->lr_rdev);
	}

	/* ARGSUSED */
	static void
	-zil_prt_rec_remove(zilog_t zilog, int txtype, void arg)
	+zil_prt_rec_remove(zilog_t zilog, int txtype, const void arg)
	{
	- lr_remove_t *lr = arg;
	+ const lr_remove_t *lr = arg;

	(void) printf("%sdoid %llu, name %s\n", tab_prefix,
	(u_longlong_t)lr->lr_doid, (char *)(lr + 1));
	}

	/* ARGSUSED */
	static void
	-zil_prt_rec_link(zilog_t zilog, int txtype, void arg)
	+zil_prt_rec_link(zilog_t zilog, int txtype, const void arg)
	{
	- lr_link_t *lr = arg;
	+ const lr_link_t *lr = arg;

	(void) printf("%sdoid %llu, link_obj %llu, name %s\n", tab_prefix,
	(u_longlong_t)lr->lr_doid, (u_longlong_t)lr->lr_link_obj,
	(char *)(lr + 1));
	}

	/* ARGSUSED */
	static void
	-zil_prt_rec_rename(zilog_t zilog, int txtype, void arg)
	+zil_prt_rec_rename(zilog_t zilog, int txtype, const void arg)
	{
	- lr_rename_t *lr = arg;
	+ const lr_rename_t *lr = arg;
	char snm = (char )(lr + 1);
	char *tnm = snm + strlen(snm) + 1;

	(void) printf("%ssdoid %llu, tdoid %llu\n", tab_prefix,
	(u_longlong_t)lr->lr_sdoid, (u_longlong_t)lr->lr_tdoid);
	(void) printf("%ssrc %s tgt %s\n", tab_prefix, snm, tnm);
	}

	/* ARGSUSED */
	static int
	zil_prt_rec_write_cb(void data, size_t len, void unused)
	{
	char *cdata = data;

	for (size_t i = 0; i < len; i++) {
	if (isprint(*cdata))
	(void) printf("%c ", *cdata);
	else
	(void) printf("%2X", *cdata);
	cdata++;
	}
	return (0);
	}

	/* ARGSUSED */
	static void
	-zil_prt_rec_write(zilog_t zilog, int txtype, void arg)
	+zil_prt_rec_write(zilog_t zilog, int txtype, const void arg)
	{
	- lr_write_t *lr = arg;
	+ const lr_write_t *lr = arg;
	abd_t *data;
	- blkptr_t *bp = &lr->lr_blkptr;
	+ const blkptr_t *bp = &lr->lr_blkptr;
	zbookmark_phys_t zb;
	int verbose = MAX(dump_opt['d'], dump_opt['i']);
	int error;

	(void) printf("%sfoid %llu, offset %llx, length %llx\n", tab_prefix,
	(u_longlong_t)lr->lr_foid, (u_longlong_t)lr->lr_offset,
	(u_longlong_t)lr->lr_length);

	if (txtype == TX_WRITE2 \|\| verbose < 5)
	return;

	if (lr->lr_common.lrc_reclen == sizeof (lr_write_t)) {
	(void) printf("%shas blkptr, %s\n", tab_prefix,
	!BP_IS_HOLE(bp) &&
	bp->blk_birth >= spa_min_claim_txg(zilog->zl_spa) ?
	"will claim" : "won't claim");
	print_log_bp(bp, tab_prefix);

	if (BP_IS_HOLE(bp)) {
	(void) printf("\t\t\tLSIZE 0x%llx\n",
	(u_longlong_t)BP_GET_LSIZE(bp));
	(void) printf("%s<hole>\n", tab_prefix);
	return;
	}
	if (bp->blk_birth < zilog->zl_header->zh_claim_txg) {
	(void) printf("%s<block already committed>\n",
	tab_prefix);
	return;
	}

	SET_BOOKMARK(&zb, dmu_objset_id(zilog->zl_os),
	lr->lr_foid, ZB_ZIL_LEVEL,
	lr->lr_offset / BP_GET_LSIZE(bp));

	data = abd_alloc(BP_GET_LSIZE(bp), B_FALSE);
	error = zio_wait(zio_read(NULL, zilog->zl_spa,
	bp, data, BP_GET_LSIZE(bp), NULL, NULL,
	ZIO_PRIORITY_SYNC_READ, ZIO_FLAG_CANFAIL, &zb));
	if (error)
	goto out;
	} else {
	/* data is stored after the end of the lr_write record */
	data = abd_alloc(lr->lr_length, B_FALSE);
	abd_copy_from_buf(data, lr + 1, lr->lr_length);
	}

	(void) printf("%s", tab_prefix);
	(void) abd_iterate_func(data,
	0, MIN(lr->lr_length, (verbose < 6 ? 20 : SPA_MAXBLOCKSIZE)),
	zil_prt_rec_write_cb, NULL);
	(void) printf("\n");

	out:
	abd_free(data);
	}

	/* ARGSUSED */
	static void
	-zil_prt_rec_truncate(zilog_t zilog, int txtype, void arg)
	+zil_prt_rec_truncate(zilog_t zilog, int txtype, const void arg)
	{
	- lr_truncate_t *lr = arg;
	+ const lr_truncate_t *lr = arg;

	(void) printf("%sfoid %llu, offset 0x%llx, length 0x%llx\n", tab_prefix,
	(u_longlong_t)lr->lr_foid, (longlong_t)lr->lr_offset,
	(u_longlong_t)lr->lr_length);
	}

	/* ARGSUSED */
	static void
	-zil_prt_rec_setattr(zilog_t zilog, int txtype, void arg)
	+zil_prt_rec_setattr(zilog_t zilog, int txtype, const void arg)
	{
	- lr_setattr_t *lr = arg;
	+ const lr_setattr_t *lr = arg;
	time_t atime = (time_t)lr->lr_atime[0];
	time_t mtime = (time_t)lr->lr_mtime[0];

	(void) printf("%sfoid %llu, mask 0x%llx\n", tab_prefix,
	(u_longlong_t)lr->lr_foid, (u_longlong_t)lr->lr_mask);

	if (lr->lr_mask & AT_MODE) {
	(void) printf("%sAT_MODE %llo\n", tab_prefix,
	(longlong_t)lr->lr_mode);
	}

	if (lr->lr_mask & AT_UID) {
	(void) printf("%sAT_UID %llu\n", tab_prefix,
	(u_longlong_t)lr->lr_uid);
	}

	if (lr->lr_mask & AT_GID) {
	(void) printf("%sAT_GID %llu\n", tab_prefix,
	(u_longlong_t)lr->lr_gid);
	}

	if (lr->lr_mask & AT_SIZE) {
	(void) printf("%sAT_SIZE %llu\n", tab_prefix,
	(u_longlong_t)lr->lr_size);
	}

	if (lr->lr_mask & AT_ATIME) {
	(void) printf("%sAT_ATIME %llu.%09llu %s", tab_prefix,
	(u_longlong_t)lr->lr_atime[0],
	(u_longlong_t)lr->lr_atime[1],
	ctime(&atime));
	}

	if (lr->lr_mask & AT_MTIME) {
	(void) printf("%sAT_MTIME %llu.%09llu %s", tab_prefix,
	(u_longlong_t)lr->lr_mtime[0],
	(u_longlong_t)lr->lr_mtime[1],
	ctime(&mtime));
	}
	}

	/* ARGSUSED */
	static void
	-zil_prt_rec_acl(zilog_t zilog, int txtype, void arg)
	+zil_prt_rec_acl(zilog_t zilog, int txtype, const void arg)
	{
	- lr_acl_t *lr = arg;
	+ const lr_acl_t *lr = arg;

	(void) printf("%sfoid %llu, aclcnt %llu\n", tab_prefix,
	(u_longlong_t)lr->lr_foid, (u_longlong_t)lr->lr_aclcnt);
	}

	-typedef void (zil_prt_rec_func_t)(zilog_t , int, void *);
	+typedef void (zil_prt_rec_func_t)(zilog_t , int, const void *);
	typedef struct zil_rec_info {
	zil_prt_rec_func_t zri_print;
	const char *zri_name;
	uint64_t zri_count;
	} zil_rec_info_t;

	static zil_rec_info_t zil_rec_info[TX_MAX_TYPE] = {
	{.zri_print = NULL, .zri_name = "Total "},
	{.zri_print = zil_prt_rec_create, .zri_name = "TX_CREATE "},
	{.zri_print = zil_prt_rec_create, .zri_name = "TX_MKDIR "},
	{.zri_print = zil_prt_rec_create, .zri_name = "TX_MKXATTR "},
	{.zri_print = zil_prt_rec_create, .zri_name = "TX_SYMLINK "},
	{.zri_print = zil_prt_rec_remove, .zri_name = "TX_REMOVE "},
	{.zri_print = zil_prt_rec_remove, .zri_name = "TX_RMDIR "},
	{.zri_print = zil_prt_rec_link, .zri_name = "TX_LINK "},
	{.zri_print = zil_prt_rec_rename, .zri_name = "TX_RENAME "},
	{.zri_print = zil_prt_rec_write, .zri_name = "TX_WRITE "},
	{.zri_print = zil_prt_rec_truncate, .zri_name = "TX_TRUNCATE "},
	{.zri_print = zil_prt_rec_setattr, .zri_name = "TX_SETATTR "},
	{.zri_print = zil_prt_rec_acl, .zri_name = "TX_ACL_V0 "},
	{.zri_print = zil_prt_rec_acl, .zri_name = "TX_ACL_ACL "},
	{.zri_print = zil_prt_rec_create, .zri_name = "TX_CREATE_ACL "},
	{.zri_print = zil_prt_rec_create, .zri_name = "TX_CREATE_ATTR "},
	{.zri_print = zil_prt_rec_create, .zri_name = "TX_CREATE_ACL_ATTR "},
	{.zri_print = zil_prt_rec_create, .zri_name = "TX_MKDIR_ACL "},
	{.zri_print = zil_prt_rec_create, .zri_name = "TX_MKDIR_ATTR "},
	{.zri_print = zil_prt_rec_create, .zri_name = "TX_MKDIR_ACL_ATTR "},
	{.zri_print = zil_prt_rec_write, .zri_name = "TX_WRITE2 "},
	};

	/* ARGSUSED */
	static int
	-print_log_record(zilog_t zilog, lr_t lr, void *arg, uint64_t claim_txg)
	+print_log_record(zilog_t zilog, const lr_t lr, void *arg, uint64_t claim_txg)
	{
	int txtype;
	int verbose = MAX(dump_opt['d'], dump_opt['i']);

	/* reduce size of txtype to strip off TX_CI bit */
	txtype = lr->lrc_txtype;

	ASSERT(txtype != 0 && (uint_t)txtype < TX_MAX_TYPE);
	ASSERT(lr->lrc_txg);

	(void) printf("\t\t%s%s len %6llu, txg %llu, seq %llu\n",
	(lr->lrc_txtype & TX_CI) ? "CI-" : "",
	zil_rec_info[txtype].zri_name,
	(u_longlong_t)lr->lrc_reclen,
	(u_longlong_t)lr->lrc_txg,
	(u_longlong_t)lr->lrc_seq);

	if (txtype && verbose >= 3) {
	if (!zilog->zl_os->os_encrypted) {
	zil_rec_info[txtype].zri_print(zilog, txtype, lr);
	} else {
	(void) printf("%s(encrypted)\n", tab_prefix);
	}
	}

	zil_rec_info[txtype].zri_count++;
	zil_rec_info[0].zri_count++;

	return (0);
	}

	/* ARGSUSED */
	static int
	-print_log_block(zilog_t zilog, blkptr_t bp, void *arg, uint64_t claim_txg)
	+print_log_block(zilog_t zilog, const blkptr_t bp, void *arg,
	+ uint64_t claim_txg)
	{
	char blkbuf[BP_SPRINTF_LEN + 10];
	int verbose = MAX(dump_opt['d'], dump_opt['i']);
	const char *claim;

	if (verbose <= 3)
	return (0);

	if (verbose >= 5) {
	(void) strcpy(blkbuf, ", ");
	snprintf_blkptr(blkbuf + strlen(blkbuf),
	sizeof (blkbuf) - strlen(blkbuf), bp);
	} else {
	blkbuf[0] = '\0';
	}

	if (claim_txg != 0)
	claim = "already claimed";
	else if (bp->blk_birth >= spa_min_claim_txg(zilog->zl_spa))
	claim = "will claim";
	else
	claim = "won't claim";

	(void) printf("\tBlock seqno %llu, %s%s\n",
	(u_longlong_t)bp->blk_cksum.zc_word[ZIL_ZC_SEQ], claim, blkbuf);

	return (0);
	}

	static void
	print_log_stats(int verbose)
	{
	unsigned i, w, p10;

	if (verbose > 3)
	(void) printf("\n");

	if (zil_rec_info[0].zri_count == 0)
	return;

	for (w = 1, p10 = 10; zil_rec_info[0].zri_count >= p10; p10 *= 10)
	w++;

	for (i = 0; i < TX_MAX_TYPE; i++)
	if (zil_rec_info[i].zri_count \|\| verbose >= 3)
	(void) printf("\t\t%s %*llu\n",
	zil_rec_info[i].zri_name, w,
	(u_longlong_t)zil_rec_info[i].zri_count);
	(void) printf("\n");
	}

	/* ARGSUSED */
	void
	dump_intent_log(zilog_t *zilog)
	{
	const zil_header_t *zh = zilog->zl_header;
	int verbose = MAX(dump_opt['d'], dump_opt['i']);
	int i;

	if (BP_IS_HOLE(&zh->zh_log) \|\| verbose < 1)
	return;

	(void) printf("\n ZIL header: claim_txg %llu, "
	"claim_blk_seq %llu, claim_lr_seq %llu",
	(u_longlong_t)zh->zh_claim_txg,
	(u_longlong_t)zh->zh_claim_blk_seq,
	(u_longlong_t)zh->zh_claim_lr_seq);
	(void) printf(" replay_seq %llu, flags 0x%llx\n",
	(u_longlong_t)zh->zh_replay_seq, (u_longlong_t)zh->zh_flags);

	for (i = 0; i < TX_MAX_TYPE; i++)
	zil_rec_info[i].zri_count = 0;

	/* see comment in zil_claim() or zil_check_log_chain() */
	if (zilog->zl_spa->spa_uberblock.ub_checkpoint_txg != 0 &&
	zh->zh_claim_txg == 0)
	return;

	if (verbose >= 2) {
	(void) printf("\n");
	(void) zil_parse(zilog, print_log_block, print_log_record, NULL,
	zh->zh_claim_txg, B_FALSE);
	print_log_stats(verbose);
	}
	}
	Index: vendor-sys/openzfs/dist/cmd/zed/agents/zfs_agents.c
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zed/agents/zfs_agents.c (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zed/agents/zfs_agents.c (revision 366774)
	@@ -1,422 +1,422 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License Version 1.0 (CDDL-1.0).
	* You can obtain a copy of the license from the top-level file
	* "OPENSOLARIS.LICENSE" or at <http://opensource.org/licenses/CDDL-1.0>.
	* You may not use this file except in compliance with the license.
	*
	* CDDL HEADER END
	*/

	/*
	* Copyright (c) 2016, Intel Corporation.
	* Copyright (c) 2018, loli10K <ezomori.nozomu@gmail.com>
	*/

	#include <libnvpair.h>
	#include <libzfs.h>
	#include <stddef.h>
	#include <stdlib.h>
	#include <string.h>
	#include <sys/list.h>
	#include <sys/time.h>
	#include <sys/sysevent/eventdefs.h>
	#include <sys/sysevent/dev.h>
	#include <sys/fm/protocol.h>
	#include <sys/fm/fs/zfs.h>
	#include <pthread.h>
	#include <unistd.h>

	#include "zfs_agents.h"
	#include "fmd_api.h"
	#include "../zed_log.h"

	/*
	* agent dispatch code
	*/

	static pthread_mutex_t agent_lock = PTHREAD_MUTEX_INITIALIZER;
	static pthread_cond_t agent_cond = PTHREAD_COND_INITIALIZER;
	static list_t agent_events; /* list of pending events */
	static int agent_exiting;

	typedef struct agent_event {
	char ae_class[64];
	char ae_subclass[32];
	nvlist_t *ae_nvl;
	list_node_t ae_node;
	} agent_event_t;

	pthread_t g_agents_tid;

	libzfs_handle_t *g_zfs_hdl;

	/* guid search data */
	typedef enum device_type {
	DEVICE_TYPE_L2ARC, /* l2arc device */
	DEVICE_TYPE_SPARE, /* spare device */
	DEVICE_TYPE_PRIMARY /* any primary pool storage device */
	} device_type_t;

	typedef struct guid_search {
	uint64_t gs_pool_guid;
	uint64_t gs_vdev_guid;
	char *gs_devid;
	device_type_t gs_vdev_type;
	uint64_t gs_vdev_expandtime; /* vdev expansion time */
	} guid_search_t;

	/*
	* Walks the vdev tree recursively looking for a matching devid.
	* Returns B_TRUE as soon as a matching device is found, B_FALSE otherwise.
	*/
	static boolean_t
	zfs_agent_iter_vdev(zpool_handle_t zhp, nvlist_t nvl, void *arg)
	{
	guid_search_t *gsp = arg;
	char *path = NULL;
	uint_t c, children;
	nvlist_t **child;

	/*
	* First iterate over any children.
	*/
	if (nvlist_lookup_nvlist_array(nvl, ZPOOL_CONFIG_CHILDREN,
	&child, &children) == 0) {
	for (c = 0; c < children; c++) {
	if (zfs_agent_iter_vdev(zhp, child[c], gsp)) {
	gsp->gs_vdev_type = DEVICE_TYPE_PRIMARY;
	return (B_TRUE);
	}
	}
	}
	/*
	* Iterate over any spares and cache devices
	*/
	if (nvlist_lookup_nvlist_array(nvl, ZPOOL_CONFIG_SPARES,
	&child, &children) == 0) {
	for (c = 0; c < children; c++) {
	if (zfs_agent_iter_vdev(zhp, child[c], gsp)) {
	gsp->gs_vdev_type = DEVICE_TYPE_L2ARC;
	return (B_TRUE);
	}
	}
	}
	if (nvlist_lookup_nvlist_array(nvl, ZPOOL_CONFIG_L2CACHE,
	&child, &children) == 0) {
	for (c = 0; c < children; c++) {
	if (zfs_agent_iter_vdev(zhp, child[c], gsp)) {
	gsp->gs_vdev_type = DEVICE_TYPE_SPARE;
	return (B_TRUE);
	}
	}
	}
	/*
	* On a devid match, grab the vdev guid and expansion time, if any.
	*/
	if (gsp->gs_devid != NULL &&
	(nvlist_lookup_string(nvl, ZPOOL_CONFIG_DEVID, &path) == 0) &&
	(strcmp(gsp->gs_devid, path) == 0)) {
	(void) nvlist_lookup_uint64(nvl, ZPOOL_CONFIG_GUID,
	&gsp->gs_vdev_guid);
	(void) nvlist_lookup_uint64(nvl, ZPOOL_CONFIG_EXPANSION_TIME,
	&gsp->gs_vdev_expandtime);
	return (B_TRUE);
	}

	return (B_FALSE);
	}

	static int
	zfs_agent_iter_pool(zpool_handle_t zhp, void arg)
	{
	guid_search_t *gsp = arg;
	nvlist_t config, nvl;

	/*
	* For each vdev in this pool, look for a match by devid
	*/
	if ((config = zpool_get_config(zhp, NULL)) != NULL) {
	if (nvlist_lookup_nvlist(config, ZPOOL_CONFIG_VDEV_TREE,
	&nvl) == 0) {
	(void) zfs_agent_iter_vdev(zhp, nvl, gsp);
	}
	}
	/*
	* if a match was found then grab the pool guid
	*/
	if (gsp->gs_vdev_guid) {
	(void) nvlist_lookup_uint64(config, ZPOOL_CONFIG_POOL_GUID,
	&gsp->gs_pool_guid);
	}

	zpool_close(zhp);
	return (gsp->gs_vdev_guid != 0);
	}

	void
	zfs_agent_post_event(const char class, const char subclass, nvlist_t *nvl)
	{
	agent_event_t *event;

	if (subclass == NULL)
	subclass = "";

	event = malloc(sizeof (agent_event_t));
	if (event == NULL \|\| nvlist_dup(nvl, &event->ae_nvl, 0) != 0) {
	if (event)
	free(event);
	return;
	}

	if (strcmp(class, "sysevent.fs.zfs.vdev_check") == 0) {
	class = EC_ZFS;
	subclass = ESC_ZFS_VDEV_CHECK;
	}

	/*
	- * On ZFS on Linux, we don't get the expected FM_RESOURCE_REMOVED
	- * ereport from vdev_disk layer after a hot unplug. Fortunately we
	- * get a EC_DEV_REMOVE from our disk monitor and it is a suitable
	+ * On Linux, we don't get the expected FM_RESOURCE_REMOVED ereport
	+ * from the vdev_disk layer after a hot unplug. Fortunately we do
	+ * get an EC_DEV_REMOVE from our disk monitor and it is a suitable
	* proxy so we remap it here for the benefit of the diagnosis engine.
	*/
	if ((strcmp(class, EC_DEV_REMOVE) == 0) &&
	(strcmp(subclass, ESC_DISK) == 0) &&
	(nvlist_exists(nvl, ZFS_EV_VDEV_GUID) \|\|
	nvlist_exists(nvl, DEV_IDENTIFIER))) {
	nvlist_t *payload = event->ae_nvl;
	struct timeval tv;
	int64_t tod[2];
	uint64_t pool_guid = 0, vdev_guid = 0;
	guid_search_t search = { 0 };
	device_type_t devtype = DEVICE_TYPE_PRIMARY;

	class = "resource.fs.zfs.removed";
	subclass = "";

	(void) nvlist_add_string(payload, FM_CLASS, class);
	(void) nvlist_lookup_uint64(nvl, ZFS_EV_POOL_GUID, &pool_guid);
	(void) nvlist_lookup_uint64(nvl, ZFS_EV_VDEV_GUID, &vdev_guid);

	(void) gettimeofday(&tv, NULL);
	tod[0] = tv.tv_sec;
	tod[1] = tv.tv_usec;
	(void) nvlist_add_int64_array(payload, FM_EREPORT_TIME, tod, 2);

	/*
	* For multipath, spare and l2arc devices ZFS_EV_VDEV_GUID or
	* ZFS_EV_POOL_GUID may be missing so find them.
	*/
	(void) nvlist_lookup_string(nvl, DEV_IDENTIFIER,
	&search.gs_devid);
	(void) zpool_iter(g_zfs_hdl, zfs_agent_iter_pool, &search);
	pool_guid = search.gs_pool_guid;
	vdev_guid = search.gs_vdev_guid;
	devtype = search.gs_vdev_type;

	/*
	* We want to avoid reporting "remove" events coming from
	* libudev for VDEVs which were expanded recently (10s) and
	* avoid activating spares in response to partitions being
	* deleted and created in rapid succession.
	*/
	if (search.gs_vdev_expandtime != 0 &&
	search.gs_vdev_expandtime + 10 > tv.tv_sec) {
	zed_log_msg(LOG_INFO, "agent post event: ignoring '%s' "
	"for recently expanded device '%s'", EC_DEV_REMOVE,
	search.gs_devid);
	goto out;
	}

	(void) nvlist_add_uint64(payload,
	FM_EREPORT_PAYLOAD_ZFS_POOL_GUID, pool_guid);
	(void) nvlist_add_uint64(payload,
	FM_EREPORT_PAYLOAD_ZFS_VDEV_GUID, vdev_guid);
	switch (devtype) {
	case DEVICE_TYPE_L2ARC:
	(void) nvlist_add_string(payload,
	FM_EREPORT_PAYLOAD_ZFS_VDEV_TYPE,
	VDEV_TYPE_L2CACHE);
	break;
	case DEVICE_TYPE_SPARE:
	(void) nvlist_add_string(payload,
	FM_EREPORT_PAYLOAD_ZFS_VDEV_TYPE, VDEV_TYPE_SPARE);
	break;
	case DEVICE_TYPE_PRIMARY:
	(void) nvlist_add_string(payload,
	FM_EREPORT_PAYLOAD_ZFS_VDEV_TYPE, VDEV_TYPE_DISK);
	break;
	}

	zed_log_msg(LOG_INFO, "agent post event: mapping '%s' to '%s'",
	EC_DEV_REMOVE, class);
	}

	(void) strlcpy(event->ae_class, class, sizeof (event->ae_class));
	(void) strlcpy(event->ae_subclass, subclass,
	sizeof (event->ae_subclass));

	(void) pthread_mutex_lock(&agent_lock);
	list_insert_tail(&agent_events, event);
	(void) pthread_mutex_unlock(&agent_lock);

	out:
	(void) pthread_cond_signal(&agent_cond);
	}

	static void
	zfs_agent_dispatch(const char class, const char subclass, nvlist_t *nvl)
	{
	/*
	* The diagnosis engine subscribes to the following events.
	* On illumos these subscriptions reside in:
	* /usr/lib/fm/fmd/plugins/zfs-diagnosis.conf
	*/
	if (strstr(class, "ereport.fs.zfs.") != NULL \|\|
	strstr(class, "resource.fs.zfs.") != NULL \|\|
	strcmp(class, "sysevent.fs.zfs.vdev_remove") == 0 \|\|
	strcmp(class, "sysevent.fs.zfs.vdev_remove_dev") == 0 \|\|
	strcmp(class, "sysevent.fs.zfs.pool_destroy") == 0) {
	fmd_module_recv(fmd_module_hdl("zfs-diagnosis"), nvl, class);
	}

	/*
	* The retire agent subscribes to the following events.
	* On illumos these subscriptions reside in:
	* /usr/lib/fm/fmd/plugins/zfs-retire.conf
	*
	* NOTE: faults events come directly from our diagnosis engine
	* and will not pass through the zfs kernel module.
	*/
	if (strcmp(class, FM_LIST_SUSPECT_CLASS) == 0 \|\|
	strcmp(class, "resource.fs.zfs.removed") == 0 \|\|
	strcmp(class, "resource.fs.zfs.statechange") == 0 \|\|
	strcmp(class, "sysevent.fs.zfs.vdev_remove") == 0) {
	fmd_module_recv(fmd_module_hdl("zfs-retire"), nvl, class);
	}

	/*
	* The SLM module only consumes disk events and vdev check events
	*
	* NOTE: disk events come directly from disk monitor and will
	* not pass through the zfs kernel module.
	*/
	if (strstr(class, "EC_dev_") != NULL \|\|
	strcmp(class, EC_ZFS) == 0) {
	(void) zfs_slm_event(class, subclass, nvl);
	}
	}

	/*
	* Events are consumed and dispatched from this thread
	* An agent can also post an event so event list lock
	* is not held when calling an agent.
	* One event is consumed at a time.
	*/
	static void *
	zfs_agent_consumer_thread(void *arg)
	{
	for (;;) {
	agent_event_t *event;

	(void) pthread_mutex_lock(&agent_lock);

	/* wait for an event to show up */
	while (!agent_exiting && list_is_empty(&agent_events))
	(void) pthread_cond_wait(&agent_cond, &agent_lock);

	if (agent_exiting) {
	(void) pthread_mutex_unlock(&agent_lock);
	zed_log_msg(LOG_INFO, "zfs_agent_consumer_thread: "
	"exiting");
	return (NULL);
	}

	if ((event = (list_head(&agent_events))) != NULL) {
	list_remove(&agent_events, event);

	(void) pthread_mutex_unlock(&agent_lock);

	/* dispatch to all event subscribers */
	zfs_agent_dispatch(event->ae_class, event->ae_subclass,
	event->ae_nvl);

	nvlist_free(event->ae_nvl);
	free(event);
	continue;
	}

	(void) pthread_mutex_unlock(&agent_lock);
	}

	return (NULL);
	}

	void
	zfs_agent_init(libzfs_handle_t *zfs_hdl)
	{
	fmd_hdl_t *hdl;

	g_zfs_hdl = zfs_hdl;

	if (zfs_slm_init() != 0)
	zed_log_die("Failed to initialize zfs slm");
	zed_log_msg(LOG_INFO, "Add Agent: init");

	hdl = fmd_module_hdl("zfs-diagnosis");
	_zfs_diagnosis_init(hdl);
	if (!fmd_module_initialized(hdl))
	zed_log_die("Failed to initialize zfs diagnosis");

	hdl = fmd_module_hdl("zfs-retire");
	_zfs_retire_init(hdl);
	if (!fmd_module_initialized(hdl))
	zed_log_die("Failed to initialize zfs retire");

	list_create(&agent_events, sizeof (agent_event_t),
	offsetof(struct agent_event, ae_node));

	if (pthread_create(&g_agents_tid, NULL, zfs_agent_consumer_thread,
	NULL) != 0) {
	list_destroy(&agent_events);
	zed_log_die("Failed to initialize agents");
	}
	}

	void
	zfs_agent_fini(void)
	{
	fmd_hdl_t *hdl;
	agent_event_t *event;

	agent_exiting = 1;
	(void) pthread_cond_signal(&agent_cond);

	/* wait for zfs_enum_pools thread to complete */
	(void) pthread_join(g_agents_tid, NULL);

	/* drain any pending events */
	while ((event = (list_head(&agent_events))) != NULL) {
	list_remove(&agent_events, event);
	nvlist_free(event->ae_nvl);
	free(event);
	}

	list_destroy(&agent_events);

	if ((hdl = fmd_module_hdl("zfs-retire")) != NULL) {
	_zfs_retire_fini(hdl);
	fmd_hdl_unregister(hdl);
	}
	if ((hdl = fmd_module_hdl("zfs-diagnosis")) != NULL) {
	_zfs_diagnosis_fini(hdl);
	fmd_hdl_unregister(hdl);
	}

	zed_log_msg(LOG_INFO, "Add Agent: fini");
	zfs_slm_fini();

	g_zfs_hdl = NULL;
	}
	Index: vendor-sys/openzfs/dist/cmd/zed/agents/zfs_mod.c
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zed/agents/zfs_mod.c (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zed/agents/zfs_mod.c (revision 366774)
	@@ -1,956 +1,954 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2007, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2012 by Delphix. All rights reserved.
	* Copyright 2014 Nexenta Systems, Inc. All rights reserved.
	* Copyright (c) 2016, 2017, Intel Corporation.
	* Copyright (c) 2017 Open-E, Inc. All Rights Reserved.
	*/

	/*
	* ZFS syseventd module.
	*
	* file origin: openzfs/usr/src/cmd/syseventd/modules/zfs_mod/zfs_mod.c
	*
	* The purpose of this module is to identify when devices are added to the
	* system, and appropriately online or replace the affected vdevs.
	*
	* When a device is added to the system:
	*
	* 1. Search for any vdevs whose devid matches that of the newly added
	* device.
	*
	* 2. If no vdevs are found, then search for any vdevs whose udev path
	* matches that of the new device.
	*
	* 3. If no vdevs match by either method, then ignore the event.
	*
	* 4. Attempt to online the device with a flag to indicate that it should
	* be unspared when resilvering completes. If this succeeds, then the
	* same device was inserted and we should continue normally.
	*
	* 5. If the pool does not have the 'autoreplace' property set, attempt to
	* online the device again without the unspare flag, which will
	* generate a FMA fault.
	*
	* 6. If the pool has the 'autoreplace' property set, and the matching vdev
	* is a whole disk, then label the new disk and attempt a 'zpool
	* replace'.
	*
	* The module responds to EC_DEV_ADD events. The special ESC_ZFS_VDEV_CHECK
	* event indicates that a device failed to open during pool load, but the
	* autoreplace property was set. In this case, we deferred the associated
	* FMA fault until our module had a chance to process the autoreplace logic.
	* If the device could not be replaced, then the second online attempt will
	* trigger the FMA fault that we skipped earlier.
	*
	- * ZFS on Linux porting notes:
	- * Linux udev provides a disk insert for both the disk and the partition
	- *
	+ * On Linux udev provides a disk insert for both the disk and the partition.
	*/

	#include <ctype.h>
	#include <fcntl.h>
	#include <libnvpair.h>
	#include <libzfs.h>
	#include <libzutil.h>
	#include <limits.h>
	#include <stddef.h>
	#include <stdlib.h>
	#include <string.h>
	#include <syslog.h>
	#include <sys/list.h>
	#include <sys/sunddi.h>
	#include <sys/sysevent/eventdefs.h>
	#include <sys/sysevent/dev.h>
	#include <thread_pool.h>
	#include <pthread.h>
	#include <unistd.h>
	#include <errno.h>
	#include "zfs_agents.h"
	#include "../zed_log.h"

	#define DEV_BYID_PATH "/dev/disk/by-id/"
	#define DEV_BYPATH_PATH "/dev/disk/by-path/"
	#define DEV_BYVDEV_PATH "/dev/disk/by-vdev/"

	typedef void (zfs_process_func_t)(zpool_handle_t , nvlist_t *, boolean_t);

	libzfs_handle_t *g_zfshdl;
	list_t g_pool_list; /* list of unavailable pools at initialization */
	list_t g_device_list; /* list of disks with asynchronous label request */
	tpool_t *g_tpool;
	boolean_t g_enumeration_done;
	pthread_t g_zfs_tid; /* zfs_enum_pools() thread */

	typedef struct unavailpool {
	zpool_handle_t *uap_zhp;
	list_node_t uap_node;
	} unavailpool_t;

	typedef struct pendingdev {
	char pd_physpath[128];
	list_node_t pd_node;
	} pendingdev_t;

	static int
	zfs_toplevel_state(zpool_handle_t *zhp)
	{
	nvlist_t *nvroot;
	vdev_stat_t *vs;
	unsigned int c;

	verify(nvlist_lookup_nvlist(zpool_get_config(zhp, NULL),
	ZPOOL_CONFIG_VDEV_TREE, &nvroot) == 0);
	verify(nvlist_lookup_uint64_array(nvroot, ZPOOL_CONFIG_VDEV_STATS,
	(uint64_t **)&vs, &c) == 0);
	return (vs->vs_state);
	}

	static int
	zfs_unavail_pool(zpool_handle_t zhp, void data)
	{
	zed_log_msg(LOG_INFO, "zfs_unavail_pool: examining '%s' (state %d)",
	zpool_get_name(zhp), (int)zfs_toplevel_state(zhp));

	if (zfs_toplevel_state(zhp) < VDEV_STATE_DEGRADED) {
	unavailpool_t *uap;
	uap = malloc(sizeof (unavailpool_t));
	uap->uap_zhp = zhp;
	list_insert_tail((list_t *)data, uap);
	} else {
	zpool_close(zhp);
	}
	return (0);
	}

	/*
	* Two stage replace on Linux
	* since we get disk notifications
	* we can wait for partitioned disk slice to show up!
	*
	* First stage tags the disk, initiates async partitioning, and returns
	* Second stage finds the tag and proceeds to ZFS labeling/replace
	*
	* disk-add --> label-disk + tag-disk --> partition-add --> zpool_vdev_attach
	*
	* 1. physical match with no fs, no partition
	* tag it top, partition disk
	*
	* 2. physical match again, see partition and tag
	*
	*/

	/*
	* The device associated with the given vdev (either by devid or physical path)
	* has been added to the system. If 'isdisk' is set, then we only attempt a
	* replacement if it's a whole disk. This also implies that we should label the
	* disk first.
	*
	* First, we attempt to online the device (making sure to undo any spare
	* operation when finished). If this succeeds, then we're done. If it fails,
	* and the new state is VDEV_CANT_OPEN, it indicates that the device was opened,
	* but that the label was not what we expected. If the 'autoreplace' property
	* is enabled, then we relabel the disk (if specified), and attempt a 'zpool
	* replace'. If the online is successful, but the new state is something else
	* (REMOVED or FAULTED), it indicates that we're out of sync or in some sort of
	* race, and we should avoid attempting to relabel the disk.
	*
	* Also can arrive here from a ESC_ZFS_VDEV_CHECK event
	*/
	static void
	zfs_process_add(zpool_handle_t zhp, nvlist_t vdev, boolean_t labeled)
	{
	char *path;
	vdev_state_t newstate;
	nvlist_t nvroot, newvd;
	pendingdev_t *device;
	uint64_t wholedisk = 0ULL;
	uint64_t offline = 0ULL;
	uint64_t guid = 0ULL;
	char physpath = NULL, new_devid = NULL, *enc_sysfs_path = NULL;
	char rawpath[PATH_MAX], fullpath[PATH_MAX];
	char devpath[PATH_MAX];
	int ret;
	boolean_t is_dm = B_FALSE;
	boolean_t is_sd = B_FALSE;
	uint_t c;
	vdev_stat_t *vs;

	if (nvlist_lookup_string(vdev, ZPOOL_CONFIG_PATH, &path) != 0)
	return;

	/* Skip healthy disks */
	verify(nvlist_lookup_uint64_array(vdev, ZPOOL_CONFIG_VDEV_STATS,
	(uint64_t **)&vs, &c) == 0);
	if (vs->vs_state == VDEV_STATE_HEALTHY) {
	zed_log_msg(LOG_INFO, "%s: %s is already healthy, skip it.",
	__func__, path);
	return;
	}

	(void) nvlist_lookup_string(vdev, ZPOOL_CONFIG_PHYS_PATH, &physpath);
	(void) nvlist_lookup_string(vdev, ZPOOL_CONFIG_VDEV_ENC_SYSFS_PATH,
	&enc_sysfs_path);
	(void) nvlist_lookup_uint64(vdev, ZPOOL_CONFIG_WHOLE_DISK, &wholedisk);
	(void) nvlist_lookup_uint64(vdev, ZPOOL_CONFIG_OFFLINE, &offline);
	(void) nvlist_lookup_uint64(vdev, ZPOOL_CONFIG_GUID, &guid);

	if (offline)
	return; /* don't intervene if it was taken offline */

	is_dm = zfs_dev_is_dm(path);
	zed_log_msg(LOG_INFO, "zfs_process_add: pool '%s' vdev '%s', phys '%s'"
	" wholedisk %d, %s dm (guid %llu)", zpool_get_name(zhp), path,
	physpath ? physpath : "NULL", wholedisk, is_dm ? "is" : "not",
	(long long unsigned int)guid);

	/*
	* The VDEV guid is preferred for identification (gets passed in path)
	*/
	if (guid != 0) {
	(void) snprintf(fullpath, sizeof (fullpath), "%llu",
	(long long unsigned int)guid);
	} else {
	/*
	* otherwise use path sans partition suffix for whole disks
	*/
	(void) strlcpy(fullpath, path, sizeof (fullpath));
	if (wholedisk) {
	char *spath = zfs_strip_partition(fullpath);
	if (!spath) {
	zed_log_msg(LOG_INFO, "%s: Can't alloc",
	__func__);
	return;
	}

	(void) strlcpy(fullpath, spath, sizeof (fullpath));
	free(spath);
	}
	}

	/*
	* Attempt to online the device.
	*/
	if (zpool_vdev_online(zhp, fullpath,
	ZFS_ONLINE_CHECKREMOVE \| ZFS_ONLINE_UNSPARE, &newstate) == 0 &&
	(newstate == VDEV_STATE_HEALTHY \|\|
	newstate == VDEV_STATE_DEGRADED)) {
	zed_log_msg(LOG_INFO, " zpool_vdev_online: vdev %s is %s",
	fullpath, (newstate == VDEV_STATE_HEALTHY) ?
	"HEALTHY" : "DEGRADED");
	return;
	}

	/*
	* vdev_id alias rule for using scsi_debug devices (FMA automated
	* testing)
	*/
	if (physpath != NULL && strcmp("scsidebug", physpath) == 0)
	is_sd = B_TRUE;

	/*
	* If the pool doesn't have the autoreplace property set, then use
	* vdev online to trigger a FMA fault by posting an ereport.
	*/
	if (!zpool_get_prop_int(zhp, ZPOOL_PROP_AUTOREPLACE, NULL) \|\|
	!(wholedisk \|\| is_dm) \|\| (physpath == NULL)) {
	(void) zpool_vdev_online(zhp, fullpath, ZFS_ONLINE_FORCEFAULT,
	&newstate);
	zed_log_msg(LOG_INFO, "Pool's autoreplace is not enabled or "
	"not a whole disk for '%s'", fullpath);
	return;
	}

	/*
	* Convert physical path into its current device node. Rawpath
	* needs to be /dev/disk/by-vdev for a scsi_debug device since
	* /dev/disk/by-path will not be present.
	*/
	(void) snprintf(rawpath, sizeof (rawpath), "%s%s",
	is_sd ? DEV_BYVDEV_PATH : DEV_BYPATH_PATH, physpath);

	if (realpath(rawpath, devpath) == NULL && !is_dm) {
	zed_log_msg(LOG_INFO, " realpath: %s failed (%s)",
	rawpath, strerror(errno));

	(void) zpool_vdev_online(zhp, fullpath, ZFS_ONLINE_FORCEFAULT,
	&newstate);

	zed_log_msg(LOG_INFO, " zpool_vdev_online: %s FORCEFAULT (%s)",
	fullpath, libzfs_error_description(g_zfshdl));
	return;
	}

	/* Only autoreplace bad disks */
	if ((vs->vs_state != VDEV_STATE_DEGRADED) &&
	(vs->vs_state != VDEV_STATE_FAULTED) &&
	(vs->vs_state != VDEV_STATE_CANT_OPEN)) {
	return;
	}

	nvlist_lookup_string(vdev, "new_devid", &new_devid);

	if (is_dm) {
	/* Don't label device mapper or multipath disks. */
	} else if (!labeled) {
	/*
	* we're auto-replacing a raw disk, so label it first
	*/
	char *leafname;

	/*
	* If this is a request to label a whole disk, then attempt to
	* write out the label. Before we can label the disk, we need
	* to map the physical string that was matched on to the under
	* lying device node.
	*
	* If any part of this process fails, then do a force online
	* to trigger a ZFS fault for the device (and any hot spare
	* replacement).
	*/
	leafname = strrchr(devpath, '/') + 1;

	/*
	* If this is a request to label a whole disk, then attempt to
	* write out the label.
	*/
	if (zpool_label_disk(g_zfshdl, zhp, leafname) != 0) {
	zed_log_msg(LOG_INFO, " zpool_label_disk: could not "
	"label '%s' (%s)", leafname,
	libzfs_error_description(g_zfshdl));

	(void) zpool_vdev_online(zhp, fullpath,
	ZFS_ONLINE_FORCEFAULT, &newstate);
	return;
	}

	/*
	* The disk labeling is asynchronous on Linux. Just record
	* this label request and return as there will be another
	* disk add event for the partition after the labeling is
	* completed.
	*/
	device = malloc(sizeof (pendingdev_t));
	(void) strlcpy(device->pd_physpath, physpath,
	sizeof (device->pd_physpath));
	list_insert_tail(&g_device_list, device);

	zed_log_msg(LOG_INFO, " zpool_label_disk: async '%s' (%llu)",
	leafname, (u_longlong_t)guid);

	return; /* resumes at EC_DEV_ADD.ESC_DISK for partition */

	} else /* labeled */ {
	boolean_t found = B_FALSE;
	/*
	* match up with request above to label the disk
	*/
	for (device = list_head(&g_device_list); device != NULL;
	device = list_next(&g_device_list, device)) {
	if (strcmp(physpath, device->pd_physpath) == 0) {
	list_remove(&g_device_list, device);
	free(device);
	found = B_TRUE;
	break;
	}
	zed_log_msg(LOG_INFO, "zpool_label_disk: %s != %s",
	physpath, device->pd_physpath);
	}
	if (!found) {
	/* unexpected partition slice encountered */
	zed_log_msg(LOG_INFO, "labeled disk %s unexpected here",
	fullpath);
	(void) zpool_vdev_online(zhp, fullpath,
	ZFS_ONLINE_FORCEFAULT, &newstate);
	return;
	}

	zed_log_msg(LOG_INFO, " zpool_label_disk: resume '%s' (%llu)",
	physpath, (u_longlong_t)guid);

	(void) snprintf(devpath, sizeof (devpath), "%s%s",
	DEV_BYID_PATH, new_devid);
	}

	/*
	* Construct the root vdev to pass to zpool_vdev_attach(). While adding
	* the entire vdev structure is harmless, we construct a reduced set of
	* path/physpath/wholedisk to keep it simple.
	*/
	if (nvlist_alloc(&nvroot, NV_UNIQUE_NAME, 0) != 0) {
	zed_log_msg(LOG_WARNING, "zfs_mod: nvlist_alloc out of memory");
	return;
	}
	if (nvlist_alloc(&newvd, NV_UNIQUE_NAME, 0) != 0) {
	zed_log_msg(LOG_WARNING, "zfs_mod: nvlist_alloc out of memory");
	nvlist_free(nvroot);
	return;
	}

	if (nvlist_add_string(newvd, ZPOOL_CONFIG_TYPE, VDEV_TYPE_DISK) != 0 \|\|
	nvlist_add_string(newvd, ZPOOL_CONFIG_PATH, path) != 0 \|\|
	nvlist_add_string(newvd, ZPOOL_CONFIG_DEVID, new_devid) != 0 \|\|
	(physpath != NULL && nvlist_add_string(newvd,
	ZPOOL_CONFIG_PHYS_PATH, physpath) != 0) \|\|
	(enc_sysfs_path != NULL && nvlist_add_string(newvd,
	ZPOOL_CONFIG_VDEV_ENC_SYSFS_PATH, enc_sysfs_path) != 0) \|\|
	nvlist_add_uint64(newvd, ZPOOL_CONFIG_WHOLE_DISK, wholedisk) != 0 \|\|
	nvlist_add_string(nvroot, ZPOOL_CONFIG_TYPE, VDEV_TYPE_ROOT) != 0 \|\|
	nvlist_add_nvlist_array(nvroot, ZPOOL_CONFIG_CHILDREN, &newvd,
	1) != 0) {
	zed_log_msg(LOG_WARNING, "zfs_mod: unable to add nvlist pairs");
	nvlist_free(newvd);
	nvlist_free(nvroot);
	return;
	}

	nvlist_free(newvd);

	/*
	* Wait for udev to verify the links exist, then auto-replace
	* the leaf disk at same physical location.
	*/
	if (zpool_label_disk_wait(path, 3000) != 0) {
	zed_log_msg(LOG_WARNING, "zfs_mod: expected replacement "
	"disk %s is missing", path);
	nvlist_free(nvroot);
	return;
	}

	ret = zpool_vdev_attach(zhp, fullpath, path, nvroot, B_TRUE, B_FALSE);

	zed_log_msg(LOG_INFO, " zpool_vdev_replace: %s with %s (%s)",
	fullpath, path, (ret == 0) ? "no errors" :
	libzfs_error_description(g_zfshdl));

	nvlist_free(nvroot);
	}

	/*
	* Utility functions to find a vdev matching given criteria.
	*/
	typedef struct dev_data {
	const char *dd_compare;
	const char *dd_prop;
	zfs_process_func_t dd_func;
	boolean_t dd_found;
	boolean_t dd_islabeled;
	uint64_t dd_pool_guid;
	uint64_t dd_vdev_guid;
	const char *dd_new_devid;
	} dev_data_t;

	static void
	zfs_iter_vdev(zpool_handle_t zhp, nvlist_t nvl, void *data)
	{
	dev_data_t *dp = data;
	char *path = NULL;
	uint_t c, children;
	nvlist_t **child;

	/*
	* First iterate over any children.
	*/
	if (nvlist_lookup_nvlist_array(nvl, ZPOOL_CONFIG_CHILDREN,
	&child, &children) == 0) {
	for (c = 0; c < children; c++)
	zfs_iter_vdev(zhp, child[c], data);
	}

	/*
	* Iterate over any spares and cache devices
	*/
	if (nvlist_lookup_nvlist_array(nvl, ZPOOL_CONFIG_SPARES,
	&child, &children) == 0) {
	for (c = 0; c < children; c++)
	zfs_iter_vdev(zhp, child[c], data);
	}
	if (nvlist_lookup_nvlist_array(nvl, ZPOOL_CONFIG_L2CACHE,
	&child, &children) == 0) {
	for (c = 0; c < children; c++)
	zfs_iter_vdev(zhp, child[c], data);
	}

	/* once a vdev was matched and processed there is nothing left to do */
	if (dp->dd_found)
	return;

	/*
	* Match by GUID if available otherwise fallback to devid or physical
	*/
	if (dp->dd_vdev_guid != 0) {
	uint64_t guid;

	if (nvlist_lookup_uint64(nvl, ZPOOL_CONFIG_GUID,
	&guid) != 0 \|\| guid != dp->dd_vdev_guid) {
	return;
	}
	zed_log_msg(LOG_INFO, " zfs_iter_vdev: matched on %llu", guid);
	dp->dd_found = B_TRUE;

	} else if (dp->dd_compare != NULL) {
	/*
	* NOTE: On Linux there is an event for partition, so unlike
	* illumos, substring matching is not required to accommodate
	* the partition suffix. An exact match will be present in
	* the dp->dd_compare value.
	*/
	if (nvlist_lookup_string(nvl, dp->dd_prop, &path) != 0 \|\|
	strcmp(dp->dd_compare, path) != 0)
	return;

	zed_log_msg(LOG_INFO, " zfs_iter_vdev: matched %s on %s",
	dp->dd_prop, path);
	dp->dd_found = B_TRUE;

	/* pass the new devid for use by replacing code */
	if (dp->dd_new_devid != NULL) {
	(void) nvlist_add_string(nvl, "new_devid",
	dp->dd_new_devid);
	}
	}

	(dp->dd_func)(zhp, nvl, dp->dd_islabeled);
	}

	static void
	zfs_enable_ds(void *arg)
	{
	unavailpool_t pool = (unavailpool_t )arg;

	(void) zpool_enable_datasets(pool->uap_zhp, NULL, 0);
	zpool_close(pool->uap_zhp);
	free(pool);
	}

	static int
	zfs_iter_pool(zpool_handle_t zhp, void data)
	{
	nvlist_t config, nvl;
	dev_data_t *dp = data;
	uint64_t pool_guid;
	unavailpool_t *pool;

	zed_log_msg(LOG_INFO, "zfs_iter_pool: evaluating vdevs on %s (by %s)",
	zpool_get_name(zhp), dp->dd_vdev_guid ? "GUID" : dp->dd_prop);

	/*
	* For each vdev in this pool, look for a match to apply dd_func
	*/
	if ((config = zpool_get_config(zhp, NULL)) != NULL) {
	if (dp->dd_pool_guid == 0 \|\|
	(nvlist_lookup_uint64(config, ZPOOL_CONFIG_POOL_GUID,
	&pool_guid) == 0 && pool_guid == dp->dd_pool_guid)) {
	(void) nvlist_lookup_nvlist(config,
	ZPOOL_CONFIG_VDEV_TREE, &nvl);
	zfs_iter_vdev(zhp, nvl, data);
	}
	}

	/*
	* if this pool was originally unavailable,
	* then enable its datasets asynchronously
	*/
	if (g_enumeration_done) {
	for (pool = list_head(&g_pool_list); pool != NULL;
	pool = list_next(&g_pool_list, pool)) {

	if (strcmp(zpool_get_name(zhp),
	zpool_get_name(pool->uap_zhp)))
	continue;
	if (zfs_toplevel_state(zhp) >= VDEV_STATE_DEGRADED) {
	list_remove(&g_pool_list, pool);
	(void) tpool_dispatch(g_tpool, zfs_enable_ds,
	pool);
	break;
	}
	}
	}

	zpool_close(zhp);
	return (dp->dd_found); /* cease iteration after a match */
	}

	/*
	* Given a physical device location, iterate over all
	* (pool, vdev) pairs which correspond to that location.
	*/
	static boolean_t
	devphys_iter(const char physical, const char devid, zfs_process_func_t func,
	boolean_t is_slice)
	{
	dev_data_t data = { 0 };

	data.dd_compare = physical;
	data.dd_func = func;
	data.dd_prop = ZPOOL_CONFIG_PHYS_PATH;
	data.dd_found = B_FALSE;
	data.dd_islabeled = is_slice;
	data.dd_new_devid = devid; /* used by auto replace code */

	(void) zpool_iter(g_zfshdl, zfs_iter_pool, &data);

	return (data.dd_found);
	}

	/*
	* Given a device identifier, find any vdevs with a matching devid.
	* On Linux we can match devid directly which is always a whole disk.
	*/
	static boolean_t
	devid_iter(const char *devid, zfs_process_func_t func, boolean_t is_slice)
	{
	dev_data_t data = { 0 };

	data.dd_compare = devid;
	data.dd_func = func;
	data.dd_prop = ZPOOL_CONFIG_DEVID;
	data.dd_found = B_FALSE;
	data.dd_islabeled = is_slice;
	data.dd_new_devid = devid;

	(void) zpool_iter(g_zfshdl, zfs_iter_pool, &data);

	return (data.dd_found);
	}

	/*
	* Handle a EC_DEV_ADD.ESC_DISK event.
	*
	* illumos
	* Expects: DEV_PHYS_PATH string in schema
	* Matches: vdev's ZPOOL_CONFIG_PHYS_PATH or ZPOOL_CONFIG_DEVID
	*
	* path: '/dev/dsk/c0t1d0s0' (persistent)
	* devid: 'id1,sd@SATA_____Hitachi_HDS72101______JP2940HZ3H74MC/a'
	* phys_path: '/pci@0,0/pci103c,1609@11/disk@1,0:a'
	*
	* linux
	* provides: DEV_PHYS_PATH and DEV_IDENTIFIER strings in schema
	* Matches: vdev's ZPOOL_CONFIG_PHYS_PATH or ZPOOL_CONFIG_DEVID
	*
	* path: '/dev/sdc1' (not persistent)
	* devid: 'ata-SAMSUNG_HD204UI_S2HGJD2Z805891-part1'
	* phys_path: 'pci-0000:04:00.0-sas-0x4433221106000000-lun-0'
	*/
	static int
	zfs_deliver_add(nvlist_t *nvl, boolean_t is_lofi)
	{
	char devpath = NULL, devid;
	boolean_t is_slice;

	/*
	* Expecting a devid string and an optional physical location
	*/
	if (nvlist_lookup_string(nvl, DEV_IDENTIFIER, &devid) != 0)
	return (-1);

	(void) nvlist_lookup_string(nvl, DEV_PHYS_PATH, &devpath);

	is_slice = (nvlist_lookup_boolean(nvl, DEV_IS_PART) == 0);

	zed_log_msg(LOG_INFO, "zfs_deliver_add: adding %s (%s) (is_slice %d)",
	devid, devpath ? devpath : "NULL", is_slice);

	/*
	* Iterate over all vdevs looking for a match in the following order:
	* 1. ZPOOL_CONFIG_DEVID (identifies the unique disk)
	* 2. ZPOOL_CONFIG_PHYS_PATH (identifies disk physical location).
	*
	* For disks, we only want to pay attention to vdevs marked as whole
	* disks or are a multipath device.
	*/
	if (!devid_iter(devid, zfs_process_add, is_slice) && devpath != NULL)
	(void) devphys_iter(devpath, devid, zfs_process_add, is_slice);

	return (0);
	}

	/*
	* Called when we receive a VDEV_CHECK event, which indicates a device could not
	* be opened during initial pool open, but the autoreplace property was set on
	* the pool. In this case, we treat it as if it were an add event.
	*/
	static int
	zfs_deliver_check(nvlist_t *nvl)
	{
	dev_data_t data = { 0 };

	if (nvlist_lookup_uint64(nvl, ZFS_EV_POOL_GUID,
	&data.dd_pool_guid) != 0 \|\|
	nvlist_lookup_uint64(nvl, ZFS_EV_VDEV_GUID,
	&data.dd_vdev_guid) != 0 \|\|
	data.dd_vdev_guid == 0)
	return (0);

	zed_log_msg(LOG_INFO, "zfs_deliver_check: pool '%llu', vdev %llu",
	data.dd_pool_guid, data.dd_vdev_guid);

	data.dd_func = zfs_process_add;

	(void) zpool_iter(g_zfshdl, zfs_iter_pool, &data);

	return (0);
	}

	static int
	zfsdle_vdev_online(zpool_handle_t zhp, void data)
	{
	char *devname = data;
	boolean_t avail_spare, l2cache;
	nvlist_t *tgt;
	int error;

	zed_log_msg(LOG_INFO, "zfsdle_vdev_online: searching for '%s' in '%s'",
	devname, zpool_get_name(zhp));

	if ((tgt = zpool_find_vdev_by_physpath(zhp, devname,
	&avail_spare, &l2cache, NULL)) != NULL) {
	char *path, fullpath[MAXPATHLEN];
	uint64_t wholedisk;

	error = nvlist_lookup_string(tgt, ZPOOL_CONFIG_PATH, &path);
	if (error) {
	zpool_close(zhp);
	return (0);
	}

	error = nvlist_lookup_uint64(tgt, ZPOOL_CONFIG_WHOLE_DISK,
	&wholedisk);
	if (error)
	wholedisk = 0;

	if (wholedisk) {
	path = strrchr(path, '/');
	if (path != NULL) {
	path = zfs_strip_partition(path + 1);
	if (path == NULL) {
	zpool_close(zhp);
	return (0);
	}
	} else {
	zpool_close(zhp);
	return (0);
	}

	(void) strlcpy(fullpath, path, sizeof (fullpath));
	free(path);

	/*
	* We need to reopen the pool associated with this
	* device so that the kernel can update the size of
	* the expanded device. When expanding there is no
	* need to restart the scrub from the beginning.
	*/
	boolean_t scrub_restart = B_FALSE;
	(void) zpool_reopen_one(zhp, &scrub_restart);
	} else {
	(void) strlcpy(fullpath, path, sizeof (fullpath));
	}

	if (zpool_get_prop_int(zhp, ZPOOL_PROP_AUTOEXPAND, NULL)) {
	vdev_state_t newstate;

	if (zpool_get_state(zhp) != POOL_STATE_UNAVAIL) {
	error = zpool_vdev_online(zhp, fullpath, 0,
	&newstate);
	zed_log_msg(LOG_INFO, "zfsdle_vdev_online: "
	"setting device '%s' to ONLINE state "
	"in pool '%s': %d", fullpath,
	zpool_get_name(zhp), error);
	}
	}
	zpool_close(zhp);
	return (1);
	}
	zpool_close(zhp);
	return (0);
	}

	/*
	* This function handles the ESC_DEV_DLE device change event. Use the
	* provided vdev guid when looking up a disk or partition, when the guid
	* is not present assume the entire disk is owned by ZFS and append the
	* expected -part1 partition information then lookup by physical path.
	*/
	static int
	zfs_deliver_dle(nvlist_t *nvl)
	{
	char *devname, name[MAXPATHLEN];
	uint64_t guid;

	if (nvlist_lookup_uint64(nvl, ZFS_EV_VDEV_GUID, &guid) == 0) {
	sprintf(name, "%llu", (u_longlong_t)guid);
	} else if (nvlist_lookup_string(nvl, DEV_PHYS_PATH, &devname) == 0) {
	strlcpy(name, devname, MAXPATHLEN);
	zfs_append_partition(name, MAXPATHLEN);
	} else {
	zed_log_msg(LOG_INFO, "zfs_deliver_dle: no guid or physpath");
	}

	if (zpool_iter(g_zfshdl, zfsdle_vdev_online, name) != 1) {
	zed_log_msg(LOG_INFO, "zfs_deliver_dle: device '%s' not "
	"found", name);
	return (1);
	}

	return (0);
	}

	/*
	* syseventd daemon module event handler
	*
	* Handles syseventd daemon zfs device related events:
	*
	* EC_DEV_ADD.ESC_DISK
	* EC_DEV_STATUS.ESC_DEV_DLE
	* EC_ZFS.ESC_ZFS_VDEV_CHECK
	*
	* Note: assumes only one thread active at a time (not thread safe)
	*/
	static int
	zfs_slm_deliver_event(const char class, const char subclass, nvlist_t *nvl)
	{
	int ret;
	boolean_t is_lofi = B_FALSE, is_check = B_FALSE, is_dle = B_FALSE;

	if (strcmp(class, EC_DEV_ADD) == 0) {
	/*
	* We're mainly interested in disk additions, but we also listen
	* for new loop devices, to allow for simplified testing.
	*/
	if (strcmp(subclass, ESC_DISK) == 0)
	is_lofi = B_FALSE;
	else if (strcmp(subclass, ESC_LOFI) == 0)
	is_lofi = B_TRUE;
	else
	return (0);

	is_check = B_FALSE;
	} else if (strcmp(class, EC_ZFS) == 0 &&
	strcmp(subclass, ESC_ZFS_VDEV_CHECK) == 0) {
	/*
	* This event signifies that a device failed to open
	* during pool load, but the 'autoreplace' property was
	* set, so we should pretend it's just been added.
	*/
	is_check = B_TRUE;
	} else if (strcmp(class, EC_DEV_STATUS) == 0 &&
	strcmp(subclass, ESC_DEV_DLE) == 0) {
	is_dle = B_TRUE;
	} else {
	return (0);
	}

	if (is_dle)
	ret = zfs_deliver_dle(nvl);
	else if (is_check)
	ret = zfs_deliver_check(nvl);
	else
	ret = zfs_deliver_add(nvl, is_lofi);

	return (ret);
	}

	/ARGSUSED/
	static void *
	zfs_enum_pools(void *arg)
	{
	(void) zpool_iter(g_zfshdl, zfs_unavail_pool, (void *)&g_pool_list);
	/*
	* Linux - instead of using a thread pool, each list entry
	* will spawn a thread when an unavailable pool transitions
	* to available. zfs_slm_fini will wait for these threads.
	*/
	g_enumeration_done = B_TRUE;
	return (NULL);
	}

	/*
	* called from zed daemon at startup
	*
	* sent messages from zevents or udev monitor
	*
	* For now, each agent has its own libzfs instance
	*/
	int
	zfs_slm_init()
	{
	if ((g_zfshdl = libzfs_init()) == NULL)
	return (-1);

	/*
	* collect a list of unavailable pools (asynchronously,
	* since this can take a while)
	*/
	list_create(&g_pool_list, sizeof (struct unavailpool),
	offsetof(struct unavailpool, uap_node));

	if (pthread_create(&g_zfs_tid, NULL, zfs_enum_pools, NULL) != 0) {
	list_destroy(&g_pool_list);
	libzfs_fini(g_zfshdl);
	return (-1);
	}

	list_create(&g_device_list, sizeof (struct pendingdev),
	offsetof(struct pendingdev, pd_node));

	return (0);
	}

	void
	zfs_slm_fini()
	{
	unavailpool_t *pool;
	pendingdev_t *device;

	/* wait for zfs_enum_pools thread to complete */
	(void) pthread_join(g_zfs_tid, NULL);
	/* destroy the thread pool */
	if (g_tpool != NULL) {
	tpool_wait(g_tpool);
	tpool_destroy(g_tpool);
	}

	while ((pool = (list_head(&g_pool_list))) != NULL) {
	list_remove(&g_pool_list, pool);
	zpool_close(pool->uap_zhp);
	free(pool);
	}
	list_destroy(&g_pool_list);

	while ((device = (list_head(&g_device_list))) != NULL) {
	list_remove(&g_device_list, device);
	free(device);
	}
	list_destroy(&g_device_list);

	libzfs_fini(g_zfshdl);
	}

	void
	zfs_slm_event(const char class, const char subclass, nvlist_t *nvl)
	{
	zed_log_msg(LOG_INFO, "zfs_slm_event: %s.%s", class, subclass);
	(void) zfs_slm_deliver_event(class, subclass, nvl);
	}
	Index: vendor-sys/openzfs/dist/cmd/zed/agents/zfs_retire.c
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zed/agents/zfs_retire.c (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zed/agents/zfs_retire.c (revision 366774)
	@@ -1,557 +1,557 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2006, 2010, Oracle and/or its affiliates. All rights reserved.
	*
	* Copyright (c) 2016, Intel Corporation.
	* Copyright (c) 2018, loli10K <ezomori.nozomu@gmail.com>
	*/

	/*
	* The ZFS retire agent is responsible for managing hot spares across all pools.
	* When we see a device fault or a device removal, we try to open the associated
	* pool and look for any hot spares. We iterate over any available hot spares
	* and attempt a 'zpool replace' for each one.
	*
	* For vdevs diagnosed as faulty, the agent is also responsible for proactively
	* marking the vdev FAULTY (for I/O errors) or DEGRADED (for checksum errors).
	*/

	#include <sys/fs/zfs.h>
	#include <sys/fm/protocol.h>
	#include <sys/fm/fs/zfs.h>
	#include <libzfs.h>
	#include <string.h>

	#include "zfs_agents.h"
	#include "fmd_api.h"


	typedef struct zfs_retire_repaired {
	struct zfs_retire_repaired *zrr_next;
	uint64_t zrr_pool;
	uint64_t zrr_vdev;
	} zfs_retire_repaired_t;

	typedef struct zfs_retire_data {
	libzfs_handle_t *zrd_hdl;
	zfs_retire_repaired_t *zrd_repaired;
	} zfs_retire_data_t;

	static void
	zfs_retire_clear_data(fmd_hdl_t hdl, zfs_retire_data_t zdp)
	{
	zfs_retire_repaired_t *zrp;

	while ((zrp = zdp->zrd_repaired) != NULL) {
	zdp->zrd_repaired = zrp->zrr_next;
	fmd_hdl_free(hdl, zrp, sizeof (zfs_retire_repaired_t));
	}
	}

	/*
	* Find a pool with a matching GUID.
	*/
	typedef struct find_cbdata {
	uint64_t cb_guid;
	zpool_handle_t *cb_zhp;
	nvlist_t *cb_vdev;
	} find_cbdata_t;

	static int
	find_pool(zpool_handle_t zhp, void data)
	{
	find_cbdata_t *cbp = data;

	if (cbp->cb_guid ==
	zpool_get_prop_int(zhp, ZPOOL_PROP_GUID, NULL)) {
	cbp->cb_zhp = zhp;
	return (1);
	}

	zpool_close(zhp);
	return (0);
	}

	/*
	* Find a vdev within a tree with a matching GUID.
	*/
	static nvlist_t *
	find_vdev(libzfs_handle_t zhdl, nvlist_t nv, uint64_t search_guid)
	{
	uint64_t guid;
	nvlist_t **child;
	uint_t c, children;
	nvlist_t *ret;

	if (nvlist_lookup_uint64(nv, ZPOOL_CONFIG_GUID, &guid) == 0 &&
	guid == search_guid) {
	fmd_hdl_debug(fmd_module_hdl("zfs-retire"),
	"matched vdev %llu", guid);
	return (nv);
	}

	if (nvlist_lookup_nvlist_array(nv, ZPOOL_CONFIG_CHILDREN,
	&child, &children) != 0)
	return (NULL);

	for (c = 0; c < children; c++) {
	if ((ret = find_vdev(zhdl, child[c], search_guid)) != NULL)
	return (ret);
	}

	if (nvlist_lookup_nvlist_array(nv, ZPOOL_CONFIG_L2CACHE,
	&child, &children) != 0)
	return (NULL);

	for (c = 0; c < children; c++) {
	if ((ret = find_vdev(zhdl, child[c], search_guid)) != NULL)
	return (ret);
	}

	if (nvlist_lookup_nvlist_array(nv, ZPOOL_CONFIG_SPARES,
	&child, &children) != 0)
	return (NULL);

	for (c = 0; c < children; c++) {
	if ((ret = find_vdev(zhdl, child[c], search_guid)) != NULL)
	return (ret);
	}

	return (NULL);
	}

	/*
	* Given a (pool, vdev) GUID pair, find the matching pool and vdev.
	*/
	static zpool_handle_t *
	find_by_guid(libzfs_handle_t *zhdl, uint64_t pool_guid, uint64_t vdev_guid,
	nvlist_t **vdevp)
	{
	find_cbdata_t cb;
	zpool_handle_t *zhp;
	nvlist_t config, nvroot;

	/*
	* Find the corresponding pool and make sure the vdev still exists.
	*/
	cb.cb_guid = pool_guid;
	if (zpool_iter(zhdl, find_pool, &cb) != 1)
	return (NULL);

	zhp = cb.cb_zhp;
	config = zpool_get_config(zhp, NULL);
	if (nvlist_lookup_nvlist(config, ZPOOL_CONFIG_VDEV_TREE,
	&nvroot) != 0) {
	zpool_close(zhp);
	return (NULL);
	}

	if (vdev_guid != 0) {
	if ((*vdevp = find_vdev(zhdl, nvroot, vdev_guid)) == NULL) {
	zpool_close(zhp);
	return (NULL);
	}
	}

	return (zhp);
	}

	/*
	* Given a vdev, attempt to replace it with every known spare until one
	* succeeds or we run out of devices to try.
	* Return whether we were successful or not in replacing the device.
	*/
	static boolean_t
	replace_with_spare(fmd_hdl_t hdl, zpool_handle_t zhp, nvlist_t *vdev)
	{
	nvlist_t config, nvroot, *replacement;
	nvlist_t **spares;
	uint_t s, nspares;
	char *dev_name;
	zprop_source_t source;
	int ashift;

	config = zpool_get_config(zhp, NULL);
	if (nvlist_lookup_nvlist(config, ZPOOL_CONFIG_VDEV_TREE,
	&nvroot) != 0)
	return (B_FALSE);

	/*
	* Find out if there are any hot spares available in the pool.
	*/
	if (nvlist_lookup_nvlist_array(nvroot, ZPOOL_CONFIG_SPARES,
	&spares, &nspares) != 0)
	return (B_FALSE);

	/*
	* lookup "ashift" pool property, we may need it for the replacement
	*/
	ashift = zpool_get_prop_int(zhp, ZPOOL_PROP_ASHIFT, &source);

	replacement = fmd_nvl_alloc(hdl, FMD_SLEEP);

	(void) nvlist_add_string(replacement, ZPOOL_CONFIG_TYPE,
	VDEV_TYPE_ROOT);

	dev_name = zpool_vdev_name(NULL, zhp, vdev, B_FALSE);

	/*
	* Try to replace each spare, ending when we successfully
	* replace it.
	*/
	for (s = 0; s < nspares; s++) {
	char *spare_name;

	if (nvlist_lookup_string(spares[s], ZPOOL_CONFIG_PATH,
	&spare_name) != 0)
	continue;

	/* if set, add the "ashift" pool property to the spare nvlist */
	if (source != ZPROP_SRC_DEFAULT)
	(void) nvlist_add_uint64(spares[s],
	ZPOOL_CONFIG_ASHIFT, ashift);

	(void) nvlist_add_nvlist_array(replacement,
	ZPOOL_CONFIG_CHILDREN, &spares[s], 1);

	fmd_hdl_debug(hdl, "zpool_vdev_replace '%s' with spare '%s'",
	dev_name, basename(spare_name));

	if (zpool_vdev_attach(zhp, dev_name, spare_name,
	replacement, B_TRUE, B_FALSE) == 0) {
	free(dev_name);
	nvlist_free(replacement);
	return (B_TRUE);
	}
	}

	free(dev_name);
	nvlist_free(replacement);

	return (B_FALSE);
	}

	/*
	* Repair this vdev if we had diagnosed a 'fault.fs.zfs.device' and
	* ASRU is now usable. ZFS has found the device to be present and
	* functioning.
	*/
	/ARGSUSED/
	static void
	zfs_vdev_repair(fmd_hdl_t hdl, nvlist_t nvl)
	{
	zfs_retire_data_t *zdp = fmd_hdl_getspecific(hdl);
	zfs_retire_repaired_t *zrp;
	uint64_t pool_guid, vdev_guid;
	if (nvlist_lookup_uint64(nvl, FM_EREPORT_PAYLOAD_ZFS_POOL_GUID,
	&pool_guid) != 0 \|\| nvlist_lookup_uint64(nvl,
	FM_EREPORT_PAYLOAD_ZFS_VDEV_GUID, &vdev_guid) != 0)
	return;

	/*
	* Before checking the state of the ASRU, go through and see if we've
	* already made an attempt to repair this ASRU. This list is cleared
	* whenever we receive any kind of list event, and is designed to
	* prevent us from generating a feedback loop when we attempt repairs
	* against a faulted pool. The problem is that checking the unusable
	* state of the ASRU can involve opening the pool, which can post
	* statechange events but otherwise leave the pool in the faulted
	* state. This list allows us to detect when a statechange event is
	* due to our own request.
	*/
	for (zrp = zdp->zrd_repaired; zrp != NULL; zrp = zrp->zrr_next) {
	if (zrp->zrr_pool == pool_guid &&
	zrp->zrr_vdev == vdev_guid)
	return;
	}

	zrp = fmd_hdl_alloc(hdl, sizeof (zfs_retire_repaired_t), FMD_SLEEP);
	zrp->zrr_next = zdp->zrd_repaired;
	zrp->zrr_pool = pool_guid;
	zrp->zrr_vdev = vdev_guid;
	zdp->zrd_repaired = zrp;

	fmd_hdl_debug(hdl, "marking repaired vdev %llu on pool %llu",
	vdev_guid, pool_guid);
	}

	/ARGSUSED/
	static void
	zfs_retire_recv(fmd_hdl_t hdl, fmd_event_t ep, nvlist_t *nvl,
	const char *class)
	{
	uint64_t pool_guid, vdev_guid;
	zpool_handle_t *zhp;
	nvlist_t resource, fault;
	nvlist_t **faults;
	uint_t f, nfaults;
	zfs_retire_data_t *zdp = fmd_hdl_getspecific(hdl);
	libzfs_handle_t *zhdl = zdp->zrd_hdl;
	boolean_t fault_device, degrade_device;
	boolean_t is_repair;
	char *scheme;
	nvlist_t *vdev = NULL;
	char *uuid;
	int repair_done = 0;
	boolean_t retire;
	boolean_t is_disk;
	vdev_aux_t aux;
	uint64_t state = 0;

	fmd_hdl_debug(hdl, "zfs_retire_recv: '%s'", class);

	nvlist_lookup_uint64(nvl, FM_EREPORT_PAYLOAD_ZFS_VDEV_STATE, &state);

	/*
	* If this is a resource notifying us of device removal then simply
	* check for an available spare and continue unless the device is a
	* l2arc vdev, in which case we just offline it.
	*/
	if (strcmp(class, "resource.fs.zfs.removed") == 0 \|\|
	(strcmp(class, "resource.fs.zfs.statechange") == 0 &&
	state == VDEV_STATE_REMOVED)) {
	char *devtype;
	char *devname;

	if (nvlist_lookup_uint64(nvl, FM_EREPORT_PAYLOAD_ZFS_POOL_GUID,
	&pool_guid) != 0 \|\|
	nvlist_lookup_uint64(nvl, FM_EREPORT_PAYLOAD_ZFS_VDEV_GUID,
	&vdev_guid) != 0)
	return;

	if ((zhp = find_by_guid(zhdl, pool_guid, vdev_guid,
	&vdev)) == NULL)
	return;

	devname = zpool_vdev_name(NULL, zhp, vdev, B_FALSE);

	/* Can't replace l2arc with a spare: offline the device */
	if (nvlist_lookup_string(nvl, FM_EREPORT_PAYLOAD_ZFS_VDEV_TYPE,
	&devtype) == 0 && strcmp(devtype, VDEV_TYPE_L2CACHE) == 0) {
	fmd_hdl_debug(hdl, "zpool_vdev_offline '%s'", devname);
	zpool_vdev_offline(zhp, devname, B_TRUE);
	} else if (!fmd_prop_get_int32(hdl, "spare_on_remove") \|\|
	replace_with_spare(hdl, zhp, vdev) == B_FALSE) {
	/* Could not handle with spare */
	fmd_hdl_debug(hdl, "no spare for '%s'", devname);
	}

	free(devname);
	zpool_close(zhp);
	return;
	}

	if (strcmp(class, FM_LIST_RESOLVED_CLASS) == 0)
	return;

	/*
	- * Note: on zfsonlinux statechange events are more than just
	+ * Note: on Linux statechange events are more than just
	* healthy ones so we need to confirm the actual state value.
	*/
	if (strcmp(class, "resource.fs.zfs.statechange") == 0 &&
	state == VDEV_STATE_HEALTHY) {
	zfs_vdev_repair(hdl, nvl);
	return;
	}
	if (strcmp(class, "sysevent.fs.zfs.vdev_remove") == 0) {
	zfs_vdev_repair(hdl, nvl);
	return;
	}

	zfs_retire_clear_data(hdl, zdp);

	if (strcmp(class, FM_LIST_REPAIRED_CLASS) == 0)
	is_repair = B_TRUE;
	else
	is_repair = B_FALSE;

	/*
	* We subscribe to zfs faults as well as all repair events.
	*/
	if (nvlist_lookup_nvlist_array(nvl, FM_SUSPECT_FAULT_LIST,
	&faults, &nfaults) != 0)
	return;

	for (f = 0; f < nfaults; f++) {
	fault = faults[f];

	fault_device = B_FALSE;
	degrade_device = B_FALSE;
	is_disk = B_FALSE;

	if (nvlist_lookup_boolean_value(fault, FM_SUSPECT_RETIRE,
	&retire) == 0 && retire == 0)
	continue;

	/*
	* While we subscribe to fault.fs.zfs.*, we only take action
	* for faults targeting a specific vdev (open failure or SERD
	* failure). We also subscribe to fault.io.* events, so that
	* faulty disks will be faulted in the ZFS configuration.
	*/
	if (fmd_nvl_class_match(hdl, fault, "fault.fs.zfs.vdev.io")) {
	fault_device = B_TRUE;
	} else if (fmd_nvl_class_match(hdl, fault,
	"fault.fs.zfs.vdev.checksum")) {
	degrade_device = B_TRUE;
	} else if (fmd_nvl_class_match(hdl, fault,
	"fault.fs.zfs.device")) {
	fault_device = B_FALSE;
	} else if (fmd_nvl_class_match(hdl, fault, "fault.io.*")) {
	is_disk = B_TRUE;
	fault_device = B_TRUE;
	} else {
	continue;
	}

	if (is_disk) {
	continue;
	} else {
	/*
	* This is a ZFS fault. Lookup the resource, and
	* attempt to find the matching vdev.
	*/
	if (nvlist_lookup_nvlist(fault, FM_FAULT_RESOURCE,
	&resource) != 0 \|\|
	nvlist_lookup_string(resource, FM_FMRI_SCHEME,
	&scheme) != 0)
	continue;

	if (strcmp(scheme, FM_FMRI_SCHEME_ZFS) != 0)
	continue;

	if (nvlist_lookup_uint64(resource, FM_FMRI_ZFS_POOL,
	&pool_guid) != 0)
	continue;

	if (nvlist_lookup_uint64(resource, FM_FMRI_ZFS_VDEV,
	&vdev_guid) != 0) {
	if (is_repair)
	vdev_guid = 0;
	else
	continue;
	}

	if ((zhp = find_by_guid(zhdl, pool_guid, vdev_guid,
	&vdev)) == NULL)
	continue;

	aux = VDEV_AUX_ERR_EXCEEDED;
	}

	if (vdev_guid == 0) {
	/*
	* For pool-level repair events, clear the entire pool.
	*/
	fmd_hdl_debug(hdl, "zpool_clear of pool '%s'",
	zpool_get_name(zhp));
	(void) zpool_clear(zhp, NULL, NULL);
	zpool_close(zhp);
	continue;
	}

	/*
	* If this is a repair event, then mark the vdev as repaired and
	* continue.
	*/
	if (is_repair) {
	repair_done = 1;
	fmd_hdl_debug(hdl, "zpool_clear of pool '%s' vdev %llu",
	zpool_get_name(zhp), vdev_guid);
	(void) zpool_vdev_clear(zhp, vdev_guid);
	zpool_close(zhp);
	continue;
	}

	/*
	* Actively fault the device if needed.
	*/
	if (fault_device)
	(void) zpool_vdev_fault(zhp, vdev_guid, aux);
	if (degrade_device)
	(void) zpool_vdev_degrade(zhp, vdev_guid, aux);

	if (fault_device \|\| degrade_device)
	fmd_hdl_debug(hdl, "zpool_vdev_%s: vdev %llu on '%s'",
	fault_device ? "fault" : "degrade", vdev_guid,
	zpool_get_name(zhp));

	/*
	* Attempt to substitute a hot spare.
	*/
	(void) replace_with_spare(hdl, zhp, vdev);
	zpool_close(zhp);
	}

	if (strcmp(class, FM_LIST_REPAIRED_CLASS) == 0 && repair_done &&
	nvlist_lookup_string(nvl, FM_SUSPECT_UUID, &uuid) == 0)
	fmd_case_uuresolved(hdl, uuid);
	}

	static const fmd_hdl_ops_t fmd_ops = {
	zfs_retire_recv, /* fmdo_recv */
	NULL, /* fmdo_timeout */
	NULL, /* fmdo_close */
	NULL, /* fmdo_stats */
	NULL, /* fmdo_gc */
	};

	static const fmd_prop_t fmd_props[] = {
	{ "spare_on_remove", FMD_TYPE_BOOL, "true" },
	{ NULL, 0, NULL }
	};

	static const fmd_hdl_info_t fmd_info = {
	"ZFS Retire Agent", "1.0", &fmd_ops, fmd_props
	};

	void
	_zfs_retire_init(fmd_hdl_t *hdl)
	{
	zfs_retire_data_t *zdp;
	libzfs_handle_t *zhdl;

	if ((zhdl = libzfs_init()) == NULL)
	return;

	if (fmd_hdl_register(hdl, FMD_API_VERSION, &fmd_info) != 0) {
	libzfs_fini(zhdl);
	return;
	}

	zdp = fmd_hdl_zalloc(hdl, sizeof (zfs_retire_data_t), FMD_SLEEP);
	zdp->zrd_hdl = zhdl;

	fmd_hdl_setspecific(hdl, zdp);
	}

	void
	_zfs_retire_fini(fmd_hdl_t *hdl)
	{
	zfs_retire_data_t *zdp = fmd_hdl_getspecific(hdl);

	if (zdp != NULL) {
	zfs_retire_clear_data(hdl, zdp);
	libzfs_fini(zdp->zrd_hdl);
	fmd_hdl_free(hdl, zdp, sizeof (zfs_retire_data_t));
	}
	}
	Index: vendor-sys/openzfs/dist/cmd/zed/zed.c
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zed/zed.c (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zed/zed.c (revision 366774)
	@@ -1,306 +1,306 @@
	/*
	- * This file is part of the ZFS Event Daemon (ZED)
	- * for ZFS on Linux (ZoL) <http://zfsonlinux.org/>.
	+ * This file is part of the ZFS Event Daemon (ZED).
	+ *
	* Developed at Lawrence Livermore National Laboratory (LLNL-CODE-403049).
	* Copyright (C) 2013-2014 Lawrence Livermore National Security, LLC.
	* Refer to the ZoL git commit log for authoritative copyright attribution.
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License Version 1.0 (CDDL-1.0).
	* You can obtain a copy of the license from the top-level file
	* "OPENSOLARIS.LICENSE" or at <http://opensource.org/licenses/CDDL-1.0>.
	* You may not use this file except in compliance with the license.
	*/

	#include <errno.h>
	#include <fcntl.h>
	#include <signal.h>
	#include <stdio.h>
	#include <stdlib.h>
	#include <string.h>
	#include <sys/mman.h>
	#include <sys/stat.h>
	#include <unistd.h>
	#include "zed.h"
	#include "zed_conf.h"
	#include "zed_event.h"
	#include "zed_file.h"
	#include "zed_log.h"

	static volatile sig_atomic_t _got_exit = 0;
	static volatile sig_atomic_t _got_hup = 0;

	/*
	* Signal handler for SIGINT & SIGTERM.
	*/
	static void
	_exit_handler(int signum)
	{
	_got_exit = 1;
	}

	/*
	* Signal handler for SIGHUP.
	*/
	static void
	_hup_handler(int signum)
	{
	_got_hup = 1;
	}

	/*
	* Register signal handlers.
	*/
	static void
	_setup_sig_handlers(void)
	{
	struct sigaction sa;

	if (sigemptyset(&sa.sa_mask) < 0)
	zed_log_die("Failed to initialize sigset");

	sa.sa_flags = SA_RESTART;
	sa.sa_handler = SIG_IGN;

	if (sigaction(SIGPIPE, &sa, NULL) < 0)
	zed_log_die("Failed to ignore SIGPIPE");

	sa.sa_handler = _exit_handler;
	if (sigaction(SIGINT, &sa, NULL) < 0)
	zed_log_die("Failed to register SIGINT handler");

	if (sigaction(SIGTERM, &sa, NULL) < 0)
	zed_log_die("Failed to register SIGTERM handler");

	sa.sa_handler = _hup_handler;
	if (sigaction(SIGHUP, &sa, NULL) < 0)
	zed_log_die("Failed to register SIGHUP handler");
	}

	/*
	* Lock all current and future pages in the virtual memory address space.
	* Access to locked pages will never be delayed by a page fault.
	*
	* EAGAIN is tested up to max_tries in case this is a transient error.
	*
	* Note that memory locks are not inherited by a child created via fork()
	* and are automatically removed during an execve(). As such, this must
	* be called after the daemon fork()s (when running in the background).
	*/
	static void
	_lock_memory(void)
	{
	#if HAVE_MLOCKALL
	int i = 0;
	const int max_tries = 10;

	for (i = 0; i < max_tries; i++) {
	if (mlockall(MCL_CURRENT \| MCL_FUTURE) == 0) {
	zed_log_msg(LOG_INFO, "Locked all pages in memory");
	return;
	}
	if (errno != EAGAIN)
	break;
	}
	zed_log_die("Failed to lock memory pages: %s", strerror(errno));

	#else /* HAVE_MLOCKALL */
	zed_log_die("Failed to lock memory pages: mlockall() not supported");
	#endif /* HAVE_MLOCKALL */
	}

	/*
	* Start daemonization of the process including the double fork().
	*
	* The parent process will block here until _finish_daemonize() is called
	* (in the grandchild process), at which point the parent process will exit.
	* This prevents the parent process from exiting until initialization is
	* complete.
	*/
	static void
	_start_daemonize(void)
	{
	pid_t pid;
	struct sigaction sa;

	/* Create pipe for communicating with child during daemonization. */
	zed_log_pipe_open();

	/* Background process and ensure child is not process group leader. */
	pid = fork();
	if (pid < 0) {
	zed_log_die("Failed to create child process: %s",
	strerror(errno));
	} else if (pid > 0) {

	/* Close writes since parent will only read from pipe. */
	zed_log_pipe_close_writes();

	/* Wait for notification that daemonization is complete. */
	zed_log_pipe_wait();

	zed_log_pipe_close_reads();
	_exit(EXIT_SUCCESS);
	}

	/* Close reads since child will only write to pipe. */
	zed_log_pipe_close_reads();

	/* Create independent session and detach from terminal. */
	if (setsid() < 0)
	zed_log_die("Failed to create new session: %s",
	strerror(errno));

	/* Prevent child from terminating on HUP when session leader exits. */
	if (sigemptyset(&sa.sa_mask) < 0)
	zed_log_die("Failed to initialize sigset");

	sa.sa_flags = 0;
	sa.sa_handler = SIG_IGN;

	if (sigaction(SIGHUP, &sa, NULL) < 0)
	zed_log_die("Failed to ignore SIGHUP");

	/* Ensure process cannot re-acquire terminal. */
	pid = fork();
	if (pid < 0) {
	zed_log_die("Failed to create grandchild process: %s",
	strerror(errno));
	} else if (pid > 0) {
	_exit(EXIT_SUCCESS);
	}
	}

	/*
	* Finish daemonization of the process by closing stdin/stdout/stderr.
	*
	* This must be called at the end of initialization after all external
	* communication channels are established and accessible.
	*/
	static void
	_finish_daemonize(void)
	{
	int devnull;

	/* Preserve fd 0/1/2, but discard data to/from stdin/stdout/stderr. */
	devnull = open("/dev/null", O_RDWR);
	if (devnull < 0)
	zed_log_die("Failed to open /dev/null: %s", strerror(errno));

	if (dup2(devnull, STDIN_FILENO) < 0)
	zed_log_die("Failed to dup /dev/null onto stdin: %s",
	strerror(errno));

	if (dup2(devnull, STDOUT_FILENO) < 0)
	zed_log_die("Failed to dup /dev/null onto stdout: %s",
	strerror(errno));

	if (dup2(devnull, STDERR_FILENO) < 0)
	zed_log_die("Failed to dup /dev/null onto stderr: %s",
	strerror(errno));

	if ((devnull > STDERR_FILENO) && (close(devnull) < 0))
	zed_log_die("Failed to close /dev/null: %s", strerror(errno));

	/* Notify parent that daemonization is complete. */
	zed_log_pipe_close_writes();
	}

	/*
	* ZFS Event Daemon (ZED).
	*/
	int
	main(int argc, char *argv[])
	{
	struct zed_conf *zcp;
	uint64_t saved_eid;
	int64_t saved_etime[2];

	zed_log_init(argv[0]);
	zed_log_stderr_open(LOG_NOTICE);
	zcp = zed_conf_create();
	zed_conf_parse_opts(zcp, argc, argv);
	if (zcp->do_verbose)
	zed_log_stderr_open(LOG_INFO);

	if (geteuid() != 0)
	zed_log_die("Must be run as root");

	zed_conf_parse_file(zcp);

	zed_file_close_from(STDERR_FILENO + 1);

	(void) umask(0);

	if (chdir("/") < 0)
	zed_log_die("Failed to change to root directory");

	if (zed_conf_scan_dir(zcp) < 0)
	exit(EXIT_FAILURE);

	if (!zcp->do_foreground) {
	_start_daemonize();
	zed_log_syslog_open(LOG_DAEMON);
	}
	_setup_sig_handlers();

	if (zcp->do_memlock)
	_lock_memory();

	if ((zed_conf_write_pid(zcp) < 0) && (!zcp->do_force))
	exit(EXIT_FAILURE);

	if (!zcp->do_foreground)
	_finish_daemonize();

	zed_log_msg(LOG_NOTICE,
	"ZFS Event Daemon %s-%s (PID %d)",
	ZFS_META_VERSION, ZFS_META_RELEASE, (int)getpid());

	if (zed_conf_open_state(zcp) < 0)
	exit(EXIT_FAILURE);

	if (zed_conf_read_state(zcp, &saved_eid, saved_etime) < 0)
	exit(EXIT_FAILURE);

	idle:
	/*
	* If -I is specified, attempt to open /dev/zfs repeatedly until
	* successful.
	*/
	do {
	if (!zed_event_init(zcp))
	break;
	/* Wait for some time and try again. tunable? */
	sleep(30);
	} while (!_got_exit && zcp->do_idle);

	if (_got_exit)
	goto out;

	zed_event_seek(zcp, saved_eid, saved_etime);

	while (!_got_exit) {
	int rv;
	if (_got_hup) {
	_got_hup = 0;
	(void) zed_conf_scan_dir(zcp);
	}
	rv = zed_event_service(zcp);

	/* ENODEV: When kernel module is unloaded (osx) */
	if (rv == ENODEV)
	break;
	}

	zed_log_msg(LOG_NOTICE, "Exiting");
	zed_event_fini(zcp);

	if (zcp->do_idle && !_got_exit)
	goto idle;

	out:
	zed_conf_destroy(zcp);
	zed_log_fini();
	exit(EXIT_SUCCESS);
	}
	Index: vendor-sys/openzfs/dist/cmd/zed/zed.h
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zed/zed.h (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zed/zed.h (revision 366774)
	@@ -1,58 +1,58 @@
	/*
	- * This file is part of the ZFS Event Daemon (ZED)
	- * for ZFS on Linux (ZoL) <http://zfsonlinux.org/>.
	+ * This file is part of the ZFS Event Daemon (ZED).
	+ *
	* Developed at Lawrence Livermore National Laboratory (LLNL-CODE-403049).
	* Copyright (C) 2013-2014 Lawrence Livermore National Security, LLC.
	* Refer to the ZoL git commit log for authoritative copyright attribution.
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License Version 1.0 (CDDL-1.0).
	* You can obtain a copy of the license from the top-level file
	* "OPENSOLARIS.LICENSE" or at <http://opensource.org/licenses/CDDL-1.0>.
	* You may not use this file except in compliance with the license.
	*/

	#ifndef ZED_H
	#define ZED_H

	/*
	* Absolute path for the default zed configuration file.
	*/
	#define ZED_CONF_FILE SYSCONFDIR "/zfs/zed.conf"

	/*
	* Absolute path for the default zed pid file.
	*/
	#define ZED_PID_FILE RUNSTATEDIR "/zed.pid"

	/*
	* Absolute path for the default zed state file.
	*/
	#define ZED_STATE_FILE RUNSTATEDIR "/zed.state"

	/*
	* Absolute path for the default zed zedlet directory.
	*/
	#define ZED_ZEDLET_DIR SYSCONFDIR "/zfs/zed.d"

	/*
	* Reserved for future use.
	*/
	#define ZED_MAX_EVENTS 0

	/*
	* Reserved for future use.
	*/
	#define ZED_MIN_EVENTS 0

	/*
	* String prefix for ZED variables passed via environment variables.
	*/
	#define ZED_VAR_PREFIX "ZED_"

	/*
	* String prefix for ZFS event names passed via environment variables.
	*/
	#define ZEVENT_VAR_PREFIX "ZEVENT_"

	#endif /* !ZED_H */
	Index: vendor-sys/openzfs/dist/cmd/zed/zed_conf.c
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zed/zed_conf.c (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zed/zed_conf.c (revision 366774)
	@@ -1,735 +1,735 @@
	/*
	- * This file is part of the ZFS Event Daemon (ZED)
	- * for ZFS on Linux (ZoL) <http://zfsonlinux.org/>.
	+ * This file is part of the ZFS Event Daemon (ZED).
	+ *
	* Developed at Lawrence Livermore National Laboratory (LLNL-CODE-403049).
	* Copyright (C) 2013-2014 Lawrence Livermore National Security, LLC.
	* Refer to the ZoL git commit log for authoritative copyright attribution.
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License Version 1.0 (CDDL-1.0).
	* You can obtain a copy of the license from the top-level file
	* "OPENSOLARIS.LICENSE" or at <http://opensource.org/licenses/CDDL-1.0>.
	* You may not use this file except in compliance with the license.
	*/

	#include <assert.h>
	#include <ctype.h>
	#include <dirent.h>
	#include <errno.h>
	#include <fcntl.h>
	#include <libgen.h>
	#include <limits.h>
	#include <stdio.h>
	#include <stdlib.h>
	#include <string.h>
	#include <sys/stat.h>
	#include <sys/uio.h>
	#include <unistd.h>
	#include "zed.h"
	#include "zed_conf.h"
	#include "zed_file.h"
	#include "zed_log.h"
	#include "zed_strings.h"

	/*
	* Return a new configuration with default values.
	*/
	struct zed_conf *
	zed_conf_create(void)
	{
	struct zed_conf *zcp;

	zcp = calloc(1, sizeof (*zcp));
	if (!zcp)
	goto nomem;

	zcp->syslog_facility = LOG_DAEMON;
	zcp->min_events = ZED_MIN_EVENTS;
	zcp->max_events = ZED_MAX_EVENTS;
	zcp->pid_fd = -1;
	zcp->zedlets = NULL; /* created via zed_conf_scan_dir() */
	zcp->state_fd = -1; /* opened via zed_conf_open_state() */
	zcp->zfs_hdl = NULL; /* opened via zed_event_init() */
	zcp->zevent_fd = -1; /* opened via zed_event_init() */

	if (!(zcp->conf_file = strdup(ZED_CONF_FILE)))
	goto nomem;

	if (!(zcp->pid_file = strdup(ZED_PID_FILE)))
	goto nomem;

	if (!(zcp->zedlet_dir = strdup(ZED_ZEDLET_DIR)))
	goto nomem;

	if (!(zcp->state_file = strdup(ZED_STATE_FILE)))
	goto nomem;

	return (zcp);

	nomem:
	zed_log_die("Failed to create conf: %s", strerror(errno));
	return (NULL);
	}

	/*
	* Destroy the configuration [zcp].
	*
	* Note: zfs_hdl & zevent_fd are destroyed via zed_event_fini().
	*/
	void
	zed_conf_destroy(struct zed_conf *zcp)
	{
	if (!zcp)
	return;

	if (zcp->state_fd >= 0) {
	if (close(zcp->state_fd) < 0)
	zed_log_msg(LOG_WARNING,
	"Failed to close state file \"%s\": %s",
	zcp->state_file, strerror(errno));
	zcp->state_fd = -1;
	}
	if (zcp->pid_file) {
	if ((unlink(zcp->pid_file) < 0) && (errno != ENOENT))
	zed_log_msg(LOG_WARNING,
	"Failed to remove PID file \"%s\": %s",
	zcp->pid_file, strerror(errno));
	}
	if (zcp->pid_fd >= 0) {
	if (close(zcp->pid_fd) < 0)
	zed_log_msg(LOG_WARNING,
	"Failed to close PID file \"%s\": %s",
	zcp->pid_file, strerror(errno));
	zcp->pid_fd = -1;
	}
	if (zcp->conf_file) {
	free(zcp->conf_file);
	zcp->conf_file = NULL;
	}
	if (zcp->pid_file) {
	free(zcp->pid_file);
	zcp->pid_file = NULL;
	}
	if (zcp->zedlet_dir) {
	free(zcp->zedlet_dir);
	zcp->zedlet_dir = NULL;
	}
	if (zcp->state_file) {
	free(zcp->state_file);
	zcp->state_file = NULL;
	}
	if (zcp->zedlets) {
	zed_strings_destroy(zcp->zedlets);
	zcp->zedlets = NULL;
	}
	free(zcp);
	}

	/*
	* Display command-line help and exit.
	*
	* If [got_err] is 0, output to stdout and exit normally;
	* otherwise, output to stderr and exit with a failure status.
	*/
	static void
	_zed_conf_display_help(const char *prog, int got_err)
	{
	FILE *fp = got_err ? stderr : stdout;
	int w1 = 4; /* width of leading whitespace */
	int w2 = 8; /* width of L-justified option field */

	fprintf(fp, "Usage: %s [OPTION]...\n", (prog ? prog : "zed"));
	fprintf(fp, "\n");
	fprintf(fp, "%c%s %s\n", w1, 0x20, -w2, "-h",
	"Display help.");
	fprintf(fp, "%c%s %s\n", w1, 0x20, -w2, "-L",
	"Display license information.");
	fprintf(fp, "%c%s %s\n", w1, 0x20, -w2, "-V",
	"Display version information.");
	fprintf(fp, "\n");
	fprintf(fp, "%c%s %s\n", w1, 0x20, -w2, "-v",
	"Be verbose.");
	fprintf(fp, "%c%s %s\n", w1, 0x20, -w2, "-f",
	"Force daemon to run.");
	fprintf(fp, "%c%s %s\n", w1, 0x20, -w2, "-F",
	"Run daemon in the foreground.");
	fprintf(fp, "%c%s %s\n", w1, 0x20, -w2, "-I",
	"Idle daemon until kernel module is (re)loaded.");
	fprintf(fp, "%c%s %s\n", w1, 0x20, -w2, "-M",
	"Lock all pages in memory.");
	fprintf(fp, "%c%s %s\n", w1, 0x20, -w2, "-P",
	"$PATH for ZED to use (only used by ZTS).");
	fprintf(fp, "%c%s %s\n", w1, 0x20, -w2, "-Z",
	"Zero state file.");
	fprintf(fp, "\n");
	#if 0
	fprintf(fp, "%c%s %s [%s]\n", w1, 0x20, -w2, "-c FILE",
	"Read configuration from FILE.", ZED_CONF_FILE);
	#endif
	fprintf(fp, "%c%s %s [%s]\n", w1, 0x20, -w2, "-d DIR",
	"Read enabled ZEDLETs from DIR.", ZED_ZEDLET_DIR);
	fprintf(fp, "%c%s %s [%s]\n", w1, 0x20, -w2, "-p FILE",
	"Write daemon's PID to FILE.", ZED_PID_FILE);
	fprintf(fp, "%c%s %s [%s]\n", w1, 0x20, -w2, "-s FILE",
	"Write daemon's state to FILE.", ZED_STATE_FILE);
	fprintf(fp, "\n");

	exit(got_err ? EXIT_FAILURE : EXIT_SUCCESS);
	}

	/*
	* Display license information to stdout and exit.
	*/
	static void
	_zed_conf_display_license(void)
	{
	const char **pp;
	const char *text[] = {
	"The ZFS Event Daemon (ZED) is distributed under the terms of the",
	" Common Development and Distribution License (CDDL-1.0)",
	" <http://opensource.org/licenses/CDDL-1.0>.",
	"",
	"Developed at Lawrence Livermore National Laboratory"
	" (LLNL-CODE-403049).",
	"",
	NULL
	};

	for (pp = text; *pp; pp++)
	printf("%s\n", *pp);

	exit(EXIT_SUCCESS);
	}

	/*
	* Display version information to stdout and exit.
	*/
	static void
	_zed_conf_display_version(void)
	{
	printf("%s-%s-%s\n",
	ZFS_META_NAME, ZFS_META_VERSION, ZFS_META_RELEASE);

	exit(EXIT_SUCCESS);
	}

	/*
	* Copy the [path] string to the [resultp] ptr.
	* If [path] is not an absolute path, prefix it with the current working dir.
	* If [resultp] is non-null, free its existing string before assignment.
	*/
	static void
	_zed_conf_parse_path(char *resultp, const char path)
	{
	char buf[PATH_MAX];

	assert(resultp != NULL);
	assert(path != NULL);

	if (*resultp)
	free(*resultp);

	if (path[0] == '/') {
	*resultp = strdup(path);
	} else if (!getcwd(buf, sizeof (buf))) {
	zed_log_die("Failed to get current working dir: %s",
	strerror(errno));
	} else if (strlcat(buf, "/", sizeof (buf)) >= sizeof (buf)) {
	zed_log_die("Failed to copy path: %s", strerror(ENAMETOOLONG));
	} else if (strlcat(buf, path, sizeof (buf)) >= sizeof (buf)) {
	zed_log_die("Failed to copy path: %s", strerror(ENAMETOOLONG));
	} else {
	*resultp = strdup(buf);
	}
	if (!*resultp)
	zed_log_die("Failed to copy path: %s", strerror(ENOMEM));
	}

	/*
	* Parse the command-line options into the configuration [zcp].
	*/
	void
	zed_conf_parse_opts(struct zed_conf zcp, int argc, char *argv)
	{
	const char * const opts = ":hLVc:d:p:P:s:vfFMZI";
	int opt;

	if (!zcp \|\| !argv \|\| !argv[0])
	zed_log_die("Failed to parse options: Internal error");

	opterr = 0; /* suppress default getopt err msgs */

	while ((opt = getopt(argc, argv, opts)) != -1) {
	switch (opt) {
	case 'h':
	_zed_conf_display_help(argv[0], EXIT_SUCCESS);
	break;
	case 'L':
	_zed_conf_display_license();
	break;
	case 'V':
	_zed_conf_display_version();
	break;
	case 'c':
	_zed_conf_parse_path(&zcp->conf_file, optarg);
	break;
	case 'd':
	_zed_conf_parse_path(&zcp->zedlet_dir, optarg);
	break;
	case 'I':
	zcp->do_idle = 1;
	break;
	case 'p':
	_zed_conf_parse_path(&zcp->pid_file, optarg);
	break;
	case 'P':
	_zed_conf_parse_path(&zcp->path, optarg);
	break;
	case 's':
	_zed_conf_parse_path(&zcp->state_file, optarg);
	break;
	case 'v':
	zcp->do_verbose = 1;
	break;
	case 'f':
	zcp->do_force = 1;
	break;
	case 'F':
	zcp->do_foreground = 1;
	break;
	case 'M':
	zcp->do_memlock = 1;
	break;
	case 'Z':
	zcp->do_zero = 1;
	break;
	case '?':
	default:
	if (optopt == '?')
	_zed_conf_display_help(argv[0], EXIT_SUCCESS);

	fprintf(stderr, "%s: %s '-%c'\n\n", argv[0],
	"Invalid option", optopt);
	_zed_conf_display_help(argv[0], EXIT_FAILURE);
	break;
	}
	}
	}

	/*
	* Parse the configuration file into the configuration [zcp].
	*
	* FIXME: Not yet implemented.
	*/
	void
	zed_conf_parse_file(struct zed_conf *zcp)
	{
	if (!zcp)
	zed_log_die("Failed to parse config: %s", strerror(EINVAL));
	}

	/*
	* Scan the [zcp] zedlet_dir for files to exec based on the event class.
	* Files must be executable by user, but not writable by group or other.
	* Dotfiles are ignored.
	*
	* Return 0 on success with an updated set of zedlets,
	* or -1 on error with errno set.
	*
	* FIXME: Check if zedlet_dir and all parent dirs are secure.
	*/
	int
	zed_conf_scan_dir(struct zed_conf *zcp)
	{
	zed_strings_t *zedlets;
	DIR *dirp;
	struct dirent *direntp;
	char pathname[PATH_MAX];
	struct stat st;
	int n;

	if (!zcp) {
	errno = EINVAL;
	zed_log_msg(LOG_ERR, "Failed to scan zedlet dir: %s",
	strerror(errno));
	return (-1);
	}
	zedlets = zed_strings_create();
	if (!zedlets) {
	errno = ENOMEM;
	zed_log_msg(LOG_WARNING, "Failed to scan dir \"%s\": %s",
	zcp->zedlet_dir, strerror(errno));
	return (-1);
	}
	dirp = opendir(zcp->zedlet_dir);
	if (!dirp) {
	int errno_bak = errno;
	zed_log_msg(LOG_WARNING, "Failed to open dir \"%s\": %s",
	zcp->zedlet_dir, strerror(errno));
	zed_strings_destroy(zedlets);
	errno = errno_bak;
	return (-1);
	}
	while ((direntp = readdir(dirp))) {
	if (direntp->d_name[0] == '.')
	continue;

	n = snprintf(pathname, sizeof (pathname),
	"%s/%s", zcp->zedlet_dir, direntp->d_name);
	if ((n < 0) \|\| (n >= sizeof (pathname))) {
	zed_log_msg(LOG_WARNING, "Failed to stat \"%s\": %s",
	direntp->d_name, strerror(ENAMETOOLONG));
	continue;
	}
	if (stat(pathname, &st) < 0) {
	zed_log_msg(LOG_WARNING, "Failed to stat \"%s\": %s",
	pathname, strerror(errno));
	continue;
	}
	if (!S_ISREG(st.st_mode)) {
	zed_log_msg(LOG_INFO,
	"Ignoring \"%s\": not a regular file",
	direntp->d_name);
	continue;
	}
	if ((st.st_uid != 0) && !zcp->do_force) {
	zed_log_msg(LOG_NOTICE,
	"Ignoring \"%s\": not owned by root",
	direntp->d_name);
	continue;
	}
	if (!(st.st_mode & S_IXUSR)) {
	zed_log_msg(LOG_INFO,
	"Ignoring \"%s\": not executable by user",
	direntp->d_name);
	continue;
	}
	if ((st.st_mode & S_IWGRP) && !zcp->do_force) {
	zed_log_msg(LOG_NOTICE,
	"Ignoring \"%s\": writable by group",
	direntp->d_name);
	continue;
	}
	if ((st.st_mode & S_IWOTH) && !zcp->do_force) {
	zed_log_msg(LOG_NOTICE,
	"Ignoring \"%s\": writable by other",
	direntp->d_name);
	continue;
	}
	if (zed_strings_add(zedlets, NULL, direntp->d_name) < 0) {
	zed_log_msg(LOG_WARNING,
	"Failed to register \"%s\": %s",
	direntp->d_name, strerror(errno));
	continue;
	}
	if (zcp->do_verbose)
	zed_log_msg(LOG_INFO,
	"Registered zedlet \"%s\"", direntp->d_name);
	}
	if (closedir(dirp) < 0) {
	int errno_bak = errno;
	zed_log_msg(LOG_WARNING, "Failed to close dir \"%s\": %s",
	zcp->zedlet_dir, strerror(errno));
	zed_strings_destroy(zedlets);
	errno = errno_bak;
	return (-1);
	}
	if (zcp->zedlets)
	zed_strings_destroy(zcp->zedlets);

	zcp->zedlets = zedlets;
	return (0);
	}

	/*
	* Write the PID file specified in [zcp].
	* Return 0 on success, -1 on error.
	*
	* This must be called after fork()ing to become a daemon (so the correct PID
	* is recorded), but before daemonization is complete and the parent process
	* exits (for synchronization with systemd).
	*/
	int
	zed_conf_write_pid(struct zed_conf *zcp)
	{
	const mode_t dirmode = S_IRWXU \| S_IRGRP \| S_IXGRP \| S_IROTH \| S_IXOTH;
	const mode_t filemode = S_IRUSR \| S_IWUSR \| S_IRGRP \| S_IROTH;
	char buf[PATH_MAX];
	int n;
	char *p;
	mode_t mask;
	int rv;

	if (!zcp \|\| !zcp->pid_file) {
	errno = EINVAL;
	zed_log_msg(LOG_ERR, "Failed to create PID file: %s",
	strerror(errno));
	return (-1);
	}
	assert(zcp->pid_fd == -1);
	/*
	* Create PID file directory if needed.
	*/
	n = strlcpy(buf, zcp->pid_file, sizeof (buf));
	if (n >= sizeof (buf)) {
	errno = ENAMETOOLONG;
	zed_log_msg(LOG_ERR, "Failed to create PID file: %s",
	strerror(errno));
	goto err;
	}
	p = strrchr(buf, '/');
	if (p)
	*p = '\0';

	if ((mkdirp(buf, dirmode) < 0) && (errno != EEXIST)) {
	zed_log_msg(LOG_ERR, "Failed to create directory \"%s\": %s",
	buf, strerror(errno));
	goto err;
	}
	/*
	* Obtain PID file lock.
	*/
	mask = umask(0);
	umask(mask \| 022);
	zcp->pid_fd = open(zcp->pid_file, (O_RDWR \| O_CREAT), filemode);
	umask(mask);
	if (zcp->pid_fd < 0) {
	zed_log_msg(LOG_ERR, "Failed to open PID file \"%s\": %s",
	zcp->pid_file, strerror(errno));
	goto err;
	}
	rv = zed_file_lock(zcp->pid_fd);
	if (rv < 0) {
	zed_log_msg(LOG_ERR, "Failed to lock PID file \"%s\": %s",
	zcp->pid_file, strerror(errno));
	goto err;
	} else if (rv > 0) {
	pid_t pid = zed_file_is_locked(zcp->pid_fd);
	if (pid < 0) {
	zed_log_msg(LOG_ERR,
	"Failed to test lock on PID file \"%s\"",
	zcp->pid_file);
	} else if (pid > 0) {
	zed_log_msg(LOG_ERR,
	"Found PID %d bound to PID file \"%s\"",
	pid, zcp->pid_file);
	} else {
	zed_log_msg(LOG_ERR,
	"Inconsistent lock state on PID file \"%s\"",
	zcp->pid_file);
	}
	goto err;
	}
	/*
	* Write PID file.
	*/
	n = snprintf(buf, sizeof (buf), "%d\n", (int)getpid());
	if ((n < 0) \|\| (n >= sizeof (buf))) {
	errno = ERANGE;
	zed_log_msg(LOG_ERR, "Failed to write PID file \"%s\": %s",
	zcp->pid_file, strerror(errno));
	} else if (zed_file_write_n(zcp->pid_fd, buf, n) != n) {
	zed_log_msg(LOG_ERR, "Failed to write PID file \"%s\": %s",
	zcp->pid_file, strerror(errno));
	} else if (fdatasync(zcp->pid_fd) < 0) {
	zed_log_msg(LOG_ERR, "Failed to sync PID file \"%s\": %s",
	zcp->pid_file, strerror(errno));
	} else {
	return (0);
	}

	err:
	if (zcp->pid_fd >= 0) {
	(void) close(zcp->pid_fd);
	zcp->pid_fd = -1;
	}
	return (-1);
	}

	/*
	* Open and lock the [zcp] state_file.
	* Return 0 on success, -1 on error.
	*
	* FIXME: Move state information into kernel.
	*/
	int
	zed_conf_open_state(struct zed_conf *zcp)
	{
	char dirbuf[PATH_MAX];
	mode_t dirmode = S_IRWXU \| S_IRGRP \| S_IXGRP \| S_IROTH \| S_IXOTH;
	int n;
	char *p;
	int rv;

	if (!zcp \|\| !zcp->state_file) {
	errno = EINVAL;
	zed_log_msg(LOG_ERR, "Failed to open state file: %s",
	strerror(errno));
	return (-1);
	}
	n = strlcpy(dirbuf, zcp->state_file, sizeof (dirbuf));
	if (n >= sizeof (dirbuf)) {
	errno = ENAMETOOLONG;
	zed_log_msg(LOG_WARNING, "Failed to open state file: %s",
	strerror(errno));
	return (-1);
	}
	p = strrchr(dirbuf, '/');
	if (p)
	*p = '\0';

	if ((mkdirp(dirbuf, dirmode) < 0) && (errno != EEXIST)) {
	zed_log_msg(LOG_WARNING,
	"Failed to create directory \"%s\": %s",
	dirbuf, strerror(errno));
	return (-1);
	}
	if (zcp->state_fd >= 0) {
	if (close(zcp->state_fd) < 0) {
	zed_log_msg(LOG_WARNING,
	"Failed to close state file \"%s\": %s",
	zcp->state_file, strerror(errno));
	return (-1);
	}
	}
	if (zcp->do_zero)
	(void) unlink(zcp->state_file);

	zcp->state_fd = open(zcp->state_file,
	(O_RDWR \| O_CREAT), (S_IRUSR \| S_IWUSR \| S_IRGRP \| S_IROTH));
	if (zcp->state_fd < 0) {
	zed_log_msg(LOG_WARNING, "Failed to open state file \"%s\": %s",
	zcp->state_file, strerror(errno));
	return (-1);
	}
	rv = zed_file_lock(zcp->state_fd);
	if (rv < 0) {
	zed_log_msg(LOG_WARNING, "Failed to lock state file \"%s\": %s",
	zcp->state_file, strerror(errno));
	return (-1);
	}
	if (rv > 0) {
	pid_t pid = zed_file_is_locked(zcp->state_fd);
	if (pid < 0) {
	zed_log_msg(LOG_WARNING,
	"Failed to test lock on state file \"%s\"",
	zcp->state_file);
	} else if (pid > 0) {
	zed_log_msg(LOG_WARNING,
	"Found PID %d bound to state file \"%s\"",
	pid, zcp->state_file);
	} else {
	zed_log_msg(LOG_WARNING,
	"Inconsistent lock state on state file \"%s\"",
	zcp->state_file);
	}
	return (-1);
	}
	return (0);
	}

	/*
	* Read the opened [zcp] state_file to obtain the eid & etime of the last event
	* processed. Write the state from the last event to the [eidp] & [etime] args
	* passed by reference. Note that etime[] is an array of size 2.
	* Return 0 on success, -1 on error.
	*/
	int
	zed_conf_read_state(struct zed_conf zcp, uint64_t eidp, int64_t etime[])
	{
	ssize_t len;
	struct iovec iov[3];
	ssize_t n;

	if (!zcp \|\| !eidp \|\| !etime) {
	errno = EINVAL;
	zed_log_msg(LOG_ERR,
	"Failed to read state file: %s", strerror(errno));
	return (-1);
	}
	if (lseek(zcp->state_fd, 0, SEEK_SET) == (off_t)-1) {
	zed_log_msg(LOG_WARNING,
	"Failed to reposition state file offset: %s",
	strerror(errno));
	return (-1);
	}
	len = 0;
	iov[0].iov_base = eidp;
	len += iov[0].iov_len = sizeof (*eidp);
	iov[1].iov_base = &etime[0];
	len += iov[1].iov_len = sizeof (etime[0]);
	iov[2].iov_base = &etime[1];
	len += iov[2].iov_len = sizeof (etime[1]);

	n = readv(zcp->state_fd, iov, 3);
	if (n == 0) {
	*eidp = 0;
	} else if (n < 0) {
	zed_log_msg(LOG_WARNING,
	"Failed to read state file \"%s\": %s",
	zcp->state_file, strerror(errno));
	return (-1);
	} else if (n != len) {
	errno = EIO;
	zed_log_msg(LOG_WARNING,
	"Failed to read state file \"%s\": Read %d of %d bytes",
	zcp->state_file, n, len);
	return (-1);
	}
	return (0);
	}

	/*
	* Write the [eid] & [etime] of the last processed event to the opened
	* [zcp] state_file. Note that etime[] is an array of size 2.
	* Return 0 on success, -1 on error.
	*/
	int
	zed_conf_write_state(struct zed_conf *zcp, uint64_t eid, int64_t etime[])
	{
	ssize_t len;
	struct iovec iov[3];
	ssize_t n;

	if (!zcp) {
	errno = EINVAL;
	zed_log_msg(LOG_ERR,
	"Failed to write state file: %s", strerror(errno));
	return (-1);
	}
	if (lseek(zcp->state_fd, 0, SEEK_SET) == (off_t)-1) {
	zed_log_msg(LOG_WARNING,
	"Failed to reposition state file offset: %s",
	strerror(errno));
	return (-1);
	}
	len = 0;
	iov[0].iov_base = &eid;
	len += iov[0].iov_len = sizeof (eid);
	iov[1].iov_base = &etime[0];
	len += iov[1].iov_len = sizeof (etime[0]);
	iov[2].iov_base = &etime[1];
	len += iov[2].iov_len = sizeof (etime[1]);

	n = writev(zcp->state_fd, iov, 3);
	if (n < 0) {
	zed_log_msg(LOG_WARNING,
	"Failed to write state file \"%s\": %s",
	zcp->state_file, strerror(errno));
	return (-1);
	}
	if (n != len) {
	errno = EIO;
	zed_log_msg(LOG_WARNING,
	"Failed to write state file \"%s\": Wrote %d of %d bytes",
	zcp->state_file, n, len);
	return (-1);
	}
	if (fdatasync(zcp->state_fd) < 0) {
	zed_log_msg(LOG_WARNING,
	"Failed to sync state file \"%s\": %s",
	zcp->state_file, strerror(errno));
	return (-1);
	}
	return (0);
	}
	Index: vendor-sys/openzfs/dist/cmd/zed/zed_conf.h
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zed/zed_conf.h (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zed/zed_conf.h (revision 366774)
	@@ -1,62 +1,62 @@
	/*
	- * This file is part of the ZFS Event Daemon (ZED)
	- * for ZFS on Linux (ZoL) <http://zfsonlinux.org/>.
	+ * This file is part of the ZFS Event Daemon (ZED).
	+ *
	* Developed at Lawrence Livermore National Laboratory (LLNL-CODE-403049).
	* Copyright (C) 2013-2014 Lawrence Livermore National Security, LLC.
	* Refer to the ZoL git commit log for authoritative copyright attribution.
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License Version 1.0 (CDDL-1.0).
	* You can obtain a copy of the license from the top-level file
	* "OPENSOLARIS.LICENSE" or at <http://opensource.org/licenses/CDDL-1.0>.
	* You may not use this file except in compliance with the license.
	*/

	#ifndef ZED_CONF_H
	#define ZED_CONF_H

	#include <libzfs.h>
	#include <stdint.h>
	#include "zed_strings.h"

	struct zed_conf {
	unsigned do_force:1; /* true if force enabled */
	unsigned do_foreground:1; /* true if run in foreground */
	unsigned do_memlock:1; /* true if locking memory */
	unsigned do_verbose:1; /* true if verbosity enabled */
	unsigned do_zero:1; /* true if zeroing state */
	unsigned do_idle:1; /* true if idle enabled */
	int syslog_facility; /* syslog facility value */
	int min_events; /* RESERVED FOR FUTURE USE */
	int max_events; /* RESERVED FOR FUTURE USE */
	char conf_file; / abs path to config file */
	char pid_file; / abs path to pid file */
	int pid_fd; /* fd to pid file for lock */
	char zedlet_dir; / abs path to zedlet dir */
	zed_strings_t zedlets; / names of enabled zedlets */
	char state_file; / abs path to state file */
	int state_fd; /* fd to state file */
	libzfs_handle_t zfs_hdl; / handle to libzfs */
	int zevent_fd; /* fd for access to zevents */
	char path; / custom $PATH for zedlets to use */
	};

	struct zed_conf *zed_conf_create(void);

	void zed_conf_destroy(struct zed_conf *zcp);

	void zed_conf_parse_opts(struct zed_conf zcp, int argc, char *argv);

	void zed_conf_parse_file(struct zed_conf *zcp);

	int zed_conf_scan_dir(struct zed_conf *zcp);

	int zed_conf_write_pid(struct zed_conf *zcp);

	int zed_conf_open_state(struct zed_conf *zcp);

	int zed_conf_read_state(struct zed_conf zcp, uint64_t eidp, int64_t etime[]);

	int zed_conf_write_state(struct zed_conf *zcp, uint64_t eid, int64_t etime[]);

	#endif /* !ZED_CONF_H */
	Index: vendor-sys/openzfs/dist/cmd/zed/zed_event.c
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zed/zed_event.c (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zed/zed_event.c (revision 366774)
	@@ -1,965 +1,965 @@
	/*
	- * This file is part of the ZFS Event Daemon (ZED)
	- * for ZFS on Linux (ZoL) <http://zfsonlinux.org/>.
	+ * This file is part of the ZFS Event Daemon (ZED).
	+ *
	* Developed at Lawrence Livermore National Laboratory (LLNL-CODE-403049).
	* Copyright (C) 2013-2014 Lawrence Livermore National Security, LLC.
	* Refer to the ZoL git commit log for authoritative copyright attribution.
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License Version 1.0 (CDDL-1.0).
	* You can obtain a copy of the license from the top-level file
	* "OPENSOLARIS.LICENSE" or at <http://opensource.org/licenses/CDDL-1.0>.
	* You may not use this file except in compliance with the license.
	*/

	#include <ctype.h>
	#include <errno.h>
	#include <fcntl.h>
	#include <libzfs.h> /* FIXME: Replace with libzfs_core. */
	#include <paths.h>
	#include <stdarg.h>
	#include <stdio.h>
	#include <stdlib.h>
	#include <string.h>
	#include <sys/zfs_ioctl.h>
	#include <time.h>
	#include <unistd.h>
	#include <sys/fm/fs/zfs.h>
	#include "zed.h"
	#include "zed_conf.h"
	#include "zed_disk_event.h"
	#include "zed_event.h"
	#include "zed_exec.h"
	#include "zed_file.h"
	#include "zed_log.h"
	#include "zed_strings.h"

	#include "agents/zfs_agents.h"

	#define MAXBUF 4096

	/*
	* Open the libzfs interface.
	*/
	int
	zed_event_init(struct zed_conf *zcp)
	{
	if (!zcp)
	zed_log_die("Failed zed_event_init: %s", strerror(EINVAL));

	zcp->zfs_hdl = libzfs_init();
	if (!zcp->zfs_hdl) {
	if (zcp->do_idle)
	return (-1);
	zed_log_die("Failed to initialize libzfs");
	}

	zcp->zevent_fd = open(ZFS_DEV, O_RDWR);
	if (zcp->zevent_fd < 0) {
	if (zcp->do_idle)
	return (-1);
	zed_log_die("Failed to open \"%s\": %s",
	ZFS_DEV, strerror(errno));
	}

	zfs_agent_init(zcp->zfs_hdl);

	if (zed_disk_event_init() != 0) {
	if (zcp->do_idle)
	return (-1);
	zed_log_die("Failed to initialize disk events");
	}

	return (0);
	}

	/*
	* Close the libzfs interface.
	*/
	void
	zed_event_fini(struct zed_conf *zcp)
	{
	if (!zcp)
	zed_log_die("Failed zed_event_fini: %s", strerror(EINVAL));

	zed_disk_event_fini();
	zfs_agent_fini();

	if (zcp->zevent_fd >= 0) {
	if (close(zcp->zevent_fd) < 0)
	zed_log_msg(LOG_WARNING, "Failed to close \"%s\": %s",
	ZFS_DEV, strerror(errno));

	zcp->zevent_fd = -1;
	}
	if (zcp->zfs_hdl) {
	libzfs_fini(zcp->zfs_hdl);
	zcp->zfs_hdl = NULL;
	}
	}

	/*
	* Seek to the event specified by [saved_eid] and [saved_etime].
	* This protects against processing a given event more than once.
	* Return 0 upon a successful seek to the specified event, or -1 otherwise.
	*
	* A zevent is considered to be uniquely specified by its (eid,time) tuple.
	* The unsigned 64b eid is set to 1 when the kernel module is loaded, and
	* incremented by 1 for each new event. Since the state file can persist
	* across a kernel module reload, the time must be checked to ensure a match.
	*/
	int
	zed_event_seek(struct zed_conf *zcp, uint64_t saved_eid, int64_t saved_etime[])
	{
	uint64_t eid;
	int found;
	nvlist_t *nvl;
	int n_dropped;
	int64_t *etime;
	uint_t nelem;
	int rv;

	if (!zcp) {
	errno = EINVAL;
	zed_log_msg(LOG_ERR, "Failed to seek zevent: %s",
	strerror(errno));
	return (-1);
	}
	eid = 0;
	found = 0;
	while ((eid < saved_eid) && !found) {
	rv = zpool_events_next(zcp->zfs_hdl, &nvl, &n_dropped,
	ZEVENT_NONBLOCK, zcp->zevent_fd);

	if ((rv != 0) \|\| !nvl)
	break;

	if (n_dropped > 0) {
	zed_log_msg(LOG_WARNING, "Missed %d events", n_dropped);
	/*
	* FIXME: Increase max size of event nvlist in
	* /sys/module/zfs/parameters/zfs_zevent_len_max ?
	*/
	}
	if (nvlist_lookup_uint64(nvl, "eid", &eid) != 0) {
	zed_log_msg(LOG_WARNING, "Failed to lookup zevent eid");
	} else if (nvlist_lookup_int64_array(nvl, "time",
	&etime, &nelem) != 0) {
	zed_log_msg(LOG_WARNING,
	"Failed to lookup zevent time (eid=%llu)", eid);
	} else if (nelem != 2) {
	zed_log_msg(LOG_WARNING,
	"Failed to lookup zevent time (eid=%llu, nelem=%u)",
	eid, nelem);
	} else if ((eid != saved_eid) \|\|
	(etime[0] != saved_etime[0]) \|\|
	(etime[1] != saved_etime[1])) {
	/* no-op */
	} else {
	found = 1;
	}
	free(nvl);
	}
	if (!found && (saved_eid > 0)) {
	if (zpool_events_seek(zcp->zfs_hdl, ZEVENT_SEEK_START,
	zcp->zevent_fd) < 0)
	zed_log_msg(LOG_WARNING, "Failed to seek to eid=0");
	else
	eid = 0;
	}
	zed_log_msg(LOG_NOTICE, "Processing events since eid=%llu", eid);
	return (found ? 0 : -1);
	}

	/*
	* Return non-zero if nvpair [name] should be formatted in hex; o/w, return 0.
	*/
	static int
	_zed_event_value_is_hex(const char *name)
	{
	const char *hex_suffix[] = {
	"_guid",
	"_guids",
	NULL
	};
	const char **pp;
	char *p;

	if (!name)
	return (0);

	for (pp = hex_suffix; *pp; pp++) {
	p = strstr(name, *pp);
	if (p && strlen(p) == strlen(*pp))
	return (1);
	}
	return (0);
	}

	/*
	* Add an environment variable for [eid] to the container [zsp].
	*
	* The variable name is the concatenation of [prefix] and [name] converted to
	* uppercase with non-alphanumeric characters converted to underscores;
	* [prefix] is optional, and [name] must begin with an alphabetic character.
	* If the converted variable name already exists within the container [zsp],
	* its existing value will be replaced with the new value.
	*
	* The variable value is specified by the format string [fmt].
	*
	* Returns 0 on success, and -1 on error (with errno set).
	*
	* All environment variables in [zsp] should be added through this function.
	*/
	static int
	_zed_event_add_var(uint64_t eid, zed_strings_t *zsp,
	const char prefix, const char name, const char *fmt, ...)
	{
	char keybuf[MAXBUF];
	char valbuf[MAXBUF];
	char *dstp;
	const char *srcp;
	const char *lastp;
	int n;
	int buflen;
	va_list vargs;

	assert(zsp != NULL);
	assert(fmt != NULL);

	if (!name) {
	errno = EINVAL;
	zed_log_msg(LOG_WARNING,
	"Failed to add variable for eid=%llu: Name is empty", eid);
	return (-1);
	} else if (!isalpha(name[0])) {
	errno = EINVAL;
	zed_log_msg(LOG_WARNING,
	"Failed to add variable for eid=%llu: "
	"Name \"%s\" is invalid", eid, name);
	return (-1);
	}
	/*
	* Construct the string key by converting PREFIX (if present) and NAME.
	*/
	dstp = keybuf;
	lastp = keybuf + sizeof (keybuf);
	if (prefix) {
	for (srcp = prefix; *srcp && (dstp < lastp); srcp++)
	dstp++ = isalnum(srcp) ? toupper(*srcp) : '_';
	}
	for (srcp = name; *srcp && (dstp < lastp); srcp++)
	dstp++ = isalnum(srcp) ? toupper(*srcp) : '_';

	if (dstp == lastp) {
	errno = ENAMETOOLONG;
	zed_log_msg(LOG_WARNING,
	"Failed to add variable for eid=%llu: Name too long", eid);
	return (-1);
	}
	*dstp = '\0';
	/*
	* Construct the string specified by "[PREFIX][NAME]=[FMT]".
	*/
	dstp = valbuf;
	buflen = sizeof (valbuf);
	n = strlcpy(dstp, keybuf, buflen);
	if (n >= sizeof (valbuf)) {
	errno = EMSGSIZE;
	zed_log_msg(LOG_WARNING, "Failed to add %s for eid=%llu: %s",
	keybuf, eid, "Exceeded buffer size");
	return (-1);
	}
	dstp += n;
	buflen -= n;

	*dstp++ = '=';
	buflen--;

	if (buflen <= 0) {
	errno = EMSGSIZE;
	zed_log_msg(LOG_WARNING, "Failed to add %s for eid=%llu: %s",
	keybuf, eid, "Exceeded buffer size");
	return (-1);
	}

	va_start(vargs, fmt);
	n = vsnprintf(dstp, buflen, fmt, vargs);
	va_end(vargs);

	if ((n < 0) \|\| (n >= buflen)) {
	errno = EMSGSIZE;
	zed_log_msg(LOG_WARNING, "Failed to add %s for eid=%llu: %s",
	keybuf, eid, "Exceeded buffer size");
	return (-1);
	} else if (zed_strings_add(zsp, keybuf, valbuf) < 0) {
	zed_log_msg(LOG_WARNING, "Failed to add %s for eid=%llu: %s",
	keybuf, eid, strerror(errno));
	return (-1);
	}
	return (0);
	}

	static int
	_zed_event_add_array_err(uint64_t eid, const char *name)
	{
	errno = EMSGSIZE;
	zed_log_msg(LOG_WARNING,
	"Failed to convert nvpair \"%s\" for eid=%llu: "
	"Exceeded buffer size", name, eid);
	return (-1);
	}

	static int
	_zed_event_add_int8_array(uint64_t eid, zed_strings_t *zsp,
	const char prefix, nvpair_t nvp)
	{
	char buf[MAXBUF];
	int buflen = sizeof (buf);
	const char *name;
	int8_t *i8p;
	uint_t nelem;
	uint_t i;
	char *p;
	int n;

	assert((nvp != NULL) && (nvpair_type(nvp) == DATA_TYPE_INT8_ARRAY));

	name = nvpair_name(nvp);
	(void) nvpair_value_int8_array(nvp, &i8p, &nelem);
	for (i = 0, p = buf; (i < nelem) && (buflen > 0); i++) {
	n = snprintf(p, buflen, "%d ", i8p[i]);
	if ((n < 0) \|\| (n >= buflen))
	return (_zed_event_add_array_err(eid, name));
	p += n;
	buflen -= n;
	}
	if (nelem > 0)
	*--p = '\0';

	return (_zed_event_add_var(eid, zsp, prefix, name, "%s", buf));
	}

	static int
	_zed_event_add_uint8_array(uint64_t eid, zed_strings_t *zsp,
	const char prefix, nvpair_t nvp)
	{
	char buf[MAXBUF];
	int buflen = sizeof (buf);
	const char *name;
	uint8_t *u8p;
	uint_t nelem;
	uint_t i;
	char *p;
	int n;

	assert((nvp != NULL) && (nvpair_type(nvp) == DATA_TYPE_UINT8_ARRAY));

	name = nvpair_name(nvp);
	(void) nvpair_value_uint8_array(nvp, &u8p, &nelem);
	for (i = 0, p = buf; (i < nelem) && (buflen > 0); i++) {
	n = snprintf(p, buflen, "%u ", u8p[i]);
	if ((n < 0) \|\| (n >= buflen))
	return (_zed_event_add_array_err(eid, name));
	p += n;
	buflen -= n;
	}
	if (nelem > 0)
	*--p = '\0';

	return (_zed_event_add_var(eid, zsp, prefix, name, "%s", buf));
	}

	static int
	_zed_event_add_int16_array(uint64_t eid, zed_strings_t *zsp,
	const char prefix, nvpair_t nvp)
	{
	char buf[MAXBUF];
	int buflen = sizeof (buf);
	const char *name;
	int16_t *i16p;
	uint_t nelem;
	uint_t i;
	char *p;
	int n;

	assert((nvp != NULL) && (nvpair_type(nvp) == DATA_TYPE_INT16_ARRAY));

	name = nvpair_name(nvp);
	(void) nvpair_value_int16_array(nvp, &i16p, &nelem);
	for (i = 0, p = buf; (i < nelem) && (buflen > 0); i++) {
	n = snprintf(p, buflen, "%d ", i16p[i]);
	if ((n < 0) \|\| (n >= buflen))
	return (_zed_event_add_array_err(eid, name));
	p += n;
	buflen -= n;
	}
	if (nelem > 0)
	*--p = '\0';

	return (_zed_event_add_var(eid, zsp, prefix, name, "%s", buf));
	}

	static int
	_zed_event_add_uint16_array(uint64_t eid, zed_strings_t *zsp,
	const char prefix, nvpair_t nvp)
	{
	char buf[MAXBUF];
	int buflen = sizeof (buf);
	const char *name;
	uint16_t *u16p;
	uint_t nelem;
	uint_t i;
	char *p;
	int n;

	assert((nvp != NULL) && (nvpair_type(nvp) == DATA_TYPE_UINT16_ARRAY));

	name = nvpair_name(nvp);
	(void) nvpair_value_uint16_array(nvp, &u16p, &nelem);
	for (i = 0, p = buf; (i < nelem) && (buflen > 0); i++) {
	n = snprintf(p, buflen, "%u ", u16p[i]);
	if ((n < 0) \|\| (n >= buflen))
	return (_zed_event_add_array_err(eid, name));
	p += n;
	buflen -= n;
	}
	if (nelem > 0)
	*--p = '\0';

	return (_zed_event_add_var(eid, zsp, prefix, name, "%s", buf));
	}

	static int
	_zed_event_add_int32_array(uint64_t eid, zed_strings_t *zsp,
	const char prefix, nvpair_t nvp)
	{
	char buf[MAXBUF];
	int buflen = sizeof (buf);
	const char *name;
	int32_t *i32p;
	uint_t nelem;
	uint_t i;
	char *p;
	int n;

	assert((nvp != NULL) && (nvpair_type(nvp) == DATA_TYPE_INT32_ARRAY));

	name = nvpair_name(nvp);
	(void) nvpair_value_int32_array(nvp, &i32p, &nelem);
	for (i = 0, p = buf; (i < nelem) && (buflen > 0); i++) {
	n = snprintf(p, buflen, "%d ", i32p[i]);
	if ((n < 0) \|\| (n >= buflen))
	return (_zed_event_add_array_err(eid, name));
	p += n;
	buflen -= n;
	}
	if (nelem > 0)
	*--p = '\0';

	return (_zed_event_add_var(eid, zsp, prefix, name, "%s", buf));
	}

	static int
	_zed_event_add_uint32_array(uint64_t eid, zed_strings_t *zsp,
	const char prefix, nvpair_t nvp)
	{
	char buf[MAXBUF];
	int buflen = sizeof (buf);
	const char *name;
	uint32_t *u32p;
	uint_t nelem;
	uint_t i;
	char *p;
	int n;

	assert((nvp != NULL) && (nvpair_type(nvp) == DATA_TYPE_UINT32_ARRAY));

	name = nvpair_name(nvp);
	(void) nvpair_value_uint32_array(nvp, &u32p, &nelem);
	for (i = 0, p = buf; (i < nelem) && (buflen > 0); i++) {
	n = snprintf(p, buflen, "%u ", u32p[i]);
	if ((n < 0) \|\| (n >= buflen))
	return (_zed_event_add_array_err(eid, name));
	p += n;
	buflen -= n;
	}
	if (nelem > 0)
	*--p = '\0';

	return (_zed_event_add_var(eid, zsp, prefix, name, "%s", buf));
	}

	static int
	_zed_event_add_int64_array(uint64_t eid, zed_strings_t *zsp,
	const char prefix, nvpair_t nvp)
	{
	char buf[MAXBUF];
	int buflen = sizeof (buf);
	const char *name;
	int64_t *i64p;
	uint_t nelem;
	uint_t i;
	char *p;
	int n;

	assert((nvp != NULL) && (nvpair_type(nvp) == DATA_TYPE_INT64_ARRAY));

	name = nvpair_name(nvp);
	(void) nvpair_value_int64_array(nvp, &i64p, &nelem);
	for (i = 0, p = buf; (i < nelem) && (buflen > 0); i++) {
	n = snprintf(p, buflen, "%lld ", (u_longlong_t)i64p[i]);
	if ((n < 0) \|\| (n >= buflen))
	return (_zed_event_add_array_err(eid, name));
	p += n;
	buflen -= n;
	}
	if (nelem > 0)
	*--p = '\0';

	return (_zed_event_add_var(eid, zsp, prefix, name, "%s", buf));
	}

	static int
	_zed_event_add_uint64_array(uint64_t eid, zed_strings_t *zsp,
	const char prefix, nvpair_t nvp)
	{
	char buf[MAXBUF];
	int buflen = sizeof (buf);
	const char *name;
	const char *fmt;
	uint64_t *u64p;
	uint_t nelem;
	uint_t i;
	char *p;
	int n;

	assert((nvp != NULL) && (nvpair_type(nvp) == DATA_TYPE_UINT64_ARRAY));

	name = nvpair_name(nvp);
	fmt = _zed_event_value_is_hex(name) ? "0x%.16llX " : "%llu ";
	(void) nvpair_value_uint64_array(nvp, &u64p, &nelem);
	for (i = 0, p = buf; (i < nelem) && (buflen > 0); i++) {
	n = snprintf(p, buflen, fmt, (u_longlong_t)u64p[i]);
	if ((n < 0) \|\| (n >= buflen))
	return (_zed_event_add_array_err(eid, name));
	p += n;
	buflen -= n;
	}
	if (nelem > 0)
	*--p = '\0';

	return (_zed_event_add_var(eid, zsp, prefix, name, "%s", buf));
	}

	static int
	_zed_event_add_string_array(uint64_t eid, zed_strings_t *zsp,
	const char prefix, nvpair_t nvp)
	{
	char buf[MAXBUF];
	int buflen = sizeof (buf);
	const char *name;
	char **strp;
	uint_t nelem;
	uint_t i;
	char *p;
	int n;

	assert((nvp != NULL) && (nvpair_type(nvp) == DATA_TYPE_STRING_ARRAY));

	name = nvpair_name(nvp);
	(void) nvpair_value_string_array(nvp, &strp, &nelem);
	for (i = 0, p = buf; (i < nelem) && (buflen > 0); i++) {
	n = snprintf(p, buflen, "%s ", strp[i] ? strp[i] : "<NULL>");
	if ((n < 0) \|\| (n >= buflen))
	return (_zed_event_add_array_err(eid, name));
	p += n;
	buflen -= n;
	}
	if (nelem > 0)
	*--p = '\0';

	return (_zed_event_add_var(eid, zsp, prefix, name, "%s", buf));
	}

	/*
	* Convert the nvpair [nvp] to a string which is added to the environment
	* of the child process.
	* Return 0 on success, -1 on error.
	*
	* FIXME: Refactor with cmd/zpool/zpool_main.c:zpool_do_events_nvprint()?
	*/
	static void
	_zed_event_add_nvpair(uint64_t eid, zed_strings_t zsp, nvpair_t nvp)
	{
	const char *name;
	data_type_t type;
	const char *prefix = ZEVENT_VAR_PREFIX;
	boolean_t b;
	double d;
	uint8_t i8;
	uint16_t i16;
	uint32_t i32;
	uint64_t i64;
	char *str;

	assert(zsp != NULL);
	assert(nvp != NULL);

	name = nvpair_name(nvp);
	type = nvpair_type(nvp);

	switch (type) {
	case DATA_TYPE_BOOLEAN:
	_zed_event_add_var(eid, zsp, prefix, name, "%s", "1");
	break;
	case DATA_TYPE_BOOLEAN_VALUE:
	(void) nvpair_value_boolean_value(nvp, &b);
	_zed_event_add_var(eid, zsp, prefix, name, "%s", b ? "1" : "0");
	break;
	case DATA_TYPE_BYTE:
	(void) nvpair_value_byte(nvp, &i8);
	_zed_event_add_var(eid, zsp, prefix, name, "%d", i8);
	break;
	case DATA_TYPE_INT8:
	(void) nvpair_value_int8(nvp, (int8_t *)&i8);
	_zed_event_add_var(eid, zsp, prefix, name, "%d", i8);
	break;
	case DATA_TYPE_UINT8:
	(void) nvpair_value_uint8(nvp, &i8);
	_zed_event_add_var(eid, zsp, prefix, name, "%u", i8);
	break;
	case DATA_TYPE_INT16:
	(void) nvpair_value_int16(nvp, (int16_t *)&i16);
	_zed_event_add_var(eid, zsp, prefix, name, "%d", i16);
	break;
	case DATA_TYPE_UINT16:
	(void) nvpair_value_uint16(nvp, &i16);
	_zed_event_add_var(eid, zsp, prefix, name, "%u", i16);
	break;
	case DATA_TYPE_INT32:
	(void) nvpair_value_int32(nvp, (int32_t *)&i32);
	_zed_event_add_var(eid, zsp, prefix, name, "%d", i32);
	break;
	case DATA_TYPE_UINT32:
	(void) nvpair_value_uint32(nvp, &i32);
	_zed_event_add_var(eid, zsp, prefix, name, "%u", i32);
	break;
	case DATA_TYPE_INT64:
	(void) nvpair_value_int64(nvp, (int64_t *)&i64);
	_zed_event_add_var(eid, zsp, prefix, name,
	"%lld", (longlong_t)i64);
	break;
	case DATA_TYPE_UINT64:
	(void) nvpair_value_uint64(nvp, &i64);
	_zed_event_add_var(eid, zsp, prefix, name,
	(_zed_event_value_is_hex(name) ? "0x%.16llX" : "%llu"),
	(u_longlong_t)i64);
	/*
	* shadow readable strings for vdev state pairs
	*/
	if (strcmp(name, FM_EREPORT_PAYLOAD_ZFS_VDEV_STATE) == 0 \|\|
	strcmp(name, FM_EREPORT_PAYLOAD_ZFS_VDEV_LASTSTATE) == 0) {
	char alt[32];

	(void) snprintf(alt, sizeof (alt), "%s_str", name);
	_zed_event_add_var(eid, zsp, prefix, alt, "%s",
	zpool_state_to_name(i64, VDEV_AUX_NONE));
	} else
	/*
	* shadow readable strings for pool state
	*/
	if (strcmp(name, FM_EREPORT_PAYLOAD_ZFS_POOL_STATE) == 0) {
	char alt[32];

	(void) snprintf(alt, sizeof (alt), "%s_str", name);
	_zed_event_add_var(eid, zsp, prefix, alt, "%s",
	zpool_pool_state_to_name(i64));
	}
	break;
	case DATA_TYPE_DOUBLE:
	(void) nvpair_value_double(nvp, &d);
	_zed_event_add_var(eid, zsp, prefix, name, "%g", d);
	break;
	case DATA_TYPE_HRTIME:
	(void) nvpair_value_hrtime(nvp, (hrtime_t *)&i64);
	_zed_event_add_var(eid, zsp, prefix, name,
	"%llu", (u_longlong_t)i64);
	break;
	case DATA_TYPE_NVLIST:
	_zed_event_add_var(eid, zsp, prefix, name,
	"%s", "_NOT_IMPLEMENTED_"); /* FIXME */
	break;
	case DATA_TYPE_STRING:
	(void) nvpair_value_string(nvp, &str);
	_zed_event_add_var(eid, zsp, prefix, name,
	"%s", (str ? str : "<NULL>"));
	break;
	case DATA_TYPE_BOOLEAN_ARRAY:
	_zed_event_add_var(eid, zsp, prefix, name,
	"%s", "_NOT_IMPLEMENTED_"); /* FIXME */
	break;
	case DATA_TYPE_BYTE_ARRAY:
	_zed_event_add_var(eid, zsp, prefix, name,
	"%s", "_NOT_IMPLEMENTED_"); /* FIXME */
	break;
	case DATA_TYPE_INT8_ARRAY:
	_zed_event_add_int8_array(eid, zsp, prefix, nvp);
	break;
	case DATA_TYPE_UINT8_ARRAY:
	_zed_event_add_uint8_array(eid, zsp, prefix, nvp);
	break;
	case DATA_TYPE_INT16_ARRAY:
	_zed_event_add_int16_array(eid, zsp, prefix, nvp);
	break;
	case DATA_TYPE_UINT16_ARRAY:
	_zed_event_add_uint16_array(eid, zsp, prefix, nvp);
	break;
	case DATA_TYPE_INT32_ARRAY:
	_zed_event_add_int32_array(eid, zsp, prefix, nvp);
	break;
	case DATA_TYPE_UINT32_ARRAY:
	_zed_event_add_uint32_array(eid, zsp, prefix, nvp);
	break;
	case DATA_TYPE_INT64_ARRAY:
	_zed_event_add_int64_array(eid, zsp, prefix, nvp);
	break;
	case DATA_TYPE_UINT64_ARRAY:
	_zed_event_add_uint64_array(eid, zsp, prefix, nvp);
	break;
	case DATA_TYPE_STRING_ARRAY:
	_zed_event_add_string_array(eid, zsp, prefix, nvp);
	break;
	case DATA_TYPE_NVLIST_ARRAY:
	_zed_event_add_var(eid, zsp, prefix, name,
	"%s", "_NOT_IMPLEMENTED_"); /* FIXME */
	break;
	default:
	errno = EINVAL;
	zed_log_msg(LOG_WARNING,
	"Failed to convert nvpair \"%s\" for eid=%llu: "
	"Unrecognized type=%u", name, eid, (unsigned int) type);
	break;
	}
	}

	/*
	* Restrict various environment variables to safe and sane values
	* when constructing the environment for the child process, unless
	* we're running with a custom $PATH (like under the ZFS test suite).
	*
	* Reference: Secure Programming Cookbook by Viega & Messier, Section 1.1.
	*/
	static void
	_zed_event_add_env_restrict(uint64_t eid, zed_strings_t *zsp,
	const char *path)
	{
	const char *env_restrict[][2] = {
	{ "IFS", " \t\n" },
	{ "PATH", _PATH_STDPATH },
	{ "ZDB", SBINDIR "/zdb" },
	{ "ZED", SBINDIR "/zed" },
	{ "ZFS", SBINDIR "/zfs" },
	{ "ZINJECT", SBINDIR "/zinject" },
	{ "ZPOOL", SBINDIR "/zpool" },
	{ "ZFS_ALIAS", ZFS_META_ALIAS },
	{ "ZFS_VERSION", ZFS_META_VERSION },
	{ "ZFS_RELEASE", ZFS_META_RELEASE },
	{ NULL, NULL }
	};

	/*
	* If we have a custom $PATH, use the default ZFS binary locations
	* instead of the hard-coded ones.
	*/
	const char *env_path[][2] = {
	{ "IFS", " \t\n" },
	{ "PATH", NULL }, /* $PATH copied in later on */
	{ "ZDB", "zdb" },
	{ "ZED", "zed" },
	{ "ZFS", "zfs" },
	{ "ZINJECT", "zinject" },
	{ "ZPOOL", "zpool" },
	{ "ZFS_ALIAS", ZFS_META_ALIAS },
	{ "ZFS_VERSION", ZFS_META_VERSION },
	{ "ZFS_RELEASE", ZFS_META_RELEASE },
	{ NULL, NULL }
	};
	const char (pa)[2];

	assert(zsp != NULL);

	pa = path != NULL ? env_path : env_restrict;

	for (; (pa); pa++) {
	/* Use our custom $PATH if we have one */
	if (path != NULL && strcmp((*pa)[0], "PATH") == 0)
	(*pa)[1] = path;

	_zed_event_add_var(eid, zsp, NULL, (pa)[0], "%s", (pa)[1]);
	}
	}

	/*
	* Preserve specified variables from the parent environment
	* when constructing the environment for the child process.
	*
	* Reference: Secure Programming Cookbook by Viega & Messier, Section 1.1.
	*/
	static void
	_zed_event_add_env_preserve(uint64_t eid, zed_strings_t *zsp)
	{
	const char *env_preserve[] = {
	"TZ",
	NULL
	};
	const char **keyp;
	const char *val;

	assert(zsp != NULL);

	for (keyp = env_preserve; *keyp; keyp++) {
	if ((val = getenv(*keyp)))
	_zed_event_add_var(eid, zsp, NULL, *keyp, "%s", val);
	}
	}

	/*
	* Compute the "subclass" by removing the first 3 components of [class]
	* (which will always be of the form "*.fs.zfs"). Return a pointer inside
	* the string [class], or NULL if insufficient components exist.
	*/
	static const char *
	_zed_event_get_subclass(const char *class)
	{
	const char *p;
	int i;

	if (!class)
	return (NULL);

	p = class;
	for (i = 0; i < 3; i++) {
	p = strchr(p, '.');
	if (!p)
	break;
	p++;
	}
	return (p);
	}

	/*
	* Convert the zevent time from a 2-element array of 64b integers
	* into a more convenient form:
	* - TIME_SECS is the second component of the time.
	* - TIME_NSECS is the nanosecond component of the time.
	* - TIME_STRING is an almost-RFC3339-compliant string representation.
	*/
	static void
	_zed_event_add_time_strings(uint64_t eid, zed_strings_t *zsp, int64_t etime[])
	{
	struct tm *stp;
	char buf[32];

	assert(zsp != NULL);
	assert(etime != NULL);

	_zed_event_add_var(eid, zsp, ZEVENT_VAR_PREFIX, "TIME_SECS",
	"%lld", (long long int) etime[0]);
	_zed_event_add_var(eid, zsp, ZEVENT_VAR_PREFIX, "TIME_NSECS",
	"%lld", (long long int) etime[1]);

	if (!(stp = localtime((const time_t *) &etime[0]))) {
	zed_log_msg(LOG_WARNING, "Failed to add %s%s for eid=%llu: %s",
	ZEVENT_VAR_PREFIX, "TIME_STRING", eid, "localtime error");
	} else if (!strftime(buf, sizeof (buf), "%Y-%m-%d %H:%M:%S%z", stp)) {
	zed_log_msg(LOG_WARNING, "Failed to add %s%s for eid=%llu: %s",
	ZEVENT_VAR_PREFIX, "TIME_STRING", eid, "strftime error");
	} else {
	_zed_event_add_var(eid, zsp, ZEVENT_VAR_PREFIX, "TIME_STRING",
	"%s", buf);
	}
	}

	/*
	* Service the next zevent, blocking until one is available.
	*/
	int
	zed_event_service(struct zed_conf *zcp)
	{
	nvlist_t *nvl;
	nvpair_t *nvp;
	int n_dropped;
	zed_strings_t *zsp;
	uint64_t eid;
	int64_t *etime;
	uint_t nelem;
	char *class;
	const char *subclass;
	int rv;

	if (!zcp) {
	errno = EINVAL;
	zed_log_msg(LOG_ERR, "Failed to service zevent: %s",
	strerror(errno));
	return (EINVAL);
	}
	rv = zpool_events_next(zcp->zfs_hdl, &nvl, &n_dropped, ZEVENT_NONE,
	zcp->zevent_fd);

	if ((rv != 0) \|\| !nvl)
	return (errno);

	if (n_dropped > 0) {
	zed_log_msg(LOG_WARNING, "Missed %d events", n_dropped);
	/*
	* FIXME: Increase max size of event nvlist in
	* /sys/module/zfs/parameters/zfs_zevent_len_max ?
	*/
	}
	if (nvlist_lookup_uint64(nvl, "eid", &eid) != 0) {
	zed_log_msg(LOG_WARNING, "Failed to lookup zevent eid");
	} else if (nvlist_lookup_int64_array(
	nvl, "time", &etime, &nelem) != 0) {
	zed_log_msg(LOG_WARNING,
	"Failed to lookup zevent time (eid=%llu)", eid);
	} else if (nelem != 2) {
	zed_log_msg(LOG_WARNING,
	"Failed to lookup zevent time (eid=%llu, nelem=%u)",
	eid, nelem);
	} else if (nvlist_lookup_string(nvl, "class", &class) != 0) {
	zed_log_msg(LOG_WARNING,
	"Failed to lookup zevent class (eid=%llu)", eid);
	} else {
	/* let internal modules see this event first */
	zfs_agent_post_event(class, NULL, nvl);

	zsp = zed_strings_create();

	nvp = NULL;
	while ((nvp = nvlist_next_nvpair(nvl, nvp)))
	_zed_event_add_nvpair(eid, zsp, nvp);

	_zed_event_add_env_restrict(eid, zsp, zcp->path);
	_zed_event_add_env_preserve(eid, zsp);

	_zed_event_add_var(eid, zsp, ZED_VAR_PREFIX, "PID",
	"%d", (int)getpid());
	_zed_event_add_var(eid, zsp, ZED_VAR_PREFIX, "ZEDLET_DIR",
	"%s", zcp->zedlet_dir);
	subclass = _zed_event_get_subclass(class);
	_zed_event_add_var(eid, zsp, ZEVENT_VAR_PREFIX, "SUBCLASS",
	"%s", (subclass ? subclass : class));

	_zed_event_add_time_strings(eid, zsp, etime);

	zed_exec_process(eid, class, subclass,
	zcp->zedlet_dir, zcp->zedlets, zsp, zcp->zevent_fd);

	zed_conf_write_state(zcp, eid, etime);

	zed_strings_destroy(zsp);
	}
	nvlist_free(nvl);
	return (0);
	}
	Index: vendor-sys/openzfs/dist/cmd/zed/zed_event.h
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zed/zed_event.h (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zed/zed_event.h (revision 366774)
	@@ -1,29 +1,29 @@
	/*
	- * This file is part of the ZFS Event Daemon (ZED)
	- * for ZFS on Linux (ZoL) <http://zfsonlinux.org/>.
	+ * This file is part of the ZFS Event Daemon (ZED).
	+ *
	* Developed at Lawrence Livermore National Laboratory (LLNL-CODE-403049).
	* Copyright (C) 2013-2014 Lawrence Livermore National Security, LLC.
	* Refer to the ZoL git commit log for authoritative copyright attribution.
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License Version 1.0 (CDDL-1.0).
	* You can obtain a copy of the license from the top-level file
	* "OPENSOLARIS.LICENSE" or at <http://opensource.org/licenses/CDDL-1.0>.
	* You may not use this file except in compliance with the license.
	*/

	#ifndef ZED_EVENT_H
	#define ZED_EVENT_H

	#include <stdint.h>

	int zed_event_init(struct zed_conf *zcp);

	void zed_event_fini(struct zed_conf *zcp);

	int zed_event_seek(struct zed_conf *zcp, uint64_t saved_eid,
	int64_t saved_etime[]);

	int zed_event_service(struct zed_conf *zcp);

	#endif /* !ZED_EVENT_H */
	Index: vendor-sys/openzfs/dist/cmd/zed/zed_exec.c
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zed/zed_exec.c (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zed/zed_exec.c (revision 366774)
	@@ -1,232 +1,232 @@
	/*
	- * This file is part of the ZFS Event Daemon (ZED)
	- * for ZFS on Linux (ZoL) <http://zfsonlinux.org/>.
	+ * This file is part of the ZFS Event Daemon (ZED).
	+ *
	* Developed at Lawrence Livermore National Laboratory (LLNL-CODE-403049).
	* Copyright (C) 2013-2014 Lawrence Livermore National Security, LLC.
	* Refer to the ZoL git commit log for authoritative copyright attribution.
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License Version 1.0 (CDDL-1.0).
	* You can obtain a copy of the license from the top-level file
	* "OPENSOLARIS.LICENSE" or at <http://opensource.org/licenses/CDDL-1.0>.
	* You may not use this file except in compliance with the license.
	*/

	#include <assert.h>
	#include <ctype.h>
	#include <errno.h>
	#include <fcntl.h>
	#include <stdlib.h>
	#include <string.h>
	#include <sys/stat.h>
	#include <sys/wait.h>
	#include <time.h>
	#include <unistd.h>
	#include "zed_exec.h"
	#include "zed_file.h"
	#include "zed_log.h"
	#include "zed_strings.h"

	#define ZEVENT_FILENO 3

	/*
	* Create an environment string array for passing to execve() using the
	* NAME=VALUE strings in container [zsp].
	* Return a newly-allocated environment, or NULL on error.
	*/
	static char **
	_zed_exec_create_env(zed_strings_t *zsp)
	{
	int num_ptrs;
	int buflen;
	char *buf;
	char **pp;
	char *p;
	const char *q;
	int i;
	int len;

	num_ptrs = zed_strings_count(zsp) + 1;
	buflen = num_ptrs * sizeof (char *);
	for (q = zed_strings_first(zsp); q; q = zed_strings_next(zsp))
	buflen += strlen(q) + 1;

	buf = calloc(1, buflen);
	if (!buf)
	return (NULL);

	pp = (char **)buf;
	p = buf + (num_ptrs * sizeof (char *));
	i = 0;
	for (q = zed_strings_first(zsp); q; q = zed_strings_next(zsp)) {
	pp[i] = p;
	len = strlen(q) + 1;
	memcpy(p, q, len);
	p += len;
	i++;
	}
	pp[i] = NULL;
	assert(buf + buflen == p);
	return ((char **)buf);
	}

	/*
	* Fork a child process to handle event [eid]. The program [prog]
	* in directory [dir] is executed with the environment [env].
	*
	* The file descriptor [zfd] is the zevent_fd used to track the
	* current cursor location within the zevent nvlist.
	*/
	static void
	_zed_exec_fork_child(uint64_t eid, const char dir, const char prog,
	char *env[], int zfd)
	{
	char path[PATH_MAX];
	int n;
	pid_t pid;
	int fd;
	pid_t wpid;
	int status;

	assert(dir != NULL);
	assert(prog != NULL);
	assert(env != NULL);
	assert(zfd >= 0);

	n = snprintf(path, sizeof (path), "%s/%s", dir, prog);
	if ((n < 0) \|\| (n >= sizeof (path))) {
	zed_log_msg(LOG_WARNING,
	"Failed to fork \"%s\" for eid=%llu: %s",
	prog, eid, strerror(ENAMETOOLONG));
	return;
	}
	pid = fork();
	if (pid < 0) {
	zed_log_msg(LOG_WARNING,
	"Failed to fork \"%s\" for eid=%llu: %s",
	prog, eid, strerror(errno));
	return;
	} else if (pid == 0) {
	(void) umask(022);
	if ((fd = open("/dev/null", O_RDWR)) != -1) {
	(void) dup2(fd, STDIN_FILENO);
	(void) dup2(fd, STDOUT_FILENO);
	(void) dup2(fd, STDERR_FILENO);
	}
	(void) dup2(zfd, ZEVENT_FILENO);
	zed_file_close_from(ZEVENT_FILENO + 1);
	execle(path, prog, NULL, env);
	_exit(127);
	}

	/* parent process */

	zed_log_msg(LOG_INFO, "Invoking \"%s\" eid=%llu pid=%d",
	prog, eid, pid);

	/* FIXME: Timeout rogue child processes with sigalarm? */

	/*
	* Wait for child process using WNOHANG to limit
	* the time spent waiting to 10 seconds (10,000ms).
	*/
	for (n = 0; n < 1000; n++) {
	wpid = waitpid(pid, &status, WNOHANG);
	if (wpid == (pid_t)-1) {
	if (errno == EINTR)
	continue;
	zed_log_msg(LOG_WARNING,
	"Failed to wait for \"%s\" eid=%llu pid=%d",
	prog, eid, pid);
	break;
	} else if (wpid == 0) {
	struct timespec t;

	/* child still running */
	t.tv_sec = 0;
	t.tv_nsec = 10000000; /* 10ms */
	(void) nanosleep(&t, NULL);
	continue;
	}

	if (WIFEXITED(status)) {
	zed_log_msg(LOG_INFO,
	"Finished \"%s\" eid=%llu pid=%d exit=%d",
	prog, eid, pid, WEXITSTATUS(status));
	} else if (WIFSIGNALED(status)) {
	zed_log_msg(LOG_INFO,
	"Finished \"%s\" eid=%llu pid=%d sig=%d/%s",
	prog, eid, pid, WTERMSIG(status),
	strsignal(WTERMSIG(status)));
	} else {
	zed_log_msg(LOG_INFO,
	"Finished \"%s\" eid=%llu pid=%d status=0x%X",
	prog, eid, (unsigned int) status);
	}
	break;
	}

	/*
	* kill child process after 10 seconds
	*/
	if (wpid == 0) {
	zed_log_msg(LOG_WARNING, "Killing hung \"%s\" pid=%d",
	prog, pid);
	(void) kill(pid, SIGKILL);
	}
	}

	/*
	* Process the event [eid] by synchronously invoking all zedlets with a
	* matching class prefix.
	*
	* Each executable in [zedlets] from the directory [dir] is matched against
	* the event's [class], [subclass], and the "all" class (which matches
	* all events). Every zedlet with a matching class prefix is invoked.
	* The NAME=VALUE strings in [envs] will be passed to the zedlet as
	* environment variables.
	*
	* The file descriptor [zfd] is the zevent_fd used to track the
	* current cursor location within the zevent nvlist.
	*
	* Return 0 on success, -1 on error.
	*/
	int
	zed_exec_process(uint64_t eid, const char class, const char subclass,
	const char dir, zed_strings_t zedlets, zed_strings_t *envs, int zfd)
	{
	const char *class_strings[4];
	const char *allclass = "all";
	const char **csp;
	const char *z;
	char **e;
	int n;

	if (!dir \|\| !zedlets \|\| !envs \|\| zfd < 0)
	return (-1);

	csp = class_strings;

	if (class)
	*csp++ = class;

	if (subclass)
	*csp++ = subclass;

	if (allclass)
	*csp++ = allclass;

	*csp = NULL;

	e = _zed_exec_create_env(envs);

	for (z = zed_strings_first(zedlets); z; z = zed_strings_next(zedlets)) {
	for (csp = class_strings; *csp; csp++) {
	n = strlen(*csp);
	if ((strncmp(z, *csp, n) == 0) && !isalpha(z[n]))
	_zed_exec_fork_child(eid, dir, z, e, zfd);
	}
	}
	free(e);
	return (0);
	}
	Index: vendor-sys/openzfs/dist/cmd/zed/zed_exec.h
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zed/zed_exec.h (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zed/zed_exec.h (revision 366774)
	@@ -1,25 +1,25 @@
	/*
	- * This file is part of the ZFS Event Daemon (ZED)
	- * for ZFS on Linux (ZoL) <http://zfsonlinux.org/>.
	+ * This file is part of the ZFS Event Daemon (ZED).
	+ *
	* Developed at Lawrence Livermore National Laboratory (LLNL-CODE-403049).
	* Copyright (C) 2013-2014 Lawrence Livermore National Security, LLC.
	* Refer to the ZoL git commit log for authoritative copyright attribution.
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License Version 1.0 (CDDL-1.0).
	* You can obtain a copy of the license from the top-level file
	* "OPENSOLARIS.LICENSE" or at <http://opensource.org/licenses/CDDL-1.0>.
	* You may not use this file except in compliance with the license.
	*/

	#ifndef ZED_EXEC_H
	#define ZED_EXEC_H

	#include <stdint.h>
	#include "zed_strings.h"

	int zed_exec_process(uint64_t eid, const char class, const char subclass,
	const char dir, zed_strings_t zedlets, zed_strings_t *envs,
	int zevent_fd);

	#endif /* !ZED_EXEC_H */
	Index: vendor-sys/openzfs/dist/cmd/zed/zed_file.c
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zed/zed_file.c (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zed/zed_file.c (revision 366774)
	@@ -1,217 +1,217 @@
	/*
	- * This file is part of the ZFS Event Daemon (ZED)
	- * for ZFS on Linux (ZoL) <http://zfsonlinux.org/>.
	+ * This file is part of the ZFS Event Daemon (ZED).
	+ *
	* Developed at Lawrence Livermore National Laboratory (LLNL-CODE-403049).
	* Copyright (C) 2013-2014 Lawrence Livermore National Security, LLC.
	* Refer to the ZoL git commit log for authoritative copyright attribution.
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License Version 1.0 (CDDL-1.0).
	* You can obtain a copy of the license from the top-level file
	* "OPENSOLARIS.LICENSE" or at <http://opensource.org/licenses/CDDL-1.0>.
	* You may not use this file except in compliance with the license.
	*/

	#include <errno.h>
	#include <fcntl.h>
	#include <limits.h>
	#include <string.h>
	#include <sys/resource.h>
	#include <sys/stat.h>
	#include <sys/types.h>
	#include <unistd.h>
	#include "zed_file.h"
	#include "zed_log.h"

	/*
	* Read up to [n] bytes from [fd] into [buf].
	* Return the number of bytes read, 0 on EOF, or -1 on error.
	*/
	ssize_t
	zed_file_read_n(int fd, void *buf, size_t n)
	{
	unsigned char *p;
	size_t n_left;
	ssize_t n_read;

	p = buf;
	n_left = n;
	while (n_left > 0) {
	if ((n_read = read(fd, p, n_left)) < 0) {
	if (errno == EINTR)
	continue;
	else
	return (-1);

	} else if (n_read == 0) {
	break;
	}
	n_left -= n_read;
	p += n_read;
	}
	return (n - n_left);
	}

	/*
	* Write [n] bytes from [buf] out to [fd].
	* Return the number of bytes written, or -1 on error.
	*/
	ssize_t
	zed_file_write_n(int fd, void *buf, size_t n)
	{
	const unsigned char *p;
	size_t n_left;
	ssize_t n_written;

	p = buf;
	n_left = n;
	while (n_left > 0) {
	if ((n_written = write(fd, p, n_left)) < 0) {
	if (errno == EINTR)
	continue;
	else
	return (-1);

	}
	n_left -= n_written;
	p += n_written;
	}
	return (n);
	}

	/*
	* Set an exclusive advisory lock on the open file descriptor [fd].
	* Return 0 on success, 1 if a conflicting lock is held by another process,
	* or -1 on error (with errno set).
	*/
	int
	zed_file_lock(int fd)
	{
	struct flock lock;

	if (fd < 0) {
	errno = EBADF;
	return (-1);
	}
	lock.l_type = F_WRLCK;
	lock.l_whence = SEEK_SET;
	lock.l_start = 0;
	lock.l_len = 0;

	if (fcntl(fd, F_SETLK, &lock) < 0) {
	if ((errno == EACCES) \|\| (errno == EAGAIN))
	return (1);

	return (-1);
	}
	return (0);
	}

	/*
	* Release an advisory lock held on the open file descriptor [fd].
	* Return 0 on success, or -1 on error (with errno set).
	*/
	int
	zed_file_unlock(int fd)
	{
	struct flock lock;

	if (fd < 0) {
	errno = EBADF;
	return (-1);
	}
	lock.l_type = F_UNLCK;
	lock.l_whence = SEEK_SET;
	lock.l_start = 0;
	lock.l_len = 0;

	if (fcntl(fd, F_SETLK, &lock) < 0)
	return (-1);

	return (0);
	}

	/*
	* Test whether an exclusive advisory lock could be obtained for the open
	* file descriptor [fd].
	* Return 0 if the file is not locked, >0 for the PID of another process
	* holding a conflicting lock, or -1 on error (with errno set).
	*/
	pid_t
	zed_file_is_locked(int fd)
	{
	struct flock lock;

	if (fd < 0) {
	errno = EBADF;
	return (-1);
	}
	lock.l_type = F_WRLCK;
	lock.l_whence = SEEK_SET;
	lock.l_start = 0;
	lock.l_len = 0;

	if (fcntl(fd, F_GETLK, &lock) < 0)
	return (-1);

	if (lock.l_type == F_UNLCK)
	return (0);

	return (lock.l_pid);
	}

	/*
	* Close all open file descriptors greater than or equal to [lowfd].
	* Any errors encountered while closing file descriptors are ignored.
	*/
	void
	zed_file_close_from(int lowfd)
	{
	const int maxfd_def = 256;
	int errno_bak;
	struct rlimit rl;
	int maxfd;
	int fd;

	errno_bak = errno;

	if (getrlimit(RLIMIT_NOFILE, &rl) < 0) {
	maxfd = maxfd_def;
	} else if (rl.rlim_max == RLIM_INFINITY) {
	maxfd = maxfd_def;
	} else {
	maxfd = rl.rlim_max;
	}
	for (fd = lowfd; fd < maxfd; fd++)
	(void) close(fd);

	errno = errno_bak;
	}

	/*
	* Set the CLOEXEC flag on file descriptor [fd] so it will be automatically
	* closed upon successful execution of one of the exec functions.
	* Return 0 on success, or -1 on error.
	*
	* FIXME: No longer needed?
	*/
	int
	zed_file_close_on_exec(int fd)
	{
	int flags;

	if (fd < 0) {
	errno = EBADF;
	return (-1);
	}
	flags = fcntl(fd, F_GETFD);
	if (flags == -1)
	return (-1);

	flags \|= FD_CLOEXEC;

	if (fcntl(fd, F_SETFD, flags) == -1)
	return (-1);

	return (0);
	}
	Index: vendor-sys/openzfs/dist/cmd/zed/zed_file.h
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zed/zed_file.h (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zed/zed_file.h (revision 366774)
	@@ -1,35 +1,35 @@
	/*
	- * This file is part of the ZFS Event Daemon (ZED)
	- * for ZFS on Linux (ZoL) <http://zfsonlinux.org/>.
	+ * This file is part of the ZFS Event Daemon (ZED).
	+ *
	* Developed at Lawrence Livermore National Laboratory (LLNL-CODE-403049).
	* Copyright (C) 2013-2014 Lawrence Livermore National Security, LLC.
	* Refer to the ZoL git commit log for authoritative copyright attribution.
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License Version 1.0 (CDDL-1.0).
	* You can obtain a copy of the license from the top-level file
	* "OPENSOLARIS.LICENSE" or at <http://opensource.org/licenses/CDDL-1.0>.
	* You may not use this file except in compliance with the license.
	*/

	#ifndef ZED_FILE_H
	#define ZED_FILE_H

	#include <sys/types.h>
	#include <unistd.h>

	ssize_t zed_file_read_n(int fd, void *buf, size_t n);

	ssize_t zed_file_write_n(int fd, void *buf, size_t n);

	int zed_file_lock(int fd);

	int zed_file_unlock(int fd);

	pid_t zed_file_is_locked(int fd);

	void zed_file_close_from(int fd);

	int zed_file_close_on_exec(int fd);

	#endif /* !ZED_FILE_H */
	Index: vendor-sys/openzfs/dist/cmd/zed/zed_log.c
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zed/zed_log.c (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zed/zed_log.c (revision 366774)
	@@ -1,256 +1,256 @@
	/*
	- * This file is part of the ZFS Event Daemon (ZED)
	- * for ZFS on Linux (ZoL) <http://zfsonlinux.org/>.
	+ * This file is part of the ZFS Event Daemon (ZED).
	+ *
	* Developed at Lawrence Livermore National Laboratory (LLNL-CODE-403049).
	* Copyright (C) 2013-2014 Lawrence Livermore National Security, LLC.
	* Refer to the ZoL git commit log for authoritative copyright attribution.
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License Version 1.0 (CDDL-1.0).
	* You can obtain a copy of the license from the top-level file
	* "OPENSOLARIS.LICENSE" or at <http://opensource.org/licenses/CDDL-1.0>.
	* You may not use this file except in compliance with the license.
	*/

	#include <assert.h>
	#include <errno.h>
	#include <stdarg.h>
	#include <stdio.h>
	#include <stdlib.h>
	#include <string.h>
	#include <sys/types.h>
	#include <syslog.h>
	#include <unistd.h>
	#include "zed_log.h"

	#define ZED_LOG_MAX_LOG_LEN 1024

	static struct {
	unsigned do_stderr:1;
	unsigned do_syslog:1;
	const char *identity;
	int priority;
	int pipe_fd[2];
	} _ctx;

	/*
	* Initialize the logging subsystem.
	*/
	void
	zed_log_init(const char *identity)
	{
	if (identity) {
	const char *p = strrchr(identity, '/');
	_ctx.identity = (p != NULL) ? p + 1 : identity;
	} else {
	_ctx.identity = NULL;
	}
	_ctx.pipe_fd[0] = -1;
	_ctx.pipe_fd[1] = -1;
	}

	/*
	* Shutdown the logging subsystem.
	*/
	void
	zed_log_fini(void)
	{
	zed_log_stderr_close();
	zed_log_syslog_close();
	}

	/*
	* Create pipe for communicating daemonization status between the parent and
	* child processes across the double-fork().
	*/
	void
	zed_log_pipe_open(void)
	{
	if ((_ctx.pipe_fd[0] != -1) \|\| (_ctx.pipe_fd[1] != -1))
	zed_log_die("Invalid use of zed_log_pipe_open in PID %d",
	(int)getpid());

	if (pipe(_ctx.pipe_fd) < 0)
	zed_log_die("Failed to create daemonize pipe in PID %d: %s",
	(int)getpid(), strerror(errno));
	}

	/*
	* Close the read-half of the daemonize pipe.
	*
	* This should be called by the child after fork()ing from the parent since
	* the child will never read from this pipe.
	*/
	void
	zed_log_pipe_close_reads(void)
	{
	if (_ctx.pipe_fd[0] < 0)
	zed_log_die(
	"Invalid use of zed_log_pipe_close_reads in PID %d",
	(int)getpid());

	if (close(_ctx.pipe_fd[0]) < 0)
	zed_log_die(
	"Failed to close reads on daemonize pipe in PID %d: %s",
	(int)getpid(), strerror(errno));

	_ctx.pipe_fd[0] = -1;
	}

	/*
	* Close the write-half of the daemonize pipe.
	*
	* This should be called by the parent after fork()ing its child since the
	* parent will never write to this pipe.
	*
	* This should also be called by the child once initialization is complete
	* in order to signal the parent that it can safely exit.
	*/
	void
	zed_log_pipe_close_writes(void)
	{
	if (_ctx.pipe_fd[1] < 0)
	zed_log_die(
	"Invalid use of zed_log_pipe_close_writes in PID %d",
	(int)getpid());

	if (close(_ctx.pipe_fd[1]) < 0)
	zed_log_die(
	"Failed to close writes on daemonize pipe in PID %d: %s",
	(int)getpid(), strerror(errno));

	_ctx.pipe_fd[1] = -1;
	}

	/*
	* Block on reading from the daemonize pipe until signaled by the child
	* (via zed_log_pipe_close_writes()) that initialization is complete.
	*
	* This should only be called by the parent while waiting to exit after
	* fork()ing the child.
	*/
	void
	zed_log_pipe_wait(void)
	{
	ssize_t n;
	char c;

	if (_ctx.pipe_fd[0] < 0)
	zed_log_die("Invalid use of zed_log_pipe_wait in PID %d",
	(int)getpid());

	for (;;) {
	n = read(_ctx.pipe_fd[0], &c, sizeof (c));
	if (n < 0) {
	if (errno == EINTR)
	continue;
	zed_log_die(
	"Failed to read from daemonize pipe in PID %d: %s",
	(int)getpid(), strerror(errno));
	}
	if (n == 0) {
	break;
	}
	}
	}

	/*
	* Start logging messages at the syslog [priority] level or higher to stderr.
	* Refer to syslog(3) for valid priority values.
	*/
	void
	zed_log_stderr_open(int priority)
	{
	_ctx.do_stderr = 1;
	_ctx.priority = priority;
	}

	/*
	* Stop logging messages to stderr.
	*/
	void
	zed_log_stderr_close(void)
	{
	if (_ctx.do_stderr)
	_ctx.do_stderr = 0;
	}

	/*
	* Start logging messages to syslog.
	* Refer to syslog(3) for valid option/facility values.
	*/
	void
	zed_log_syslog_open(int facility)
	{
	_ctx.do_syslog = 1;
	openlog(_ctx.identity, LOG_NDELAY \| LOG_PID, facility);
	}

	/*
	* Stop logging messages to syslog.
	*/
	void
	zed_log_syslog_close(void)
	{
	if (_ctx.do_syslog) {
	_ctx.do_syslog = 0;
	closelog();
	}
	}

	/*
	* Auxiliary function to log a message to syslog and/or stderr.
	*/
	static void
	_zed_log_aux(int priority, const char *fmt, va_list vargs)
	{
	char buf[ZED_LOG_MAX_LOG_LEN];
	int n;

	if (!fmt)
	return;

	n = vsnprintf(buf, sizeof (buf), fmt, vargs);
	if ((n < 0) \|\| (n >= sizeof (buf))) {
	buf[sizeof (buf) - 2] = '+';
	buf[sizeof (buf) - 1] = '\0';
	}

	if (_ctx.do_syslog)
	syslog(priority, "%s", buf);

	if (_ctx.do_stderr && (priority <= _ctx.priority))
	fprintf(stderr, "%s\n", buf);
	}

	/*
	* Log a message at the given [priority] level specified by the printf-style
	* format string [fmt].
	*/
	void
	zed_log_msg(int priority, const char *fmt, ...)
	{
	va_list vargs;

	if (fmt) {
	va_start(vargs, fmt);
	_zed_log_aux(priority, fmt, vargs);
	va_end(vargs);
	}
	}

	/*
	* Log a fatal error message specified by the printf-style format string [fmt].
	*/
	void
	zed_log_die(const char *fmt, ...)
	{
	va_list vargs;

	if (fmt) {
	va_start(vargs, fmt);
	_zed_log_aux(LOG_ERR, fmt, vargs);
	va_end(vargs);
	}
	exit(EXIT_FAILURE);
	}
	Index: vendor-sys/openzfs/dist/cmd/zed/zed_log.h
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zed/zed_log.h (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zed/zed_log.h (revision 366774)
	@@ -1,44 +1,44 @@
	/*
	- * This file is part of the ZFS Event Daemon (ZED)
	- * for ZFS on Linux (ZoL) <http://zfsonlinux.org/>.
	+ * This file is part of the ZFS Event Daemon (ZED).
	+ *
	* Developed at Lawrence Livermore National Laboratory (LLNL-CODE-403049).
	* Copyright (C) 2013-2014 Lawrence Livermore National Security, LLC.
	* Refer to the ZoL git commit log for authoritative copyright attribution.
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License Version 1.0 (CDDL-1.0).
	* You can obtain a copy of the license from the top-level file
	* "OPENSOLARIS.LICENSE" or at <http://opensource.org/licenses/CDDL-1.0>.
	* You may not use this file except in compliance with the license.
	*/

	#ifndef ZED_LOG_H
	#define ZED_LOG_H

	#include <syslog.h>

	void zed_log_init(const char *identity);

	void zed_log_fini(void);

	void zed_log_pipe_open(void);

	void zed_log_pipe_close_reads(void);

	void zed_log_pipe_close_writes(void);

	void zed_log_pipe_wait(void);

	void zed_log_stderr_open(int priority);

	void zed_log_stderr_close(void);

	void zed_log_syslog_open(int facility);

	void zed_log_syslog_close(void);

	void zed_log_msg(int priority, const char *fmt, ...);

	void zed_log_die(const char *fmt, ...);

	#endif /* !ZED_LOG_H */
	Index: vendor-sys/openzfs/dist/cmd/zed/zed_strings.c
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zed/zed_strings.c (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zed/zed_strings.c (revision 366774)
	@@ -1,247 +1,247 @@
	/*
	- * This file is part of the ZFS Event Daemon (ZED)
	- * for ZFS on Linux (ZoL) <http://zfsonlinux.org/>.
	+ * This file is part of the ZFS Event Daemon (ZED).
	+ *
	* Developed at Lawrence Livermore National Laboratory (LLNL-CODE-403049).
	* Copyright (C) 2013-2014 Lawrence Livermore National Security, LLC.
	* Refer to the ZoL git commit log for authoritative copyright attribution.
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License Version 1.0 (CDDL-1.0).
	* You can obtain a copy of the license from the top-level file
	* "OPENSOLARIS.LICENSE" or at <http://opensource.org/licenses/CDDL-1.0>.
	* You may not use this file except in compliance with the license.
	*/

	#include <assert.h>
	#include <errno.h>
	#include <stddef.h>
	#include <stdlib.h>
	#include <string.h>
	#include <sys/avl.h>
	#include <sys/sysmacros.h>
	#include "zed_strings.h"

	struct zed_strings {
	avl_tree_t tree;
	avl_node_t *iteratorp;
	};

	struct zed_strings_node {
	avl_node_t node;
	char *key;
	char *val;
	};

	typedef struct zed_strings_node zed_strings_node_t;

	/*
	* Compare zed_strings_node_t nodes [x1] and [x2].
	* As required for the AVL tree, return -1 for <, 0 for ==, and +1 for >.
	*/
	static int
	_zed_strings_node_compare(const void x1, const void x2)
	{
	const char *s1;
	const char *s2;
	int rv;

	assert(x1 != NULL);
	assert(x2 != NULL);

	s1 = ((const zed_strings_node_t *) x1)->key;
	assert(s1 != NULL);
	s2 = ((const zed_strings_node_t *) x2)->key;
	assert(s2 != NULL);
	rv = strcmp(s1, s2);

	if (rv < 0)
	return (-1);

	if (rv > 0)
	return (1);

	return (0);
	}

	/*
	* Return a new string container, or NULL on error.
	*/
	zed_strings_t *
	zed_strings_create(void)
	{
	zed_strings_t *zsp;

	zsp = calloc(1, sizeof (*zsp));
	if (!zsp)
	return (NULL);

	avl_create(&zsp->tree, _zed_strings_node_compare,
	sizeof (zed_strings_node_t), offsetof(zed_strings_node_t, node));

	zsp->iteratorp = NULL;
	return (zsp);
	}

	/*
	* Destroy the string node [np].
	*/
	static void
	_zed_strings_node_destroy(zed_strings_node_t *np)
	{
	if (!np)
	return;

	if (np->key) {
	if (np->key != np->val)
	free(np->key);
	np->key = NULL;
	}
	if (np->val) {
	free(np->val);
	np->val = NULL;
	}
	free(np);
	}

	/*
	* Return a new string node for storing the string [val], or NULL on error.
	* If [key] is specified, it will be used to index the node; otherwise,
	* the string [val] will be used.
	*/
	static zed_strings_node_t *
	_zed_strings_node_create(const char key, const char val)
	{
	zed_strings_node_t *np;

	assert(val != NULL);

	np = calloc(1, sizeof (*np));
	if (!np)
	return (NULL);

	np->val = strdup(val);
	if (!np->val)
	goto nomem;

	if (key) {
	np->key = strdup(key);
	if (!np->key)
	goto nomem;
	} else {
	np->key = np->val;
	}
	return (np);

	nomem:
	_zed_strings_node_destroy(np);
	return (NULL);
	}

	/*
	* Destroy the string container [zsp] and all nodes within.
	*/
	void
	zed_strings_destroy(zed_strings_t *zsp)
	{
	void *cookie;
	zed_strings_node_t *np;

	if (!zsp)
	return;

	cookie = NULL;
	while ((np = avl_destroy_nodes(&zsp->tree, &cookie)))
	_zed_strings_node_destroy(np);

	avl_destroy(&zsp->tree);
	free(zsp);
	}

	/*
	* Add a copy of the string [s] indexed by [key] to the container [zsp].
	* If [key] already exists within the container [zsp], it will be replaced
	* with the new string [s].
	* If [key] is NULL, the string [s] will be used as the key.
	* Return 0 on success, or -1 on error.
	*/
	int
	zed_strings_add(zed_strings_t zsp, const char key, const char *s)
	{
	zed_strings_node_t newp, oldp;

	if (!zsp \|\| !s) {
	errno = EINVAL;
	return (-1);
	}
	if (key == s)
	key = NULL;

	newp = _zed_strings_node_create(key, s);
	if (!newp)
	return (-1);

	oldp = avl_find(&zsp->tree, newp, NULL);
	if (oldp) {
	avl_remove(&zsp->tree, oldp);
	_zed_strings_node_destroy(oldp);
	}
	avl_add(&zsp->tree, newp);
	return (0);
	}

	/*
	* Return the first string in container [zsp].
	* Return NULL if there are no strings, or on error.
	* This can be called multiple times to re-traverse [zsp].
	* XXX: Not thread-safe.
	*/
	const char *
	zed_strings_first(zed_strings_t *zsp)
	{
	if (!zsp) {
	errno = EINVAL;
	return (NULL);
	}
	zsp->iteratorp = avl_first(&zsp->tree);
	if (!zsp->iteratorp)
	return (NULL);

	return (((zed_strings_node_t *)zsp->iteratorp)->val);

	}

	/*
	* Return the next string in container [zsp].
	* Return NULL after the last string, or on error.
	* This must be called after zed_strings_first().
	* XXX: Not thread-safe.
	*/
	const char *
	zed_strings_next(zed_strings_t *zsp)
	{
	if (!zsp) {
	errno = EINVAL;
	return (NULL);
	}
	if (!zsp->iteratorp)
	return (NULL);

	zsp->iteratorp = AVL_NEXT(&zsp->tree, zsp->iteratorp);
	if (!zsp->iteratorp)
	return (NULL);

	return (((zed_strings_node_t *)zsp->iteratorp)->val);
	}

	/*
	* Return the number of strings in container [zsp], or -1 on error.
	*/
	int
	zed_strings_count(zed_strings_t *zsp)
	{
	if (!zsp) {
	errno = EINVAL;
	return (-1);
	}
	return (avl_numnodes(&zsp->tree));
	}
	Index: vendor-sys/openzfs/dist/cmd/zed/zed_strings.h
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zed/zed_strings.h (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zed/zed_strings.h (revision 366774)
	@@ -1,27 +1,27 @@
	/*
	- * This file is part of the ZFS Event Daemon (ZED)
	- * for ZFS on Linux (ZoL) <http://zfsonlinux.org/>.
	+ * This file is part of the ZFS Event Daemon (ZED).
	+ *
	* Developed at Lawrence Livermore National Laboratory (LLNL-CODE-403049).
	* Copyright (C) 2013-2014 Lawrence Livermore National Security, LLC.
	* Refer to the ZoL git commit log for authoritative copyright attribution.
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License Version 1.0 (CDDL-1.0).
	* You can obtain a copy of the license from the top-level file
	* "OPENSOLARIS.LICENSE" or at <http://opensource.org/licenses/CDDL-1.0>.
	* You may not use this file except in compliance with the license.
	*/

	#ifndef ZED_STRINGS_H
	#define ZED_STRINGS_H

	typedef struct zed_strings zed_strings_t;

	zed_strings_t *zed_strings_create(void);
	void zed_strings_destroy(zed_strings_t *zsp);
	int zed_strings_add(zed_strings_t zsp, const char key, const char *s);
	const char zed_strings_first(zed_strings_t zsp);
	const char zed_strings_next(zed_strings_t zsp);
	int zed_strings_count(zed_strings_t *zsp);

	#endif /* !ZED_STRINGS_H */
	Index: vendor-sys/openzfs/dist/cmd/zvol_id/Makefile.am
	===================================================================
	--- vendor-sys/openzfs/dist/cmd/zvol_id/Makefile.am (revision 366773)
	+++ vendor-sys/openzfs/dist/cmd/zvol_id/Makefile.am (revision 366774)
	@@ -1,10 +1,10 @@
	include $(top_srcdir)/config/Rules.am

	# Disable GCC stack protection for zvol_id. This is a kludge and should be
	-# removed once https://github.com/zfsonlinux/zfs/issues/569 is resolved.
	+# removed once https://github.com/openzfs/zfs/issues/569 is resolved.
	AM_CFLAGS += -fno-stack-protector

	udev_PROGRAMS = zvol_id

	zvol_id_SOURCES = \
	zvol_id_main.c
	Index: vendor-sys/openzfs/dist/config/kernel-bio.m4
	===================================================================
	--- vendor-sys/openzfs/dist/config/kernel-bio.m4 (revision 366773)
	+++ vendor-sys/openzfs/dist/config/kernel-bio.m4 (revision 366774)
	@@ -1,403 +1,403 @@
	dnl #
	dnl # 2.6.36 API change,
	dnl # REQ_FAILFAST_{DEV\|TRANSPORT\|DRIVER}
	dnl # REQ_DISCARD
	dnl # REQ_FLUSH
	dnl #
	dnl # 4.8 - 4.9 API,
	dnl # REQ_FLUSH was renamed to REQ_PREFLUSH
	dnl #
	AC_DEFUN([ZFS_AC_KERNEL_SRC_REQ], [
	ZFS_LINUX_TEST_SRC([req_failfast_mask], [
	#include <linux/bio.h>
	],[
	int flags __attribute__ ((unused));
	flags = REQ_FAILFAST_MASK;
	])

	ZFS_LINUX_TEST_SRC([req_discard], [
	#include <linux/bio.h>
	],[
	int flags __attribute__ ((unused));
	flags = REQ_DISCARD;
	])

	ZFS_LINUX_TEST_SRC([req_flush], [
	#include <linux/bio.h>
	],[
	int flags __attribute__ ((unused));
	flags = REQ_FLUSH;
	])

	ZFS_LINUX_TEST_SRC([req_preflush], [
	#include <linux/bio.h>
	],[
	int flags __attribute__ ((unused));
	flags = REQ_PREFLUSH;
	])
	])

	AC_DEFUN([ZFS_AC_KERNEL_BIO_REQ_FAILFAST_MASK], [
	AC_MSG_CHECKING([whether REQ_FAILFAST_MASK is defined])
	ZFS_LINUX_TEST_RESULT([req_failfast_mask], [
	AC_MSG_RESULT(yes)
	],[
	ZFS_LINUX_TEST_ERROR([REQ_FAILFAST_MASK])
	])
	])

	AC_DEFUN([ZFS_AC_KERNEL_BIO_REQ_DISCARD], [
	AC_MSG_CHECKING([whether REQ_DISCARD is defined])
	ZFS_LINUX_TEST_RESULT([req_discard], [
	AC_MSG_RESULT(yes)
	AC_DEFINE(HAVE_REQ_DISCARD, 1, [REQ_DISCARD is defined])
	],[
	AC_MSG_RESULT(no)
	])
	])

	AC_DEFUN([ZFS_AC_KERNEL_BIO_REQ_FLUSH], [
	AC_MSG_CHECKING([whether REQ_FLUSH is defined])
	ZFS_LINUX_TEST_RESULT([req_flush], [
	AC_MSG_RESULT(yes)
	AC_DEFINE(HAVE_REQ_FLUSH, 1, [REQ_FLUSH is defined])
	],[
	AC_MSG_RESULT(no)
	])
	])

	AC_DEFUN([ZFS_AC_KERNEL_BIO_REQ_PREFLUSH], [
	AC_MSG_CHECKING([whether REQ_PREFLUSH is defined])
	ZFS_LINUX_TEST_RESULT([req_preflush], [
	AC_MSG_RESULT(yes)
	AC_DEFINE(HAVE_REQ_PREFLUSH, 1, [REQ_PREFLUSH is defined])
	],[
	AC_MSG_RESULT(no)
	])
	])

	dnl #
	dnl # Linux 4.8 API,
	dnl #
	dnl # The bio_op() helper was introduced as a replacement for explicitly
	dnl # checking the bio->bi_rw flags. The following checks are used to
	dnl # detect if a specific operation is supported.
	dnl #
	AC_DEFUN([ZFS_AC_KERNEL_SRC_BIO_OPS], [
	ZFS_LINUX_TEST_SRC([req_op_discard], [
	#include <linux/blk_types.h>
	],[
	int op __attribute__ ((unused)) = REQ_OP_DISCARD;
	])

	ZFS_LINUX_TEST_SRC([req_op_secure_erase], [
	#include <linux/blk_types.h>
	],[
	int op __attribute__ ((unused)) = REQ_OP_SECURE_ERASE;
	])

	ZFS_LINUX_TEST_SRC([req_op_flush], [
	#include <linux/blk_types.h>
	],[
	int op __attribute__ ((unused)) = REQ_OP_FLUSH;
	])

	ZFS_LINUX_TEST_SRC([bio_bi_opf], [
	#include <linux/bio.h>
	],[
	struct bio bio __attribute__ ((unused));
	bio.bi_opf = 0;
	])

	ZFS_LINUX_TEST_SRC([bio_set_op_attrs], [
	#include <linux/bio.h>
	],[
	struct bio *bio __attribute__ ((unused)) = NULL;
	bio_set_op_attrs(bio, 0, 0);
	])
	])

	AC_DEFUN([ZFS_AC_KERNEL_BIO_REQ_OP_DISCARD], [
	AC_MSG_CHECKING([whether REQ_OP_DISCARD is defined])
	ZFS_LINUX_TEST_RESULT([req_op_discard], [
	AC_MSG_RESULT(yes)
	AC_DEFINE(HAVE_REQ_OP_DISCARD, 1, [REQ_OP_DISCARD is defined])
	],[
	AC_MSG_RESULT(no)
	])
	])

	AC_DEFUN([ZFS_AC_KERNEL_BIO_REQ_OP_SECURE_ERASE], [
	AC_MSG_CHECKING([whether REQ_OP_SECURE_ERASE is defined])
	ZFS_LINUX_TEST_RESULT([req_op_secure_erase], [
	AC_MSG_RESULT(yes)
	AC_DEFINE(HAVE_REQ_OP_SECURE_ERASE, 1,
	[REQ_OP_SECURE_ERASE is defined])
	],[
	AC_MSG_RESULT(no)
	])
	])

	AC_DEFUN([ZFS_AC_KERNEL_BIO_REQ_OP_FLUSH], [
	AC_MSG_CHECKING([whether REQ_OP_FLUSH is defined])
	ZFS_LINUX_TEST_RESULT([req_op_flush], [
	AC_MSG_RESULT(yes)
	AC_DEFINE(HAVE_REQ_OP_FLUSH, 1, [REQ_OP_FLUSH is defined])
	],[
	AC_MSG_RESULT(no)
	])
	])

	AC_DEFUN([ZFS_AC_KERNEL_BIO_BI_OPF], [
	AC_MSG_CHECKING([whether bio->bi_opf is defined])
	ZFS_LINUX_TEST_RESULT([bio_bi_opf], [
	AC_MSG_RESULT(yes)
	AC_DEFINE(HAVE_BIO_BI_OPF, 1, [bio->bi_opf is defined])
	],[
	AC_MSG_RESULT(no)
	])
	])

	AC_DEFUN([ZFS_AC_KERNEL_BIO_SET_OP_ATTRS], [
	AC_MSG_CHECKING([whether bio_set_op_attrs is available])
	ZFS_LINUX_TEST_RESULT([bio_set_op_attrs], [
	AC_MSG_RESULT(yes)
	AC_DEFINE(HAVE_BIO_SET_OP_ATTRS, 1,
	[bio_set_op_attrs is available])
	],[
	AC_MSG_RESULT(no)
	])
	])

	dnl #
	dnl # Linux 4.14 API,
	dnl #
	dnl # The bio_set_dev() helper macro was introduced as part of the transition
	dnl # to have struct gendisk in struct bio.
	dnl #
	dnl # Linux 5.0 API,
	dnl #
	dnl # The bio_set_dev() helper macro was updated to internally depend on
	dnl # bio_associate_blkg() symbol which is exported GPL-only.
	dnl #
	AC_DEFUN([ZFS_AC_KERNEL_SRC_BIO_SET_DEV], [
	ZFS_LINUX_TEST_SRC([bio_set_dev], [
	#include <linux/bio.h>
	#include <linux/fs.h>
	],[
	struct block_device *bdev = NULL;
	struct bio *bio = NULL;
	bio_set_dev(bio, bdev);
	], [], [$ZFS_META_LICENSE])
	])

	AC_DEFUN([ZFS_AC_KERNEL_BIO_SET_DEV], [
	AC_MSG_CHECKING([whether bio_set_dev() is available])
	ZFS_LINUX_TEST_RESULT([bio_set_dev], [
	AC_MSG_RESULT(yes)
	AC_DEFINE(HAVE_BIO_SET_DEV, 1, [bio_set_dev() is available])

	AC_MSG_CHECKING([whether bio_set_dev() is GPL-only])
	ZFS_LINUX_TEST_RESULT([bio_set_dev_license], [
	AC_MSG_RESULT(no)
	],[
	AC_MSG_RESULT(yes)
	AC_DEFINE(HAVE_BIO_SET_DEV_GPL_ONLY, 1,
	[bio_set_dev() GPL-only])
	])
	],[
	AC_MSG_RESULT(no)
	])
	])

	dnl #
	dnl # 4.3 API change
	dnl # Error argument dropped from bio_endio in favor of newly introduced
	dnl # bio->bi_error. This also replaces bio->bi_flags value BIO_UPTODATE.
	dnl # Introduced by torvalds/linux@4246a0b63bd8f56a1469b12eafeb875b1041a451
	dnl # ("block: add a bi_error field to struct bio").
	dnl #
	AC_DEFUN([ZFS_AC_KERNEL_SRC_BIO_END_IO_T_ARGS], [
	ZFS_LINUX_TEST_SRC([bio_end_io_t_args], [
	#include <linux/bio.h>
	void wanted_end_io(struct bio *bio) { return; }
	bio_end_io_t *end_io __attribute__ ((unused)) = wanted_end_io;
	], [])
	])

	AC_DEFUN([ZFS_AC_KERNEL_BIO_END_IO_T_ARGS], [
	AC_MSG_CHECKING([whether bio_end_io_t wants 1 arg])
	ZFS_LINUX_TEST_RESULT([bio_end_io_t_args], [
	AC_MSG_RESULT(yes)
	AC_DEFINE(HAVE_1ARG_BIO_END_IO_T, 1,
	[bio_end_io_t wants 1 arg])
	], [
	AC_MSG_RESULT(no)
	])
	])

	dnl #
	dnl # 4.13 API change
	dnl # The bio->bi_error field was replaced with bio->bi_status which is an
	dnl # enum which describes all possible error types.
	dnl #
	AC_DEFUN([ZFS_AC_KERNEL_SRC_BIO_BI_STATUS], [
	ZFS_LINUX_TEST_SRC([bio_bi_status], [
	#include <linux/bio.h>
	], [
	struct bio bio __attribute__ ((unused));
	blk_status_t status __attribute__ ((unused)) = BLK_STS_OK;
	bio.bi_status = status;
	])
	])

	AC_DEFUN([ZFS_AC_KERNEL_BIO_BI_STATUS], [
	AC_MSG_CHECKING([whether bio->bi_status exists])
	ZFS_LINUX_TEST_RESULT([bio_bi_status], [
	AC_MSG_RESULT(yes)
	AC_DEFINE(HAVE_BIO_BI_STATUS, 1, [bio->bi_status exists])
	],[
	AC_MSG_RESULT(no)
	])
	])

	dnl #
	dnl # 3.14 API change,
	dnl # Immutable biovecs. A number of fields of struct bio are moved to
	dnl # struct bvec_iter.
	dnl #
	AC_DEFUN([ZFS_AC_KERNEL_SRC_BIO_BVEC_ITER], [
	ZFS_LINUX_TEST_SRC([bio_bvec_iter], [
	#include <linux/bio.h>
	],[
	struct bio bio;
	bio.bi_iter.bi_sector = 0;
	])
	])

	AC_DEFUN([ZFS_AC_KERNEL_BIO_BVEC_ITER], [
	AC_MSG_CHECKING([whether bio has bi_iter])
	ZFS_LINUX_TEST_RESULT([bio_bvec_iter], [
	AC_MSG_RESULT(yes)
	AC_DEFINE(HAVE_BIO_BVEC_ITER, 1, [bio has bi_iter])
	],[
	AC_MSG_RESULT(no)
	])
	])

	dnl #
	dnl # 4.8 API change
	dnl # The rw argument has been removed from submit_bio/submit_bio_wait.
	dnl # Callers are now expected to set bio->bi_rw instead of passing it in.
	dnl #
	AC_DEFUN([ZFS_AC_KERNEL_SRC_BIO_SUBMIT_BIO], [
	ZFS_LINUX_TEST_SRC([submit_bio], [
	#include <linux/bio.h>
	],[
	blk_qc_t blk_qc;
	struct bio *bio = NULL;
	blk_qc = submit_bio(bio);
	])
	])

	AC_DEFUN([ZFS_AC_KERNEL_BIO_SUBMIT_BIO], [
	AC_MSG_CHECKING([whether submit_bio() wants 1 arg])
	ZFS_LINUX_TEST_RESULT([submit_bio], [
	AC_MSG_RESULT(yes)
	AC_DEFINE(HAVE_1ARG_SUBMIT_BIO, 1, [submit_bio() wants 1 arg])
	],[
	AC_MSG_RESULT(no)
	])
	])

	dnl #
	dnl # 2.6.34 API change
	dnl # current->bio_list
	dnl #
	AC_DEFUN([ZFS_AC_KERNEL_SRC_BIO_CURRENT_BIO_LIST], [
	ZFS_LINUX_TEST_SRC([current_bio_list], [
	#include <linux/sched.h>
	], [
	current->bio_list = (struct bio_list *) NULL;
	])
	])

	AC_DEFUN([ZFS_AC_KERNEL_BIO_CURRENT_BIO_LIST], [
	AC_MSG_CHECKING([whether current->bio_list exists])
	ZFS_LINUX_TEST_RESULT([current_bio_list], [
	AC_MSG_RESULT(yes)
	],[
	ZFS_LINUX_TEST_ERROR([bio_list])
	])
	])

	dnl #
	dnl # Linux 5.5 API,
	dnl #
	dnl # The Linux 5.5 kernel updated percpu_ref_tryget() which is inlined by
	dnl # blkg_tryget() to use rcu_read_lock() instead of rcu_read_lock_sched().
	dnl # As a side effect the function was converted to GPL-only.
	dnl #
	AC_DEFUN([ZFS_AC_KERNEL_SRC_BLKG_TRYGET], [
	ZFS_LINUX_TEST_SRC([blkg_tryget], [
	#include <linux/blk-cgroup.h>
	#include <linux/bio.h>
	#include <linux/fs.h>
	],[
	- struct blkcg_gq blkg __attribute__ ((unused));
	+ struct blkcg_gq blkg __attribute__ ((unused)) = {};
	bool rc __attribute__ ((unused));
	rc = blkg_tryget(&blkg);
	], [], [$ZFS_META_LICENSE])
	])

	AC_DEFUN([ZFS_AC_KERNEL_BLKG_TRYGET], [
	AC_MSG_CHECKING([whether blkg_tryget() is available])
	ZFS_LINUX_TEST_RESULT([blkg_tryget], [
	AC_MSG_RESULT(yes)
	AC_DEFINE(HAVE_BLKG_TRYGET, 1, [blkg_tryget() is available])

	AC_MSG_CHECKING([whether blkg_tryget() is GPL-only])
	ZFS_LINUX_TEST_RESULT([blkg_tryget_license], [
	AC_MSG_RESULT(no)
	],[
	AC_MSG_RESULT(yes)
	AC_DEFINE(HAVE_BLKG_TRYGET_GPL_ONLY, 1,
	[blkg_tryget() GPL-only])
	])
	],[
	AC_MSG_RESULT(no)
	])
	])

	AC_DEFUN([ZFS_AC_KERNEL_SRC_BIO], [
	ZFS_AC_KERNEL_SRC_REQ
	ZFS_AC_KERNEL_SRC_BIO_OPS
	ZFS_AC_KERNEL_SRC_BIO_SET_DEV
	ZFS_AC_KERNEL_SRC_BIO_END_IO_T_ARGS
	ZFS_AC_KERNEL_SRC_BIO_BI_STATUS
	ZFS_AC_KERNEL_SRC_BIO_BVEC_ITER
	ZFS_AC_KERNEL_SRC_BIO_SUBMIT_BIO
	ZFS_AC_KERNEL_SRC_BIO_CURRENT_BIO_LIST
	ZFS_AC_KERNEL_SRC_BLKG_TRYGET
	])

	AC_DEFUN([ZFS_AC_KERNEL_BIO], [
	ZFS_AC_KERNEL_BIO_REQ_FAILFAST_MASK
	ZFS_AC_KERNEL_BIO_REQ_DISCARD
	ZFS_AC_KERNEL_BIO_REQ_FLUSH
	ZFS_AC_KERNEL_BIO_REQ_PREFLUSH

	ZFS_AC_KERNEL_BIO_REQ_OP_DISCARD
	ZFS_AC_KERNEL_BIO_REQ_OP_SECURE_ERASE
	ZFS_AC_KERNEL_BIO_REQ_OP_FLUSH
	ZFS_AC_KERNEL_BIO_BI_OPF
	ZFS_AC_KERNEL_BIO_SET_OP_ATTRS

	ZFS_AC_KERNEL_BIO_SET_DEV
	ZFS_AC_KERNEL_BIO_END_IO_T_ARGS
	ZFS_AC_KERNEL_BIO_BI_STATUS
	ZFS_AC_KERNEL_BIO_BVEC_ITER
	ZFS_AC_KERNEL_BIO_SUBMIT_BIO
	ZFS_AC_KERNEL_BIO_CURRENT_BIO_LIST
	ZFS_AC_KERNEL_BLKG_TRYGET
	])
	Index: vendor-sys/openzfs/dist/configure.ac
	===================================================================
	--- vendor-sys/openzfs/dist/configure.ac (revision 366773)
	+++ vendor-sys/openzfs/dist/configure.ac (revision 366774)
	@@ -1,412 +1,413 @@
	/*
	* This file is part of the ZFS Linux port.
	*
	* Copyright (c) 2009 Lawrence Livermore National Security, LLC.
	* Produced at Lawrence Livermore National Laboratory
	* Written by:
	* Brian Behlendorf <behlendorf1@llnl.gov>,
	* Herb Wartens <wartens2@llnl.gov>,
	* Jim Garlick <garlick@llnl.gov>
	* LLNL-CODE-403049
	*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License, Version 1.0 only
	* (the "License"). You may not use this file except in compliance
	* with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/

	AC_INIT(m4_esyscmd(grep ^Name: META \| cut -d ':' -f 2 \| tr -d ' \n'),
	m4_esyscmd(grep ^Version: META \| cut -d ':' -f 2 \| tr -d ' \n'))
	AC_LANG(C)
	ZFS_AC_META
	AC_CONFIG_AUX_DIR([config])
	AC_CONFIG_MACRO_DIR([config])
	AC_CANONICAL_SYSTEM
	AM_MAINTAINER_MODE
	m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])])
	AM_INIT_AUTOMAKE([subdir-objects])
	AC_CONFIG_HEADERS([zfs_config.h], [
	(mv zfs_config.h zfs_config.h.tmp &&
	awk -f ${ac_srcdir}/config/config.awk zfs_config.h.tmp >zfs_config.h &&
	rm zfs_config.h.tmp) \|\| exit 1])

	AC_PROG_INSTALL
	AC_PROG_CC
	AC_PROG_LIBTOOL
	PKG_PROG_PKG_CONFIG
	AM_PROG_AS
	AM_PROG_CC_C_O
	AX_CODE_COVERAGE
	_AM_PROG_TAR(pax)

	ZFS_AC_LICENSE
	ZFS_AC_CONFIG
	ZFS_AC_PACKAGE
	ZFS_AC_DEBUG
	ZFS_AC_DEBUGINFO
	ZFS_AC_DEBUG_KMEM
	ZFS_AC_DEBUG_KMEM_TRACKING

	AC_CONFIG_FILES([
	Makefile
	cmd/Makefile
	cmd/arc_summary/Makefile
	cmd/arcstat/Makefile
	cmd/dbufstat/Makefile
	cmd/fsck_zfs/Makefile
	cmd/mount_zfs/Makefile
	cmd/raidz_test/Makefile
	cmd/vdev_id/Makefile
	cmd/zdb/Makefile
	cmd/zed/Makefile
	cmd/zed/zed.d/Makefile
	cmd/zfs/Makefile
	cmd/zfs_ids_to_path/Makefile
	cmd/zgenhostid/Makefile
	cmd/zhack/Makefile
	cmd/zinject/Makefile
	cmd/zpool/Makefile
	cmd/zstream/Makefile
	cmd/zstreamdump/Makefile
	cmd/ztest/Makefile
	cmd/zvol_id/Makefile
	cmd/zvol_wait/Makefile
	contrib/Makefile
	contrib/bash_completion.d/Makefile
	contrib/bpftrace/Makefile
	contrib/dracut/02zfsexpandknowledge/Makefile
	contrib/dracut/90zfs/Makefile
	contrib/dracut/Makefile
	contrib/initramfs/Makefile
	contrib/initramfs/conf.d/Makefile
	contrib/initramfs/conf-hooks.d/Makefile
	contrib/initramfs/hooks/Makefile
	contrib/initramfs/scripts/Makefile
	contrib/initramfs/scripts/local-top/Makefile
	contrib/pam_zfs_key/Makefile
	contrib/pyzfs/Makefile
	contrib/pyzfs/setup.py
	contrib/zcp/Makefile
	etc/Makefile
	etc/default/Makefile
	etc/init.d/Makefile
	etc/modules-load.d/Makefile
	etc/sudoers.d/Makefile
	etc/systemd/Makefile
	etc/systemd/system-generators/Makefile
	etc/systemd/system/Makefile
	etc/zfs/Makefile
	include/Makefile
	include/os/Makefile
	include/os/freebsd/Makefile
	include/os/freebsd/linux/Makefile
	include/os/freebsd/spl/Makefile
	include/os/freebsd/spl/acl/Makefile
	include/os/freebsd/spl/rpc/Makefile
	include/os/freebsd/spl/sys/Makefile
	include/os/freebsd/zfs/Makefile
	include/os/freebsd/zfs/sys/Makefile
	include/os/linux/Makefile
	include/os/linux/kernel/Makefile
	include/os/linux/kernel/linux/Makefile
	include/os/linux/spl/Makefile
	include/os/linux/spl/rpc/Makefile
	include/os/linux/spl/sys/Makefile
	include/os/linux/zfs/Makefile
	include/os/linux/zfs/sys/Makefile
	include/sys/Makefile
	include/sys/crypto/Makefile
	include/sys/fm/Makefile
	include/sys/fm/fs/Makefile
	include/sys/fs/Makefile
	include/sys/lua/Makefile
	include/sys/sysevent/Makefile
	include/sys/zstd/Makefile
	lib/Makefile
	lib/libavl/Makefile
	lib/libefi/Makefile
	lib/libicp/Makefile
	lib/libnvpair/Makefile
	lib/libshare/Makefile
	lib/libspl/Makefile
	lib/libspl/include/Makefile
	lib/libspl/include/ia32/Makefile
	lib/libspl/include/ia32/sys/Makefile
	lib/libspl/include/os/Makefile
	lib/libspl/include/os/freebsd/Makefile
	lib/libspl/include/os/freebsd/sys/Makefile
	lib/libspl/include/os/linux/Makefile
	lib/libspl/include/os/linux/sys/Makefile
	lib/libspl/include/rpc/Makefile
	lib/libspl/include/sys/Makefile
	lib/libspl/include/sys/dktp/Makefile
	lib/libspl/include/util/Makefile
	lib/libtpool/Makefile
	lib/libunicode/Makefile
	lib/libuutil/Makefile
	lib/libzfs/Makefile
	lib/libzfs/libzfs.pc
	lib/libzfsbootenv/Makefile
	lib/libzfsbootenv/libzfsbootenv.pc
	lib/libzfs_core/Makefile
	lib/libzfs_core/libzfs_core.pc
	lib/libzpool/Makefile
	lib/libzstd/Makefile
	lib/libzutil/Makefile
	man/Makefile
	man/man1/Makefile
	man/man5/Makefile
	man/man8/Makefile
	module/Kbuild
	module/Makefile
	module/avl/Makefile
	module/icp/Makefile
	module/lua/Makefile
	module/nvpair/Makefile
	module/os/linux/spl/Makefile
	module/os/linux/zfs/Makefile
	module/spl/Makefile
	module/unicode/Makefile
	module/zcommon/Makefile
	module/zfs/Makefile
	module/zstd/Makefile
	rpm/Makefile
	rpm/generic/Makefile
	rpm/generic/zfs-dkms.spec
	rpm/generic/zfs-kmod.spec
	rpm/generic/zfs.spec
	rpm/redhat/Makefile
	rpm/redhat/zfs-dkms.spec
	rpm/redhat/zfs-kmod.spec
	rpm/redhat/zfs.spec
	scripts/Makefile
	tests/Makefile
	tests/runfiles/Makefile
	tests/test-runner/Makefile
	tests/test-runner/bin/Makefile
	tests/test-runner/include/Makefile
	tests/test-runner/man/Makefile
	tests/zfs-tests/Makefile
	tests/zfs-tests/callbacks/Makefile
	tests/zfs-tests/cmd/Makefile
	tests/zfs-tests/cmd/badsend/Makefile
	tests/zfs-tests/cmd/btree_test/Makefile
	tests/zfs-tests/cmd/chg_usr_exec/Makefile
	tests/zfs-tests/cmd/devname2devid/Makefile
	tests/zfs-tests/cmd/dir_rd_update/Makefile
	tests/zfs-tests/cmd/file_check/Makefile
	tests/zfs-tests/cmd/file_trunc/Makefile
	tests/zfs-tests/cmd/file_write/Makefile
	tests/zfs-tests/cmd/get_diff/Makefile
	tests/zfs-tests/cmd/largest_file/Makefile
	tests/zfs-tests/cmd/libzfs_input_check/Makefile
	tests/zfs-tests/cmd/mkbusy/Makefile
	tests/zfs-tests/cmd/mkfile/Makefile
	tests/zfs-tests/cmd/mkfiles/Makefile
	tests/zfs-tests/cmd/mktree/Makefile
	tests/zfs-tests/cmd/mmap_exec/Makefile
	tests/zfs-tests/cmd/mmap_libaio/Makefile
	tests/zfs-tests/cmd/mmapwrite/Makefile
	tests/zfs-tests/cmd/nvlist_to_lua/Makefile
	tests/zfs-tests/cmd/randfree_file/Makefile
	tests/zfs-tests/cmd/randwritecomp/Makefile
	tests/zfs-tests/cmd/readmmap/Makefile
	tests/zfs-tests/cmd/rename_dir/Makefile
	tests/zfs-tests/cmd/rm_lnkcnt_zero_file/Makefile
	tests/zfs-tests/cmd/stride_dd/Makefile
	tests/zfs-tests/cmd/threadsappend/Makefile
	tests/zfs-tests/cmd/user_ns_exec/Makefile
	tests/zfs-tests/cmd/xattrtest/Makefile
	tests/zfs-tests/include/Makefile
	tests/zfs-tests/tests/Makefile
	tests/zfs-tests/tests/functional/Makefile
	tests/zfs-tests/tests/functional/acl/Makefile
	tests/zfs-tests/tests/functional/acl/posix/Makefile
	+ tests/zfs-tests/tests/functional/acl/posix-sa/Makefile
	tests/zfs-tests/tests/functional/alloc_class/Makefile
	tests/zfs-tests/tests/functional/arc/Makefile
	tests/zfs-tests/tests/functional/atime/Makefile
	tests/zfs-tests/tests/functional/bootfs/Makefile
	tests/zfs-tests/tests/functional/btree/Makefile
	tests/zfs-tests/tests/functional/cache/Makefile
	tests/zfs-tests/tests/functional/cachefile/Makefile
	tests/zfs-tests/tests/functional/casenorm/Makefile
	tests/zfs-tests/tests/functional/channel_program/Makefile
	tests/zfs-tests/tests/functional/channel_program/lua_core/Makefile
	tests/zfs-tests/tests/functional/channel_program/synctask_core/Makefile
	tests/zfs-tests/tests/functional/chattr/Makefile
	tests/zfs-tests/tests/functional/checksum/Makefile
	tests/zfs-tests/tests/functional/clean_mirror/Makefile
	tests/zfs-tests/tests/functional/cli_root/Makefile
	tests/zfs-tests/tests/functional/cli_root/zdb/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_bookmark/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_change-key/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_clone/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_copies/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_create/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_destroy/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_diff/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_get/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_ids_to_path/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_inherit/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_jail/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_load-key/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_mount/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_program/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_promote/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_property/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_receive/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_rename/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_reservation/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_rollback/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_send/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_set/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_share/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_snapshot/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_sysfs/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_unload-key/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_unmount/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_unshare/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_upgrade/Makefile
	tests/zfs-tests/tests/functional/cli_root/zfs_wait/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_add/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_attach/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_clear/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_create/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_destroy/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_detach/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_events/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_expand/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_export/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_get/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_history/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_import/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_import/blockfiles/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_initialize/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_labelclear/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_offline/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_online/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_remove/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_reopen/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_replace/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_resilver/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_scrub/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_set/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_split/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_status/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_sync/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_trim/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_upgrade/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_upgrade/blockfiles/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_wait/Makefile
	tests/zfs-tests/tests/functional/cli_root/zpool_wait/scan/Makefile
	tests/zfs-tests/tests/functional/cli_user/Makefile
	tests/zfs-tests/tests/functional/cli_user/misc/Makefile
	tests/zfs-tests/tests/functional/cli_user/zfs_list/Makefile
	tests/zfs-tests/tests/functional/cli_user/zpool_iostat/Makefile
	tests/zfs-tests/tests/functional/cli_user/zpool_list/Makefile
	tests/zfs-tests/tests/functional/cli_user/zpool_status/Makefile
	tests/zfs-tests/tests/functional/compression/Makefile
	tests/zfs-tests/tests/functional/cp_files/Makefile
	tests/zfs-tests/tests/functional/ctime/Makefile
	tests/zfs-tests/tests/functional/deadman/Makefile
	tests/zfs-tests/tests/functional/delegate/Makefile
	tests/zfs-tests/tests/functional/devices/Makefile
	tests/zfs-tests/tests/functional/events/Makefile
	tests/zfs-tests/tests/functional/exec/Makefile
	tests/zfs-tests/tests/functional/fallocate/Makefile
	tests/zfs-tests/tests/functional/fault/Makefile
	tests/zfs-tests/tests/functional/features/Makefile
	tests/zfs-tests/tests/functional/features/async_destroy/Makefile
	tests/zfs-tests/tests/functional/features/large_dnode/Makefile
	tests/zfs-tests/tests/functional/grow/Makefile
	tests/zfs-tests/tests/functional/history/Makefile
	tests/zfs-tests/tests/functional/hkdf/Makefile
	tests/zfs-tests/tests/functional/inheritance/Makefile
	tests/zfs-tests/tests/functional/inuse/Makefile
	tests/zfs-tests/tests/functional/io/Makefile
	tests/zfs-tests/tests/functional/large_files/Makefile
	tests/zfs-tests/tests/functional/largest_pool/Makefile
	tests/zfs-tests/tests/functional/libzfs/Makefile
	tests/zfs-tests/tests/functional/limits/Makefile
	tests/zfs-tests/tests/functional/link_count/Makefile
	tests/zfs-tests/tests/functional/log_spacemap/Makefile
	tests/zfs-tests/tests/functional/migration/Makefile
	tests/zfs-tests/tests/functional/mmap/Makefile
	tests/zfs-tests/tests/functional/mmp/Makefile
	tests/zfs-tests/tests/functional/mount/Makefile
	tests/zfs-tests/tests/functional/mv_files/Makefile
	tests/zfs-tests/tests/functional/nestedfs/Makefile
	tests/zfs-tests/tests/functional/no_space/Makefile
	tests/zfs-tests/tests/functional/nopwrite/Makefile
	tests/zfs-tests/tests/functional/online_offline/Makefile
	tests/zfs-tests/tests/functional/pam/Makefile
	tests/zfs-tests/tests/functional/persist_l2arc/Makefile
	tests/zfs-tests/tests/functional/pool_checkpoint/Makefile
	tests/zfs-tests/tests/functional/pool_names/Makefile
	tests/zfs-tests/tests/functional/poolversion/Makefile
	tests/zfs-tests/tests/functional/privilege/Makefile
	tests/zfs-tests/tests/functional/procfs/Makefile
	tests/zfs-tests/tests/functional/projectquota/Makefile
	tests/zfs-tests/tests/functional/pyzfs/Makefile
	tests/zfs-tests/tests/functional/quota/Makefile
	tests/zfs-tests/tests/functional/raidz/Makefile
	tests/zfs-tests/tests/functional/redacted_send/Makefile
	tests/zfs-tests/tests/functional/redundancy/Makefile
	tests/zfs-tests/tests/functional/refquota/Makefile
	tests/zfs-tests/tests/functional/refreserv/Makefile
	tests/zfs-tests/tests/functional/removal/Makefile
	tests/zfs-tests/tests/functional/rename_dirs/Makefile
	tests/zfs-tests/tests/functional/replacement/Makefile
	tests/zfs-tests/tests/functional/reservation/Makefile
	tests/zfs-tests/tests/functional/rootpool/Makefile
	tests/zfs-tests/tests/functional/rsend/Makefile
	tests/zfs-tests/tests/functional/scrub_mirror/Makefile
	tests/zfs-tests/tests/functional/slog/Makefile
	tests/zfs-tests/tests/functional/snapshot/Makefile
	tests/zfs-tests/tests/functional/snapused/Makefile
	tests/zfs-tests/tests/functional/sparse/Makefile
	tests/zfs-tests/tests/functional/suid/Makefile
	tests/zfs-tests/tests/functional/threadsappend/Makefile
	tests/zfs-tests/tests/functional/tmpfile/Makefile
	tests/zfs-tests/tests/functional/trim/Makefile
	tests/zfs-tests/tests/functional/truncate/Makefile
	tests/zfs-tests/tests/functional/upgrade/Makefile
	tests/zfs-tests/tests/functional/user_namespace/Makefile
	tests/zfs-tests/tests/functional/userquota/Makefile
	tests/zfs-tests/tests/functional/vdev_zaps/Makefile
	tests/zfs-tests/tests/functional/write_dirs/Makefile
	tests/zfs-tests/tests/functional/xattr/Makefile
	tests/zfs-tests/tests/functional/zvol/Makefile
	tests/zfs-tests/tests/functional/zvol/zvol_ENOSPC/Makefile
	tests/zfs-tests/tests/functional/zvol/zvol_cli/Makefile
	tests/zfs-tests/tests/functional/zvol/zvol_misc/Makefile
	tests/zfs-tests/tests/functional/zvol/zvol_swap/Makefile
	tests/zfs-tests/tests/perf/Makefile
	tests/zfs-tests/tests/perf/fio/Makefile
	tests/zfs-tests/tests/perf/regression/Makefile
	tests/zfs-tests/tests/perf/scripts/Makefile
	tests/zfs-tests/tests/stress/Makefile
	udev/Makefile
	udev/rules.d/Makefile
	zfs.release
	])


	AC_OUTPUT
	Index: vendor-sys/openzfs/dist/contrib/dracut/90zfs/module-setup.sh.in
	===================================================================
	--- vendor-sys/openzfs/dist/contrib/dracut/90zfs/module-setup.sh.in (revision 366773)
	+++ vendor-sys/openzfs/dist/contrib/dracut/90zfs/module-setup.sh.in (revision 366774)
	@@ -1,116 +1,117 @@
	#!/usr/bin/env bash

	check() {
	# We depend on udev-rules being loaded
	[ "${1}" = "-d" ] && return 0

	# Verify the zfs tool chain
	for tool in "@bindir@/zgenhostid" "@sbindir@/zpool" "@sbindir@/zfs" "@mounthelperdir@/mount.zfs" ; do
	test -x "$tool" \|\| return 1
	done
	# Verify grep exists
	which grep >/dev/null 2>&1 \|\| return 1

	return 0
	}

	depends() {
	echo udev-rules
	return 0
	}

	installkernel() {
	instmods zfs
	instmods zcommon
	instmods znvpair
	instmods zavl
	instmods zunicode
	instmods zlua
	instmods icp
	instmods spl
	instmods zlib_deflate
	instmods zlib_inflate
	}

	install() {
	inst_rules @udevruledir@/90-zfs.rules
	inst_rules @udevruledir@/69-vdev.rules
	inst_rules @udevruledir@/60-zvol.rules
	dracut_install hostid
	dracut_install grep
	dracut_install @bindir@/zgenhostid
	dracut_install @sbindir@/zfs
	dracut_install @sbindir@/zpool
	- # Workaround for zfsonlinux/zfs#4749 by ensuring libgcc_s.so(.1) is included
	+ # Workaround for https://github.com/openzfs/zfs/issues/4749 by
	+ # ensuring libgcc_s.so(.1) is included
	if [[ -n "$(ldd @sbindir@/zpool \| grep -F 'libgcc_s.so')" ]]; then
	# Dracut will have already tracked and included it
	:;
	elif command -v gcc-config 2>&1 1>/dev/null; then
	# On systems with gcc-config (Gentoo, Funtoo, etc.):
	# Use the current profile to resolve the appropriate path
	dracut_install "/usr/lib/gcc/$(s=$(gcc-config -c); echo ${s%-}/${s##-})/libgcc_s.so.1"
	elif [[ -n "$(ls /usr/lib/libgcc_s.so* 2>/dev/null)" ]]; then
	# Try a simple path first
	dracut_install /usr/lib/libgcc_s.so*
	else
	# Fallback: Guess the path and include all matches
	dracut_install /usr/lib/gcc///libgcc_s.so*
	fi
	dracut_install @mounthelperdir@/mount.zfs
	dracut_install @udevdir@/vdev_id
	dracut_install awk
	dracut_install basename
	dracut_install cut
	dracut_install head
	dracut_install @udevdir@/zvol_id
	inst_hook cmdline 95 "${moddir}/parse-zfs.sh"
	if [ -n "$systemdutildir" ] ; then
	inst_script "${moddir}/zfs-generator.sh" "$systemdutildir"/system-generators/dracut-zfs-generator
	fi
	inst_hook pre-mount 90 "${moddir}/zfs-load-key.sh"
	inst_hook mount 98 "${moddir}/mount-zfs.sh"
	inst_hook cleanup 99 "${moddir}/zfs-needshutdown.sh"
	inst_hook shutdown 20 "${moddir}/export-zfs.sh"

	inst_simple "${moddir}/zfs-lib.sh" "/lib/dracut-zfs-lib.sh"
	if [ -e @sysconfdir@/zfs/zpool.cache ]; then
	inst @sysconfdir@/zfs/zpool.cache
	type mark_hostonly >/dev/null 2>&1 && mark_hostonly @sysconfdir@/zfs/zpool.cache
	fi

	if [ -e @sysconfdir@/zfs/vdev_id.conf ]; then
	inst @sysconfdir@/zfs/vdev_id.conf
	type mark_hostonly >/dev/null 2>&1 && mark_hostonly @sysconfdir@/zfs/vdev_id.conf
	fi

	# Synchronize initramfs and system hostid
	zgenhostid -o "${initdir}/etc/hostid" "$(hostid)"

	if dracut_module_included "systemd"; then
	mkdir -p "${initdir}/$systemdsystemunitdir/zfs-import.target.wants"
	for _item in scan cache ; do
	dracut_install @systemdunitdir@/zfs-import-$_item.service
	if ! [ -L "${initdir}/$systemdsystemunitdir/zfs-import.target.wants"/zfs-import-$_item.service ]; then
	ln -s ../zfs-import-$_item.service "${initdir}/$systemdsystemunitdir/zfs-import.target.wants"/zfs-import-$_item.service
	type mark_hostonly >/dev/null 2>&1 && mark_hostonly @systemdunitdir@/zfs-import-$_item.service
	fi
	done
	inst "${moddir}"/zfs-env-bootfs.service "${systemdsystemunitdir}"/zfs-env-bootfs.service
	ln -s ../zfs-env-bootfs.service "${initdir}/${systemdsystemunitdir}/zfs-import.target.wants"/zfs-env-bootfs.service
	type mark_hostonly >/dev/null 2>&1 && mark_hostonly @systemdunitdir@/zfs-env-bootfs.service
	dracut_install systemd-ask-password
	dracut_install systemd-tty-ask-password-agent
	mkdir -p "${initdir}/$systemdsystemunitdir/initrd.target.wants"
	dracut_install @systemdunitdir@/zfs-import.target
	if ! [ -L "${initdir}/$systemdsystemunitdir/initrd.target.wants"/zfs-import.target ]; then
	ln -s ../zfs-import.target "${initdir}/$systemdsystemunitdir/initrd.target.wants"/zfs-import.target
	type mark_hostonly >/dev/null 2>&1 && mark_hostonly @systemdunitdir@/zfs-import.target
	fi
	for _service in zfs-snapshot-bootfs.service zfs-rollback-bootfs.service ; do
	inst "${moddir}"/$_service "${systemdsystemunitdir}"/$_service
	if ! [ -L "${initdir}/$systemdsystemunitdir/initrd.target.wants"/$_service ]; then
	ln -s ../$_service "${initdir}/$systemdsystemunitdir/initrd.target.wants"/$_service
	fi
	done
	fi
	}
	Index: vendor-sys/openzfs/dist/contrib/pyzfs/README
	===================================================================
	--- vendor-sys/openzfs/dist/contrib/pyzfs/README (revision 366773)
	+++ vendor-sys/openzfs/dist/contrib/pyzfs/README (revision 366774)
	@@ -1,28 +1,28 @@
	This package provides a wrapper for libzfs_core C library.

	libzfs_core is intended to be a stable interface for programmatic
	administration of ZFS.
	This wrapper provides one-to-one wrappers for libzfs_core API functions,
	but the signatures and types are more natural to Python.
	nvlists are wrapped as dictionaries or lists depending on their usage.
	Some parameters have default values depending on typical use for
	increased convenience.
	Enumerations and bit flags become strings and lists of strings in Python.
	Errors are reported as exceptions rather than integer errno-style
	error codes. The wrapper takes care to provide one-to-many mapping
	of the error codes to the exceptions by interpreting a context
	in which the error code is produced.

	Unit tests and automated test for the libzfs_core API are provided
	with this package.
	Please note that the API tests perform lots of ZFS dataset level
	operations and ZFS tries hard to ensure that any modifications
	do reach stable storage. That means that the operations are done
	synchronously and that, for example, disk caches are flushed.
	Thus, the tests can be very slow on real hardware.
	It is recommended to place the default temporary directory or
	a temporary directory specified by, for instance, TMP environment
	variable on a memory backed filesystem.

	Package documentation: http://pyzfs.readthedocs.org
	-Package development: https://github.com/zfsonlinux/zfs
	+Package development: https://github.com/openzfs/zfs
	Index: vendor-sys/openzfs/dist/contrib/pyzfs/libzfs_core/__init__.py
	===================================================================
	--- vendor-sys/openzfs/dist/contrib/pyzfs/libzfs_core/__init__.py (revision 366773)
	+++ vendor-sys/openzfs/dist/contrib/pyzfs/libzfs_core/__init__.py (revision 366774)
	@@ -1,154 +1,154 @@
	#
	# Copyright 2015 ClusterHQ
	#
	# Licensed under the Apache License, Version 2.0 (the "License");
	# you may not use this file except in compliance with the License.
	# You may obtain a copy of the License at
	#
	# http://www.apache.org/licenses/LICENSE-2.0
	#
	# Unless required by applicable law or agreed to in writing, software
	# distributed under the License is distributed on an "AS IS" BASIS,
	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	# See the License for the specific language governing permissions and
	# limitations under the License.
	#

	'''
	Python wrappers for libzfs_core library.

	libzfs_core is intended to be a stable, committed interface for programmatic
	administration of ZFS.
	This wrapper provides one-to-one wrappers for libzfs_core API functions,
	but the signatures and types are more natural to Python.
	nvlists are wrapped as dictionaries or lists depending on their usage.
	Some parameters have default values depending on typical use for
	increased convenience.
	Output parameters are not used and return values are directly returned.
	Enumerations and bit flags become strings and lists of strings in Python.
	Errors are reported as exceptions rather than integer errno-style
	error codes. The wrapper takes care to provide one-to-many mapping
	of the error codes to the exceptions by interpreting a context
	in which the error code is produced.

	To submit an issue or contribute to development of this package
	-please visit its `GitHub repository <https://github.com/zfsonlinux/zfs>`_.
	+please visit its `GitHub repository <https://github.com/openzfs/zfs>`_.

	.. data:: MAXNAMELEN

	Maximum length of any ZFS name.
	'''
	from __future__ import absolute_import, division, print_function

	from ._constants import (
	MAXNAMELEN,
	ZCP_DEFAULT_INSTRLIMIT,
	ZCP_DEFAULT_MEMLIMIT,
	WRAPPING_KEY_LEN,
	zfs_key_location,
	zfs_keyformat,
	zio_encrypt
	)

	from ._libzfs_core import (
	lzc_bookmark,
	lzc_change_key,
	lzc_channel_program,
	lzc_channel_program_nosync,
	lzc_clone,
	lzc_create,
	lzc_destroy_bookmarks,
	lzc_destroy_snaps,
	lzc_exists,
	lzc_get_bookmarks,
	lzc_get_holds,
	lzc_hold,
	lzc_load_key,
	lzc_pool_checkpoint,
	lzc_pool_checkpoint_discard,
	lzc_promote,
	lzc_receive,
	lzc_receive_one,
	lzc_receive_resumable,
	lzc_receive_with_cmdprops,
	lzc_receive_with_header,
	lzc_release,
	lzc_reopen,
	lzc_rollback,
	lzc_rollback_to,
	lzc_send,
	lzc_send_resume,
	lzc_send_space,
	lzc_snaprange_space,
	lzc_snapshot,
	lzc_sync,
	lzc_unload_key,
	is_supported,
	lzc_recv,
	lzc_snap,
	lzc_rename,
	lzc_destroy,
	lzc_inherit_prop,
	lzc_get_props,
	lzc_set_props,
	lzc_list_children,
	lzc_list_snaps,
	receive_header,
	)

	__all__ = [
	'ctypes',
	'exceptions',
	'MAXNAMELEN',
	'ZCP_DEFAULT_INSTRLIMIT',
	'ZCP_DEFAULT_MEMLIMIT',
	'WRAPPING_KEY_LEN',
	'zfs_key_location',
	'zfs_keyformat',
	'zio_encrypt',
	'lzc_bookmark',
	'lzc_change_key',
	'lzc_channel_program',
	'lzc_channel_program_nosync',
	'lzc_clone',
	'lzc_create',
	'lzc_destroy_bookmarks',
	'lzc_destroy_snaps',
	'lzc_exists',
	'lzc_get_bookmarks',
	'lzc_get_holds',
	'lzc_hold',
	'lzc_load_key',
	'lzc_pool_checkpoint',
	'lzc_pool_checkpoint_discard',
	'lzc_promote',
	'lzc_receive',
	'lzc_receive_one',
	'lzc_receive_resumable',
	'lzc_receive_with_cmdprops',
	'lzc_receive_with_header',
	'lzc_release',
	'lzc_reopen',
	'lzc_rollback',
	'lzc_rollback_to',
	'lzc_send',
	'lzc_send_resume',
	'lzc_send_space',
	'lzc_snaprange_space',
	'lzc_snapshot',
	'lzc_sync',
	'lzc_unload_key',
	'is_supported',
	'lzc_recv',
	'lzc_snap',
	'lzc_rename',
	'lzc_destroy',
	'lzc_inherit_prop',
	'lzc_get_props',
	'lzc_set_props',
	'lzc_list_children',
	'lzc_list_snaps',
	'receive_header',
	]

	# vim: softtabstop=4 tabstop=4 expandtab shiftwidth=4
	Index: vendor-sys/openzfs/dist/copy-builtin
	===================================================================
	--- vendor-sys/openzfs/dist/copy-builtin (revision 366773)
	+++ vendor-sys/openzfs/dist/copy-builtin (revision 366774)
	@@ -1,79 +1,79 @@
	#!/usr/bin/env bash

	set -e

	usage()
	{
	echo "usage: $0 <kernel source tree>" >&2
	exit 1
	}

	[ "$#" -eq 1 ] \|\| usage
	KERNEL_DIR="$(readlink --canonicalize-existing "$1")"

	if ! [ -e 'zfs_config.h' ]
	then
	echo >&2
	echo " $0: you did not run configure, or you're not in the ZFS source directory." >&2
	echo " $0: run configure with --with-linux=$KERNEL_DIR and --enable-linux-builtin." >&2
	echo >&2
	exit 1
	fi

	make clean \|\| true
	make gitrev

	rm -rf "$KERNEL_DIR/include/zfs" "$KERNEL_DIR/fs/zfs"
	cp --recursive include "$KERNEL_DIR/include/zfs"
	cp --recursive module "$KERNEL_DIR/fs/zfs"
	cp zfs_config.h "$KERNEL_DIR/include/zfs/"

	cat > "$KERNEL_DIR/fs/zfs/Kconfig" <<"EOF"
	config ZFS
	tristate "ZFS filesystem support"
	depends on EFI_PARTITION
	select ZLIB_INFLATE
	select ZLIB_DEFLATE
	help
	- This is the ZFS filesystem from the ZFS On Linux project.
	+ This is the ZFS filesystem from the OpenZFS project.

	- See https://zfsonlinux.org/
	+ See https://github.com/openzfs/zfs

	To compile this file system support as a module, choose M here.

	If unsure, say N.
	EOF

	add_after()
	{
	local FILE="$1"
	local MARKER="$2"
	local NEW="$3"
	local LINE

	while IFS='' read -r LINE
	do
	echo "$LINE"

	if [ -n "$MARKER" -a "$LINE" = "$MARKER" ]
	then
	echo "$NEW"
	MARKER=''
	if IFS='' read -r LINE
	then
	[ "$LINE" != "$NEW" ] && echo "$LINE"
	fi
	fi
	done < "$FILE" > "$FILE.new"

	mv "$FILE.new" "$FILE"
	}

	add_after "$KERNEL_DIR/fs/Kconfig" 'if BLOCK' 'source "fs/zfs/Kconfig"'
	add_after "$KERNEL_DIR/fs/Makefile" 'endif' 'obj-$(CONFIG_ZFS) += zfs/'

	echo >&2
	echo " $0: done." >&2
	echo " $0: now you can build the kernel with ZFS support." >&2
	echo " $0: make sure you enable ZFS support (CONFIG_ZFS) before building." >&2
	echo >&2
	Index: vendor-sys/openzfs/dist/etc/init.d/README.md
	===================================================================
	--- vendor-sys/openzfs/dist/etc/init.d/README.md (revision 366773)
	+++ vendor-sys/openzfs/dist/etc/init.d/README.md (revision 366774)
	@@ -1,72 +1,72 @@
	DESCRIPTION
	These script were written with the primary intention of being portable and
	usable on as many systems as possible.

	This is, in practice, usually not possible. But the intention is there.
	And it is a good one.

	They have been tested successfully on:

	* Debian GNU/Linux Wheezy
	* Debian GNU/Linux Jessie
	* Ubuntu Trusty
	* CentOS 6.0
	* CentOS 6.6
	* Gentoo

	SUPPORT
	If you find that they don't work for your platform, please report this
	- at the ZFS On Linux issue tracker at https://github.com/zfsonlinux/zfs/issues.
	+ at the OpenZFS issue tracker at https://github.com/openzfs/zfs/issues.

	Please include:

	* Distribution name
	* Distribution version
	* Where to find an install CD image
	* Architecture

	If you have code to share that fixes the problem, that is much better.
	But please remember to try your best keep portability in mind. If you
	suspect that what you're writing/modifying won't work on anything else
	than your distribution, please make sure to put that code in appropriate
	if/else/fi code.

	It currently MUST be bash (or fully compatible) for this to work.

	If you're making your own distribution and you want the scripts to
	work on that, the biggest problem you'll (probably) have is the part
	at the beginning of the "zfs-functions" file which sets up the
	logging output.

	INSTALLING INIT SCRIPT LINKS
	To setup the init script links in /etc/rc?.d manually on a Debian GNU/Linux
	(or derived) system, run the following commands (the order is important!):

	update-rc.d zfs-import start 07 S . stop 07 0 1 6 .
	update-rc.d zfs-mount start 02 2 3 4 5 . stop 06 0 1 6 .
	update-rc.d zfs-zed start 07 2 3 4 5 . stop 08 0 1 6 .
	update-rc.d zfs-share start 27 2 3 4 5 . stop 05 0 1 6 .

	To do the same on RedHat, Fedora and/or CentOS:

	chkconfig zfs-import
	chkconfig zfs-mount
	chkconfig zfs-zed
	chkconfig zfs-share

	On Gentoo:

	rc-update add zfs-import boot
	rc-update add zfs-mount boot
	rc-update add zfs-zed default
	rc-update add zfs-share default

	The idea here is to make sure all of the ZFS filesystems, including possibly
	separate datasets like /var, are mounted before anything else is started.

	Then, ZED, which depends on /var, can be started. It will consume and act
	on events that occurred before it started. ZED may also play a role in
	sharing filesystems in the future, so it is important to start before the
	'share' service.

	Finally, we share filesystems configured with the share\* property.
	Index: vendor-sys/openzfs/dist/include/os/freebsd/linux/compiler.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/linux/compiler.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/linux/compiler.h (revision 366774)
	@@ -1,101 +1,101 @@
	/*
	* Copyright (c) 2010 Isilon Systems, Inc.
	* Copyright (c) 2010 iXsystems, Inc.
	* Copyright (c) 2010 Panasas, Inc.
	* Copyright (c) 2013-2016 Mellanox Technologies, Ltd.
	* Copyright (c) 2015 François Tigeot
	* All rights reserved.
	*
	* Redistribution and use in source and binary forms, with or without
	* modification, are permitted provided that the following conditions
	* are met:
	* 1. Redistributions of source code must retain the above copyright
	* notice unmodified, this list of conditions, and the following
	* disclaimer.
	* 2. Redistributions in binary form must reproduce the above copyright
	* notice, this list of conditions and the following disclaimer in the
	* documentation and/or other materials provided with the distribution.
	*
	* THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
	* IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
	* OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
	* IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
	* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
	* NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
	* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
	* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
	* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
	* THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
	*
	* $FreeBSD$
	*/
	#ifndef _LINUX_COMPILER_H_
	#define _LINUX_COMPILER_H_

	#include <sys/cdefs.h>

	#define __user
	#define __kernel
	#define __safe
	#define __force
	#define __nocast
	#define __iomem
	#define __chk_user_ptr(x) ((void)0)
	#define __chk_io_ptr(x) ((void)0)
	#define __builtin_warning(x, y...) (1)
	#define __acquires(x)
	#define __releases(x)
	#define __acquire(x) do { } while (0)
	#define __release(x) do { } while (0)
	#define __cond_lock(x, c) (c)
	#define __bitwise
	#define __devinitdata
	#define __deprecated
	#define __init
	#define __initconst
	#define __devinit
	#define __devexit
	#define __exit
	#define __rcu
	#define __percpu
	#define __weak __weak_symbol
	#define __malloc
	#define ___stringify(...) #__VA_ARGS__
	#define __stringify(...) ___stringify(__VA_ARGS__)
	#define __attribute_const__ __attribute__((__const__))
	#undef __always_inline
	#define __always_inline inline
	#define noinline __noinline
	#define ____cacheline_aligned __aligned(CACHE_LINE_SIZE)

	-#ifndef _KERNEL
	+#if !defined(_KERNEL) && !defined(_STANDALONE)
	#define likely(x) __builtin_expect(!!(x), 1)
	#define unlikely(x) __builtin_expect(!!(x), 0)
	#endif
	#define typeof(x) __typeof(x)

	#define uninitialized_var(x) x = x
	#define __maybe_unused __unused
	#define __always_unused __unused
	#define __must_check __result_use_check

	#define __printf(a, b) __printflike(a, b)

	#define barrier() __asm__ __volatile__("": : :"memory")
	#define smp_rmb() rmb()
	#define ___PASTE(a, b) a##b
	#define __PASTE(a, b) ___PASTE(a, b)

	#define ACCESS_ONCE(x) ((volatile __typeof(x) )&(x))

	#define WRITE_ONCE(x, v) do { \
	barrier(); \
	ACCESS_ONCE(x) = (v); \
	barrier(); \
	} while (0)

	#define lockless_dereference(p) READ_ONCE(p)

	#define _AT(T, X) ((T)(X))

	#endif /* _LINUX_COMPILER_H_ */
	Index: vendor-sys/openzfs/dist/include/os/freebsd/spl/rpc/xdr.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/spl/rpc/xdr.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/spl/rpc/xdr.h (revision 366774)
	@@ -1,71 +1,71 @@
	/*
	* Sun RPC is a product of Sun Microsystems, Inc. and is provided for
	* unrestricted use provided that this legend is included on all tape
	* media and as a part of the software program in whole or part. Users
	* may copy or modify Sun RPC without charge, but are not authorized
	* to license or distribute it to anyone else except as part of a product or
	* program developed by the user.
	*
	* SUN RPC IS PROVIDED AS IS WITH NO WARRANTIES OF ANY KIND INCLUDING THE
	* WARRANTIES OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR
	* PURPOSE, OR ARISING FROM A COURSE OF DEALING, USAGE OR TRADE PRACTICE.
	*
	* Sun RPC is provided with no support and without any obligation on the
	* part of Sun Microsystems, Inc. to assist in its use, correction,
	* modification or enhancement.
	*
	* SUN MICROSYSTEMS, INC. SHALL HAVE NO LIABILITY WITH RESPECT TO THE
	* INFRINGEMENT OF COPYRIGHTS, TRADE SECRETS OR ANY PATENTS BY SUN RPC
	* OR ANY PART THEREOF.
	*
	* In no event will Sun Microsystems, Inc. be liable for any lost revenue
	* or profits or other special, indirect and consequential damages, even if
	* Sun has been advised of the possibility of such damages.
	*
	* Sun Microsystems, Inc.
	* 2550 Garcia Avenue
	* Mountain View, California 94043
	*/

	#ifndef _OPENSOLARIS_RPC_XDR_H_
	#define _OPENSOLARIS_RPC_XDR_H_

	#include <rpc/types.h>
	#include_next <rpc/xdr.h>

	-#ifndef _KERNEL
	+#if !defined(_KERNEL) && !defined(_STANDALONE)

	#include <assert.h>

	/*
	* Taken from sys/xdr/xdr_mem.c.
	*
	* FreeBSD's userland XDR doesn't implement control method (only the kernel),
	* but OpenSolaris nvpair still depend on it, so we have to implement it here.
	*/
	static __inline bool_t
	xdrmem_control(XDR xdrs, int request, void info)
	{
	xdr_bytesrec *xptr;

	switch (request) {
	case XDR_GET_BYTES_AVAIL:
	xptr = (xdr_bytesrec *)info;
	xptr->xc_is_last_record = TRUE;
	xptr->xc_num_avail = xdrs->x_handy;
	return (TRUE);
	default:
	assert(!"unexpected request");
	}
	return (FALSE);
	}

	#undef XDR_CONTROL
	#define XDR_CONTROL(xdrs, req, op) \
	(((xdrs)->x_ops->x_control == NULL) ? \
	xdrmem_control((xdrs), (req), (op)) : \
	(*(xdrs)->x_ops->x_control)(xdrs, req, op))

	-#endif /* !_KERNEL */
	+#endif /* !_KERNEL && !_STANDALONE */

	#endif /* !_OPENSOLARIS_RPC_XDR_H_ */
	Index: vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/atomic.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/atomic.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/atomic.h (revision 366774)
	@@ -1,182 +1,193 @@
	/*
	* Copyright (c) 2007 Pawel Jakub Dawidek <pjd@FreeBSD.org>
	* All rights reserved.
	*
	* Redistribution and use in source and binary forms, with or without
	* modification, are permitted provided that the following conditions
	* are met:
	* 1. Redistributions of source code must retain the above copyright
	* notice, this list of conditions and the following disclaimer.
	* 2. Redistributions in binary form must reproduce the above copyright
	* notice, this list of conditions and the following disclaimer in the
	* documentation and/or other materials provided with the distribution.
	*
	* THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
	* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
	* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
	* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
	* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
	* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
	* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
	* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
	* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
	* SUCH DAMAGE.
	*
	* $FreeBSD$
	*/

	#ifndef _OPENSOLARIS_SYS_ATOMIC_H_
	#define _OPENSOLARIS_SYS_ATOMIC_H_

	+#ifndef _STANDALONE
	+
	#include <sys/types.h>
	#include <machine/atomic.h>

	#define atomic_sub_64 atomic_subtract_64

	#if defined(__i386__) && (defined(_KERNEL) \|\| defined(KLD_MODULE))
	#define I386_HAVE_ATOMIC64
	#endif

	#if defined(__i386__) \|\| defined(__amd64__) \|\| defined(__arm__)
	/* No spurious failures from fcmpset. */
	#define STRONG_FCMPSET
	#endif

	#if !defined(__LP64__) && !defined(__mips_n32) && \
	!defined(ARM_HAVE_ATOMIC64) && !defined(I386_HAVE_ATOMIC64) && \
	!defined(HAS_EMULATED_ATOMIC64)
	extern void atomic_add_64(volatile uint64_t *target, int64_t delta);
	extern void atomic_dec_64(volatile uint64_t *target);
	extern uint64_t atomic_swap_64(volatile uint64_t *a, uint64_t value);
	extern uint64_t atomic_load_64(volatile uint64_t *a);
	extern uint64_t atomic_add_64_nv(volatile uint64_t *target, int64_t delta);
	extern uint64_t atomic_cas_64(volatile uint64_t *target, uint64_t cmp,
	uint64_t newval);
	#endif

	#define membar_producer atomic_thread_fence_rel

	static __inline uint32_t
	atomic_add_32_nv(volatile uint32_t *target, int32_t delta)
	{
	return (atomic_fetchadd_32(target, delta) + delta);
	}

	static __inline uint_t
	atomic_add_int_nv(volatile uint_t *target, int delta)
	{
	return (atomic_add_32_nv(target, delta));
	}

	static __inline void
	atomic_inc_32(volatile uint32_t *target)
	{
	atomic_add_32(target, 1);
	}

	static __inline uint32_t
	atomic_inc_32_nv(volatile uint32_t *target)
	{
	return (atomic_add_32_nv(target, 1));
	}

	static __inline void
	atomic_dec_32(volatile uint32_t *target)
	{
	atomic_subtract_32(target, 1);
	}

	static __inline uint32_t
	atomic_dec_32_nv(volatile uint32_t *target)
	{
	return (atomic_add_32_nv(target, -1));
	}

	#ifndef __sparc64__
	static inline uint32_t
	atomic_cas_32(volatile uint32_t *target, uint32_t cmp, uint32_t newval)
	{
	#ifdef STRONG_FCMPSET
	(void) atomic_fcmpset_32(target, &cmp, newval);
	#else
	uint32_t expected = cmp;

	do {
	if (atomic_fcmpset_32(target, &cmp, newval))
	break;
	} while (cmp == expected);
	#endif
	return (cmp);
	}
	#endif

	#if defined(__LP64__) \|\| defined(__mips_n32) \|\| \
	defined(ARM_HAVE_ATOMIC64) \|\| defined(I386_HAVE_ATOMIC64) \|\| \
	defined(HAS_EMULATED_ATOMIC64)
	static __inline void
	atomic_dec_64(volatile uint64_t *target)
	{
	atomic_subtract_64(target, 1);
	}

	static inline uint64_t
	atomic_add_64_nv(volatile uint64_t *target, int64_t delta)
	{
	return (atomic_fetchadd_64(target, delta) + delta);
	}

	#ifndef __sparc64__
	static inline uint64_t
	atomic_cas_64(volatile uint64_t *target, uint64_t cmp, uint64_t newval)
	{
	#ifdef STRONG_FCMPSET
	(void) atomic_fcmpset_64(target, &cmp, newval);
	#else
	uint64_t expected = cmp;

	do {
	if (atomic_fcmpset_64(target, &cmp, newval))
	break;
	} while (cmp == expected);
	#endif
	return (cmp);
	}
	#endif
	#endif

	static __inline void
	atomic_inc_64(volatile uint64_t *target)
	{
	atomic_add_64(target, 1);
	}

	static __inline uint64_t
	atomic_inc_64_nv(volatile uint64_t *target)
	{
	return (atomic_add_64_nv(target, 1));
	}

	static __inline uint64_t
	atomic_dec_64_nv(volatile uint64_t *target)
	{
	return (atomic_add_64_nv(target, -1));
	}

	#if !defined(COMPAT_32BIT) && defined(__LP64__)
	static __inline void *
	atomic_cas_ptr(volatile void target, void cmp, void *newval)
	{
	return ((void )atomic_cas_64((volatile uint64_t )target,
	(uint64_t)cmp, (uint64_t)newval));
	}
	#else
	static __inline void *
	atomic_cas_ptr(volatile void target, void cmp, void *newval)
	{
	return ((void )atomic_cas_32((volatile uint32_t )target,
	(uint32_t)cmp, (uint32_t)newval));
	}
	#endif /* !defined(COMPAT_32BIT) && defined(__LP64__) */
	+
	+#else /* _STANDALONE */
	+/*
	+ * sometimes atomic_add_64 is defined, sometimes not, but the
	+ * following is always right for the boot loader.
	+ */
	+#undef atomic_add_64
	+#define atomic_add_64(ptr, val) *(ptr) += val
	+#endif /* !_STANDALONE */

	#endif /* !_OPENSOLARIS_SYS_ATOMIC_H_ */
	Index: vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/byteorder.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/byteorder.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/byteorder.h (revision 366774)
	@@ -1,107 +1,109 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*
	* $FreeBSD$
	*/

	/*
	* Copyright 2005 Sun Microsystems, Inc. All rights reserved.
	* Use is subject to license terms.
	*/

	/* Copyright (c) 1983, 1984, 1985, 1986, 1987, 1988, 1989 AT&T */
	/* All Rights Reserved */

	/*
	* University Copyright- Copyright (c) 1982, 1986, 1988
	* The Regents of the University of California
	* All Rights Reserved
	*
	* University Acknowledgment- Portions of this document are derived from
	* software developed by the University of California, Berkeley, and its
	* contributors.
	*/

	#ifndef _OPENSOLARIS_SYS_BYTEORDER_H_
	#define _OPENSOLARIS_SYS_BYTEORDER_H_

	#include <sys/endian.h>

	/*
	* Macros to reverse byte order
	*/
	#define BSWAP_8(x) ((x) & 0xff)
	#define BSWAP_16(x) ((BSWAP_8(x) << 8) \| BSWAP_8((x) >> 8))
	#define BSWAP_32(x) ((BSWAP_16(x) << 16) \| BSWAP_16((x) >> 16))
	#define BSWAP_64(x) ((BSWAP_32(x) << 32) \| BSWAP_32((x) >> 32))

	#define BMASK_8(x) ((x) & 0xff)
	#define BMASK_16(x) ((x) & 0xffff)
	#define BMASK_32(x) ((x) & 0xffffffff)
	#define BMASK_64(x) (x)

	/*
	* Macros to convert from a specific byte order to/from native byte order
	*/
	#if BYTE_ORDER == _BIG_ENDIAN
	#define BE_8(x) BMASK_8(x)
	#define BE_16(x) BMASK_16(x)
	#define BE_32(x) BMASK_32(x)
	#define BE_64(x) BMASK_64(x)
	#define LE_8(x) BSWAP_8(x)
	#define LE_16(x) BSWAP_16(x)
	#define LE_32(x) BSWAP_32(x)
	#define LE_64(x) BSWAP_64(x)
	#else
	#define LE_8(x) BMASK_8(x)
	#define LE_16(x) BMASK_16(x)
	#define LE_32(x) BMASK_32(x)
	#define LE_64(x) BMASK_64(x)
	#define BE_8(x) BSWAP_8(x)
	#define BE_16(x) BSWAP_16(x)
	#define BE_32(x) BSWAP_32(x)
	#define BE_64(x) BSWAP_64(x)
	#endif

	+#if !defined(_STANDALONE)
	#if BYTE_ORDER == _BIG_ENDIAN
	#define htonll(x) BMASK_64(x)
	#define ntohll(x) BMASK_64(x)
	-#else
	+#else /* BYTE_ORDER == _LITTLE_ENDIAN */
	#ifndef __LP64__
	static __inline__ uint64_t
	htonll(uint64_t n)
	{
	return ((((uint64_t)htonl(n)) << 32) + htonl(n >> 32));
	}

	static __inline__ uint64_t
	ntohll(uint64_t n)
	{
	return ((((uint64_t)ntohl(n)) << 32) + ntohl(n >> 32));
	}
	-#else
	+#else /* !__LP64__ */
	#define htonll(x) BSWAP_64(x)
	#define ntohll(x) BSWAP_64(x)
	-#endif
	-#endif
	+#endif /* __LP64__ */
	+#endif /* BYTE_ORDER */
	+#endif /* _STANDALONE */

	#define BE_IN32(xa) htonl(((uint32_t )(void *)(xa)))

	#endif /* _OPENSOLARIS_SYS_BYTEORDER_H_ */
	Index: vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/ccompile.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/ccompile.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/ccompile.h (revision 366774)
	@@ -1,274 +1,280 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License, Version 1.0 only
	* (the "License"). You may not use this file except in compliance
	* with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright 2004 Sun Microsystems, Inc. All rights reserved.
	* Use is subject to license terms.
	*/

	#ifndef _SYS_CCOMPILE_H
	#define _SYS_CCOMPILE_H

	/*
	* This file contains definitions designed to enable different compilers
	* to be used harmoniously on Solaris systems.
	*/

	#ifdef __cplusplus
	extern "C" {
	#endif

	/*
	* Allow for version tests for compiler bugs and features.
	*/
	#if defined(__GNUC__)
	#define __GNUC_VERSION \
	(__GNUC__ * 10000 + __GNUC_MINOR__ * 100 + __GNUC_PATCHLEVEL__)
	#else
	#define __GNUC_VERSION 0
	#endif

	#if defined(__ATTRIBUTE_IMPLEMENTED) \|\| defined(__GNUC__)

	/*
	* analogous to lint's PRINTFLIKEn
	*/
	#define __sun_attr___PRINTFLIKE__(__n) \
	__attribute__((__format__(printf, __n, (__n)+1)))
	#define __sun_attr___VPRINTFLIKE__(__n) \
	__attribute__((__format__(printf, __n, 0)))

	/*
	* Handle the kernel printf routines that can take '%b' too
	*/
	#if __GNUC_VERSION < 30402
	/*
	* XX64 at least this doesn't work correctly yet with 3.4.1 anyway!
	*/
	#define __sun_attr___KPRINTFLIKE__ __sun_attr___PRINTFLIKE__
	#define __sun_attr___KVPRINTFLIKE__ __sun_attr___VPRINTFLIKE__
	#else
	#define __sun_attr___KPRINTFLIKE__(__n) \
	__attribute__((__format__(cmn_err, __n, (__n)+1)))
	#define __sun_attr___KVPRINTFLIKE__(__n) \
	__attribute__((__format__(cmn_err, __n, 0)))
	#endif

	/*
	* This one's pretty obvious -- the function never returns
	*/
	#define __sun_attr___noreturn__ __attribute__((__noreturn__))


	/*
	* This is an appropriate label for functions that do not
	* modify their arguments, e.g. strlen()
	*/
	#define __sun_attr___pure__ __attribute__((__pure__))

	/*
	* This is a stronger form of __pure__. Can be used for functions
	* that do not modify their arguments and don't depend on global
	* memory.
	*/
	#define __sun_attr___const__ __attribute__((__const__))

	/*
	* structure packing like #pragma pack(1)
	*/
	#define __sun_attr___packed__ __attribute__((__packed__))

	#define ___sun_attr_inner(__a) __sun_attr_##__a
	#define __sun_attr__(__a) ___sun_attr_inner __a

	#else /* __ATTRIBUTE_IMPLEMENTED \|\| __GNUC__ */

	#define __sun_attr__(__a)

	#endif /* __ATTRIBUTE_IMPLEMENTED \|\| __GNUC__ */

	/*
	* Shorthand versions for readability
	*/

	#define __PRINTFLIKE(__n) __sun_attr__((__PRINTFLIKE__(__n)))
	#define __VPRINTFLIKE(__n) __sun_attr__((__VPRINTFLIKE__(__n)))
	#define __KPRINTFLIKE(__n) __sun_attr__((__KPRINTFLIKE__(__n)))
	#define __KVPRINTFLIKE(__n) __sun_attr__((__KVPRINTFLIKE__(__n)))
	-#ifdef _KERNEL
	+#if defined(_KERNEL) \|\| defined(_STANDALONE)
	#define __NORETURN __sun_attr__((__noreturn__))
	-#endif
	+#endif /* _KERNEL \|\| _STANDALONE */
	#define __CONST __sun_attr__((__const__))
	#define __PURE __sun_attr__((__pure__))

	#if defined(INVARIANTS) && !defined(ZFS_DEBUG)
	#define ZFS_DEBUG
	#undef NDEBUG
	#endif

	#define EXPORT_SYMBOL(x)
	#define MODULE_AUTHOR(s)
	#define MODULE_DESCRIPTION(s)
	#define MODULE_LICENSE(s)
	#define module_param(a, b, c)
	#define module_param_call(a, b, c, d, e)
	#define module_param_named(a, b, c, d)
	#define MODULE_PARM_DESC(a, b)
	#define asm __asm
	#ifdef ZFS_DEBUG
	#undef NDEBUG
	#endif
	#if !defined(ZFS_DEBUG) && !defined(NDEBUG)
	#define NDEBUG
	#endif

	#ifndef EINTEGRITY
	#define EINTEGRITY 97 /* EINTEGRITY is new in 13 */
	#endif

	/*
	* These are bespoke errnos used in ZFS. We map them to their closest FreeBSD
	* equivalents. This gives us more useful error messages from strerror(3).
	*/
	#define ECKSUM EINTEGRITY
	#define EFRAGS ENOSPC

	/* Similar for ENOACTIVE */
	#define ENOTACTIVE ECANCELED

	#define EREMOTEIO EREMOTE
	#define ECHRNG ENXIO
	#define ETIME ETIMEDOUT

	#define O_LARGEFILE 0
	#define O_RSYNC 0
	#define O_DSYNC 0

	#ifndef LOCORE
	#ifndef HAVE_RPC_TYPES
	typedef int bool_t;
	typedef int enum_t;
	#endif
	#endif

	#ifndef __cplusplus
	#define __init
	#define __exit
	#endif

	-#ifdef _KERNEL
	+#if defined(_KERNEL) \|\| defined(_STANDALONE)
	#define param_set_charp(a, b) (0)
	#define ATTR_UID AT_UID
	#define ATTR_GID AT_GID
	#define ATTR_MODE AT_MODE
	#define ATTR_XVATTR AT_XVATTR
	#define ATTR_CTIME AT_CTIME
	#define ATTR_MTIME AT_MTIME
	#define ATTR_ATIME AT_ATIME
	+#if defined(_STANDALONE)
	+#define vmem_free kmem_free
	+#define vmem_zalloc kmem_zalloc
	+#define vmem_alloc kmem_zalloc
	+#else
	#define vmem_free zfs_kmem_free
	#define vmem_zalloc(size, flags) zfs_kmem_alloc(size, flags \| M_ZERO)
	#define vmem_alloc zfs_kmem_alloc
	+#endif
	#define MUTEX_NOLOCKDEP 0
	#define RW_NOLOCKDEP 0

	#else
	#define FALSE 0
	#define TRUE 1
	/*
	* XXX We really need to consolidate on standard
	* error codes in the common code
	*/
	#define ENOSTR ENOTCONN
	#define ENODATA EINVAL


	#define __BSD_VISIBLE 1
	#ifndef IN_BASE
	#define __POSIX_VISIBLE 201808
	#define __XSI_VISIBLE 1000
	#endif
	#define ARRAY_SIZE(a) (sizeof (a) / sizeof (a[0]))
	#define mmap64 mmap
	/* Note: this file can be used on linux/macOS when bootstrapping tools. */
	#if defined(__FreeBSD__)
	#define open64 open
	#define pwrite64 pwrite
	#define ftruncate64 ftruncate
	#define lseek64 lseek
	#define pread64 pread
	#define stat64 stat
	#define lstat64 lstat
	#define statfs64 statfs
	#define readdir64 readdir
	#define dirent64 dirent
	#endif
	#define P2ALIGN(x, align) ((x) & -(align))
	#define P2CROSS(x, y, align) (((x) ^ (y)) > (align) - 1)
	#define P2ROUNDUP(x, align) ((((x) - 1) \| ((align) - 1)) + 1)
	#define P2PHASE(x, align) ((x) & ((align) - 1))
	#define P2NPHASE(x, align) (-(x) & ((align) - 1))
	#define ISP2(x) (((x) & ((x) - 1)) == 0)
	#define IS_P2ALIGNED(v, a) ((((uintptr_t)(v)) & ((uintptr_t)(a) - 1)) == 0)
	#define P2BOUNDARY(off, len, align) \
	(((off) ^ ((off) + (len) - 1)) > (align) - 1)

	/*
	* Typed version of the P2* macros. These macros should be used to ensure
	* that the result is correctly calculated based on the data type of (x),
	* which is passed in as the last argument, regardless of the data
	* type of the alignment. For example, if (x) is of type uint64_t,
	* and we want to round it up to a page boundary using "PAGESIZE" as
	* the alignment, we can do either
	*
	* P2ROUNDUP(x, (uint64_t)PAGESIZE)
	* or
	* P2ROUNDUP_TYPED(x, PAGESIZE, uint64_t)
	*/
	#define P2ALIGN_TYPED(x, align, type) \
	((type)(x) & -(type)(align))
	#define P2PHASE_TYPED(x, align, type) \
	((type)(x) & ((type)(align) - 1))
	#define P2NPHASE_TYPED(x, align, type) \
	(-(type)(x) & ((type)(align) - 1))
	#define P2ROUNDUP_TYPED(x, align, type) \
	((((type)(x) - 1) \| ((type)(align) - 1)) + 1)
	#define P2END_TYPED(x, align, type) \
	(-(~(type)(x) & -(type)(align)))
	#define P2PHASEUP_TYPED(x, align, phase, type) \
	((type)(phase) - (((type)(phase) - (type)(x)) & -(type)(align)))
	#define P2CROSS_TYPED(x, y, align, type) \
	(((type)(x) ^ (type)(y)) > (type)(align) - 1)
	#define P2SAMEHIGHBIT_TYPED(x, y, type) \
	(((type)(x) ^ (type)(y)) < ((type)(x) & (type)(y)))

	#define DIV_ROUND_UP(n, d) (((n) + (d) - 1) / (d))
	#define RLIM64_INFINITY RLIM_INFINITY
	#ifndef HAVE_ERESTART
	#define ERESTART EAGAIN
	#endif
	#define ABS(a) ((a) < 0 ? -(a) : (a))

	#endif
	#ifdef __cplusplus
	}
	#endif

	#endif /* _SYS_CCOMPILE_H */
	Index: vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/cmn_err.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/cmn_err.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/cmn_err.h (revision 366774)
	@@ -1,98 +1,89 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License, Version 1.0 only
	* (the "License"). You may not use this file except in compliance
	* with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/* Copyright (c) 1984, 1986, 1987, 1988, 1989 AT&T */
	/* All Rights Reserved */


	/*
	* Copyright 2004 Sun Microsystems, Inc. All rights reserved.
	* Use is subject to license terms.
	*/

	#ifndef _SYS_CMN_ERR_H
	#define _SYS_CMN_ERR_H

	#if !defined(_ASM)
	#include <sys/_stdarg.h>
	#endif

	#ifdef __cplusplus
	extern "C" {
	#endif

	/* Common error handling severity levels */

	#define CE_CONT 0 /* continuation */
	#define CE_NOTE 1 /* notice */
	#define CE_WARN 2 /* warning */
	#define CE_PANIC 3 /* panic */
	#define CE_IGNORE 4 /* print nothing */

	#ifndef _ASM

	/PRINTFLIKE2/
	extern void cmn_err(int, const char *, ...)
	__KPRINTFLIKE(2);
	-#pragma rarely_called(cmn_err)

	extern void vzcmn_err(zoneid_t, int, const char *, __va_list)
	__KVPRINTFLIKE(3);
	-#pragma rarely_called(vzcmn_err)

	extern void vcmn_err(int, const char *, __va_list)
	__KVPRINTFLIKE(2);
	-#pragma rarely_called(vcmn_err)

	/PRINTFLIKE3/
	extern void zcmn_err(zoneid_t, int, const char *, ...)
	__KPRINTFLIKE(3);
	-#pragma rarely_called(zcmn_err)

	extern void vzprintf(zoneid_t, const char *, __va_list)
	__KVPRINTFLIKE(2);
	-#pragma rarely_called(vzprintf)

	/PRINTFLIKE2/
	extern void zprintf(zoneid_t, const char *, ...)
	__KPRINTFLIKE(2);
	-#pragma rarely_called(zprintf)

	extern void vuprintf(const char *, __va_list)
	__KVPRINTFLIKE(1);
	-#pragma rarely_called(vuprintf)

	/PRINTFLIKE1/
	extern void panic(const char *, ...)
	__KPRINTFLIKE(1) __NORETURN;
	-#pragma rarely_called(panic)

	extern void vpanic(const char *, __va_list)
	__KVPRINTFLIKE(1) __NORETURN;
	-#pragma rarely_called(vpanic)

	#endif /* !_ASM */

	#ifdef __cplusplus
	}
	#endif

	#endif /* _SYS_CMN_ERR_H */
	Index: vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/condvar.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/condvar.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/condvar.h (revision 366774)
	@@ -1,211 +1,212 @@
	/*
	* Copyright (c) 2007 Pawel Jakub Dawidek <pjd@FreeBSD.org>
	* Copyright (c) 2013 iXsystems, Inc.
	* All rights reserved.
	*
	* Redistribution and use in source and binary forms, with or without
	* modification, are permitted provided that the following conditions
	* are met:
	* 1. Redistributions of source code must retain the above copyright
	* notice, this list of conditions and the following disclaimer.
	* 2. Redistributions in binary form must reproduce the above copyright
	* notice, this list of conditions and the following disclaimer in the
	* documentation and/or other materials provided with the distribution.
	*
	* THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
	* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
	* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
	* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
	* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
	* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
	* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
	* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
	* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
	* SUCH DAMAGE.
	*
	* $FreeBSD$
	*/

	#ifndef _OPENSOLARIS_SYS_CONDVAR_H_
	#define _OPENSOLARIS_SYS_CONDVAR_H_

	#include <sys/param.h>
	#include <sys/systm.h>

	#include <sys/spl_condvar.h>
	#include <sys/mutex.h>
	#include <sys/time.h>
	+#include <sys/errno.h>

	/*
	* cv_timedwait() is similar to cv_wait() except that it additionally expects
	* a timeout value specified in ticks. When woken by cv_signal() or
	* cv_broadcast() it returns 1, otherwise when the timeout is reached -1 is
	* returned.
	*
	* cv_timedwait_sig() behaves the same as cv_timedwait() but blocks
	* interruptibly and can be woken by a signal (EINTR, ERESTART). When
	* this occurs 0 is returned.
	*
	* cv_timedwait_io() and cv_timedwait_sig_io() are variants of cv_timedwait()
	* and cv_timedwait_sig() which should be used when waiting for outstanding
	* IO to complete. They are responsible for updating the iowait accounting
	* when this is supported by the platform.
	*
	* cv_timedwait_hires() and cv_timedwait_sig_hires() are high resolution
	* versions of cv_timedwait() and cv_timedwait_sig(). They expect the timeout
	* to be specified as a hrtime_t allowing for timeouts of less than a tick.
	*
	* N.B. The return values differ slightly from the illumos implementation
	* which returns the time remaining, instead of 1, when woken. They both
	* return -1 on timeout. Consumers which need to know the time remaining
	* are responsible for tracking it themselves.
	*/

	static __inline sbintime_t
	zfs_nstosbt(int64_t _ns)
	{
	sbintime_t sb = 0;

	#ifdef KASSERT
	KASSERT(_ns >= 0, ("Negative values illegal for nstosbt: %jd", _ns));
	#endif
	if (_ns >= SBT_1S) {
	sb = (_ns / 1000000000) * SBT_1S;
	_ns = _ns % 1000000000;
	}
	/* 9223372037 = ceil(2^63 / 1000000000) */
	sb += ((_ns * 9223372037ull) + 0x7fffffff) >> 31;
	return (sb);
	}


	typedef struct cv kcondvar_t;
	#define CALLOUT_FLAG_ABSOLUTE C_ABSOLUTE

	typedef enum {
	CV_DEFAULT,
	CV_DRIVER
	} kcv_type_t;

	#define zfs_cv_init(cv, name, type, arg) do { \
	const char *_name; \
	ASSERT((type) == CV_DEFAULT); \
	for (_name = #cv; *_name != '\0'; _name++) { \
	if (_name >= 'a' && _name <= 'z') \
	break; \
	} \
	if (*_name == '\0') \
	_name = #cv; \
	cv_init((cv), _name); \
	} while (0)
	#define cv_init(cv, name, type, arg) zfs_cv_init(cv, name, type, arg)


	static inline int
	cv_wait_sig(kcondvar_t cvp, kmutex_t mp)
	{

	return (_cv_wait_sig(cvp, &(mp)->lock_object) == 0);
	}

	static inline int
	cv_timedwait(kcondvar_t cvp, kmutex_t mp, clock_t timo)
	{
	int rc;

	timo -= ddi_get_lbolt();
	if (timo <= 0)
	return (-1);
	rc = _cv_timedwait_sbt((cvp), &(mp)->lock_object, \
	tick_sbt * (timo), 0, C_HARDCLOCK);
	if (rc == EWOULDBLOCK)
	return (-1);
	return (1);
	}

	static inline int
	cv_timedwait_sig(kcondvar_t cvp, kmutex_t mp, clock_t timo)
	{
	int rc;

	timo -= ddi_get_lbolt();
	if (timo <= 0)
	return (-1);
	rc = _cv_timedwait_sig_sbt(cvp, &(mp)->lock_object, \
	tick_sbt * (timo), 0, C_HARDCLOCK);
	if (rc == EWOULDBLOCK)
	return (-1);
	if (rc == EINTR \|\| rc == ERESTART)
	return (0);

	return (1);
	}

	#define cv_timedwait_io cv_timedwait
	#define cv_timedwait_idle cv_timedwait
	#define cv_timedwait_sig_io cv_timedwait_sig
	#define cv_wait_io cv_wait
	#define cv_wait_io_sig cv_wait_sig
	#define cv_wait_idle cv_wait
	#define cv_timedwait_io_hires cv_timedwait_hires
	#define cv_timedwait_idle_hires cv_timedwait_hires

	static inline int
	cv_timedwait_hires(kcondvar_t cvp, kmutex_t mp, hrtime_t tim, hrtime_t res,
	int flag)
	{
	hrtime_t hrtime;
	int rc;

	ASSERT(tim >= res);

	hrtime = gethrtime();
	if (flag == 0)
	tim += hrtime;

	if (hrtime >= tim)
	return (-1);
	rc = cv_timedwait_sbt(cvp, mp, zfs_nstosbt(tim),
	zfs_nstosbt(res), C_ABSOLUTE);

	if (rc == EWOULDBLOCK)
	return (-1);

	KASSERT(rc == 0, ("unexpected rc value %d", rc));
	return (1);
	}

	static inline int
	cv_timedwait_sig_hires(kcondvar_t cvp, kmutex_t mp, hrtime_t tim,
	hrtime_t res, int flag)
	{
	sbintime_t sbt;
	hrtime_t hrtime;
	int rc;

	ASSERT(tim >= res);

	hrtime = gethrtime();
	if (flag == 0)
	tim += hrtime;

	if (hrtime >= tim)
	return (-1);

	sbt = zfs_nstosbt(tim);
	rc = cv_timedwait_sig_sbt(cvp, mp, sbt, zfs_nstosbt(res), C_ABSOLUTE);

	switch (rc) {
	case EWOULDBLOCK:
	return (-1);
	case EINTR:
	case ERESTART:
	return (0);
	default:
	KASSERT(rc == 0, ("unexpected rc value %d", rc));
	return (1);
	}
	}

	#endif /* _OPENSOLARIS_SYS_CONDVAR_H_ */
	Index: vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/kmem.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/kmem.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/kmem.h (revision 366774)
	@@ -1,97 +1,107 @@
	/*
	* Copyright (c) 2007 Pawel Jakub Dawidek <pjd@FreeBSD.org>
	* All rights reserved.
	*
	* Redistribution and use in source and binary forms, with or without
	* modification, are permitted provided that the following conditions
	* are met:
	* 1. Redistributions of source code must retain the above copyright
	* notice, this list of conditions and the following disclaimer.
	* 2. Redistributions in binary form must reproduce the above copyright
	* notice, this list of conditions and the following disclaimer in the
	* documentation and/or other materials provided with the distribution.
	*
	* THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
	* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
	* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
	* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
	* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
	* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
	* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
	* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
	* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
	* SUCH DAMAGE.
	*
	* $FreeBSD$
	*/

	#ifndef _OPENSOLARIS_SYS_KMEM_H_
	#define _OPENSOLARIS_SYS_KMEM_H_

	+#ifdef _KERNEL
	#include <sys/param.h>
	#include <sys/malloc.h>
	#include <sys/vmem.h>
	#include <sys/counter.h>

	#include <vm/uma.h>
	#include <vm/vm.h>
	#include <vm/vm_extern.h>

	MALLOC_DECLARE(M_SOLARIS);

	#define POINTER_IS_VALID(p) (!((uintptr_t)(p) & 0x3))
	#define POINTER_INVALIDATE(pp) ((pp) = (void )((uintptr_t)(*(pp)) \| 0x1))

	#define KM_SLEEP M_WAITOK
	#define KM_PUSHPAGE M_WAITOK
	#define KM_NOSLEEP M_NOWAIT
	#define KM_NORMALPRI 0
	#define KMC_NODEBUG UMA_ZONE_NODUMP

	typedef struct vmem vmem_t;

	extern char kmem_asprintf(const char , ...);
	extern char kmem_vasprintf(const char fmt, va_list ap);

	typedef struct kmem_cache {
	char kc_name[32];
	#if !defined(KMEM_DEBUG)
	uma_zone_t kc_zone;
	#else
	size_t kc_size;
	#endif
	int (kc_constructor)(void , void *, int);
	void (kc_destructor)(void , void *);
	void *kc_private;
	} kmem_cache_t;

	extern uint64_t spl_kmem_cache_inuse(kmem_cache_t *cache);
	extern uint64_t spl_kmem_cache_entry_size(kmem_cache_t *cache);

	void *zfs_kmem_alloc(size_t size, int kmflags);
	void zfs_kmem_free(void *buf, size_t size);
	uint64_t kmem_size(void);
	kmem_cache_t kmem_cache_create(char name, size_t bufsize, size_t align,
	int (constructor)(void , void , int), void (destructor)(void , void ),
	void (reclaim)(void ) __unused, void private, vmem_t vmp, int cflags);
	void kmem_cache_destroy(kmem_cache_t *cache);
	void kmem_cache_alloc(kmem_cache_t cache, int flags);
	void kmem_cache_free(kmem_cache_t cache, void buf);
	boolean_t kmem_cache_reap_active(void);
	void kmem_cache_reap_soon(kmem_cache_t *);
	void kmem_reap(void);
	int kmem_debugging(void);
	void *calloc(size_t n, size_t s);


	#define kmem_cache_reap_now kmem_cache_reap_soon
	#define freemem vm_free_count()
	#define minfree vm_cnt.v_free_min
	#define kmem_alloc(size, kmflags) zfs_kmem_alloc((size), (kmflags))
	#define kmem_zalloc(size, kmflags) \
	zfs_kmem_alloc((size), (kmflags) \| M_ZERO)
	#define kmem_free(buf, size) zfs_kmem_free((buf), (size))

	+#endif /* _KERNEL */
	+
	+#ifdef _STANDALONE
	+/*
	+ * At the moment, we just need it for the type. We redirect the alloc/free
	+ * routines to the usual Free and Malloc in that environment.
	+ */
	+typedef int kmem_cache_t;
	+#endif /* _STANDALONE */

	#endif /* _OPENSOLARIS_SYS_KMEM_H_ */
	Index: vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/kmem_cache.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/kmem_cache.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/kmem_cache.h (revision 366774)
	@@ -1,49 +1,52 @@
	/*
	* Copyright (c) 2020 iXsystems, Inc.
	* All rights reserved.
	*
	* Redistribution and use in source and binary forms, with or without
	* modification, are permitted provided that the following conditions
	* are met:
	* 1. Redistributions of source code must retain the above copyright
	* notice, this list of conditions and the following disclaimer.
	* 2. Redistributions in binary form must reproduce the above copyright
	* notice, this list of conditions and the following disclaimer in the
	* documentation and/or other materials provided with the distribution.
	*
	* THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
	* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
	* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
	* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
	* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
	* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
	* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
	* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
	* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
	* SUCH DAMAGE.
	*
	* $FreeBSD$
	*/


	#ifndef _SPL_KMEM_CACHE_H
	#define _SPL_KMEM_CACHE_H

	+#ifdef _KERNEL
	#include <sys/taskq.h>

	/* kmem move callback return values */
	typedef enum kmem_cbrc {
	KMEM_CBRC_YES = 0, /* Object moved */
	KMEM_CBRC_NO = 1, /* Object not moved */
	KMEM_CBRC_LATER = 2, /* Object not moved, try again later */
	KMEM_CBRC_DONT_NEED = 3, /* Neither object is needed */
	KMEM_CBRC_DONT_KNOW = 4, /* Object unknown */
	} kmem_cbrc_t;

	extern void spl_kmem_cache_set_move(kmem_cache_t *,
	kmem_cbrc_t ()(void , void , size_t, void ));

	#define kmem_cache_set_move(skc, move) spl_kmem_cache_set_move(skc, move)
	+
	+#endif /* _KERNEL */

	#endif
	Index: vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/kstat.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/kstat.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/kstat.h (revision 366774)
	@@ -1,225 +1,234 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_KSTAT_H
	#define _SPL_KSTAT_H
	+
	#include <sys/types.h>
	+#ifndef _STANDALONE
	#include <sys/sysctl.h>
	+#endif
	struct list_head {};
	#include <sys/mutex.h>
	#include <sys/proc.h>

	#define KSTAT_STRLEN 255
	#define KSTAT_RAW_MAX (128*1024)

	/*
	* For reference valid classes are:
	* disk, tape, net, controller, vm, kvm, hat, streams, kstat, misc
	*/

	#define KSTAT_TYPE_RAW 0 /* can be anything; ks_ndata >= 1 */
	#define KSTAT_TYPE_NAMED 1 /* name/value pair; ks_ndata >= 1 */
	#define KSTAT_TYPE_INTR 2 /* interrupt stats; ks_ndata == 1 */
	#define KSTAT_TYPE_IO 3 /* I/O stats; ks_ndata == 1 */
	#define KSTAT_TYPE_TIMER 4 /* event timer; ks_ndata >= 1 */
	#define KSTAT_NUM_TYPES 5

	#define KSTAT_DATA_CHAR 0
	#define KSTAT_DATA_INT32 1
	#define KSTAT_DATA_UINT32 2
	#define KSTAT_DATA_INT64 3
	#define KSTAT_DATA_UINT64 4
	#define KSTAT_DATA_LONG 5
	#define KSTAT_DATA_ULONG 6
	#define KSTAT_DATA_STRING 7
	#define KSTAT_NUM_DATAS 8

	#define KSTAT_INTR_HARD 0
	#define KSTAT_INTR_SOFT 1
	#define KSTAT_INTR_WATCHDOG 2
	#define KSTAT_INTR_SPURIOUS 3
	#define KSTAT_INTR_MULTSVC 4
	#define KSTAT_NUM_INTRS 5

	#define KSTAT_FLAG_VIRTUAL 0x01
	#define KSTAT_FLAG_VAR_SIZE 0x02
	#define KSTAT_FLAG_WRITABLE 0x04
	#define KSTAT_FLAG_PERSISTENT 0x08
	#define KSTAT_FLAG_DORMANT 0x10
	#define KSTAT_FLAG_INVALID 0x20
	#define KSTAT_FLAG_LONGSTRINGS 0x40
	#define KSTAT_FLAG_NO_HEADERS 0x80

	#define KS_MAGIC 0x9d9d9d9d

	/* Dynamic updates */
	#define KSTAT_READ 0
	#define KSTAT_WRITE 1

	struct kstat_s;
	typedef struct kstat_s kstat_t;

	typedef int kid_t; /* unique kstat id */
	typedef int kstat_update_t(struct kstat_s , int); / dynamic update cb */

	struct seq_file {
	char *sf_buf;
	size_t sf_size;
	};

	void seq_printf(struct seq_file m, const char fmt, ...);


	typedef struct kstat_module {
	char ksm_name[KSTAT_STRLEN+1]; /* module name */
	struct list_head ksm_module_list; /* module linkage */
	struct list_head ksm_kstat_list; /* list of kstat entries */
	struct proc_dir_entry ksm_proc; / proc entry */
	} kstat_module_t;

	typedef struct kstat_raw_ops {
	int (headers)(char buf, size_t size);
	int (seq_headers)(struct seq_file );
	int (data)(char buf, size_t size, void *data);
	void (addr)(kstat_t *ksp, loff_t index);
	} kstat_raw_ops_t;

	struct kstat_s {
	int ks_magic; /* magic value */
	kid_t ks_kid; /* unique kstat ID */
	hrtime_t ks_crtime; /* creation time */
	hrtime_t ks_snaptime; /* last access time */
	char ks_module[KSTAT_STRLEN+1]; /* provider module name */
	int ks_instance; /* provider module instance */
	char ks_name[KSTAT_STRLEN+1]; /* kstat name */
	char ks_class[KSTAT_STRLEN+1]; /* kstat class */
	uchar_t ks_type; /* kstat data type */
	uchar_t ks_flags; /* kstat flags */
	void ks_data; / kstat type-specific data */
	uint_t ks_ndata; /* # of data records */
	size_t ks_data_size; /* size of kstat data section */
	kstat_update_t ks_update; / dynamic updates */
	void ks_private; / private data */
	void ks_private1; / private data */
	kmutex_t ks_private_lock; /* kstat private data lock */
	kmutex_t ks_lock; / kstat data lock */
	struct list_head ks_list; /* kstat linkage */
	kstat_module_t ks_owner; / kstat module linkage */
	kstat_raw_ops_t ks_raw_ops; /* ops table for raw type */
	char ks_raw_buf; / buf used for raw ops */
	size_t ks_raw_bufsize; /* size of raw ops buffer */
	+#ifndef _STANDALONE
	struct sysctl_ctx_list ks_sysctl_ctx;
	struct sysctl_oid *ks_sysctl_root;
	-
	+#endif /* _STANDALONE */
	};

	typedef struct kstat_named_s {
	char name[KSTAT_STRLEN]; /* name of counter */
	uchar_t data_type; /* data type */
	union {
	char c[16]; /* 128-bit int */
	int32_t i32; /* 32-bit signed int */
	uint32_t ui32; /* 32-bit unsigned int */
	int64_t i64; /* 64-bit signed int */
	uint64_t ui64; /* 64-bit unsigned int */
	long l; /* native signed long */
	ulong_t ul; /* native unsigned long */
	struct {
	union {
	char ptr; / NULL-term string */
	char __pad[8]; /* 64-bit padding */
	} addr;
	uint32_t len; /* # bytes for strlen + '\0' */
	} string;
	} value;
	} kstat_named_t;

	#define KSTAT_NAMED_STR_PTR(knptr) ((knptr)->value.string.addr.ptr)
	#define KSTAT_NAMED_STR_BUFLEN(knptr) ((knptr)->value.string.len)

	typedef struct kstat_intr {
	uint_t intrs[KSTAT_NUM_INTRS];
	} kstat_intr_t;

	typedef struct kstat_io {
	u_longlong_t nread; /* number of bytes read */
	u_longlong_t nwritten; /* number of bytes written */
	uint_t reads; /* number of read operations */
	uint_t writes; /* number of write operations */
	hrtime_t wtime; /* cumulative wait (pre-service) time */
	hrtime_t wlentime; /* cumulative wait lentime product /
	hrtime_t wlastupdate; /* last time wait queue changed */
	hrtime_t rtime; /* cumulative run (service) time */
	hrtime_t rlentime; /* cumulative run lengthtime product /
	hrtime_t rlastupdate; /* last time run queue changed */
	uint_t wcnt; /* count of elements in wait state */
	uint_t rcnt; /* count of elements in run state */
	} kstat_io_t;

	typedef struct kstat_timer {
	char name[KSTAT_STRLEN+1]; /* event name */
	u_longlong_t num_events; /* number of events */
	hrtime_t elapsed_time; /* cumulative elapsed time */
	hrtime_t min_time; /* shortest event duration */
	hrtime_t max_time; /* longest event duration */
	hrtime_t start_time; /* previous event start time */
	hrtime_t stop_time; /* previous event stop time */
	} kstat_timer_t;

	int spl_kstat_init(void);
	void spl_kstat_fini(void);

	extern void __kstat_set_raw_ops(kstat_t *ksp,
	int (headers)(char buf, size_t size),
	int (data)(char buf, size_t size, void *data),
	void* (addr)(kstat_t ksp, loff_t index));

	extern void __kstat_set_seq_raw_ops(kstat_t *ksp,
	int (headers)(struct seq_file ),
	int (data)(char buf, size_t size, void *data),
	void* (addr)(kstat_t ksp, loff_t index));


	extern kstat_t __kstat_create(const char ks_module, int ks_instance,
	const char ks_name, const char ks_class, uchar_t ks_type,
	uint_t ks_ndata, uchar_t ks_flags);

	extern void __kstat_install(kstat_t *ksp);
	extern void __kstat_delete(kstat_t *ksp);
	extern void kstat_waitq_enter(kstat_io_t *);
	extern void kstat_waitq_exit(kstat_io_t *);
	extern void kstat_runq_enter(kstat_io_t *);
	extern void kstat_runq_exit(kstat_io_t *);

	#define kstat_set_seq_raw_ops(k, h, d, a) \
	__kstat_set_seq_raw_ops(k, h, d, a)
	#define kstat_set_raw_ops(k, h, d, a) \
	__kstat_set_raw_ops(k, h, d, a)
	+#ifndef _STANDALONE
	#define kstat_create(m, i, n, c, t, s, f) \
	__kstat_create(m, i, n, c, t, s, f)

	#define kstat_install(k) __kstat_install(k)
	#define kstat_delete(k) __kstat_delete(k)
	+#else
	+#define kstat_create(m, i, n, c, t, s, f) ((kstat_t *)0)
	+#define kstat_install(k)
	+#define kstat_delete(k)
	+#endif

	#endif /* _SPL_KSTAT_H */
	Index: vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/proc.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/proc.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/proc.h (revision 366774)
	@@ -1,114 +1,115 @@
	/*
	* Copyright (c) 2007 Pawel Jakub Dawidek <pjd@FreeBSD.org>
	* All rights reserved.
	*
	* Redistribution and use in source and binary forms, with or without
	* modification, are permitted provided that the following conditions
	* are met:
	* 1. Redistributions of source code must retain the above copyright
	* notice, this list of conditions and the following disclaimer.
	* 2. Redistributions in binary form must reproduce the above copyright
	* notice, this list of conditions and the following disclaimer in the
	* documentation and/or other materials provided with the distribution.
	*
	* THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
	* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
	* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
	* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
	* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
	* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
	* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
	* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
	* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
	* SUCH DAMAGE.
	*
	* $FreeBSD$
	*/

	#ifndef _OPENSOLARIS_SYS_PROC_H_
	#define _OPENSOLARIS_SYS_PROC_H_

	#include <sys/param.h>
	#include <sys/kthread.h>
	#include_next <sys/proc.h>
	#include <sys/stdint.h>
	#include <sys/smp.h>
	#include <sys/sched.h>
	#include <sys/lock.h>
	#include <sys/mutex.h>
	#include <sys/unistd.h>
	#include <sys/kmem.h>
	#include <sys/malloc.h>

	-
	+#ifdef _KERNEL
	#define CPU curcpu
	#define minclsyspri PRIBIO
	#define defclsyspri minclsyspri
	#define maxclsyspri PVM
	#define max_ncpus (mp_maxid + 1)
	#define boot_max_ncpus (mp_maxid + 1)

	#define TS_RUN 0

	#define p0 proc0

	#define t_tid td_tid

	typedef short pri_t;
	typedef struct thread _kthread;
	typedef struct thread kthread_t;
	typedef struct thread *kthread_id_t;
	typedef struct proc proc_t;

	extern proc_t *system_proc;

	static __inline kthread_t *
	do_thread_create(caddr_t stk, size_t stksize, void (proc)(void ), void *arg,
	size_t len, proc_t pp, int state, pri_t pri, const char name)
	{
	kthread_t *td = NULL;
	proc_t **ppp;
	int error;

	/*
	* Be sure there are no surprises.
	*/
	ASSERT(stk == NULL);
	ASSERT(len == 0);
	ASSERT(state == TS_RUN);

	if (pp == &p0)
	ppp = &system_proc;
	else
	ppp = &pp;
	error = kproc_kthread_add(proc, arg, ppp, &td, RFSTOPPED,
	stksize / PAGE_SIZE, "zfskern", "%s", name);
	if (error == 0) {
	thread_lock(td);
	sched_prio(td, pri);
	sched_add(td, SRQ_BORING);
	#if __FreeBSD_version < 1300068
	thread_unlock(td);
	#endif
	}
	return (td);
	}

	#define thread_create_named(name, stk, stksize, proc, arg, len, \
	pp, state, pri) \
	do_thread_create(stk, stksize, proc, arg, len, pp, state, pri, name)
	#define thread_create(stk, stksize, proc, arg, len, pp, state, pri) \
	do_thread_create(stk, stksize, proc, arg, len, pp, state, pri, #proc)
	#define thread_exit() kthread_exit()

	int uread(proc_t , void , size_t, uintptr_t);
	int uwrite(proc_t , void , size_t, uintptr_t);

	static inline boolean_t
	zfs_proc_is_caller(proc_t *p)
	{
	return (p == curproc);
	}

	+#endif /* _KERNEL */
	#endif /* _OPENSOLARIS_SYS_PROC_H_ */
	Index: vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/procfs_list.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/procfs_list.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/procfs_list.h (revision 366774)
	@@ -1,67 +1,73 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2018 by Delphix. All rights reserved.
	*/

	#ifndef _SPL_PROCFS_LIST_H
	#define _SPL_PROCFS_LIST_H

	+#ifndef _STANDALONE
	+
	#include <sys/kstat.h>
	#include <sys/mutex.h>


	/*
	* procfs list manipulation
	*/

	typedef struct procfs_list procfs_list_t;
	struct procfs_list {
	void *pl_private;
	void *pl_next_data;
	kmutex_t pl_lock;
	list_t pl_list;
	uint64_t pl_next_id;
	int (pl_show)(struct seq_file f, void *p);
	int (pl_show_header)(struct seq_file f);
	int (pl_clear)(procfs_list_t procfs_list);
	size_t pl_node_offset;
	};

	typedef struct procfs_list_node {
	list_node_t pln_link;
	uint64_t pln_id;
	} procfs_list_node_t;

	void procfs_list_install(const char *module,
	const char *submodule,
	const char *name,
	mode_t mode,
	procfs_list_t *procfs_list,
	int (show)(struct seq_file f, void *p),
	int (show_header)(struct seq_file f),
	int (clear)(procfs_list_t procfs_list),
	size_t procfs_list_node_off);
	void procfs_list_uninstall(procfs_list_t *procfs_list);
	void procfs_list_destroy(procfs_list_t *procfs_list);
	void procfs_list_add(procfs_list_t procfs_list, void p);
	+
	+#else
	+typedef int procfs_list_t;
	+#endif /* !_STANDALONE */

	#endif /* _SPL_PROCFS_LIST_H */
	Index: vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/sig.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/sig.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/sig.h (revision 366774)
	@@ -1,65 +1,70 @@
	/*
	* Copyright (c) 2008 Pawel Jakub Dawidek <pjd@FreeBSD.org>
	* All rights reserved.
	*
	* Redistribution and use in source and binary forms, with or without
	* modification, are permitted provided that the following conditions
	* are met:
	* 1. Redistributions of source code must retain the above copyright
	* notice, this list of conditions and the following disclaimer.
	* 2. Redistributions in binary form must reproduce the above copyright
	* notice, this list of conditions and the following disclaimer in the
	* documentation and/or other materials provided with the distribution.
	*
	* THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
	* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
	* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
	* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
	* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
	* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
	* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
	* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
	* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
	* SUCH DAMAGE.
	*
	* $FreeBSD$
	*/

	#ifndef _OPENSOLARIS_SYS_SIG_H_
	#define _OPENSOLARIS_SYS_SIG_H_

	+#ifndef _STANDALONE
	+
	#include_next <sys/signal.h>
	#include <sys/param.h>
	#include <sys/lock.h>
	#include <sys/mutex.h>
	#include <sys/proc.h>
	#include <sys/signalvar.h>
	#include <sys/debug.h>

	#define FORREAL 0
	#define JUSTLOOKING 1

	static __inline int
	issig(int why)
	{
	struct thread *td = curthread;
	struct proc *p;
	int sig;

	ASSERT(why == FORREAL \|\| why == JUSTLOOKING);
	if (SIGPENDING(td)) {
	if (why == JUSTLOOKING)
	return (1);
	p = td->td_proc;
	PROC_LOCK(p);
	mtx_lock(&p->p_sigacts->ps_mtx);
	sig = cursig(td);
	mtx_unlock(&p->p_sigacts->ps_mtx);
	PROC_UNLOCK(p);
	if (sig != 0)
	return (1);
	}
	return (0);
	}
	+
	+#endif /* !_STANDALONE */
	+
	#endif /* _OPENSOLARIS_SYS_SIG_H_ */
	Index: vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/simd_x86.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/simd_x86.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/simd_x86.h (revision 366774)
	@@ -1,300 +1,297 @@
	/*
	* Copyright (c) 2020 iXsystems, Inc.
	* All rights reserved.
	*
	* Redistribution and use in source and binary forms, with or without
	* modification, are permitted provided that the following conditions
	* are met:
	* 1. Redistributions of source code must retain the above copyright
	* notice, this list of conditions and the following disclaimer.
	* 2. Redistributions in binary form must reproduce the above copyright
	* notice, this list of conditions and the following disclaimer in the
	* documentation and/or other materials provided with the distribution.
	*
	* THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
	* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
	* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
	* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
	* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
	* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
	* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
	* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
	* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
	* SUCH DAMAGE.
	*
	* $FreeBSD$
	*/

	-#include <sys/cdefs.h>
	#include <sys/types.h>
	-#include <sys/systm.h>
	+#include <sys/cdefs.h>
	#include <sys/proc.h>
	-#ifdef __i386__
	-#include <x86/fpu.h>
	-#else
	-#include <machine/fpu.h>
	-#endif
	+#include <sys/systm.h>
	+
	+#include <machine/pcb.h>
	#include <x86/x86_var.h>
	#include <x86/specialreg.h>

	#define kfpu_init() (0)
	#define kfpu_fini() do {} while (0)
	#define kfpu_allowed() 1
	#define kfpu_initialize(tsk) do {} while (0)

	#define kfpu_begin() { \
	if (__predict_false(!is_fpu_kern_thread(0))) \
	fpu_kern_enter(curthread, NULL, FPU_KERN_NOCTX);\
	}

	#define kfpu_end() { \
	if (__predict_false(curpcb->pcb_flags & PCB_FPUNOSAVE)) \
	fpu_kern_leave(curthread, NULL); \
	}

	/*
	* Check if OS supports AVX and AVX2 by checking XCR0
	* Only call this function if CPUID indicates that AVX feature is
	* supported by the CPU, otherwise it might be an illegal instruction.
	*/
	static inline uint64_t
	xgetbv(uint32_t index)
	{
	uint32_t eax, edx;
	/* xgetbv - instruction byte code */
	__asm__ __volatile__(".byte 0x0f; .byte 0x01; .byte 0xd0"
	: "=a" (eax), "=d" (edx)
	: "c" (index));

	return ((((uint64_t)edx)<<32) \| (uint64_t)eax);
	}


	/*
	* Detect register set support
	*/
	static inline boolean_t
	__simd_state_enabled(const uint64_t state)
	{
	boolean_t has_osxsave;
	uint64_t xcr0;

	has_osxsave = !!(cpu_feature2 & CPUID2_OSXSAVE);

	if (!has_osxsave)
	return (B_FALSE);

	xcr0 = xgetbv(0);
	return ((xcr0 & state) == state);
	}

	#define _XSTATE_SSE_AVX (0x2 \| 0x4)
	#define _XSTATE_AVX512 (0xE0 \| _XSTATE_SSE_AVX)

	#define __ymm_enabled() __simd_state_enabled(_XSTATE_SSE_AVX)
	#define __zmm_enabled() __simd_state_enabled(_XSTATE_AVX512)


	/*
	* Check if SSE instruction set is available
	*/
	static inline boolean_t
	zfs_sse_available(void)
	{
	return (!!(cpu_feature & CPUID_SSE));
	}

	/*
	* Check if SSE2 instruction set is available
	*/
	static inline boolean_t
	zfs_sse2_available(void)
	{
	return (!!(cpu_feature & CPUID_SSE2));
	}

	/*
	* Check if SSE3 instruction set is available
	*/
	static inline boolean_t
	zfs_sse3_available(void)
	{
	return (!!(cpu_feature2 & CPUID2_SSE3));
	}

	/*
	* Check if SSSE3 instruction set is available
	*/
	static inline boolean_t
	zfs_ssse3_available(void)
	{
	return (!!(cpu_feature2 & CPUID2_SSSE3));
	}

	/*
	* Check if SSE4.1 instruction set is available
	*/
	static inline boolean_t
	zfs_sse4_1_available(void)
	{
	return (!!(cpu_feature2 & CPUID2_SSE41));
	}

	/*
	* Check if SSE4.2 instruction set is available
	*/
	static inline boolean_t
	zfs_sse4_2_available(void)
	{
	return (!!(cpu_feature2 & CPUID2_SSE42));
	}

	/*
	* Check if AVX instruction set is available
	*/
	static inline boolean_t
	zfs_avx_available(void)
	{
	boolean_t has_avx;

	has_avx = !!(cpu_feature2 & CPUID2_AVX);

	return (has_avx && __ymm_enabled());
	}

	/*
	* Check if AVX2 instruction set is available
	*/
	static inline boolean_t
	zfs_avx2_available(void)
	{
	boolean_t has_avx2;

	has_avx2 = !!(cpu_stdext_feature & CPUID_STDEXT_AVX2);

	return (has_avx2 && __ymm_enabled());
	}

	/*
	* AVX-512 family of instruction sets:
	*
	* AVX512F Foundation
	* AVX512CD Conflict Detection Instructions
	* AVX512ER Exponential and Reciprocal Instructions
	* AVX512PF Prefetch Instructions
	*
	* AVX512BW Byte and Word Instructions
	* AVX512DQ Double-word and Quadword Instructions
	* AVX512VL Vector Length Extensions
	*
	* AVX512IFMA Integer Fused Multiply Add (Not supported by kernel 4.4)
	* AVX512VBMI Vector Byte Manipulation Instructions
	*/


	/* Check if AVX512F instruction set is available */
	static inline boolean_t
	zfs_avx512f_available(void)
	{
	boolean_t has_avx512;

	has_avx512 = !!(cpu_stdext_feature & CPUID_STDEXT_AVX512F);

	return (has_avx512 && __zmm_enabled());
	}

	/* Check if AVX512CD instruction set is available */
	static inline boolean_t
	zfs_avx512cd_available(void)
	{
	boolean_t has_avx512;

	has_avx512 = !!(cpu_stdext_feature & CPUID_STDEXT_AVX512F) &&
	!!(cpu_stdext_feature & CPUID_STDEXT_AVX512CD);

	return (has_avx512 && __zmm_enabled());
	}

	/* Check if AVX512ER instruction set is available */
	static inline boolean_t
	zfs_avx512er_available(void)
	{
	boolean_t has_avx512;

	has_avx512 = !!(cpu_stdext_feature & CPUID_STDEXT_AVX512F) &&
	!!(cpu_stdext_feature & CPUID_STDEXT_AVX512CD);

	return (has_avx512 && __zmm_enabled());
	}

	/* Check if AVX512PF instruction set is available */
	static inline boolean_t
	zfs_avx512pf_available(void)
	{
	boolean_t has_avx512;

	has_avx512 = !!(cpu_stdext_feature & CPUID_STDEXT_AVX512F) &&
	!!(cpu_stdext_feature & CPUID_STDEXT_AVX512PF);

	return (has_avx512 && __zmm_enabled());
	}

	/* Check if AVX512BW instruction set is available */
	static inline boolean_t
	zfs_avx512bw_available(void)
	{
	boolean_t has_avx512 = B_FALSE;

	has_avx512 = !!(cpu_stdext_feature & CPUID_STDEXT_AVX512BW);

	return (has_avx512 && __zmm_enabled());
	}

	/* Check if AVX512DQ instruction set is available */
	static inline boolean_t
	zfs_avx512dq_available(void)
	{
	boolean_t has_avx512;

	has_avx512 = !!(cpu_stdext_feature & CPUID_STDEXT_AVX512F) &&
	!!(cpu_stdext_feature & CPUID_STDEXT_AVX512DQ);

	return (has_avx512 && __zmm_enabled());
	}

	/* Check if AVX512VL instruction set is available */
	static inline boolean_t
	zfs_avx512vl_available(void)
	{
	boolean_t has_avx512;

	has_avx512 = !!(cpu_stdext_feature & CPUID_STDEXT_AVX512F) &&
	!!(cpu_stdext_feature & CPUID_STDEXT_AVX512VL);

	return (has_avx512 && __zmm_enabled());
	}

	/* Check if AVX512IFMA instruction set is available */
	static inline boolean_t
	zfs_avx512ifma_available(void)
	{
	boolean_t has_avx512;

	has_avx512 = !!(cpu_stdext_feature & CPUID_STDEXT_AVX512F) &&
	!!(cpu_stdext_feature & CPUID_STDEXT_AVX512IFMA);

	return (has_avx512 && __zmm_enabled());
	}

	/* Check if AVX512VBMI instruction set is available */
	static inline boolean_t
	zfs_avx512vbmi_available(void)
	{
	boolean_t has_avx512;

	has_avx512 = !!(cpu_stdext_feature & CPUID_STDEXT_AVX512F) &&
	!!(cpu_stdext_feature & CPUID_STDEXT_BMI1);

	return (has_avx512 && __zmm_enabled());
	}
	Index: vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/sunddi.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/sunddi.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/sunddi.h (revision 366774)
	@@ -1,69 +1,68 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_SUNDDI_H
	#define _SPL_SUNDDI_H

	#include <sys/cred.h>
	#include <sys/uio.h>
	#include <sys/mutex.h>
	#include <sys/u8_textprep.h>
	#ifdef BUILDING_ZFS
	#include <sys/vnode.h>
	#endif

	typedef int ddi_devid_t;

	#define DDI_DEV_T_NONE ((dev_t)-1)
	#define DDI_DEV_T_ANY ((dev_t)-2)
	#define DI_MAJOR_T_UNKNOWN ((major_t)0)

	#define DDI_PROP_DONTPASS 0x0001
	#define DDI_PROP_CANSLEEP 0x0002

	#define DDI_SUCCESS 0
	#define DDI_FAILURE -1

	#define ddi_prop_lookup_string(x1, x2, x3, x4, x5) (*x5 = NULL)
	#define ddi_prop_free(x) (void)0
	#define ddi_root_node() (void)0

	extern int ddi_strtoul(const char , char , int, unsigned long );
	extern int ddi_strtol(const char , char , int, long );
	extern int ddi_strtoull(const char , char , int, unsigned long long );
	extern int ddi_strtoll(const char , char , int, long long );

	extern int ddi_copyin(const void from, void to, size_t len, int flags);
	extern int ddi_copyout(const void from, void to, size_t len, int flags);
	extern void ddi_sysevent_init(void);


	int ddi_soft_state_init(void **statep, size_t size, size_t nitems);
	void ddi_soft_state_fini(void **statep);

	void ddi_get_soft_state(void state, int item);
	int ddi_soft_state_zalloc(void *state, int item);
	void ddi_soft_state_free(void *state, int item);

	#endif /* SPL_SUNDDI_H */
	Index: vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/sysmacros.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/sysmacros.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/sysmacros.h (revision 366774)
	@@ -1,404 +1,409 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/* Copyright (c) 1984, 1986, 1987, 1988, 1989 AT&T */
	/* All Rights Reserved */


	/*
	* Copyright 2008 Sun Microsystems, Inc. All rights reserved.
	* Use is subject to license terms.
	*/

	#ifndef _SYS_SYSMACROS_H
	#define _SYS_SYSMACROS_H

	#include <sys/param.h>
	+#include <sys/systm.h>
	#include <sys/isa_defs.h>
	#include <sys/libkern.h>
	#include <sys/zone.h>
	#include <sys/condvar.h>

	#ifdef __cplusplus
	extern "C" {
	#endif

	/*
	* Some macros for units conversion
	*/
	/*
	* Disk blocks (sectors) and bytes.
	*/
	#define dtob(DD) ((DD) << DEV_BSHIFT)
	#define btod(BB) (((BB) + DEV_BSIZE - 1) >> DEV_BSHIFT)
	#define btodt(BB) ((BB) >> DEV_BSHIFT)
	#define lbtod(BB) (((offset_t)(BB) + DEV_BSIZE - 1) >> DEV_BSHIFT)

	/* common macros */
	#ifndef MIN
	#define MIN(a, b) ((a) < (b) ? (a) : (b))
	#endif
	#ifndef MAX
	#define MAX(a, b) ((a) < (b) ? (b) : (a))
	#endif
	#ifndef ABS
	#define ABS(a) ((a) < 0 ? -(a) : (a))
	#endif
	#ifndef SIGNOF
	#define SIGNOF(a) ((a) < 0 ? -1 : (a) > 0)
	#endif
	#ifndef ARRAY_SIZE
	#define ARRAY_SIZE(a) (sizeof (a) / sizeof (a[0]))
	#endif
	#ifndef DIV_ROUND_UP
	#define DIV_ROUND_UP(n, d) (((n) + (d) - 1) / (d))
	#endif

	+#ifdef _STANDALONE
	+#define boot_ncpus 1
	+#else /* _STANDALONE */
	#define boot_ncpus mp_ncpus
	+#endif /* _STANDALONE */
	#define kpreempt_disable() critical_enter()
	#define kpreempt_enable() critical_exit()
	#define CPU_SEQID curcpu
	#define is_system_labeled() 0
	/*
	* Convert a single byte to/from binary-coded decimal (BCD).
	*/
	extern unsigned char byte_to_bcd[256];
	extern unsigned char bcd_to_byte[256];

	#define BYTE_TO_BCD(x) byte_to_bcd[(x) & 0xff]
	#define BCD_TO_BYTE(x) bcd_to_byte[(x) & 0xff]

	/*
	* WARNING: The device number macros defined here should not be used by device
	* drivers or user software. Device drivers should use the device functions
	* defined in the DDI/DKI interface (see also ddi.h). Application software
	* should make use of the library routines available in makedev(3). A set of
	* new device macros are provided to operate on the expanded device number
	* format supported in SVR4. Macro versions of the DDI device functions are
	* provided for use by kernel proper routines only. Macro routines bmajor(),
	* major(), minor(), emajor(), eminor(), and makedev() will be removed or
	* their definitions changed at the next major release following SVR4.
	*/

	#define O_BITSMAJOR 7 /* # of SVR3 major device bits */
	#define O_BITSMINOR 8 /* # of SVR3 minor device bits */
	#define O_MAXMAJ 0x7f /* SVR3 max major value */
	#define O_MAXMIN 0xff /* SVR3 max minor value */


	#define L_BITSMAJOR32 14 /* # of SVR4 major device bits */
	#define L_BITSMINOR32 18 /* # of SVR4 minor device bits */
	#define L_MAXMAJ32 0x3fff /* SVR4 max major value */
	#define L_MAXMIN32 0x3ffff /* MAX minor for 3b2 software drivers. */
	/* For 3b2 hardware devices the minor is */
	/* restricted to 256 (0-255) */

	#ifdef _LP64
	#define L_BITSMAJOR 32 /* # of major device bits in 64-bit Solaris */
	#define L_BITSMINOR 32 /* # of minor device bits in 64-bit Solaris */
	#define L_MAXMAJ 0xfffffffful /* max major value */
	#define L_MAXMIN 0xfffffffful /* max minor value */
	#else
	#define L_BITSMAJOR L_BITSMAJOR32
	#define L_BITSMINOR L_BITSMINOR32
	#define L_MAXMAJ L_MAXMAJ32
	#define L_MAXMIN L_MAXMIN32
	#endif

	/*
	* These are versions of the kernel routines for compressing and
	* expanding long device numbers that don't return errors.
	*/
	#if (L_BITSMAJOR32 == L_BITSMAJOR) && (L_BITSMINOR32 == L_BITSMINOR)

	#define DEVCMPL(x) (x)
	#define DEVEXPL(x) (x)

	#else

	#define DEVCMPL(x) \
	(dev32_t)((((x) >> L_BITSMINOR) > L_MAXMAJ32 \|\| \
	((x) & L_MAXMIN) > L_MAXMIN32) ? NODEV32 : \
	((((x) >> L_BITSMINOR) << L_BITSMINOR32) \| ((x) & L_MAXMIN32)))

	#define DEVEXPL(x) \
	(((x) == NODEV32) ? NODEV : \
	makedevice(((x) >> L_BITSMINOR32) & L_MAXMAJ32, (x) & L_MAXMIN32))

	#endif /* L_BITSMAJOR32 ... */

	/* convert to old (SVR3.2) dev format */

	#define cmpdev(x) \
	(o_dev_t)((((x) >> L_BITSMINOR) > O_MAXMAJ \|\| \
	((x) & L_MAXMIN) > O_MAXMIN) ? NODEV : \
	((((x) >> L_BITSMINOR) << O_BITSMINOR) \| ((x) & O_MAXMIN)))

	/* convert to new (SVR4) dev format */

	#define expdev(x) \
	(dev_t)(((dev_t)(((x) >> O_BITSMINOR) & O_MAXMAJ) << L_BITSMINOR) \| \
	((x) & O_MAXMIN))

	/*
	* Macro for checking power of 2 address alignment.
	*/
	#define IS_P2ALIGNED(v, a) ((((uintptr_t)(v)) & ((uintptr_t)(a) - 1)) == 0)

	/*
	* Macros for counting and rounding.
	*/
	#define howmany(x, y) (((x)+((y)-1))/(y))
	#define roundup(x, y) ((((x)+((y)-1))/(y))*(y))

	/*
	* Macro to determine if value is a power of 2
	*/
	#define ISP2(x) (((x) & ((x) - 1)) == 0)

	/*
	* Macros for various sorts of alignment and rounding. The "align" must
	* be a power of 2. Often times it is a block, sector, or page.
	*/

	/*
	* return x rounded down to an align boundary
	* eg, P2ALIGN(1200, 1024) == 1024 (1*align)
	* eg, P2ALIGN(1024, 1024) == 1024 (1*align)
	* eg, P2ALIGN(0x1234, 0x100) == 0x1200 (0x12*align)
	* eg, P2ALIGN(0x5600, 0x100) == 0x5600 (0x56*align)
	*/
	#define P2ALIGN(x, align) ((x) & -(align))

	/*
	* return x % (mod) align
	* eg, P2PHASE(0x1234, 0x100) == 0x34 (x-0x12*align)
	* eg, P2PHASE(0x5600, 0x100) == 0x00 (x-0x56*align)
	*/
	#define P2PHASE(x, align) ((x) & ((align) - 1))

	/*
	* return how much space is left in this block (but if it's perfectly
	* aligned, return 0).
	* eg, P2NPHASE(0x1234, 0x100) == 0xcc (0x13*align-x)
	* eg, P2NPHASE(0x5600, 0x100) == 0x00 (0x56*align-x)
	*/
	#define P2NPHASE(x, align) (-(x) & ((align) - 1))

	/*
	* return x rounded up to an align boundary
	* eg, P2ROUNDUP(0x1234, 0x100) == 0x1300 (0x13*align)
	* eg, P2ROUNDUP(0x5600, 0x100) == 0x5600 (0x56*align)
	*/
	#define P2ROUNDUP(x, align) (-(-(x) & -(align)))

	/*
	* return the ending address of the block that x is in
	* eg, P2END(0x1234, 0x100) == 0x12ff (0x13*align - 1)
	* eg, P2END(0x5600, 0x100) == 0x56ff (0x57*align - 1)
	*/
	#define P2END(x, align) (-(~(x) & -(align)))

	/*
	* return x rounded up to the next phase (offset) within align.
	* phase should be < align.
	* eg, P2PHASEUP(0x1234, 0x100, 0x10) == 0x1310 (0x13*align + phase)
	* eg, P2PHASEUP(0x5600, 0x100, 0x10) == 0x5610 (0x56*align + phase)
	*/
	#define P2PHASEUP(x, align, phase) ((phase) - (((phase) - (x)) & -(align)))

	/*
	* return TRUE if adding len to off would cause it to cross an align
	* boundary.
	* eg, P2BOUNDARY(0x1234, 0xe0, 0x100) == TRUE (0x1234 + 0xe0 == 0x1314)
	* eg, P2BOUNDARY(0x1234, 0x50, 0x100) == FALSE (0x1234 + 0x50 == 0x1284)
	*/
	#define P2BOUNDARY(off, len, align) \
	(((off) ^ ((off) + (len) - 1)) > (align) - 1)

	/*
	* Return TRUE if they have the same highest bit set.
	* eg, P2SAMEHIGHBIT(0x1234, 0x1001) == TRUE (the high bit is 0x1000)
	* eg, P2SAMEHIGHBIT(0x1234, 0x3010) == FALSE (high bit of 0x3010 is 0x2000)
	*/
	#define P2SAMEHIGHBIT(x, y) (((x) ^ (y)) < ((x) & (y)))

	/*
	* Typed version of the P2* macros. These macros should be used to ensure
	* that the result is correctly calculated based on the data type of (x),
	* which is passed in as the last argument, regardless of the data
	* type of the alignment. For example, if (x) is of type uint64_t,
	* and we want to round it up to a page boundary using "PAGESIZE" as
	* the alignment, we can do either
	* P2ROUNDUP(x, (uint64_t)PAGESIZE)
	* or
	* P2ROUNDUP_TYPED(x, PAGESIZE, uint64_t)
	*/
	#define P2ALIGN_TYPED(x, align, type) \
	((type)(x) & -(type)(align))
	#define P2PHASE_TYPED(x, align, type) \
	((type)(x) & ((type)(align) - 1))
	#define P2NPHASE_TYPED(x, align, type) \
	(-(type)(x) & ((type)(align) - 1))
	#define P2ROUNDUP_TYPED(x, align, type) \
	(-(-(type)(x) & -(type)(align)))
	#define P2END_TYPED(x, align, type) \
	(-(~(type)(x) & -(type)(align)))
	#define P2PHASEUP_TYPED(x, align, phase, type) \
	((type)(phase) - (((type)(phase) - (type)(x)) & -(type)(align)))
	#define P2CROSS_TYPED(x, y, align, type) \
	(((type)(x) ^ (type)(y)) > (type)(align) - 1)
	#define P2SAMEHIGHBIT_TYPED(x, y, type) \
	(((type)(x) ^ (type)(y)) < ((type)(x) & (type)(y)))

	/*
	* Macros to atomically increment/decrement a variable. mutex and var
	* must be pointers.
	*/
	#define INCR_COUNT(var, mutex) mutex_enter(mutex), (*(var))++, mutex_exit(mutex)
	#define DECR_COUNT(var, mutex) mutex_enter(mutex), (*(var))--, mutex_exit(mutex)

	/*
	* Macros to declare bitfields - the order in the parameter list is
	* Low to High - that is, declare bit 0 first. We only support 8-bit bitfields
	* because if a field crosses a byte boundary it's not likely to be meaningful
	* without reassembly in its nonnative endianness.
	*/
	#if defined(_BIT_FIELDS_LTOH)
	#define DECL_BITFIELD2(_a, _b) \
	uint8_t _a, _b
	#define DECL_BITFIELD3(_a, _b, _c) \
	uint8_t _a, _b, _c
	#define DECL_BITFIELD4(_a, _b, _c, _d) \
	uint8_t _a, _b, _c, _d
	#define DECL_BITFIELD5(_a, _b, _c, _d, _e) \
	uint8_t _a, _b, _c, _d, _e
	#define DECL_BITFIELD6(_a, _b, _c, _d, _e, _f) \
	uint8_t _a, _b, _c, _d, _e, _f
	#define DECL_BITFIELD7(_a, _b, _c, _d, _e, _f, _g) \
	uint8_t _a, _b, _c, _d, _e, _f, _g
	#define DECL_BITFIELD8(_a, _b, _c, _d, _e, _f, _g, _h) \
	uint8_t _a, _b, _c, _d, _e, _f, _g, _h
	#elif defined(_BIT_FIELDS_HTOL)
	#define DECL_BITFIELD2(_a, _b) \
	uint8_t _b, _a
	#define DECL_BITFIELD3(_a, _b, _c) \
	uint8_t _c, _b, _a
	#define DECL_BITFIELD4(_a, _b, _c, _d) \
	uint8_t _d, _c, _b, _a
	#define DECL_BITFIELD5(_a, _b, _c, _d, _e) \
	uint8_t _e, _d, _c, _b, _a
	#define DECL_BITFIELD6(_a, _b, _c, _d, _e, _f) \
	uint8_t _f, _e, _d, _c, _b, _a
	#define DECL_BITFIELD7(_a, _b, _c, _d, _e, _f, _g) \
	uint8_t _g, _f, _e, _d, _c, _b, _a
	#define DECL_BITFIELD8(_a, _b, _c, _d, _e, _f, _g, _h) \
	uint8_t _h, _g, _f, _e, _d, _c, _b, _a
	#else
	#error One of _BIT_FIELDS_LTOH or _BIT_FIELDS_HTOL must be defined
	#endif /* _BIT_FIELDS_LTOH */

	#if !defined(_KMEMUSER) && !defined(offsetof)

	/* avoid any possibility of clashing with <stddef.h> version */

	-#define offsetof(s, m) ((size_t)(&(((s *)0)->m)))
	+#define offsetof(type, field) __offsetof(type, field)
	#endif

	/*
	* Find highest one bit set.
	* Returns bit number + 1 of highest bit that is set, otherwise returns 0.
	* High order bit is 31 (or 63 in _LP64 kernel).
	*/
	static __inline int
	highbit(ulong_t i)
	{
	#if defined(HAVE_INLINE_FLSL)
	return (flsl(i));
	#else
	int h = 1;

	if (i == 0)
	return (0);
	#ifdef _LP64
	if (i & 0xffffffff00000000ul) {
	h += 32; i >>= 32;
	}
	#endif
	if (i & 0xffff0000) {
	h += 16; i >>= 16;
	}
	if (i & 0xff00) {
	h += 8; i >>= 8;
	}
	if (i & 0xf0) {
	h += 4; i >>= 4;
	}
	if (i & 0xc) {
	h += 2; i >>= 2;
	}
	if (i & 0x2) {
	h += 1;
	}
	return (h);
	#endif
	}

	/*
	* Find highest one bit set.
	* Returns bit number + 1 of highest bit that is set, otherwise returns 0.
	*/
	static __inline int
	highbit64(uint64_t i)
	{
	#if defined(HAVE_INLINE_FLSLL)
	return (flsll(i));
	#else
	int h = 1;

	if (i == 0)
	return (0);
	if (i & 0xffffffff00000000ULL) {
	h += 32; i >>= 32;
	}
	if (i & 0xffff0000) {
	h += 16; i >>= 16;
	}
	if (i & 0xff00) {
	h += 8; i >>= 8;
	}
	if (i & 0xf0) {
	h += 4; i >>= 4;
	}
	if (i & 0xc) {
	h += 2; i >>= 2;
	}
	if (i & 0x2) {
	h += 1;
	}
	return (h);
	#endif
	}

	#ifdef __cplusplus
	}
	#endif

	#endif /* _SYS_SYSMACROS_H */
	Index: vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/taskq.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/taskq.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/taskq.h (revision 366774)
	@@ -1,115 +1,124 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright 2009 Sun Microsystems, Inc. All rights reserved.
	* Use is subject to license terms.
	*/

	#ifndef _SYS_TASKQ_H
	#define _SYS_TASKQ_H

	+#ifdef _KERNEL
	+
	#include <sys/types.h>
	#include <sys/proc.h>
	#include <sys/taskqueue.h>
	#include <sys/thread.h>
	#include <sys/ck.h>

	#ifdef __cplusplus
	extern "C" {
	#endif

	#define TASKQ_NAMELEN 31

	typedef struct taskq {
	struct taskqueue *tq_queue;
	} taskq_t;

	typedef uintptr_t taskqid_t;
	typedef void (task_func_t)(void *);

	typedef struct taskq_ent {
	struct task tqent_task;
	struct timeout_task tqent_timeout_task;
	task_func_t *tqent_func;
	void *tqent_arg;
	taskqid_t tqent_id;
	CK_LIST_ENTRY(taskq_ent) tqent_hash;
	uint8_t tqent_type;
	uint8_t tqent_registered;
	uint8_t tqent_cancelled;
	volatile uint32_t tqent_rc;
	} taskq_ent_t;

	/*
	* Public flags for taskq_create(): bit range 0-15
	*/
	#define TASKQ_PREPOPULATE 0x0001 /* Prepopulate with threads and data */
	#define TASKQ_CPR_SAFE 0x0002 /* Use CPR safe protocol */
	#define TASKQ_DYNAMIC 0x0004 /* Use dynamic thread scheduling */
	#define TASKQ_THREADS_CPU_PCT 0x0008 /* number of threads as % of ncpu */
	#define TASKQ_DC_BATCH 0x0010 /* Taskq uses SDC in batch mode */

	/*
	* Flags for taskq_dispatch. TQ_SLEEP/TQ_NOSLEEP should be same as
	* KM_SLEEP/KM_NOSLEEP.
	*/
	#define TQ_SLEEP 0x00 /* Can block for memory */
	#define TQ_NOSLEEP 0x01 /* cannot block for memory; may fail */
	#define TQ_NOQUEUE 0x02 /* Do not enqueue if can't dispatch */
	#define TQ_NOALLOC 0x04 /* cannot allocate memory; may fail */
	#define TQ_FRONT 0x08 /* Put task at the front of the queue */

	#define TASKQID_INVALID ((taskqid_t)0)

	#define taskq_init_ent(x)
	extern taskq_t *system_taskq;
	/* Global dynamic task queue for long delay */
	extern taskq_t *system_delay_taskq;

	extern taskqid_t taskq_dispatch(taskq_t , task_func_t, void , uint_t);
	extern taskqid_t taskq_dispatch_delay(taskq_t , task_func_t, void ,
	uint_t, clock_t);
	extern void taskq_dispatch_ent(taskq_t , task_func_t, void , uint_t,
	taskq_ent_t *);
	extern int taskq_empty_ent(taskq_ent_t *);
	taskq_t taskq_create(const char , int, pri_t, int, int, uint_t);
	taskq_t taskq_create_instance(const char , int, int, pri_t, int, int, uint_t);
	taskq_t taskq_create_proc(const char , int, pri_t, int, int,
	struct proc *, uint_t);
	taskq_t taskq_create_sysdc(const char , int, int, int,
	struct proc *, uint_t, uint_t);
	void nulltask(void *);
	extern void taskq_destroy(taskq_t *);
	extern void taskq_wait_id(taskq_t *, taskqid_t);
	extern void taskq_wait_outstanding(taskq_t *, taskqid_t);
	extern void taskq_wait(taskq_t *);
	extern int taskq_cancel_id(taskq_t *, taskqid_t);
	extern int taskq_member(taskq_t , kthread_t );
	extern taskq_t *taskq_of_curthread(void);
	void taskq_suspend(taskq_t *);
	int taskq_suspended(taskq_t *);
	void taskq_resume(taskq_t *);

	#ifdef __cplusplus
	}
	#endif
	+
	+#endif /* _KERNEL */
	+
	+#ifdef _STANDALONE
	+typedef int taskq_ent_t;
	+#define taskq_init_ent(x)
	+#endif /* _STANDALONE */

	#endif /* _SYS_TASKQ_H */
	Index: vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/uio.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/uio.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/spl/sys/uio.h (revision 366774)
	@@ -1,110 +1,114 @@
	/*
	* Copyright (c) 2010 Pawel Jakub Dawidek <pjd@FreeBSD.org>
	* All rights reserved.
	*
	* Redistribution and use in source and binary forms, with or without
	* modification, are permitted provided that the following conditions
	* are met:
	* 1. Redistributions of source code must retain the above copyright
	* notice, this list of conditions and the following disclaimer.
	* 2. Redistributions in binary form must reproduce the above copyright
	* notice, this list of conditions and the following disclaimer in the
	* documentation and/or other materials provided with the distribution.
	*
	* THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
	* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
	* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
	* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
	* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
	* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
	* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
	* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
	* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
	* SUCH DAMAGE.
	*
	* $FreeBSD$
	*/

	#ifndef _OPENSOLARIS_SYS_UIO_H_
	#define _OPENSOLARIS_SYS_UIO_H_

	+#ifndef _STANDALONE
	+
	#include_next <sys/uio.h>
	#include <sys/_uio.h>
	#include <sys/debug.h>



	#define uio_loffset uio_offset

	typedef struct uio uio_t;
	typedef struct iovec iovec_t;
	typedef enum uio_seg uio_seg_t;

	typedef enum xuio_type {
	UIOTYPE_ASYNCIO,
	UIOTYPE_ZEROCOPY
	} xuio_type_t;

	typedef struct xuio {
	uio_t xu_uio;

	/* Extended uio fields */
	enum xuio_type xu_type; /* What kind of uio structure? */
	union {
	struct {
	int xu_zc_rw;
	void *xu_zc_priv;
	} xu_zc;
	} xu_ext;
	} xuio_t;

	#define XUIO_XUZC_PRIV(xuio) xuio->xu_ext.xu_zc.xu_zc_priv
	#define XUIO_XUZC_RW(xuio) xuio->xu_ext.xu_zc.xu_zc_rw

	static __inline int
	zfs_uiomove(void cp, size_t n, enum uio_rw dir, uio_t uio)
	{

	ASSERT(uio->uio_rw == dir);
	return (uiomove(cp, (int)n, uio));
	}
	#define uiomove(cp, n, dir, uio) zfs_uiomove((cp), (n), (dir), (uio))

	int uiocopy(void p, size_t n, enum uio_rw rw, struct uio uio, size_t *cbytes);
	void uioskip(uio_t *uiop, size_t n);

	#define uio_segflg(uio) (uio)->uio_segflg
	#define uio_offset(uio) (uio)->uio_loffset
	#define uio_resid(uio) (uio)->uio_resid
	#define uio_iovcnt(uio) (uio)->uio_iovcnt
	#define uio_iovlen(uio, idx) (uio)->uio_iov[(idx)].iov_len
	#define uio_iovbase(uio, idx) (uio)->uio_iov[(idx)].iov_base

	static inline void
	uio_iov_at_index(uio_t uio, uint_t idx, void base, uint64_t len)
	{
	*base = uio_iovbase(uio, idx);
	*len = uio_iovlen(uio, idx);
	}

	static inline void
	uio_advance(uio_t *uio, size_t size)
	{
	uio->uio_resid -= size;
	uio->uio_loffset += size;
	}

	static inline offset_t
	uio_index_at_offset(uio_t uio, offset_t off, uint_t vec_idx)
	{
	*vec_idx = 0;
	while (vec_idx < uio_iovcnt(uio) && off >= uio_iovlen(uio, vec_idx)) {
	off -= uio_iovlen(uio, *vec_idx);
	(*vec_idx)++;
	}

	return (off);
	}
	+
	+#endif /* !_STANDALONE */

	#endif /* !_OPENSOLARIS_SYS_UIO_H_ */
	Index: vendor-sys/openzfs/dist/include/os/freebsd/zfs/sys/zfs_ctldir.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/zfs/sys/zfs_ctldir.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/zfs/sys/zfs_ctldir.h (revision 366774)
	@@ -1,65 +1,65 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	*/

	#ifndef _ZFS_CTLDIR_H
	#define _ZFS_CTLDIR_H

	#include <sys/vnode.h>
	#include <sys/zfs_vfsops.h>
	#include <sys/zfs_znode.h>

	#ifdef __cplusplus
	extern "C" {
	#endif

	#define ZFS_CTLDIR_NAME ".zfs"

	#define zfs_has_ctldir(zdp) \
	((zdp)->z_id == (zdp)->z_zfsvfs->z_root && \
	((zdp)->z_zfsvfs->z_ctldir != NULL))
	#define zfs_show_ctldir(zdp) \
	(zfs_has_ctldir(zdp) && \
	((zdp)->z_zfsvfs->z_show_ctldir))

	void zfsctl_create(zfsvfs_t *);
	void zfsctl_destroy(zfsvfs_t *);
	int zfsctl_root(zfsvfs_t , int, vnode_t *);
	void zfsctl_init(void);
	void zfsctl_fini(void);
	boolean_t zfsctl_is_node(vnode_t *);
	-int zfsctl_snapshot_unmount(char *snapname, int flags);
	+int zfsctl_snapshot_unmount(const char *snapname, int flags);
	int zfsctl_rename_snapshot(const char from, const char to);
	int zfsctl_destroy_snapshot(const char *snapname, int force);
	int zfsctl_umount_snapshots(vfs_t , int, cred_t );

	int zfsctl_lookup_objset(vfs_t vfsp, uint64_t objsetid, zfsvfs_t *zfsvfsp);

	#define ZFSCTL_INO_ROOT 0x1
	#define ZFSCTL_INO_SNAPDIR 0x2

	#ifdef __cplusplus
	}
	#endif

	#endif /* _ZFS_CTLDIR_H */
	Index: vendor-sys/openzfs/dist/include/os/freebsd/zfs/sys/zfs_dir.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/zfs/sys/zfs_dir.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/zfs/sys/zfs_dir.h (revision 366774)
	@@ -1,74 +1,70 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright 2010 Sun Microsystems, Inc. All rights reserved.
	* Use is subject to license terms.
	*/

	#ifndef _SYS_FS_ZFS_DIR_H
	#define _SYS_FS_ZFS_DIR_H

	#include <sys/pathname.h>
	#include <sys/dmu.h>
	#include <sys/zfs_znode.h>

	#ifdef __cplusplus
	extern "C" {
	#endif

	/* zfs_dirent_lock() flags */
	#define ZNEW 0x0001 /* entry should not exist */
	#define ZEXISTS 0x0002 /* entry should exist */
	#define ZSHARED 0x0004 /* shared access (zfs_dirlook()) */
	#define ZXATTR 0x0008 /* we want the xattr dir */
	#define ZRENAMING 0x0010 /* znode is being renamed */
	#define ZCILOOK 0x0020 /* case-insensitive lookup requested */
	#define ZCIEXACT 0x0040 /* c-i requires c-s match (rename) */
	#define ZHAVELOCK 0x0080 /* z_name_lock is already held */

	/* mknode flags */
	#define IS_ROOT_NODE 0x01 /* create a root node */
	#define IS_XATTR 0x02 /* create an extended attribute node */

	extern int zfs_dirent_lookup(znode_t , const char , znode_t **, int);
	extern int zfs_link_create(znode_t , const char , znode_t , dmu_tx_t , int);
	extern int zfs_link_destroy(znode_t , const char , znode_t , dmu_tx_t , int,
	boolean_t *);
	-#if 0
	-extern int zfs_dirlook(vnode_t , const char , vnode_t **, int);
	-#else
	extern int zfs_dirlook(znode_t , const char name, znode_t **);
	-#endif
	extern void zfs_mknode(znode_t , vattr_t , dmu_tx_t , cred_t ,
	uint_t, znode_t *, zfs_acl_ids_t );
	extern void zfs_rmnode(znode_t *);
	extern boolean_t zfs_dirempty(znode_t *);
	extern void zfs_unlinked_add(znode_t , dmu_tx_t );
	extern void zfs_unlinked_drain(zfsvfs_t *zfsvfs);
	extern int zfs_sticky_remove_access(znode_t , znode_t , cred_t *cr);
	extern int zfs_get_xattrdir(znode_t , znode_t , cred_t , int);
	extern int zfs_make_xattrdir(znode_t , vattr_t , znode_t *, cred_t );

	#ifdef __cplusplus
	}
	#endif

	#endif /* _SYS_FS_ZFS_DIR_H */
	Index: vendor-sys/openzfs/dist/include/os/freebsd/zfs/sys/zfs_vfsops_os.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/zfs/sys/zfs_vfsops_os.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/zfs/sys/zfs_vfsops_os.h (revision 366774)
	@@ -1,239 +1,240 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2011 Pawel Jakub Dawidek <pawel@dawidek.net>.
	* All rights reserved.
	*/

	#ifndef _SYS_FS_ZFS_VFSOPS_H
	#define _SYS_FS_ZFS_VFSOPS_H

	#if __FreeBSD_version >= 1300109
	#define TEARDOWN_INACTIVE_RMS
	#endif

	#include <sys/dataset_kstats.h>
	#include <sys/list.h>
	#include <sys/vfs.h>
	#include <sys/zil.h>
	#include <sys/sa.h>
	#include <sys/rrwlock.h>
	#ifdef TEARDOWN_INACTIVE_RMS
	#include <sys/rmlock.h>
	#endif
	#include <sys/zfs_ioctl.h>

	#ifdef __cplusplus
	extern "C" {
	#endif

	#ifdef TEARDOWN_INACTIVE_RMS
	typedef struct rmslock zfs_teardown_lock_t;
	#else
	#define zfs_teardown_lock_t krwlock_t
	#endif

	typedef struct zfsvfs zfsvfs_t;
	struct znode;

	struct zfsvfs {
	vfs_t z_vfs; / generic fs struct */
	zfsvfs_t z_parent; / parent fs */
	objset_t z_os; / objset reference */
	uint64_t z_flags; /* super_block flags */
	uint64_t z_root; /* id of root znode */
	uint64_t z_unlinkedobj; /* id of unlinked zapobj */
	uint64_t z_max_blksz; /* maximum block size for files */
	uint64_t z_fuid_obj; /* fuid table object number */
	uint64_t z_fuid_size; /* fuid table size */
	avl_tree_t z_fuid_idx; /* fuid tree keyed by index */
	avl_tree_t z_fuid_domain; /* fuid tree keyed by domain */
	krwlock_t z_fuid_lock; /* fuid lock */
	boolean_t z_fuid_loaded; /* fuid tables are loaded */
	boolean_t z_fuid_dirty; /* need to sync fuid table ? */
	struct zfs_fuid_info z_fuid_replay; / fuid info for replay */
	zilog_t z_log; / intent log pointer */
	+ uint_t z_acl_type; /* type of acl usable on this fs */
	uint_t z_acl_mode; /* acl chmod/mode behavior */
	uint_t z_acl_inherit; /* acl inheritance behavior */
	zfs_case_t z_case; /* case-sense */
	boolean_t z_utf8; /* utf8-only */
	int z_norm; /* normalization flags */
	boolean_t z_atime; /* enable atimes mount option */
	boolean_t z_unmounted; /* unmounted */
	rrmlock_t z_teardown_lock;
	zfs_teardown_lock_t z_teardown_inactive_lock;
	list_t z_all_znodes; /* all vnodes in the fs */
	uint64_t z_nr_znodes; /* number of znodes in the fs */
	kmutex_t z_znodes_lock; /* lock for z_all_znodes */
	struct zfsctl_root z_ctldir; / .zfs directory pointer */
	boolean_t z_show_ctldir; /* expose .zfs in the root dir */
	boolean_t z_issnap; /* true if this is a snapshot */
	boolean_t z_vscan; /* virus scan on/off */
	boolean_t z_use_fuids; /* version allows fuids */
	boolean_t z_replay; /* set during ZIL replay */
	boolean_t z_use_sa; /* version allow system attributes */
	boolean_t z_xattr_sa; /* allow xattrs to be stores as SA */
	boolean_t z_use_namecache; /* make use of FreeBSD name cache */
	uint8_t z_xattr; /* xattr type in use */
	uint64_t z_version; /* ZPL version */
	uint64_t z_shares_dir; /* hidden shares dir */
	dataset_kstats_t z_kstat; /* fs kstats */
	kmutex_t z_lock;
	uint64_t z_userquota_obj;
	uint64_t z_groupquota_obj;
	uint64_t z_userobjquota_obj;
	uint64_t z_groupobjquota_obj;
	uint64_t z_projectquota_obj;
	uint64_t z_projectobjquota_obj;
	uint64_t z_replay_eof; /* New end of file - replay only */
	sa_attr_type_t z_attr_table; / SA attr mapping->id */
	#define ZFS_OBJ_MTX_SZ 64
	kmutex_t z_hold_mtx[ZFS_OBJ_MTX_SZ]; /* znode hold locks */
	struct task z_unlinked_drain_task;
	};

	#ifdef TEARDOWN_INACTIVE_RMS
	#define ZFS_INIT_TEARDOWN_INACTIVE(zfsvfs) \
	rms_init(&(zfsvfs)->z_teardown_inactive_lock, "zfs teardown inactive")

	#define ZFS_DESTROY_TEARDOWN_INACTIVE(zfsvfs) \
	rms_destroy(&(zfsvfs)->z_teardown_inactive_lock)

	#define ZFS_TRYRLOCK_TEARDOWN_INACTIVE(zfsvfs) \
	rms_try_rlock(&(zfsvfs)->z_teardown_inactive_lock)

	#define ZFS_RLOCK_TEARDOWN_INACTIVE(zfsvfs) \
	rms_rlock(&(zfsvfs)->z_teardown_inactive_lock)

	#define ZFS_RUNLOCK_TEARDOWN_INACTIVE(zfsvfs) \
	rms_runlock(&(zfsvfs)->z_teardown_inactive_lock)

	#define ZFS_WLOCK_TEARDOWN_INACTIVE(zfsvfs) \
	rms_wlock(&(zfsvfs)->z_teardown_inactive_lock)

	#define ZFS_WUNLOCK_TEARDOWN_INACTIVE(zfsvfs) \
	rms_wunlock(&(zfsvfs)->z_teardown_inactive_lock)

	#define ZFS_TEARDOWN_INACTIVE_WLOCKED(zfsvfs) \
	rms_wowned(&(zfsvfs)->z_teardown_inactive_lock)
	#else
	#define ZFS_INIT_TEARDOWN_INACTIVE(zfsvfs) \
	rw_init(&(zfsvfs)->z_teardown_inactive_lock, NULL, RW_DEFAULT, NULL)

	#define ZFS_DESTROY_TEARDOWN_INACTIVE(zfsvfs) \
	rw_destroy(&(zfsvfs)->z_teardown_inactive_lock)

	#define ZFS_TRYRLOCK_TEARDOWN_INACTIVE(zfsvfs) \
	rw_tryenter(&(zfsvfs)->z_teardown_inactive_lock, RW_READER)

	#define ZFS_RLOCK_TEARDOWN_INACTIVE(zfsvfs) \
	rw_enter(&(zfsvfs)->z_teardown_inactive_lock, RW_READER)

	#define ZFS_RUNLOCK_TEARDOWN_INACTIVE(zfsvfs) \
	rw_exit(&(zfsvfs)->z_teardown_inactive_lock)

	#define ZFS_WLOCK_TEARDOWN_INACTIVE(zfsvfs) \
	rw_enter(&(zfsvfs)->z_teardown_inactive_lock, RW_WRITER)

	#define ZFS_WUNLOCK_TEARDOWN_INACTIVE(zfsvfs) \
	rw_exit(&(zfsvfs)->z_teardown_inactive_lock)

	#define ZFS_TEARDOWN_INACTIVE_WLOCKED(zfsvfs) \
	RW_WRITE_HELD(&(zfsvfs)->z_teardown_inactive_lock)
	#endif

	#define ZSB_XATTR 0x0001 /* Enable user xattrs */
	/*
	* Normal filesystems (those not under .zfs/snapshot) have a total
	* file ID size limited to 12 bytes (including the length field) due to
	* NFSv2 protocol's limitation of 32 bytes for a filehandle. For historical
	* reasons, this same limit is being imposed by the Solaris NFSv3 implementation
	* (although the NFSv3 protocol actually permits a maximum of 64 bytes). It
	* is not possible to expand beyond 12 bytes without abandoning support
	* of NFSv2.
	*
	* For normal filesystems, we partition up the available space as follows:
	* 2 bytes fid length (required)
	* 6 bytes object number (48 bits)
	* 4 bytes generation number (32 bits)
	*
	* We reserve only 48 bits for the object number, as this is the limit
	* currently defined and imposed by the DMU.
	*/
	typedef struct zfid_short {
	uint16_t zf_len;
	uint8_t zf_object[6]; /* obj[i] = obj >> (8 * i) */
	uint8_t zf_gen[4]; /* gen[i] = gen >> (8 * i) */
	} zfid_short_t;

	/*
	* Filesystems under .zfs/snapshot have a total file ID size of 22[*] bytes
	* (including the length field). This makes files under .zfs/snapshot
	* accessible by NFSv3 and NFSv4, but not NFSv2.
	*
	* For files under .zfs/snapshot, we partition up the available space
	* as follows:
	* 2 bytes fid length (required)
	* 6 bytes object number (48 bits)
	* 4 bytes generation number (32 bits)
	* 6 bytes objset id (48 bits)
	* 4 bytes[**] currently just zero (32 bits)
	*
	* We reserve only 48 bits for the object number and objset id, as these are
	* the limits currently defined and imposed by the DMU.
	*
	* [*] 20 bytes on FreeBSD to fit into the size of struct fid.
	* [**] 2 bytes on FreeBSD for the above reason.
	*/
	typedef struct zfid_long {
	zfid_short_t z_fid;
	uint8_t zf_setid[6]; /* obj[i] = obj >> (8 * i) */
	uint8_t zf_setgen[2]; /* gen[i] = gen >> (8 * i) */
	} zfid_long_t;

	#define SHORT_FID_LEN (sizeof (zfid_short_t) - sizeof (uint16_t))
	#define LONG_FID_LEN (sizeof (zfid_long_t) - sizeof (uint16_t))

	extern uint_t zfs_fsyncer_key;
	extern int zfs_super_owner;

	extern void zfs_init(void);
	extern void zfs_fini(void);

	extern int zfs_suspend_fs(zfsvfs_t *zfsvfs);
	extern int zfs_resume_fs(zfsvfs_t zfsvfs, struct dsl_dataset ds);
	extern int zfs_end_fs(zfsvfs_t zfsvfs, struct dsl_dataset ds);
	extern int zfs_set_version(zfsvfs_t *zfsvfs, uint64_t newvers);
	extern int zfsvfs_create(const char name, boolean_t readonly, zfsvfs_t *zfvp);
	extern int zfsvfs_create_impl(zfsvfs_t *zfvp, zfsvfs_t zfsvfs, objset_t *os);
	extern void zfsvfs_free(zfsvfs_t *zfsvfs);
	extern int zfs_check_global_label(const char dsname, const char hexsl);
	extern boolean_t zfs_is_readonly(zfsvfs_t *zfsvfs);
	extern int zfs_get_temporary_prop(struct dsl_dataset *ds, zfs_prop_t zfs_prop,
	uint64_t val, char setpoint);
	extern int zfs_busy(void);

	#ifdef __cplusplus
	}
	#endif

	#endif /* _SYS_FS_ZFS_VFSOPS_H */
	Index: vendor-sys/openzfs/dist/include/os/freebsd/zfs/sys/zfs_vnops.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/freebsd/zfs/sys/zfs_vnops.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/freebsd/zfs/sys/zfs_vnops.h (revision 366774)
	@@ -1,56 +1,56 @@
	/*
	* Copyright (c) 2020 iXsystems, Inc.
	* All rights reserved.
	*
	* Redistribution and use in source and binary forms, with or without
	* modification, are permitted provided that the following conditions
	* are met:
	* 1. Redistributions of source code must retain the above copyright
	* notice, this list of conditions and the following disclaimer.
	* 2. Redistributions in binary form must reproduce the above copyright
	* notice, this list of conditions and the following disclaimer in the
	* documentation and/or other materials provided with the distribution.
	*
	* THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
	* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
	* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
	* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
	* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
	* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
	* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
	* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
	* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
	* SUCH DAMAGE.
	*
	* $FreeBSD$
	*/

	#ifndef _SYS_ZFS_VNOPS_H_
	#define _SYS_ZFS_VNOPS_H_
	int dmu_write_pages(objset_t *os, uint64_t object, uint64_t offset,
	uint64_t size, struct vm_page *ppa, dmu_tx_t tx);
	int dmu_read_pages(objset_t os, uint64_t object, vm_page_t ma, int count,
	int rbehind, int rahead, int last_size);
	-extern int zfs_remove(znode_t dzp, char name, cred_t *cr, int flags);
	-extern int zfs_mkdir(znode_t dzp, char dirname, vattr_t *vap,
	+extern int zfs_remove(znode_t dzp, const char name, cred_t *cr, int flags);
	+extern int zfs_mkdir(znode_t dzp, const char dirname, vattr_t *vap,
	znode_t *zpp, cred_t cr, int flags, vsecattr_t *vsecp);
	-extern int zfs_rmdir(znode_t dzp, char name, znode_t *cwd,
	+extern int zfs_rmdir(znode_t dzp, const char name, znode_t *cwd,
	cred_t *cr, int flags);
	extern int zfs_setattr(znode_t zp, vattr_t vap, int flag, cred_t *cr);
	-extern int zfs_rename(znode_t sdzp, char snm, znode_t *tdzp,
	- char tnm, cred_t cr, int flags);
	+extern int zfs_rename(znode_t sdzp, const char snm, znode_t *tdzp,
	+ const char tnm, cred_t cr, int flags);
	extern int zfs_symlink(znode_t dzp, const char name, vattr_t *vap,
	const char link, znode_t zpp, cred_t cr, int flags);
	extern int zfs_link(znode_t tdzp, znode_t sp,
	- char name, cred_t cr, int flags);
	+ const char name, cred_t cr, int flags);
	extern int zfs_space(znode_t zp, int cmd, struct flock bfp, int flag,
	offset_t offset, cred_t *cr);
	-extern int zfs_create(znode_t dzp, char name, vattr_t *vap, int excl,
	+extern int zfs_create(znode_t dzp, const char name, vattr_t *vap, int excl,
	int mode, znode_t *zpp, cred_t cr, int flag, vsecattr_t *vsecp);
	extern int zfs_setsecattr(znode_t zp, vsecattr_t vsecp, int flag,
	cred_t *cr);
	extern int zfs_write_simple(znode_t zp, const void data, size_t len,
	loff_t pos, size_t *resid);

	#endif
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/rpc/xdr.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/rpc/xdr.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/rpc/xdr.h (revision 366774)
	@@ -1,156 +1,155 @@
	/*
	* Copyright (c) 2008 Sun Microsystems, Inc.
	* Written by Ricardo Correia <Ricardo.M.Correia@Sun.COM>
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_RPC_XDR_H
	#define _SPL_RPC_XDR_H

	#include <sys/types.h>

	typedef int bool_t;

	/*
	* XDR enums and types.
	*/
	enum xdr_op {
	XDR_ENCODE,
	XDR_DECODE
	};

	struct xdr_ops;

	typedef struct {
	struct xdr_ops x_ops; / Let caller know xdrmem_create() succeeds */
	caddr_t x_addr; /* Current buffer addr */
	caddr_t x_addr_end; /* End of the buffer */
	enum xdr_op x_op; /* Stream direction */
	} XDR;

	typedef bool_t (xdrproc_t)(XDR xdrs, void *ptr);

	struct xdr_ops {
	bool_t (xdr_control)(XDR , int, void *);

	bool_t (xdr_char)(XDR , char *);
	bool_t (xdr_u_short)(XDR , unsigned short *);
	bool_t (xdr_u_int)(XDR , unsigned *);
	bool_t (xdr_u_longlong_t)(XDR , u_longlong_t *);

	bool_t (xdr_opaque)(XDR , caddr_t, const uint_t);
	bool_t (xdr_string)(XDR , char **, const uint_t);
	bool_t (xdr_array)(XDR , caddr_t , uint_t , const uint_t,
	const uint_t, const xdrproc_t);
	};

	/*
	* XDR control operator.
	*/
	#define XDR_GET_BYTES_AVAIL 1

	struct xdr_bytesrec {
	bool_t xc_is_last_record;
	size_t xc_num_avail;
	};

	/*
	* XDR functions.
	*/
	void xdrmem_create(XDR *xdrs, const caddr_t addr, const uint_t size,
	const enum xdr_op op);

	/* Currently not needed. If needed later, we'll add it to struct xdr_ops */
	#define xdr_destroy(xdrs) ((void) 0)

	#define xdr_control(xdrs, req, info) \
	(xdrs)->x_ops->xdr_control((xdrs), (req), (info))

	/*
	* For precaution, the following are defined as static inlines instead of macros
	* to get some amount of type safety.
	*
	* Also, macros wouldn't work in the case where typecasting is done, because it
	* must be possible to reference the functions' addresses by these names.
	*/
	static inline bool_t xdr_char(XDR xdrs, char cp)
	{
	return (xdrs->x_ops->xdr_char(xdrs, cp));
	}

	static inline bool_t xdr_u_short(XDR xdrs, unsigned short usp)
	{
	return (xdrs->x_ops->xdr_u_short(xdrs, usp));
	}

	static inline bool_t xdr_short(XDR xdrs, short sp)
	{
	BUILD_BUG_ON(sizeof (short) != 2);
	return (xdrs->x_ops->xdr_u_short(xdrs, (unsigned short *) sp));
	}

	static inline bool_t xdr_u_int(XDR xdrs, unsigned up)
	{
	return (xdrs->x_ops->xdr_u_int(xdrs, up));
	}

	static inline bool_t xdr_int(XDR xdrs, int ip)
	{
	BUILD_BUG_ON(sizeof (int) != 4);
	return (xdrs->x_ops->xdr_u_int(xdrs, (unsigned *)ip));
	}

	static inline bool_t xdr_u_longlong_t(XDR xdrs, u_longlong_t ullp)
	{
	return (xdrs->x_ops->xdr_u_longlong_t(xdrs, ullp));
	}

	static inline bool_t xdr_longlong_t(XDR xdrs, longlong_t llp)
	{
	BUILD_BUG_ON(sizeof (longlong_t) != 8);
	return (xdrs->x_ops->xdr_u_longlong_t(xdrs, (u_longlong_t *)llp));
	}

	/*
	* Fixed-length opaque data.
	*/
	static inline bool_t xdr_opaque(XDR *xdrs, caddr_t cp, const uint_t cnt)
	{
	return (xdrs->x_ops->xdr_opaque(xdrs, cp, cnt));
	}

	/*
	* Variable-length string.
	* The *sp buffer must have (maxsize + 1) bytes.
	*/
	static inline bool_t xdr_string(XDR xdrs, char *sp, const uint_t maxsize)
	{
	return (xdrs->x_ops->xdr_string(xdrs, sp, maxsize));
	}

	/*
	* Variable-length arrays.
	*/
	static inline bool_t xdr_array(XDR xdrs, caddr_t arrp, uint_t *sizep,
	const uint_t maxsize, const uint_t elsize, const xdrproc_t elproc)
	{
	return xdrs->x_ops->xdr_array(xdrs, arrp, sizep, maxsize, elsize,
	elproc);
	}

	#endif /* SPL_RPC_XDR_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/acl.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/acl.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/acl.h (revision 366774)
	@@ -1,119 +1,118 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_ACL_H
	#define _SPL_ACL_H

	#include <sys/types.h>

	typedef struct ace {
	uid_t a_who;
	uint32_t a_access_mask;
	uint16_t a_flags;
	uint16_t a_type;
	} ace_t;

	typedef struct ace_object {
	uid_t a_who; /* uid or gid */
	uint32_t a_access_mask; /* read,write,... */
	uint16_t a_flags; /* see below */
	uint16_t a_type; /* allow or deny */
	uint8_t a_obj_type[16]; /* obj type */
	uint8_t a_inherit_obj_type[16]; /* inherit obj */
	} ace_object_t;

	#define MAX_ACL_ENTRIES 1024

	#define ACE_READ_DATA 0x00000001
	#define ACE_LIST_DIRECTORY 0x00000001
	#define ACE_WRITE_DATA 0x00000002
	#define ACE_ADD_FILE 0x00000002
	#define ACE_APPEND_DATA 0x00000004
	#define ACE_ADD_SUBDIRECTORY 0x00000004
	#define ACE_READ_NAMED_ATTRS 0x00000008
	#define ACE_WRITE_NAMED_ATTRS 0x00000010
	#define ACE_EXECUTE 0x00000020
	#define ACE_DELETE_CHILD 0x00000040
	#define ACE_READ_ATTRIBUTES 0x00000080
	#define ACE_WRITE_ATTRIBUTES 0x00000100
	#define ACE_DELETE 0x00010000
	#define ACE_READ_ACL 0x00020000
	#define ACE_WRITE_ACL 0x00040000
	#define ACE_WRITE_OWNER 0x00080000
	#define ACE_SYNCHRONIZE 0x00100000

	#define ACE_FILE_INHERIT_ACE 0x0001
	#define ACE_DIRECTORY_INHERIT_ACE 0x0002
	#define ACE_NO_PROPAGATE_INHERIT_ACE 0x0004
	#define ACE_INHERIT_ONLY_ACE 0x0008
	#define ACE_SUCCESSFUL_ACCESS_ACE_FLAG 0x0010
	#define ACE_FAILED_ACCESS_ACE_FLAG 0x0020
	#define ACE_IDENTIFIER_GROUP 0x0040
	#define ACE_INHERITED_ACE 0x0080
	#define ACE_OWNER 0x1000
	#define ACE_GROUP 0x2000
	#define ACE_EVERYONE 0x4000

	#define ACE_ACCESS_ALLOWED_ACE_TYPE 0x0000
	#define ACE_ACCESS_DENIED_ACE_TYPE 0x0001
	#define ACE_SYSTEM_AUDIT_ACE_TYPE 0x0002
	#define ACE_SYSTEM_ALARM_ACE_TYPE 0x0003

	#define ACL_AUTO_INHERIT 0x0001
	#define ACL_PROTECTED 0x0002
	#define ACL_DEFAULTED 0x0004
	#define ACL_FLAGS_ALL (ACL_AUTO_INHERIT\|ACL_PROTECTED\|ACL_DEFAULTED)

	#define ACE_ACCESS_ALLOWED_COMPOUND_ACE_TYPE 0x04
	#define ACE_ACCESS_ALLOWED_OBJECT_ACE_TYPE 0x05
	#define ACE_ACCESS_DENIED_OBJECT_ACE_TYPE 0x06
	#define ACE_SYSTEM_AUDIT_OBJECT_ACE_TYPE 0x07
	#define ACE_SYSTEM_ALARM_OBJECT_ACE_TYPE 0x08
	#define ACE_ACCESS_ALLOWED_CALLBACK_ACE_TYPE 0x09
	#define ACE_ACCESS_DENIED_CALLBACK_ACE_TYPE 0x0A
	#define ACE_ACCESS_ALLOWED_CALLBACK_OBJECT_ACE_TYPE 0x0B
	#define ACE_ACCESS_DENIED_CALLBACK_OBJECT_ACE_TYPE 0x0C
	#define ACE_SYSTEM_AUDIT_CALLBACK_ACE_TYPE 0x0D
	#define ACE_SYSTEM_ALARM_CALLBACK_ACE_TYPE 0x0E
	#define ACE_SYSTEM_AUDIT_CALLBACK_OBJECT_ACE_TYPE 0x0F
	#define ACE_SYSTEM_ALARM_CALLBACK_OBJECT_ACE_TYPE 0x10

	#define ACE_ALL_TYPES 0x001F

	#define ACE_TYPE_FLAGS (ACE_OWNER\|ACE_GROUP\|ACE_EVERYONE\|ACE_IDENTIFIER_GROUP)

	/* BEGIN CSTYLED */
	#define ACE_ALL_PERMS (ACE_READ_DATA\|ACE_LIST_DIRECTORY\|ACE_WRITE_DATA\| \
	ACE_ADD_FILE\|ACE_APPEND_DATA\|ACE_ADD_SUBDIRECTORY\|ACE_READ_NAMED_ATTRS\| \
	ACE_WRITE_NAMED_ATTRS\|ACE_EXECUTE\|ACE_DELETE_CHILD\|ACE_READ_ATTRIBUTES\| \
	ACE_WRITE_ATTRIBUTES\|ACE_DELETE\|ACE_READ_ACL\|ACE_WRITE_ACL\| \
	ACE_WRITE_OWNER\|ACE_SYNCHRONIZE)
	/* END CSTYLED */

	#define VSA_ACE 0x0010
	#define VSA_ACECNT 0x0020
	#define VSA_ACE_ALLTYPES 0x0040
	#define VSA_ACE_ACLFLAGS 0x0080

	#endif /* _SPL_ACL_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/atomic.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/atomic.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/atomic.h (revision 366774)
	@@ -1,79 +1,78 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_ATOMIC_H
	#define _SPL_ATOMIC_H

	#include <linux/module.h>
	#include <linux/spinlock.h>
	#include <sys/types.h>

	/*
	* Map the atomic_* functions to the Linux counterparts. This relies on the
	* fact that the atomic types are internally really a uint32 or uint64. If
	* this were to change an alternate approach would be needed.
	*
	* N.B. Due to the limitations of the original API atomicity is not strictly
	* preserved when using the 64-bit functions on a 32-bit system. In order
	* to support this all consumers would need to be updated to use the Linux
	* provided atomic_t and atomic64_t types.
	*/
	#define atomic_inc_32(v) atomic_inc((atomic_t *)(v))
	#define atomic_dec_32(v) atomic_dec((atomic_t *)(v))
	#define atomic_add_32(v, i) atomic_add((i), (atomic_t *)(v))
	#define atomic_sub_32(v, i) atomic_sub((i), (atomic_t *)(v))
	#define atomic_inc_32_nv(v) atomic_inc_return((atomic_t *)(v))
	#define atomic_dec_32_nv(v) atomic_dec_return((atomic_t *)(v))
	#define atomic_add_32_nv(v, i) atomic_add_return((i), (atomic_t *)(v))
	#define atomic_sub_32_nv(v, i) atomic_sub_return((i), (atomic_t *)(v))
	#define atomic_cas_32(v, x, y) atomic_cmpxchg((atomic_t *)(v), x, y)
	#define atomic_swap_32(v, x) atomic_xchg((atomic_t *)(v), x)
	#define atomic_inc_64(v) atomic64_inc((atomic64_t *)(v))
	#define atomic_dec_64(v) atomic64_dec((atomic64_t *)(v))
	#define atomic_add_64(v, i) atomic64_add((i), (atomic64_t *)(v))
	#define atomic_sub_64(v, i) atomic64_sub((i), (atomic64_t *)(v))
	#define atomic_inc_64_nv(v) atomic64_inc_return((atomic64_t *)(v))
	#define atomic_dec_64_nv(v) atomic64_dec_return((atomic64_t *)(v))
	#define atomic_add_64_nv(v, i) atomic64_add_return((i), (atomic64_t *)(v))
	#define atomic_sub_64_nv(v, i) atomic64_sub_return((i), (atomic64_t *)(v))
	#define atomic_cas_64(v, x, y) atomic64_cmpxchg((atomic64_t *)(v), x, y)
	#define atomic_swap_64(v, x) atomic64_xchg((atomic64_t *)(v), x)

	#ifdef _LP64
	static __inline__ void *
	atomic_cas_ptr(volatile void target, void cmp, void *newval)
	{
	return ((void )atomic_cas_64((volatile uint64_t )target,
	(uint64_t)cmp, (uint64_t)newval));
	}
	#else /* _LP64 */
	static __inline__ void *
	atomic_cas_ptr(volatile void target, void cmp, void *newval)
	{
	return ((void )atomic_cas_32((volatile uint32_t )target,
	(uint32_t)cmp, (uint32_t)newval));
	}
	#endif /* _LP64 */

	#endif /* _SPL_ATOMIC_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/byteorder.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/byteorder.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/byteorder.h (revision 366774)
	@@ -1,87 +1,86 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_BYTEORDER_H
	#define _SPL_BYTEORDER_H

	#include <asm/byteorder.h>

	#if defined(__BIG_ENDIAN) && !defined(_ZFS_BIG_ENDIAN)
	#define _ZFS_BIG_ENDIAN
	#endif

	#if defined(__LITTLE_ENDIAN) && !defined(_ZFS_LITTLE_ENDIAN)
	#define _ZFS_LITTLE_ENDIAN
	#endif

	#include <sys/isa_defs.h>

	#define BSWAP_8(x) ((x) & 0xff)
	#define BSWAP_16(x) ((BSWAP_8(x) << 8) \| BSWAP_8((x) >> 8))
	#define BSWAP_32(x) ((BSWAP_16(x) << 16) \| BSWAP_16((x) >> 16))
	#define BSWAP_64(x) ((BSWAP_32(x) << 32) \| BSWAP_32((x) >> 32))

	#define LE_16(x) cpu_to_le16(x)
	#define LE_32(x) cpu_to_le32(x)
	#define LE_64(x) cpu_to_le64(x)
	#define BE_16(x) cpu_to_be16(x)
	#define BE_32(x) cpu_to_be32(x)
	#define BE_64(x) cpu_to_be64(x)

	#define BE_IN8(xa) \
	((uint8_t )(xa))

	#define BE_IN16(xa) \
	(((uint16_t)BE_IN8(xa) << 8) \| BE_IN8((uint8_t *)(xa)+1))

	#define BE_IN32(xa) \
	(((uint32_t)BE_IN16(xa) << 16) \| BE_IN16((uint8_t *)(xa)+2))

	#ifdef _ZFS_BIG_ENDIAN
	static __inline__ uint64_t
	htonll(uint64_t n)
	{
	return (n);
	}

	static __inline__ uint64_t
	ntohll(uint64_t n)
	{
	return (n);
	}
	#else
	static __inline__ uint64_t
	htonll(uint64_t n)
	{
	return ((((uint64_t)htonl(n)) << 32) + htonl(n >> 32));
	}

	static __inline__ uint64_t
	ntohll(uint64_t n)
	{
	return ((((uint64_t)ntohl(n)) << 32) + ntohl(n >> 32));
	}
	#endif

	#endif /* SPL_BYTEORDER_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/callb.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/callb.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/callb.h (revision 366774)
	@@ -1,54 +1,53 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_CALLB_H
	#define _SPL_CALLB_H

	#include <linux/module.h>
	#include <sys/mutex.h>

	#define CALLB_CPR_ASSERT(cp) ASSERT(MUTEX_HELD((cp)->cc_lockp));

	typedef struct callb_cpr {
	kmutex_t *cc_lockp;
	} callb_cpr_t;

	#define CALLB_CPR_INIT(cp, lockp, func, name) { \
	(cp)->cc_lockp = lockp; \
	}

	#define CALLB_CPR_SAFE_BEGIN(cp) { \
	CALLB_CPR_ASSERT(cp); \
	}

	#define CALLB_CPR_SAFE_END(cp, lockp) { \
	CALLB_CPR_ASSERT(cp); \
	}

	#define CALLB_CPR_EXIT(cp) { \
	ASSERT(MUTEX_HELD((cp)->cc_lockp)); \
	mutex_exit((cp)->cc_lockp); \
	}

	#endif /* _SPL_CALLB_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/callo.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/callo.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/callo.h (revision 366774)
	@@ -1,52 +1,51 @@
	/*
	* Copyright (C) 2007-2013 Lawrence Livermore National Security, LLC.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_CALLO_H
	#define _SPL_CALLO_H

	/*
	* Callout flags:
	*
	* CALLOUT_FLAG_ROUNDUP
	* Roundup the expiration time to the next resolution boundary.
	* If this flag is not specified, the expiration time is rounded down.
	* CALLOUT_FLAG_ABSOLUTE
	* Normally, the expiration passed to the timeout API functions is an
	* expiration interval. If this flag is specified, then it is
	* interpreted as the expiration time itself.
	* CALLOUT_FLAG_HRESTIME
	* Normally, callouts are not affected by changes to system time
	* (hrestime). This flag is used to create a callout that is affected
	* by system time. If system time changes, these timers must be
	* handled in a special way (see callout.c). These are used by condition
	* variables and LWP timers that need this behavior.
	* CALLOUT_FLAG_32BIT
	* Legacy interfaces timeout() and realtime_timeout() pass this flag
	* to timeout_generic() to indicate that a 32-bit ID should be allocated.
	*/
	#define CALLOUT_FLAG_ROUNDUP 0x1
	#define CALLOUT_FLAG_ABSOLUTE 0x2
	#define CALLOUT_FLAG_HRESTIME 0x4
	#define CALLOUT_FLAG_32BIT 0x8

	#endif /* _SPL_CALLB_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/cmn_err.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/cmn_err.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/cmn_err.h (revision 366774)
	@@ -1,42 +1,41 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_CMN_ERR_H
	#define _SPL_CMN_ERR_H

	#include <stdarg.h>

	#define CE_CONT 0 /* continuation */
	#define CE_NOTE 1 /* notice */
	#define CE_WARN 2 /* warning */
	#define CE_PANIC 3 /* panic */
	#define CE_IGNORE 4 /* print nothing */

	extern void cmn_err(int, const char *, ...);
	extern void vcmn_err(int, const char *, va_list);
	extern void vpanic(const char *, va_list);

	#define fm_panic panic

	#endif /* SPL_CMN_ERR_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/condvar.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/condvar.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/condvar.h (revision 366774)
	@@ -1,120 +1,119 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_CONDVAR_H
	#define _SPL_CONDVAR_H

	#include <linux/module.h>
	#include <sys/kmem.h>
	#include <sys/mutex.h>
	#include <sys/callo.h>
	#include <sys/wait.h>
	#include <sys/time.h>

	/*
	* cv_timedwait() is similar to cv_wait() except that it additionally expects
	* a timeout value specified in ticks. When woken by cv_signal() or
	* cv_broadcast() it returns 1, otherwise when the timeout is reached -1 is
	* returned.
	*
	* cv_timedwait_sig() behaves the same as cv_timedwait() but blocks
	* interruptibly and can be woken by a signal (EINTR, ERESTART). When
	* this occurs 0 is returned.
	*
	* cv_timedwait_io() and cv_timedwait_sig_io() are variants of cv_timedwait()
	* and cv_timedwait_sig() which should be used when waiting for outstanding
	* IO to complete. They are responsible for updating the iowait accounting
	* when this is supported by the platform.
	*
	* cv_timedwait_hires() and cv_timedwait_sig_hires() are high resolution
	* versions of cv_timedwait() and cv_timedwait_sig(). They expect the timeout
	* to be specified as a hrtime_t allowing for timeouts of less than a tick.
	*
	* N.B. The return values differ slightly from the illumos implementation
	* which returns the time remaining, instead of 1, when woken. They both
	* return -1 on timeout. Consumers which need to know the time remaining
	* are responsible for tracking it themselves.
	*/


	/*
	* The kcondvar_t struct is protected by mutex taken externally before
	* calling any of the wait/signal funs, and passed into the wait funs.
	*/
	#define CV_MAGIC 0x346545f4
	#define CV_DESTROY 0x346545f5

	typedef struct {
	int cv_magic;
	spl_wait_queue_head_t cv_event;
	spl_wait_queue_head_t cv_destroy;
	atomic_t cv_refs;
	atomic_t cv_waiters;
	kmutex_t *cv_mutex;
	} kcondvar_t;

	typedef enum { CV_DEFAULT = 0, CV_DRIVER } kcv_type_t;

	extern void __cv_init(kcondvar_t , char , kcv_type_t, void *);
	extern void __cv_destroy(kcondvar_t *);
	extern void __cv_wait(kcondvar_t , kmutex_t );
	extern void __cv_wait_io(kcondvar_t , kmutex_t );
	extern void __cv_wait_idle(kcondvar_t , kmutex_t );
	extern int __cv_wait_io_sig(kcondvar_t , kmutex_t );
	extern int __cv_wait_sig(kcondvar_t , kmutex_t );
	extern int __cv_timedwait(kcondvar_t , kmutex_t , clock_t);
	extern int __cv_timedwait_io(kcondvar_t , kmutex_t , clock_t);
	extern int __cv_timedwait_sig(kcondvar_t , kmutex_t , clock_t);
	extern int __cv_timedwait_idle(kcondvar_t , kmutex_t , clock_t);
	extern int cv_timedwait_hires(kcondvar_t , kmutex_t , hrtime_t,
	hrtime_t res, int flag);
	extern int cv_timedwait_sig_hires(kcondvar_t , kmutex_t , hrtime_t,
	hrtime_t res, int flag);
	extern int cv_timedwait_idle_hires(kcondvar_t , kmutex_t , hrtime_t,
	hrtime_t res, int flag);
	extern void __cv_signal(kcondvar_t *);
	extern void __cv_broadcast(kcondvar_t *c);

	#define cv_init(cvp, name, type, arg) __cv_init(cvp, name, type, arg)
	#define cv_destroy(cvp) __cv_destroy(cvp)
	#define cv_wait(cvp, mp) __cv_wait(cvp, mp)
	#define cv_wait_io(cvp, mp) __cv_wait_io(cvp, mp)
	#define cv_wait_idle(cvp, mp) __cv_wait_idle(cvp, mp)
	#define cv_wait_io_sig(cvp, mp) __cv_wait_io_sig(cvp, mp)
	#define cv_wait_sig(cvp, mp) __cv_wait_sig(cvp, mp)
	#define cv_signal(cvp) __cv_signal(cvp)
	#define cv_broadcast(cvp) __cv_broadcast(cvp)

	/*
	* NB: There is no way to reliably distinguish between having been signalled
	* and having timed out on Linux. If the client code needs to reliably
	* distinguish between the two it should use the hires variant.
	*/
	#define cv_timedwait(cvp, mp, t) __cv_timedwait(cvp, mp, t)
	#define cv_timedwait_io(cvp, mp, t) __cv_timedwait_io(cvp, mp, t)
	#define cv_timedwait_sig(cvp, mp, t) __cv_timedwait_sig(cvp, mp, t)
	#define cv_timedwait_idle(cvp, mp, t) __cv_timedwait_idle(cvp, mp, t)


	#endif /* _SPL_CONDVAR_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/console.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/console.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/console.h (revision 366774)
	@@ -1,31 +1,30 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_CONSOLE_H
	#define _SPL_CONSOLE_H

	#define console_vprintf vprintk
	#define console_printf printk

	#endif /* _SPL_CONSOLE_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/cred.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/cred.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/cred.h (revision 366774)
	@@ -1,63 +1,62 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_CRED_H
	#define _SPL_CRED_H

	#include <linux/module.h>
	#include <linux/cred.h>
	#include <sys/types.h>
	#include <sys/vfs.h>

	typedef struct cred cred_t;

	#define kcred ((cred_t *)(init_task.cred))
	#define CRED() ((cred_t *)current_cred())

	/* Linux 4.9 API change, GROUP_AT was removed */
	#ifndef GROUP_AT
	#define GROUP_AT(gi, i) ((gi)->gid[i])
	#endif

	#define KUID_TO_SUID(x) (__kuid_val(x))
	#define KGID_TO_SGID(x) (__kgid_val(x))
	#define SUID_TO_KUID(x) (KUIDT_INIT(x))
	#define SGID_TO_KGID(x) (KGIDT_INIT(x))
	#define KGIDP_TO_SGIDP(x) (&(x)->val)

	extern void crhold(cred_t *cr);
	extern void crfree(cred_t *cr);
	extern uid_t crgetuid(const cred_t *cr);
	extern uid_t crgetruid(const cred_t *cr);
	extern uid_t crgetsuid(const cred_t *cr);
	extern uid_t crgetfsuid(const cred_t *cr);
	extern gid_t crgetgid(const cred_t *cr);
	extern gid_t crgetrgid(const cred_t *cr);
	extern gid_t crgetsgid(const cred_t *cr);
	extern gid_t crgetfsgid(const cred_t *cr);
	extern int crgetngroups(const cred_t *cr);
	extern gid_t crgetgroups(const cred_t cr);
	extern int groupmember(gid_t gid, const cred_t *cr);

	#endif /* _SPL_CRED_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/ctype.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/ctype.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/ctype.h (revision 366774)
	@@ -1,30 +1,29 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_CTYPE_H
	#define _SPL_CTYPE_H

	#include <linux/ctype.h>

	#endif /* SPL_CTYPE_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/debug.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/debug.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/debug.h (revision 366774)
	@@ -1,169 +1,168 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	/*
	* Available Solaris debug functions. All of the ASSERT() macros will be
	* compiled out when NDEBUG is defined, this is the default behavior for
	* the SPL. To enable assertions use the --enable-debug with configure.
	* The VERIFY() functions are never compiled out and cannot be disabled.
	*
	* PANIC() - Panic the node and print message.
	* ASSERT() - Assert X is true, if not panic.
	* ASSERT3B() - Assert boolean X OP Y is true, if not panic.
	* ASSERT3S() - Assert signed X OP Y is true, if not panic.
	* ASSERT3U() - Assert unsigned X OP Y is true, if not panic.
	* ASSERT3P() - Assert pointer X OP Y is true, if not panic.
	* ASSERT0() - Assert value is zero, if not panic.
	* VERIFY() - Verify X is true, if not panic.
	* VERIFY3B() - Verify boolean X OP Y is true, if not panic.
	* VERIFY3S() - Verify signed X OP Y is true, if not panic.
	* VERIFY3U() - Verify unsigned X OP Y is true, if not panic.
	* VERIFY3P() - Verify pointer X OP Y is true, if not panic.
	* VERIFY0() - Verify value is zero, if not panic.
	*/

	#ifndef _SPL_DEBUG_H
	#define _SPL_DEBUG_H

	/*
	* Common DEBUG functionality.
	*/
	#define __printflike(a, b) __printf(a, b)

	#ifndef __maybe_unused
	#define __maybe_unused __attribute__((unused))
	#endif

	int spl_panic(const char file, const char func, int line,
	const char *fmt, ...);
	void spl_dumpstack(void);

	/* BEGIN CSTYLED */
	#define PANIC(fmt, a...) \
	spl_panic(__FILE__, __FUNCTION__, __LINE__, fmt, ## a)

	#define VERIFY(cond) \
	(void) (unlikely(!(cond)) && \
	spl_panic(__FILE__, __FUNCTION__, __LINE__, \
	"%s", "VERIFY(" #cond ") failed\n"))

	#define VERIFY3B(LEFT, OP, RIGHT) do { \
	boolean_t _verify3_left = (boolean_t)(LEFT); \
	boolean_t _verify3_right = (boolean_t)(RIGHT); \
	if (!(_verify3_left OP _verify3_right)) \
	spl_panic(__FILE__, __FUNCTION__, __LINE__, \
	"VERIFY3(" #LEFT " " #OP " " #RIGHT ") " \
	"failed (%d " #OP " %d)\n", \
	(boolean_t) (_verify3_left), \
	(boolean_t) (_verify3_right)); \
	} while (0)

	#define VERIFY3S(LEFT, OP, RIGHT) do { \
	int64_t _verify3_left = (int64_t)(LEFT); \
	int64_t _verify3_right = (int64_t)(RIGHT); \
	if (!(_verify3_left OP _verify3_right)) \
	spl_panic(__FILE__, __FUNCTION__, __LINE__, \
	"VERIFY3(" #LEFT " " #OP " " #RIGHT ") " \
	"failed (%lld " #OP " %lld)\n", \
	(long long) (_verify3_left), \
	(long long) (_verify3_right)); \
	} while (0)

	#define VERIFY3U(LEFT, OP, RIGHT) do { \
	uint64_t _verify3_left = (uint64_t)(LEFT); \
	uint64_t _verify3_right = (uint64_t)(RIGHT); \
	if (!(_verify3_left OP _verify3_right)) \
	spl_panic(__FILE__, __FUNCTION__, __LINE__, \
	"VERIFY3(" #LEFT " " #OP " " #RIGHT ") " \
	"failed (%llu " #OP " %llu)\n", \
	(unsigned long long) (_verify3_left), \
	(unsigned long long) (_verify3_right)); \
	} while (0)

	#define VERIFY3P(LEFT, OP, RIGHT) do { \
	uintptr_t _verify3_left = (uintptr_t)(LEFT); \
	uintptr_t _verify3_right = (uintptr_t)(RIGHT); \
	if (!(_verify3_left OP _verify3_right)) \
	spl_panic(__FILE__, __FUNCTION__, __LINE__, \
	"VERIFY3(" #LEFT " " #OP " " #RIGHT ") " \
	"failed (%px " #OP " %px)\n", \
	(void *) (_verify3_left), \
	(void *) (_verify3_right)); \
	} while (0)

	#define VERIFY0(RIGHT) do { \
	int64_t _verify3_left = (int64_t)(0); \
	int64_t _verify3_right = (int64_t)(RIGHT); \
	if (!(_verify3_left == _verify3_right)) \
	spl_panic(__FILE__, __FUNCTION__, __LINE__, \
	"VERIFY3(0 == " #RIGHT ") " \
	"failed (0 == %lld)\n", \
	(long long) (_verify3_right)); \
	} while (0)

	#define CTASSERT_GLOBAL(x) _CTASSERT(x, __LINE__)
	#define CTASSERT(x) { _CTASSERT(x, __LINE__); }
	#define _CTASSERT(x, y) __CTASSERT(x, y)
	#define __CTASSERT(x, y) \
	typedef char __attribute__ ((unused)) \
	__compile_time_assertion__ ## y[(x) ? 1 : -1]

	/*
	* Debugging disabled (--disable-debug)
	*/
	#ifdef NDEBUG

	#define ASSERT(x) ((void)0)
	#define ASSERT3B(x,y,z) ((void)0)
	#define ASSERT3S(x,y,z) ((void)0)
	#define ASSERT3U(x,y,z) ((void)0)
	#define ASSERT3P(x,y,z) ((void)0)
	#define ASSERT0(x) ((void)0)
	#define IMPLY(A, B) ((void)0)
	#define EQUIV(A, B) ((void)0)

	/*
	* Debugging enabled (--enable-debug)
	*/
	#else

	#define ASSERT3B VERIFY3B
	#define ASSERT3S VERIFY3S
	#define ASSERT3U VERIFY3U
	#define ASSERT3P VERIFY3P
	#define ASSERT0 VERIFY0
	#define ASSERT VERIFY
	#define IMPLY(A, B) \
	((void)(((!(A)) \|\| (B)) \|\| \
	spl_panic(__FILE__, __FUNCTION__, __LINE__, \
	"(" #A ") implies (" #B ")")))
	#define EQUIV(A, B) \
	((void)((!!(A) == !!(B)) \|\| \
	spl_panic(__FILE__, __FUNCTION__, __LINE__, \
	"(" #A ") is equivalent to (" #B ")")))
	/* END CSTYLED */

	#endif /* NDEBUG */

	#endif /* SPL_DEBUG_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/disp.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/disp.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/disp.h (revision 366774)
	@@ -1,34 +1,33 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_DISP_H
	#define _SPL_DISP_H

	#include <linux/preempt.h>

	#define kpreempt(unused) schedule()
	#define kpreempt_disable() preempt_disable()
	#define kpreempt_enable() preempt_enable()

	#endif /* SPL_DISP_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/dkio.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/dkio.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/dkio.h (revision 366774)
	@@ -1,40 +1,39 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_DKIO_H
	#define _SPL_DKIO_H

	#define DFL_SZ(num_exts) \
	(sizeof (dkioc_free_list_t) + (num_exts - 1) * 16)

	#define DKIOC (0x04 << 8)
	#define DKIOCFLUSHWRITECACHE (DKIOC\|34) /* flush cache to phys medium */

	/*
	* ioctl to free space (e.g. SCSI UNMAP) off a disk.
	* Pass a dkioc_free_list_t containing a list of extents to be freed.
	*/
	#define DKIOCFREE (DKIOC\|50)

	#endif /* _SPL_DKIO_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/fcntl.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/fcntl.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/fcntl.h (revision 366774)
	@@ -1,37 +1,36 @@
	/*
	* Copyright (C) 2010 Lawrence Livermore National Security, LLC.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_FCNTL_H
	#define _SPL_FCNTL_H

	#include <asm/fcntl.h>

	#define F_FREESP 11

	#ifdef CONFIG_64BIT
	typedef struct flock flock64_t;
	#else
	typedef struct flock64 flock64_t;
	#endif /* CONFIG_64BIT */

	#endif /* _SPL_FCNTL_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/file.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/file.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/file.h (revision 366774)
	@@ -1,52 +1,51 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_FILE_H
	#define _SPL_FILE_H

	#define FIGNORECASE 0x00080000
	#define FKIOCTL 0x80000000
	#define ED_CASE_CONFLICT 0x10

	#ifdef HAVE_INODE_LOCK_SHARED
	#define spl_inode_lock(ip) inode_lock(ip)
	#define spl_inode_unlock(ip) inode_unlock(ip)
	#define spl_inode_lock_shared(ip) inode_lock_shared(ip)
	#define spl_inode_unlock_shared(ip) inode_unlock_shared(ip)
	#define spl_inode_trylock(ip) inode_trylock(ip)
	#define spl_inode_trylock_shared(ip) inode_trylock_shared(ip)
	#define spl_inode_is_locked(ip) inode_is_locked(ip)
	#define spl_inode_lock_nested(ip, s) inode_lock_nested(ip, s)
	#else
	#define spl_inode_lock(ip) mutex_lock(&(ip)->i_mutex)
	#define spl_inode_unlock(ip) mutex_unlock(&(ip)->i_mutex)
	#define spl_inode_lock_shared(ip) mutex_lock(&(ip)->i_mutex)
	#define spl_inode_unlock_shared(ip) mutex_unlock(&(ip)->i_mutex)
	#define spl_inode_trylock(ip) mutex_trylock(&(ip)->i_mutex)
	#define spl_inode_trylock_shared(ip) mutex_trylock(&(ip)->i_mutex)
	#define spl_inode_is_locked(ip) mutex_is_locked(&(ip)->i_mutex)
	#define spl_inode_lock_nested(ip, s) mutex_lock_nested(&(ip)->i_mutex, s)
	#endif

	#endif /* SPL_FILE_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/inttypes.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/inttypes.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/inttypes.h (revision 366774)
	@@ -1,28 +1,27 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_INTTYPES_H
	#define _SPL_INTTYPES_H

	#endif /* SPL_INTTYPES_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/isa_defs.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/isa_defs.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/isa_defs.h (revision 366774)
	@@ -1,254 +1,253 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_ISA_DEFS_H
	#define _SPL_ISA_DEFS_H

	/* x86_64 arch specific defines */
	#if defined(__x86_64) \|\| defined(__x86_64__)

	#if !defined(__x86_64)
	#define __x86_64
	#endif

	#if !defined(__amd64)
	#define __amd64
	#endif

	#if !defined(__x86)
	#define __x86
	#endif

	#if defined(_ILP32)
	/* x32-specific defines; careful to not define _LP64 here */
	#else
	#if !defined(_LP64)
	#define _LP64
	#endif
	#endif

	#define _ALIGNMENT_REQUIRED 1


	/* i386 arch specific defines */
	#elif defined(__i386) \|\| defined(__i386__)

	#if !defined(__i386)
	#define __i386
	#endif

	#if !defined(__x86)
	#define __x86
	#endif

	#if !defined(_ILP32)
	#define _ILP32
	#endif

	#define _ALIGNMENT_REQUIRED 0

	/* powerpc (ppc64) arch specific defines */
	#elif defined(__powerpc) \|\| defined(__powerpc__) \|\| defined(__powerpc64__)

	#if !defined(__powerpc)
	#define __powerpc
	#endif

	#if !defined(__powerpc__)
	#define __powerpc__
	#endif

	#if defined(__powerpc64__)
	#if !defined(_LP64)
	#define _LP64
	#endif
	#else
	#if !defined(_ILP32)
	#define _ILP32
	#endif
	#endif

	/*
	* Illumos doesn't define _ALIGNMENT_REQUIRED for PPC, so default to 1
	* out of paranoia.
	*/
	#define _ALIGNMENT_REQUIRED 1

	/* arm arch specific defines */
	#elif defined(__arm) \|\| defined(__arm__) \|\| defined(__aarch64__)

	#if !defined(__arm)
	#define __arm
	#endif

	#if !defined(__arm__)
	#define __arm__
	#endif

	#if defined(__aarch64__)
	#if !defined(_LP64)
	#define _LP64
	#endif
	#else
	#if !defined(_ILP32)
	#define _ILP32
	#endif
	#endif

	#if defined(__ARMEL__) \|\| defined(__AARCH64EL__)
	#define _ZFS_LITTLE_ENDIAN
	#else
	#define _ZFS_BIG_ENDIAN
	#endif

	/*
	* Illumos doesn't define _ALIGNMENT_REQUIRED for ARM, so default to 1
	* out of paranoia.
	*/
	#define _ALIGNMENT_REQUIRED 1

	/* sparc arch specific defines */
	#elif defined(__sparc) \|\| defined(__sparc__)

	#if !defined(__sparc)
	#define __sparc
	#endif

	#if !defined(__sparc__)
	#define __sparc__
	#endif

	#if defined(__arch64__)
	#if !defined(_LP64)
	#define _LP64
	#endif
	#else
	#if !defined(_ILP32)
	#define _ILP32
	#endif
	#endif

	#define _ZFS_BIG_ENDIAN
	#define _SUNOS_VTOC_16
	#define _ALIGNMENT_REQUIRED 1

	/* s390 arch specific defines */
	#elif defined(__s390__)
	#if defined(__s390x__)
	#if !defined(_LP64)
	#define _LP64
	#endif
	#else
	#if !defined(_ILP32)
	#define _ILP32
	#endif
	#endif

	#define _ZFS_BIG_ENDIAN

	/*
	* Illumos doesn't define _ALIGNMENT_REQUIRED for s390, so default to 1
	* out of paranoia.
	*/
	#define _ALIGNMENT_REQUIRED 1

	/* MIPS arch specific defines */
	#elif defined(__mips__)

	#if defined(__MIPSEB__)
	#define _ZFS_BIG_ENDIAN
	#elif defined(__MIPSEL__)
	#define _ZFS_LITTLE_ENDIAN
	#else
	#error MIPS no endian specified
	#endif

	#ifndef _LP64
	#define _ILP32
	#endif

	#define _SUNOS_VTOC_16

	/*
	* Illumos doesn't define _ALIGNMENT_REQUIRED for MIPS, so default to 1
	* out of paranoia.
	*/
	#define _ALIGNMENT_REQUIRED 1

	/*
	* RISC-V arch specific defines
	* only RV64G (including atomic) LP64 is supported yet
	*/
	#elif defined(__riscv) && defined(_LP64) && _LP64 && \
	defined(__riscv_atomic) && __riscv_atomic

	#ifndef __riscv__
	#define __riscv__
	#endif

	#ifndef __rv64g__
	#define __rv64g__
	#endif

	#define _ZFS_LITTLE_ENDIAN

	#define _SUNOS_VTOC_16

	#define _ALIGNMENT_REQUIRED 1

	#else
	/*
	* Currently supported:
	* x86_64, x32, i386, arm, powerpc, s390, sparc, mips, and RV64G
	*/
	#error "Unsupported ISA type"
	#endif

	#if defined(_ILP32) && defined(_LP64)
	#error "Both _ILP32 and _LP64 are defined"
	#endif

	#if !defined(_ILP32) && !defined(_LP64)
	#error "Neither _ILP32 or _LP64 are defined"
	#endif

	#include <sys/byteorder.h>

	/*
	* CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS will be defined by the Linux
	* kernel for architectures which support efficient unaligned access.
	*/
	#if defined(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)
	#define HAVE_EFFICIENT_UNALIGNED_ACCESS
	#endif

	#if defined(_ZFS_LITTLE_ENDIAN) && defined(_ZFS_BIG_ENDIAN)
	#error "Both _ZFS_LITTLE_ENDIAN and _ZFS_BIG_ENDIAN are defined"
	#endif

	#if !defined(_ZFS_LITTLE_ENDIAN) && !defined(_ZFS_BIG_ENDIAN)
	#error "Neither _ZFS_LITTLE_ENDIAN or _ZFS_BIG_ENDIAN are defined"
	#endif

	#endif /* _SPL_ISA_DEFS_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/kmem.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/kmem.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/kmem.h (revision 366774)
	@@ -1,211 +1,210 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_KMEM_H
	#define _SPL_KMEM_H

	#include <sys/debug.h>
	#include <linux/slab.h>
	#include <linux/sched.h>
	#include <linux/mm.h>
	#include <linux/vmalloc.h>

	extern int kmem_debugging(void);
	extern char kmem_vasprintf(const char fmt, va_list ap);
	extern char kmem_asprintf(const char fmt, ...);
	extern char kmem_strdup(const char str);
	extern void kmem_strfree(char *str);

	/*
	* Memory allocation interfaces
	*/
	#define KM_SLEEP 0x0000 /* can block for memory; success guaranteed */
	#define KM_NOSLEEP 0x0001 /* cannot block for memory; may fail */
	#define KM_PUSHPAGE 0x0004 /* can block for memory; may use reserve */
	#define KM_ZERO 0x1000 /* zero the allocation */
	#define KM_VMEM 0x2000 /* caller is vmem_* wrapper */

	#define KM_PUBLIC_MASK (KM_SLEEP \| KM_NOSLEEP \| KM_PUSHPAGE)

	static int spl_fstrans_check(void);
	void *spl_kvmalloc(size_t size, gfp_t flags);

	/*
	* Convert a KM_* flags mask to its Linux GFP_* counterpart. The conversion
	* function is context aware which means that KM_SLEEP allocations can be
	* safely used in syncing contexts which have set PF_FSTRANS.
	*/
	static inline gfp_t
	kmem_flags_convert(int flags)
	{
	gfp_t lflags = __GFP_NOWARN \| __GFP_COMP;

	if (flags & KM_NOSLEEP) {
	lflags \|= GFP_ATOMIC \| __GFP_NORETRY;
	} else {
	lflags \|= GFP_KERNEL;
	if (spl_fstrans_check())
	lflags &= ~(__GFP_IO\|__GFP_FS);
	}

	if (flags & KM_PUSHPAGE)
	lflags \|= __GFP_HIGH;

	if (flags & KM_ZERO)
	lflags \|= __GFP_ZERO;

	return (lflags);
	}

	typedef struct {
	struct task_struct *fstrans_thread;
	unsigned int saved_flags;
	} fstrans_cookie_t;

	/*
	* Introduced in Linux 3.9, however this cannot be solely relied on before
	* Linux 3.18 as it doesn't turn off __GFP_FS as it should.
	*/
	#ifdef PF_MEMALLOC_NOIO
	#define __SPL_PF_MEMALLOC_NOIO (PF_MEMALLOC_NOIO)
	#else
	#define __SPL_PF_MEMALLOC_NOIO (0)
	#endif

	/*
	* PF_FSTRANS is removed from Linux 4.12
	*/
	#ifdef PF_FSTRANS
	#define __SPL_PF_FSTRANS (PF_FSTRANS)
	#else
	#define __SPL_PF_FSTRANS (0)
	#endif

	#define SPL_FSTRANS (__SPL_PF_FSTRANS\|__SPL_PF_MEMALLOC_NOIO)

	static inline fstrans_cookie_t
	spl_fstrans_mark(void)
	{
	fstrans_cookie_t cookie;

	BUILD_BUG_ON(SPL_FSTRANS == 0);

	cookie.fstrans_thread = current;
	cookie.saved_flags = current->flags & SPL_FSTRANS;
	current->flags \|= SPL_FSTRANS;

	return (cookie);
	}

	static inline void
	spl_fstrans_unmark(fstrans_cookie_t cookie)
	{
	ASSERT3P(cookie.fstrans_thread, ==, current);
	ASSERT((current->flags & SPL_FSTRANS) == SPL_FSTRANS);

	current->flags &= ~SPL_FSTRANS;
	current->flags \|= cookie.saved_flags;
	}

	static inline int
	spl_fstrans_check(void)
	{
	return (current->flags & SPL_FSTRANS);
	}

	/*
	* specifically used to check PF_FSTRANS flag, cannot be relied on for
	* checking spl_fstrans_mark().
	*/
	static inline int
	__spl_pf_fstrans_check(void)
	{
	return (current->flags & __SPL_PF_FSTRANS);
	}

	/*
	* Kernel compatibility for GFP flags
	*/
	/* < 4.13 */
	#ifndef __GFP_RETRY_MAYFAIL
	#define __GFP_RETRY_MAYFAIL __GFP_REPEAT
	#endif
	/* < 4.4 */
	#ifndef __GFP_RECLAIM
	#define __GFP_RECLAIM __GFP_WAIT
	#endif

	#ifdef HAVE_ATOMIC64_T
	#define kmem_alloc_used_add(size) atomic64_add(size, &kmem_alloc_used)
	#define kmem_alloc_used_sub(size) atomic64_sub(size, &kmem_alloc_used)
	#define kmem_alloc_used_read() atomic64_read(&kmem_alloc_used)
	#define kmem_alloc_used_set(size) atomic64_set(&kmem_alloc_used, size)
	extern atomic64_t kmem_alloc_used;
	extern unsigned long long kmem_alloc_max;
	#else /* HAVE_ATOMIC64_T */
	#define kmem_alloc_used_add(size) atomic_add(size, &kmem_alloc_used)
	#define kmem_alloc_used_sub(size) atomic_sub(size, &kmem_alloc_used)
	#define kmem_alloc_used_read() atomic_read(&kmem_alloc_used)
	#define kmem_alloc_used_set(size) atomic_set(&kmem_alloc_used, size)
	extern atomic_t kmem_alloc_used;
	extern unsigned long long kmem_alloc_max;
	#endif /* HAVE_ATOMIC64_T */

	extern unsigned int spl_kmem_alloc_warn;
	extern unsigned int spl_kmem_alloc_max;

	#define kmem_alloc(sz, fl) spl_kmem_alloc((sz), (fl), __func__, __LINE__)
	#define kmem_zalloc(sz, fl) spl_kmem_zalloc((sz), (fl), __func__, __LINE__)
	#define kmem_free(ptr, sz) spl_kmem_free((ptr), (sz))
	#define kmem_cache_reap_active spl_kmem_cache_reap_active

	extern void spl_kmem_alloc(size_t sz, int fl, const char func, int line);
	extern void spl_kmem_zalloc(size_t sz, int fl, const char func, int line);
	extern void spl_kmem_free(const void *ptr, size_t sz);

	/*
	* 5.8 API change, pgprot_t argument removed.
	*/
	#ifdef HAVE_VMALLOC_PAGE_KERNEL
	#define spl_vmalloc(size, flags) __vmalloc(size, flags, PAGE_KERNEL)
	#else
	#define spl_vmalloc(size, flags) __vmalloc(size, flags)
	#endif

	/*
	* The following functions are only available for internal use.
	*/
	extern void *spl_kmem_alloc_impl(size_t size, int flags, int node);
	extern void *spl_kmem_alloc_debug(size_t size, int flags, int node);
	extern void *spl_kmem_alloc_track(size_t size, int flags,
	const char *func, int line, int node);
	extern void spl_kmem_free_impl(const void *buf, size_t size);
	extern void spl_kmem_free_debug(const void *buf, size_t size);
	extern void spl_kmem_free_track(const void *buf, size_t size);

	extern int spl_kmem_init(void);
	extern void spl_kmem_fini(void);
	extern int spl_kmem_cache_reap_active(void);

	#endif /* _SPL_KMEM_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/kmem_cache.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/kmem_cache.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/kmem_cache.h (revision 366774)
	@@ -1,216 +1,215 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_KMEM_CACHE_H
	#define _SPL_KMEM_CACHE_H

	#include <sys/taskq.h>

	/*
	* Slab allocation interfaces. The SPL slab differs from the standard
	* Linux SLAB or SLUB primarily in that each cache may be backed by slabs
	* allocated from the physical or virtual memory address space. The virtual
	* slabs allow for good behavior when allocation large objects of identical
	* size. This slab implementation also supports both constructors and
	* destructors which the Linux slab does not.
	*/
	typedef enum kmc_bit {
	KMC_BIT_NODEBUG = 1, /* Default behavior */
	KMC_BIT_KVMEM = 7, /* Use kvmalloc linux allocator */
	KMC_BIT_SLAB = 8, /* Use Linux slab cache */
	KMC_BIT_DEADLOCKED = 14, /* Deadlock detected */
	KMC_BIT_GROWING = 15, /* Growing in progress */
	KMC_BIT_REAPING = 16, /* Reaping in progress */
	KMC_BIT_DESTROY = 17, /* Destroy in progress */
	KMC_BIT_TOTAL = 18, /* Proc handler helper bit */
	KMC_BIT_ALLOC = 19, /* Proc handler helper bit */
	KMC_BIT_MAX = 20, /* Proc handler helper bit */
	} kmc_bit_t;

	/* kmem move callback return values */
	typedef enum kmem_cbrc {
	KMEM_CBRC_YES = 0, /* Object moved */
	KMEM_CBRC_NO = 1, /* Object not moved */
	KMEM_CBRC_LATER = 2, /* Object not moved, try again later */
	KMEM_CBRC_DONT_NEED = 3, /* Neither object is needed */
	KMEM_CBRC_DONT_KNOW = 4, /* Object unknown */
	} kmem_cbrc_t;

	#define KMC_NODEBUG (1 << KMC_BIT_NODEBUG)
	#define KMC_KVMEM (1 << KMC_BIT_KVMEM)
	#define KMC_SLAB (1 << KMC_BIT_SLAB)
	#define KMC_DEADLOCKED (1 << KMC_BIT_DEADLOCKED)
	#define KMC_GROWING (1 << KMC_BIT_GROWING)
	#define KMC_REAPING (1 << KMC_BIT_REAPING)
	#define KMC_DESTROY (1 << KMC_BIT_DESTROY)
	#define KMC_TOTAL (1 << KMC_BIT_TOTAL)
	#define KMC_ALLOC (1 << KMC_BIT_ALLOC)
	#define KMC_MAX (1 << KMC_BIT_MAX)

	#define KMC_REAP_CHUNK INT_MAX
	#define KMC_DEFAULT_SEEKS 1

	#define KMC_RECLAIM_ONCE 0x1 /* Force a single shrinker pass */

	extern struct list_head spl_kmem_cache_list;
	extern struct rw_semaphore spl_kmem_cache_sem;

	#define SKM_MAGIC 0x2e2e2e2e
	#define SKO_MAGIC 0x20202020
	#define SKS_MAGIC 0x22222222
	#define SKC_MAGIC 0x2c2c2c2c

	#define SPL_KMEM_CACHE_OBJ_PER_SLAB 8 /* Target objects per slab */
	#define SPL_KMEM_CACHE_ALIGN 8 /* Default object alignment */
	#ifdef _LP64
	#define SPL_KMEM_CACHE_MAX_SIZE 32 /* Max slab size in MB */
	#else
	#define SPL_KMEM_CACHE_MAX_SIZE 4 /* Max slab size in MB */
	#endif

	#define SPL_MAX_ORDER (MAX_ORDER - 3)
	#define SPL_MAX_ORDER_NR_PAGES (1 << (SPL_MAX_ORDER - 1))

	#ifdef CONFIG_SLUB
	#define SPL_MAX_KMEM_CACHE_ORDER PAGE_ALLOC_COSTLY_ORDER
	#define SPL_MAX_KMEM_ORDER_NR_PAGES (1 << (SPL_MAX_KMEM_CACHE_ORDER - 1))
	#else
	#define SPL_MAX_KMEM_ORDER_NR_PAGES (KMALLOC_MAX_SIZE >> PAGE_SHIFT)
	#endif

	#define POINTER_IS_VALID(p) 0 /* Unimplemented */
	#define POINTER_INVALIDATE(pp) /* Unimplemented */

	typedef int (spl_kmem_ctor_t)(void , void *, int);
	typedef void (spl_kmem_dtor_t)(void , void *);

	typedef struct spl_kmem_magazine {
	uint32_t skm_magic; /* Sanity magic */
	uint32_t skm_avail; /* Available objects */
	uint32_t skm_size; /* Magazine size */
	uint32_t skm_refill; /* Batch refill size */
	struct spl_kmem_cache skm_cache; / Owned by cache */
	unsigned int skm_cpu; /* Owned by cpu */
	void skm_objs[0]; / Object pointers */
	} spl_kmem_magazine_t;

	typedef struct spl_kmem_obj {
	uint32_t sko_magic; /* Sanity magic */
	void sko_addr; / Buffer address */
	struct spl_kmem_slab sko_slab; / Owned by slab */
	struct list_head sko_list; /* Free object list linkage */
	} spl_kmem_obj_t;

	typedef struct spl_kmem_slab {
	uint32_t sks_magic; /* Sanity magic */
	uint32_t sks_objs; /* Objects per slab */
	struct spl_kmem_cache sks_cache; / Owned by cache */
	struct list_head sks_list; /* Slab list linkage */
	struct list_head sks_free_list; /* Free object list */
	unsigned long sks_age; /* Last modify jiffie */
	uint32_t sks_ref; /* Ref count used objects */
	} spl_kmem_slab_t;

	typedef struct spl_kmem_alloc {
	struct spl_kmem_cache ska_cache; / Owned by cache */
	int ska_flags; /* Allocation flags */
	taskq_ent_t ska_tqe; /* Task queue entry */
	} spl_kmem_alloc_t;

	typedef struct spl_kmem_emergency {
	struct rb_node ske_node; /* Emergency tree linkage */
	unsigned long ske_obj; /* Buffer address */
	} spl_kmem_emergency_t;

	typedef struct spl_kmem_cache {
	uint32_t skc_magic; /* Sanity magic */
	uint32_t skc_name_size; /* Name length */
	char skc_name; / Name string */
	spl_kmem_magazine_t *skc_mag; / Per-CPU warm cache */
	uint32_t skc_mag_size; /* Magazine size */
	uint32_t skc_mag_refill; /* Magazine refill count */
	spl_kmem_ctor_t skc_ctor; /* Constructor */
	spl_kmem_dtor_t skc_dtor; /* Destructor */
	void skc_private; / Private data */
	void skc_vmp; / Unused */
	struct kmem_cache skc_linux_cache; / Linux slab cache if used */
	unsigned long skc_flags; /* Flags */
	uint32_t skc_obj_size; /* Object size */
	uint32_t skc_obj_align; /* Object alignment */
	uint32_t skc_slab_objs; /* Objects per slab */
	uint32_t skc_slab_size; /* Slab size */
	atomic_t skc_ref; /* Ref count callers */
	taskqid_t skc_taskqid; /* Slab reclaim task */
	struct list_head skc_list; /* List of caches linkage */
	struct list_head skc_complete_list; /* Completely alloc'ed */
	struct list_head skc_partial_list; /* Partially alloc'ed */
	struct rb_root skc_emergency_tree; /* Min sized objects */
	spinlock_t skc_lock; /* Cache lock */
	spl_wait_queue_head_t skc_waitq; /* Allocation waiters */
	uint64_t skc_slab_fail; /* Slab alloc failures */
	uint64_t skc_slab_create; /* Slab creates */
	uint64_t skc_slab_destroy; /* Slab destroys */
	uint64_t skc_slab_total; /* Slab total current */
	uint64_t skc_slab_alloc; /* Slab alloc current */
	uint64_t skc_slab_max; /* Slab max historic */
	uint64_t skc_obj_total; /* Obj total current */
	uint64_t skc_obj_alloc; /* Obj alloc current */
	struct percpu_counter skc_linux_alloc; /* Linux-backed Obj alloc */
	uint64_t skc_obj_max; /* Obj max historic */
	uint64_t skc_obj_deadlock; /* Obj emergency deadlocks */
	uint64_t skc_obj_emergency; /* Obj emergency current */
	uint64_t skc_obj_emergency_max; /* Obj emergency max */
	} spl_kmem_cache_t;
	#define kmem_cache_t spl_kmem_cache_t

	extern spl_kmem_cache_t spl_kmem_cache_create(char name, size_t size,
	size_t align, spl_kmem_ctor_t ctor, spl_kmem_dtor_t dtor,
	void reclaim, void priv, void *vmp, int flags);
	extern void spl_kmem_cache_set_move(spl_kmem_cache_t *,
	kmem_cbrc_t ()(void , void , size_t, void ));
	extern void spl_kmem_cache_destroy(spl_kmem_cache_t *skc);
	extern void spl_kmem_cache_alloc(spl_kmem_cache_t skc, int flags);
	extern void spl_kmem_cache_free(spl_kmem_cache_t skc, void obj);
	extern void spl_kmem_cache_set_allocflags(spl_kmem_cache_t *skc, gfp_t flags);
	extern void spl_kmem_cache_reap_now(spl_kmem_cache_t *skc);
	extern void spl_kmem_reap(void);
	extern uint64_t spl_kmem_cache_inuse(kmem_cache_t *cache);
	extern uint64_t spl_kmem_cache_entry_size(kmem_cache_t *cache);

	#define kmem_cache_create(name, size, align, ctor, dtor, rclm, priv, vmp, fl) \
	spl_kmem_cache_create(name, size, align, ctor, dtor, rclm, priv, vmp, fl)
	#define kmem_cache_set_move(skc, move) spl_kmem_cache_set_move(skc, move)
	#define kmem_cache_destroy(skc) spl_kmem_cache_destroy(skc)
	#define kmem_cache_alloc(skc, flags) spl_kmem_cache_alloc(skc, flags)
	#define kmem_cache_free(skc, obj) spl_kmem_cache_free(skc, obj)
	#define kmem_cache_reap_now(skc) spl_kmem_cache_reap_now(skc)
	#define kmem_reap() spl_kmem_reap()

	/*
	* The following functions are only available for internal use.
	*/
	extern int spl_kmem_cache_init(void);
	extern void spl_kmem_cache_fini(void);

	#endif /* _SPL_KMEM_CACHE_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/kstat.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/kstat.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/kstat.h (revision 366774)
	@@ -1,223 +1,222 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_KSTAT_H
	#define _SPL_KSTAT_H

	#include <linux/module.h>
	#include <sys/types.h>
	#include <sys/time.h>
	#include <sys/kmem.h>
	#include <sys/mutex.h>
	#include <sys/proc.h>

	#define KSTAT_STRLEN 255
	#define KSTAT_RAW_MAX (128*1024)

	/*
	* For reference valid classes are:
	* disk, tape, net, controller, vm, kvm, hat, streams, kstat, misc
	*/

	#define KSTAT_TYPE_RAW 0 /* can be anything; ks_ndata >= 1 */
	#define KSTAT_TYPE_NAMED 1 /* name/value pair; ks_ndata >= 1 */
	#define KSTAT_TYPE_INTR 2 /* interrupt stats; ks_ndata == 1 */
	#define KSTAT_TYPE_IO 3 /* I/O stats; ks_ndata == 1 */
	#define KSTAT_TYPE_TIMER 4 /* event timer; ks_ndata >= 1 */
	#define KSTAT_NUM_TYPES 5

	#define KSTAT_DATA_CHAR 0
	#define KSTAT_DATA_INT32 1
	#define KSTAT_DATA_UINT32 2
	#define KSTAT_DATA_INT64 3
	#define KSTAT_DATA_UINT64 4
	#define KSTAT_DATA_LONG 5
	#define KSTAT_DATA_ULONG 6
	#define KSTAT_DATA_STRING 7
	#define KSTAT_NUM_DATAS 8

	#define KSTAT_INTR_HARD 0
	#define KSTAT_INTR_SOFT 1
	#define KSTAT_INTR_WATCHDOG 2
	#define KSTAT_INTR_SPURIOUS 3
	#define KSTAT_INTR_MULTSVC 4
	#define KSTAT_NUM_INTRS 5

	#define KSTAT_FLAG_VIRTUAL 0x01
	#define KSTAT_FLAG_VAR_SIZE 0x02
	#define KSTAT_FLAG_WRITABLE 0x04
	#define KSTAT_FLAG_PERSISTENT 0x08
	#define KSTAT_FLAG_DORMANT 0x10
	#define KSTAT_FLAG_INVALID 0x20
	#define KSTAT_FLAG_LONGSTRINGS 0x40
	#define KSTAT_FLAG_NO_HEADERS 0x80

	#define KS_MAGIC 0x9d9d9d9d

	/* Dynamic updates */
	#define KSTAT_READ 0
	#define KSTAT_WRITE 1

	struct kstat_s;
	typedef struct kstat_s kstat_t;

	typedef int kid_t; /* unique kstat id */
	typedef int kstat_update_t(struct kstat_s , int); / dynamic update cb */

	typedef struct kstat_module {
	char ksm_name[KSTAT_STRLEN+1]; /* module name */
	struct list_head ksm_module_list; /* module linkage */
	struct list_head ksm_kstat_list; /* list of kstat entries */
	struct proc_dir_entry ksm_proc; / proc entry */
	} kstat_module_t;

	typedef struct kstat_raw_ops {
	int (headers)(char buf, size_t size);
	int (data)(char buf, size_t size, void *data);
	void (addr)(kstat_t *ksp, loff_t index);
	} kstat_raw_ops_t;

	typedef struct kstat_proc_entry {
	char kpe_name[KSTAT_STRLEN+1]; /* kstat name */
	char kpe_module[KSTAT_STRLEN+1]; /* provider module name */
	kstat_module_t kpe_owner; / kstat module linkage */
	struct list_head kpe_list; /* kstat linkage */
	struct proc_dir_entry kpe_proc; / procfs entry */
	} kstat_proc_entry_t;

	struct kstat_s {
	int ks_magic; /* magic value */
	kid_t ks_kid; /* unique kstat ID */
	hrtime_t ks_crtime; /* creation time */
	hrtime_t ks_snaptime; /* last access time */
	int ks_instance; /* provider module instance */
	char ks_class[KSTAT_STRLEN+1]; /* kstat class */
	uchar_t ks_type; /* kstat data type */
	uchar_t ks_flags; /* kstat flags */
	void ks_data; / kstat type-specific data */
	uint_t ks_ndata; /* # of data records */
	size_t ks_data_size; /* size of kstat data section */
	kstat_update_t ks_update; / dynamic updates */
	void ks_private; / private data */
	kmutex_t ks_private_lock; /* kstat private data lock */
	kmutex_t ks_lock; / kstat data lock */
	kstat_raw_ops_t ks_raw_ops; /* ops table for raw type */
	char ks_raw_buf; / buf used for raw ops */
	size_t ks_raw_bufsize; /* size of raw ops buffer */
	kstat_proc_entry_t ks_proc; /* data for procfs entry */
	};

	typedef struct kstat_named_s {
	char name[KSTAT_STRLEN]; /* name of counter */
	uchar_t data_type; /* data type */
	union {
	char c[16]; /* 128-bit int */
	int32_t i32; /* 32-bit signed int */
	uint32_t ui32; /* 32-bit unsigned int */
	int64_t i64; /* 64-bit signed int */
	uint64_t ui64; /* 64-bit unsigned int */
	long l; /* native signed long */
	ulong_t ul; /* native unsigned long */
	struct {
	union {
	char ptr; / NULL-term string */
	char __pad[8]; /* 64-bit padding */
	} addr;
	uint32_t len; /* # bytes for strlen + '\0' */
	} string;
	} value;
	} kstat_named_t;

	#define KSTAT_NAMED_STR_PTR(knptr) ((knptr)->value.string.addr.ptr)
	#define KSTAT_NAMED_STR_BUFLEN(knptr) ((knptr)->value.string.len)

	#ifdef HAVE_PROC_OPS_STRUCT
	typedef struct proc_ops kstat_proc_op_t;
	#else
	typedef struct file_operations kstat_proc_op_t;
	#endif

	typedef struct kstat_intr {
	uint_t intrs[KSTAT_NUM_INTRS];
	} kstat_intr_t;

	typedef struct kstat_io {
	u_longlong_t nread; /* number of bytes read */
	u_longlong_t nwritten; /* number of bytes written */
	uint_t reads; /* number of read operations */
	uint_t writes; /* number of write operations */
	hrtime_t wtime; /* cumulative wait (pre-service) time */
	hrtime_t wlentime; /* cumulative wait lentime product /
	hrtime_t wlastupdate; /* last time wait queue changed */
	hrtime_t rtime; /* cumulative run (service) time */
	hrtime_t rlentime; /* cumulative run lengthtime product /
	hrtime_t rlastupdate; /* last time run queue changed */
	uint_t wcnt; /* count of elements in wait state */
	uint_t rcnt; /* count of elements in run state */
	} kstat_io_t;

	typedef struct kstat_timer {
	char name[KSTAT_STRLEN+1]; /* event name */
	u_longlong_t num_events; /* number of events */
	hrtime_t elapsed_time; /* cumulative elapsed time */
	hrtime_t min_time; /* shortest event duration */
	hrtime_t max_time; /* longest event duration */
	hrtime_t start_time; /* previous event start time */
	hrtime_t stop_time; /* previous event stop time */
	} kstat_timer_t;

	int spl_kstat_init(void);
	void spl_kstat_fini(void);

	extern void __kstat_set_raw_ops(kstat_t *ksp,
	int (headers)(char buf, size_t size),
	int (data)(char buf, size_t size, void *data),
	void* (addr)(kstat_t ksp, loff_t index));

	extern kstat_t __kstat_create(const char ks_module, int ks_instance,
	const char ks_name, const char ks_class, uchar_t ks_type,
	uint_t ks_ndata, uchar_t ks_flags);

	extern void kstat_proc_entry_init(kstat_proc_entry_t *kpep,
	const char module, const char name);
	extern void kstat_proc_entry_delete(kstat_proc_entry_t *kpep);
	extern void kstat_proc_entry_install(kstat_proc_entry_t *kpep, mode_t mode,
	const kstat_proc_op_t file_ops, void data);

	extern void __kstat_install(kstat_t *ksp);
	extern void __kstat_delete(kstat_t *ksp);
	extern void kstat_waitq_enter(kstat_io_t *);
	extern void kstat_waitq_exit(kstat_io_t *);
	extern void kstat_runq_enter(kstat_io_t *);
	extern void kstat_runq_exit(kstat_io_t *);

	#define kstat_set_raw_ops(k, h, d, a) \
	__kstat_set_raw_ops(k, h, d, a)
	#define kstat_create(m, i, n, c, t, s, f) \
	__kstat_create(m, i, n, c, t, s, f)

	#define kstat_install(k) __kstat_install(k)
	#define kstat_delete(k) __kstat_delete(k)

	#endif /* _SPL_KSTAT_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/list.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/list.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/list.h (revision 366774)
	@@ -1,210 +1,209 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_LIST_H
	#define _SPL_LIST_H

	#include <sys/types.h>
	#include <sys/debug.h>
	#include <linux/list.h>

	/*
	* NOTE: I have implemented the Solaris list API in terms of the native
	* linux API. This has certain advantages in terms of leveraging the linux
	* list debugging infrastructure, but it also means that the internals of a
	* list differ slightly than on Solaris. This is not a problem as long as
	* all callers stick to the published API. The two major differences are:
	*
	* 1) A list_node_t is mapped to a linux list_head struct which changes
	* the name of the list_next/list_prev pointers to next/prev respectively.
	*
	* 2) A list_node_t which is not attached to a list on Solaris is denoted
	* by having its list_next/list_prev pointers set to NULL. Under linux
	* the next/prev pointers are set to LIST_POISON1 and LIST_POISON2
	* respectively. At this moment this only impacts the implementation
	* of the list_link_init() and list_link_active() functions.
	*/

	typedef struct list_head list_node_t;

	typedef struct list {
	size_t list_size;
	size_t list_offset;
	list_node_t list_head;
	} list_t;

	#define list_d2l(a, obj) ((list_node_t )(((char )obj) + (a)->list_offset))
	#define list_object(a, node) ((void )(((char )node) - (a)->list_offset))

	static inline int
	list_is_empty(list_t *list)
	{
	return (list_empty(&list->list_head));
	}

	static inline void
	list_link_init(list_node_t *node)
	{
	node->next = LIST_POISON1;
	node->prev = LIST_POISON2;
	}

	static inline void
	list_create(list_t *list, size_t size, size_t offset)
	{
	list->list_size = size;
	list->list_offset = offset;
	INIT_LIST_HEAD(&list->list_head);
	}

	static inline void
	list_destroy(list_t *list)
	{
	list_del(&list->list_head);
	}

	static inline void
	list_insert_head(list_t list, void object)
	{
	list_add(list_d2l(list, object), &list->list_head);
	}

	static inline void
	list_insert_tail(list_t list, void object)
	{
	list_add_tail(list_d2l(list, object), &list->list_head);
	}

	static inline void
	list_insert_after(list_t list, void object, void *nobject)
	{
	if (object == NULL)
	list_insert_head(list, nobject);
	else
	list_add(list_d2l(list, nobject), list_d2l(list, object));
	}

	static inline void
	list_insert_before(list_t list, void object, void *nobject)
	{
	if (object == NULL)
	list_insert_tail(list, nobject);
	else
	list_add_tail(list_d2l(list, nobject), list_d2l(list, object));
	}

	static inline void
	list_remove(list_t list, void object)
	{
	list_del(list_d2l(list, object));
	}

	static inline void *
	list_remove_head(list_t *list)
	{
	list_node_t *head = list->list_head.next;
	if (head == &list->list_head)
	return (NULL);

	list_del(head);
	return (list_object(list, head));
	}

	static inline void *
	list_remove_tail(list_t *list)
	{
	list_node_t *tail = list->list_head.prev;
	if (tail == &list->list_head)
	return (NULL);

	list_del(tail);
	return (list_object(list, tail));
	}

	static inline void *
	list_head(list_t *list)
	{
	if (list_is_empty(list))
	return (NULL);

	return (list_object(list, list->list_head.next));
	}

	static inline void *
	list_tail(list_t *list)
	{
	if (list_is_empty(list))
	return (NULL);

	return (list_object(list, list->list_head.prev));
	}

	static inline void *
	list_next(list_t list, void object)
	{
	list_node_t *node = list_d2l(list, object);

	if (node->next != &list->list_head)
	return (list_object(list, node->next));

	return (NULL);
	}

	static inline void *
	list_prev(list_t list, void object)
	{
	list_node_t *node = list_d2l(list, object);

	if (node->prev != &list->list_head)
	return (list_object(list, node->prev));

	return (NULL);
	}

	static inline int
	list_link_active(list_node_t *node)
	{
	EQUIV(node->next == LIST_POISON1, node->prev == LIST_POISON2);
	return (node->next != LIST_POISON1);
	}

	static inline void
	spl_list_move_tail(list_t dst, list_t src)
	{
	list_splice_init(&src->list_head, dst->list_head.prev);
	}

	#define list_move_tail(dst, src) spl_list_move_tail(dst, src)

	static inline void
	list_link_replace(list_node_t old_node, list_node_t new_node)
	{
	new_node->next = old_node->next;
	new_node->prev = old_node->prev;
	old_node->prev->next = new_node;
	old_node->next->prev = new_node;
	list_link_init(old_node);
	}

	#endif /* SPL_LIST_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/mod_os.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/mod_os.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/mod_os.h (revision 366774)
	@@ -1,28 +1,27 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/
	#ifndef _SPL_MOD_H
	#define _SPL_MOD_H
	#include <linux/mod_compat.h>

	#endif /* SPL_MOD_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/mutex.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/mutex.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/mutex.h (revision 366774)
	@@ -1,185 +1,184 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_MUTEX_H
	#define _SPL_MUTEX_H

	#include <sys/types.h>
	#include <linux/sched.h>
	#include <linux/mutex.h>
	#include <linux/lockdep.h>
	#include <linux/compiler_compat.h>

	typedef enum {
	MUTEX_DEFAULT = 0,
	MUTEX_SPIN = 1,
	MUTEX_ADAPTIVE = 2,
	MUTEX_NOLOCKDEP = 3
	} kmutex_type_t;

	typedef struct {
	struct mutex m_mutex;
	spinlock_t m_lock; /* used for serializing mutex_exit */
	kthread_t *m_owner;
	#ifdef CONFIG_LOCKDEP
	kmutex_type_t m_type;
	#endif /* CONFIG_LOCKDEP */
	} kmutex_t;

	#define MUTEX(mp) (&((mp)->m_mutex))

	static inline void
	spl_mutex_set_owner(kmutex_t *mp)
	{
	mp->m_owner = current;
	}

	static inline void
	spl_mutex_clear_owner(kmutex_t *mp)
	{
	mp->m_owner = NULL;
	}

	#define mutex_owner(mp) (READ_ONCE((mp)->m_owner))
	#define mutex_owned(mp) (mutex_owner(mp) == current)
	#define MUTEX_HELD(mp) mutex_owned(mp)
	#define MUTEX_NOT_HELD(mp) (!MUTEX_HELD(mp))

	#ifdef CONFIG_LOCKDEP
	static inline void
	spl_mutex_set_type(kmutex_t *mp, kmutex_type_t type)
	{
	mp->m_type = type;
	}
	static inline void
	spl_mutex_lockdep_off_maybe(kmutex_t *mp) \
	{ \
	if (mp && mp->m_type == MUTEX_NOLOCKDEP) \
	lockdep_off(); \
	}
	static inline void
	spl_mutex_lockdep_on_maybe(kmutex_t *mp) \
	{ \
	if (mp && mp->m_type == MUTEX_NOLOCKDEP) \
	lockdep_on(); \
	}
	#else /* CONFIG_LOCKDEP */
	#define spl_mutex_set_type(mp, type)
	#define spl_mutex_lockdep_off_maybe(mp)
	#define spl_mutex_lockdep_on_maybe(mp)
	#endif /* CONFIG_LOCKDEP */

	/*
	* The following functions must be a #define and not static inline.
	* This ensures that the native linux mutex functions (lock/unlock)
	* will be correctly located in the users code which is important
	* for the built in kernel lock analysis tools
	*/
	#undef mutex_init
	#define mutex_init(mp, name, type, ibc) \
	{ \
	static struct lock_class_key __key; \
	ASSERT(type == MUTEX_DEFAULT \|\| type == MUTEX_NOLOCKDEP); \
	\
	__mutex_init(MUTEX(mp), (name) ? (#name) : (#mp), &__key); \
	spin_lock_init(&(mp)->m_lock); \
	spl_mutex_clear_owner(mp); \
	spl_mutex_set_type(mp, type); \
	}

	#undef mutex_destroy
	#define mutex_destroy(mp) \
	{ \
	VERIFY3P(mutex_owner(mp), ==, NULL); \
	}

	/* BEGIN CSTYLED */
	#define mutex_tryenter(mp) \
	({ \
	int _rc_; \
	\
	spl_mutex_lockdep_off_maybe(mp); \
	if ((_rc_ = mutex_trylock(MUTEX(mp))) == 1) \
	spl_mutex_set_owner(mp); \
	spl_mutex_lockdep_on_maybe(mp); \
	\
	_rc_; \
	})
	/* END CSTYLED */

	#define NESTED_SINGLE 1

	#ifdef CONFIG_DEBUG_LOCK_ALLOC
	#define mutex_enter_nested(mp, subclass) \
	{ \
	ASSERT3P(mutex_owner(mp), !=, current); \
	spl_mutex_lockdep_off_maybe(mp); \
	mutex_lock_nested(MUTEX(mp), (subclass)); \
	spl_mutex_lockdep_on_maybe(mp); \
	spl_mutex_set_owner(mp); \
	}
	#else /* CONFIG_DEBUG_LOCK_ALLOC */
	#define mutex_enter_nested(mp, subclass) \
	{ \
	ASSERT3P(mutex_owner(mp), !=, current); \
	spl_mutex_lockdep_off_maybe(mp); \
	mutex_lock(MUTEX(mp)); \
	spl_mutex_lockdep_on_maybe(mp); \
	spl_mutex_set_owner(mp); \
	}
	#endif /* CONFIG_DEBUG_LOCK_ALLOC */

	#define mutex_enter(mp) mutex_enter_nested((mp), 0)

	/*
	* The reason for the spinlock:
	*
	* The Linux mutex is designed with a fast-path/slow-path design such that it
	* does not guarantee serialization upon itself, allowing a race where latter
	* acquirers finish mutex_unlock before former ones.
	*
	* The race renders it unsafe to be used for serializing the freeing of an
	* object in which the mutex is embedded, where the latter acquirer could go
	* on to free the object while the former one is still doing mutex_unlock and
	* causing memory corruption.
	*
	* However, there are many places in ZFS where the mutex is used for
	* serializing object freeing, and the code is shared among other OSes without
	* this issue. Thus, we need the spinlock to force the serialization on
	* mutex_exit().
	*
	* See http://lwn.net/Articles/575477/ for the information about the race.
	*/
	#define mutex_exit(mp) \
	{ \
	spl_mutex_clear_owner(mp); \
	spin_lock(&(mp)->m_lock); \
	spl_mutex_lockdep_off_maybe(mp); \
	mutex_unlock(MUTEX(mp)); \
	spl_mutex_lockdep_on_maybe(mp); \
	spin_unlock(&(mp)->m_lock); \
	/* NOTE: do not dereference mp after this point */ \
	}

	#endif /* _SPL_MUTEX_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/param.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/param.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/param.h (revision 366774)
	@@ -1,36 +1,35 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_PARAM_H
	#define _SPL_PARAM_H

	#include <asm/page.h>

	/* Pages to bytes and back */
	#define ptob(pages) ((pages) << PAGE_SHIFT)
	#define btop(bytes) ((bytes) >> PAGE_SHIFT)

	#define MAXUID UINT32_MAX

	#endif /* SPL_PARAM_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/proc.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/proc.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/proc.h (revision 366774)
	@@ -1,42 +1,41 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_PROC_H
	#define _SPL_PROC_H

	#include <linux/proc_fs.h>
	#include <linux/sched.h>

	extern struct proc_dir_entry *proc_spl_kstat;

	int spl_proc_init(void);
	void spl_proc_fini(void);

	static inline boolean_t
	zfs_proc_is_caller(struct task_struct *t)
	{
	return (t->group_leader == current->group_leader);
	}

	#endif /* SPL_PROC_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/processor.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/processor.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/processor.h (revision 366774)
	@@ -1,32 +1,31 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_PROCESSOR_H
	#define _SPL_PROCESSOR_H

	#define getcpuid() smp_processor_id()

	typedef int processorid_t;

	#endif /* _SPL_PROCESSOR_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/random.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/random.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/random.h (revision 366774)
	@@ -1,40 +1,39 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_RANDOM_H
	#define _SPL_RANDOM_H

	#include <linux/module.h>
	#include <linux/random.h>

	static __inline__ int
	random_get_bytes(uint8_t *ptr, size_t len)
	{
	get_random_bytes((void *)ptr, (int)len);
	return (0);
	}

	extern int random_get_pseudo_bytes(uint8_t *ptr, size_t len);

	#endif /* _SPL_RANDOM_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/rwlock.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/rwlock.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/rwlock.h (revision 366774)
	@@ -1,201 +1,200 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_RWLOCK_H
	#define _SPL_RWLOCK_H

	#include <sys/types.h>
	#include <linux/rwsem.h>
	#include <linux/sched.h>

	typedef enum {
	RW_DRIVER = 2,
	RW_DEFAULT = 4,
	RW_NOLOCKDEP = 5
	} krw_type_t;

	typedef enum {
	RW_NONE = 0,
	RW_WRITER = 1,
	RW_READER = 2
	} krw_t;

	typedef struct {
	struct rw_semaphore rw_rwlock;
	kthread_t *rw_owner;
	#ifdef CONFIG_LOCKDEP
	krw_type_t rw_type;
	#endif /* CONFIG_LOCKDEP */
	} krwlock_t;

	#define SEM(rwp) (&(rwp)->rw_rwlock)

	static inline void
	spl_rw_set_owner(krwlock_t *rwp)
	{
	rwp->rw_owner = current;
	}

	static inline void
	spl_rw_clear_owner(krwlock_t *rwp)
	{
	rwp->rw_owner = NULL;
	}

	static inline kthread_t *
	rw_owner(krwlock_t *rwp)
	{
	return (rwp->rw_owner);
	}

	#ifdef CONFIG_LOCKDEP
	static inline void
	spl_rw_set_type(krwlock_t *rwp, krw_type_t type)
	{
	rwp->rw_type = type;
	}
	static inline void
	spl_rw_lockdep_off_maybe(krwlock_t *rwp) \
	{ \
	if (rwp && rwp->rw_type == RW_NOLOCKDEP) \
	lockdep_off(); \
	}
	static inline void
	spl_rw_lockdep_on_maybe(krwlock_t *rwp) \
	{ \
	if (rwp && rwp->rw_type == RW_NOLOCKDEP) \
	lockdep_on(); \
	}
	#else /* CONFIG_LOCKDEP */
	#define spl_rw_set_type(rwp, type)
	#define spl_rw_lockdep_off_maybe(rwp)
	#define spl_rw_lockdep_on_maybe(rwp)
	#endif /* CONFIG_LOCKDEP */

	static inline int
	RW_LOCK_HELD(krwlock_t *rwp)
	{
	return (rwsem_is_locked(SEM(rwp)));
	}

	static inline int
	RW_WRITE_HELD(krwlock_t *rwp)
	{
	return (rw_owner(rwp) == current);
	}

	static inline int
	RW_READ_HELD(krwlock_t *rwp)
	{
	return (RW_LOCK_HELD(rwp) && rw_owner(rwp) == NULL);
	}

	/*
	* The following functions must be a #define and not static inline.
	* This ensures that the native linux semaphore functions (down/up)
	* will be correctly located in the users code which is important
	* for the built in kernel lock analysis tools
	*/
	/* BEGIN CSTYLED */
	#define rw_init(rwp, name, type, arg) \
	({ \
	static struct lock_class_key __key; \
	ASSERT(type == RW_DEFAULT \|\| type == RW_NOLOCKDEP); \
	\
	__init_rwsem(SEM(rwp), #rwp, &__key); \
	spl_rw_clear_owner(rwp); \
	spl_rw_set_type(rwp, type); \
	})

	/*
	* The Linux rwsem implementation does not require a matching destroy.
	*/
	#define rw_destroy(rwp) ((void) 0)

	/*
	* Upgrading a rwsem from a reader to a writer is not supported by the
	* Linux kernel. The lock must be dropped and reacquired as a writer.
	*/
	#define rw_tryupgrade(rwp) RW_WRITE_HELD(rwp)

	#define rw_tryenter(rwp, rw) \
	({ \
	int _rc_ = 0; \
	\
	spl_rw_lockdep_off_maybe(rwp); \
	switch (rw) { \
	case RW_READER: \
	_rc_ = down_read_trylock(SEM(rwp)); \
	break; \
	case RW_WRITER: \
	if ((_rc_ = down_write_trylock(SEM(rwp)))) \
	spl_rw_set_owner(rwp); \
	break; \
	default: \
	VERIFY(0); \
	} \
	spl_rw_lockdep_on_maybe(rwp); \
	_rc_; \
	})

	#define rw_enter(rwp, rw) \
	({ \
	spl_rw_lockdep_off_maybe(rwp); \
	switch (rw) { \
	case RW_READER: \
	down_read(SEM(rwp)); \
	break; \
	case RW_WRITER: \
	down_write(SEM(rwp)); \
	spl_rw_set_owner(rwp); \
	break; \
	default: \
	VERIFY(0); \
	} \
	spl_rw_lockdep_on_maybe(rwp); \
	})

	#define rw_exit(rwp) \
	({ \
	spl_rw_lockdep_off_maybe(rwp); \
	if (RW_WRITE_HELD(rwp)) { \
	spl_rw_clear_owner(rwp); \
	up_write(SEM(rwp)); \
	} else { \
	ASSERT(RW_READ_HELD(rwp)); \
	up_read(SEM(rwp)); \
	} \
	spl_rw_lockdep_on_maybe(rwp); \
	})

	#define rw_downgrade(rwp) \
	({ \
	spl_rw_lockdep_off_maybe(rwp); \
	spl_rw_clear_owner(rwp); \
	downgrade_write(SEM(rwp)); \
	spl_rw_lockdep_on_maybe(rwp); \
	})
	/* END CSTYLED */

	#endif /* _SPL_RWLOCK_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/shrinker.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/shrinker.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/shrinker.h (revision 366774)
	@@ -1,110 +1,109 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_SHRINKER_H
	#define _SPL_SHRINKER_H

	#include <linux/mm.h>
	#include <linux/fs.h>

	/*
	* Due to frequent changes in the shrinker API the following
	* compatibility wrappers should be used. They are as follows:
	*
	* SPL_SHRINKER_DECLARE(varname, countfunc, scanfunc, seek_cost);
	*
	* SPL_SHRINKER_DECLARE is used to declare a shrinker with the name varname,
	* which is passed to spl_register_shrinker()/spl_unregister_shrinker().
	* The countfunc returns the number of free-able objects.
	* The scanfunc returns the number of objects that were freed.
	* The callbacks can return SHRINK_STOP if further calls can't make any more
	* progress. Note that a return value of SHRINK_EMPTY is currently not
	* supported.
	*
	* Example:
	*
	* static unsigned long
	* my_count(struct shrinker shrink, struct shrink_control sc)
	* {
	* ...calculate number of objects in the cache...
	*
	* return (number of objects in the cache);
	* }
	*
	* static unsigned long
	* my_scan(struct shrinker shrink, struct shrink_control sc)
	* {
	* ...scan objects in the cache and reclaim them...
	* }
	*
	* SPL_SHRINKER_DECLARE(my_shrinker, my_count, my_scan, DEFAULT_SEEKS);
	*
	* void my_init_func(void) {
	* spl_register_shrinker(&my_shrinker);
	* }
	*/

	#define spl_register_shrinker(x) register_shrinker(x)
	#define spl_unregister_shrinker(x) unregister_shrinker(x)

	/*
	* Linux 3.0 to 3.11 Shrinker API Compatibility.
	*/
	#if defined(HAVE_SINGLE_SHRINKER_CALLBACK)
	#define SPL_SHRINKER_DECLARE(varname, countfunc, scanfunc, seek_cost) \
	static int \
	__ ## varname ## _wrapper(struct shrinker shrink, struct shrink_control sc)\
	{ \
	if (sc->nr_to_scan != 0) { \
	(void) scanfunc(shrink, sc); \
	} \
	return (countfunc(shrink, sc)); \
	} \
	\
	static struct shrinker varname = { \
	.shrink = __ ## varname ## _wrapper, \
	.seeks = seek_cost, \
	}

	#define SHRINK_STOP (-1)

	/*
	* Linux 3.12 and later Shrinker API Compatibility.
	*/
	#elif defined(HAVE_SPLIT_SHRINKER_CALLBACK)
	#define SPL_SHRINKER_DECLARE(varname, countfunc, scanfunc, seek_cost) \
	static struct shrinker varname = { \
	.count_objects = countfunc, \
	.scan_objects = scanfunc, \
	.seeks = seek_cost, \
	}

	#else
	/*
	* Linux 2.x to 2.6.22, or a newer shrinker API has been introduced.
	*/
	#error "Unknown shrinker callback"
	#endif

	#endif /* SPL_SHRINKER_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/sid.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/sid.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/sid.h (revision 366774)
	@@ -1,61 +1,60 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_SID_H
	#define _SPL_SID_H

	typedef struct ksiddomain {
	char *kd_name;
	} ksiddomain_t;

	typedef enum ksid_index {
	KSID_USER,
	KSID_GROUP,
	KSID_OWNER,
	KSID_COUNT
	} ksid_index_t;

	typedef int ksid_t;

	static inline ksiddomain_t *
	ksid_lookupdomain(const char *dom)
	{
	ksiddomain_t *kd;
	int len = strlen(dom);

	kd = kmem_zalloc(sizeof (ksiddomain_t), KM_SLEEP);
	kd->kd_name = kmem_zalloc(len + 1, KM_SLEEP);
	memcpy(kd->kd_name, dom, len);

	return (kd);
	}

	static inline void
	ksiddomain_rele(ksiddomain_t *ksid)
	{
	kmem_free(ksid->kd_name, strlen(ksid->kd_name) + 1);
	kmem_free(ksid, sizeof (ksiddomain_t));
	}

	#endif /* _SPL_SID_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/signal.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/signal.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/signal.h (revision 366774)
	@@ -1,55 +1,54 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_SIGNAL_H
	#define _SPL_SIGNAL_H

	#include <linux/sched.h>

	#ifdef HAVE_SCHED_SIGNAL_HEADER
	#include <linux/sched/signal.h>
	#endif

	#define FORREAL 0 /* Usual side-effects */
	#define JUSTLOOKING 1 /* Don't stop the process */

	/*
	* The "why" argument indicates the allowable side-effects of the call:
	*
	* FORREAL: Extract the next pending signal from p_sig into p_cursig;
	* stop the process if a stop has been requested or if a traced signal
	* is pending.
	*
	* JUSTLOOKING: Don't stop the process, just indicate whether or not
	* a signal might be pending (FORREAL is needed to tell for sure).
	*/
	static __inline__ int
	issig(int why)
	{
	ASSERT(why == FORREAL \|\| why == JUSTLOOKING);

	return (signal_pending(current));
	}

	#endif /* SPL_SIGNAL_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/simd.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/simd.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/simd.h (revision 366774)
	@@ -1,31 +1,30 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_SYS_SIMD_H
	#define _SPL_SYS_SIMD_H

	#include <sys/isa_defs.h>
	#include <linux/simd.h>

	#endif /* _SPL_SYS_SIMD_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/stat.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/stat.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/stat.h (revision 366774)
	@@ -1,30 +1,29 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_STAT_H
	#define _SPL_STAT_H

	#include <linux/stat.h>

	#endif /* SPL_STAT_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/strings.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/strings.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/strings.h (revision 366774)
	@@ -1,31 +1,30 @@
	/*
	* Copyright (C) 2018 Lawrence Livermore National Security, LLC.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/
	#ifndef _SPL_SYS_STRINGS_H
	#define _SPL_SYS_STRINGS_H

	#include <linux/string.h>

	#define bzero(ptr, size) memset(ptr, 0, size)
	#define bcopy(src, dest, size) memmove(dest, src, size)
	#define bcmp(src, dest, size) memcmp((src), (dest), (size_t)(size))

	#endif /* _SPL_SYS_STRINGS_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/sunddi.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/sunddi.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/sunddi.h (revision 366774)
	@@ -1,58 +1,57 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_SUNDDI_H
	#define _SPL_SUNDDI_H

	#include <sys/cred.h>
	#include <sys/uio.h>
	#include <sys/mutex.h>
	#include <sys/u8_textprep.h>
	#include <sys/vnode.h>

	typedef int ddi_devid_t;

	#define DDI_DEV_T_NONE ((dev_t)-1)
	#define DDI_DEV_T_ANY ((dev_t)-2)
	#define DI_MAJOR_T_UNKNOWN ((major_t)0)

	#define DDI_PROP_DONTPASS 0x0001
	#define DDI_PROP_CANSLEEP 0x0002

	#define DDI_SUCCESS 0
	#define DDI_FAILURE -1

	#define ddi_prop_lookup_string(x1, x2, x3, x4, x5) (*x5 = NULL)
	#define ddi_prop_free(x) (void)0
	#define ddi_root_node() (void)0

	extern int ddi_strtoul(const char , char , int, unsigned long );
	extern int ddi_strtol(const char , char , int, long );
	extern int ddi_strtoull(const char , char , int, unsigned long long );
	extern int ddi_strtoll(const char , char , int, long long );

	extern int ddi_copyin(const void from, void to, size_t len, int flags);
	extern int ddi_copyout(const void from, void to, size_t len, int flags);

	#endif /* SPL_SUNDDI_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/sysmacros.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/sysmacros.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/sysmacros.h (revision 366774)
	@@ -1,206 +1,205 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_SYSMACROS_H
	#define _SPL_SYSMACROS_H

	#include <linux/module.h>
	#include <linux/sched.h>
	#include <linux/sched/rt.h>
	#include <linux/cpumask.h>
	#include <sys/debug.h>
	#include <sys/zone.h>
	#include <sys/signal.h>
	#include <asm/page.h>


	#ifndef _KERNEL
	#define _KERNEL __KERNEL__
	#endif

	#define FALSE 0
	#define TRUE 1

	#define INT8_MAX (127)
	#define INT8_MIN (-128)
	#define UINT8_MAX (255)
	#define UINT8_MIN (0)

	#define INT16_MAX (32767)
	#define INT16_MIN (-32768)
	#define UINT16_MAX (65535)
	#define UINT16_MIN (0)

	#define INT32_MAX INT_MAX
	#define INT32_MIN INT_MIN
	#define UINT32_MAX UINT_MAX
	#define UINT32_MIN UINT_MIN

	#define INT64_MAX LLONG_MAX
	#define INT64_MIN LLONG_MIN
	#define UINT64_MAX ULLONG_MAX
	#define UINT64_MIN ULLONG_MIN

	#define NBBY 8

	#define MAXMSGLEN 256
	#define MAXNAMELEN 256
	#define MAXPATHLEN 4096
	#define MAXOFFSET_T LLONG_MAX
	#define MAXBSIZE 8192
	#define DEV_BSIZE 512
	#define DEV_BSHIFT 9 /* log2(DEV_BSIZE) */

	#define proc_pageout NULL
	#define curproc current
	#define max_ncpus num_possible_cpus()
	#define boot_ncpus num_online_cpus()
	#define CPU_SEQID smp_processor_id()
	#define is_system_labeled() 0

	#ifndef RLIM64_INFINITY
	#define RLIM64_INFINITY (~0ULL)
	#endif

	/*
	* 0..MAX_PRIO-1: Process priority
	* 0..MAX_RT_PRIO-1: RT priority tasks
	* MAX_RT_PRIO..MAX_PRIO-1: SCHED_NORMAL tasks
	*
	* Treat shim tasks as SCHED_NORMAL tasks
	*/
	#define minclsyspri (MAX_PRIO-1)
	#define maxclsyspri (MAX_RT_PRIO)
	#define defclsyspri (DEFAULT_PRIO)

	#ifndef NICE_TO_PRIO
	#define NICE_TO_PRIO(nice) (MAX_RT_PRIO + (nice) + 20)
	#endif
	#ifndef PRIO_TO_NICE
	#define PRIO_TO_NICE(prio) ((prio) - MAX_RT_PRIO - 20)
	#endif

	/*
	* Missing macros
	*/
	#ifndef PAGESIZE
	#define PAGESIZE PAGE_SIZE
	#endif

	#ifndef PAGESHIFT
	#define PAGESHIFT PAGE_SHIFT
	#endif

	/* Missing globals */
	extern char spl_gitrev[64];
	extern unsigned long spl_hostid;

	/* Missing misc functions */
	extern uint32_t zone_get_hostid(void *zone);
	extern void spl_setup(void);
	extern void spl_cleanup(void);

	#define highbit(x) __fls(x)
	#define lowbit(x) __ffs(x)

	#define highbit64(x) fls64(x)
	#define makedevice(maj, min) makedev(maj, min)

	/* common macros */
	#ifndef MIN
	#define MIN(a, b) ((a) < (b) ? (a) : (b))
	#endif
	#ifndef MAX
	#define MAX(a, b) ((a) < (b) ? (b) : (a))
	#endif
	#ifndef ABS
	#define ABS(a) ((a) < 0 ? -(a) : (a))
	#endif
	#ifndef DIV_ROUND_UP
	#define DIV_ROUND_UP(n, d) (((n) + (d) - 1) / (d))
	#endif
	#ifndef roundup
	#define roundup(x, y) ((((x) + ((y) - 1)) / (y)) * (y))
	#endif
	#ifndef howmany
	#define howmany(x, y) (((x) + ((y) - 1)) / (y))
	#endif

	/*
	* Compatibility macros/typedefs needed for Solaris -> Linux port
	*/
	#define P2ALIGN(x, align) ((x) & -(align))
	#define P2CROSS(x, y, align) (((x) ^ (y)) > (align) - 1)
	#define P2ROUNDUP(x, align) ((((x) - 1) \| ((align) - 1)) + 1)
	#define P2PHASE(x, align) ((x) & ((align) - 1))
	#define P2NPHASE(x, align) (-(x) & ((align) - 1))
	#define ISP2(x) (((x) & ((x) - 1)) == 0)
	#define IS_P2ALIGNED(v, a) ((((uintptr_t)(v)) & ((uintptr_t)(a) - 1)) == 0)
	#define P2BOUNDARY(off, len, align) \
	(((off) ^ ((off) + (len) - 1)) > (align) - 1)

	/*
	* Typed version of the P2* macros. These macros should be used to ensure
	* that the result is correctly calculated based on the data type of (x),
	* which is passed in as the last argument, regardless of the data
	* type of the alignment. For example, if (x) is of type uint64_t,
	* and we want to round it up to a page boundary using "PAGESIZE" as
	* the alignment, we can do either
	*
	* P2ROUNDUP(x, (uint64_t)PAGESIZE)
	* or
	* P2ROUNDUP_TYPED(x, PAGESIZE, uint64_t)
	*/
	#define P2ALIGN_TYPED(x, align, type) \
	((type)(x) & -(type)(align))
	#define P2PHASE_TYPED(x, align, type) \
	((type)(x) & ((type)(align) - 1))
	#define P2NPHASE_TYPED(x, align, type) \
	(-(type)(x) & ((type)(align) - 1))
	#define P2ROUNDUP_TYPED(x, align, type) \
	((((type)(x) - 1) \| ((type)(align) - 1)) + 1)
	#define P2END_TYPED(x, align, type) \
	(-(~(type)(x) & -(type)(align)))
	#define P2PHASEUP_TYPED(x, align, phase, type) \
	((type)(phase) - (((type)(phase) - (type)(x)) & -(type)(align)))
	#define P2CROSS_TYPED(x, y, align, type) \
	(((type)(x) ^ (type)(y)) > (type)(align) - 1)
	#define P2SAMEHIGHBIT_TYPED(x, y, type) \
	(((type)(x) ^ (type)(y)) < ((type)(x) & (type)(y)))

	#define SET_ERROR(err) \
	(__set_error(__FILE__, __func__, __LINE__, err), err)

	#include <linux/sort.h>
	#define qsort(base, num, size, cmp) \
	sort(base, num, size, cmp, NULL)

	#if !defined(_KMEMUSER) && !defined(offsetof)

	/* avoid any possibility of clashing with <stddef.h> version */

	#define offsetof(s, m) ((size_t)(&(((s *)0)->m)))
	#endif

	#endif /* _SPL_SYSMACROS_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/systeminfo.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/systeminfo.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/systeminfo.h (revision 366774)
	@@ -1,36 +1,35 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_SYSTEMINFO_H
	#define _SPL_SYSTEMINFO_H

	#define HW_HOSTID_LEN 11 /* minimum buffer size needed */
	/* to hold a decimal or hex */
	/* hostid string */

	/* Supplemental definitions for Linux. */
	#define HW_HOSTID_PATH "/etc/hostid" /* binary configuration file */
	#define HW_HOSTID_MASK 0xFFFFFFFF /* significant hostid bits */

	#endif /* SPL_SYSTEMINFO_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/taskq.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/taskq.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/taskq.h (revision 366774)
	@@ -1,164 +1,163 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_TASKQ_H
	#define _SPL_TASKQ_H

	#include <linux/module.h>
	#include <linux/gfp.h>
	#include <linux/slab.h>
	#include <linux/interrupt.h>
	#include <linux/kthread.h>
	#include <sys/types.h>
	#include <sys/thread.h>
	#include <sys/rwlock.h>
	#include <sys/wait.h>

	#define TASKQ_NAMELEN 31

	#define TASKQ_PREPOPULATE 0x00000001
	#define TASKQ_CPR_SAFE 0x00000002
	#define TASKQ_DYNAMIC 0x00000004
	#define TASKQ_THREADS_CPU_PCT 0x00000008
	#define TASKQ_DC_BATCH 0x00000010
	#define TASKQ_ACTIVE 0x80000000

	/*
	* Flags for taskq_dispatch. TQ_SLEEP/TQ_NOSLEEP should be same as
	* KM_SLEEP/KM_NOSLEEP. TQ_NOQUEUE/TQ_NOALLOC are set particularly
	* large so as not to conflict with already used GFP_* defines.
	*/
	#define TQ_SLEEP 0x00000000
	#define TQ_NOSLEEP 0x00000001
	#define TQ_PUSHPAGE 0x00000002
	#define TQ_NOQUEUE 0x01000000
	#define TQ_NOALLOC 0x02000000
	#define TQ_NEW 0x04000000
	#define TQ_FRONT 0x08000000

	/*
	* Reserved taskqid values.
	*/
	#define TASKQID_INVALID ((taskqid_t)0)
	#define TASKQID_INITIAL ((taskqid_t)1)

	/*
	* spin_lock(lock) and spin_lock_nested(lock,0) are equivalent,
	* so TQ_LOCK_DYNAMIC must not evaluate to 0
	*/
	typedef enum tq_lock_role {
	TQ_LOCK_GENERAL = 0,
	TQ_LOCK_DYNAMIC = 1,
	} tq_lock_role_t;

	typedef unsigned long taskqid_t;
	typedef void (task_func_t)(void *);

	typedef struct taskq {
	spinlock_t tq_lock; /* protects taskq_t */
	char tq_name; / taskq name */
	int tq_instance; /* instance of tq_name */
	struct list_head tq_thread_list; /* list of all threads */
	struct list_head tq_active_list; /* list of active threads */
	int tq_nactive; /* # of active threads */
	int tq_nthreads; /* # of existing threads */
	int tq_nspawn; /* # of threads being spawned */
	int tq_maxthreads; /* # of threads maximum */
	int tq_pri; /* priority */
	int tq_minalloc; /* min taskq_ent_t pool size */
	int tq_maxalloc; /* max taskq_ent_t pool size */
	int tq_nalloc; /* cur taskq_ent_t pool size */
	uint_t tq_flags; /* flags */
	taskqid_t tq_next_id; /* next pend/work id */
	taskqid_t tq_lowest_id; /* lowest pend/work id */
	struct list_head tq_free_list; /* free taskq_ent_t's */
	struct list_head tq_pend_list; /* pending taskq_ent_t's */
	struct list_head tq_prio_list; /* priority taskq_ent_t's */
	struct list_head tq_delay_list; /* delayed taskq_ent_t's */
	struct list_head tq_taskqs; /* all taskq_t's */
	spl_wait_queue_head_t tq_work_waitq; /* new work waitq */
	spl_wait_queue_head_t tq_wait_waitq; /* wait waitq */
	tq_lock_role_t tq_lock_class; /* class when taking tq_lock */
	} taskq_t;

	typedef struct taskq_ent {
	spinlock_t tqent_lock;
	spl_wait_queue_head_t tqent_waitq;
	struct timer_list tqent_timer;
	struct list_head tqent_list;
	taskqid_t tqent_id;
	task_func_t *tqent_func;
	void *tqent_arg;
	taskq_t *tqent_taskq;
	uintptr_t tqent_flags;
	unsigned long tqent_birth;
	} taskq_ent_t;

	#define TQENT_FLAG_PREALLOC 0x1
	#define TQENT_FLAG_CANCEL 0x2

	typedef struct taskq_thread {
	struct list_head tqt_thread_list;
	struct list_head tqt_active_list;
	struct task_struct *tqt_thread;
	taskq_t *tqt_tq;
	taskqid_t tqt_id;
	taskq_ent_t *tqt_task;
	uintptr_t tqt_flags;
	} taskq_thread_t;

	/* Global system-wide dynamic task queue available for all consumers */
	extern taskq_t *system_taskq;
	/* Global dynamic task queue for long delay */
	extern taskq_t *system_delay_taskq;

	/* List of all taskqs */
	extern struct list_head tq_list;
	extern struct rw_semaphore tq_list_sem;

	extern taskqid_t taskq_dispatch(taskq_t , task_func_t, void , uint_t);
	extern taskqid_t taskq_dispatch_delay(taskq_t , task_func_t, void ,
	uint_t, clock_t);
	extern void taskq_dispatch_ent(taskq_t , task_func_t, void , uint_t,
	taskq_ent_t *);
	extern int taskq_empty_ent(taskq_ent_t *);
	extern void taskq_init_ent(taskq_ent_t *);
	extern taskq_t taskq_create(const char , int, pri_t, int, int, uint_t);
	extern void taskq_destroy(taskq_t *);
	extern void taskq_wait_id(taskq_t *, taskqid_t);
	extern void taskq_wait_outstanding(taskq_t *, taskqid_t);
	extern void taskq_wait(taskq_t *);
	extern int taskq_cancel_id(taskq_t *, taskqid_t);
	extern int taskq_member(taskq_t , kthread_t );
	extern taskq_t *taskq_of_curthread(void);

	#define taskq_create_proc(name, nthreads, pri, min, max, proc, flags) \
	taskq_create(name, nthreads, pri, min, max, flags)
	#define taskq_create_sysdc(name, nthreads, min, max, proc, dc, flags) \
	taskq_create(name, nthreads, maxclsyspri, min, max, flags)

	int spl_taskq_init(void);
	void spl_taskq_fini(void);

	#endif /* _SPL_TASKQ_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/thread.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/thread.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/thread.h (revision 366774)
	@@ -1,74 +1,73 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_THREAD_H
	#define _SPL_THREAD_H

	#include <linux/module.h>
	#include <linux/mm.h>
	#include <linux/spinlock.h>
	#include <linux/kthread.h>
	#include <sys/types.h>
	#include <sys/sysmacros.h>
	#include <sys/tsd.h>

	/*
	* Thread interfaces
	*/
	#define TP_MAGIC 0x53535353

	#define TS_SLEEP TASK_INTERRUPTIBLE
	#define TS_RUN TASK_RUNNING
	#define TS_ZOMB EXIT_ZOMBIE
	#define TS_STOPPED TASK_STOPPED

	typedef void (thread_func_t)(void );

	#define thread_create_named(name, stk, stksize, func, arg, len, \
	pp, state, pri) \
	__thread_create(stk, stksize, (thread_func_t)func, \
	name, arg, len, pp, state, pri)

	/* BEGIN CSTYLED */
	#define thread_create(stk, stksize, func, arg, len, pp, state, pri) \
	__thread_create(stk, stksize, (thread_func_t)func, \
	#func, arg, len, pp, state, pri)
	/* END CSTYLED */

	#define thread_exit() __thread_exit()
	#define thread_join(t) VERIFY(0)
	#define curthread current
	#define getcomm() current->comm
	#define getpid() current->pid

	extern kthread_t *__thread_create(caddr_t stk, size_t stksize,
	thread_func_t func, const char name, void args, size_t len, proc_t *pp,
	int state, pri_t pri);
	extern void __thread_exit(void);
	extern struct task_struct spl_kthread_create(int (func)(void *),
	void *data, const char namefmt[], ...);

	extern proc_t p0;

	#endif /* _SPL_THREAD_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/time.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/time.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/time.h (revision 366774)
	@@ -1,118 +1,117 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_TIME_H
	#define _SPL_TIME_H

	#include <linux/module.h>
	#include <linux/time.h>
	#include <sys/types.h>
	#include <sys/timer.h>

	#if defined(CONFIG_64BIT)
	#define TIME_MAX INT64_MAX
	#define TIME_MIN INT64_MIN
	#else
	#define TIME_MAX INT32_MAX
	#define TIME_MIN INT32_MIN
	#endif

	#define SEC 1
	#define MILLISEC 1000
	#define MICROSEC 1000000
	#define NANOSEC 1000000000

	#define MSEC2NSEC(m) ((hrtime_t)(m) * (NANOSEC / MILLISEC))
	#define NSEC2MSEC(n) ((n) / (NANOSEC / MILLISEC))

	#define USEC2NSEC(m) ((hrtime_t)(m) * (NANOSEC / MICROSEC))
	#define NSEC2USEC(n) ((n) / (NANOSEC / MICROSEC))

	#define NSEC2SEC(n) ((n) / (NANOSEC / SEC))
	#define SEC2NSEC(m) ((hrtime_t)(m) * (NANOSEC / SEC))

	static const int hz = HZ;

	typedef longlong_t hrtime_t;
	typedef struct timespec timespec_t;

	#define TIMESPEC_OVERFLOW(ts) \
	((ts)->tv_sec < TIME_MIN \|\| (ts)->tv_sec > TIME_MAX)

	#if defined(HAVE_INODE_TIMESPEC64_TIMES)
	typedef struct timespec64 inode_timespec_t;
	#else
	typedef struct timespec inode_timespec_t;
	#endif

	/* Include for Lustre compatibility */
	#define timestruc_t inode_timespec_t

	static inline void
	gethrestime(inode_timespec_t *ts)
	{
	#if defined(HAVE_INODE_TIMESPEC64_TIMES)

	#if defined(HAVE_KTIME_GET_COARSE_REAL_TS64)
	ktime_get_coarse_real_ts64(ts);
	#else
	*ts = current_kernel_time64();
	#endif /* HAVE_KTIME_GET_COARSE_REAL_TS64 */

	#else
	*ts = current_kernel_time();
	#endif
	}

	static inline uint64_t
	gethrestime_sec(void)
	{
	#if defined(HAVE_INODE_TIMESPEC64_TIMES)
	#if defined(HAVE_KTIME_GET_COARSE_REAL_TS64)
	inode_timespec_t ts;
	ktime_get_coarse_real_ts64(&ts);
	#else
	inode_timespec_t ts = current_kernel_time64();
	#endif /* HAVE_KTIME_GET_COARSE_REAL_TS64 */

	#else
	inode_timespec_t ts = current_kernel_time();
	#endif
	return (ts.tv_sec);
	}

	static inline hrtime_t
	gethrtime(void)
	{
	#if defined(HAVE_KTIME_GET_RAW_TS64)
	struct timespec64 ts;
	ktime_get_raw_ts64(&ts);
	#else
	struct timespec ts;
	getrawmonotonic(&ts);
	#endif
	return (((hrtime_t)ts.tv_sec * NSEC_PER_SEC) + ts.tv_nsec);
	}

	#endif /* _SPL_TIME_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/timer.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/timer.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/timer.h (revision 366774)
	@@ -1,86 +1,85 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_TIMER_H
	#define _SPL_TIMER_H

	#include <linux/module.h>
	#include <linux/delay.h>
	#include <linux/sched.h>
	#include <linux/time.h>
	#include <linux/timer.h>

	#define lbolt ((clock_t)jiffies)
	#define lbolt64 ((int64_t)get_jiffies_64())

	#define ddi_get_lbolt() ((clock_t)jiffies)
	#define ddi_get_lbolt64() ((int64_t)get_jiffies_64())

	#define ddi_time_before(a, b) (typecheck(clock_t, a) && \
	typecheck(clock_t, b) && \
	((a) - (b) < 0))
	#define ddi_time_after(a, b) ddi_time_before(b, a)
	#define ddi_time_before_eq(a, b) (!ddi_time_after(a, b))
	#define ddi_time_after_eq(a, b) ddi_time_before_eq(b, a)

	#define ddi_time_before64(a, b) (typecheck(int64_t, a) && \
	typecheck(int64_t, b) && \
	((a) - (b) < 0))
	#define ddi_time_after64(a, b) ddi_time_before64(b, a)
	#define ddi_time_before_eq64(a, b) (!ddi_time_after64(a, b))
	#define ddi_time_after_eq64(a, b) ddi_time_before_eq64(b, a)

	#define delay(ticks) schedule_timeout_uninterruptible(ticks)

	#define SEC_TO_TICK(sec) ((sec) * HZ)
	#define MSEC_TO_TICK(ms) msecs_to_jiffies(ms)
	#define USEC_TO_TICK(us) usecs_to_jiffies(us)
	#define NSEC_TO_TICK(ns) usecs_to_jiffies(ns / NSEC_PER_USEC)

	#ifndef from_timer
	#define from_timer(var, timer, timer_field) \
	container_of(timer, typeof(*var), timer_field)
	#endif

	#ifdef HAVE_KERNEL_TIMER_FUNCTION_TIMER_LIST
	typedef struct timer_list *spl_timer_list_t;
	#else
	typedef unsigned long spl_timer_list_t;
	#endif

	#ifndef HAVE_KERNEL_TIMER_SETUP

	static inline void
	timer_setup(struct timer_list timer, void (func)(spl_timer_list_t), u32 fl)
	{
	#ifdef HAVE_KERNEL_TIMER_LIST_FLAGS
	(timer)->flags = fl;
	#endif
	init_timer(timer);
	setup_timer(timer, func, (spl_timer_list_t)(timer));
	}

	#endif /* HAVE_KERNEL_TIMER_SETUP */

	#endif /* _SPL_TIMER_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/tsd.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/tsd.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/tsd.h (revision 366774)
	@@ -1,46 +1,45 @@
	/*
	* Copyright (C) 2010 Lawrence Livermore National Security, LLC.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_TSD_H
	#define _SPL_TSD_H

	#include <sys/types.h>

	#define TSD_HASH_TABLE_BITS_DEFAULT 9
	#define TSD_KEYS_MAX 32768
	#define DTOR_PID (PID_MAX_LIMIT+1)
	#define PID_KEY (TSD_KEYS_MAX+1)

	typedef void (dtor_func_t)(void );

	extern int tsd_set(uint_t, void *);
	extern void *tsd_get(uint_t);
	extern void tsd_get_by_thread(uint_t, kthread_t );
	extern void tsd_create(uint_t *, dtor_func_t);
	extern void tsd_destroy(uint_t *);
	extern void tsd_exit(void);

	int spl_tsd_init(void);
	void spl_tsd_fini(void);

	#endif /* _SPL_TSD_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/types.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/types.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/types.h (revision 366774)
	@@ -1,58 +1,57 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_TYPES_H
	#define _SPL_TYPES_H

	#include <linux/types.h>

	typedef enum {
	B_FALSE = 0,
	B_TRUE = 1
	} boolean_t;

	typedef unsigned char uchar_t;
	typedef unsigned short ushort_t;
	typedef unsigned int uint_t;
	typedef unsigned long ulong_t;
	typedef unsigned long long u_longlong_t;
	typedef long long longlong_t;

	typedef unsigned long intptr_t;
	typedef unsigned long long rlim64_t;

	typedef struct task_struct kthread_t;
	typedef struct task_struct proc_t;

	typedef int id_t;
	typedef short pri_t;
	typedef short index_t;
	typedef longlong_t offset_t;
	typedef u_longlong_t u_offset_t;
	typedef ulong_t pgcnt_t;

	typedef int major_t;
	typedef int minor_t;

	#endif /* _SPL_TYPES_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/types32.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/types32.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/types32.h (revision 366774)
	@@ -1,35 +1,34 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_TYPES32_H
	#define _SPL_TYPES32_H

	#include <sys/types.h>

	typedef uint32_t caddr32_t;
	typedef int32_t daddr32_t;
	typedef int32_t time32_t;
	typedef uint32_t size32_t;

	#endif /* _SPL_TYPES32_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/uio.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/uio.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/uio.h (revision 366774)
	@@ -1,144 +1,143 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Copyright (c) 2015 by Chunwei Chen. All rights reserved.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_UIO_H
	#define _SPL_UIO_H

	#include <sys/debug.h>
	#include <linux/uio.h>
	#include <linux/blkdev.h>
	#include <linux/blkdev_compat.h>
	#include <linux/mm.h>
	#include <linux/bio.h>
	#include <asm/uaccess.h>
	#include <sys/types.h>

	typedef struct iovec iovec_t;

	typedef enum uio_rw {
	UIO_READ = 0,
	UIO_WRITE = 1,
	} uio_rw_t;

	typedef enum uio_seg {
	UIO_USERSPACE = 0,
	UIO_SYSSPACE = 1,
	UIO_USERISPACE = 2,
	UIO_BVEC = 3,
	} uio_seg_t;

	typedef struct uio {
	union {
	const struct iovec *uio_iov;
	const struct bio_vec *uio_bvec;
	};
	int uio_iovcnt;
	offset_t uio_loffset;
	uio_seg_t uio_segflg;
	boolean_t uio_fault_disable;
	uint16_t uio_fmode;
	uint16_t uio_extflg;
	offset_t uio_limit;
	ssize_t uio_resid;
	size_t uio_skip;
	} uio_t;

	typedef struct aio_req {
	uio_t *aio_uio;
	void *aio_private;
	} aio_req_t;

	typedef enum xuio_type {
	UIOTYPE_ASYNCIO,
	UIOTYPE_ZEROCOPY,
	} xuio_type_t;


	#define UIOA_IOV_MAX 16

	typedef struct uioa_page_s {
	int uioa_pfncnt;
	void **uioa_ppp;
	caddr_t uioa_base;
	size_t uioa_len;
	} uioa_page_t;

	typedef struct xuio {
	uio_t xu_uio;
	enum xuio_type xu_type;
	union {
	struct {
	uint32_t xu_a_state;
	ssize_t xu_a_mbytes;
	uioa_page_t *xu_a_lcur;
	void **xu_a_lppp;
	void *xu_a_hwst[4];
	uioa_page_t xu_a_locked[UIOA_IOV_MAX];
	} xu_aio;

	struct {
	int xu_zc_rw;
	void *xu_zc_priv;
	} xu_zc;
	} xu_ext;
	} xuio_t;

	#define XUIO_XUZC_PRIV(xuio) xuio->xu_ext.xu_zc.xu_zc_priv
	#define XUIO_XUZC_RW(xuio) xuio->xu_ext.xu_zc.xu_zc_rw

	#define uio_segflg(uio) (uio)->uio_segflg
	#define uio_offset(uio) (uio)->uio_loffset
	#define uio_resid(uio) (uio)->uio_resid
	#define uio_iovcnt(uio) (uio)->uio_iovcnt
	#define uio_iovlen(uio, idx) (uio)->uio_iov[(idx)].iov_len
	#define uio_iovbase(uio, idx) (uio)->uio_iov[(idx)].iov_base

	static inline void
	uio_iov_at_index(uio_t uio, uint_t idx, void base, uint64_t len)
	{
	*base = uio_iovbase(uio, idx);
	*len = uio_iovlen(uio, idx);
	}

	static inline void
	uio_advance(uio_t *uio, size_t size)
	{
	uio->uio_resid -= size;
	uio->uio_loffset += size;
	}

	static inline offset_t
	uio_index_at_offset(uio_t uio, offset_t off, uint_t vec_idx)
	{
	*vec_idx = 0;
	while (vec_idx < uio_iovcnt(uio) && off >= uio_iovlen(uio, vec_idx)) {
	off -= uio_iovlen(uio, *vec_idx);
	(*vec_idx)++;
	}

	return (off);
	}

	#endif /* SPL_UIO_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/user.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/user.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/user.h (revision 366774)
	@@ -1,42 +1,41 @@
	/*
	* Copyright (C) 2015 Cluster Inc.
	* Produced at ClusterHQ Inc (cf, DISCLAIMER).
	* Written by Richard Yao <richard.yao@clusterhq.com>.
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_USER_H
	#define _SPL_USER_H

	/*
	* We have uf_info_t for areleasef(). We implement areleasef() using a global
	* linked list of all open file descriptors with the task structs referenced,
	* so accessing the correct descriptor from areleasef() only requires knowing
	* about the Linux task_struct. Since this is internal to our compatibility
	* layer, we make it an opaque type.
	*
	* XXX: If the descriptor changes under us and we do not do a getf() between
	* the change and using it, we would get an incorrect reference.
	*/

	struct uf_info;
	typedef struct uf_info uf_info_t;

	#define P_FINFO(x) ((uf_info_t *)x)

	#endif /* SPL_USER_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/vfs.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/vfs.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/vfs.h (revision 366774)
	@@ -1,51 +1,50 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_ZFS_H
	#define _SPL_ZFS_H

	#include <linux/mount.h>
	#include <linux/fs.h>
	#include <linux/dcache.h>
	#include <linux/statfs.h>
	#include <linux/xattr.h>
	#include <linux/security.h>
	#include <linux/seq_file.h>

	#define MAXFIDSZ 64

	typedef struct spl_fid {
	union {
	long fid_pad;
	struct {
	ushort_t len; /* length of data in bytes */
	char data[MAXFIDSZ]; /* data (variable len) */
	} _fid;
	} un;
	} fid_t;

	#define fid_len un._fid.len
	#define fid_data un._fid.data

	#endif /* SPL_ZFS_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/vmem.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/vmem.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/vmem.h (revision 366774)
	@@ -1,102 +1,101 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_VMEM_H
	#define _SPL_VMEM_H

	#include <sys/kmem.h>
	#include <linux/sched.h>
	#include <linux/vmalloc.h>

	typedef struct vmem { } vmem_t;

	/*
	* Memory allocation interfaces
	*/
	#define VMEM_ALLOC 0x01
	#define VMEM_FREE 0x02

	#ifndef VMALLOC_TOTAL
	#define VMALLOC_TOTAL (VMALLOC_END - VMALLOC_START)
	#endif

	/*
	* vmem_* is an interface to a low level arena-based memory allocator on
	* Illumos that is used to allocate virtual address space. The kmem SLAB
	* allocator allocates slabs from it. Then the generic allocation functions
	* kmem_{alloc,zalloc,free}() are layered on top of SLAB allocators.
	*
	* On Linux, the primary means of doing allocations is via kmalloc(), which
	* is similarly layered on top of something called the buddy allocator. The
	* buddy allocator is not available to kernel modules, it uses physical
	* memory addresses rather than virtual memory addresses and is prone to
	* fragmentation.
	*
	* Linux sets aside a relatively small address space for in-kernel virtual
	* memory from which allocations can be done using vmalloc(). It might seem
	* like a good idea to use vmalloc() to implement something similar to
	* Illumos' allocator. However, this has the following problems:
	*
	* 1. Page directory table allocations are hard coded to use GFP_KERNEL.
	* Consequently, any KM_PUSHPAGE or KM_NOSLEEP allocations done using
	* vmalloc() will not have proper semantics.
	*
	* 2. Address space exhaustion is a real issue on 32-bit platforms where
	* only a few 100MB are available. The kernel will handle it by spinning
	* when it runs out of address space.
	*
	* 3. All vmalloc() allocations and frees are protected by a single global
	* lock which serializes all allocations.
	*
	* 4. Accessing /proc/meminfo and /proc/vmallocinfo will iterate the entire
	* list. The former will sum the allocations while the latter will print
	* them to user space in a way that user space can keep the lock held
	* indefinitely. When the total number of mapped allocations is large
	* (several 100,000) a large amount of time will be spent waiting on locks.
	*
	* 5. Linux has a wait_on_bit() locking primitive that assumes physical
	* memory is used, it simply does not work on virtual memory. Certain
	* Linux structures (e.g. the superblock) use them and might be embedded
	* into a structure from Illumos. This makes using Linux virtual memory
	* unsafe in certain situations.
	*
	* It follows that we cannot obtain identical semantics to those on Illumos.
	* Consequently, we implement the kmem_{alloc,zalloc,free}() functions in
	* such a way that they can be used as drop-in replacements for small vmem_*
	* allocations (8MB in size or smaller) and map vmem_{alloc,zalloc,free}()
	* to them.
	*/

	#define vmem_alloc(sz, fl) spl_vmem_alloc((sz), (fl), __func__, __LINE__)
	#define vmem_zalloc(sz, fl) spl_vmem_zalloc((sz), (fl), __func__, __LINE__)
	#define vmem_free(ptr, sz) spl_vmem_free((ptr), (sz))

	extern void spl_vmem_alloc(size_t sz, int fl, const char func, int line);
	extern void spl_vmem_zalloc(size_t sz, int fl, const char func, int line);
	extern void spl_vmem_free(const void *ptr, size_t sz);

	int spl_vmem_init(void);
	void spl_vmem_fini(void);

	#endif /* _SPL_VMEM_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/vmsystm.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/vmsystm.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/vmsystm.h (revision 366774)
	@@ -1,92 +1,91 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_VMSYSTM_H
	#define _SPL_VMSYSTM_H

	#include <linux/mmzone.h>
	#include <linux/mm.h>
	#include <linux/swap.h>
	#include <linux/highmem.h>
	#include <linux/vmalloc.h>
	#include <sys/types.h>
	#include <asm/uaccess.h>

	#ifdef HAVE_TOTALRAM_PAGES_FUNC
	#define zfs_totalram_pages totalram_pages()
	#else
	#define zfs_totalram_pages totalram_pages
	#endif

	#ifdef HAVE_TOTALHIGH_PAGES
	#define zfs_totalhigh_pages totalhigh_pages()
	#else
	#define zfs_totalhigh_pages totalhigh_pages
	#endif

	#define membar_producer() smp_wmb()
	#define physmem zfs_totalram_pages

	#define xcopyin(from, to, size) copy_from_user(to, from, size)
	#define xcopyout(from, to, size) copy_to_user(to, from, size)

	static __inline__ int
	copyin(const void from, void to, size_t len)
	{
	/* On error copyin routine returns -1 */
	if (xcopyin(from, to, len))
	return (-1);

	return (0);
	}

	static __inline__ int
	copyout(const void from, void to, size_t len)
	{
	/* On error copyout routine returns -1 */
	if (xcopyout(from, to, len))
	return (-1);

	return (0);
	}

	static __inline__ int
	copyinstr(const void from, void to, size_t len, size_t *done)
	{
	size_t rc;

	if (len == 0)
	return (-ENAMETOOLONG);

	/* XXX: Should return ENAMETOOLONG if 'strlen(from) > len' */

	memset(to, 0, len);
	rc = copyin(from, to, len - 1);
	if (done != NULL)
	*done = rc;

	return (0);
	}

	#endif /* SPL_VMSYSTM_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/vnode.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/vnode.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/vnode.h (revision 366774)
	@@ -1,107 +1,106 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_VNODE_H
	#define _SPL_VNODE_H

	#include <linux/module.h>
	#include <linux/syscalls.h>
	#include <linux/fcntl.h>
	#include <linux/buffer_head.h>
	#include <linux/dcache.h>
	#include <linux/namei.h>
	#include <linux/file.h>
	#include <linux/fs.h>
	#include <linux/fs_struct.h>
	#include <linux/mount.h>
	#include <sys/kmem.h>
	#include <sys/mutex.h>
	#include <sys/types.h>
	#include <sys/time.h>
	#include <sys/uio.h>
	#include <sys/user.h>

	/*
	* Prior to linux-2.6.33 only O_DSYNC semantics were implemented and
	* they used the O_SYNC flag. As of linux-2.6.33 the this behavior
	* was properly split in to O_SYNC and O_DSYNC respectively.
	*/
	#ifndef O_DSYNC
	#define O_DSYNC O_SYNC
	#endif

	#define F_FREESP 11 /* Free file space */

	/*
	* The vnode AT_ flags are mapped to the Linux ATTR_* flags.
	* This allows them to be used safely with an iattr structure.
	* The AT_XVATTR flag has been added and mapped to the upper
	* bit range to avoid conflicting with the standard Linux set.
	*/
	#undef AT_UID
	#undef AT_GID

	#define AT_MODE ATTR_MODE
	#define AT_UID ATTR_UID
	#define AT_GID ATTR_GID
	#define AT_SIZE ATTR_SIZE
	#define AT_ATIME ATTR_ATIME
	#define AT_MTIME ATTR_MTIME
	#define AT_CTIME ATTR_CTIME

	#define ATTR_XVATTR (1U << 31)
	#define AT_XVATTR ATTR_XVATTR

	#define ATTR_IATTR_MASK (ATTR_MODE \| ATTR_UID \| ATTR_GID \| ATTR_SIZE \| \
	ATTR_ATIME \| ATTR_MTIME \| ATTR_CTIME \| ATTR_FILE)

	#define CRCREAT 0x01
	#define RMFILE 0x02

	#define B_INVAL 0x01
	#define B_TRUNC 0x02

	#define LOOKUP_DIR 0x01
	#define LOOKUP_XATTR 0x02
	#define CREATE_XATTR_DIR 0x04
	#define ATTR_NOACLCHECK 0x20

	typedef struct vattr {
	uint32_t va_mask; /* attribute bit-mask */
	ushort_t va_mode; /* acc mode */
	uid_t va_uid; /* owner uid */
	gid_t va_gid; /* owner gid */
	long va_fsid; /* fs id */
	long va_nodeid; /* node # */
	uint32_t va_nlink; /* # links */
	uint64_t va_size; /* file size */
	inode_timespec_t va_atime; /* last acc */
	inode_timespec_t va_mtime; /* last mod */
	inode_timespec_t va_ctime; /* last chg */
	dev_t va_rdev; /* dev */
	uint64_t va_nblocks; /* space used */
	uint32_t va_blksize; /* block size */
	struct dentry va_dentry; / dentry to wire */
	} vattr_t;
	#endif /* SPL_VNODE_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/wait.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/wait.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/wait.h (revision 366774)
	@@ -1,55 +1,54 @@
	/*
	* Copyright (C) 2007-2014 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_WAIT_H
	#define _SPL_WAIT_H

	#include <linux/sched.h>
	#include <linux/wait.h>

	#ifndef HAVE_WAIT_ON_BIT_ACTION
	#define spl_wait_on_bit(word, bit, mode) wait_on_bit(word, bit, mode)
	#else

	static inline int
	spl_bit_wait(void *word)
	{
	schedule();
	return (0);
	}

	#define spl_wait_on_bit(word, bit, mode) \
	wait_on_bit(word, bit, spl_bit_wait, mode)

	#endif /* HAVE_WAIT_ON_BIT_ACTION */

	#ifdef HAVE_WAIT_QUEUE_ENTRY_T
	typedef wait_queue_head_t spl_wait_queue_head_t;
	typedef wait_queue_entry_t spl_wait_queue_entry_t;
	#else
	typedef wait_queue_head_t spl_wait_queue_head_t;
	typedef wait_queue_t spl_wait_queue_entry_t;
	#endif

	#endif /* SPL_WAIT_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/zmod.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/zmod.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/zmod.h (revision 366774)
	@@ -1,70 +1,69 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*
	*
	* z_compress_level/z_uncompress are nearly identical copies of the
	* compress2/uncompress functions provided by the official zlib package
	* available at http://zlib.net/. The only changes made we to slightly
	* adapt the functions called to match the linux kernel implementation
	* of zlib. The full zlib license follows:
	*
	* zlib.h -- interface of the 'zlib' general purpose compression library
	* version 1.2.5, April 19th, 2010
	*
	* Copyright (C) 1995-2010 Jean-loup Gailly and Mark Adler
	*
	* This software is provided 'as-is', without any express or implied
	* warranty. In no event will the authors be held liable for any damages
	* arising from the use of this software.
	*
	* Permission is granted to anyone to use this software for any purpose,
	* including commercial applications, and to alter it and redistribute it
	* freely, subject to the following restrictions:
	*
	* 1. The origin of this software must not be misrepresented; you must not
	* claim that you wrote the original software. If you use this software
	* in a product, an acknowledgment in the product documentation would be
	* appreciated but is not required.
	* 2. Altered source versions must be plainly marked as such, and must not be
	* misrepresented as being the original software.
	* 3. This notice may not be removed or altered from any source distribution.
	*
	* Jean-loup Gailly
	* Mark Adler
	*/

	#ifndef _SPL_ZMOD_H
	#define _SPL_ZMOD_H

	#include <sys/types.h>
	#include <linux/zlib.h>

	extern int z_compress_level(void dest, size_t destLen, const void *source,
	size_t sourceLen, int level);
	extern int z_uncompress(void dest, size_t destLen, const void *source,
	size_t sourceLen);

	int spl_zlib_init(void);
	void spl_zlib_fini(void);

	#endif /* SPL_ZMOD_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/spl/sys/zone.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/spl/sys/zone.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/spl/sys/zone.h (revision 366774)
	@@ -1,36 +1,35 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#ifndef _SPL_ZONE_H
	#define _SPL_ZONE_H

	#include <sys/byteorder.h>

	#define GLOBAL_ZONEID 0

	#define zone_dataset_visible(x, y) (1)
	#define crgetzoneid(x) (GLOBAL_ZONEID)
	#define INGLOBALZONE(z) (1)

	#endif /* SPL_ZONE_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/zfs/sys/zfs_ctldir.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/zfs/sys/zfs_ctldir.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/zfs/sys/zfs_ctldir.h (revision 366774)
	@@ -1,103 +1,103 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (C) 2011 Lawrence Livermore National Security, LLC.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* LLNL-CODE-403049.
	* Rewritten for Linux by:
	* Rohan Puri <rohan.puri15@gmail.com>
	* Brian Behlendorf <behlendorf1@llnl.gov>
	*/

	#ifndef _ZFS_CTLDIR_H
	#define _ZFS_CTLDIR_H

	#include <sys/vnode.h>
	#include <sys/pathname.h>
	#include <sys/zfs_vfsops.h>
	#include <sys/zfs_znode.h>

	#define ZFS_CTLDIR_NAME ".zfs"
	#define ZFS_SNAPDIR_NAME "snapshot"
	#define ZFS_SHAREDIR_NAME "shares"

	#define zfs_has_ctldir(zdp) \
	((zdp)->z_id == ZTOZSB(zdp)->z_root && \
	(ZTOZSB(zdp)->z_ctldir != NULL))
	#define zfs_show_ctldir(zdp) \
	(zfs_has_ctldir(zdp) && \
	(ZTOZSB(zdp)->z_show_ctldir))

	extern int zfs_expire_snapshot;

	/* zfsctl generic functions */
	extern int zfsctl_create(zfsvfs_t *);
	extern void zfsctl_destroy(zfsvfs_t *);
	extern struct inode zfsctl_root(znode_t );
	extern void zfsctl_init(void);
	extern void zfsctl_fini(void);
	extern boolean_t zfsctl_is_node(struct inode *ip);
	extern boolean_t zfsctl_is_snapdir(struct inode *ip);
	extern int zfsctl_fid(struct inode ip, fid_t fidp);

	/* zfsctl '.zfs' functions */
	-extern int zfsctl_root_lookup(struct inode dip, char name,
	+extern int zfsctl_root_lookup(struct inode dip, const char name,
	struct inode *ipp, int flags, cred_t cr, int *direntflags,
	pathname_t *realpnp);

	/* zfsctl '.zfs/snapshot' functions */
	-extern int zfsctl_snapdir_lookup(struct inode dip, char name,
	+extern int zfsctl_snapdir_lookup(struct inode dip, const char name,
	struct inode *ipp, int flags, cred_t cr, int *direntflags,
	pathname_t *realpnp);
	-extern int zfsctl_snapdir_rename(struct inode sdip, char sname,
	- struct inode tdip, char tname, cred_t *cr, int flags);
	-extern int zfsctl_snapdir_remove(struct inode dip, char name, cred_t *cr,
	- int flags);
	-extern int zfsctl_snapdir_mkdir(struct inode dip, char dirname, vattr_t *vap,
	- struct inode *ipp, cred_t cr, int flags);
	+extern int zfsctl_snapdir_rename(struct inode sdip, const char sname,
	+ struct inode tdip, const char tname, cred_t *cr, int flags);
	+extern int zfsctl_snapdir_remove(struct inode dip, const char name,
	+ cred_t *cr, int flags);
	+extern int zfsctl_snapdir_mkdir(struct inode dip, const char dirname,
	+ vattr_t vap, struct inode ipp, cred_t cr, int flags);
	extern int zfsctl_snapshot_mount(struct path *path, int flags);
	-extern int zfsctl_snapshot_unmount(char *snapname, int flags);
	+extern int zfsctl_snapshot_unmount(const char *snapname, int flags);
	extern int zfsctl_snapshot_unmount_delay(spa_t *spa, uint64_t objsetid,
	int delay);
	extern int zfsctl_snapdir_vget(struct super_block *sb, uint64_t objsetid,
	int gen, struct inode **ipp);

	/* zfsctl '.zfs/shares' functions */
	extern int zfsctl_shares_lookup(struct inode dip, char name,
	struct inode *ipp, int flags, cred_t cr, int *direntflags,
	pathname_t *realpnp);

	/*
	* These inodes numbers are reserved for the .zfs control directory.
	* It is important that they be no larger that 48-bits because only
	* 6 bytes are reserved in the NFS file handle for the object number.
	* However, they should be as large as possible to avoid conflicts
	* with the objects which are assigned monotonically by the dmu.
	*/
	#define ZFSCTL_INO_ROOT 0x0000FFFFFFFFFFFFULL
	#define ZFSCTL_INO_SHARES 0x0000FFFFFFFFFFFEULL
	#define ZFSCTL_INO_SNAPDIR 0x0000FFFFFFFFFFFDULL
	#define ZFSCTL_INO_SNAPDIRS 0x0000FFFFFFFFFFFCULL

	#define ZFSCTL_EXPIRE_SNAPSHOT 300

	#endif /* _ZFS_CTLDIR_H */
	Index: vendor-sys/openzfs/dist/include/os/linux/zfs/sys/zfs_vnops.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/os/linux/zfs/sys/zfs_vnops.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/os/linux/zfs/sys/zfs_vnops.h (revision 366774)
	@@ -1,91 +1,91 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
	*/

	#ifndef _SYS_FS_ZFS_VNOPS_H
	#define _SYS_FS_ZFS_VNOPS_H

	#include <sys/vnode.h>
	#include <sys/xvattr.h>
	#include <sys/uio.h>
	#include <sys/cred.h>
	#include <sys/fcntl.h>
	#include <sys/pathname.h>
	#include <sys/zpl.h>
	#include <sys/zfs_file.h>

	#ifdef __cplusplus
	extern "C" {
	#endif

	extern int zfs_open(struct inode ip, int mode, int flag, cred_t cr);
	extern int zfs_close(struct inode ip, int flag, cred_t cr);
	extern int zfs_holey(struct inode ip, int cmd, loff_t off);
	extern int zfs_read(struct inode ip, uio_t uio, int ioflag, cred_t *cr);
	extern int zfs_write(struct inode ip, uio_t uio, int ioflag, cred_t *cr);
	extern int zfs_write_simple(znode_t zp, const void data, size_t len,
	loff_t pos, size_t *resid);
	extern int zfs_access(struct inode ip, int mode, int flag, cred_t cr);
	-extern int zfs_lookup(znode_t dzp, char nm, znode_t **zpp,
	- int flags, cred_t cr, int direntflags, pathname_t *realpnp);
	+extern int zfs_lookup(znode_t dzp, char nm, znode_t **zpp, int flags,
	+ cred_t cr, int direntflags, pathname_t *realpnp);
	extern int zfs_create(znode_t dzp, char name, vattr_t *vap, int excl,
	int mode, znode_t *zpp, cred_t cr, int flag, vsecattr_t *vsecp);
	extern int zfs_tmpfile(struct inode dip, vattr_t vapzfs, int excl,
	int mode, struct inode *ipp, cred_t cr, int flag, vsecattr_t *vsecp);
	extern int zfs_remove(znode_t dzp, char name, cred_t *cr, int flags);
	extern int zfs_mkdir(znode_t dzp, char dirname, vattr_t *vap,
	znode_t *zpp, cred_t cr, int flags, vsecattr_t *vsecp);
	extern int zfs_rmdir(znode_t dzp, char name, znode_t *cwd,
	cred_t *cr, int flags);
	extern int zfs_readdir(struct inode ip, zpl_dir_context_t ctx, cred_t *cr);
	extern int zfs_fsync(znode_t zp, int syncflag, cred_t cr);
	extern int zfs_getattr_fast(struct inode ip, struct kstat sp);
	extern int zfs_setattr(znode_t zp, vattr_t vap, int flag, cred_t *cr);
	extern int zfs_rename(znode_t sdzp, char snm, znode_t *tdzp,
	char tnm, cred_t cr, int flags);
	extern int zfs_symlink(znode_t dzp, char name, vattr_t *vap,
	char link, znode_t zpp, cred_t cr, int flags);
	extern int zfs_readlink(struct inode ip, uio_t uio, cred_t *cr);
	extern int zfs_link(znode_t tdzp, znode_t szp,
	char name, cred_t cr, int flags);
	extern void zfs_inactive(struct inode *ip);
	extern int zfs_space(znode_t zp, int cmd, flock64_t bfp, int flag,
	offset_t offset, cred_t *cr);
	extern int zfs_fid(struct inode ip, fid_t fidp);
	extern int zfs_getsecattr(struct inode ip, vsecattr_t vsecp, int flag,
	cred_t *cr);
	extern int zfs_setsecattr(znode_t zp, vsecattr_t vsecp, int flag,
	cred_t *cr);
	extern int zfs_getpage(struct inode ip, struct page pl[], int nr_pages);
	extern int zfs_putpage(struct inode ip, struct page pp,
	struct writeback_control *wbc);
	extern int zfs_dirty_inode(struct inode *ip, int flags);
	extern int zfs_map(struct inode ip, offset_t off, caddr_t addrp,
	size_t len, unsigned long vm_flags);
	extern void zfs_zrele_async(znode_t *zp);

	#ifdef __cplusplus
	}
	#endif

	#endif /* _SYS_FS_ZFS_VNOPS_H */
	Index: vendor-sys/openzfs/dist/include/sys/dmu.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/sys/dmu.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/sys/dmu.h (revision 366774)
	@@ -1,1090 +1,1089 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2011, 2020 by Delphix. All rights reserved.
	* Copyright 2011 Nexenta Systems, Inc. All rights reserved.
	* Copyright (c) 2012, Joyent, Inc. All rights reserved.
	* Copyright 2014 HybridCluster. All rights reserved.
	* Copyright (c) 2014 Spectra Logic Corporation, All rights reserved.
	* Copyright 2013 Saso Kiselkov. All rights reserved.
	* Copyright (c) 2017, Intel Corporation.
	*/

	/* Portions Copyright 2010 Robert Milkowski */

	#ifndef _SYS_DMU_H
	#define _SYS_DMU_H

	/*
	* This file describes the interface that the DMU provides for its
	* consumers.
	*
	* The DMU also interacts with the SPA. That interface is described in
	* dmu_spa.h.
	*/

	#include <sys/zfs_context.h>
	#include <sys/inttypes.h>
	#include <sys/cred.h>
	#include <sys/fs/zfs.h>
	#include <sys/zio_compress.h>
	#include <sys/zio_priority.h>
	#include <sys/uio.h>
	#include <sys/zfs_file.h>

	#ifdef __cplusplus
	extern "C" {
	#endif

	struct page;
	struct vnode;
	struct spa;
	struct zilog;
	struct zio;
	struct blkptr;
	struct zap_cursor;
	struct dsl_dataset;
	struct dsl_pool;
	struct dnode;
	struct drr_begin;
	struct drr_end;
	struct zbookmark_phys;
	struct spa;
	struct nvlist;
	struct arc_buf;
	struct zio_prop;
	struct sa_handle;
	struct dsl_crypto_params;
	struct locked_range;

	typedef struct objset objset_t;
	typedef struct dmu_tx dmu_tx_t;
	typedef struct dsl_dir dsl_dir_t;
	typedef struct dnode dnode_t;

	typedef enum dmu_object_byteswap {
	DMU_BSWAP_UINT8,
	DMU_BSWAP_UINT16,
	DMU_BSWAP_UINT32,
	DMU_BSWAP_UINT64,
	DMU_BSWAP_ZAP,
	DMU_BSWAP_DNODE,
	DMU_BSWAP_OBJSET,
	DMU_BSWAP_ZNODE,
	DMU_BSWAP_OLDACL,
	DMU_BSWAP_ACL,
	/*
	* Allocating a new byteswap type number makes the on-disk format
	* incompatible with any other format that uses the same number.
	*
	* Data can usually be structured to work with one of the
	* DMU_BSWAP_UINT* or DMU_BSWAP_ZAP types.
	*/
	DMU_BSWAP_NUMFUNCS
	} dmu_object_byteswap_t;

	#define DMU_OT_NEWTYPE 0x80
	#define DMU_OT_METADATA 0x40
	#define DMU_OT_ENCRYPTED 0x20
	#define DMU_OT_BYTESWAP_MASK 0x1f

	/*
	* Defines a uint8_t object type. Object types specify if the data
	* in the object is metadata (boolean) and how to byteswap the data
	* (dmu_object_byteswap_t). All of the types created by this method
	* are cached in the dbuf metadata cache.
	*/
	#define DMU_OT(byteswap, metadata, encrypted) \
	(DMU_OT_NEWTYPE \| \
	((metadata) ? DMU_OT_METADATA : 0) \| \
	((encrypted) ? DMU_OT_ENCRYPTED : 0) \| \
	((byteswap) & DMU_OT_BYTESWAP_MASK))

	#define DMU_OT_IS_VALID(ot) (((ot) & DMU_OT_NEWTYPE) ? \
	((ot) & DMU_OT_BYTESWAP_MASK) < DMU_BSWAP_NUMFUNCS : \
	(ot) < DMU_OT_NUMTYPES)

	#define DMU_OT_IS_METADATA_CACHED(ot) (((ot) & DMU_OT_NEWTYPE) ? \
	B_TRUE : dmu_ot[(ot)].ot_dbuf_metadata_cache)

	/*
	* MDB doesn't have dmu_ot; it defines these macros itself.
	*/
	#ifndef ZFS_MDB
	#define DMU_OT_IS_METADATA_IMPL(ot) (dmu_ot[ot].ot_metadata)
	#define DMU_OT_IS_ENCRYPTED_IMPL(ot) (dmu_ot[ot].ot_encrypt)
	#define DMU_OT_BYTESWAP_IMPL(ot) (dmu_ot[ot].ot_byteswap)
	#endif

	#define DMU_OT_IS_METADATA(ot) (((ot) & DMU_OT_NEWTYPE) ? \
	((ot) & DMU_OT_METADATA) : \
	DMU_OT_IS_METADATA_IMPL(ot))

	#define DMU_OT_IS_DDT(ot) \
	((ot) == DMU_OT_DDT_ZAP)

	#define DMU_OT_IS_ZIL(ot) \
	((ot) == DMU_OT_INTENT_LOG)

	/* Note: ztest uses DMU_OT_UINT64_OTHER as a proxy for file blocks */
	#define DMU_OT_IS_FILE(ot) \
	((ot) == DMU_OT_PLAIN_FILE_CONTENTS \|\| (ot) == DMU_OT_UINT64_OTHER)

	#define DMU_OT_IS_ENCRYPTED(ot) (((ot) & DMU_OT_NEWTYPE) ? \
	((ot) & DMU_OT_ENCRYPTED) : \
	DMU_OT_IS_ENCRYPTED_IMPL(ot))

	/*
	* These object types use bp_fill != 1 for their L0 bp's. Therefore they can't
	* have their data embedded (i.e. use a BP_IS_EMBEDDED() bp), because bp_fill
	* is repurposed for embedded BPs.
	*/
	#define DMU_OT_HAS_FILL(ot) \
	((ot) == DMU_OT_DNODE \|\| (ot) == DMU_OT_OBJSET)

	#define DMU_OT_BYTESWAP(ot) (((ot) & DMU_OT_NEWTYPE) ? \
	((ot) & DMU_OT_BYTESWAP_MASK) : \
	DMU_OT_BYTESWAP_IMPL(ot))

	typedef enum dmu_object_type {
	DMU_OT_NONE,
	/* general: */
	DMU_OT_OBJECT_DIRECTORY, /* ZAP */
	DMU_OT_OBJECT_ARRAY, /* UINT64 */
	DMU_OT_PACKED_NVLIST, /* UINT8 (XDR by nvlist_pack/unpack) */
	DMU_OT_PACKED_NVLIST_SIZE, /* UINT64 */
	DMU_OT_BPOBJ, /* UINT64 */
	DMU_OT_BPOBJ_HDR, /* UINT64 */
	/* spa: */
	DMU_OT_SPACE_MAP_HEADER, /* UINT64 */
	DMU_OT_SPACE_MAP, /* UINT64 */
	/* zil: */
	DMU_OT_INTENT_LOG, /* UINT64 */
	/* dmu: */
	DMU_OT_DNODE, /* DNODE */
	DMU_OT_OBJSET, /* OBJSET */
	/* dsl: */
	DMU_OT_DSL_DIR, /* UINT64 */
	DMU_OT_DSL_DIR_CHILD_MAP, /* ZAP */
	DMU_OT_DSL_DS_SNAP_MAP, /* ZAP */
	DMU_OT_DSL_PROPS, /* ZAP */
	DMU_OT_DSL_DATASET, /* UINT64 */
	/* zpl: */
	DMU_OT_ZNODE, /* ZNODE */
	DMU_OT_OLDACL, /* Old ACL */
	DMU_OT_PLAIN_FILE_CONTENTS, /* UINT8 */
	DMU_OT_DIRECTORY_CONTENTS, /* ZAP */
	DMU_OT_MASTER_NODE, /* ZAP */
	DMU_OT_UNLINKED_SET, /* ZAP */
	/* zvol: */
	DMU_OT_ZVOL, /* UINT8 */
	DMU_OT_ZVOL_PROP, /* ZAP */
	/* other; for testing only! */
	DMU_OT_PLAIN_OTHER, /* UINT8 */
	DMU_OT_UINT64_OTHER, /* UINT64 */
	DMU_OT_ZAP_OTHER, /* ZAP */
	/* new object types: */
	DMU_OT_ERROR_LOG, /* ZAP */
	DMU_OT_SPA_HISTORY, /* UINT8 */
	DMU_OT_SPA_HISTORY_OFFSETS, /* spa_his_phys_t */
	DMU_OT_POOL_PROPS, /* ZAP */
	DMU_OT_DSL_PERMS, /* ZAP */
	DMU_OT_ACL, /* ACL */
	DMU_OT_SYSACL, /* SYSACL */
	DMU_OT_FUID, /* FUID table (Packed NVLIST UINT8) */
	DMU_OT_FUID_SIZE, /* FUID table size UINT64 */
	DMU_OT_NEXT_CLONES, /* ZAP */
	DMU_OT_SCAN_QUEUE, /* ZAP */
	DMU_OT_USERGROUP_USED, /* ZAP */
	DMU_OT_USERGROUP_QUOTA, /* ZAP */
	DMU_OT_USERREFS, /* ZAP */
	DMU_OT_DDT_ZAP, /* ZAP */
	DMU_OT_DDT_STATS, /* ZAP */
	DMU_OT_SA, /* System attr */
	DMU_OT_SA_MASTER_NODE, /* ZAP */
	DMU_OT_SA_ATTR_REGISTRATION, /* ZAP */
	DMU_OT_SA_ATTR_LAYOUTS, /* ZAP */
	DMU_OT_SCAN_XLATE, /* ZAP */
	DMU_OT_DEDUP, /* fake dedup BP from ddt_bp_create() */
	DMU_OT_DEADLIST, /* ZAP */
	DMU_OT_DEADLIST_HDR, /* UINT64 */
	DMU_OT_DSL_CLONES, /* ZAP */
	DMU_OT_BPOBJ_SUBOBJ, /* UINT64 */
	/*
	* Do not allocate new object types here. Doing so makes the on-disk
	* format incompatible with any other format that uses the same object
	* type number.
	*
	* When creating an object which does not have one of the above types
	* use the DMU_OTN_* type with the correct byteswap and metadata
	* values.
	*
	* The DMU_OTN_* types do not have entries in the dmu_ot table,
	* use the DMU_OT_IS_METADATA() and DMU_OT_BYTESWAP() macros instead
	* of indexing into dmu_ot directly (this works for both DMU_OT_* types
	* and DMU_OTN_* types).
	*/
	DMU_OT_NUMTYPES,

	/*
	* Names for valid types declared with DMU_OT().
	*/
	DMU_OTN_UINT8_DATA = DMU_OT(DMU_BSWAP_UINT8, B_FALSE, B_FALSE),
	DMU_OTN_UINT8_METADATA = DMU_OT(DMU_BSWAP_UINT8, B_TRUE, B_FALSE),
	DMU_OTN_UINT16_DATA = DMU_OT(DMU_BSWAP_UINT16, B_FALSE, B_FALSE),
	DMU_OTN_UINT16_METADATA = DMU_OT(DMU_BSWAP_UINT16, B_TRUE, B_FALSE),
	DMU_OTN_UINT32_DATA = DMU_OT(DMU_BSWAP_UINT32, B_FALSE, B_FALSE),
	DMU_OTN_UINT32_METADATA = DMU_OT(DMU_BSWAP_UINT32, B_TRUE, B_FALSE),
	DMU_OTN_UINT64_DATA = DMU_OT(DMU_BSWAP_UINT64, B_FALSE, B_FALSE),
	DMU_OTN_UINT64_METADATA = DMU_OT(DMU_BSWAP_UINT64, B_TRUE, B_FALSE),
	DMU_OTN_ZAP_DATA = DMU_OT(DMU_BSWAP_ZAP, B_FALSE, B_FALSE),
	DMU_OTN_ZAP_METADATA = DMU_OT(DMU_BSWAP_ZAP, B_TRUE, B_FALSE),

	DMU_OTN_UINT8_ENC_DATA = DMU_OT(DMU_BSWAP_UINT8, B_FALSE, B_TRUE),
	DMU_OTN_UINT8_ENC_METADATA = DMU_OT(DMU_BSWAP_UINT8, B_TRUE, B_TRUE),
	DMU_OTN_UINT16_ENC_DATA = DMU_OT(DMU_BSWAP_UINT16, B_FALSE, B_TRUE),
	DMU_OTN_UINT16_ENC_METADATA = DMU_OT(DMU_BSWAP_UINT16, B_TRUE, B_TRUE),
	DMU_OTN_UINT32_ENC_DATA = DMU_OT(DMU_BSWAP_UINT32, B_FALSE, B_TRUE),
	DMU_OTN_UINT32_ENC_METADATA = DMU_OT(DMU_BSWAP_UINT32, B_TRUE, B_TRUE),
	DMU_OTN_UINT64_ENC_DATA = DMU_OT(DMU_BSWAP_UINT64, B_FALSE, B_TRUE),
	DMU_OTN_UINT64_ENC_METADATA = DMU_OT(DMU_BSWAP_UINT64, B_TRUE, B_TRUE),
	DMU_OTN_ZAP_ENC_DATA = DMU_OT(DMU_BSWAP_ZAP, B_FALSE, B_TRUE),
	DMU_OTN_ZAP_ENC_METADATA = DMU_OT(DMU_BSWAP_ZAP, B_TRUE, B_TRUE),
	} dmu_object_type_t;

	/*
	* These flags are intended to be used to specify the "txg_how"
	* parameter when calling the dmu_tx_assign() function. See the comment
	* above dmu_tx_assign() for more details on the meaning of these flags.
	*/
	#define TXG_NOWAIT (0ULL)
	#define TXG_WAIT (1ULL<<0)
	#define TXG_NOTHROTTLE (1ULL<<1)

	void byteswap_uint64_array(void *buf, size_t size);
	void byteswap_uint32_array(void *buf, size_t size);
	void byteswap_uint16_array(void *buf, size_t size);
	void byteswap_uint8_array(void *buf, size_t size);
	void zap_byteswap(void *buf, size_t size);
	void zfs_oldacl_byteswap(void *buf, size_t size);
	void zfs_acl_byteswap(void *buf, size_t size);
	void zfs_znode_byteswap(void *buf, size_t size);

	#define DS_FIND_SNAPSHOTS (1<<0)
	#define DS_FIND_CHILDREN (1<<1)
	#define DS_FIND_SERIALIZE (1<<2)

	/*
	* The maximum number of bytes that can be accessed as part of one
	* operation, including metadata.
	*/
	#define DMU_MAX_ACCESS (64 * 1024 * 1024) /* 64MB */
	#define DMU_MAX_DELETEBLKCNT (20480) /* ~5MB of indirect blocks */

	#define DMU_USERUSED_OBJECT (-1ULL)
	#define DMU_GROUPUSED_OBJECT (-2ULL)
	#define DMU_PROJECTUSED_OBJECT (-3ULL)

	/*
	* Zap prefix for object accounting in DMU_{USER,GROUP,PROJECT}USED_OBJECT.
	*/
	#define DMU_OBJACCT_PREFIX "obj-"
	#define DMU_OBJACCT_PREFIX_LEN 4

	/*
	* artificial blkids for bonus buffer and spill blocks
	*/
	#define DMU_BONUS_BLKID (-1ULL)
	#define DMU_SPILL_BLKID (-2ULL)

	/*
	* Public routines to create, destroy, open, and close objsets.
	*/
	typedef void dmu_objset_create_sync_func_t(objset_t os, void arg,
	cred_t cr, dmu_tx_t tx);

	int dmu_objset_hold(const char name, void tag, objset_t **osp);
	int dmu_objset_own(const char *name, dmu_objset_type_t type,
	boolean_t readonly, boolean_t key_required, void tag, objset_t *osp);
	void dmu_objset_rele(objset_t os, void tag);
	void dmu_objset_disown(objset_t os, boolean_t key_required, void tag);
	int dmu_objset_open_ds(struct dsl_dataset ds, objset_t *osp);

	void dmu_objset_evict_dbufs(objset_t *os);
	int dmu_objset_create(const char *name, dmu_objset_type_t type, uint64_t flags,
	struct dsl_crypto_params *dcp, dmu_objset_create_sync_func_t func,
	void *arg);
	int dmu_objset_clone(const char name, const char origin);
	int dsl_destroy_snapshots_nvl(struct nvlist *snaps, boolean_t defer,
	struct nvlist *errlist);
	int dmu_objset_snapshot_one(const char fsname, const char snapname);
	-int dmu_objset_snapshot_tmp(const char , const char , int);
	int dmu_objset_find(const char name, int func(const char , void ), void arg,
	int flags);
	void dmu_objset_byteswap(void *buf, size_t size);
	int dsl_dataset_rename_snapshot(const char *fsname,
	const char oldsnapname, const char newsnapname, boolean_t recursive);

	typedef struct dmu_buf {
	uint64_t db_object; /* object that this buffer is part of */
	uint64_t db_offset; /* byte offset in this object */
	uint64_t db_size; /* size of buffer in bytes */
	void db_data; / data in buffer */
	} dmu_buf_t;

	/*
	* The names of zap entries in the DIRECTORY_OBJECT of the MOS.
	*/
	#define DMU_POOL_DIRECTORY_OBJECT 1
	#define DMU_POOL_CONFIG "config"
	#define DMU_POOL_FEATURES_FOR_WRITE "features_for_write"
	#define DMU_POOL_FEATURES_FOR_READ "features_for_read"
	#define DMU_POOL_FEATURE_DESCRIPTIONS "feature_descriptions"
	#define DMU_POOL_FEATURE_ENABLED_TXG "feature_enabled_txg"
	#define DMU_POOL_ROOT_DATASET "root_dataset"
	#define DMU_POOL_SYNC_BPOBJ "sync_bplist"
	#define DMU_POOL_ERRLOG_SCRUB "errlog_scrub"
	#define DMU_POOL_ERRLOG_LAST "errlog_last"
	#define DMU_POOL_SPARES "spares"
	#define DMU_POOL_DEFLATE "deflate"
	#define DMU_POOL_HISTORY "history"
	#define DMU_POOL_PROPS "pool_props"
	#define DMU_POOL_L2CACHE "l2cache"
	#define DMU_POOL_TMP_USERREFS "tmp_userrefs"
	#define DMU_POOL_DDT "DDT-%s-%s-%s"
	#define DMU_POOL_DDT_STATS "DDT-statistics"
	#define DMU_POOL_CREATION_VERSION "creation_version"
	#define DMU_POOL_SCAN "scan"
	#define DMU_POOL_FREE_BPOBJ "free_bpobj"
	#define DMU_POOL_BPTREE_OBJ "bptree_obj"
	#define DMU_POOL_EMPTY_BPOBJ "empty_bpobj"
	#define DMU_POOL_CHECKSUM_SALT "org.illumos:checksum_salt"
	#define DMU_POOL_VDEV_ZAP_MAP "com.delphix:vdev_zap_map"
	#define DMU_POOL_REMOVING "com.delphix:removing"
	#define DMU_POOL_OBSOLETE_BPOBJ "com.delphix:obsolete_bpobj"
	#define DMU_POOL_CONDENSING_INDIRECT "com.delphix:condensing_indirect"
	#define DMU_POOL_ZPOOL_CHECKPOINT "com.delphix:zpool_checkpoint"
	#define DMU_POOL_LOG_SPACEMAP_ZAP "com.delphix:log_spacemap_zap"
	#define DMU_POOL_DELETED_CLONES "com.delphix:deleted_clones"

	/*
	* Allocate an object from this objset. The range of object numbers
	* available is (0, DN_MAX_OBJECT). Object 0 is the meta-dnode.
	*
	* The transaction must be assigned to a txg. The newly allocated
	* object will be "held" in the transaction (ie. you can modify the
	* newly allocated object in this transaction).
	*
	* dmu_object_alloc() chooses an object and returns it in *objectp.
	*
	* dmu_object_claim() allocates a specific object number. If that
	* number is already allocated, it fails and returns EEXIST.
	*
	* Return 0 on success, or ENOSPC or EEXIST as specified above.
	*/
	uint64_t dmu_object_alloc(objset_t *os, dmu_object_type_t ot,
	int blocksize, dmu_object_type_t bonus_type, int bonus_len, dmu_tx_t *tx);
	uint64_t dmu_object_alloc_ibs(objset_t *os, dmu_object_type_t ot, int blocksize,
	int indirect_blockshift,
	dmu_object_type_t bonustype, int bonuslen, dmu_tx_t *tx);
	uint64_t dmu_object_alloc_dnsize(objset_t *os, dmu_object_type_t ot,
	int blocksize, dmu_object_type_t bonus_type, int bonus_len,
	int dnodesize, dmu_tx_t *tx);
	uint64_t dmu_object_alloc_hold(objset_t *os, dmu_object_type_t ot,
	int blocksize, int indirect_blockshift, dmu_object_type_t bonustype,
	int bonuslen, int dnodesize, dnode_t *allocated_dnode, void tag,
	dmu_tx_t *tx);
	int dmu_object_claim(objset_t *os, uint64_t object, dmu_object_type_t ot,
	int blocksize, dmu_object_type_t bonus_type, int bonus_len, dmu_tx_t *tx);
	int dmu_object_claim_dnsize(objset_t *os, uint64_t object, dmu_object_type_t ot,
	int blocksize, dmu_object_type_t bonus_type, int bonus_len,
	int dnodesize, dmu_tx_t *tx);
	int dmu_object_reclaim(objset_t *os, uint64_t object, dmu_object_type_t ot,
	int blocksize, dmu_object_type_t bonustype, int bonuslen, dmu_tx_t *txp);
	int dmu_object_reclaim_dnsize(objset_t *os, uint64_t object,
	dmu_object_type_t ot, int blocksize, dmu_object_type_t bonustype,
	int bonuslen, int dnodesize, boolean_t keep_spill, dmu_tx_t *tx);
	int dmu_object_rm_spill(objset_t os, uint64_t object, dmu_tx_t tx);

	/*
	* Free an object from this objset.
	*
	* The object's data will be freed as well (ie. you don't need to call
	* dmu_free(object, 0, -1, tx)).
	*
	* The object need not be held in the transaction.
	*
	* If there are any holds on this object's buffers (via dmu_buf_hold()),
	* or tx holds on the object (via dmu_tx_hold_object()), you can not
	* free it; it fails and returns EBUSY.
	*
	* If the object is not allocated, it fails and returns ENOENT.
	*
	* Return 0 on success, or EBUSY or ENOENT as specified above.
	*/
	int dmu_object_free(objset_t os, uint64_t object, dmu_tx_t tx);

	/*
	* Find the next allocated or free object.
	*
	* The objectp parameter is in-out. It will be updated to be the next
	* object which is allocated. Ignore objects which have not been
	* modified since txg.
	*
	* XXX Can only be called on a objset with no dirty data.
	*
	* Returns 0 on success, or ENOENT if there are no more objects.
	*/
	int dmu_object_next(objset_t os, uint64_t objectp,
	boolean_t hole, uint64_t txg);

	/*
	* Set the number of levels on a dnode. nlevels must be greater than the
	* current number of levels or an EINVAL will be returned.
	*/
	int dmu_object_set_nlevels(objset_t *os, uint64_t object, int nlevels,
	dmu_tx_t *tx);

	/*
	* Set the data blocksize for an object.
	*
	* The object cannot have any blocks allocated beyond the first. If
	* the first block is allocated already, the new size must be greater
	* than the current block size. If these conditions are not met,
	* ENOTSUP will be returned.
	*
	* Returns 0 on success, or EBUSY if there are any holds on the object
	* contents, or ENOTSUP as described above.
	*/
	int dmu_object_set_blocksize(objset_t *os, uint64_t object, uint64_t size,
	int ibs, dmu_tx_t *tx);

	/*
	* Manually set the maxblkid on a dnode. This will adjust nlevels accordingly
	* to accommodate the change. When calling this function, the caller must
	* ensure that the object's nlevels can sufficiently support the new maxblkid.
	*/
	int dmu_object_set_maxblkid(objset_t *os, uint64_t object, uint64_t maxblkid,
	dmu_tx_t *tx);

	/*
	* Set the checksum property on a dnode. The new checksum algorithm will
	* apply to all newly written blocks; existing blocks will not be affected.
	*/
	void dmu_object_set_checksum(objset_t *os, uint64_t object, uint8_t checksum,
	dmu_tx_t *tx);

	/*
	* Set the compress property on a dnode. The new compression algorithm will
	* apply to all newly written blocks; existing blocks will not be affected.
	*/
	void dmu_object_set_compress(objset_t *os, uint64_t object, uint8_t compress,
	dmu_tx_t *tx);

	void dmu_write_embedded(objset_t *os, uint64_t object, uint64_t offset,
	void *data, uint8_t etype, uint8_t comp, int uncompressed_size,
	int compressed_size, int byteorder, dmu_tx_t *tx);
	void dmu_redact(objset_t *os, uint64_t object, uint64_t offset, uint64_t size,
	dmu_tx_t *tx);

	/*
	* Decide how to write a block: checksum, compression, number of copies, etc.
	*/
	#define WP_NOFILL 0x1
	#define WP_DMU_SYNC 0x2
	#define WP_SPILL 0x4

	void dmu_write_policy(objset_t os, dnode_t dn, int level, int wp,
	struct zio_prop *zp);

	/*
	* The bonus data is accessed more or less like a regular buffer.
	* You must dmu_bonus_hold() to get the buffer, which will give you a
	* dmu_buf_t with db_offset==-1ULL, and db_size = the size of the bonus
	* data. As with any normal buffer, you must call dmu_buf_will_dirty()
	* before modifying it, and the
	* object must be held in an assigned transaction before calling
	* dmu_buf_will_dirty. You may use dmu_buf_set_user() on the bonus
	* buffer as well. You must release what you hold with dmu_buf_rele().
	*
	* Returns ENOENT, EIO, or 0.
	*/
	int dmu_bonus_hold(objset_t os, uint64_t object, void tag, dmu_buf_t **dbp);
	int dmu_bonus_hold_by_dnode(dnode_t dn, void tag, dmu_buf_t **dbp,
	uint32_t flags);
	int dmu_bonus_max(void);
	int dmu_set_bonus(dmu_buf_t , int, dmu_tx_t );
	int dmu_set_bonustype(dmu_buf_t , dmu_object_type_t, dmu_tx_t );
	dmu_object_type_t dmu_get_bonustype(dmu_buf_t *);
	int dmu_rm_spill(objset_t , uint64_t, dmu_tx_t );

	/*
	* Special spill buffer support used by "SA" framework
	*/

	int dmu_spill_hold_by_bonus(dmu_buf_t bonus, uint32_t flags, void tag,
	dmu_buf_t **dbp);
	int dmu_spill_hold_by_dnode(dnode_t *dn, uint32_t flags,
	void tag, dmu_buf_t *dbp);
	int dmu_spill_hold_existing(dmu_buf_t bonus, void tag, dmu_buf_t **dbp);

	/*
	* Obtain the DMU buffer from the specified object which contains the
	* specified offset. dmu_buf_hold() puts a "hold" on the buffer, so
	* that it will remain in memory. You must release the hold with
	* dmu_buf_rele(). You must not access the dmu_buf_t after releasing
	* what you hold. You must have a hold on any dmu_buf_t* you pass to the DMU.
	*
	* You must call dmu_buf_read, dmu_buf_will_dirty, or dmu_buf_will_fill
	* on the returned buffer before reading or writing the buffer's
	* db_data. The comments for those routines describe what particular
	* operations are valid after calling them.
	*
	* The object number must be a valid, allocated object number.
	*/
	int dmu_buf_hold(objset_t *os, uint64_t object, uint64_t offset,
	void tag, dmu_buf_t *, int flags);
	int dmu_buf_hold_by_dnode(dnode_t *dn, uint64_t offset,
	void tag, dmu_buf_t *dbp, int flags);
	int dmu_buf_hold_array_by_dnode(dnode_t *dn, uint64_t offset,
	uint64_t length, boolean_t read, void tag, int numbufsp,
	dmu_buf_t ***dbpp, uint32_t flags);
	/*
	* Add a reference to a dmu buffer that has already been held via
	* dmu_buf_hold() in the current context.
	*/
	void dmu_buf_add_ref(dmu_buf_t db, void tag);

	/*
	* Attempt to add a reference to a dmu buffer that is in an unknown state,
	* using a pointer that may have been invalidated by eviction processing.
	* The request will succeed if the passed in dbuf still represents the
	* same os/object/blkid, is ineligible for eviction, and has at least
	* one hold by a user other than the syncer.
	*/
	boolean_t dmu_buf_try_add_ref(dmu_buf_t , objset_t os, uint64_t object,
	uint64_t blkid, void *tag);

	void dmu_buf_rele(dmu_buf_t db, void tag);
	uint64_t dmu_buf_refcount(dmu_buf_t *db);
	uint64_t dmu_buf_user_refcount(dmu_buf_t *db);

	/*
	* dmu_buf_hold_array holds the DMU buffers which contain all bytes in a
	* range of an object. A pointer to an array of dmu_buf_t*'s is
	* returned (in *dbpp).
	*
	* dmu_buf_rele_array releases the hold on an array of dmu_buf_t*'s, and
	* frees the array. The hold on the array of buffers MUST be released
	* with dmu_buf_rele_array. You can NOT release the hold on each buffer
	* individually with dmu_buf_rele.
	*/
	int dmu_buf_hold_array_by_bonus(dmu_buf_t *db, uint64_t offset,
	uint64_t length, boolean_t read, void *tag,
	int numbufsp, dmu_buf_t **dbpp);
	void dmu_buf_rele_array(dmu_buf_t *, int numbufs, void tag);

	typedef void dmu_buf_evict_func_t(void *user_ptr);

	/*
	* A DMU buffer user object may be associated with a dbuf for the
	* duration of its lifetime. This allows the user of a dbuf (client)
	* to attach private data to a dbuf (e.g. in-core only data such as a
	* dnode_children_t, zap_t, or zap_leaf_t) and be optionally notified
	* when that dbuf has been evicted. Clients typically respond to the
	* eviction notification by freeing their private data, thus ensuring
	* the same lifetime for both dbuf and private data.
	*
	* The mapping from a dmu_buf_user_t to any client private data is the
	* client's responsibility. All current consumers of the API with private
	* data embed a dmu_buf_user_t as the first member of the structure for
	* their private data. This allows conversions between the two types
	* with a simple cast. Since the DMU buf user API never needs access
	* to the private data, other strategies can be employed if necessary
	* or convenient for the client (e.g. using container_of() to do the
	* conversion for private data that cannot have the dmu_buf_user_t as
	* its first member).
	*
	* Eviction callbacks are executed without the dbuf mutex held or any
	* other type of mechanism to guarantee that the dbuf is still available.
	* For this reason, users must assume the dbuf has already been freed
	* and not reference the dbuf from the callback context.
	*
	* Users requesting "immediate eviction" are notified as soon as the dbuf
	* is only referenced by dirty records (dirties == holds). Otherwise the
	* notification occurs after eviction processing for the dbuf begins.
	*/
	typedef struct dmu_buf_user {
	/*
	* Asynchronous user eviction callback state.
	*/
	taskq_ent_t dbu_tqent;

	/*
	* This instance's eviction function pointers.
	*
	* dbu_evict_func_sync is called synchronously and then
	* dbu_evict_func_async is executed asynchronously on a taskq.
	*/
	dmu_buf_evict_func_t *dbu_evict_func_sync;
	dmu_buf_evict_func_t *dbu_evict_func_async;
	#ifdef ZFS_DEBUG
	/*
	* Pointer to user's dbuf pointer. NULL for clients that do
	* not associate a dbuf with their user data.
	*
	* The dbuf pointer is cleared upon eviction so as to catch
	* use-after-evict bugs in clients.
	*/
	dmu_buf_t **dbu_clear_on_evict_dbufp;
	#endif
	} dmu_buf_user_t;

	/*
	* Initialize the given dmu_buf_user_t instance with the eviction function
	* evict_func, to be called when the user is evicted.
	*
	* NOTE: This function should only be called once on a given dmu_buf_user_t.
	* To allow enforcement of this, dbu must already be zeroed on entry.
	*/
	/ARGSUSED/
	static inline void
	dmu_buf_init_user(dmu_buf_user_t dbu, dmu_buf_evict_func_t evict_func_sync,
	dmu_buf_evict_func_t *evict_func_async,
	dmu_buf_t **clear_on_evict_dbufp __maybe_unused)
	{
	ASSERT(dbu->dbu_evict_func_sync == NULL);
	ASSERT(dbu->dbu_evict_func_async == NULL);

	/* must have at least one evict func */
	IMPLY(evict_func_sync == NULL, evict_func_async != NULL);
	dbu->dbu_evict_func_sync = evict_func_sync;
	dbu->dbu_evict_func_async = evict_func_async;
	taskq_init_ent(&dbu->dbu_tqent);
	#ifdef ZFS_DEBUG
	dbu->dbu_clear_on_evict_dbufp = clear_on_evict_dbufp;
	#endif
	}

	/*
	* Attach user data to a dbuf and mark it for normal (when the dbuf's
	* data is cleared or its reference count goes to zero) eviction processing.
	*
	* Returns NULL on success, or the existing user if another user currently
	* owns the buffer.
	*/
	void dmu_buf_set_user(dmu_buf_t db, dmu_buf_user_t *user);

	/*
	* Attach user data to a dbuf and mark it for immediate (its dirty and
	* reference counts are equal) eviction processing.
	*
	* Returns NULL on success, or the existing user if another user currently
	* owns the buffer.
	*/
	void dmu_buf_set_user_ie(dmu_buf_t db, dmu_buf_user_t *user);

	/*
	* Replace the current user of a dbuf.
	*
	* If given the current user of a dbuf, replaces the dbuf's user with
	* "new_user" and returns the user data pointer that was replaced.
	* Otherwise returns the current, and unmodified, dbuf user pointer.
	*/
	void dmu_buf_replace_user(dmu_buf_t db,
	dmu_buf_user_t old_user, dmu_buf_user_t new_user);

	/*
	* Remove the specified user data for a DMU buffer.
	*
	* Returns the user that was removed on success, or the current user if
	* another user currently owns the buffer.
	*/
	void dmu_buf_remove_user(dmu_buf_t db, dmu_buf_user_t *user);

	/*
	* Returns the user data (dmu_buf_user_t *) associated with this dbuf.
	*/
	void dmu_buf_get_user(dmu_buf_t db);

	objset_t dmu_buf_get_objset(dmu_buf_t db);
	dnode_t dmu_buf_dnode_enter(dmu_buf_t db);
	void dmu_buf_dnode_exit(dmu_buf_t *db);

	/* Block until any in-progress dmu buf user evictions complete. */
	void dmu_buf_user_evict_wait(void);

	/*
	* Returns the blkptr associated with this dbuf, or NULL if not set.
	*/
	struct blkptr dmu_buf_get_blkptr(dmu_buf_t db);

	/*
	* Indicate that you are going to modify the buffer's data (db_data).
	*
	* The transaction (tx) must be assigned to a txg (ie. you've called
	* dmu_tx_assign()). The buffer's object must be held in the tx
	* (ie. you've called dmu_tx_hold_object(tx, db->db_object)).
	*/
	void dmu_buf_will_dirty(dmu_buf_t db, dmu_tx_t tx);
	boolean_t dmu_buf_is_dirty(dmu_buf_t db, dmu_tx_t tx);
	void dmu_buf_set_crypt_params(dmu_buf_t *db_fake, boolean_t byteorder,
	const uint8_t salt, const uint8_t iv, const uint8_t mac, dmu_tx_t tx);

	/*
	* You must create a transaction, then hold the objects which you will
	* (or might) modify as part of this transaction. Then you must assign
	* the transaction to a transaction group. Once the transaction has
	* been assigned, you can modify buffers which belong to held objects as
	* part of this transaction. You can't modify buffers before the
	* transaction has been assigned; you can't modify buffers which don't
	* belong to objects which this transaction holds; you can't hold
	* objects once the transaction has been assigned. You may hold an
	* object which you are going to free (with dmu_object_free()), but you
	* don't have to.
	*
	* You can abort the transaction before it has been assigned.
	*
	* Note that you may hold buffers (with dmu_buf_hold) at any time,
	* regardless of transaction state.
	*/

	#define DMU_NEW_OBJECT (-1ULL)
	#define DMU_OBJECT_END (-1ULL)

	dmu_tx_t dmu_tx_create(objset_t os);
	void dmu_tx_hold_write(dmu_tx_t *tx, uint64_t object, uint64_t off, int len);
	void dmu_tx_hold_write_by_dnode(dmu_tx_t tx, dnode_t dn, uint64_t off,
	int len);
	void dmu_tx_hold_free(dmu_tx_t *tx, uint64_t object, uint64_t off,
	uint64_t len);
	void dmu_tx_hold_free_by_dnode(dmu_tx_t tx, dnode_t dn, uint64_t off,
	uint64_t len);
	void dmu_tx_hold_zap(dmu_tx_t tx, uint64_t object, int add, const char name);
	void dmu_tx_hold_zap_by_dnode(dmu_tx_t tx, dnode_t dn, int add,
	const char *name);
	void dmu_tx_hold_bonus(dmu_tx_t *tx, uint64_t object);
	void dmu_tx_hold_bonus_by_dnode(dmu_tx_t tx, dnode_t dn);
	void dmu_tx_hold_spill(dmu_tx_t *tx, uint64_t object);
	void dmu_tx_hold_sa(dmu_tx_t tx, struct sa_handle hdl, boolean_t may_grow);
	void dmu_tx_hold_sa_create(dmu_tx_t *tx, int total_size);
	void dmu_tx_abort(dmu_tx_t *tx);
	int dmu_tx_assign(dmu_tx_t *tx, uint64_t txg_how);
	void dmu_tx_wait(dmu_tx_t *tx);
	void dmu_tx_commit(dmu_tx_t *tx);
	void dmu_tx_mark_netfree(dmu_tx_t *tx);

	/*
	* To register a commit callback, dmu_tx_callback_register() must be called.
	*
	* dcb_data is a pointer to caller private data that is passed on as a
	* callback parameter. The caller is responsible for properly allocating and
	* freeing it.
	*
	* When registering a callback, the transaction must be already created, but
	* it cannot be committed or aborted. It can be assigned to a txg or not.
	*
	* The callback will be called after the transaction has been safely written
	* to stable storage and will also be called if the dmu_tx is aborted.
	* If there is any error which prevents the transaction from being committed to
	* disk, the callback will be called with a value of error != 0.
	*
	* When multiple callbacks are registered to the transaction, the callbacks
	* will be called in reverse order to let Lustre, the only user of commit
	* callback currently, take the fast path of its commit callback handling.
	*/
	typedef void dmu_tx_callback_func_t(void *dcb_data, int error);

	void dmu_tx_callback_register(dmu_tx_t tx, dmu_tx_callback_func_t dcb_func,
	void *dcb_data);
	void dmu_tx_do_callbacks(list_t *cb_list, int error);

	/*
	* Free up the data blocks for a defined range of a file. If size is
	* -1, the range from offset to end-of-file is freed.
	*/
	int dmu_free_range(objset_t *os, uint64_t object, uint64_t offset,
	uint64_t size, dmu_tx_t *tx);
	int dmu_free_long_range(objset_t *os, uint64_t object, uint64_t offset,
	uint64_t size);
	int dmu_free_long_object(objset_t *os, uint64_t object);

	/*
	* Convenience functions.
	*
	* Canfail routines will return 0 on success, or an errno if there is a
	* nonrecoverable I/O error.
	*/
	#define DMU_READ_PREFETCH 0 /* prefetch */
	#define DMU_READ_NO_PREFETCH 1 /* don't prefetch */
	#define DMU_READ_NO_DECRYPT 2 /* don't decrypt */
	int dmu_read(objset_t *os, uint64_t object, uint64_t offset, uint64_t size,
	void *buf, uint32_t flags);
	int dmu_read_by_dnode(dnode_t dn, uint64_t offset, uint64_t size, void buf,
	uint32_t flags);
	void dmu_write(objset_t *os, uint64_t object, uint64_t offset, uint64_t size,
	const void buf, dmu_tx_t tx);
	void dmu_write_by_dnode(dnode_t *dn, uint64_t offset, uint64_t size,
	const void buf, dmu_tx_t tx);
	void dmu_prealloc(objset_t *os, uint64_t object, uint64_t offset, uint64_t size,
	dmu_tx_t *tx);
	#ifdef _KERNEL
	int dmu_read_uio(objset_t os, uint64_t object, struct uio uio, uint64_t size);
	int dmu_read_uio_dbuf(dmu_buf_t zdb, struct uio uio, uint64_t size);
	int dmu_read_uio_dnode(dnode_t dn, struct uio uio, uint64_t size);
	int dmu_write_uio(objset_t os, uint64_t object, struct uio uio, uint64_t size,
	dmu_tx_t *tx);
	int dmu_write_uio_dbuf(dmu_buf_t zdb, struct uio uio, uint64_t size,
	dmu_tx_t *tx);
	int dmu_write_uio_dnode(dnode_t dn, struct uio uio, uint64_t size,
	dmu_tx_t *tx);
	#endif
	struct arc_buf dmu_request_arcbuf(dmu_buf_t handle, int size);
	void dmu_return_arcbuf(struct arc_buf *buf);
	int dmu_assign_arcbuf_by_dnode(dnode_t *dn, uint64_t offset,
	struct arc_buf buf, dmu_tx_t tx);
	int dmu_assign_arcbuf_by_dbuf(dmu_buf_t *handle, uint64_t offset,
	struct arc_buf buf, dmu_tx_t tx);
	#define dmu_assign_arcbuf dmu_assign_arcbuf_by_dbuf
	#ifdef HAVE_UIO_ZEROCOPY
	int dmu_xuio_init(struct xuio *uio, int niov);
	void dmu_xuio_fini(struct xuio *uio);
	int dmu_xuio_add(struct xuio uio, struct arc_buf abuf, offset_t off,
	size_t n);
	int dmu_xuio_cnt(struct xuio *uio);
	struct arc_buf dmu_xuio_arcbuf(struct xuio uio, int i);
	void dmu_xuio_clear(struct xuio *uio, int i);
	#endif /* HAVE_UIO_ZEROCOPY */
	void xuio_stat_wbuf_copied(void);
	void xuio_stat_wbuf_nocopy(void);

	extern int zfs_prefetch_disable;
	extern int zfs_max_recordsize;

	/*
	* Asynchronously try to read in the data.
	*/
	void dmu_prefetch(objset_t *os, uint64_t object, int64_t level, uint64_t offset,
	uint64_t len, enum zio_priority pri);

	typedef struct dmu_object_info {
	/* All sizes are in bytes unless otherwise indicated. */
	uint32_t doi_data_block_size;
	uint32_t doi_metadata_block_size;
	dmu_object_type_t doi_type;
	dmu_object_type_t doi_bonus_type;
	uint64_t doi_bonus_size;
	uint8_t doi_indirection; /* 2 = dnode->indirect->data */
	uint8_t doi_checksum;
	uint8_t doi_compress;
	uint8_t doi_nblkptr;
	uint8_t doi_pad[4];
	uint64_t doi_dnodesize;
	uint64_t doi_physical_blocks_512; /* data + metadata, 512b blks */
	uint64_t doi_max_offset;
	uint64_t doi_fill_count; /* number of non-empty blocks */
	} dmu_object_info_t;

	typedef void (const arc_byteswap_func_t)(void buf, size_t size);

	typedef struct dmu_object_type_info {
	dmu_object_byteswap_t ot_byteswap;
	boolean_t ot_metadata;
	boolean_t ot_dbuf_metadata_cache;
	boolean_t ot_encrypt;
	char *ot_name;
	} dmu_object_type_info_t;

	typedef const struct dmu_object_byteswap_info {
	arc_byteswap_func_t ob_func;
	char *ob_name;
	} dmu_object_byteswap_info_t;

	extern const dmu_object_type_info_t dmu_ot[DMU_OT_NUMTYPES];
	extern const dmu_object_byteswap_info_t dmu_ot_byteswap[DMU_BSWAP_NUMFUNCS];

	/*
	* Get information on a DMU object.
	*
	* Return 0 on success or ENOENT if object is not allocated.
	*
	* If doi is NULL, just indicates whether the object exists.
	*/
	int dmu_object_info(objset_t os, uint64_t object, dmu_object_info_t doi);
	void __dmu_object_info_from_dnode(struct dnode dn, dmu_object_info_t doi);
	/* Like dmu_object_info, but faster if you have a held dnode in hand. */
	void dmu_object_info_from_dnode(dnode_t dn, dmu_object_info_t doi);
	/* Like dmu_object_info, but faster if you have a held dbuf in hand. */
	void dmu_object_info_from_db(dmu_buf_t db, dmu_object_info_t doi);
	/*
	* Like dmu_object_info_from_db, but faster still when you only care about
	* the size.
	*/
	void dmu_object_size_from_db(dmu_buf_t db, uint32_t blksize,
	u_longlong_t *nblk512);

	void dmu_object_dnsize_from_db(dmu_buf_t db, int dnsize);

	typedef struct dmu_objset_stats {
	uint64_t dds_num_clones; /* number of clones of this */
	uint64_t dds_creation_txg;
	uint64_t dds_guid;
	dmu_objset_type_t dds_type;
	uint8_t dds_is_snapshot;
	uint8_t dds_inconsistent;
	uint8_t dds_redacted;
	char dds_origin[ZFS_MAX_DATASET_NAME_LEN];
	} dmu_objset_stats_t;

	/*
	* Get stats on a dataset.
	*/
	void dmu_objset_fast_stat(objset_t os, dmu_objset_stats_t stat);

	/*
	* Add entries to the nvlist for all the objset's properties. See
	* zfs_prop_table[] and zfs(1m) for details on the properties.
	*/
	void dmu_objset_stats(objset_t os, struct nvlist nv);

	/*
	* Get the space usage statistics for statvfs().
	*
	* refdbytes is the amount of space "referenced" by this objset.
	* availbytes is the amount of space available to this objset, taking
	* into account quotas & reservations, assuming that no other objsets
	* use the space first. These values correspond to the 'referenced' and
	* 'available' properties, described in the zfs(1m) manpage.
	*
	* usedobjs and availobjs are the number of objects currently allocated,
	* and available.
	*/
	void dmu_objset_space(objset_t os, uint64_t refdbytesp, uint64_t *availbytesp,
	uint64_t usedobjsp, uint64_t availobjsp);

	/*
	* The fsid_guid is a 56-bit ID that can change to avoid collisions.
	* (Contrast with the ds_guid which is a 64-bit ID that will never
	* change, so there is a small probability that it will collide.)
	*/
	uint64_t dmu_objset_fsid_guid(objset_t *os);

	/*
	* Get the [cm]time for an objset's snapshot dir
	*/
	inode_timespec_t dmu_objset_snap_cmtime(objset_t *os);

	int dmu_objset_is_snapshot(objset_t *os);

	extern struct spa dmu_objset_spa(objset_t os);
	extern struct zilog dmu_objset_zil(objset_t os);
	extern struct dsl_pool dmu_objset_pool(objset_t os);
	extern struct dsl_dataset dmu_objset_ds(objset_t os);
	extern void dmu_objset_name(objset_t os, char buf);
	extern dmu_objset_type_t dmu_objset_type(objset_t *os);
	extern uint64_t dmu_objset_id(objset_t *os);
	extern uint64_t dmu_objset_dnodesize(objset_t *os);
	extern zfs_sync_type_t dmu_objset_syncprop(objset_t *os);
	extern zfs_logbias_op_t dmu_objset_logbias(objset_t *os);
	extern int dmu_objset_blksize(objset_t *os);
	extern int dmu_snapshot_list_next(objset_t os, int namelen, char name,
	uint64_t id, uint64_t offp, boolean_t *case_conflict);
	extern int dmu_snapshot_lookup(objset_t os, const char name, uint64_t *val);
	-extern int dmu_snapshot_realname(objset_t os, char name, char *real,
	+extern int dmu_snapshot_realname(objset_t os, const char name, char *real,
	int maxlen, boolean_t *conflict);
	extern int dmu_dir_list_next(objset_t os, int namelen, char name,
	uint64_t idp, uint64_t offp);

	typedef struct zfs_file_info {
	uint64_t zfi_user;
	uint64_t zfi_group;
	uint64_t zfi_project;
	uint64_t zfi_generation;
	} zfs_file_info_t;

	typedef int file_info_cb_t(dmu_object_type_t bonustype, const void *data,
	struct zfs_file_info *zoi);
	extern void dmu_objset_register_type(dmu_objset_type_t ost,
	file_info_cb_t *cb);
	extern void dmu_objset_set_user(objset_t os, void user_ptr);
	extern void dmu_objset_get_user(objset_t os);

	/*
	* Return the txg number for the given assigned transaction.
	*/
	uint64_t dmu_tx_get_txg(dmu_tx_t *tx);

	/*
	* Synchronous write.
	* If a parent zio is provided this function initiates a write on the
	* provided buffer as a child of the parent zio.
	* In the absence of a parent zio, the write is completed synchronously.
	* At write completion, blk is filled with the bp of the written block.
	* Note that while the data covered by this function will be on stable
	* storage when the write completes this new data does not become a
	* permanent part of the file until the associated transaction commits.
	*/

	/*
	* {zfs,zvol,ztest}_get_done() args
	*/
	typedef struct zgd {
	struct lwb *zgd_lwb;
	struct blkptr *zgd_bp;
	dmu_buf_t *zgd_db;
	struct zfs_locked_range *zgd_lr;
	void *zgd_private;
	} zgd_t;

	typedef void dmu_sync_cb_t(zgd_t *arg, int error);
	int dmu_sync(struct zio zio, uint64_t txg, dmu_sync_cb_t done, zgd_t *zgd);

	/*
	* Find the next hole or data block in file starting at *off
	* Return found offset in *off. Return ESRCH for end of file.
	*/
	int dmu_offset_next(objset_t *os, uint64_t object, boolean_t hole,
	uint64_t *off);

	/*
	* Initial setup and final teardown.
	*/
	extern void dmu_init(void);
	extern void dmu_fini(void);

	typedef void (dmu_traverse_cb_t)(objset_t os, void arg, struct blkptr bp,
	uint64_t object, uint64_t offset, int len);
	void dmu_traverse_objset(objset_t *os, uint64_t txg_start,
	dmu_traverse_cb_t cb, void *arg);

	int dmu_diff(const char tosnap_name, const char fromsnap_name,
	zfs_file_t fp, offset_t offp);

	/* CRC64 table */
	#define ZFS_CRC64_POLY 0xC96C5795D7870F42ULL /* ECMA-182, reflected form */
	extern uint64_t zfs_crc64_table[256];

	#ifdef __cplusplus
	}
	#endif

	#endif /* _SYS_DMU_H */
	Index: vendor-sys/openzfs/dist/include/sys/mod.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/sys/mod.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/sys/mod.h (revision 366774)
	@@ -1,41 +1,40 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/
	#ifndef _SYS_MOD_H
	#define _SYS_MOD_H

	#ifdef _KERNEL
	#include <sys/mod_os.h>
	#else
	/*
	* Exported symbols
	*/
	#define EXPORT_SYMBOL(x)

	#define ZFS_MODULE_DESCRIPTION(s)
	#define ZFS_MODULE_AUTHOR(s)
	#define ZFS_MODULE_LICENSE(s)
	#define ZFS_MODULE_VERSION(s)
	#endif

	#endif /* SYS_MOD_H */
	Index: vendor-sys/openzfs/dist/include/sys/nvpair.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/sys/nvpair.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/sys/nvpair.h (revision 366774)
	@@ -1,358 +1,358 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2012, 2018 by Delphix. All rights reserved.
	*/

	#ifndef _SYS_NVPAIR_H
	#define _SYS_NVPAIR_H

	#include <sys/types.h>
	#include <sys/time.h>
	#include <sys/errno.h>

	#ifdef __cplusplus
	extern "C" {
	#endif

	typedef enum {
	DATA_TYPE_DONTCARE = -1,
	DATA_TYPE_UNKNOWN = 0,
	DATA_TYPE_BOOLEAN,
	DATA_TYPE_BYTE,
	DATA_TYPE_INT16,
	DATA_TYPE_UINT16,
	DATA_TYPE_INT32,
	DATA_TYPE_UINT32,
	DATA_TYPE_INT64,
	DATA_TYPE_UINT64,
	DATA_TYPE_STRING,
	DATA_TYPE_BYTE_ARRAY,
	DATA_TYPE_INT16_ARRAY,
	DATA_TYPE_UINT16_ARRAY,
	DATA_TYPE_INT32_ARRAY,
	DATA_TYPE_UINT32_ARRAY,
	DATA_TYPE_INT64_ARRAY,
	DATA_TYPE_UINT64_ARRAY,
	DATA_TYPE_STRING_ARRAY,
	DATA_TYPE_HRTIME,
	DATA_TYPE_NVLIST,
	DATA_TYPE_NVLIST_ARRAY,
	DATA_TYPE_BOOLEAN_VALUE,
	DATA_TYPE_INT8,
	DATA_TYPE_UINT8,
	DATA_TYPE_BOOLEAN_ARRAY,
	DATA_TYPE_INT8_ARRAY,
	-#if !defined(_KERNEL)
	+#if !defined(_KERNEL) && !defined(_STANDALONE)
	DATA_TYPE_UINT8_ARRAY,
	DATA_TYPE_DOUBLE
	#else
	DATA_TYPE_UINT8_ARRAY
	#endif
	} data_type_t;

	typedef struct nvpair {
	int32_t nvp_size; /* size of this nvpair */
	int16_t nvp_name_sz; /* length of name string */
	int16_t nvp_reserve; /* not used */
	int32_t nvp_value_elem; /* number of elements for array types */
	data_type_t nvp_type; /* type of value */
	/* name string */
	/* aligned ptr array for string arrays */
	/* aligned array of data for value */
	} nvpair_t;

	/* nvlist header */
	typedef struct nvlist {
	int32_t nvl_version;
	uint32_t nvl_nvflag; /* persistent flags */
	uint64_t nvl_priv; /* ptr to private data if not packed */
	uint32_t nvl_flag;
	int32_t nvl_pad; /* currently not used, for alignment */
	} nvlist_t;

	/* nvp implementation version */
	#define NV_VERSION 0

	/* nvlist pack encoding */
	#define NV_ENCODE_NATIVE 0
	#define NV_ENCODE_XDR 1

	/* nvlist persistent unique name flags, stored in nvl_nvflags */
	#define NV_UNIQUE_NAME 0x1
	#define NV_UNIQUE_NAME_TYPE 0x2

	/* nvlist lookup pairs related flags */
	#define NV_FLAG_NOENTOK 0x1

	/* convenience macros */
	#define NV_ALIGN(x) (((ulong_t)(x) + 7ul) & ~7ul)
	#define NV_ALIGN4(x) (((x) + 3) & ~3)

	#define NVP_SIZE(nvp) ((nvp)->nvp_size)
	#define NVP_NAME(nvp) ((char *)(nvp) + sizeof (nvpair_t))
	#define NVP_TYPE(nvp) ((nvp)->nvp_type)
	#define NVP_NELEM(nvp) ((nvp)->nvp_value_elem)
	#define NVP_VALUE(nvp) ((char *)(nvp) + NV_ALIGN(sizeof (nvpair_t) \
	+ (nvp)->nvp_name_sz))

	#define NVL_VERSION(nvl) ((nvl)->nvl_version)
	#define NVL_SIZE(nvl) ((nvl)->nvl_size)
	#define NVL_FLAG(nvl) ((nvl)->nvl_flag)

	/* NV allocator framework */
	typedef struct nv_alloc_ops nv_alloc_ops_t;

	typedef struct nv_alloc {
	const nv_alloc_ops_t *nva_ops;
	void *nva_arg;
	} nv_alloc_t;

	struct nv_alloc_ops {
	int (nv_ao_init)(nv_alloc_t , va_list);
	void (nv_ao_fini)(nv_alloc_t );
	void (nv_ao_alloc)(nv_alloc_t *, size_t);
	void (nv_ao_free)(nv_alloc_t , void *, size_t);
	void (nv_ao_reset)(nv_alloc_t );
	};

	extern const nv_alloc_ops_t *nv_fixed_ops;
	extern nv_alloc_t *nv_alloc_nosleep;

	#if defined(_KERNEL)
	extern nv_alloc_t *nv_alloc_sleep;
	extern nv_alloc_t *nv_alloc_pushpage;
	#endif

	int nv_alloc_init(nv_alloc_t , const nv_alloc_ops_t , /* args */ ...);
	void nv_alloc_reset(nv_alloc_t *);
	void nv_alloc_fini(nv_alloc_t *);

	/* list management */
	int nvlist_alloc(nvlist_t **, uint_t, int);
	void nvlist_free(nvlist_t *);
	int nvlist_size(nvlist_t , size_t , int);
	int nvlist_pack(nvlist_t , char , size_t , int, int);
	int nvlist_unpack(char , size_t, nvlist_t *, int);
	int nvlist_dup(nvlist_t , nvlist_t *, int);
	int nvlist_merge(nvlist_t , nvlist_t , int);

	uint_t nvlist_nvflag(nvlist_t *);

	int nvlist_xalloc(nvlist_t *, uint_t, nv_alloc_t );
	int nvlist_xpack(nvlist_t , char , size_t , int, nv_alloc_t *);
	int nvlist_xunpack(char , size_t, nvlist_t , nv_alloc_t );
	int nvlist_xdup(nvlist_t , nvlist_t , nv_alloc_t );
	nv_alloc_t nvlist_lookup_nv_alloc(nvlist_t );

	int nvlist_add_nvpair(nvlist_t , nvpair_t );
	int nvlist_add_boolean(nvlist_t , const char );
	int nvlist_add_boolean_value(nvlist_t , const char , boolean_t);
	int nvlist_add_byte(nvlist_t , const char , uchar_t);
	int nvlist_add_int8(nvlist_t , const char , int8_t);
	int nvlist_add_uint8(nvlist_t , const char , uint8_t);
	int nvlist_add_int16(nvlist_t , const char , int16_t);
	int nvlist_add_uint16(nvlist_t , const char , uint16_t);
	int nvlist_add_int32(nvlist_t , const char , int32_t);
	int nvlist_add_uint32(nvlist_t , const char , uint32_t);
	int nvlist_add_int64(nvlist_t , const char , int64_t);
	int nvlist_add_uint64(nvlist_t , const char , uint64_t);
	int nvlist_add_string(nvlist_t , const char , const char *);
	int nvlist_add_nvlist(nvlist_t , const char , nvlist_t *);
	int nvlist_add_boolean_array(nvlist_t , const char , boolean_t *, uint_t);
	int nvlist_add_byte_array(nvlist_t , const char , uchar_t *, uint_t);
	int nvlist_add_int8_array(nvlist_t , const char , int8_t *, uint_t);
	int nvlist_add_uint8_array(nvlist_t , const char , uint8_t *, uint_t);
	int nvlist_add_int16_array(nvlist_t , const char , int16_t *, uint_t);
	int nvlist_add_uint16_array(nvlist_t , const char , uint16_t *, uint_t);
	int nvlist_add_int32_array(nvlist_t , const char , int32_t *, uint_t);
	int nvlist_add_uint32_array(nvlist_t , const char , uint32_t *, uint_t);
	int nvlist_add_int64_array(nvlist_t , const char , int64_t *, uint_t);
	int nvlist_add_uint64_array(nvlist_t , const char , uint64_t *, uint_t);
	int nvlist_add_string_array(nvlist_t , const char , char const , uint_t);
	int nvlist_add_nvlist_array(nvlist_t , const char , nvlist_t **, uint_t);
	int nvlist_add_hrtime(nvlist_t , const char , hrtime_t);
	-#if !defined(_KERNEL)
	+#if !defined(_KERNEL) && !defined(_STANDALONE)
	int nvlist_add_double(nvlist_t , const char , double);
	#endif

	int nvlist_remove(nvlist_t , const char , data_type_t);
	int nvlist_remove_all(nvlist_t , const char );
	int nvlist_remove_nvpair(nvlist_t , nvpair_t );

	int nvlist_lookup_boolean(nvlist_t , const char );
	int nvlist_lookup_boolean_value(nvlist_t , const char , boolean_t *);
	int nvlist_lookup_byte(nvlist_t , const char , uchar_t *);
	int nvlist_lookup_int8(nvlist_t , const char , int8_t *);
	int nvlist_lookup_uint8(nvlist_t , const char , uint8_t *);
	int nvlist_lookup_int16(nvlist_t , const char , int16_t *);
	int nvlist_lookup_uint16(nvlist_t , const char , uint16_t *);
	int nvlist_lookup_int32(nvlist_t , const char , int32_t *);
	int nvlist_lookup_uint32(nvlist_t , const char , uint32_t *);
	int nvlist_lookup_int64(nvlist_t , const char , int64_t *);
	int nvlist_lookup_uint64(nvlist_t , const char , uint64_t *);
	int nvlist_lookup_string(nvlist_t , const char , char **);
	int nvlist_lookup_nvlist(nvlist_t , const char , nvlist_t **);
	int nvlist_lookup_boolean_array(nvlist_t , const char ,
	boolean_t *, uint_t );
	int nvlist_lookup_byte_array(nvlist_t , const char , uchar_t *, uint_t );
	int nvlist_lookup_int8_array(nvlist_t , const char , int8_t *, uint_t );
	int nvlist_lookup_uint8_array(nvlist_t , const char , uint8_t *, uint_t );
	int nvlist_lookup_int16_array(nvlist_t , const char , int16_t *, uint_t );
	int nvlist_lookup_uint16_array(nvlist_t , const char , uint16_t *, uint_t );
	int nvlist_lookup_int32_array(nvlist_t , const char , int32_t *, uint_t );
	int nvlist_lookup_uint32_array(nvlist_t , const char , uint32_t *, uint_t );
	int nvlist_lookup_int64_array(nvlist_t , const char , int64_t *, uint_t );
	int nvlist_lookup_uint64_array(nvlist_t , const char , uint64_t *, uint_t );
	int nvlist_lookup_string_array(nvlist_t , const char , char **, uint_t );
	int nvlist_lookup_nvlist_array(nvlist_t , const char ,
	nvlist_t **, uint_t );
	int nvlist_lookup_hrtime(nvlist_t , const char , hrtime_t *);
	int nvlist_lookup_pairs(nvlist_t *, int, ...);
	-#if !defined(_KERNEL)
	+#if !defined(_KERNEL) && !defined(_STANDALONE)
	int nvlist_lookup_double(nvlist_t , const char , double *);
	#endif

	int nvlist_lookup_nvpair(nvlist_t , const char , nvpair_t **);
	int nvlist_lookup_nvpair_embedded_index(nvlist_t , const char , nvpair_t **,
	int , char *);
	boolean_t nvlist_exists(nvlist_t , const char );
	boolean_t nvlist_empty(nvlist_t *);

	/* processing nvpair */
	nvpair_t nvlist_next_nvpair(nvlist_t , nvpair_t *);
	nvpair_t nvlist_prev_nvpair(nvlist_t , nvpair_t *);
	char nvpair_name(nvpair_t );
	data_type_t nvpair_type(nvpair_t *);
	int nvpair_type_is_array(nvpair_t *);
	int nvpair_value_boolean_value(nvpair_t , boolean_t );
	int nvpair_value_byte(nvpair_t , uchar_t );
	int nvpair_value_int8(nvpair_t , int8_t );
	int nvpair_value_uint8(nvpair_t , uint8_t );
	int nvpair_value_int16(nvpair_t , int16_t );
	int nvpair_value_uint16(nvpair_t , uint16_t );
	int nvpair_value_int32(nvpair_t , int32_t );
	int nvpair_value_uint32(nvpair_t , uint32_t );
	int nvpair_value_int64(nvpair_t , int64_t );
	int nvpair_value_uint64(nvpair_t , uint64_t );
	int nvpair_value_string(nvpair_t , char *);
	int nvpair_value_nvlist(nvpair_t , nvlist_t *);
	int nvpair_value_boolean_array(nvpair_t , boolean_t , uint_t );
	int nvpair_value_byte_array(nvpair_t , uchar_t , uint_t );
	int nvpair_value_int8_array(nvpair_t , int8_t , uint_t );
	int nvpair_value_uint8_array(nvpair_t , uint8_t , uint_t );
	int nvpair_value_int16_array(nvpair_t , int16_t , uint_t );
	int nvpair_value_uint16_array(nvpair_t , uint16_t , uint_t );
	int nvpair_value_int32_array(nvpair_t , int32_t , uint_t );
	int nvpair_value_uint32_array(nvpair_t , uint32_t , uint_t );
	int nvpair_value_int64_array(nvpair_t , int64_t , uint_t );
	int nvpair_value_uint64_array(nvpair_t , uint64_t , uint_t );
	int nvpair_value_string_array(nvpair_t , char *, uint_t );
	int nvpair_value_nvlist_array(nvpair_t , nvlist_t *, uint_t );
	int nvpair_value_hrtime(nvpair_t , hrtime_t );
	-#if !defined(_KERNEL)
	+#if !defined(_KERNEL) && !defined(_STANDALONE)
	int nvpair_value_double(nvpair_t , double );
	#endif

	nvlist_t *fnvlist_alloc(void);
	void fnvlist_free(nvlist_t *);
	size_t fnvlist_size(nvlist_t *);
	char fnvlist_pack(nvlist_t , size_t *);
	void fnvlist_pack_free(char *, size_t);
	nvlist_t fnvlist_unpack(char , size_t);
	nvlist_t fnvlist_dup(nvlist_t );
	void fnvlist_merge(nvlist_t , nvlist_t );
	size_t fnvlist_num_pairs(nvlist_t *);

	void fnvlist_add_boolean(nvlist_t , const char );
	void fnvlist_add_boolean_value(nvlist_t , const char , boolean_t);
	void fnvlist_add_byte(nvlist_t , const char , uchar_t);
	void fnvlist_add_int8(nvlist_t , const char , int8_t);
	void fnvlist_add_uint8(nvlist_t , const char , uint8_t);
	void fnvlist_add_int16(nvlist_t , const char , int16_t);
	void fnvlist_add_uint16(nvlist_t , const char , uint16_t);
	void fnvlist_add_int32(nvlist_t , const char , int32_t);
	void fnvlist_add_uint32(nvlist_t , const char , uint32_t);
	void fnvlist_add_int64(nvlist_t , const char , int64_t);
	void fnvlist_add_uint64(nvlist_t , const char , uint64_t);
	void fnvlist_add_string(nvlist_t , const char , const char *);
	void fnvlist_add_nvlist(nvlist_t , const char , nvlist_t *);
	void fnvlist_add_nvpair(nvlist_t , nvpair_t );
	void fnvlist_add_boolean_array(nvlist_t , const char , boolean_t *, uint_t);
	void fnvlist_add_byte_array(nvlist_t , const char , uchar_t *, uint_t);
	void fnvlist_add_int8_array(nvlist_t , const char , int8_t *, uint_t);
	void fnvlist_add_uint8_array(nvlist_t , const char , uint8_t *, uint_t);
	void fnvlist_add_int16_array(nvlist_t , const char , int16_t *, uint_t);
	void fnvlist_add_uint16_array(nvlist_t , const char , uint16_t *, uint_t);
	void fnvlist_add_int32_array(nvlist_t , const char , int32_t *, uint_t);
	void fnvlist_add_uint32_array(nvlist_t , const char , uint32_t *, uint_t);
	void fnvlist_add_int64_array(nvlist_t , const char , int64_t *, uint_t);
	void fnvlist_add_uint64_array(nvlist_t , const char , uint64_t *, uint_t);
	void fnvlist_add_string_array(nvlist_t , const char , char * const *, uint_t);
	void fnvlist_add_nvlist_array(nvlist_t , const char , nvlist_t **, uint_t);

	void fnvlist_remove(nvlist_t , const char );
	void fnvlist_remove_nvpair(nvlist_t , nvpair_t );

	nvpair_t fnvlist_lookup_nvpair(nvlist_t , const char *);
	boolean_t fnvlist_lookup_boolean(nvlist_t , const char );
	boolean_t fnvlist_lookup_boolean_value(nvlist_t , const char );
	uchar_t fnvlist_lookup_byte(nvlist_t , const char );
	int8_t fnvlist_lookup_int8(nvlist_t , const char );
	int16_t fnvlist_lookup_int16(nvlist_t , const char );
	int32_t fnvlist_lookup_int32(nvlist_t , const char );
	int64_t fnvlist_lookup_int64(nvlist_t , const char );
	uint8_t fnvlist_lookup_uint8(nvlist_t , const char );
	uint16_t fnvlist_lookup_uint16(nvlist_t , const char );
	uint32_t fnvlist_lookup_uint32(nvlist_t , const char );
	uint64_t fnvlist_lookup_uint64(nvlist_t , const char );
	char fnvlist_lookup_string(nvlist_t , const char *);
	nvlist_t fnvlist_lookup_nvlist(nvlist_t , const char *);
	boolean_t fnvlist_lookup_boolean_array(nvlist_t , const char , uint_t );
	uchar_t fnvlist_lookup_byte_array(nvlist_t , const char , uint_t );
	int8_t fnvlist_lookup_int8_array(nvlist_t , const char , uint_t );
	uint8_t fnvlist_lookup_uint8_array(nvlist_t , const char , uint_t );
	int16_t fnvlist_lookup_int16_array(nvlist_t , const char , uint_t );
	uint16_t fnvlist_lookup_uint16_array(nvlist_t , const char , uint_t );
	int32_t fnvlist_lookup_int32_array(nvlist_t , const char , uint_t );
	uint32_t fnvlist_lookup_uint32_array(nvlist_t , const char , uint_t );
	int64_t fnvlist_lookup_int64_array(nvlist_t , const char , uint_t );
	uint64_t fnvlist_lookup_uint64_array(nvlist_t , const char , uint_t );

	boolean_t fnvpair_value_boolean_value(nvpair_t *nvp);
	uchar_t fnvpair_value_byte(nvpair_t *nvp);
	int8_t fnvpair_value_int8(nvpair_t *nvp);
	int16_t fnvpair_value_int16(nvpair_t *nvp);
	int32_t fnvpair_value_int32(nvpair_t *nvp);
	int64_t fnvpair_value_int64(nvpair_t *nvp);
	uint8_t fnvpair_value_uint8(nvpair_t *nvp);
	uint16_t fnvpair_value_uint16(nvpair_t *nvp);
	uint32_t fnvpair_value_uint32(nvpair_t *nvp);
	uint64_t fnvpair_value_uint64(nvpair_t *nvp);
	char fnvpair_value_string(nvpair_t nvp);
	nvlist_t fnvpair_value_nvlist(nvpair_t nvp);

	#ifdef __cplusplus
	}
	#endif

	#endif /* _SYS_NVPAIR_H */
	Index: vendor-sys/openzfs/dist/include/sys/pathname.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/sys/pathname.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/sys/pathname.h (revision 366774)
	@@ -1,72 +1,68 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/

	/*
	* Copyright (c) 1989, 2010, Oracle and/or its affiliates. All rights reserved.
	*/

	/* Copyright (c) 1983, 1984, 1985, 1986, 1987, 1988, 1989 AT&T */
	/* All Rights Reserved */

	/*
	* Portions of this source code were derived from Berkeley 4.3 BSD
	* under license from the Regents of the University of California.
	*/

	#ifndef _SYS_PATHNAME_H
	#define _SYS_PATHNAME_H

	#ifdef __cplusplus
	extern "C" {
	#endif

	/*
	* Pathname structure.
	* System calls that operate on path names gather the path name
	* from the system call into this structure and reduce it by
	* peeling off translated components. If a symbolic link is
	* encountered the new path name to be translated is also
	* assembled in this structure.
	*
	* By convention pn_buf is not changed once it's been set to point
	* to the underlying storage; routines which manipulate the pathname
	* do so by changing pn_path and pn_pathlen. pn_pathlen is redundant
	* since the path name is null-terminated, but is provided to make
	* some computations faster.
	*/
	typedef struct pathname {
	char pn_buf; / underlying storage */
	-#if 0 /* unused in ZoL */
	- char pn_path; / remaining pathname */
	- size_t pn_pathlen; /* remaining length */
	-#endif
	size_t pn_bufsize; /* total size of pn_buf */
	} pathname_t;

	extern void pn_alloc(struct pathname *);
	extern void pn_alloc_sz(struct pathname *, size_t);
	extern void pn_free(struct pathname *);

	#ifdef __cplusplus
	}
	#endif

	#endif /* _SYS_PATHNAME_H */
	Index: vendor-sys/openzfs/dist/include/sys/spa.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/sys/spa.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/sys/spa.h (revision 366774)
	@@ -1,1224 +1,1224 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2011, 2020 by Delphix. All rights reserved.
	* Copyright 2011 Nexenta Systems, Inc. All rights reserved.
	* Copyright (c) 2014 Spectra Logic Corporation, All rights reserved.
	* Copyright 2013 Saso Kiselkov. All rights reserved.
	* Copyright (c) 2014 Integros [integros.com]
	* Copyright 2017 Joyent, Inc.
	* Copyright (c) 2017, 2019, Datto Inc. All rights reserved.
	* Copyright (c) 2017, Intel Corporation.
	* Copyright (c) 2019, Allan Jude
	* Copyright (c) 2019, Klara Inc.
	*/

	#ifndef _SYS_SPA_H
	#define _SYS_SPA_H

	#include <sys/avl.h>
	#include <sys/zfs_context.h>
	#include <sys/kstat.h>
	#include <sys/nvpair.h>
	#include <sys/sysmacros.h>
	#include <sys/types.h>
	#include <sys/fs/zfs.h>
	#include <sys/spa_checksum.h>
	#include <sys/dmu.h>
	#include <sys/space_map.h>
	#include <sys/bitops.h>

	#ifdef __cplusplus
	extern "C" {
	#endif

	/*
	* Forward references that lots of things need.
	*/
	typedef struct spa spa_t;
	typedef struct vdev vdev_t;
	typedef struct metaslab metaslab_t;
	typedef struct metaslab_group metaslab_group_t;
	typedef struct metaslab_class metaslab_class_t;
	typedef struct zio zio_t;
	typedef struct zilog zilog_t;
	typedef struct spa_aux_vdev spa_aux_vdev_t;
	typedef struct ddt ddt_t;
	typedef struct ddt_entry ddt_entry_t;
	typedef struct zbookmark_phys zbookmark_phys_t;

	struct bpobj;
	struct bplist;
	struct dsl_pool;
	struct dsl_dataset;
	struct dsl_crypto_params;

	/*
	* We currently support block sizes from 512 bytes to 16MB.
	* The benefits of larger blocks, and thus larger IO, need to be weighed
	* against the cost of COWing a giant block to modify one byte, and the
	* large latency of reading or writing a large block.
	*
	* Note that although blocks up to 16MB are supported, the recordsize
	* property can not be set larger than zfs_max_recordsize (default 1MB).
	* See the comment near zfs_max_recordsize in dsl_dataset.c for details.
	*
	* Note that although the LSIZE field of the blkptr_t can store sizes up
	* to 32MB, the dnode's dn_datablkszsec can only store sizes up to
	* 32MB - 512 bytes. Therefore, we limit SPA_MAXBLOCKSIZE to 16MB.
	*/
	#define SPA_MINBLOCKSHIFT 9
	#define SPA_OLD_MAXBLOCKSHIFT 17
	#define SPA_MAXBLOCKSHIFT 24
	#define SPA_MINBLOCKSIZE (1ULL << SPA_MINBLOCKSHIFT)
	#define SPA_OLD_MAXBLOCKSIZE (1ULL << SPA_OLD_MAXBLOCKSHIFT)
	#define SPA_MAXBLOCKSIZE (1ULL << SPA_MAXBLOCKSHIFT)

	/*
	* Alignment Shift (ashift) is an immutable, internal top-level vdev property
	* which can only be set at vdev creation time. Physical writes are always done
	* according to it, which makes 2^ashift the smallest possible IO on a vdev.
	*
	* We currently allow values ranging from 512 bytes (2^9 = 512) to 64 KiB
	* (2^16 = 65,536).
	*/
	#define ASHIFT_MIN 9
	#define ASHIFT_MAX 16

	/*
	* Size of block to hold the configuration data (a packed nvlist)
	*/
	#define SPA_CONFIG_BLOCKSIZE (1ULL << 14)

	/*
	* The DVA size encodings for LSIZE and PSIZE support blocks up to 32MB.
	* The ASIZE encoding should be at least 64 times larger (6 more bits)
	* to support up to 4-way RAID-Z mirror mode with worst-case gang block
	* overhead, three DVAs per bp, plus one more bit in case we do anything
	* else that expands the ASIZE.
	*/
	#define SPA_LSIZEBITS 16 /* LSIZE up to 32M (2^16 * 512) */
	#define SPA_PSIZEBITS 16 /* PSIZE up to 32M (2^16 * 512) */
	#define SPA_ASIZEBITS 24 /* ASIZE up to 64 times larger */

	#define SPA_COMPRESSBITS 7
	#define SPA_VDEVBITS 24
	#define SPA_COMPRESSMASK ((1U << SPA_COMPRESSBITS) - 1)

	/*
	* All SPA data is represented by 128-bit data virtual addresses (DVAs).
	* The members of the dva_t should be considered opaque outside the SPA.
	*/
	typedef struct dva {
	uint64_t dva_word[2];
	} dva_t;


	/*
	* Some checksums/hashes need a 256-bit initialization salt. This salt is kept
	* secret and is suitable for use in MAC algorithms as the key.
	*/
	typedef struct zio_cksum_salt {
	uint8_t zcs_bytes[32];
	} zio_cksum_salt_t;

	/*
	* Each block is described by its DVAs, time of birth, checksum, etc.
	* The word-by-word, bit-by-bit layout of the blkptr is as follows:
	*
	* 64 56 48 40 32 24 16 8 0
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 0 \| pad \| vdev1 \| GRID \| ASIZE \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 1 \|G\| offset1 \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 2 \| pad \| vdev2 \| GRID \| ASIZE \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 3 \|G\| offset2 \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 4 \| pad \| vdev3 \| GRID \| ASIZE \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 5 \|G\| offset3 \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 6 \|BDX\|lvl\| type \| cksum \|E\| comp\| PSIZE \| LSIZE \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 7 \| padding \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 8 \| padding \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 9 \| physical birth txg \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* a \| logical birth txg \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* b \| fill count \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* c \| checksum[0] \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* d \| checksum[1] \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* e \| checksum[2] \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* f \| checksum[3] \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	*
	* Legend:
	*
	* vdev virtual device ID
	* offset offset into virtual device
	* LSIZE logical size
	* PSIZE physical size (after compression)
	* ASIZE allocated size (including RAID-Z parity and gang block headers)
	* GRID RAID-Z layout information (reserved for future use)
	* cksum checksum function
	* comp compression function
	* G gang block indicator
	* B byteorder (endianness)
	* D dedup
	* X encryption
	* E blkptr_t contains embedded data (see below)
	* lvl level of indirection
	* type DMU object type
	* phys birth txg when dva[0] was written; zero if same as logical birth txg
	* note that typically all the dva's would be written in this
	* txg, but they could be different if they were moved by
	* device removal.
	* log. birth transaction group in which the block was logically born
	* fill count number of non-zero blocks under this bp
	* checksum[4] 256-bit checksum of the data this bp describes
	*/

	/*
	* The blkptr_t's of encrypted blocks also need to store the encryption
	* parameters so that the block can be decrypted. This layout is as follows:
	*
	* 64 56 48 40 32 24 16 8 0
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 0 \| vdev1 \| GRID \| ASIZE \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 1 \|G\| offset1 \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 2 \| vdev2 \| GRID \| ASIZE \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 3 \|G\| offset2 \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 4 \| salt \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 5 \| IV1 \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 6 \|BDX\|lvl\| type \| cksum \|E\| comp\| PSIZE \| LSIZE \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 7 \| padding \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 8 \| padding \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 9 \| physical birth txg \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* a \| logical birth txg \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* b \| IV2 \| fill count \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* c \| checksum[0] \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* d \| checksum[1] \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* e \| MAC[0] \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* f \| MAC[1] \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	*
	* Legend:
	*
	* salt Salt for generating encryption keys
	* IV1 First 64 bits of encryption IV
	* X Block requires encryption handling (set to 1)
	* E blkptr_t contains embedded data (set to 0, see below)
	* fill count number of non-zero blocks under this bp (truncated to 32 bits)
	* IV2 Last 32 bits of encryption IV
	* checksum[2] 128-bit checksum of the data this bp describes
	* MAC[2] 128-bit message authentication code for this data
	*
	* The X bit being set indicates that this block is one of 3 types. If this is
	* a level 0 block with an encrypted object type, the block is encrypted
	* (see BP_IS_ENCRYPTED()). If this is a level 0 block with an unencrypted
	* object type, this block is authenticated with an HMAC (see
	* BP_IS_AUTHENTICATED()). Otherwise (if level > 0), this bp will use the MAC
	* words to store a checksum-of-MACs from the level below (see
	* BP_HAS_INDIRECT_MAC_CKSUM()). For convenience in the code, BP_IS_PROTECTED()
	* refers to both encrypted and authenticated blocks and BP_USES_CRYPT()
	* refers to any of these 3 kinds of blocks.
	*
	* The additional encryption parameters are the salt, IV, and MAC which are
	* explained in greater detail in the block comment at the top of zio_crypt.c.
	* The MAC occupies half of the checksum space since it serves a very similar
	* purpose: to prevent data corruption on disk. The only functional difference
	* is that the checksum is used to detect on-disk corruption whether or not the
	* encryption key is loaded and the MAC provides additional protection against
	* malicious disk tampering. We use the 3rd DVA to store the salt and first
	* 64 bits of the IV. As a result encrypted blocks can only have 2 copies
	* maximum instead of the normal 3. The last 32 bits of the IV are stored in
	* the upper bits of what is usually the fill count. Note that only blocks at
	* level 0 or -2 are ever encrypted, which allows us to guarantee that these
	* 32 bits are not trampled over by other code (see zio_crypt.c for details).
	* The salt and IV are not used for authenticated bps or bps with an indirect
	* MAC checksum, so these blocks can utilize all 3 DVAs and the full 64 bits
	* for the fill count.
	*/

	/*
	* "Embedded" blkptr_t's don't actually point to a block, instead they
	* have a data payload embedded in the blkptr_t itself. See the comment
	* in blkptr.c for more details.
	*
	* The blkptr_t is laid out as follows:
	*
	* 64 56 48 40 32 24 16 8 0
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 0 \| payload \|
	* 1 \| payload \|
	* 2 \| payload \|
	* 3 \| payload \|
	* 4 \| payload \|
	* 5 \| payload \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 6 \|BDX\|lvl\| type \| etype \|E\| comp\| PSIZE\| LSIZE \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* 7 \| payload \|
	* 8 \| payload \|
	* 9 \| payload \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* a \| logical birth txg \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* b \| payload \|
	* c \| payload \|
	* d \| payload \|
	* e \| payload \|
	* f \| payload \|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	*
	* Legend:
	*
	* payload contains the embedded data
	* B (byteorder) byteorder (endianness)
	* D (dedup) padding (set to zero)
	* X encryption (set to zero)
	* E (embedded) set to one
	* lvl indirection level
	* type DMU object type
	* etype how to interpret embedded data (BP_EMBEDDED_TYPE_*)
	* comp compression function of payload
	* PSIZE size of payload after compression, in bytes
	* LSIZE logical size of payload, in bytes
	* note that 25 bits is enough to store the largest
	* "normal" BP's LSIZE (2^16 * 2^9) in bytes
	* log. birth transaction group in which the block was logically born
	*
	* Note that LSIZE and PSIZE are stored in bytes, whereas for non-embedded
	* bp's they are stored in units of SPA_MINBLOCKSHIFT.
	* Generally, the generic BP_GET_*() macros can be used on embedded BP's.
	* The B, D, X, lvl, type, and comp fields are stored the same as with normal
	* BP's so the BP_SET_* macros can be used with them. etype, PSIZE, LSIZE must
	* be set with the BPE_SET_* macros. BP_SET_EMBEDDED() should be called before
	* other macros, as they assert that they are only used on BP's of the correct
	* "embedded-ness". Encrypted blkptr_t's cannot be embedded because they use
	* the payload space for encryption parameters (see the comment above on
	* how encryption parameters are stored).
	*/

	#define BPE_GET_ETYPE(bp) \
	(ASSERT(BP_IS_EMBEDDED(bp)), \
	BF64_GET((bp)->blk_prop, 40, 8))
	#define BPE_SET_ETYPE(bp, t) do { \
	ASSERT(BP_IS_EMBEDDED(bp)); \
	BF64_SET((bp)->blk_prop, 40, 8, t); \
	_NOTE(CONSTCOND) } while (0)

	#define BPE_GET_LSIZE(bp) \
	(ASSERT(BP_IS_EMBEDDED(bp)), \
	BF64_GET_SB((bp)->blk_prop, 0, 25, 0, 1))
	#define BPE_SET_LSIZE(bp, x) do { \
	ASSERT(BP_IS_EMBEDDED(bp)); \
	BF64_SET_SB((bp)->blk_prop, 0, 25, 0, 1, x); \
	_NOTE(CONSTCOND) } while (0)

	#define BPE_GET_PSIZE(bp) \
	(ASSERT(BP_IS_EMBEDDED(bp)), \
	BF64_GET_SB((bp)->blk_prop, 25, 7, 0, 1))
	#define BPE_SET_PSIZE(bp, x) do { \
	ASSERT(BP_IS_EMBEDDED(bp)); \
	BF64_SET_SB((bp)->blk_prop, 25, 7, 0, 1, x); \
	_NOTE(CONSTCOND) } while (0)

	typedef enum bp_embedded_type {
	BP_EMBEDDED_TYPE_DATA,
	BP_EMBEDDED_TYPE_RESERVED, /* Reserved for Delphix byteswap feature. */
	BP_EMBEDDED_TYPE_REDACTED,
	NUM_BP_EMBEDDED_TYPES
	} bp_embedded_type_t;

	#define BPE_NUM_WORDS 14
	#define BPE_PAYLOAD_SIZE (BPE_NUM_WORDS * sizeof (uint64_t))
	#define BPE_IS_PAYLOADWORD(bp, wp) \
	((wp) != &(bp)->blk_prop && (wp) != &(bp)->blk_birth)

	#define SPA_BLKPTRSHIFT 7 /* blkptr_t is 128 bytes */
	#define SPA_DVAS_PER_BP 3 /* Number of DVAs in a bp */
	#define SPA_SYNC_MIN_VDEVS 3 /* min vdevs to update during sync */

	/*
	* A block is a hole when it has either 1) never been written to, or
	* 2) is zero-filled. In both cases, ZFS can return all zeroes for all reads
	* without physically allocating disk space. Holes are represented in the
	* blkptr_t structure by zeroed blk_dva. Correct checking for holes is
	* done through the BP_IS_HOLE macro. For holes, the logical size, level,
	* DMU object type, and birth times are all also stored for holes that
	* were written to at some point (i.e. were punched after having been filled).
	*/
	typedef struct blkptr {
	dva_t blk_dva[SPA_DVAS_PER_BP]; /* Data Virtual Addresses */
	uint64_t blk_prop; /* size, compression, type, etc */
	uint64_t blk_pad[2]; /* Extra space for the future */
	uint64_t blk_phys_birth; /* txg when block was allocated */
	uint64_t blk_birth; /* transaction group at birth */
	uint64_t blk_fill; /* fill count */
	zio_cksum_t blk_cksum; /* 256-bit checksum */
	} blkptr_t;

	/*
	* Macros to get and set fields in a bp or DVA.
	*/
	#define DVA_GET_ASIZE(dva) \
	BF64_GET_SB((dva)->dva_word[0], 0, SPA_ASIZEBITS, SPA_MINBLOCKSHIFT, 0)
	#define DVA_SET_ASIZE(dva, x) \
	BF64_SET_SB((dva)->dva_word[0], 0, SPA_ASIZEBITS, \
	SPA_MINBLOCKSHIFT, 0, x)

	#define DVA_GET_GRID(dva) BF64_GET((dva)->dva_word[0], 24, 8)
	#define DVA_SET_GRID(dva, x) BF64_SET((dva)->dva_word[0], 24, 8, x)

	#define DVA_GET_VDEV(dva) BF64_GET((dva)->dva_word[0], 32, SPA_VDEVBITS)
	#define DVA_SET_VDEV(dva, x) \
	BF64_SET((dva)->dva_word[0], 32, SPA_VDEVBITS, x)

	#define DVA_GET_OFFSET(dva) \
	BF64_GET_SB((dva)->dva_word[1], 0, 63, SPA_MINBLOCKSHIFT, 0)
	#define DVA_SET_OFFSET(dva, x) \
	BF64_SET_SB((dva)->dva_word[1], 0, 63, SPA_MINBLOCKSHIFT, 0, x)

	#define DVA_GET_GANG(dva) BF64_GET((dva)->dva_word[1], 63, 1)
	#define DVA_SET_GANG(dva, x) BF64_SET((dva)->dva_word[1], 63, 1, x)

	#define BP_GET_LSIZE(bp) \
	(BP_IS_EMBEDDED(bp) ? \
	(BPE_GET_ETYPE(bp) == BP_EMBEDDED_TYPE_DATA ? BPE_GET_LSIZE(bp) : 0): \
	BF64_GET_SB((bp)->blk_prop, 0, SPA_LSIZEBITS, SPA_MINBLOCKSHIFT, 1))
	#define BP_SET_LSIZE(bp, x) do { \
	ASSERT(!BP_IS_EMBEDDED(bp)); \
	BF64_SET_SB((bp)->blk_prop, \
	0, SPA_LSIZEBITS, SPA_MINBLOCKSHIFT, 1, x); \
	_NOTE(CONSTCOND) } while (0)

	#define BP_GET_PSIZE(bp) \
	(BP_IS_EMBEDDED(bp) ? 0 : \
	BF64_GET_SB((bp)->blk_prop, 16, SPA_PSIZEBITS, SPA_MINBLOCKSHIFT, 1))
	#define BP_SET_PSIZE(bp, x) do { \
	ASSERT(!BP_IS_EMBEDDED(bp)); \
	BF64_SET_SB((bp)->blk_prop, \
	16, SPA_PSIZEBITS, SPA_MINBLOCKSHIFT, 1, x); \
	_NOTE(CONSTCOND) } while (0)

	#define BP_GET_COMPRESS(bp) \
	BF64_GET((bp)->blk_prop, 32, SPA_COMPRESSBITS)
	#define BP_SET_COMPRESS(bp, x) \
	BF64_SET((bp)->blk_prop, 32, SPA_COMPRESSBITS, x)

	#define BP_IS_EMBEDDED(bp) BF64_GET((bp)->blk_prop, 39, 1)
	#define BP_SET_EMBEDDED(bp, x) BF64_SET((bp)->blk_prop, 39, 1, x)

	#define BP_GET_CHECKSUM(bp) \
	(BP_IS_EMBEDDED(bp) ? ZIO_CHECKSUM_OFF : \
	BF64_GET((bp)->blk_prop, 40, 8))
	#define BP_SET_CHECKSUM(bp, x) do { \
	ASSERT(!BP_IS_EMBEDDED(bp)); \
	BF64_SET((bp)->blk_prop, 40, 8, x); \
	_NOTE(CONSTCOND) } while (0)

	#define BP_GET_TYPE(bp) BF64_GET((bp)->blk_prop, 48, 8)
	#define BP_SET_TYPE(bp, x) BF64_SET((bp)->blk_prop, 48, 8, x)

	#define BP_GET_LEVEL(bp) BF64_GET((bp)->blk_prop, 56, 5)
	#define BP_SET_LEVEL(bp, x) BF64_SET((bp)->blk_prop, 56, 5, x)

	/* encrypted, authenticated, and MAC cksum bps use the same bit */
	#define BP_USES_CRYPT(bp) BF64_GET((bp)->blk_prop, 61, 1)
	#define BP_SET_CRYPT(bp, x) BF64_SET((bp)->blk_prop, 61, 1, x)

	#define BP_IS_ENCRYPTED(bp) \
	(BP_USES_CRYPT(bp) && \
	BP_GET_LEVEL(bp) <= 0 && \
	DMU_OT_IS_ENCRYPTED(BP_GET_TYPE(bp)))

	#define BP_IS_AUTHENTICATED(bp) \
	(BP_USES_CRYPT(bp) && \
	BP_GET_LEVEL(bp) <= 0 && \
	!DMU_OT_IS_ENCRYPTED(BP_GET_TYPE(bp)))

	#define BP_HAS_INDIRECT_MAC_CKSUM(bp) \
	(BP_USES_CRYPT(bp) && BP_GET_LEVEL(bp) > 0)

	#define BP_IS_PROTECTED(bp) \
	(BP_IS_ENCRYPTED(bp) \|\| BP_IS_AUTHENTICATED(bp))

	#define BP_GET_DEDUP(bp) BF64_GET((bp)->blk_prop, 62, 1)
	#define BP_SET_DEDUP(bp, x) BF64_SET((bp)->blk_prop, 62, 1, x)

	#define BP_GET_BYTEORDER(bp) BF64_GET((bp)->blk_prop, 63, 1)
	#define BP_SET_BYTEORDER(bp, x) BF64_SET((bp)->blk_prop, 63, 1, x)

	#define BP_GET_FREE(bp) BF64_GET((bp)->blk_fill, 0, 1)
	#define BP_SET_FREE(bp, x) BF64_SET((bp)->blk_fill, 0, 1, x)

	#define BP_PHYSICAL_BIRTH(bp) \
	(BP_IS_EMBEDDED(bp) ? 0 : \
	(bp)->blk_phys_birth ? (bp)->blk_phys_birth : (bp)->blk_birth)

	#define BP_SET_BIRTH(bp, logical, physical) \
	{ \
	ASSERT(!BP_IS_EMBEDDED(bp)); \
	(bp)->blk_birth = (logical); \
	(bp)->blk_phys_birth = ((logical) == (physical) ? 0 : (physical)); \
	}

	#define BP_GET_FILL(bp) \
	((BP_IS_ENCRYPTED(bp)) ? BF64_GET((bp)->blk_fill, 0, 32) : \
	((BP_IS_EMBEDDED(bp)) ? 1 : (bp)->blk_fill))

	#define BP_SET_FILL(bp, fill) \
	{ \
	if (BP_IS_ENCRYPTED(bp)) \
	BF64_SET((bp)->blk_fill, 0, 32, fill); \
	else \
	(bp)->blk_fill = fill; \
	}

	#define BP_GET_IV2(bp) \
	(ASSERT(BP_IS_ENCRYPTED(bp)), \
	BF64_GET((bp)->blk_fill, 32, 32))
	#define BP_SET_IV2(bp, iv2) \
	{ \
	ASSERT(BP_IS_ENCRYPTED(bp)); \
	BF64_SET((bp)->blk_fill, 32, 32, iv2); \
	}

	#define BP_IS_METADATA(bp) \
	(BP_GET_LEVEL(bp) > 0 \|\| DMU_OT_IS_METADATA(BP_GET_TYPE(bp)))

	#define BP_GET_ASIZE(bp) \
	(BP_IS_EMBEDDED(bp) ? 0 : \
	DVA_GET_ASIZE(&(bp)->blk_dva[0]) + \
	DVA_GET_ASIZE(&(bp)->blk_dva[1]) + \
	(DVA_GET_ASIZE(&(bp)->blk_dva[2]) * !BP_IS_ENCRYPTED(bp)))

	#define BP_GET_UCSIZE(bp) \
	(BP_IS_METADATA(bp) ? BP_GET_PSIZE(bp) : BP_GET_LSIZE(bp))

	#define BP_GET_NDVAS(bp) \
	(BP_IS_EMBEDDED(bp) ? 0 : \
	!!DVA_GET_ASIZE(&(bp)->blk_dva[0]) + \
	!!DVA_GET_ASIZE(&(bp)->blk_dva[1]) + \
	(!!DVA_GET_ASIZE(&(bp)->blk_dva[2]) * !BP_IS_ENCRYPTED(bp)))

	#define BP_COUNT_GANG(bp) \
	(BP_IS_EMBEDDED(bp) ? 0 : \
	(DVA_GET_GANG(&(bp)->blk_dva[0]) + \
	DVA_GET_GANG(&(bp)->blk_dva[1]) + \
	(DVA_GET_GANG(&(bp)->blk_dva[2]) * !BP_IS_ENCRYPTED(bp))))

	#define DVA_EQUAL(dva1, dva2) \
	((dva1)->dva_word[1] == (dva2)->dva_word[1] && \
	(dva1)->dva_word[0] == (dva2)->dva_word[0])

	#define BP_EQUAL(bp1, bp2) \
	(BP_PHYSICAL_BIRTH(bp1) == BP_PHYSICAL_BIRTH(bp2) && \
	(bp1)->blk_birth == (bp2)->blk_birth && \
	DVA_EQUAL(&(bp1)->blk_dva[0], &(bp2)->blk_dva[0]) && \
	DVA_EQUAL(&(bp1)->blk_dva[1], &(bp2)->blk_dva[1]) && \
	DVA_EQUAL(&(bp1)->blk_dva[2], &(bp2)->blk_dva[2]))


	#define DVA_IS_VALID(dva) (DVA_GET_ASIZE(dva) != 0)

	#define BP_IDENTITY(bp) (ASSERT(!BP_IS_EMBEDDED(bp)), &(bp)->blk_dva[0])
	#define BP_IS_GANG(bp) \
	(BP_IS_EMBEDDED(bp) ? B_FALSE : DVA_GET_GANG(BP_IDENTITY(bp)))
	#define DVA_IS_EMPTY(dva) ((dva)->dva_word[0] == 0ULL && \
	(dva)->dva_word[1] == 0ULL)
	#define BP_IS_HOLE(bp) \
	(!BP_IS_EMBEDDED(bp) && DVA_IS_EMPTY(BP_IDENTITY(bp)))

	#define BP_SET_REDACTED(bp) \
	{ \
	BP_SET_EMBEDDED(bp, B_TRUE); \
	BPE_SET_ETYPE(bp, BP_EMBEDDED_TYPE_REDACTED); \
	}
	#define BP_IS_REDACTED(bp) \
	(BP_IS_EMBEDDED(bp) && BPE_GET_ETYPE(bp) == BP_EMBEDDED_TYPE_REDACTED)

	/* BP_IS_RAIDZ(bp) assumes no block compression */
	#define BP_IS_RAIDZ(bp) (DVA_GET_ASIZE(&(bp)->blk_dva[0]) > \
	BP_GET_PSIZE(bp))

	#define BP_ZERO(bp) \
	{ \
	(bp)->blk_dva[0].dva_word[0] = 0; \
	(bp)->blk_dva[0].dva_word[1] = 0; \
	(bp)->blk_dva[1].dva_word[0] = 0; \
	(bp)->blk_dva[1].dva_word[1] = 0; \
	(bp)->blk_dva[2].dva_word[0] = 0; \
	(bp)->blk_dva[2].dva_word[1] = 0; \
	(bp)->blk_prop = 0; \
	(bp)->blk_pad[0] = 0; \
	(bp)->blk_pad[1] = 0; \
	(bp)->blk_phys_birth = 0; \
	(bp)->blk_birth = 0; \
	(bp)->blk_fill = 0; \
	ZIO_SET_CHECKSUM(&(bp)->blk_cksum, 0, 0, 0, 0); \
	}

	#ifdef _ZFS_BIG_ENDIAN
	#define ZFS_HOST_BYTEORDER (0ULL)
	#else
	#define ZFS_HOST_BYTEORDER (1ULL)
	#endif

	#define BP_SHOULD_BYTESWAP(bp) (BP_GET_BYTEORDER(bp) != ZFS_HOST_BYTEORDER)

	#define BP_SPRINTF_LEN 400

	/*
	* This macro allows code sharing between zfs, libzpool, and mdb.
	* 'func' is either snprintf() or mdb_snprintf().
	* 'ws' (whitespace) can be ' ' for single-line format, '\n' for multi-line.
	*/

	#define SNPRINTF_BLKPTR(func, ws, buf, size, bp, type, checksum, compress) \
	{ \
	static const char *copyname[] = \
	{ "zero", "single", "double", "triple" }; \
	int len = 0; \
	int copies = 0; \
	const char *crypt_type; \
	if (bp != NULL) { \
	if (BP_IS_ENCRYPTED(bp)) { \
	crypt_type = "encrypted"; \
	/* LINTED E_SUSPICIOUS_COMPARISON */ \
	} else if (BP_IS_AUTHENTICATED(bp)) { \
	crypt_type = "authenticated"; \
	} else if (BP_HAS_INDIRECT_MAC_CKSUM(bp)) { \
	crypt_type = "indirect-MAC"; \
	} else { \
	crypt_type = "unencrypted"; \
	} \
	} \
	if (bp == NULL) { \
	len += func(buf + len, size - len, "<NULL>"); \
	} else if (BP_IS_HOLE(bp)) { \
	len += func(buf + len, size - len, \
	"HOLE [L%llu %s] " \
	"size=%llxL birth=%lluL", \
	(u_longlong_t)BP_GET_LEVEL(bp), \
	type, \
	(u_longlong_t)BP_GET_LSIZE(bp), \
	(u_longlong_t)bp->blk_birth); \
	} else if (BP_IS_EMBEDDED(bp)) { \
	len = func(buf + len, size - len, \
	"EMBEDDED [L%llu %s] et=%u %s " \
	"size=%llxL/%llxP birth=%lluL", \
	(u_longlong_t)BP_GET_LEVEL(bp), \
	type, \
	(int)BPE_GET_ETYPE(bp), \
	compress, \
	(u_longlong_t)BPE_GET_LSIZE(bp), \
	(u_longlong_t)BPE_GET_PSIZE(bp), \
	(u_longlong_t)bp->blk_birth); \
	} else if (BP_IS_REDACTED(bp)) { \
	len += func(buf + len, size - len, \
	"REDACTED [L%llu %s] size=%llxL birth=%lluL", \
	(u_longlong_t)BP_GET_LEVEL(bp), \
	type, \
	(u_longlong_t)BP_GET_LSIZE(bp), \
	(u_longlong_t)bp->blk_birth); \
	} else { \
	for (int d = 0; d < BP_GET_NDVAS(bp); d++) { \
	const dva_t *dva = &bp->blk_dva[d]; \
	if (DVA_IS_VALID(dva)) \
	copies++; \
	len += func(buf + len, size - len, \
	"DVA[%d]=<%llu:%llx:%llx>%c", d, \
	(u_longlong_t)DVA_GET_VDEV(dva), \
	(u_longlong_t)DVA_GET_OFFSET(dva), \
	(u_longlong_t)DVA_GET_ASIZE(dva), \
	ws); \
	} \
	if (BP_IS_ENCRYPTED(bp)) { \
	len += func(buf + len, size - len, \
	"salt=%llx iv=%llx:%llx%c", \
	(u_longlong_t)bp->blk_dva[2].dva_word[0], \
	(u_longlong_t)bp->blk_dva[2].dva_word[1], \
	(u_longlong_t)BP_GET_IV2(bp), \
	ws); \
	} \
	if (BP_IS_GANG(bp) && \
	DVA_GET_ASIZE(&bp->blk_dva[2]) <= \
	DVA_GET_ASIZE(&bp->blk_dva[1]) / 2) \
	copies--; \
	len += func(buf + len, size - len, \
	"[L%llu %s] %s %s %s %s %s %s %s%c" \
	"size=%llxL/%llxP birth=%lluL/%lluP fill=%llu%c" \
	"cksum=%llx:%llx:%llx:%llx", \
	(u_longlong_t)BP_GET_LEVEL(bp), \
	type, \
	checksum, \
	compress, \
	crypt_type, \
	BP_GET_BYTEORDER(bp) == 0 ? "BE" : "LE", \
	BP_IS_GANG(bp) ? "gang" : "contiguous", \
	BP_GET_DEDUP(bp) ? "dedup" : "unique", \
	copyname[copies], \
	ws, \
	(u_longlong_t)BP_GET_LSIZE(bp), \
	(u_longlong_t)BP_GET_PSIZE(bp), \
	(u_longlong_t)bp->blk_birth, \
	(u_longlong_t)BP_PHYSICAL_BIRTH(bp), \
	(u_longlong_t)BP_GET_FILL(bp), \
	ws, \
	(u_longlong_t)bp->blk_cksum.zc_word[0], \
	(u_longlong_t)bp->blk_cksum.zc_word[1], \
	(u_longlong_t)bp->blk_cksum.zc_word[2], \
	(u_longlong_t)bp->blk_cksum.zc_word[3]); \
	} \
	ASSERT(len < size); \
	}

	#define BP_GET_BUFC_TYPE(bp) \
	(BP_IS_METADATA(bp) ? ARC_BUFC_METADATA : ARC_BUFC_DATA)

	typedef enum spa_import_type {
	SPA_IMPORT_EXISTING,
	SPA_IMPORT_ASSEMBLE
	} spa_import_type_t;

	typedef enum spa_mode {
	SPA_MODE_UNINIT = 0,
	SPA_MODE_READ = 1,
	SPA_MODE_WRITE = 2,
	} spa_mode_t;

	/*
	* Send TRIM commands in-line during normal pool operation while deleting.
	* OFF: no
	* ON: yes
	* NB: IN_FREEBSD_BASE is defined within the FreeBSD sources.
	*/
	typedef enum {
	SPA_AUTOTRIM_OFF = 0, /* default */
	SPA_AUTOTRIM_ON,
	#ifdef IN_FREEBSD_BASE
	SPA_AUTOTRIM_DEFAULT = SPA_AUTOTRIM_ON,
	#else
	SPA_AUTOTRIM_DEFAULT = SPA_AUTOTRIM_OFF,
	#endif
	} spa_autotrim_t;

	/*
	* Reason TRIM command was issued, used internally for accounting purposes.
	*/
	typedef enum trim_type {
	TRIM_TYPE_MANUAL = 0,
	TRIM_TYPE_AUTO = 1,
	TRIM_TYPE_SIMPLE = 2
	} trim_type_t;

	/* state manipulation functions */
	extern int spa_open(const char pool, spa_t , void tag);
	extern int spa_open_rewind(const char pool, spa_t , void tag,
	nvlist_t policy, nvlist_t *config);
	extern int spa_get_stats(const char pool, nvlist_t config, char altroot,
	size_t buflen);
	extern int spa_create(const char pool, nvlist_t nvroot, nvlist_t *props,
	nvlist_t zplprops, struct dsl_crypto_params dcp);
	extern int spa_import(char pool, nvlist_t config, nvlist_t *props,
	uint64_t flags);
	extern nvlist_t spa_tryimport(nvlist_t tryconfig);
	-extern int spa_destroy(char *pool);
	+extern int spa_destroy(const char *pool);
	extern int spa_checkpoint(const char *pool);
	extern int spa_checkpoint_discard(const char *pool);
	-extern int spa_export(char pool, nvlist_t *oldconfig, boolean_t force,
	+extern int spa_export(const char pool, nvlist_t *oldconfig, boolean_t force,
	boolean_t hardforce);
	-extern int spa_reset(char *pool);
	+extern int spa_reset(const char *pool);
	extern void spa_async_request(spa_t *spa, int flag);
	extern void spa_async_unrequest(spa_t *spa, int flag);
	extern void spa_async_suspend(spa_t *spa);
	extern void spa_async_resume(spa_t *spa);
	extern int spa_async_tasks(spa_t *spa);
	extern spa_t spa_inject_addref(char pool);
	extern void spa_inject_delref(spa_t *spa);
	extern void spa_scan_stat_init(spa_t *spa);
	extern int spa_scan_get_stats(spa_t spa, pool_scan_stat_t ps);
	extern int bpobj_enqueue_alloc_cb(void arg, const blkptr_t bp, dmu_tx_t *tx);
	extern int bpobj_enqueue_free_cb(void arg, const blkptr_t bp, dmu_tx_t *tx);

	#define SPA_ASYNC_CONFIG_UPDATE 0x01
	#define SPA_ASYNC_REMOVE 0x02
	#define SPA_ASYNC_PROBE 0x04
	#define SPA_ASYNC_RESILVER_DONE 0x08
	#define SPA_ASYNC_RESILVER 0x10
	#define SPA_ASYNC_AUTOEXPAND 0x20
	#define SPA_ASYNC_REMOVE_DONE 0x40
	#define SPA_ASYNC_REMOVE_STOP 0x80
	#define SPA_ASYNC_INITIALIZE_RESTART 0x100
	#define SPA_ASYNC_TRIM_RESTART 0x200
	#define SPA_ASYNC_AUTOTRIM_RESTART 0x400
	#define SPA_ASYNC_L2CACHE_REBUILD 0x800
	#define SPA_ASYNC_L2CACHE_TRIM 0x1000
	#define SPA_ASYNC_REBUILD_DONE 0x2000

	/* device manipulation */
	extern int spa_vdev_add(spa_t spa, nvlist_t nvroot);
	extern int spa_vdev_attach(spa_t spa, uint64_t guid, nvlist_t nvroot,
	int replacing, int rebuild);
	extern int spa_vdev_detach(spa_t *spa, uint64_t guid, uint64_t pguid,
	int replace_done);
	extern int spa_vdev_remove(spa_t *spa, uint64_t guid, boolean_t unspare);
	extern boolean_t spa_vdev_remove_active(spa_t *spa);
	extern int spa_vdev_initialize(spa_t spa, nvlist_t nv, uint64_t cmd_type,
	nvlist_t *vdev_errlist);
	extern int spa_vdev_trim(spa_t spa, nvlist_t nv, uint64_t cmd_type,
	uint64_t rate, boolean_t partial, boolean_t secure, nvlist_t *vdev_errlist);
	extern int spa_vdev_setpath(spa_t spa, uint64_t guid, const char newpath);
	extern int spa_vdev_setfru(spa_t spa, uint64_t guid, const char newfru);
	extern int spa_vdev_split_mirror(spa_t spa, char newname, nvlist_t *config,
	nvlist_t *props, boolean_t exp);

	/* spare state (which is global across all pools) */
	extern void spa_spare_add(vdev_t *vd);
	extern void spa_spare_remove(vdev_t *vd);
	extern boolean_t spa_spare_exists(uint64_t guid, uint64_t pool, int refcnt);
	extern void spa_spare_activate(vdev_t *vd);

	/* L2ARC state (which is global across all pools) */
	extern void spa_l2cache_add(vdev_t *vd);
	extern void spa_l2cache_remove(vdev_t *vd);
	extern boolean_t spa_l2cache_exists(uint64_t guid, uint64_t *pool);
	extern void spa_l2cache_activate(vdev_t *vd);
	extern void spa_l2cache_drop(spa_t *spa);

	/* scanning */
	extern int spa_scan(spa_t *spa, pool_scan_func_t func);
	extern int spa_scan_stop(spa_t *spa);
	extern int spa_scrub_pause_resume(spa_t *spa, pool_scrub_cmd_t flag);

	/* spa syncing */
	extern void spa_sync(spa_t spa, uint64_t txg); / only for DMU use */
	extern void spa_sync_allpools(void);

	extern int zfs_sync_pass_deferred_free;

	/* spa namespace global mutex */
	extern kmutex_t spa_namespace_lock;

	/*
	* SPA configuration functions in spa_config.c
	*/

	#define SPA_CONFIG_UPDATE_POOL 0
	#define SPA_CONFIG_UPDATE_VDEVS 1

	extern void spa_write_cachefile(spa_t *, boolean_t, boolean_t);
	extern void spa_config_load(void);
	extern nvlist_t spa_all_configs(uint64_t );
	extern void spa_config_set(spa_t spa, nvlist_t config);
	extern nvlist_t spa_config_generate(spa_t spa, vdev_t *vd, uint64_t txg,
	int getstats);
	extern void spa_config_update(spa_t *spa, int what);
	extern int spa_config_parse(spa_t spa, vdev_t vdp, nvlist_t nv,
	vdev_t *parent, uint_t id, int atype);


	/*
	* Miscellaneous SPA routines in spa_misc.c
	*/

	/* Namespace manipulation */
	extern spa_t spa_lookup(const char name);
	extern spa_t spa_add(const char name, nvlist_t config, const char altroot);
	extern void spa_remove(spa_t *spa);
	extern spa_t spa_next(spa_t prev);

	/* Refcount functions */
	extern void spa_open_ref(spa_t spa, void tag);
	extern void spa_close(spa_t spa, void tag);
	extern void spa_async_close(spa_t spa, void tag);
	extern boolean_t spa_refcount_zero(spa_t *spa);

	#define SCL_NONE 0x00
	#define SCL_CONFIG 0x01
	#define SCL_STATE 0x02
	#define SCL_L2ARC 0x04 /* hack until L2ARC 2.0 */
	#define SCL_ALLOC 0x08
	#define SCL_ZIO 0x10
	#define SCL_FREE 0x20
	#define SCL_VDEV 0x40
	#define SCL_LOCKS 7
	#define SCL_ALL ((1 << SCL_LOCKS) - 1)
	#define SCL_STATE_ALL (SCL_STATE \| SCL_L2ARC \| SCL_ZIO)

	/* Historical pool statistics */
	typedef struct spa_history_kstat {
	kmutex_t lock;
	uint64_t count;
	uint64_t size;
	kstat_t *kstat;
	void *priv;
	list_t list;
	} spa_history_kstat_t;

	typedef struct spa_history_list {
	uint64_t size;
	procfs_list_t procfs_list;
	} spa_history_list_t;

	typedef struct spa_stats {
	spa_history_list_t read_history;
	spa_history_list_t txg_history;
	spa_history_kstat_t tx_assign_histogram;
	spa_history_kstat_t io_history;
	spa_history_list_t mmp_history;
	spa_history_kstat_t state; /* pool state */
	spa_history_kstat_t iostats;
	} spa_stats_t;

	typedef enum txg_state {
	TXG_STATE_BIRTH = 0,
	TXG_STATE_OPEN = 1,
	TXG_STATE_QUIESCED = 2,
	TXG_STATE_WAIT_FOR_SYNC = 3,
	TXG_STATE_SYNCED = 4,
	TXG_STATE_COMMITTED = 5,
	} txg_state_t;

	typedef struct txg_stat {
	vdev_stat_t vs1;
	vdev_stat_t vs2;
	uint64_t txg;
	uint64_t ndirty;
	} txg_stat_t;

	/* Assorted pool IO kstats */
	typedef struct spa_iostats {
	kstat_named_t trim_extents_written;
	kstat_named_t trim_bytes_written;
	kstat_named_t trim_extents_skipped;
	kstat_named_t trim_bytes_skipped;
	kstat_named_t trim_extents_failed;
	kstat_named_t trim_bytes_failed;
	kstat_named_t autotrim_extents_written;
	kstat_named_t autotrim_bytes_written;
	kstat_named_t autotrim_extents_skipped;
	kstat_named_t autotrim_bytes_skipped;
	kstat_named_t autotrim_extents_failed;
	kstat_named_t autotrim_bytes_failed;
	kstat_named_t simple_trim_extents_written;
	kstat_named_t simple_trim_bytes_written;
	kstat_named_t simple_trim_extents_skipped;
	kstat_named_t simple_trim_bytes_skipped;
	kstat_named_t simple_trim_extents_failed;
	kstat_named_t simple_trim_bytes_failed;
	} spa_iostats_t;

	extern void spa_stats_init(spa_t *spa);
	extern void spa_stats_destroy(spa_t *spa);
	extern void spa_read_history_add(spa_t spa, const zbookmark_phys_t zb,
	uint32_t aflags);
	extern void spa_txg_history_add(spa_t *spa, uint64_t txg, hrtime_t birth_time);
	extern int spa_txg_history_set(spa_t *spa, uint64_t txg,
	txg_state_t completed_state, hrtime_t completed_time);
	extern txg_stat_t spa_txg_history_init_io(spa_t , uint64_t,
	struct dsl_pool *);
	extern void spa_txg_history_fini_io(spa_t , txg_stat_t );
	extern void spa_tx_assign_add_nsecs(spa_t *spa, uint64_t nsecs);
	extern int spa_mmp_history_set_skip(spa_t *spa, uint64_t mmp_kstat_id);
	extern int spa_mmp_history_set(spa_t *spa, uint64_t mmp_kstat_id, int io_error,
	hrtime_t duration);
	extern void spa_mmp_history_add(spa_t *spa, uint64_t txg, uint64_t timestamp,
	uint64_t mmp_delay, vdev_t *vd, int label, uint64_t mmp_kstat_id,
	int error);
	extern void spa_iostats_trim_add(spa_t *spa, trim_type_t type,
	uint64_t extents_written, uint64_t bytes_written,
	uint64_t extents_skipped, uint64_t bytes_skipped,
	uint64_t extents_failed, uint64_t bytes_failed);
	extern void spa_import_progress_add(spa_t *spa);
	extern void spa_import_progress_remove(uint64_t spa_guid);
	extern int spa_import_progress_set_mmp_check(uint64_t pool_guid,
	uint64_t mmp_sec_remaining);
	extern int spa_import_progress_set_max_txg(uint64_t pool_guid,
	uint64_t max_txg);
	extern int spa_import_progress_set_state(uint64_t pool_guid,
	spa_load_state_t spa_load_state);

	/* Pool configuration locks */
	extern int spa_config_tryenter(spa_t spa, int locks, void tag, krw_t rw);
	extern void spa_config_enter(spa_t spa, int locks, const void tag, krw_t rw);
	extern void spa_config_exit(spa_t spa, int locks, const void tag);
	extern int spa_config_held(spa_t *spa, int locks, krw_t rw);

	/* Pool vdev add/remove lock */
	extern uint64_t spa_vdev_enter(spa_t *spa);
	extern uint64_t spa_vdev_detach_enter(spa_t *spa, uint64_t guid);
	extern uint64_t spa_vdev_config_enter(spa_t *spa);
	extern void spa_vdev_config_exit(spa_t spa, vdev_t vd, uint64_t txg,
	int error, char *tag);
	extern int spa_vdev_exit(spa_t spa, vdev_t vd, uint64_t txg, int error);

	/* Pool vdev state change lock */
	extern void spa_vdev_state_enter(spa_t *spa, int oplock);
	extern int spa_vdev_state_exit(spa_t spa, vdev_t vd, int error);

	/* Log state */
	typedef enum spa_log_state {
	SPA_LOG_UNKNOWN = 0, /* unknown log state */
	SPA_LOG_MISSING, /* missing log(s) */
	SPA_LOG_CLEAR, /* clear the log(s) */
	SPA_LOG_GOOD, /* log(s) are good */
	} spa_log_state_t;

	extern spa_log_state_t spa_get_log_state(spa_t *spa);
	extern void spa_set_log_state(spa_t *spa, spa_log_state_t state);
	extern int spa_reset_logs(spa_t *spa);

	/* Log claim callback */
	extern void spa_claim_notify(zio_t *zio);
	extern void spa_deadman(void *);

	/* Accessor functions */
	extern boolean_t spa_shutting_down(spa_t *spa);
	extern struct dsl_pool spa_get_dsl(spa_t spa);
	extern boolean_t spa_is_initializing(spa_t *spa);
	extern boolean_t spa_indirect_vdevs_loaded(spa_t *spa);
	extern blkptr_t spa_get_rootblkptr(spa_t spa);
	extern void spa_set_rootblkptr(spa_t spa, const blkptr_t bp);
	extern void spa_altroot(spa_t , char , size_t);
	extern int spa_sync_pass(spa_t *spa);
	extern char spa_name(spa_t spa);
	extern uint64_t spa_guid(spa_t *spa);
	extern uint64_t spa_load_guid(spa_t *spa);
	extern uint64_t spa_last_synced_txg(spa_t *spa);
	extern uint64_t spa_first_txg(spa_t *spa);
	extern uint64_t spa_syncing_txg(spa_t *spa);
	extern uint64_t spa_final_dirty_txg(spa_t *spa);
	extern uint64_t spa_version(spa_t *spa);
	extern pool_state_t spa_state(spa_t *spa);
	extern spa_load_state_t spa_load_state(spa_t *spa);
	extern uint64_t spa_freeze_txg(spa_t *spa);
	extern uint64_t spa_get_worst_case_asize(spa_t *spa, uint64_t lsize);
	extern uint64_t spa_get_dspace(spa_t *spa);
	extern uint64_t spa_get_checkpoint_space(spa_t *spa);
	extern uint64_t spa_get_slop_space(spa_t *spa);
	extern void spa_update_dspace(spa_t *spa);
	extern uint64_t spa_version(spa_t *spa);
	extern boolean_t spa_deflate(spa_t *spa);
	extern metaslab_class_t spa_normal_class(spa_t spa);
	extern metaslab_class_t spa_log_class(spa_t spa);
	extern metaslab_class_t spa_special_class(spa_t spa);
	extern metaslab_class_t spa_dedup_class(spa_t spa);
	extern metaslab_class_t spa_preferred_class(spa_t spa, uint64_t size,
	dmu_object_type_t objtype, uint_t level, uint_t special_smallblk);

	extern void spa_evicting_os_register(spa_t , objset_t os);
	extern void spa_evicting_os_deregister(spa_t , objset_t os);
	extern void spa_evicting_os_wait(spa_t *spa);
	extern int spa_max_replication(spa_t *spa);
	extern int spa_prev_software_version(spa_t *spa);
	extern uint64_t spa_get_failmode(spa_t *spa);
	extern uint64_t spa_get_deadman_failmode(spa_t *spa);
	extern void spa_set_deadman_failmode(spa_t spa, const char failmode);
	extern boolean_t spa_suspended(spa_t *spa);
	extern uint64_t spa_bootfs(spa_t *spa);
	extern uint64_t spa_delegation(spa_t *spa);
	extern objset_t spa_meta_objset(spa_t spa);
	extern space_map_t spa_syncing_log_sm(spa_t spa);
	extern uint64_t spa_deadman_synctime(spa_t *spa);
	extern uint64_t spa_deadman_ziotime(spa_t *spa);
	extern uint64_t spa_dirty_data(spa_t *spa);
	extern spa_autotrim_t spa_get_autotrim(spa_t *spa);

	/* Miscellaneous support routines */
	extern void spa_load_failed(spa_t spa, const char fmt, ...);
	extern void spa_load_note(spa_t spa, const char fmt, ...);
	extern void spa_activate_mos_feature(spa_t spa, const char feature,
	dmu_tx_t *tx);
	extern void spa_deactivate_mos_feature(spa_t spa, const char feature);
	extern spa_t *spa_by_guid(uint64_t pool_guid, uint64_t device_guid);
	extern boolean_t spa_guid_exists(uint64_t pool_guid, uint64_t device_guid);
	extern char spa_strdup(const char );
	extern void spa_strfree(char *);
	extern uint64_t spa_get_random(uint64_t range);
	extern uint64_t spa_generate_guid(spa_t *spa);
	extern void snprintf_blkptr(char buf, size_t buflen, const blkptr_t bp);
	extern void spa_freeze(spa_t *spa);
	extern int spa_change_guid(spa_t *spa);
	extern void spa_upgrade(spa_t *spa, uint64_t version);
	extern void spa_evict_all(void);
	extern vdev_t spa_lookup_by_guid(spa_t spa, uint64_t guid,
	boolean_t l2cache);
	extern boolean_t spa_has_spare(spa_t *, uint64_t guid);
	extern uint64_t dva_get_dsize_sync(spa_t spa, const dva_t dva);
	extern uint64_t bp_get_dsize_sync(spa_t spa, const blkptr_t bp);
	extern uint64_t bp_get_dsize(spa_t spa, const blkptr_t bp);
	extern boolean_t spa_has_slogs(spa_t *spa);
	extern boolean_t spa_is_root(spa_t *spa);
	extern boolean_t spa_writeable(spa_t *spa);
	extern boolean_t spa_has_pending_synctask(spa_t *spa);
	extern int spa_maxblocksize(spa_t *spa);
	extern int spa_maxdnodesize(spa_t *spa);
	extern boolean_t spa_has_checkpoint(spa_t *spa);
	extern boolean_t spa_importing_readonly_checkpoint(spa_t *spa);
	extern boolean_t spa_suspend_async_destroy(spa_t *spa);
	extern uint64_t spa_min_claim_txg(spa_t *spa);
	extern boolean_t zfs_dva_valid(spa_t spa, const dva_t dva,
	const blkptr_t *bp);
	typedef void (*spa_remap_cb_t)(uint64_t vdev, uint64_t offset, uint64_t size,
	void *arg);
	extern boolean_t spa_remap_blkptr(spa_t spa, blkptr_t bp,
	spa_remap_cb_t callback, void *arg);
	extern uint64_t spa_get_last_removal_txg(spa_t *spa);
	extern boolean_t spa_trust_config(spa_t *spa);
	extern uint64_t spa_missing_tvds_allowed(spa_t *spa);
	extern void spa_set_missing_tvds(spa_t *spa, uint64_t missing);
	extern boolean_t spa_top_vdevs_spacemap_addressable(spa_t *spa);
	extern uint64_t spa_total_metaslabs(spa_t *spa);
	extern boolean_t spa_multihost(spa_t *spa);
	extern uint32_t spa_get_hostid(spa_t *spa);
	extern void spa_activate_allocation_classes(spa_t , dmu_tx_t );
	extern boolean_t spa_livelist_delete_check(spa_t *spa);

	extern spa_mode_t spa_mode(spa_t *spa);
	extern uint64_t zfs_strtonum(const char str, char *nptr);

	extern char *spa_his_ievent_table[];

	extern void spa_history_create_obj(spa_t spa, dmu_tx_t tx);
	extern int spa_history_get(spa_t spa, uint64_t offset, uint64_t *len_read,
	char *his_buf);
	extern int spa_history_log(spa_t spa, const char his_buf);
	extern int spa_history_log_nvl(spa_t spa, nvlist_t nvl);
	extern void spa_history_log_version(spa_t spa, const char operation,
	dmu_tx_t *tx);
	extern void spa_history_log_internal(spa_t spa, const char operation,
	dmu_tx_t tx, const char fmt, ...) __printflike(4, 5);
	extern void spa_history_log_internal_ds(struct dsl_dataset ds, const char op,
	dmu_tx_t tx, const char fmt, ...) __printflike(4, 5);
	extern void spa_history_log_internal_dd(dsl_dir_t dd, const char operation,
	dmu_tx_t tx, const char fmt, ...) __printflike(4, 5);

	extern const char spa_state_to_name(spa_t spa);

	/* error handling */
	struct zbookmark_phys;
	extern void spa_log_error(spa_t spa, const zbookmark_phys_t zb);
	extern int zfs_ereport_post(const char clazz, spa_t spa, vdev_t *vd,
	const zbookmark_phys_t zb, zio_t zio, uint64_t state);
	extern boolean_t zfs_ereport_is_valid(const char clazz, spa_t spa, vdev_t *vd,
	zio_t *zio);
	extern void zfs_ereport_taskq_fini(void);
	extern nvlist_t zfs_event_create(spa_t spa, vdev_t vd, const char type,
	const char name, nvlist_t aux);
	extern void zfs_post_remove(spa_t spa, vdev_t vd);
	extern void zfs_post_state_change(spa_t spa, vdev_t vd, uint64_t laststate);
	extern void zfs_post_autoreplace(spa_t spa, vdev_t vd);
	extern uint64_t spa_get_errlog_size(spa_t *spa);
	extern int spa_get_errlog(spa_t spa, void uaddr, size_t *count);
	extern void spa_errlog_rotate(spa_t *spa);
	extern void spa_errlog_drain(spa_t *spa);
	extern void spa_errlog_sync(spa_t *spa, uint64_t txg);
	extern void spa_get_errlists(spa_t spa, avl_tree_t last, avl_tree_t *scrub);

	/* vdev cache */
	extern void vdev_cache_stat_init(void);
	extern void vdev_cache_stat_fini(void);

	/* vdev mirror */
	extern void vdev_mirror_stat_init(void);
	extern void vdev_mirror_stat_fini(void);

	/* Initialization and termination */
	extern void spa_init(spa_mode_t mode);
	extern void spa_fini(void);
	extern void spa_boot_init(void);

	/* properties */
	extern int spa_prop_set(spa_t spa, nvlist_t nvp);
	extern int spa_prop_get(spa_t spa, nvlist_t *nvp);
	extern void spa_prop_clear_bootfs(spa_t spa, uint64_t obj, dmu_tx_t tx);
	extern void spa_configfile_set(spa_t , nvlist_t , boolean_t);

	/* asynchronous event notification */
	extern void spa_event_notify(spa_t spa, vdev_t vdev, nvlist_t *hist_nvl,
	const char *name);

	/* waiting for pool activities to complete */
	extern int spa_wait(const char *pool, zpool_wait_activity_t activity,
	boolean_t *waited);
	extern int spa_wait_tag(const char *name, zpool_wait_activity_t activity,
	uint64_t tag, boolean_t *waited);
	extern void spa_notify_waiters(spa_t *spa);
	extern void spa_wake_waiters(spa_t *spa);

	/* module param call functions */
	int param_set_deadman_ziotime(ZFS_MODULE_PARAM_ARGS);
	int param_set_deadman_synctime(ZFS_MODULE_PARAM_ARGS);
	int param_set_slop_shift(ZFS_MODULE_PARAM_ARGS);
	int param_set_deadman_failmode(ZFS_MODULE_PARAM_ARGS);

	#ifdef ZFS_DEBUG
	#define dprintf_bp(bp, fmt, ...) do { \
	if (zfs_flags & ZFS_DEBUG_DPRINTF) { \
	char *__blkbuf = kmem_alloc(BP_SPRINTF_LEN, KM_SLEEP); \
	snprintf_blkptr(__blkbuf, BP_SPRINTF_LEN, (bp)); \
	dprintf(fmt " %s\n", __VA_ARGS__, __blkbuf); \
	kmem_free(__blkbuf, BP_SPRINTF_LEN); \
	} \
	_NOTE(CONSTCOND) } while (0)
	#else
	#define dprintf_bp(bp, fmt, ...)
	#endif

	extern spa_mode_t spa_mode_global;
	extern int zfs_deadman_enabled;
	extern unsigned long zfs_deadman_synctime_ms;
	extern unsigned long zfs_deadman_ziotime_ms;
	extern unsigned long zfs_deadman_checktime_ms;

	#ifdef __cplusplus
	}
	#endif

	#endif /* _SYS_SPA_H */
	Index: vendor-sys/openzfs/dist/include/sys/u8_textprep.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/sys/u8_textprep.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/sys/u8_textprep.h (revision 366774)
	@@ -1,113 +1,113 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright 2007 Sun Microsystems, Inc. All rights reserved.
	* Use is subject to license terms.
	*/

	#ifndef _SYS_U8_TEXTPREP_H
	#define _SYS_U8_TEXTPREP_H



	#include <sys/isa_defs.h>
	#include <sys/types.h>
	#include <sys/errno.h>

	#ifdef __cplusplus
	extern "C" {
	#endif

	/*
	* Unicode encoding conversion functions and their macros.
	*/
	#define UCONV_IN_BIG_ENDIAN 0x0001
	#define UCONV_OUT_BIG_ENDIAN 0x0002
	#define UCONV_IN_SYSTEM_ENDIAN 0x0004
	#define UCONV_OUT_SYSTEM_ENDIAN 0x0008
	#define UCONV_IN_LITTLE_ENDIAN 0x0010
	#define UCONV_OUT_LITTLE_ENDIAN 0x0020
	#define UCONV_IGNORE_NULL 0x0040
	#define UCONV_IN_ACCEPT_BOM 0x0080
	#define UCONV_OUT_EMIT_BOM 0x0100

	extern int uconv_u16tou32(const uint16_t , size_t , uint32_t , size_t ,
	int);
	extern int uconv_u16tou8(const uint16_t , size_t , uchar_t , size_t , int);
	extern int uconv_u32tou16(const uint32_t , size_t , uint16_t , size_t ,
	int);
	extern int uconv_u32tou8(const uint32_t , size_t , uchar_t , size_t , int);
	extern int uconv_u8tou16(const uchar_t , size_t , uint16_t , size_t , int);
	extern int uconv_u8tou32(const uchar_t , size_t , uint32_t , size_t , int);

	/*
	* UTF-8 text preparation functions and their macros.
	*
	* Among the macros defined, U8_CANON_DECOMP, U8_COMPAT_DECOMP, and
	* U8_CANON_COMP are not public interfaces and must not be used directly
	* at the flag input argument.
	*/
	#define U8_STRCMP_CS (0x00000001)
	#define U8_STRCMP_CI_UPPER (0x00000002)
	#define U8_STRCMP_CI_LOWER (0x00000004)

	#define U8_CANON_DECOMP (0x00000010)
	#define U8_COMPAT_DECOMP (0x00000020)
	#define U8_CANON_COMP (0x00000040)

	#define U8_STRCMP_NFD (U8_CANON_DECOMP)
	#define U8_STRCMP_NFC (U8_CANON_DECOMP \| U8_CANON_COMP)
	#define U8_STRCMP_NFKD (U8_COMPAT_DECOMP)
	#define U8_STRCMP_NFKC (U8_COMPAT_DECOMP \| U8_CANON_COMP)

	#define U8_TEXTPREP_TOUPPER (U8_STRCMP_CI_UPPER)
	#define U8_TEXTPREP_TOLOWER (U8_STRCMP_CI_LOWER)

	#define U8_TEXTPREP_NFD (U8_STRCMP_NFD)
	#define U8_TEXTPREP_NFC (U8_STRCMP_NFC)
	#define U8_TEXTPREP_NFKD (U8_STRCMP_NFKD)
	#define U8_TEXTPREP_NFKC (U8_STRCMP_NFKC)

	#define U8_TEXTPREP_IGNORE_NULL (0x00010000)
	#define U8_TEXTPREP_IGNORE_INVALID (0x00020000)
	#define U8_TEXTPREP_NOWAIT (0x00040000)

	#define U8_UNICODE_320 (0)
	#define U8_UNICODE_500 (1)
	#define U8_UNICODE_LATEST (U8_UNICODE_500)

	#define U8_VALIDATE_ENTIRE (0x00100000)
	#define U8_VALIDATE_CHECK_ADDITIONAL (0x00200000)
	#define U8_VALIDATE_UCS2_RANGE (0x00400000)

	#define U8_ILLEGAL_CHAR (-1)
	#define U8_OUT_OF_RANGE_CHAR (-2)

	-extern int u8_validate(char , size_t, char , int, int );
	+extern int u8_validate(const char , size_t, char , int, int );
	extern int u8_strcmp(const char , const char , size_t, int, size_t, int *);
	extern size_t u8_textprep_str(char , size_t , char , size_t , int, size_t,
	int *);

	#ifdef __cplusplus
	}
	#endif

	#endif /* _SYS_U8_TEXTPREP_H */
	Index: vendor-sys/openzfs/dist/include/sys/zfs_context.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/sys/zfs_context.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/sys/zfs_context.h (revision 366774)
	@@ -1,768 +1,776 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright 2011 Nexenta Systems, Inc. All rights reserved.
	* Copyright (c) 2012, 2018 by Delphix. All rights reserved.
	* Copyright (c) 2012, Joyent, Inc. All rights reserved.
	*/

	#ifndef _SYS_ZFS_CONTEXT_H
	#define _SYS_ZFS_CONTEXT_H

	#ifdef __cplusplus
	extern "C" {
	#endif

	-#ifdef __KERNEL__
	+/*
	+ * This code compiles in three different contexts. When __KERNEL__ is defined,
	+ * the code uses "unix-like" kernel interfaces. When _STANDALONE is defined, the
	+ * code is running in a reduced capacity environment of the boot loader which is
	+ * generally a subset of both POSIX and kernel interfaces (with a few unique
	+ * interfaces too). When neither are defined, it's in a userland POSIX or
	+ * similar environment.
	+ */
	+#if defined(__KERNEL__) \|\| defined(_STANDALONE)
	#include <sys/note.h>
	#include <sys/types.h>
	#include <sys/atomic.h>
	#include <sys/sysmacros.h>
	#include <sys/vmsystm.h>
	#include <sys/condvar.h>
	#include <sys/cmn_err.h>
	#include <sys/kmem.h>
	#include <sys/kmem_cache.h>
	#include <sys/vmem.h>
	#include <sys/taskq.h>
	#include <sys/param.h>
	#include <sys/disp.h>
	#include <sys/debug.h>
	#include <sys/random.h>
	#include <sys/strings.h>
	#include <sys/byteorder.h>
	#include <sys/list.h>
	#include <sys/time.h>
	#include <sys/zone.h>
	#include <sys/kstat.h>
	#include <sys/zfs_debug.h>
	#include <sys/sysevent.h>
	#include <sys/sysevent/eventdefs.h>
	#include <sys/zfs_delay.h>
	#include <sys/sunddi.h>
	#include <sys/ctype.h>
	#include <sys/disp.h>
	#include <sys/trace.h>
	#include <sys/procfs_list.h>
	#include <sys/mod.h>
	#include <sys/zfs_context_os.h>
	-#else /* _KERNEL */
	+#else /* _KERNEL \|\| _STANDALONE */

	#define _SYS_MUTEX_H
	#define _SYS_RWLOCK_H
	#define _SYS_CONDVAR_H
	#define _SYS_VNODE_H
	#define _SYS_VFS_H
	#define _SYS_SUNDDI_H
	#define _SYS_CALLB_H

	#include <stdio.h>
	#include <stdlib.h>
	#include <stddef.h>
	#include <stdarg.h>
	#include <fcntl.h>
	#include <unistd.h>
	#include <errno.h>
	#include <string.h>
	#include <strings.h>
	#include <pthread.h>
	#include <setjmp.h>
	#include <assert.h>
	#include <umem.h>
	#include <limits.h>
	#include <atomic.h>
	#include <dirent.h>
	#include <time.h>
	#include <ctype.h>
	#include <signal.h>
	#include <sys/mman.h>
	#include <sys/note.h>
	#include <sys/types.h>
	#include <sys/cred.h>
	#include <sys/sysmacros.h>
	#include <sys/resource.h>
	#include <sys/byteorder.h>
	#include <sys/list.h>
	#include <sys/mod.h>
	#include <sys/uio.h>
	#include <sys/zfs_debug.h>
	#include <sys/kstat.h>
	#include <sys/u8_textprep.h>
	#include <sys/sysevent.h>
	#include <sys/sysevent/eventdefs.h>
	#include <sys/sunddi.h>
	#include <sys/debug.h>
	#include <sys/utsname.h>
	#include <sys/trace_zfs.h>

	#include <sys/zfs_context_os.h>

	/*
	* Stack
	*/

	#define noinline __attribute__((noinline))
	#define likely(x) __builtin_expect((x), 1)
	#define unlikely(x) __builtin_expect((x), 0)

	/*
	* Debugging
	*/

	/*
	* Note that we are not using the debugging levels.
	*/

	#define CE_CONT 0 /* continuation */
	#define CE_NOTE 1 /* notice */
	#define CE_WARN 2 /* warning */
	#define CE_PANIC 3 /* panic */
	#define CE_IGNORE 4 /* print nothing */

	/*
	* ZFS debugging
	*/

	extern void dprintf_setup(int argc, char *argv);

	extern void cmn_err(int, const char *, ...);
	extern void vcmn_err(int, const char *, va_list);
	extern void panic(const char *, ...) __NORETURN;
	extern void vpanic(const char *, va_list) __NORETURN;

	#define fm_panic panic

	extern int aok;

	/*
	* DTrace SDT probes have different signatures in userland than they do in
	* the kernel. If they're being used in kernel code, re-define them out of
	* existence for their counterparts in libzpool.
	*
	* Here's an example of how to use the set-error probes in userland:
	* zfs$target:::set-error /arg0 == EBUSY/ {stack();}
	*
	* Here's an example of how to use DTRACE_PROBE probes in userland:
	* If there is a probe declared as follows:
	* DTRACE_PROBE2(zfs__probe_name, uint64_t, blkid, dnode_t *, dn);
	* Then you can use it as follows:
	* zfs$target:::probe2 /copyinstr(arg0) == "zfs__probe_name"/
	* {printf("%u %p\n", arg1, arg2);}
	*/

	#ifdef DTRACE_PROBE
	#undef DTRACE_PROBE
	#endif /* DTRACE_PROBE */
	#define DTRACE_PROBE(a)

	#ifdef DTRACE_PROBE1
	#undef DTRACE_PROBE1
	#endif /* DTRACE_PROBE1 */
	#define DTRACE_PROBE1(a, b, c)

	#ifdef DTRACE_PROBE2
	#undef DTRACE_PROBE2
	#endif /* DTRACE_PROBE2 */
	#define DTRACE_PROBE2(a, b, c, d, e)

	#ifdef DTRACE_PROBE3
	#undef DTRACE_PROBE3
	#endif /* DTRACE_PROBE3 */
	#define DTRACE_PROBE3(a, b, c, d, e, f, g)

	#ifdef DTRACE_PROBE4
	#undef DTRACE_PROBE4
	#endif /* DTRACE_PROBE4 */
	#define DTRACE_PROBE4(a, b, c, d, e, f, g, h, i)

	/*
	* Tunables.
	*/
	typedef struct zfs_kernel_param {
	const char name; / unused stub */
	} zfs_kernel_param_t;

	#define ZFS_MODULE_PARAM(scope_prefix, name_prefix, name, type, perm, desc)
	#define ZFS_MODULE_PARAM_ARGS void
	#define ZFS_MODULE_PARAM_CALL(scope_prefix, name_prefix, name, setfunc, \
	getfunc, perm, desc)

	/*
	* Threads.
	*/
	typedef pthread_t kthread_t;

	#define TS_RUN 0x00000002
	#define TS_JOINABLE 0x00000004

	#define curthread ((void *)(uintptr_t)pthread_self())
	#define kpreempt(x) yield()
	#define getcomm() "unknown"

	#define thread_create_named(name, stk, stksize, func, arg, len, \
	pp, state, pri) \
	zk_thread_create(func, arg, stksize, state)
	#define thread_create(stk, stksize, func, arg, len, pp, state, pri) \
	zk_thread_create(func, arg, stksize, state)
	#define thread_exit() pthread_exit(NULL)
	#define thread_join(t) pthread_join((pthread_t)(t), NULL)

	#define newproc(f, a, cid, pri, ctp, pid) (ENOSYS)

	/* in libzpool, p0 exists only to have its address taken */
	typedef struct proc {
	uintptr_t this_is_never_used_dont_dereference_it;
	} proc_t;

	extern struct proc p0;
	#define curproc (&p0)

	#define PS_NONE -1

	extern kthread_t zk_thread_create(void (func)(void ), void arg,
	size_t stksize, int state);

	#define issig(why) (FALSE)
	#define ISSIG(thr, why) (FALSE)

	#define kpreempt_disable() ((void)0)
	#define kpreempt_enable() ((void)0)
	#define cond_resched() sched_yield()

	/*
	* Mutexes
	*/
	typedef struct kmutex {
	pthread_mutex_t m_lock;
	pthread_t m_owner;
	} kmutex_t;

	#define MUTEX_DEFAULT 0
	#define MUTEX_NOLOCKDEP MUTEX_DEFAULT
	#define MUTEX_HELD(mp) pthread_equal((mp)->m_owner, pthread_self())
	#define MUTEX_NOT_HELD(mp) !MUTEX_HELD(mp)

	extern void mutex_init(kmutex_t mp, char name, int type, void *cookie);
	extern void mutex_destroy(kmutex_t *mp);
	extern void mutex_enter(kmutex_t *mp);
	extern void mutex_exit(kmutex_t *mp);
	extern int mutex_tryenter(kmutex_t *mp);

	#define NESTED_SINGLE 1
	#define mutex_enter_nested(mp, class) mutex_enter(mp)
	/*
	* RW locks
	*/
	typedef struct krwlock {
	pthread_rwlock_t rw_lock;
	pthread_t rw_owner;
	uint_t rw_readers;
	} krwlock_t;

	typedef int krw_t;

	#define RW_READER 0
	#define RW_WRITER 1
	#define RW_DEFAULT RW_READER
	#define RW_NOLOCKDEP RW_READER

	#define RW_READ_HELD(rw) ((rw)->rw_readers > 0)
	#define RW_WRITE_HELD(rw) pthread_equal((rw)->rw_owner, pthread_self())
	#define RW_LOCK_HELD(rw) (RW_READ_HELD(rw) \|\| RW_WRITE_HELD(rw))

	extern void rw_init(krwlock_t rwlp, char name, int type, void *arg);
	extern void rw_destroy(krwlock_t *rwlp);
	extern void rw_enter(krwlock_t *rwlp, krw_t rw);
	extern int rw_tryenter(krwlock_t *rwlp, krw_t rw);
	extern int rw_tryupgrade(krwlock_t *rwlp);
	extern void rw_exit(krwlock_t *rwlp);
	#define rw_downgrade(rwlp) do { } while (0)

	/*
	* Credentials
	*/
	extern uid_t crgetuid(cred_t *cr);
	extern uid_t crgetruid(cred_t *cr);
	extern gid_t crgetgid(cred_t *cr);
	extern int crgetngroups(cred_t *cr);
	extern gid_t crgetgroups(cred_t cr);

	/*
	* Condition variables
	*/
	typedef pthread_cond_t kcondvar_t;

	#define CV_DEFAULT 0
	#define CALLOUT_FLAG_ABSOLUTE 0x2

	extern void cv_init(kcondvar_t cv, char name, int type, void *arg);
	extern void cv_destroy(kcondvar_t *cv);
	extern void cv_wait(kcondvar_t cv, kmutex_t mp);
	extern int cv_wait_sig(kcondvar_t cv, kmutex_t mp);
	extern int cv_timedwait(kcondvar_t cv, kmutex_t mp, clock_t abstime);
	extern int cv_timedwait_hires(kcondvar_t cvp, kmutex_t mp, hrtime_t tim,
	hrtime_t res, int flag);
	extern void cv_signal(kcondvar_t *cv);
	extern void cv_broadcast(kcondvar_t *cv);

	#define cv_timedwait_io(cv, mp, at) cv_timedwait(cv, mp, at)
	#define cv_timedwait_idle(cv, mp, at) cv_timedwait(cv, mp, at)
	#define cv_timedwait_sig(cv, mp, at) cv_timedwait(cv, mp, at)
	#define cv_wait_io(cv, mp) cv_wait(cv, mp)
	#define cv_wait_idle(cv, mp) cv_wait(cv, mp)
	#define cv_wait_io_sig(cv, mp) cv_wait_sig(cv, mp)
	#define cv_timedwait_sig_hires(cv, mp, t, r, f) \
	cv_timedwait_hires(cv, mp, t, r, f)
	#define cv_timedwait_idle_hires(cv, mp, t, r, f) \
	cv_timedwait_hires(cv, mp, t, r, f)

	/*
	* Thread-specific data
	*/
	#define tsd_get(k) pthread_getspecific(k)
	#define tsd_set(k, v) pthread_setspecific(k, v)
	#define tsd_create(kp, d) pthread_key_create((pthread_key_t *)kp, d)
	#define tsd_destroy(kp) /* nothing */
	#ifdef __FreeBSD__
	typedef off_t loff_t;
	#endif

	/*
	* kstat creation, installation and deletion
	*/
	extern kstat_t kstat_create(const char , int,
	const char , const char , uchar_t, ulong_t, uchar_t);
	extern void kstat_install(kstat_t *);
	extern void kstat_delete(kstat_t *);
	extern void kstat_waitq_enter(kstat_io_t *);
	extern void kstat_waitq_exit(kstat_io_t *);
	extern void kstat_runq_enter(kstat_io_t *);
	extern void kstat_runq_exit(kstat_io_t *);
	extern void kstat_waitq_to_runq(kstat_io_t *);
	extern void kstat_runq_back_to_waitq(kstat_io_t *);
	extern void kstat_set_raw_ops(kstat_t *ksp,
	int (headers)(char buf, size_t size),
	int (data)(char buf, size_t size, void *data),
	void (addr)(kstat_t *ksp, loff_t index));

	/*
	* procfs list manipulation
	*/

	typedef struct procfs_list {
	void *pl_private;
	kmutex_t pl_lock;
	list_t pl_list;
	uint64_t pl_next_id;
	size_t pl_node_offset;
	} procfs_list_t;

	#ifndef __cplusplus
	struct seq_file { };
	void seq_printf(struct seq_file m, const char fmt, ...);

	typedef struct procfs_list_node {
	list_node_t pln_link;
	uint64_t pln_id;
	} procfs_list_node_t;

	void procfs_list_install(const char *module,
	const char *submodule,
	const char *name,
	mode_t mode,
	procfs_list_t *procfs_list,
	int (show)(struct seq_file f, void *p),
	int (show_header)(struct seq_file f),
	int (clear)(procfs_list_t procfs_list),
	size_t procfs_list_node_off);
	void procfs_list_uninstall(procfs_list_t *procfs_list);
	void procfs_list_destroy(procfs_list_t *procfs_list);
	void procfs_list_add(procfs_list_t procfs_list, void p);
	#endif

	/*
	* Kernel memory
	*/
	#define KM_SLEEP UMEM_NOFAIL
	#define KM_PUSHPAGE KM_SLEEP
	#define KM_NOSLEEP UMEM_DEFAULT
	#define KM_NORMALPRI 0 /* not needed with UMEM_DEFAULT */
	#define KMC_NODEBUG UMC_NODEBUG
	#define KMC_KVMEM 0x0
	#define kmem_alloc(_s, _f) umem_alloc(_s, _f)
	#define kmem_zalloc(_s, _f) umem_zalloc(_s, _f)
	#define kmem_free(_b, _s) umem_free(_b, _s)
	#define vmem_alloc(_s, _f) kmem_alloc(_s, _f)
	#define vmem_zalloc(_s, _f) kmem_zalloc(_s, _f)
	#define vmem_free(_b, _s) kmem_free(_b, _s)
	#define kmem_cache_create(_a, _b, _c, _d, _e, _f, _g, _h, _i) \
	umem_cache_create(_a, _b, _c, _d, _e, _f, _g, _h, _i)
	#define kmem_cache_destroy(_c) umem_cache_destroy(_c)
	#define kmem_cache_alloc(_c, _f) umem_cache_alloc(_c, _f)
	#define kmem_cache_free(_c, _b) umem_cache_free(_c, _b)
	#define kmem_debugging() 0
	#define kmem_cache_reap_now(_c) umem_cache_reap_now(_c);
	#define kmem_cache_set_move(_c, _cb) /* nothing */
	#define POINTER_INVALIDATE(_pp) /* nothing */
	#define POINTER_IS_VALID(_p) 0

	typedef umem_cache_t kmem_cache_t;

	typedef enum kmem_cbrc {
	KMEM_CBRC_YES,
	KMEM_CBRC_NO,
	KMEM_CBRC_LATER,
	KMEM_CBRC_DONT_NEED,
	KMEM_CBRC_DONT_KNOW
	} kmem_cbrc_t;

	/*
	* Task queues
	*/

	#define TASKQ_NAMELEN 31

	typedef uintptr_t taskqid_t;
	typedef void (task_func_t)(void *);

	typedef struct taskq_ent {
	struct taskq_ent *tqent_next;
	struct taskq_ent *tqent_prev;
	task_func_t *tqent_func;
	void *tqent_arg;
	uintptr_t tqent_flags;
	} taskq_ent_t;

	typedef struct taskq {
	char tq_name[TASKQ_NAMELEN + 1];
	kmutex_t tq_lock;
	krwlock_t tq_threadlock;
	kcondvar_t tq_dispatch_cv;
	kcondvar_t tq_wait_cv;
	kthread_t **tq_threadlist;
	int tq_flags;
	int tq_active;
	int tq_nthreads;
	int tq_nalloc;
	int tq_minalloc;
	int tq_maxalloc;
	kcondvar_t tq_maxalloc_cv;
	int tq_maxalloc_wait;
	taskq_ent_t *tq_freelist;
	taskq_ent_t tq_task;
	} taskq_t;

	#define TQENT_FLAG_PREALLOC 0x1 /* taskq_dispatch_ent used */

	#define TASKQ_PREPOPULATE 0x0001
	#define TASKQ_CPR_SAFE 0x0002 /* Use CPR safe protocol */
	#define TASKQ_DYNAMIC 0x0004 /* Use dynamic thread scheduling */
	#define TASKQ_THREADS_CPU_PCT 0x0008 /* Scale # threads by # cpus */
	#define TASKQ_DC_BATCH 0x0010 /* Mark threads as batch */

	#define TQ_SLEEP KM_SLEEP /* Can block for memory */
	#define TQ_NOSLEEP KM_NOSLEEP /* cannot block for memory; may fail */
	#define TQ_NOQUEUE 0x02 /* Do not enqueue if can't dispatch */
	#define TQ_FRONT 0x08 /* Queue in front */

	#define TASKQID_INVALID ((taskqid_t)0)

	extern taskq_t *system_taskq;
	extern taskq_t *system_delay_taskq;

	extern taskq_t taskq_create(const char , int, pri_t, int, int, uint_t);
	#define taskq_create_proc(a, b, c, d, e, p, f) \
	(taskq_create(a, b, c, d, e, f))
	#define taskq_create_sysdc(a, b, d, e, p, dc, f) \
	(taskq_create(a, b, maxclsyspri, d, e, f))
	extern taskqid_t taskq_dispatch(taskq_t , task_func_t, void , uint_t);
	extern taskqid_t taskq_dispatch_delay(taskq_t , task_func_t, void , uint_t,
	clock_t);
	extern void taskq_dispatch_ent(taskq_t , task_func_t, void , uint_t,
	taskq_ent_t *);
	extern int taskq_empty_ent(taskq_ent_t *);
	extern void taskq_init_ent(taskq_ent_t *);
	extern void taskq_destroy(taskq_t *);
	extern void taskq_wait(taskq_t *);
	extern void taskq_wait_id(taskq_t *, taskqid_t);
	extern void taskq_wait_outstanding(taskq_t *, taskqid_t);
	extern int taskq_member(taskq_t , kthread_t );
	extern taskq_t *taskq_of_curthread(void);
	extern int taskq_cancel_id(taskq_t *, taskqid_t);
	extern void system_taskq_init(void);
	extern void system_taskq_fini(void);

	#define XVA_MAPSIZE 3
	#define XVA_MAGIC 0x78766174

	extern char *vn_dumpdir;
	#define AV_SCANSTAMP_SZ 32 /* length of anti-virus scanstamp */

	typedef struct xoptattr {
	inode_timespec_t xoa_createtime; /* Create time of file */
	uint8_t xoa_archive;
	uint8_t xoa_system;
	uint8_t xoa_readonly;
	uint8_t xoa_hidden;
	uint8_t xoa_nounlink;
	uint8_t xoa_immutable;
	uint8_t xoa_appendonly;
	uint8_t xoa_nodump;
	uint8_t xoa_settable;
	uint8_t xoa_opaque;
	uint8_t xoa_av_quarantined;
	uint8_t xoa_av_modified;
	uint8_t xoa_av_scanstamp[AV_SCANSTAMP_SZ];
	uint8_t xoa_reparse;
	uint8_t xoa_offline;
	uint8_t xoa_sparse;
	} xoptattr_t;

	typedef struct vattr {
	uint_t va_mask; /* bit-mask of attributes */
	u_offset_t va_size; /* file size in bytes */
	} vattr_t;


	typedef struct xvattr {
	vattr_t xva_vattr; /* Embedded vattr structure */
	uint32_t xva_magic; /* Magic Number */
	uint32_t xva_mapsize; /* Size of attr bitmap (32-bit words) */
	uint32_t xva_rtnattrmapp; / Ptr to xva_rtnattrmap[] */
	uint32_t xva_reqattrmap[XVA_MAPSIZE]; /* Requested attrs */
	uint32_t xva_rtnattrmap[XVA_MAPSIZE]; /* Returned attrs */
	xoptattr_t xva_xoptattrs; /* Optional attributes */
	} xvattr_t;

	typedef struct vsecattr {
	uint_t vsa_mask; /* See below */
	int vsa_aclcnt; /* ACL entry count */
	void vsa_aclentp; / pointer to ACL entries */
	int vsa_dfaclcnt; /* default ACL entry count */
	void vsa_dfaclentp; / pointer to default ACL entries */
	size_t vsa_aclentsz; /* ACE size in bytes of vsa_aclentp */
	} vsecattr_t;

	#define AT_MODE 0x00002
	#define AT_UID 0x00004
	#define AT_GID 0x00008
	#define AT_FSID 0x00010
	#define AT_NODEID 0x00020
	#define AT_NLINK 0x00040
	#define AT_SIZE 0x00080
	#define AT_ATIME 0x00100
	#define AT_MTIME 0x00200
	#define AT_CTIME 0x00400
	#define AT_RDEV 0x00800
	#define AT_BLKSIZE 0x01000
	#define AT_NBLOCKS 0x02000
	#define AT_SEQ 0x08000
	#define AT_XVATTR 0x10000

	#define CRCREAT 0

	#define F_FREESP 11
	#define FIGNORECASE 0x80000 /* request case-insensitive lookups */

	/*
	* Random stuff
	*/
	#define ddi_get_lbolt() (gethrtime() >> 23)
	#define ddi_get_lbolt64() (gethrtime() >> 23)
	#define hz 119 /* frequency when using gethrtime() >> 23 for lbolt */

	#define ddi_time_before(a, b) (a < b)
	#define ddi_time_after(a, b) ddi_time_before(b, a)
	#define ddi_time_before_eq(a, b) (!ddi_time_after(a, b))
	#define ddi_time_after_eq(a, b) ddi_time_before_eq(b, a)

	#define ddi_time_before64(a, b) (a < b)
	#define ddi_time_after64(a, b) ddi_time_before64(b, a)
	#define ddi_time_before_eq64(a, b) (!ddi_time_after64(a, b))
	#define ddi_time_after_eq64(a, b) ddi_time_before_eq64(b, a)

	extern void delay(clock_t ticks);

	#define SEC_TO_TICK(sec) ((sec) * hz)
	#define MSEC_TO_TICK(msec) (howmany((hrtime_t)(msec) * hz, MILLISEC))
	#define USEC_TO_TICK(usec) (howmany((hrtime_t)(usec) * hz, MICROSEC))
	#define NSEC_TO_TICK(nsec) (howmany((hrtime_t)(nsec) * hz, NANOSEC))

	#define max_ncpus 64
	#define boot_ncpus (sysconf(_SC_NPROCESSORS_ONLN))

	/*
	* Process priorities as defined by setpriority(2) and getpriority(2).
	*/
	#define minclsyspri 19
	#define maxclsyspri -20
	#define defclsyspri 0

	#define CPU_SEQID ((uintptr_t)pthread_self() & (max_ncpus - 1))

	#define kcred NULL
	#define CRED() NULL

	#define ptob(x) ((x) * PAGESIZE)

	#define NN_DIVISOR_1000 (1U << 0)
	#define NN_NUMBUF_SZ (6)

	extern uint64_t physmem;
	extern char *random_path;
	extern char *urandom_path;

	extern int highbit64(uint64_t i);
	extern int lowbit64(uint64_t i);
	extern int random_get_bytes(uint8_t *ptr, size_t len);
	extern int random_get_pseudo_bytes(uint8_t *ptr, size_t len);

	extern void kernel_init(int mode);
	extern void kernel_fini(void);
	extern void random_init(void);
	extern void random_fini(void);

	struct spa;
	extern void show_pool_stats(struct spa *);
	extern int set_global_var(char *arg);

	typedef struct callb_cpr {
	kmutex_t *cc_lockp;
	} callb_cpr_t;

	#define CALLB_CPR_INIT(cp, lockp, func, name) { \
	(cp)->cc_lockp = lockp; \
	}

	#define CALLB_CPR_SAFE_BEGIN(cp) { \
	ASSERT(MUTEX_HELD((cp)->cc_lockp)); \
	}

	#define CALLB_CPR_SAFE_END(cp, lockp) { \
	ASSERT(MUTEX_HELD((cp)->cc_lockp)); \
	}

	#define CALLB_CPR_EXIT(cp) { \
	ASSERT(MUTEX_HELD((cp)->cc_lockp)); \
	mutex_exit((cp)->cc_lockp); \
	}

	#define zone_dataset_visible(x, y) (1)
	#define INGLOBALZONE(z) (1)
	extern uint32_t zone_get_hostid(void *zonep);

	extern char kmem_vasprintf(const char fmt, va_list adx);
	extern char kmem_asprintf(const char fmt, ...);
	#define kmem_strfree(str) kmem_free((str), strlen(str) + 1)
	#define kmem_strdup(s) strdup(s)

	/*
	* Hostname information
	*/
	extern char hw_serial[]; /* for userland-emulated hostid access */
	extern int ddi_strtoul(const char str, char *nptr, int base,
	unsigned long *result);

	extern int ddi_strtoull(const char str, char *nptr, int base,
	u_longlong_t *result);

	typedef struct utsname utsname_t;
	extern utsname_t *utsname(void);

	/* ZFS Boot Related stuff. */

	struct _buf {
	intptr_t _fd;
	};

	struct bootstat {
	uint64_t st_size;
	};

	typedef struct ace_object {
	uid_t a_who;
	uint32_t a_access_mask;
	uint16_t a_flags;
	uint16_t a_type;
	uint8_t a_obj_type[16];
	uint8_t a_inherit_obj_type[16];
	} ace_object_t;


	#define ACE_ACCESS_ALLOWED_OBJECT_ACE_TYPE 0x05
	#define ACE_ACCESS_DENIED_OBJECT_ACE_TYPE 0x06
	#define ACE_SYSTEM_AUDIT_OBJECT_ACE_TYPE 0x07
	#define ACE_SYSTEM_ALARM_OBJECT_ACE_TYPE 0x08

	extern int zfs_secpolicy_snapshot_perms(const char name, cred_t cr);
	extern int zfs_secpolicy_rename_perms(const char from, const char to,
	cred_t *cr);
	extern int zfs_secpolicy_destroy_perms(const char name, cred_t cr);
	extern int secpolicy_zfs(const cred_t *cr);
	extern int secpolicy_zfs_proc(const cred_t cr, proc_t proc);
	extern zoneid_t getzoneid(void);

	/* SID stuff */
	typedef struct ksiddomain {
	uint_t kd_ref;
	uint_t kd_len;
	char *kd_name;
	} ksiddomain_t;

	ksiddomain_t ksid_lookupdomain(const char );
	void ksiddomain_rele(ksiddomain_t *);

	#define DDI_SLEEP KM_SLEEP
	#define ddi_log_sysevent(_a, _b, _c, _d, _e, _f, _g) \
	sysevent_post_event(_c, _d, _b, "libzpool", _e, _f)

	#define zfs_sleep_until(wakeup) \
	do { \
	hrtime_t delta = wakeup - gethrtime(); \
	struct timespec ts; \
	ts.tv_sec = delta / NANOSEC; \
	ts.tv_nsec = delta % NANOSEC; \
	(void) nanosleep(&ts, NULL); \
	} while (0)

	typedef int fstrans_cookie_t;

	extern fstrans_cookie_t spl_fstrans_mark(void);
	extern void spl_fstrans_unmark(fstrans_cookie_t);
	extern int __spl_pf_fstrans_check(void);
	extern int kmem_cache_reap_active(void);

	#define ____cacheline_aligned

	/*
	* Kernel modules
	*/
	#define __init
	#define __exit

	-#endif /* _KERNEL */
	+#endif /* _KERNEL \|\| _STANDALONE */

	#ifdef __cplusplus
	};
	#endif

	#endif /* _SYS_ZFS_CONTEXT_H */
	Index: vendor-sys/openzfs/dist/include/sys/zfs_ioctl.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/sys/zfs_ioctl.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/sys/zfs_ioctl.h (revision 366774)
	@@ -1,581 +1,582 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2012, 2020 by Delphix. All rights reserved.
	* Copyright 2016 RackTop Systems.
	* Copyright (c) 2017, Intel Corporation.
	*/

	#ifndef _SYS_ZFS_IOCTL_H
	#define _SYS_ZFS_IOCTL_H

	#include <sys/cred.h>
	#include <sys/dmu.h>
	#include <sys/zio.h>
	#include <sys/dsl_deleg.h>
	#include <sys/spa.h>
	#include <sys/zfs_stat.h>

	#ifdef _KERNEL
	#include <sys/nvpair.h>
	#endif /* _KERNEL */

	#ifdef __cplusplus
	extern "C" {
	#endif

	/*
	* The structures in this file are passed between userland and the
	* kernel. Userland may be running a 32-bit process, while the kernel
	* is 64-bit. Therefore, these structures need to compile the same in
	* 32-bit and 64-bit. This means not using type "long", and adding
	* explicit padding so that the 32-bit structure will not be packed more
	* tightly than the 64-bit structure (which requires 64-bit alignment).
	*/

	/*
	* Property values for snapdir
	*/
	#define ZFS_SNAPDIR_HIDDEN 0
	#define ZFS_SNAPDIR_VISIBLE 1

	/*
	* Property values for snapdev
	*/
	#define ZFS_SNAPDEV_HIDDEN 0
	#define ZFS_SNAPDEV_VISIBLE 1
	/*
	* Property values for acltype
	*/
	#define ZFS_ACLTYPE_OFF 0
	#define ZFS_ACLTYPE_POSIX 1
	+#define ZFS_ACLTYPE_NFSV4 2

	/*
	* Field manipulation macros for the drr_versioninfo field of the
	* send stream header.
	*/

	/*
	* Header types for zfs send streams.
	*/
	typedef enum drr_headertype {
	DMU_SUBSTREAM = 0x1,
	DMU_COMPOUNDSTREAM = 0x2
	} drr_headertype_t;

	#define DMU_GET_STREAM_HDRTYPE(vi) BF64_GET((vi), 0, 2)
	#define DMU_SET_STREAM_HDRTYPE(vi, x) BF64_SET((vi), 0, 2, x)

	#define DMU_GET_FEATUREFLAGS(vi) BF64_GET((vi), 2, 30)
	#define DMU_SET_FEATUREFLAGS(vi, x) BF64_SET((vi), 2, 30, x)

	/*
	* Feature flags for zfs send streams (flags in drr_versioninfo)
	*/

	#define DMU_BACKUP_FEATURE_DEDUP (1 << 0)
	#define DMU_BACKUP_FEATURE_DEDUPPROPS (1 << 1)
	#define DMU_BACKUP_FEATURE_SA_SPILL (1 << 2)
	/* flags #3 - #15 are reserved for incompatible closed-source implementations */
	#define DMU_BACKUP_FEATURE_EMBED_DATA (1 << 16)
	#define DMU_BACKUP_FEATURE_LZ4 (1 << 17)
	/* flag #18 is reserved for a Delphix feature */
	#define DMU_BACKUP_FEATURE_LARGE_BLOCKS (1 << 19)
	#define DMU_BACKUP_FEATURE_RESUMING (1 << 20)
	#define DMU_BACKUP_FEATURE_REDACTED (1 << 21)
	#define DMU_BACKUP_FEATURE_COMPRESSED (1 << 22)
	#define DMU_BACKUP_FEATURE_LARGE_DNODE (1 << 23)
	#define DMU_BACKUP_FEATURE_RAW (1 << 24)
	#define DMU_BACKUP_FEATURE_ZSTD (1 << 25)
	#define DMU_BACKUP_FEATURE_HOLDS (1 << 26)
	/*
	* The SWITCH_TO_LARGE_BLOCKS feature indicates that we can receive
	* incremental LARGE_BLOCKS streams (those with WRITE records of >128KB) even
	* if the previous send did not use LARGE_BLOCKS, and thus its large blocks
	* were split into multiple 128KB WRITE records. (See
	* flush_write_batch_impl() and receive_object()). Older software that does
	* not support this flag may encounter a bug when switching to large blocks,
	* which causes files to incorrectly be zeroed.
	*
	* This flag is currently not set on any send streams. In the future, we
	* intend for incremental send streams of snapshots that have large blocks to
	* use LARGE_BLOCKS by default, and these streams will also have the
	* SWITCH_TO_LARGE_BLOCKS feature set. This ensures that streams from the
	* default use of "zfs send" won't encounter the bug mentioned above.
	*/
	#define DMU_BACKUP_FEATURE_SWITCH_TO_LARGE_BLOCKS (1 << 27)

	/*
	* Mask of all supported backup features
	*/
	#define DMU_BACKUP_FEATURE_MASK (DMU_BACKUP_FEATURE_SA_SPILL \| \
	DMU_BACKUP_FEATURE_EMBED_DATA \| DMU_BACKUP_FEATURE_LZ4 \| \
	DMU_BACKUP_FEATURE_RESUMING \| DMU_BACKUP_FEATURE_LARGE_BLOCKS \| \
	DMU_BACKUP_FEATURE_COMPRESSED \| DMU_BACKUP_FEATURE_LARGE_DNODE \| \
	DMU_BACKUP_FEATURE_RAW \| DMU_BACKUP_FEATURE_HOLDS \| \
	DMU_BACKUP_FEATURE_REDACTED \| DMU_BACKUP_FEATURE_SWITCH_TO_LARGE_BLOCKS \| \
	DMU_BACKUP_FEATURE_ZSTD)

	/* Are all features in the given flag word currently supported? */
	#define DMU_STREAM_SUPPORTED(x) (!((x) & ~DMU_BACKUP_FEATURE_MASK))

	typedef enum dmu_send_resume_token_version {
	ZFS_SEND_RESUME_TOKEN_VERSION = 1
	} dmu_send_resume_token_version_t;

	/*
	* The drr_versioninfo field of the dmu_replay_record has the
	* following layout:
	*
	* 64 56 48 40 32 24 16 8 0
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	* \| reserved \| feature-flags \|C\|S\|
	* +-------+-------+-------+-------+-------+-------+-------+-------+
	*
	* The low order two bits indicate the header type: SUBSTREAM (0x1)
	* or COMPOUNDSTREAM (0x2). Using two bits for this is historical:
	* this field used to be a version number, where the two version types
	* were 1 and 2. Using two bits for this allows earlier versions of
	* the code to be able to recognize send streams that don't use any
	* of the features indicated by feature flags.
	*/

	#define DMU_BACKUP_MAGIC 0x2F5bacbacULL

	/*
	* Send stream flags. Bits 24-31 are reserved for vendor-specific
	* implementations and should not be used.
	*/
	#define DRR_FLAG_CLONE (1<<0)
	#define DRR_FLAG_CI_DATA (1<<1)
	/*
	* This send stream, if it is a full send, includes the FREE and FREEOBJECT
	* records that are created by the sending process. This means that the send
	* stream can be received as a clone, even though it is not an incremental.
	* This is not implemented as a feature flag, because the receiving side does
	* not need to have implemented it to receive this stream; it is fully backwards
	* compatible. We need a flag, though, because full send streams without it
	* cannot necessarily be received as a clone correctly.
	*/
	#define DRR_FLAG_FREERECORDS (1<<2)
	/*
	* When DRR_FLAG_SPILL_BLOCK is set it indicates the DRR_OBJECT_SPILL
	* and DRR_SPILL_UNMODIFIED flags are meaningful in the send stream.
	*
	* When DRR_FLAG_SPILL_BLOCK is set, DRR_OBJECT records will have
	* DRR_OBJECT_SPILL set if and only if they should have a spill block
	* (either an existing one, or a new one in the send stream). When clear
	* the object does not have a spill block and any existing spill block
	* should be freed.
	*
	* Similarly, when DRR_FLAG_SPILL_BLOCK is set, DRR_SPILL records will
	* have DRR_SPILL_UNMODIFIED set if and only if they were included for
	* backward compatibility purposes, and can be safely ignored by new versions
	* of zfs receive. Previous versions of ZFS which do not understand the
	* DRR_FLAG_SPILL_BLOCK will process this record and recreate any missing
	* spill blocks.
	*/
	#define DRR_FLAG_SPILL_BLOCK (1<<3)

	/*
	* flags in the drr_flags field in the DRR_WRITE, DRR_SPILL, DRR_OBJECT,
	* DRR_WRITE_BYREF, and DRR_OBJECT_RANGE blocks
	*/
	#define DRR_CHECKSUM_DEDUP (1<<0) /* not used for SPILL records */
	#define DRR_RAW_BYTESWAP (1<<1)
	#define DRR_OBJECT_SPILL (1<<2) /* OBJECT record has a spill block */
	#define DRR_SPILL_UNMODIFIED (1<<2) /* SPILL record for unmodified block */

	#define DRR_IS_DEDUP_CAPABLE(flags) ((flags) & DRR_CHECKSUM_DEDUP)
	#define DRR_IS_RAW_BYTESWAPPED(flags) ((flags) & DRR_RAW_BYTESWAP)
	#define DRR_OBJECT_HAS_SPILL(flags) ((flags) & DRR_OBJECT_SPILL)
	#define DRR_SPILL_IS_UNMODIFIED(flags) ((flags) & DRR_SPILL_UNMODIFIED)

	/* deal with compressed drr_write replay records */
	#define DRR_WRITE_COMPRESSED(drrw) ((drrw)->drr_compressiontype != 0)
	#define DRR_WRITE_PAYLOAD_SIZE(drrw) \
	(DRR_WRITE_COMPRESSED(drrw) ? (drrw)->drr_compressed_size : \
	(drrw)->drr_logical_size)
	#define DRR_SPILL_PAYLOAD_SIZE(drrs) \
	((drrs)->drr_compressed_size ? \
	(drrs)->drr_compressed_size : (drrs)->drr_length)
	#define DRR_OBJECT_PAYLOAD_SIZE(drro) \
	((drro)->drr_raw_bonuslen != 0 ? \
	(drro)->drr_raw_bonuslen : P2ROUNDUP((drro)->drr_bonuslen, 8))

	/*
	* zfs ioctl command structure
	*/

	/* Header is used in C++ so can't forward declare untagged struct */
	struct drr_begin {
	uint64_t drr_magic;
	uint64_t drr_versioninfo; /* was drr_version */
	uint64_t drr_creation_time;
	dmu_objset_type_t drr_type;
	uint32_t drr_flags;
	uint64_t drr_toguid;
	uint64_t drr_fromguid;
	char drr_toname[MAXNAMELEN];
	};

	typedef struct dmu_replay_record {
	enum {
	DRR_BEGIN, DRR_OBJECT, DRR_FREEOBJECTS,
	DRR_WRITE, DRR_FREE, DRR_END, DRR_WRITE_BYREF,
	DRR_SPILL, DRR_WRITE_EMBEDDED, DRR_OBJECT_RANGE, DRR_REDACT,
	DRR_NUMTYPES
	} drr_type;
	uint32_t drr_payloadlen;
	union {
	struct drr_begin drr_begin;
	struct drr_end {
	zio_cksum_t drr_checksum;
	uint64_t drr_toguid;
	} drr_end;
	struct drr_object {
	uint64_t drr_object;
	dmu_object_type_t drr_type;
	dmu_object_type_t drr_bonustype;
	uint32_t drr_blksz;
	uint32_t drr_bonuslen;
	uint8_t drr_checksumtype;
	uint8_t drr_compress;
	uint8_t drr_dn_slots;
	uint8_t drr_flags;
	uint32_t drr_raw_bonuslen;
	uint64_t drr_toguid;
	/* only (possibly) nonzero for raw streams */
	uint8_t drr_indblkshift;
	uint8_t drr_nlevels;
	uint8_t drr_nblkptr;
	uint8_t drr_pad[5];
	uint64_t drr_maxblkid;
	/* bonus content follows */
	} drr_object;
	struct drr_freeobjects {
	uint64_t drr_firstobj;
	uint64_t drr_numobjs;
	uint64_t drr_toguid;
	} drr_freeobjects;
	struct drr_write {
	uint64_t drr_object;
	dmu_object_type_t drr_type;
	uint32_t drr_pad;
	uint64_t drr_offset;
	uint64_t drr_logical_size;
	uint64_t drr_toguid;
	uint8_t drr_checksumtype;
	uint8_t drr_flags;
	uint8_t drr_compressiontype;
	uint8_t drr_pad2[5];
	/* deduplication key */
	ddt_key_t drr_key;
	/* only nonzero if drr_compressiontype is not 0 */
	uint64_t drr_compressed_size;
	/* only nonzero for raw streams */
	uint8_t drr_salt[ZIO_DATA_SALT_LEN];
	uint8_t drr_iv[ZIO_DATA_IV_LEN];
	uint8_t drr_mac[ZIO_DATA_MAC_LEN];
	/* content follows */
	} drr_write;
	struct drr_free {
	uint64_t drr_object;
	uint64_t drr_offset;
	uint64_t drr_length;
	uint64_t drr_toguid;
	} drr_free;
	struct drr_write_byref {
	/* where to put the data */
	uint64_t drr_object;
	uint64_t drr_offset;
	uint64_t drr_length;
	uint64_t drr_toguid;
	/* where to find the prior copy of the data */
	uint64_t drr_refguid;
	uint64_t drr_refobject;
	uint64_t drr_refoffset;
	/* properties of the data */
	uint8_t drr_checksumtype;
	uint8_t drr_flags;
	uint8_t drr_pad2[6];
	ddt_key_t drr_key; /* deduplication key */
	} drr_write_byref;
	struct drr_spill {
	uint64_t drr_object;
	uint64_t drr_length;
	uint64_t drr_toguid;
	uint8_t drr_flags;
	uint8_t drr_compressiontype;
	uint8_t drr_pad[6];
	/* only nonzero for raw streams */
	uint64_t drr_compressed_size;
	uint8_t drr_salt[ZIO_DATA_SALT_LEN];
	uint8_t drr_iv[ZIO_DATA_IV_LEN];
	uint8_t drr_mac[ZIO_DATA_MAC_LEN];
	dmu_object_type_t drr_type;
	/* spill data follows */
	} drr_spill;
	struct drr_write_embedded {
	uint64_t drr_object;
	uint64_t drr_offset;
	/* logical length, should equal blocksize */
	uint64_t drr_length;
	uint64_t drr_toguid;
	uint8_t drr_compression;
	uint8_t drr_etype;
	uint8_t drr_pad[6];
	uint32_t drr_lsize; /* uncompressed size of payload */
	uint32_t drr_psize; /* compr. (real) size of payload */
	/* (possibly compressed) content follows */
	} drr_write_embedded;
	struct drr_object_range {
	uint64_t drr_firstobj;
	uint64_t drr_numslots;
	uint64_t drr_toguid;
	uint8_t drr_salt[ZIO_DATA_SALT_LEN];
	uint8_t drr_iv[ZIO_DATA_IV_LEN];
	uint8_t drr_mac[ZIO_DATA_MAC_LEN];
	uint8_t drr_flags;
	uint8_t drr_pad[3];
	} drr_object_range;
	struct drr_redact {
	uint64_t drr_object;
	uint64_t drr_offset;
	uint64_t drr_length;
	uint64_t drr_toguid;
	} drr_redact;

	/*
	* Note: drr_checksum is overlaid with all record types
	* except DRR_BEGIN. Therefore its (non-pad) members
	* must not overlap with members from the other structs.
	* We accomplish this by putting its members at the very
	* end of the struct.
	*/
	struct drr_checksum {
	uint64_t drr_pad[34];
	/*
	* fletcher-4 checksum of everything preceding the
	* checksum.
	*/
	zio_cksum_t drr_checksum;
	} drr_checksum;
	} drr_u;
	} dmu_replay_record_t;

	/* diff record range types */
	typedef enum diff_type {
	DDR_NONE = 0x1,
	DDR_INUSE = 0x2,
	DDR_FREE = 0x4
	} diff_type_t;

	/*
	* The diff reports back ranges of free or in-use objects.
	*/
	typedef struct dmu_diff_record {
	uint64_t ddr_type;
	uint64_t ddr_first;
	uint64_t ddr_last;
	} dmu_diff_record_t;

	typedef struct zinject_record {
	uint64_t zi_objset;
	uint64_t zi_object;
	uint64_t zi_start;
	uint64_t zi_end;
	uint64_t zi_guid;
	uint32_t zi_level;
	uint32_t zi_error;
	uint64_t zi_type;
	uint32_t zi_freq;
	uint32_t zi_failfast;
	char zi_func[MAXNAMELEN];
	uint32_t zi_iotype;
	int32_t zi_duration;
	uint64_t zi_timer;
	uint64_t zi_nlanes;
	uint32_t zi_cmd;
	uint32_t zi_dvas;
	} zinject_record_t;

	#define ZINJECT_NULL 0x1
	#define ZINJECT_FLUSH_ARC 0x2
	#define ZINJECT_UNLOAD_SPA 0x4
	#define ZINJECT_CALC_RANGE 0x8

	#define ZEVENT_NONE 0x0
	#define ZEVENT_NONBLOCK 0x1
	#define ZEVENT_SIZE 1024

	#define ZEVENT_SEEK_START 0
	#define ZEVENT_SEEK_END UINT64_MAX

	/* scaled frequency ranges */
	#define ZI_PERCENTAGE_MIN 4294UL
	#define ZI_PERCENTAGE_MAX UINT32_MAX

	#define ZI_NO_DVA (-1)

	typedef enum zinject_type {
	ZINJECT_UNINITIALIZED,
	ZINJECT_DATA_FAULT,
	ZINJECT_DEVICE_FAULT,
	ZINJECT_LABEL_FAULT,
	ZINJECT_IGNORED_WRITES,
	ZINJECT_PANIC,
	ZINJECT_DELAY_IO,
	ZINJECT_DECRYPT_FAULT,
	} zinject_type_t;

	typedef struct zfs_share {
	uint64_t z_exportdata;
	uint64_t z_sharedata;
	uint64_t z_sharetype; /* 0 = share, 1 = unshare */
	uint64_t z_sharemax; /* max length of share string */
	} zfs_share_t;

	/*
	* ZFS file systems may behave the usual, POSIX-compliant way, where
	* name lookups are case-sensitive. They may also be set up so that
	* all the name lookups are case-insensitive, or so that only some
	* lookups, the ones that set an FIGNORECASE flag, are case-insensitive.
	*/
	typedef enum zfs_case {
	ZFS_CASE_SENSITIVE,
	ZFS_CASE_INSENSITIVE,
	ZFS_CASE_MIXED
	} zfs_case_t;

	/*
	* Note: this struct must have the same layout in 32-bit and 64-bit, so
	* that 32-bit processes (like /sbin/zfs) can pass it to the 64-bit
	* kernel. Therefore, we add padding to it so that no "hidden" padding
	* is automatically added on 64-bit (but not on 32-bit).
	*/
	typedef struct zfs_cmd {
	char zc_name[MAXPATHLEN]; /* name of pool or dataset */
	uint64_t zc_nvlist_src; /* really (char ) /
	uint64_t zc_nvlist_src_size;
	uint64_t zc_nvlist_dst; /* really (char ) /
	uint64_t zc_nvlist_dst_size;
	boolean_t zc_nvlist_dst_filled; /* put an nvlist in dst? */
	int zc_pad2;

	/*
	* The following members are for legacy ioctls which haven't been
	* converted to the new method.
	*/
	uint64_t zc_history; /* really (char ) /
	char zc_value[MAXPATHLEN * 2];
	char zc_string[MAXNAMELEN];
	uint64_t zc_guid;
	uint64_t zc_nvlist_conf; /* really (char ) /
	uint64_t zc_nvlist_conf_size;
	uint64_t zc_cookie;
	uint64_t zc_objset_type;
	uint64_t zc_perm_action;
	uint64_t zc_history_len;
	uint64_t zc_history_offset;
	uint64_t zc_obj;
	uint64_t zc_iflags; /* internal to zfs(7fs) */
	zfs_share_t zc_share;
	dmu_objset_stats_t zc_objset_stats;
	struct drr_begin zc_begin_record;
	zinject_record_t zc_inject_record;
	uint32_t zc_defer_destroy;
	uint32_t zc_flags;
	uint64_t zc_action_handle;
	int zc_cleanup_fd;
	uint8_t zc_simple;
	uint8_t zc_pad[3]; /* alignment */
	uint64_t zc_sendobj;
	uint64_t zc_fromobj;
	uint64_t zc_createtxg;
	zfs_stat_t zc_stat;
	uint64_t zc_zoneid;
	} zfs_cmd_t;

	typedef struct zfs_useracct {
	char zu_domain[256];
	uid_t zu_rid;
	uint32_t zu_pad;
	uint64_t zu_space;
	} zfs_useracct_t;

	#define ZFSDEV_MAX_MINOR (1 << 16)
	#define ZFS_MIN_MINOR (ZFSDEV_MAX_MINOR + 1)

	#define ZPOOL_EXPORT_AFTER_SPLIT 0x1

	#ifdef _KERNEL
	struct objset;
	struct zfsvfs;

	typedef struct zfs_creat {
	nvlist_t *zct_zplprops;
	nvlist_t *zct_props;
	} zfs_creat_t;

	extern int zfs_secpolicy_snapshot_perms(const char , cred_t );
	extern int zfs_secpolicy_rename_perms(const char , const char , cred_t *);
	extern int zfs_secpolicy_destroy_perms(const char , cred_t );
	extern void zfs_unmount_snap(const char *);
	extern void zfs_destroy_unmount_origin(const char *);
	extern int getzfsvfs_impl(struct objset , struct zfsvfs *);
	extern int getzfsvfs(const char , struct zfsvfs *);

	enum zfsdev_state_type {
	ZST_ONEXIT,
	ZST_ZEVENT,
	ZST_ALL,
	};

	/*
	* The zfsdev_state_t structure is managed as a singly-linked list
	* from which items are never deleted. This allows for lock-free
	* reading of the list so long as assignments to the zs_next and
	* reads from zs_minor are performed atomically. Empty items are
	* indicated by storing -1 into zs_minor.
	*/
	typedef struct zfsdev_state {
	struct zfsdev_state zs_next; / next zfsdev_state_t link */
	minor_t zs_minor; /* made up minor number */
	void zs_onexit; / onexit data */
	void zs_zevent; / zevent data */
	} zfsdev_state_t;

	extern void *zfsdev_get_state(minor_t minor, enum zfsdev_state_type which);
	extern int zfsdev_getminor(int fd, minor_t *minorp);
	extern minor_t zfsdev_minor_alloc(void);

	extern uint_t zfs_fsyncer_key;
	extern uint_t zfs_allow_log_key;

	#endif /* _KERNEL */

	#ifdef __cplusplus
	}
	#endif

	#endif /* _SYS_ZFS_IOCTL_H */
	Index: vendor-sys/openzfs/dist/include/sys/zfs_znode.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/sys/zfs_znode.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/sys/zfs_znode.h (revision 366774)
	@@ -1,295 +1,296 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2012, 2018 by Delphix. All rights reserved.
	* Copyright 2016 Nexenta Systems, Inc. All rights reserved.
	*/

	#ifndef _SYS_FS_ZFS_ZNODE_H
	#define _SYS_FS_ZFS_ZNODE_H

	#include <sys/zfs_acl.h>
	#include <sys/zil.h>
	#include <sys/zfs_project.h>

	#ifdef __cplusplus
	extern "C" {
	#endif

	/*
	* Additional file level attributes, that are stored
	* in the upper half of zp_flags
	*/
	#define ZFS_READONLY 0x0000000100000000ull
	#define ZFS_HIDDEN 0x0000000200000000ull
	#define ZFS_SYSTEM 0x0000000400000000ull
	#define ZFS_ARCHIVE 0x0000000800000000ull
	#define ZFS_IMMUTABLE 0x0000001000000000ull
	#define ZFS_NOUNLINK 0x0000002000000000ull
	#define ZFS_APPENDONLY 0x0000004000000000ull
	#define ZFS_NODUMP 0x0000008000000000ull
	#define ZFS_OPAQUE 0x0000010000000000ull
	#define ZFS_AV_QUARANTINED 0x0000020000000000ull
	#define ZFS_AV_MODIFIED 0x0000040000000000ull
	#define ZFS_REPARSE 0x0000080000000000ull
	#define ZFS_OFFLINE 0x0000100000000000ull
	#define ZFS_SPARSE 0x0000200000000000ull

	/*
	* PROJINHERIT attribute is used to indicate that the child object under the
	* directory which has the PROJINHERIT attribute needs to inherit its parent
	* project ID that is used by project quota.
	*/
	#define ZFS_PROJINHERIT 0x0000400000000000ull

	/*
	* PROJID attr is used internally to indicate that the object has project ID.
	*/
	#define ZFS_PROJID 0x0000800000000000ull

	#define ZFS_ATTR_SET(zp, attr, value, pflags, tx) \
	{ \
	if (value) \
	pflags \|= attr; \
	else \
	pflags &= ~attr; \
	VERIFY(0 == sa_update(zp->z_sa_hdl, SA_ZPL_FLAGS(ZTOZSB(zp)), \
	&pflags, sizeof (pflags), tx)); \
	}

	/*
	* Define special zfs pflags
	*/
	#define ZFS_XATTR 0x1 /* is an extended attribute */
	#define ZFS_INHERIT_ACE 0x2 /* ace has inheritable ACEs */
	#define ZFS_ACL_TRIVIAL 0x4 /* files ACL is trivial */
	#define ZFS_ACL_OBJ_ACE 0x8 /* ACL has CMPLX Object ACE */
	#define ZFS_ACL_PROTECTED 0x10 /* ACL protected */
	#define ZFS_ACL_DEFAULTED 0x20 /* ACL should be defaulted */
	#define ZFS_ACL_AUTO_INHERIT 0x40 /* ACL should be inherited */
	#define ZFS_BONUS_SCANSTAMP 0x80 /* Scanstamp in bonus area */
	#define ZFS_NO_EXECS_DENIED 0x100 /* exec was given to everyone */

	#define SA_ZPL_ATIME(z) z->z_attr_table[ZPL_ATIME]
	#define SA_ZPL_MTIME(z) z->z_attr_table[ZPL_MTIME]
	#define SA_ZPL_CTIME(z) z->z_attr_table[ZPL_CTIME]
	#define SA_ZPL_CRTIME(z) z->z_attr_table[ZPL_CRTIME]
	#define SA_ZPL_GEN(z) z->z_attr_table[ZPL_GEN]
	#define SA_ZPL_DACL_ACES(z) z->z_attr_table[ZPL_DACL_ACES]
	#define SA_ZPL_XATTR(z) z->z_attr_table[ZPL_XATTR]
	#define SA_ZPL_SYMLINK(z) z->z_attr_table[ZPL_SYMLINK]
	#define SA_ZPL_RDEV(z) z->z_attr_table[ZPL_RDEV]
	#define SA_ZPL_SCANSTAMP(z) z->z_attr_table[ZPL_SCANSTAMP]
	#define SA_ZPL_UID(z) z->z_attr_table[ZPL_UID]
	#define SA_ZPL_GID(z) z->z_attr_table[ZPL_GID]
	#define SA_ZPL_PARENT(z) z->z_attr_table[ZPL_PARENT]
	#define SA_ZPL_LINKS(z) z->z_attr_table[ZPL_LINKS]
	#define SA_ZPL_MODE(z) z->z_attr_table[ZPL_MODE]
	#define SA_ZPL_DACL_COUNT(z) z->z_attr_table[ZPL_DACL_COUNT]
	#define SA_ZPL_FLAGS(z) z->z_attr_table[ZPL_FLAGS]
	#define SA_ZPL_SIZE(z) z->z_attr_table[ZPL_SIZE]
	#define SA_ZPL_ZNODE_ACL(z) z->z_attr_table[ZPL_ZNODE_ACL]
	#define SA_ZPL_DXATTR(z) z->z_attr_table[ZPL_DXATTR]
	#define SA_ZPL_PAD(z) z->z_attr_table[ZPL_PAD]
	#define SA_ZPL_PROJID(z) z->z_attr_table[ZPL_PROJID]

	/*
	* Is ID ephemeral?
	*/
	#define IS_EPHEMERAL(x) (x > MAXUID)

	/*
	* Should we use FUIDs?
	*/
	#define USE_FUIDS(version, os) (version >= ZPL_VERSION_FUID && \
	spa_version(dmu_objset_spa(os)) >= SPA_VERSION_FUID)
	#define USE_SA(version, os) (version >= ZPL_VERSION_SA && \
	spa_version(dmu_objset_spa(os)) >= SPA_VERSION_SA)

	#define MASTER_NODE_OBJ 1

	/*
	* Special attributes for master node.
	* "userquota@", "groupquota@" and "projectquota@" are also valid (from
	* zfs_userquota_prop_prefixes[]).
	*/
	#define ZFS_FSID "FSID"
	#define ZFS_UNLINKED_SET "DELETE_QUEUE"
	#define ZFS_ROOT_OBJ "ROOT"
	#define ZPL_VERSION_STR "VERSION"
	#define ZFS_FUID_TABLES "FUID"
	#define ZFS_SHARES_DIR "SHARES"
	#define ZFS_SA_ATTRS "SA_ATTRS"

	/*
	* Convert mode bits (zp_mode) to BSD-style DT_* values for storing in
	* the directory entries. On Linux systems this value is already
	* defined correctly as part of the /usr/include/dirent.h header file.
	*/
	#ifndef IFTODT
	#define IFTODT(mode) (((mode) & S_IFMT) >> 12)
	#endif

	/*
	* The directory entry has the type (currently unused on Solaris) in the
	* top 4 bits, and the object number in the low 48 bits. The "middle"
	* 12 bits are unused.
	*/
	#define ZFS_DIRENT_TYPE(de) BF64_GET(de, 60, 4)
	#define ZFS_DIRENT_OBJ(de) BF64_GET(de, 0, 48)

	extern int zfs_obj_to_path(objset_t osp, uint64_t obj, char buf, int len);

	#ifdef _KERNEL
	#include <sys/zfs_znode_impl.h>

	/*
	* Directory entry locks control access to directory entries.
	* They are used to protect creates, deletes, and renames.
	* Each directory znode has a mutex and a list of locked names.
	*/
	typedef struct zfs_dirlock {
	char dl_name; / directory entry being locked */
	uint32_t dl_sharecnt; /* 0 if exclusive, > 0 if shared */
	uint8_t dl_namelock; /* 1 if z_name_lock is NOT held */
	uint16_t dl_namesize; /* set if dl_name was allocated */
	kcondvar_t dl_cv; /* wait for entry to be unlocked */
	struct znode dl_dzp; / directory znode */
	struct zfs_dirlock dl_next; / next in z_dirlocks list */
	} zfs_dirlock_t;

	typedef struct znode {
	uint64_t z_id; /* object ID for this znode */
	kmutex_t z_lock; /* znode modification lock */
	krwlock_t z_parent_lock; /* parent lock for directories */
	krwlock_t z_name_lock; /* "master" lock for dirent locks */
	zfs_dirlock_t z_dirlocks; / directory entry lock list */
	zfs_rangelock_t z_rangelock; /* file range locks */
	boolean_t z_unlinked; /* file has been unlinked */
	boolean_t z_atime_dirty; /* atime needs to be synced */
	boolean_t z_zn_prefetch; /* Prefetch znodes? */
	boolean_t z_moved; /* Has this znode been moved? */
	boolean_t z_is_sa; /* are we native sa? */
	boolean_t z_is_mapped; /* are we mmap'ed */
	boolean_t z_is_ctldir; /* are we .zfs entry */
	boolean_t z_is_stale; /* are we stale due to rollback? */
	boolean_t z_suspended; /* extra ref from a suspend? */
	uint_t z_blksz; /* block size in bytes */
	uint_t z_seq; /* modification sequence number */
	uint64_t z_mapcnt; /* number of pages mapped to file */
	uint64_t z_dnodesize; /* dnode size */
	uint64_t z_size; /* file size (cached) */
	uint64_t z_pflags; /* pflags (cached) */
	uint32_t z_sync_cnt; /* synchronous open count */
	mode_t z_mode; /* mode (cached) */
	kmutex_t z_acl_lock; /* acl data lock */
	zfs_acl_t z_acl_cached; / cached acl */
	krwlock_t z_xattr_lock; /* xattr data lock */
	nvlist_t z_xattr_cached; / cached xattrs */
	uint64_t z_xattr_parent; /* parent obj for this xattr */
	uint64_t z_projid; /* project ID */
	list_node_t z_link_node; /* all znodes in fs link */
	sa_handle_t z_sa_hdl; / handle to sa data */

	/*
	* Platform specific field, defined by each platform and only
	* accessible from platform specific code.
	*/
	ZNODE_OS_FIELDS;
	} znode_t;

	typedef struct znode_hold {
	uint64_t zh_obj; /* object id */
	kmutex_t zh_lock; /* lock serializing object access */
	avl_node_t zh_node; /* avl tree linkage */
	zfs_refcount_t zh_refcount; /* active consumer reference count */
	} znode_hold_t;

	static inline uint64_t
	zfs_inherit_projid(znode_t *dzp)
	{
	return ((dzp->z_pflags & ZFS_PROJINHERIT) ? dzp->z_projid :
	ZFS_DEFAULT_PROJID);
	}

	/*
	* Timestamp defines
	*/
	#define ACCESSED (ATTR_ATIME)
	#define STATE_CHANGED (ATTR_CTIME)
	#define CONTENT_MODIFIED (ATTR_MTIME \| ATTR_CTIME)

	extern int zfs_init_fs(zfsvfs_t , znode_t *);
	extern void zfs_set_dataprop(objset_t *);
	extern void zfs_create_fs(objset_t os, cred_t cr, nvlist_t *,
	dmu_tx_t *tx);
	extern void zfs_tstamp_update_setup(znode_t *, uint_t, uint64_t [2],
	uint64_t [2]);
	extern void zfs_grow_blocksize(znode_t , uint64_t, dmu_tx_t );
	extern int zfs_freesp(znode_t *, uint64_t, uint64_t, int, boolean_t);
	extern void zfs_znode_init(void);
	extern void zfs_znode_fini(void);
	extern int zfs_znode_hold_compare(const void , const void );
	extern int zfs_zget(zfsvfs_t , uint64_t, znode_t *);
	extern int zfs_rezget(znode_t *);
	extern void zfs_zinactive(znode_t *);
	extern void zfs_znode_delete(znode_t , dmu_tx_t );
	extern void zfs_remove_op_tables(void);
	extern int zfs_create_op_tables(void);
	extern dev_t zfs_cmpldev(uint64_t);
	extern int zfs_get_zplprop(objset_t os, zfs_prop_t prop, uint64_t value);
	extern int zfs_get_stats(objset_t os, nvlist_t nv);
	extern boolean_t zfs_get_vfs_flag_unmounted(objset_t *os);
	extern void zfs_znode_dmu_fini(znode_t *);

	extern void zfs_log_create(zilog_t zilog, dmu_tx_t tx, uint64_t txtype,
	- znode_t dzp, znode_t zp, char name, vsecattr_t , zfs_fuid_info_t *,
	- vattr_t *vap);
	+ znode_t dzp, znode_t zp, const char name, vsecattr_t ,
	+ zfs_fuid_info_t , vattr_t vap);
	extern int zfs_log_create_txtype(zil_create_t, vsecattr_t *vsecp,
	vattr_t *vap);
	extern void zfs_log_remove(zilog_t zilog, dmu_tx_t tx, uint64_t txtype,
	- znode_t dzp, char name, uint64_t foid, boolean_t unlinked);
	+ znode_t dzp, const char name, uint64_t foid, boolean_t unlinked);
	#define ZFS_NO_OBJECT 0 /* no object id */
	extern void zfs_log_link(zilog_t zilog, dmu_tx_t tx, uint64_t txtype,
	- znode_t dzp, znode_t zp, char *name);
	+ znode_t dzp, znode_t zp, const char *name);
	extern void zfs_log_symlink(zilog_t zilog, dmu_tx_t tx, uint64_t txtype,
	- znode_t dzp, znode_t zp, char name, char link);
	+ znode_t dzp, znode_t zp, const char name, const char link);
	extern void zfs_log_rename(zilog_t zilog, dmu_tx_t tx, uint64_t txtype,
	- znode_t sdzp, char sname, znode_t tdzp, char dname, znode_t *szp);
	+ znode_t sdzp, const char sname, znode_t tdzp, const char dname,
	+ znode_t *szp);
	extern void zfs_log_write(zilog_t zilog, dmu_tx_t tx, int txtype,
	znode_t *zp, offset_t off, ssize_t len, int ioflag,
	zil_callback_t callback, void *callback_data);
	extern void zfs_log_truncate(zilog_t zilog, dmu_tx_t tx, int txtype,
	znode_t *zp, uint64_t off, uint64_t len);
	extern void zfs_log_setattr(zilog_t zilog, dmu_tx_t tx, int txtype,
	znode_t zp, vattr_t vap, uint_t mask_applied, zfs_fuid_info_t *fuidp);
	extern void zfs_log_acl(zilog_t zilog, dmu_tx_t tx, znode_t *zp,
	vsecattr_t vsecp, zfs_fuid_info_t fuidp);
	extern void zfs_xvattr_set(znode_t zp, xvattr_t xvap, dmu_tx_t *tx);
	extern void zfs_upgrade(zfsvfs_t zfsvfs, dmu_tx_t tx);

	#endif
	#ifdef __cplusplus
	}
	#endif

	#endif /* _SYS_FS_ZFS_ZNODE_H */
	Index: vendor-sys/openzfs/dist/include/sys/zil.h
	===================================================================
	--- vendor-sys/openzfs/dist/include/sys/zil.h (revision 366773)
	+++ vendor-sys/openzfs/dist/include/sys/zil.h (revision 366774)
	@@ -1,529 +1,529 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2012, 2018 by Delphix. All rights reserved.
	*/

	/* Portions Copyright 2010 Robert Milkowski */

	#ifndef _SYS_ZIL_H
	#define _SYS_ZIL_H

	#include <sys/types.h>
	#include <sys/spa.h>
	#include <sys/zio.h>
	#include <sys/dmu.h>
	#include <sys/zio_crypt.h>

	#ifdef __cplusplus
	extern "C" {
	#endif

	struct dsl_pool;
	struct dsl_dataset;
	struct lwb;

	/*
	* Intent log format:
	*
	* Each objset has its own intent log. The log header (zil_header_t)
	* for objset N's intent log is kept in the Nth object of the SPA's
	* intent_log objset. The log header points to a chain of log blocks,
	* each of which contains log records (i.e., transactions) followed by
	* a log block trailer (zil_trailer_t). The format of a log record
	* depends on the record (or transaction) type, but all records begin
	* with a common structure that defines the type, length, and txg.
	*/

	/*
	* Intent log header - this on disk structure holds fields to manage
	* the log. All fields are 64 bit to easily handle cross architectures.
	*/
	typedef struct zil_header {
	uint64_t zh_claim_txg; /* txg in which log blocks were claimed */
	uint64_t zh_replay_seq; /* highest replayed sequence number */
	blkptr_t zh_log; /* log chain */
	uint64_t zh_claim_blk_seq; /* highest claimed block sequence number */
	uint64_t zh_flags; /* header flags */
	uint64_t zh_claim_lr_seq; /* highest claimed lr sequence number */
	uint64_t zh_pad[3];
	} zil_header_t;

	/*
	* zh_flags bit settings
	*/
	#define ZIL_REPLAY_NEEDED 0x1 /* replay needed - internal only */
	#define ZIL_CLAIM_LR_SEQ_VALID 0x2 /* zh_claim_lr_seq field is valid */

	/*
	* Log block chaining.
	*
	* Log blocks are chained together. Originally they were chained at the
	* end of the block. For performance reasons the chain was moved to the
	* beginning of the block which allows writes for only the data being used.
	* The older position is supported for backwards compatibility.
	*
	* The zio_eck_t contains a zec_cksum which for the intent log is
	* the sequence number of this log block. A seq of 0 is invalid.
	* The zec_cksum is checked by the SPA against the sequence
	* number passed in the blk_cksum field of the blkptr_t
	*/
	typedef struct zil_chain {
	uint64_t zc_pad;
	blkptr_t zc_next_blk; /* next block in chain */
	uint64_t zc_nused; /* bytes in log block used */
	zio_eck_t zc_eck; /* block trailer */
	} zil_chain_t;

	#define ZIL_MIN_BLKSZ 4096ULL

	/*
	* ziltest is by and large an ugly hack, but very useful in
	* checking replay without tedious work.
	* When running ziltest we want to keep all itx's and so maintain
	* a single list in the zl_itxg[] that uses a high txg: ZILTEST_TXG
	* We subtract TXG_CONCURRENT_STATES to allow for common code.
	*/
	#define ZILTEST_TXG (UINT64_MAX - TXG_CONCURRENT_STATES)

	/*
	* The words of a log block checksum.
	*/
	#define ZIL_ZC_GUID_0 0
	#define ZIL_ZC_GUID_1 1
	#define ZIL_ZC_OBJSET 2
	#define ZIL_ZC_SEQ 3

	typedef enum zil_create {
	Z_FILE,
	Z_DIR,
	Z_XATTRDIR,
	} zil_create_t;

	/*
	* size of xvattr log section.
	* its composed of lr_attr_t + xvattr bitmap + 2 64 bit timestamps
	* for create time and a single 64 bit integer for all of the attributes,
	* and 4 64 bit integers (32 bytes) for the scanstamp.
	*
	*/

	#define ZIL_XVAT_SIZE(mapsize) \
	sizeof (lr_attr_t) + (sizeof (uint32_t) * (mapsize - 1)) + \
	(sizeof (uint64_t) * 7)

	/*
	* Size of ACL in log. The ACE data is padded out to properly align
	* on 8 byte boundary.
	*/

	#define ZIL_ACE_LENGTH(x) (roundup(x, sizeof (uint64_t)))

	/*
	* Intent log transaction types and record structures
	*/
	#define TX_COMMIT 0 /* Commit marker (no on-disk state) */
	#define TX_CREATE 1 /* Create file */
	#define TX_MKDIR 2 /* Make directory */
	#define TX_MKXATTR 3 /* Make XATTR directory */
	#define TX_SYMLINK 4 /* Create symbolic link to a file */
	#define TX_REMOVE 5 /* Remove file */
	#define TX_RMDIR 6 /* Remove directory */
	#define TX_LINK 7 /* Create hard link to a file */
	#define TX_RENAME 8 /* Rename a file */
	#define TX_WRITE 9 /* File write */
	#define TX_TRUNCATE 10 /* Truncate a file */
	#define TX_SETATTR 11 /* Set file attributes */
	#define TX_ACL_V0 12 /* Set old formatted ACL */
	#define TX_ACL 13 /* Set ACL */
	#define TX_CREATE_ACL 14 /* create with ACL */
	#define TX_CREATE_ATTR 15 /* create + attrs */
	#define TX_CREATE_ACL_ATTR 16 /* create with ACL + attrs */
	#define TX_MKDIR_ACL 17 /* mkdir with ACL */
	#define TX_MKDIR_ATTR 18 /* mkdir with attr */
	#define TX_MKDIR_ACL_ATTR 19 /* mkdir with ACL + attrs */
	#define TX_WRITE2 20 /* dmu_sync EALREADY write */
	#define TX_MAX_TYPE 21 /* Max transaction type */

	/*
	* The transactions for mkdir, symlink, remove, rmdir, link, and rename
	* may have the following bit set, indicating the original request
	* specified case-insensitive handling of names.
	*/
	#define TX_CI ((uint64_t)0x1 << 63) /* case-insensitive behavior requested */

	/*
	* Transactions for write, truncate, setattr, acl_v0, and acl can be logged
	* out of order. For convenience in the code, all such records must have
	* lr_foid at the same offset.
	*/
	#define TX_OOO(txtype) \
	((txtype) == TX_WRITE \|\| \
	(txtype) == TX_TRUNCATE \|\| \
	(txtype) == TX_SETATTR \|\| \
	(txtype) == TX_ACL_V0 \|\| \
	(txtype) == TX_ACL \|\| \
	(txtype) == TX_WRITE2)

	/*
	* The number of dnode slots consumed by the object is stored in the 8
	* unused upper bits of the object ID. We subtract 1 from the value
	* stored on disk for compatibility with implementations that don't
	* support large dnodes. The slot count for a single-slot dnode will
	* contain 0 for those bits to preserve the log record format for
	* "small" dnodes.
	*/
	#define LR_FOID_GET_SLOTS(oid) (BF64_GET((oid), 56, 8) + 1)
	#define LR_FOID_SET_SLOTS(oid, x) BF64_SET((oid), 56, 8, (x) - 1)
	#define LR_FOID_GET_OBJ(oid) BF64_GET((oid), 0, DN_MAX_OBJECT_SHIFT)
	#define LR_FOID_SET_OBJ(oid, x) BF64_SET((oid), 0, DN_MAX_OBJECT_SHIFT, (x))

	/*
	* Format of log records.
	* The fields are carefully defined to allow them to be aligned
	* and sized the same on sparc & intel architectures.
	* Each log record has a common structure at the beginning.
	*
	* The log record on disk (lrc_seq) holds the sequence number of all log
	* records which is used to ensure we don't replay the same record.
	*/
	typedef struct { /* common log record header */
	uint64_t lrc_txtype; /* intent log transaction type */
	uint64_t lrc_reclen; /* transaction record length */
	uint64_t lrc_txg; /* dmu transaction group number */
	uint64_t lrc_seq; /* see comment above */
	} lr_t;

	/*
	* Common start of all out-of-order record types (TX_OOO() above).
	*/
	typedef struct {
	lr_t lr_common; /* common portion of log record */
	uint64_t lr_foid; /* object id */
	} lr_ooo_t;

	/*
	* Handle option extended vattr attributes.
	*
	* Whenever new attributes are added the version number
	* will need to be updated as will code in
	* zfs_log.c and zfs_replay.c
	*/
	typedef struct {
	uint32_t lr_attr_masksize; /* number of elements in array */
	uint32_t lr_attr_bitmap; /* First entry of array */
	/* remainder of array and any additional fields */
	} lr_attr_t;

	/*
	* log record for creates without optional ACL.
	* This log record does support optional xvattr_t attributes.
	*/
	typedef struct {
	lr_t lr_common; /* common portion of log record */
	uint64_t lr_doid; /* object id of directory */
	uint64_t lr_foid; /* object id of created file object */
	uint64_t lr_mode; /* mode of object */
	uint64_t lr_uid; /* uid of object */
	uint64_t lr_gid; /* gid of object */
	uint64_t lr_gen; /* generation (txg of creation) */
	uint64_t lr_crtime[2]; /* creation time */
	uint64_t lr_rdev; /* rdev of object to create */
	/* name of object to create follows this */
	/* for symlinks, link content follows name */
	/* for creates with xvattr data, the name follows the xvattr info */
	} lr_create_t;

	/*
	* FUID ACL record will be an array of ACEs from the original ACL.
	* If this array includes ephemeral IDs, the record will also include
	* an array of log-specific FUIDs to replace the ephemeral IDs.
	* Only one copy of each unique domain will be present, so the log-specific
	* FUIDs will use an index into a compressed domain table. On replay this
	* information will be used to construct real FUIDs (and bypass idmap,
	* since it may not be available).
	*/

	/*
	* Log record for creates with optional ACL
	* This log record is also used for recording any FUID
	* information needed for replaying the create. If the
	* file doesn't have any actual ACEs then the lr_aclcnt
	* would be zero.
	*
	* After lr_acl_flags, there are a lr_acl_bytes number of variable sized ace's.
	* If create is also setting xvattr's, then acl data follows xvattr.
	* If ACE FUIDs are needed then they will follow the xvattr_t. Following
	* the FUIDs will be the domain table information. The FUIDs for the owner
	* and group will be in lr_create. Name follows ACL data.
	*/
	typedef struct {
	lr_create_t lr_create; /* common create portion */
	uint64_t lr_aclcnt; /* number of ACEs in ACL */
	uint64_t lr_domcnt; /* number of unique domains */
	uint64_t lr_fuidcnt; /* number of real fuids */
	uint64_t lr_acl_bytes; /* number of bytes in ACL */
	uint64_t lr_acl_flags; /* ACL flags */
	} lr_acl_create_t;

	typedef struct {
	lr_t lr_common; /* common portion of log record */
	uint64_t lr_doid; /* obj id of directory */
	/* name of object to remove follows this */
	} lr_remove_t;

	typedef struct {
	lr_t lr_common; /* common portion of log record */
	uint64_t lr_doid; /* obj id of directory */
	uint64_t lr_link_obj; /* obj id of link */
	/* name of object to link follows this */
	} lr_link_t;

	typedef struct {
	lr_t lr_common; /* common portion of log record */
	uint64_t lr_sdoid; /* obj id of source directory */
	uint64_t lr_tdoid; /* obj id of target directory */
	/* 2 strings: names of source and destination follow this */
	} lr_rename_t;

	typedef struct {
	lr_t lr_common; /* common portion of log record */
	uint64_t lr_foid; /* file object to write */
	uint64_t lr_offset; /* offset to write to */
	uint64_t lr_length; /* user data length to write */
	uint64_t lr_blkoff; /* no longer used */
	blkptr_t lr_blkptr; /* spa block pointer for replay */
	/* write data will follow for small writes */
	} lr_write_t;

	typedef struct {
	lr_t lr_common; /* common portion of log record */
	uint64_t lr_foid; /* object id of file to truncate */
	uint64_t lr_offset; /* offset to truncate from */
	uint64_t lr_length; /* length to truncate */
	} lr_truncate_t;

	typedef struct {
	lr_t lr_common; /* common portion of log record */
	uint64_t lr_foid; /* file object to change attributes */
	uint64_t lr_mask; /* mask of attributes to set */
	uint64_t lr_mode; /* mode to set */
	uint64_t lr_uid; /* uid to set */
	uint64_t lr_gid; /* gid to set */
	uint64_t lr_size; /* size to set */
	uint64_t lr_atime[2]; /* access time */
	uint64_t lr_mtime[2]; /* modification time */
	/* optional attribute lr_attr_t may be here */
	} lr_setattr_t;

	typedef struct {
	lr_t lr_common; /* common portion of log record */
	uint64_t lr_foid; /* obj id of file */
	uint64_t lr_aclcnt; /* number of acl entries */
	/* lr_aclcnt number of ace_t entries follow this */
	} lr_acl_v0_t;

	typedef struct {
	lr_t lr_common; /* common portion of log record */
	uint64_t lr_foid; /* obj id of file */
	uint64_t lr_aclcnt; /* number of ACEs in ACL */
	uint64_t lr_domcnt; /* number of unique domains */
	uint64_t lr_fuidcnt; /* number of real fuids */
	uint64_t lr_acl_bytes; /* number of bytes in ACL */
	uint64_t lr_acl_flags; /* ACL flags */
	/* lr_acl_bytes number of variable sized ace's follows */
	} lr_acl_t;

	/*
	* ZIL structure definitions, interface function prototype and globals.
	*/

	/*
	* Writes are handled in three different ways:
	*
	* WR_INDIRECT:
	* In this mode, if we need to commit the write later, then the block
	* is immediately written into the file system (using dmu_sync),
	* and a pointer to the block is put into the log record.
	* When the txg commits the block is linked in.
	* This saves additionally writing the data into the log record.
	* There are a few requirements for this to occur:
	* - write is greater than zfs/zvol_immediate_write_sz
	* - not using slogs (as slogs are assumed to always be faster
	* than writing into the main pool)
	* - the write occupies only one block
	* WR_COPIED:
	* If we know we'll immediately be committing the
	* transaction (O_SYNC or O_DSYNC), then we allocate a larger
	* log record here for the data and copy the data in.
	* WR_NEED_COPY:
	* Otherwise we don't allocate a buffer, and if we need to
	* flush the write later then a buffer is allocated and
	* we retrieve the data using the dmu.
	*/
	typedef enum {
	WR_INDIRECT, /* indirect - a large write (dmu_sync() data */
	/* and put blkptr in log, rather than actual data) */
	WR_COPIED, /* immediate - data is copied into lr_write_t */
	WR_NEED_COPY, /* immediate - data needs to be copied if pushed */
	WR_NUM_STATES /* number of states */
	} itx_wr_state_t;

	typedef void (zil_callback_t)(void data);

	typedef struct itx {
	list_node_t itx_node; /* linkage on zl_itx_list */
	void itx_private; / type-specific opaque data */
	itx_wr_state_t itx_wr_state; /* write state */
	uint8_t itx_sync; /* synchronous transaction */
	zil_callback_t itx_callback; /* Called when the itx is persistent */
	void itx_callback_data; / User data for the callback */
	size_t itx_size; /* allocated itx structure size */
	uint64_t itx_oid; /* object id */
	lr_t itx_lr; /* common part of log record */
	/* followed by type-specific part of lr_xx_t and its immediate data */
	} itx_t;

	/*
	* Used for zil kstat.
	*/
	typedef struct zil_stats {
	/*
	* Number of times a ZIL commit (e.g. fsync) has been requested.
	*/
	kstat_named_t zil_commit_count;

	/*
	* Number of times the ZIL has been flushed to stable storage.
	* This is less than zil_commit_count when commits are "merged"
	* (see the documentation above zil_commit()).
	*/
	kstat_named_t zil_commit_writer_count;

	/*
	* Number of transactions (reads, writes, renames, etc.)
	* that have been committed.
	*/
	kstat_named_t zil_itx_count;

	/*
	* See the documentation for itx_wr_state_t above.
	* Note that "bytes" accumulates the length of the transactions
	* (i.e. data), not the actual log record sizes.
	*/
	kstat_named_t zil_itx_indirect_count;
	kstat_named_t zil_itx_indirect_bytes;
	kstat_named_t zil_itx_copied_count;
	kstat_named_t zil_itx_copied_bytes;
	kstat_named_t zil_itx_needcopy_count;
	kstat_named_t zil_itx_needcopy_bytes;

	/*
	* Transactions which have been allocated to the "normal"
	* (i.e. not slog) storage pool. Note that "bytes" accumulate
	* the actual log record sizes - which do not include the actual
	* data in case of indirect writes.
	*/
	kstat_named_t zil_itx_metaslab_normal_count;
	kstat_named_t zil_itx_metaslab_normal_bytes;

	/*
	* Transactions which have been allocated to the "slog" storage pool.
	* If there are no separate log devices, this is the same as the
	* "normal" pool.
	*/
	kstat_named_t zil_itx_metaslab_slog_count;
	kstat_named_t zil_itx_metaslab_slog_bytes;
	} zil_stats_t;

	extern zil_stats_t zil_stats;

	#define ZIL_STAT_INCR(stat, val) \
	atomic_add_64(&zil_stats.stat.value.ui64, (val));
	#define ZIL_STAT_BUMP(stat) \
	ZIL_STAT_INCR(stat, 1);

	-typedef int zil_parse_blk_func_t(zilog_t zilog, blkptr_t bp, void *arg,
	+typedef int zil_parse_blk_func_t(zilog_t zilog, const blkptr_t bp, void *arg,
	uint64_t txg);
	-typedef int zil_parse_lr_func_t(zilog_t zilog, lr_t lr, void *arg,
	+typedef int zil_parse_lr_func_t(zilog_t zilog, const lr_t lr, void *arg,
	uint64_t txg);
	typedef int zil_replay_func_t(void arg1, void arg2, boolean_t byteswap);
	typedef int zil_get_data_t(void arg, lr_write_t lr, char *dbuf,
	struct lwb lwb, zio_t zio);

	extern int zil_parse(zilog_t zilog, zil_parse_blk_func_t parse_blk_func,
	zil_parse_lr_func_t parse_lr_func, void arg, uint64_t txg,
	boolean_t decrypt);

	extern void zil_init(void);
	extern void zil_fini(void);

	extern zilog_t zil_alloc(objset_t os, zil_header_t *zh_phys);
	extern void zil_free(zilog_t *zilog);

	extern zilog_t zil_open(objset_t os, zil_get_data_t *get_data);
	extern void zil_close(zilog_t *zilog);

	extern void zil_replay(objset_t os, void arg,
	zil_replay_func_t *replay_func[TX_MAX_TYPE]);
	extern boolean_t zil_replaying(zilog_t zilog, dmu_tx_t tx);
	extern void zil_destroy(zilog_t *zilog, boolean_t keep_first);
	extern void zil_destroy_sync(zilog_t zilog, dmu_tx_t tx);

	extern itx_t *zil_itx_create(uint64_t txtype, size_t lrsize);
	extern void zil_itx_destroy(itx_t *itx);
	extern void zil_itx_assign(zilog_t zilog, itx_t itx, dmu_tx_t *tx);

	extern void zil_async_to_sync(zilog_t *zilog, uint64_t oid);
	extern void zil_commit(zilog_t *zilog, uint64_t oid);
	extern void zil_commit_impl(zilog_t *zilog, uint64_t oid);
	extern void zil_remove_async(zilog_t *zilog, uint64_t oid);

	extern int zil_reset(const char osname, void txarg);
	extern int zil_claim(struct dsl_pool *dp,
	struct dsl_dataset ds, void txarg);
	extern int zil_check_log_chain(struct dsl_pool *dp,
	struct dsl_dataset ds, void tx);
	extern void zil_sync(zilog_t zilog, dmu_tx_t tx);
	extern void zil_clean(zilog_t *zilog, uint64_t synced_txg);

	extern int zil_suspend(const char osname, void *cookiep);
	extern void zil_resume(void *cookie);

	extern void zil_lwb_add_block(struct lwb lwb, const blkptr_t bp);
	extern void zil_lwb_add_txg(struct lwb *lwb, uint64_t txg);
	extern int zil_bp_tree_add(zilog_t zilog, const blkptr_t bp);

	extern void zil_set_sync(zilog_t *zilog, uint64_t syncval);

	extern void zil_set_logbias(zilog_t *zilog, uint64_t slogval);

	extern uint64_t zil_max_copied_data(zilog_t *zilog);
	extern uint64_t zil_max_log_data(zilog_t *zilog);

	extern int zil_replay_disable;

	#ifdef __cplusplus
	}
	#endif

	#endif /* _SYS_ZIL_H */
	Index: vendor-sys/openzfs/dist/lib/libspl/include/os/freebsd/sys/param.h
	===================================================================
	--- vendor-sys/openzfs/dist/lib/libspl/include/os/freebsd/sys/param.h (revision 366773)
	+++ vendor-sys/openzfs/dist/lib/libspl/include/os/freebsd/sys/param.h (revision 366774)
	@@ -1,66 +1,62 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License, Version 1.0 only
	* (the "License"). You may not use this file except in compliance
	* with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright 2006 Sun Microsystems, Inc. All rights reserved.
	* Use is subject to license terms.
	*/

	#ifndef _LIBSPL_SYS_PARAM_H
	#define _LIBSPL_SYS_PARAM_H

	#include_next <sys/param.h>
	#include <unistd.h>

	/*
	* File system parameters and macros.
	*
	* The file system is made out of blocks of at most MAXBSIZE units,
	* with smaller units (fragments) only in the last direct block.
	* MAXBSIZE primarily determines the size of buffers in the buffer
	* pool. It may be made larger without any effect on existing
	* file systems; however making it smaller may make some file
	* systems unmountable.
	*
	* Note that the blocked devices are assumed to have DEV_BSIZE
	* "sectors" and that fragments must be some multiple of this size.
	*/
	#define MAXNAMELEN 256

	-#ifndef IN_BASE
	-#define UID_NOBODY 60001 /* user ID no body */
	-#define GID_NOBODY UID_NOBODY
	-#endif
	#define UID_NOACCESS 60002 /* user ID no access */

	#define MAXUID UINT32_MAX /* max user id */
	#define MAXPROJID MAXUID /* max project id */

	#ifdef PAGESIZE
	#undef PAGESIZE
	#endif /* PAGESIZE */

	extern size_t spl_pagesize(void);
	#define PAGESIZE (spl_pagesize())

	extern int execvpe(const char name, char const argv[], char * const envp[]);

	#endif
	Index: vendor-sys/openzfs/dist/lib/libspl/include/sys/acl.h
	===================================================================
	--- vendor-sys/openzfs/dist/lib/libspl/include/sys/acl.h (revision 366773)
	+++ vendor-sys/openzfs/dist/lib/libspl/include/sys/acl.h (revision 366774)
	@@ -1,287 +1,303 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	- * Copyright 2007 Sun Microsystems, Inc. All rights reserved.
	+ * Copyright 2009 Sun Microsystems, Inc. All rights reserved.
	* Use is subject to license terms.
	+ *
	+ * Copyright 2014 Garrett D'Amore <garrett@damore.org>
	+ * Copyright 2014 Nexenta Systems, Inc. All rights reserved.
	+ * Copyright 2017 RackTop Systems.
	*/

	#ifndef _SYS_ACL_H
	#define _SYS_ACL_H

	#include <sys/types.h>
	#include <sys/acl_impl.h>

	#ifdef __cplusplus
	extern "C" {
	#endif

	#define MAX_ACL_ENTRIES (1024) /* max entries of each type */
	typedef struct acl {
	int a_type; /* the type of ACL entry */
	uid_t a_id; /* the entry in -uid or gid */
	o_mode_t a_perm; /* the permission field */
	} aclent_t;

	typedef struct ace {
	uid_t a_who; /* uid or gid */
	uint32_t a_access_mask; /* read,write,... */
	uint16_t a_flags; /* see below */
	uint16_t a_type; /* allow or deny */
	} ace_t;

	typedef struct acl_info acl_t;

	/*
	* The following are Defined types for an aclent_t.
	*/
	#define USER_OBJ (0x01) /* object owner */
	#define USER (0x02) /* additional users */
	#define GROUP_OBJ (0x04) /* owning group of the object */
	#define GROUP (0x08) /* additional groups */
	#define CLASS_OBJ (0x10) /* file group class and mask entry */
	#define OTHER_OBJ (0x20) /* other entry for the object */
	#define ACL_DEFAULT (0x1000) /* default flag */
	/* default object owner */
	#define DEF_USER_OBJ (ACL_DEFAULT \| USER_OBJ)
	/* default additional users */
	#define DEF_USER (ACL_DEFAULT \| USER)
	/* default owning group */
	#define DEF_GROUP_OBJ (ACL_DEFAULT \| GROUP_OBJ)
	/* default additional groups */
	#define DEF_GROUP (ACL_DEFAULT \| GROUP)
	/* default mask entry */
	#define DEF_CLASS_OBJ (ACL_DEFAULT \| CLASS_OBJ)
	/* default other entry */
	#define DEF_OTHER_OBJ (ACL_DEFAULT \| OTHER_OBJ)

	/*
	* The following are defined for ace_t.
	*/
	-#define ACE_READ_DATA 0x00000001
	-#define ACE_LIST_DIRECTORY 0x00000001
	-#define ACE_WRITE_DATA 0x00000002
	-#define ACE_ADD_FILE 0x00000002
	-#define ACE_APPEND_DATA 0x00000004
	-#define ACE_ADD_SUBDIRECTORY 0x00000004
	-#define ACE_READ_NAMED_ATTRS 0x00000008
	-#define ACE_WRITE_NAMED_ATTRS 0x00000010
	-#define ACE_EXECUTE 0x00000020
	-#define ACE_DELETE_CHILD 0x00000040
	-#define ACE_READ_ATTRIBUTES 0x00000080
	-#define ACE_WRITE_ATTRIBUTES 0x00000100
	-#define ACE_DELETE 0x00010000
	-#define ACE_READ_ACL 0x00020000
	-#define ACE_WRITE_ACL 0x00040000
	-#define ACE_WRITE_OWNER 0x00080000
	-#define ACE_SYNCHRONIZE 0x00100000
	+#define ACE_READ_DATA 0x00000001 /* file: read data */
	+#define ACE_LIST_DIRECTORY 0x00000001 /* dir: list files */
	+#define ACE_WRITE_DATA 0x00000002 /* file: write data */
	+#define ACE_ADD_FILE 0x00000002 /* dir: create file */
	+#define ACE_APPEND_DATA 0x00000004 /* file: append data */
	+#define ACE_ADD_SUBDIRECTORY 0x00000004 /* dir: create subdir */
	+#define ACE_READ_NAMED_ATTRS 0x00000008 /* FILE_READ_EA */
	+#define ACE_WRITE_NAMED_ATTRS 0x00000010 /* FILE_WRITE_EA */
	+#define ACE_EXECUTE 0x00000020 /* file: execute */
	+#define ACE_TRAVERSE 0x00000020 /* dir: lookup name */
	+#define ACE_DELETE_CHILD 0x00000040 /* dir: unlink child */
	+#define ACE_READ_ATTRIBUTES 0x00000080 /* (all) stat, etc. */
	+#define ACE_WRITE_ATTRIBUTES 0x00000100 /* (all) utimes, etc. */
	+#define ACE_DELETE 0x00010000 /* (all) unlink self */
	+#define ACE_READ_ACL 0x00020000 /* (all) getsecattr */
	+#define ACE_WRITE_ACL 0x00040000 /* (all) setsecattr */
	+#define ACE_WRITE_OWNER 0x00080000 /* (all) chown */
	+#define ACE_SYNCHRONIZE 0x00100000 /* (all) */

	#define ACE_FILE_INHERIT_ACE 0x0001
	#define ACE_DIRECTORY_INHERIT_ACE 0x0002
	#define ACE_NO_PROPAGATE_INHERIT_ACE 0x0004
	#define ACE_INHERIT_ONLY_ACE 0x0008
	#define ACE_SUCCESSFUL_ACCESS_ACE_FLAG 0x0010
	#define ACE_FAILED_ACCESS_ACE_FLAG 0x0020
	#define ACE_IDENTIFIER_GROUP 0x0040
	#define ACE_INHERITED_ACE 0x0080
	#define ACE_OWNER 0x1000
	#define ACE_GROUP 0x2000
	#define ACE_EVERYONE 0x4000

	#define ACE_ACCESS_ALLOWED_ACE_TYPE 0x0000
	#define ACE_ACCESS_DENIED_ACE_TYPE 0x0001
	#define ACE_SYSTEM_AUDIT_ACE_TYPE 0x0002
	#define ACE_SYSTEM_ALARM_ACE_TYPE 0x0003

	#define ACL_AUTO_INHERIT 0x0001
	#define ACL_PROTECTED 0x0002
	#define ACL_DEFAULTED 0x0004
	#define ACL_FLAGS_ALL (ACL_AUTO_INHERIT\|ACL_PROTECTED\| \
	ACL_DEFAULTED)

	-#ifdef _KERNEL
	-
	/*
	* These are only applicable in a CIFS context.
	*/
	#define ACE_ACCESS_ALLOWED_COMPOUND_ACE_TYPE 0x04
	#define ACE_ACCESS_ALLOWED_OBJECT_ACE_TYPE 0x05
	#define ACE_ACCESS_DENIED_OBJECT_ACE_TYPE 0x06
	#define ACE_SYSTEM_AUDIT_OBJECT_ACE_TYPE 0x07
	#define ACE_SYSTEM_ALARM_OBJECT_ACE_TYPE 0x08
	#define ACE_ACCESS_ALLOWED_CALLBACK_ACE_TYPE 0x09
	#define ACE_ACCESS_DENIED_CALLBACK_ACE_TYPE 0x0A
	#define ACE_ACCESS_ALLOWED_CALLBACK_OBJECT_ACE_TYPE 0x0B
	#define ACE_ACCESS_DENIED_CALLBACK_OBJECT_ACE_TYPE 0x0C
	#define ACE_SYSTEM_AUDIT_CALLBACK_ACE_TYPE 0x0D
	#define ACE_SYSTEM_ALARM_CALLBACK_ACE_TYPE 0x0E
	#define ACE_SYSTEM_AUDIT_CALLBACK_OBJECT_ACE_TYPE 0x0F
	#define ACE_SYSTEM_ALARM_CALLBACK_OBJECT_ACE_TYPE 0x10

	#define ACE_ALL_TYPES 0x001F

	+#if defined(_KERNEL)
	+
	typedef struct ace_object {
	uid_t a_who; /* uid or gid */
	uint32_t a_access_mask; /* read,write,... */
	uint16_t a_flags; /* see below */
	uint16_t a_type; /* allow or deny */
	uint8_t a_obj_type[16]; /* obj type */
	uint8_t a_inherit_obj_type[16]; /* inherit obj */
	} ace_object_t;

	#endif

	#define ACE_ALL_PERMS (ACE_READ_DATA\|ACE_LIST_DIRECTORY\|ACE_WRITE_DATA\| \
	ACE_ADD_FILE\|ACE_APPEND_DATA\|ACE_ADD_SUBDIRECTORY\|ACE_READ_NAMED_ATTRS\| \
	ACE_WRITE_NAMED_ATTRS\|ACE_EXECUTE\|ACE_DELETE_CHILD\|ACE_READ_ATTRIBUTES\| \
	ACE_WRITE_ATTRIBUTES\|ACE_DELETE\|ACE_READ_ACL\|ACE_WRITE_ACL\| \
	ACE_WRITE_OWNER\|ACE_SYNCHRONIZE)

	+#define ACE_ALL_WRITE_PERMS (ACE_WRITE_DATA\|ACE_APPEND_DATA\| \
	+ ACE_WRITE_ATTRIBUTES\|ACE_WRITE_NAMED_ATTRS\|ACE_WRITE_ACL\| \
	+ ACE_WRITE_OWNER\|ACE_DELETE\|ACE_DELETE_CHILD)
	+
	+#define ACE_READ_PERMS (ACE_READ_DATA\|ACE_READ_ACL\|ACE_READ_ATTRIBUTES\| \
	+ ACE_READ_NAMED_ATTRS)
	+
	+#define ACE_WRITE_PERMS (ACE_WRITE_DATA\|ACE_APPEND_DATA\|ACE_WRITE_ATTRIBUTES\| \
	+ ACE_WRITE_NAMED_ATTRS)
	+
	+#define ACE_MODIFY_PERMS (ACE_READ_DATA\|ACE_LIST_DIRECTORY\|ACE_WRITE_DATA\| \
	+ ACE_ADD_FILE\|ACE_APPEND_DATA\|ACE_ADD_SUBDIRECTORY\|ACE_READ_NAMED_ATTRS\| \
	+ ACE_WRITE_NAMED_ATTRS\|ACE_EXECUTE\|ACE_DELETE_CHILD\|ACE_READ_ATTRIBUTES\| \
	+ ACE_WRITE_ATTRIBUTES\|ACE_DELETE\|ACE_READ_ACL\|ACE_SYNCHRONIZE)
	+
	/*
	* The following flags are supported by both NFSv4 ACLs and ace_t.
	*/
	#define ACE_NFSV4_SUP_FLAGS (ACE_FILE_INHERIT_ACE \| \
	ACE_DIRECTORY_INHERIT_ACE \| \
	ACE_NO_PROPAGATE_INHERIT_ACE \| \
	ACE_INHERIT_ONLY_ACE \| \
	ACE_IDENTIFIER_GROUP)

	#define ACE_TYPE_FLAGS (ACE_OWNER\|ACE_GROUP\|ACE_EVERYONE\| \
	ACE_IDENTIFIER_GROUP)
	#define ACE_INHERIT_FLAGS (ACE_FILE_INHERIT_ACE\| \
	ACE_DIRECTORY_INHERIT_ACE\|ACE_NO_PROPAGATE_INHERIT_ACE\|ACE_INHERIT_ONLY_ACE)

	/* cmd args to acl(2) for aclent_t */
	#define GETACL 1
	#define SETACL 2
	#define GETACLCNT 3

	/* cmd's to manipulate ace acls. */
	#define ACE_GETACL 4
	#define ACE_SETACL 5
	#define ACE_GETACLCNT 6

	/* minimal acl entries from GETACLCNT */
	#define MIN_ACL_ENTRIES 4

	#if !defined(_KERNEL)

	/* acl check errors */
	#define GRP_ERROR 1
	#define USER_ERROR 2
	#define OTHER_ERROR 3
	#define CLASS_ERROR 4
	#define DUPLICATE_ERROR 5
	#define MISS_ERROR 6
	#define MEM_ERROR 7
	#define ENTRY_ERROR 8


	/*
	* similar to ufs_acl.h: changed to char type for user commands (tar, cpio)
	* Attribute types
	*/
	#define UFSD_FREE ('0') /* Free entry */
	#define UFSD_ACL ('1') /* Access Control Lists */
	#define UFSD_DFACL ('2') /* reserved for future use */
	#define ACE_ACL ('3') /* ace_t style acls */

	/*
	* flag to [f]acl_get()
	* controls whether a trivial acl should be returned.
	*/
	#define ACL_NO_TRIVIAL 0x2


	/*
	* Flags to control acl_totext()
	*/

	#define ACL_APPEND_ID 0x1 /* append uid/gid to user/group entries */
	#define ACL_COMPACT_FMT 0x2 /* build ACL in ls -V format */
	#define ACL_NORESOLVE 0x4 /* don't do name service lookups */
	+#define ACL_SID_FMT 0x8 /* use usersid/groupsid when appropriate */

	/*
	* Legacy aclcheck errors for aclent_t ACLs
	*/
	#define EACL_GRP_ERROR GRP_ERROR
	#define EACL_USER_ERROR USER_ERROR
	#define EACL_OTHER_ERROR OTHER_ERROR
	#define EACL_CLASS_ERROR CLASS_ERROR
	#define EACL_DUPLICATE_ERROR DUPLICATE_ERROR
	#define EACL_MISS_ERROR MISS_ERROR
	#define EACL_MEM_ERROR MEM_ERROR
	#define EACL_ENTRY_ERROR ENTRY_ERROR

	#define EACL_INHERIT_ERROR 9 /* invalid inherit flags */
	#define EACL_FLAGS_ERROR 10 /* unknown flag value */
	#define EACL_PERM_MASK_ERROR 11 /* unknown permission */
	#define EACL_COUNT_ERROR 12 /* invalid acl count */

	#define EACL_INVALID_SLOT 13 /* invalid acl slot */
	#define EACL_NO_ACL_ENTRY 14 /* Entry doesn't exist */
	#define EACL_DIFF_TYPE 15 /* acls aren't same type */

	#define EACL_INVALID_USER_GROUP 16 /* need user/group name */
	#define EACL_INVALID_STR 17 /* invalid acl string */
	#define EACL_FIELD_NOT_BLANK 18 /* can't have blank field */
	#define EACL_INVALID_ACCESS_TYPE 19 /* invalid access type */
	#define EACL_UNKNOWN_DATA 20 /* Unrecognized data in ACL */
	#define EACL_MISSING_FIELDS 21 /* missing fields in acl */

	#define EACL_INHERIT_NOTDIR 22 /* Need dir for inheritance */

	extern int aclcheck(aclent_t , int, int );
	extern int acltomode(aclent_t , int, mode_t );
	extern int aclfrommode(aclent_t , int, mode_t );
	extern int aclsort(int, int, aclent_t *);
	extern char acltotext(aclent_t , int);
	extern aclent_t aclfromtext(char , int *);
	extern void acl_free(acl_t *);
	extern int acl_get(const char , int, acl_t *);
	extern int facl_get(int, int, acl_t **);
	extern int acl_set(const char , acl_t acl);
	extern int facl_set(int, acl_t *acl);
	extern int acl_strip(const char *, uid_t, gid_t, mode_t);
	extern int acl_trivial(const char *);
	extern char acl_totext(acl_t , int);
	extern int acl_fromtext(const char , acl_t *);
	extern int acl_check(acl_t *, int);

	#else /* !defined(_KERNEL) */

	extern void ksort(caddr_t, int, int, int ()(void , void *));
	extern int cmp2acls(void , void );

	#endif /* !defined(_KERNEL) */

	-#if defined(__STDC__)
	extern int acl(const char path, int cmd, int cnt, void buf);
	extern int facl(int fd, int cmd, int cnt, void *buf);
	-#else /* !__STDC__ */
	-extern int acl();
	-extern int facl();
	-#endif /* defined(__STDC__) */

	#ifdef __cplusplus
	}
	#endif

	#endif /* _SYS_ACL_H */
	Index: vendor-sys/openzfs/dist/lib/libspl/os/freebsd/getmntany.c
	===================================================================
	--- vendor-sys/openzfs/dist/lib/libspl/os/freebsd/getmntany.c (revision 366773)
	+++ vendor-sys/openzfs/dist/lib/libspl/os/freebsd/getmntany.c (revision 366774)
	@@ -1,67 +1,63 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License, Version 1.0 only
	* (the "License"). You may not use this file except in compliance
	* with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright 2005 Sun Microsystems, Inc. All rights reserved.
	* Copyright 2006 Ricardo Correia. All rights reserved.
	* Use is subject to license terms.
	*/

	/* Copyright (c) 1988 AT&T */
	/* All Rights Reserved */

	#include <stdio.h>
	#include <string.h>
	#include <sys/errno.h>
	#include <sys/mnttab.h>
	#include <sys/types.h>
	#include <sys/sysmacros.h>
	#include <sys/stat.h>
	#include <unistd.h>

	-#define BUFSIZE (MNT_LINE_MAX + 2)
	-
	-__thread char buf[BUFSIZE];
	-
	int
	getextmntent(const char path, struct extmnttab entry, struct stat64 *statbuf)
	{
	struct statfs sfs;

	if (strlen(path) >= MAXPATHLEN) {
	(void) fprintf(stderr, "invalid object; pathname too long\n");
	return (-1);
	}

	if (stat64(path, statbuf) != 0) {
	(void) fprintf(stderr, "cannot open '%s': %s\n",
	path, strerror(errno));
	return (-1);
	}

	if (statfs(path, &sfs) != 0) {
	(void) fprintf(stderr, "%s: %s\n", path,
	strerror(errno));
	return (-1);
	}
	statfs2mnttab(&sfs, (struct mnttab *)entry);
	return (0);
	}
	Index: vendor-sys/openzfs/dist/lib/libzfs/libzfs_sendrecv.c
	===================================================================
	--- vendor-sys/openzfs/dist/lib/libzfs/libzfs_sendrecv.c (revision 366773)
	+++ vendor-sys/openzfs/dist/lib/libzfs/libzfs_sendrecv.c (revision 366774)
	@@ -1,5203 +1,5188 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/

	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2011, 2020 by Delphix. All rights reserved.
	* Copyright (c) 2012, Joyent, Inc. All rights reserved.
	* Copyright (c) 2012 Pawel Jakub Dawidek <pawel@dawidek.net>.
	* All rights reserved
	* Copyright (c) 2013 Steven Hartland. All rights reserved.
	* Copyright 2015, OmniTI Computer Consulting, Inc. All rights reserved.
	* Copyright 2016 Igor Kozhukhov <ikozhukhov@gmail.com>
	* Copyright (c) 2018, loli10K <ezomori.nozomu@gmail.com>. All rights reserved.
	* Copyright (c) 2019 Datto Inc.
	*/

	#include <assert.h>
	#include <ctype.h>
	#include <errno.h>
	#include <libintl.h>
	#include <stdio.h>
	#include <stdlib.h>
	#include <strings.h>
	#include <unistd.h>
	#include <stddef.h>
	#include <fcntl.h>
	#include <sys/mount.h>
	#include <sys/mntent.h>
	#include <sys/mnttab.h>
	#include <sys/avl.h>
	#include <sys/debug.h>
	#include <sys/stat.h>
	#include <stddef.h>
	#include <pthread.h>
	#include <umem.h>
	#include <time.h>

	#include <libzfs.h>
	#include <libzfs_core.h>
	#include <libzutil.h>

	#include "zfs_namecheck.h"
	#include "zfs_prop.h"
	#include "zfs_fletcher.h"
	#include "libzfs_impl.h"
	#include <cityhash.h>
	#include <zlib.h>
	#include <sys/zio_checksum.h>
	#include <sys/dsl_crypt.h>
	#include <sys/ddt.h>
	#include <sys/socket.h>
	#include <sys/sha2.h>

	static int zfs_receive_impl(libzfs_handle_t , const char , const char *,
	recvflags_t , int, const char , nvlist_t , avl_tree_t , char **,
	const char , nvlist_t );
	static int guid_to_name_redact_snaps(libzfs_handle_t hdl, const char parent,
	uint64_t guid, boolean_t bookmark_ok, uint64_t *redact_snap_guids,
	uint64_t num_redact_snaps, char *name);
	static int guid_to_name(libzfs_handle_t , const char ,
	uint64_t, boolean_t, char *);

	typedef struct progress_arg {
	zfs_handle_t *pa_zhp;
	int pa_fd;
	boolean_t pa_parsable;
	boolean_t pa_estimate;
	int pa_verbosity;
	} progress_arg_t;

	static int
	dump_record(dmu_replay_record_t drr, void payload, int payload_len,
	zio_cksum_t *zc, int outfd)
	{
	ASSERT3U(offsetof(dmu_replay_record_t, drr_u.drr_checksum.drr_checksum),
	==, sizeof (dmu_replay_record_t) - sizeof (zio_cksum_t));
	fletcher_4_incremental_native(drr,
	offsetof(dmu_replay_record_t, drr_u.drr_checksum.drr_checksum), zc);
	if (drr->drr_type != DRR_BEGIN) {
	ASSERT(ZIO_CHECKSUM_IS_ZERO(&drr->drr_u.
	drr_checksum.drr_checksum));
	drr->drr_u.drr_checksum.drr_checksum = *zc;
	}
	fletcher_4_incremental_native(&drr->drr_u.drr_checksum.drr_checksum,
	sizeof (zio_cksum_t), zc);
	if (write(outfd, drr, sizeof (*drr)) == -1)
	return (errno);
	if (payload_len != 0) {
	fletcher_4_incremental_native(payload, payload_len, zc);
	if (write(outfd, payload, payload_len) == -1)
	return (errno);
	}
	return (0);
	}

	/*
	* Routines for dealing with the AVL tree of fs-nvlists
	*/
	typedef struct fsavl_node {
	avl_node_t fn_node;
	nvlist_t *fn_nvfs;
	char *fn_snapname;
	uint64_t fn_guid;
	} fsavl_node_t;

	static int
	fsavl_compare(const void arg1, const void arg2)
	{
	const fsavl_node_t fn1 = (const fsavl_node_t )arg1;
	const fsavl_node_t fn2 = (const fsavl_node_t )arg2;

	return (TREE_CMP(fn1->fn_guid, fn2->fn_guid));
	}

	/*
	* Given the GUID of a snapshot, find its containing filesystem and
	* (optionally) name.
	*/
	static nvlist_t *
	fsavl_find(avl_tree_t avl, uint64_t snapguid, char *snapname)
	{
	fsavl_node_t fn_find;
	fsavl_node_t *fn;

	fn_find.fn_guid = snapguid;

	fn = avl_find(avl, &fn_find, NULL);
	if (fn) {
	if (snapname)
	*snapname = fn->fn_snapname;
	return (fn->fn_nvfs);
	}
	return (NULL);
	}

	static void
	fsavl_destroy(avl_tree_t *avl)
	{
	fsavl_node_t *fn;
	void *cookie;

	if (avl == NULL)
	return;

	cookie = NULL;
	while ((fn = avl_destroy_nodes(avl, &cookie)) != NULL)
	free(fn);
	avl_destroy(avl);
	free(avl);
	}

	/*
	* Given an nvlist, produce an avl tree of snapshots, ordered by guid
	*/
	static avl_tree_t *
	fsavl_create(nvlist_t *fss)
	{
	avl_tree_t *fsavl;
	nvpair_t *fselem = NULL;

	if ((fsavl = malloc(sizeof (avl_tree_t))) == NULL)
	return (NULL);

	avl_create(fsavl, fsavl_compare, sizeof (fsavl_node_t),
	offsetof(fsavl_node_t, fn_node));

	while ((fselem = nvlist_next_nvpair(fss, fselem)) != NULL) {
	nvlist_t nvfs, snaps;
	nvpair_t *snapelem = NULL;

	VERIFY(0 == nvpair_value_nvlist(fselem, &nvfs));
	VERIFY(0 == nvlist_lookup_nvlist(nvfs, "snaps", &snaps));

	while ((snapelem =
	nvlist_next_nvpair(snaps, snapelem)) != NULL) {
	fsavl_node_t *fn;
	uint64_t guid;

	VERIFY(0 == nvpair_value_uint64(snapelem, &guid));
	if ((fn = malloc(sizeof (fsavl_node_t))) == NULL) {
	fsavl_destroy(fsavl);
	return (NULL);
	}
	fn->fn_nvfs = nvfs;
	fn->fn_snapname = nvpair_name(snapelem);
	fn->fn_guid = guid;

	/*
	* Note: if there are multiple snaps with the
	* same GUID, we ignore all but one.
	*/
	if (avl_find(fsavl, fn, NULL) == NULL)
	avl_add(fsavl, fn);
	else
	free(fn);
	}
	}

	return (fsavl);
	}

	/*
	* Routines for dealing with the giant nvlist of fs-nvlists, etc.
	*/
	typedef struct send_data {
	/*
	* assigned inside every recursive call,
	* restored from *_save on return:
	*
	* guid of fromsnap snapshot in parent dataset
	* txg of fromsnap snapshot in current dataset
	* txg of tosnap snapshot in current dataset
	*/

	uint64_t parent_fromsnap_guid;
	uint64_t fromsnap_txg;
	uint64_t tosnap_txg;

	/* the nvlists get accumulated during depth-first traversal */
	nvlist_t *parent_snaps;
	nvlist_t *fss;
	nvlist_t *snapprops;
	nvlist_t snapholds; / user holds */

	/* send-receive configuration, does not change during traversal */
	const char *fsname;
	const char *fromsnap;
	const char *tosnap;
	boolean_t recursive;
	boolean_t raw;
	boolean_t doall;
	boolean_t replicate;
	boolean_t verbose;
	boolean_t backup;
	boolean_t seenfrom;
	boolean_t seento;
	boolean_t holds; /* were holds requested with send -h */
	boolean_t props;

	/*
	* The header nvlist is of the following format:
	* {
	* "tosnap" -> string
	* "fromsnap" -> string (if incremental)
	* "fss" -> {
	* id -> {
	*
	* "name" -> string (full name; for debugging)
	* "parentfromsnap" -> number (guid of fromsnap in parent)
	*
	* "props" -> { name -> value (only if set here) }
	* "snaps" -> { name (lastname) -> number (guid) }
	* "snapprops" -> { name (lastname) -> { name -> value } }
	* "snapholds" -> { name (lastname) -> { holdname -> crtime } }
	*
	* "origin" -> number (guid) (if clone)
	* "is_encroot" -> boolean
	* "sent" -> boolean (not on-disk)
	* }
	* }
	* }
	*
	*/
	} send_data_t;

	static void
	send_iterate_prop(zfs_handle_t zhp, boolean_t received_only, nvlist_t nv);

	static int
	send_iterate_snap(zfs_handle_t zhp, void arg)
	{
	send_data_t *sd = arg;
	uint64_t guid = zhp->zfs_dmustats.dds_guid;
	uint64_t txg = zhp->zfs_dmustats.dds_creation_txg;
	char *snapname;
	nvlist_t *nv;
	boolean_t isfromsnap, istosnap, istosnapwithnofrom;

	snapname = strrchr(zhp->zfs_name, '@')+1;
	isfromsnap = (sd->fromsnap != NULL &&
	strcmp(sd->fromsnap, snapname) == 0);
	istosnap = (sd->tosnap != NULL && (strcmp(sd->tosnap, snapname) == 0));
	istosnapwithnofrom = (istosnap && sd->fromsnap == NULL);

	if (sd->tosnap_txg != 0 && txg > sd->tosnap_txg) {
	if (sd->verbose) {
	(void) fprintf(stderr, dgettext(TEXT_DOMAIN,
	"skipping snapshot %s because it was created "
	"after the destination snapshot (%s)\n"),
	zhp->zfs_name, sd->tosnap);
	}
	zfs_close(zhp);
	return (0);
	}

	VERIFY(0 == nvlist_add_uint64(sd->parent_snaps, snapname, guid));
	/*
	* NB: if there is no fromsnap here (it's a newly created fs in
	* an incremental replication), we will substitute the tosnap.
	*/
	if (isfromsnap \|\| (sd->parent_fromsnap_guid == 0 && istosnap)) {
	sd->parent_fromsnap_guid = guid;
	}

	if (!sd->recursive) {
	if (!sd->seenfrom && isfromsnap) {
	sd->seenfrom = B_TRUE;
	zfs_close(zhp);
	return (0);
	}

	if ((sd->seento \|\| !sd->seenfrom) && !istosnapwithnofrom) {
	zfs_close(zhp);
	return (0);
	}

	if (istosnap)
	sd->seento = B_TRUE;
	}

	VERIFY(0 == nvlist_alloc(&nv, NV_UNIQUE_NAME, 0));
	send_iterate_prop(zhp, sd->backup, nv);
	VERIFY(0 == nvlist_add_nvlist(sd->snapprops, snapname, nv));
	nvlist_free(nv);
	if (sd->holds) {
	nvlist_t *holds = fnvlist_alloc();
	int err = lzc_get_holds(zhp->zfs_name, &holds);
	if (err == 0) {
	VERIFY(0 == nvlist_add_nvlist(sd->snapholds,
	snapname, holds));
	}
	fnvlist_free(holds);
	}

	zfs_close(zhp);
	return (0);
	}

	static void
	send_iterate_prop(zfs_handle_t zhp, boolean_t received_only, nvlist_t nv)
	{
	nvlist_t *props = NULL;
	nvpair_t *elem = NULL;

	if (received_only)
	props = zfs_get_recvd_props(zhp);
	else
	props = zhp->zfs_props;

	while ((elem = nvlist_next_nvpair(props, elem)) != NULL) {
	char *propname = nvpair_name(elem);
	zfs_prop_t prop = zfs_name_to_prop(propname);
	nvlist_t *propnv;

	if (!zfs_prop_user(propname)) {
	/*
	* Realistically, this should never happen. However,
	* we want the ability to add DSL properties without
	* needing to make incompatible version changes. We
	* need to ignore unknown properties to allow older
	* software to still send datasets containing these
	* properties, with the unknown properties elided.
	*/
	if (prop == ZPROP_INVAL)
	continue;

	if (zfs_prop_readonly(prop))
	continue;
	}

	verify(nvpair_value_nvlist(elem, &propnv) == 0);
	if (prop == ZFS_PROP_QUOTA \|\| prop == ZFS_PROP_RESERVATION \|\|
	prop == ZFS_PROP_REFQUOTA \|\|
	prop == ZFS_PROP_REFRESERVATION) {
	char *source;
	uint64_t value;
	verify(nvlist_lookup_uint64(propnv,
	ZPROP_VALUE, &value) == 0);
	if (zhp->zfs_type == ZFS_TYPE_SNAPSHOT)
	continue;
	/*
	* May have no source before SPA_VERSION_RECVD_PROPS,
	* but is still modifiable.
	*/
	if (nvlist_lookup_string(propnv,
	ZPROP_SOURCE, &source) == 0) {
	if ((strcmp(source, zhp->zfs_name) != 0) &&
	(strcmp(source,
	ZPROP_SOURCE_VAL_RECVD) != 0))
	continue;
	}
	} else {
	char *source;
	if (nvlist_lookup_string(propnv,
	ZPROP_SOURCE, &source) != 0)
	continue;
	if ((strcmp(source, zhp->zfs_name) != 0) &&
	(strcmp(source, ZPROP_SOURCE_VAL_RECVD) != 0))
	continue;
	}

	if (zfs_prop_user(propname) \|\|
	zfs_prop_get_type(prop) == PROP_TYPE_STRING) {
	char *value;
	verify(nvlist_lookup_string(propnv,
	ZPROP_VALUE, &value) == 0);
	VERIFY(0 == nvlist_add_string(nv, propname, value));
	} else {
	uint64_t value;
	verify(nvlist_lookup_uint64(propnv,
	ZPROP_VALUE, &value) == 0);
	VERIFY(0 == nvlist_add_uint64(nv, propname, value));
	}
	}
	}

	/*
	* returns snapshot creation txg
	* and returns 0 if the snapshot does not exist
	*/
	static uint64_t
	get_snap_txg(libzfs_handle_t hdl, const char fs, const char *snap)
	{
	char name[ZFS_MAX_DATASET_NAME_LEN];
	uint64_t txg = 0;

	if (fs == NULL \|\| fs[0] == '\0' \|\| snap == NULL \|\| snap[0] == '\0')
	return (txg);

	(void) snprintf(name, sizeof (name), "%s@%s", fs, snap);
	if (zfs_dataset_exists(hdl, name, ZFS_TYPE_SNAPSHOT)) {
	zfs_handle_t *zhp = zfs_open(hdl, name, ZFS_TYPE_SNAPSHOT);
	if (zhp != NULL) {
	txg = zfs_prop_get_int(zhp, ZFS_PROP_CREATETXG);
	zfs_close(zhp);
	}
	}

	return (txg);
	}

	/*
	* recursively generate nvlists describing datasets. See comment
	* for the data structure send_data_t above for description of contents
	* of the nvlist.
	*/
	static int
	send_iterate_fs(zfs_handle_t zhp, void arg)
	{
	send_data_t *sd = arg;
	nvlist_t nvfs = NULL, nv = NULL;
	int rv = 0;
	uint64_t min_txg = 0, max_txg = 0;
	uint64_t parent_fromsnap_guid_save = sd->parent_fromsnap_guid;
	uint64_t fromsnap_txg_save = sd->fromsnap_txg;
	uint64_t tosnap_txg_save = sd->tosnap_txg;
	uint64_t txg = zhp->zfs_dmustats.dds_creation_txg;
	uint64_t guid = zhp->zfs_dmustats.dds_guid;
	uint64_t fromsnap_txg, tosnap_txg;
	char guidstring[64];

	fromsnap_txg = get_snap_txg(zhp->zfs_hdl, zhp->zfs_name, sd->fromsnap);
	if (fromsnap_txg != 0)
	sd->fromsnap_txg = fromsnap_txg;

	tosnap_txg = get_snap_txg(zhp->zfs_hdl, zhp->zfs_name, sd->tosnap);
	if (tosnap_txg != 0)
	sd->tosnap_txg = tosnap_txg;

	/*
	* on the send side, if the current dataset does not have tosnap,
	* perform two additional checks:
	*
	* - skip sending the current dataset if it was created later than
	* the parent tosnap
	* - return error if the current dataset was created earlier than
	* the parent tosnap
	*/
	if (sd->tosnap != NULL && tosnap_txg == 0) {
	if (sd->tosnap_txg != 0 && txg > sd->tosnap_txg) {
	if (sd->verbose) {
	(void) fprintf(stderr, dgettext(TEXT_DOMAIN,
	"skipping dataset %s: snapshot %s does "
	"not exist\n"), zhp->zfs_name, sd->tosnap);
	}
	} else {
	(void) fprintf(stderr, dgettext(TEXT_DOMAIN,
	"cannot send %s@%s%s: snapshot %s@%s does not "
	"exist\n"), sd->fsname, sd->tosnap, sd->recursive ?
	dgettext(TEXT_DOMAIN, " recursively") : "",
	zhp->zfs_name, sd->tosnap);
	rv = EZFS_NOENT;
	}
	goto out;
	}

	nvfs = fnvlist_alloc();
	fnvlist_add_string(nvfs, "name", zhp->zfs_name);
	fnvlist_add_uint64(nvfs, "parentfromsnap",
	sd->parent_fromsnap_guid);

	if (zhp->zfs_dmustats.dds_origin[0]) {
	zfs_handle_t *origin = zfs_open(zhp->zfs_hdl,
	zhp->zfs_dmustats.dds_origin, ZFS_TYPE_SNAPSHOT);
	if (origin == NULL) {
	rv = -1;
	goto out;
	}
	fnvlist_add_uint64(nvfs, "origin",
	origin->zfs_dmustats.dds_guid);

	zfs_close(origin);
	}

	/* iterate over props */
	if (sd->props \|\| sd->backup \|\| sd->recursive) {
	nv = fnvlist_alloc();
	send_iterate_prop(zhp, sd->backup, nv);
	}
	if (zfs_prop_get_int(zhp, ZFS_PROP_ENCRYPTION) != ZIO_CRYPT_OFF) {
	boolean_t encroot;

	/* determine if this dataset is an encryption root */
	if (zfs_crypto_get_encryption_root(zhp, &encroot, NULL) != 0) {
	rv = -1;
	goto out;
	}

	if (encroot)
	fnvlist_add_boolean(nvfs, "is_encroot");

	/*
	* Encrypted datasets can only be sent with properties if
	* the raw flag is specified because the receive side doesn't
	* currently have a mechanism for recursively asking the user
	* for new encryption parameters.
	*/
	if (!sd->raw) {
	(void) fprintf(stderr, dgettext(TEXT_DOMAIN,
	"cannot send %s@%s: encrypted dataset %s may not "
	"be sent with properties without the raw flag\n"),
	sd->fsname, sd->tosnap, zhp->zfs_name);
	rv = -1;
	goto out;
	}

	}

	if (nv != NULL)
	fnvlist_add_nvlist(nvfs, "props", nv);

	/* iterate over snaps, and set sd->parent_fromsnap_guid */
	sd->parent_fromsnap_guid = 0;
	sd->parent_snaps = fnvlist_alloc();
	sd->snapprops = fnvlist_alloc();
	if (sd->holds)
	VERIFY(0 == nvlist_alloc(&sd->snapholds, NV_UNIQUE_NAME, 0));


	/*
	* If this is a "doall" send, a replicate send or we're just trying
	* to gather a list of previous snapshots, iterate through all the
	* snaps in the txg range. Otherwise just look at the one we're
	* interested in.
	*/
	if (sd->doall \|\| sd->replicate \|\| sd->tosnap == NULL) {
	if (!sd->replicate && fromsnap_txg != 0)
	min_txg = fromsnap_txg;
	if (!sd->replicate && tosnap_txg != 0)
	max_txg = tosnap_txg;
	(void) zfs_iter_snapshots_sorted(zhp, send_iterate_snap, sd,
	min_txg, max_txg);
	} else {
	char snapname[MAXPATHLEN] = { 0 };
	zfs_handle_t *snap;

	(void) snprintf(snapname, sizeof (snapname), "%s@%s",
	zhp->zfs_name, sd->tosnap);
	if (sd->fromsnap != NULL)
	sd->seenfrom = B_TRUE;
	snap = zfs_open(zhp->zfs_hdl, snapname,
	ZFS_TYPE_SNAPSHOT);
	if (snap != NULL)
	(void) send_iterate_snap(snap, sd);
	}

	fnvlist_add_nvlist(nvfs, "snaps", sd->parent_snaps);
	fnvlist_add_nvlist(nvfs, "snapprops", sd->snapprops);
	if (sd->holds)
	fnvlist_add_nvlist(nvfs, "snapholds", sd->snapholds);
	fnvlist_free(sd->parent_snaps);
	fnvlist_free(sd->snapprops);
	fnvlist_free(sd->snapholds);

	/* Do not allow the size of the properties list to exceed the limit */
	if ((fnvlist_size(nvfs) + fnvlist_size(sd->fss)) >
	zhp->zfs_hdl->libzfs_max_nvlist) {
	(void) fprintf(stderr, dgettext(TEXT_DOMAIN,
	"warning: cannot send %s@%s: the size of the list of "
	"snapshots and properties is too large to be received "
	"successfully.\n"
	"Select a smaller number of snapshots to send.\n"),
	zhp->zfs_name, sd->tosnap);
	rv = EZFS_NOSPC;
	goto out;
	}
	/* add this fs to nvlist */
	(void) snprintf(guidstring, sizeof (guidstring),
	"0x%llx", (longlong_t)guid);
	fnvlist_add_nvlist(sd->fss, guidstring, nvfs);

	/* iterate over children */
	if (sd->recursive)
	rv = zfs_iter_filesystems(zhp, send_iterate_fs, sd);

	out:
	sd->parent_fromsnap_guid = parent_fromsnap_guid_save;
	sd->fromsnap_txg = fromsnap_txg_save;
	sd->tosnap_txg = tosnap_txg_save;
	fnvlist_free(nv);
	fnvlist_free(nvfs);

	zfs_close(zhp);
	return (rv);
	}

	static int
	gather_nvlist(libzfs_handle_t hdl, const char fsname, const char *fromsnap,
	const char *tosnap, boolean_t recursive, boolean_t raw, boolean_t doall,
	boolean_t replicate, boolean_t verbose, boolean_t backup, boolean_t holds,
	boolean_t props, nvlist_t nvlp, avl_tree_t avlp)
	{
	zfs_handle_t *zhp;
	send_data_t sd = { 0 };
	int error;

	zhp = zfs_open(hdl, fsname, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME);
	if (zhp == NULL)
	return (EZFS_BADTYPE);

	VERIFY(0 == nvlist_alloc(&sd.fss, NV_UNIQUE_NAME, 0));
	sd.fsname = fsname;
	sd.fromsnap = fromsnap;
	sd.tosnap = tosnap;
	sd.recursive = recursive;
	sd.raw = raw;
	sd.doall = doall;
	sd.replicate = replicate;
	sd.verbose = verbose;
	sd.backup = backup;
	sd.holds = holds;
	sd.props = props;

	if ((error = send_iterate_fs(zhp, &sd)) != 0) {
	nvlist_free(sd.fss);
	if (avlp != NULL)
	*avlp = NULL;
	*nvlp = NULL;
	return (error);
	}

	if (avlp != NULL && (*avlp = fsavl_create(sd.fss)) == NULL) {
	nvlist_free(sd.fss);
	*nvlp = NULL;
	return (EZFS_NOMEM);
	}

	*nvlp = sd.fss;
	return (0);
	}

	/*
	* Routines specific to "zfs send"
	*/
	typedef struct send_dump_data {
	/* these are all just the short snapname (the part after the @) */
	const char *fromsnap;
	const char *tosnap;
	char prevsnap[ZFS_MAX_DATASET_NAME_LEN];
	uint64_t prevsnap_obj;
	boolean_t seenfrom, seento, replicate, doall, fromorigin;
	boolean_t dryrun, parsable, progress, embed_data, std_out;
	boolean_t large_block, compress, raw, holds;
	int outfd;
	boolean_t err;
	nvlist_t *fss;
	nvlist_t *snapholds;
	avl_tree_t *fsavl;
	snapfilter_cb_t *filter_cb;
	void *filter_cb_arg;
	nvlist_t *debugnv;
	char holdtag[ZFS_MAX_DATASET_NAME_LEN];
	int cleanup_fd;
	int verbosity;
	uint64_t size;
	} send_dump_data_t;

	static int
	zfs_send_space(zfs_handle_t zhp, const char snapname, const char *from,
	enum lzc_send_flags flags, uint64_t *spacep)
	{
	libzfs_handle_t *hdl = zhp->zfs_hdl;
	int error;

	assert(snapname != NULL);
	error = lzc_send_space(snapname, from, flags, spacep);

	if (error != 0) {
	char errbuf[1024];
	(void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN,
	"warning: cannot estimate space for '%s'"), snapname);

	switch (error) {
	case EXDEV:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"not an earlier snapshot from the same fs"));
	return (zfs_error(hdl, EZFS_CROSSTARGET, errbuf));

	case ENOENT:
	if (zfs_dataset_exists(hdl, snapname,
	ZFS_TYPE_SNAPSHOT)) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"incremental source (%s) does not exist"),
	snapname);
	}
	return (zfs_error(hdl, EZFS_NOENT, errbuf));

	case EDQUOT:
	case EFBIG:
	case EIO:
	case ENOLINK:
	case ENOSPC:
	case ENOSTR:
	case ENXIO:
	case EPIPE:
	case ERANGE:
	case EFAULT:
	case EROFS:
	case EINVAL:
	zfs_error_aux(hdl, strerror(error));
	return (zfs_error(hdl, EZFS_BADBACKUP, errbuf));

	default:
	return (zfs_standard_error(hdl, error, errbuf));
	}
	}

	return (0);
	}

	/*
	* Dumps a backup of the given snapshot (incremental from fromsnap if it's not
	* NULL) to the file descriptor specified by outfd.
	*/
	static int
	dump_ioctl(zfs_handle_t zhp, const char fromsnap, uint64_t fromsnap_obj,
	boolean_t fromorigin, int outfd, enum lzc_send_flags flags,
	nvlist_t *debugnv)
	{
	zfs_cmd_t zc = {"\0"};
	libzfs_handle_t *hdl = zhp->zfs_hdl;
	nvlist_t *thisdbg;

	assert(zhp->zfs_type == ZFS_TYPE_SNAPSHOT);
	assert(fromsnap_obj == 0 \|\| !fromorigin);

	(void) strlcpy(zc.zc_name, zhp->zfs_name, sizeof (zc.zc_name));
	zc.zc_cookie = outfd;
	zc.zc_obj = fromorigin;
	zc.zc_sendobj = zfs_prop_get_int(zhp, ZFS_PROP_OBJSETID);
	zc.zc_fromobj = fromsnap_obj;
	zc.zc_flags = flags;

	VERIFY(0 == nvlist_alloc(&thisdbg, NV_UNIQUE_NAME, 0));
	if (fromsnap && fromsnap[0] != '\0') {
	VERIFY(0 == nvlist_add_string(thisdbg,
	"fromsnap", fromsnap));
	}

	if (zfs_ioctl(zhp->zfs_hdl, ZFS_IOC_SEND, &zc) != 0) {
	char errbuf[1024];
	(void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN,
	"warning: cannot send '%s'"), zhp->zfs_name);

	VERIFY(0 == nvlist_add_uint64(thisdbg, "error", errno));
	if (debugnv) {
	VERIFY(0 == nvlist_add_nvlist(debugnv,
	zhp->zfs_name, thisdbg));
	}
	nvlist_free(thisdbg);

	switch (errno) {
	case EXDEV:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"not an earlier snapshot from the same fs"));
	return (zfs_error(hdl, EZFS_CROSSTARGET, errbuf));

	case EACCES:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"source key must be loaded"));
	return (zfs_error(hdl, EZFS_CRYPTOFAILED, errbuf));

	case ENOENT:
	if (zfs_dataset_exists(hdl, zc.zc_name,
	ZFS_TYPE_SNAPSHOT)) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"incremental source (@%s) does not exist"),
	zc.zc_value);
	}
	return (zfs_error(hdl, EZFS_NOENT, errbuf));

	case EDQUOT:
	case EFBIG:
	case EIO:
	case ENOLINK:
	case ENOSPC:
	case ENOSTR:
	case ENXIO:
	case EPIPE:
	case ERANGE:
	case EFAULT:
	case EROFS:
	zfs_error_aux(hdl, strerror(errno));
	return (zfs_error(hdl, EZFS_BADBACKUP, errbuf));

	default:
	return (zfs_standard_error(hdl, errno, errbuf));
	}
	}

	if (debugnv)
	VERIFY(0 == nvlist_add_nvlist(debugnv, zhp->zfs_name, thisdbg));
	nvlist_free(thisdbg);

	return (0);
	}

	static void
	gather_holds(zfs_handle_t zhp, send_dump_data_t sdd)
	{
	assert(zhp->zfs_type == ZFS_TYPE_SNAPSHOT);

	/*
	* zfs_send() only sets snapholds for sends that need them,
	* e.g. replication and doall.
	*/
	if (sdd->snapholds == NULL)
	return;

	fnvlist_add_string(sdd->snapholds, zhp->zfs_name, sdd->holdtag);
	}

	int
	zfs_send_progress(zfs_handle_t zhp, int fd, uint64_t bytes_written,
	uint64_t *blocks_visited)
	{
	zfs_cmd_t zc = {"\0"};

	(void) strlcpy(zc.zc_name, zhp->zfs_name, sizeof (zc.zc_name));
	zc.zc_cookie = fd;
	if (zfs_ioctl(zhp->zfs_hdl, ZFS_IOC_SEND_PROGRESS, &zc) != 0)
	return (errno);
	if (bytes_written != NULL)
	*bytes_written = zc.zc_cookie;
	if (blocks_visited != NULL)
	*blocks_visited = zc.zc_objset_type;
	return (0);
	}

	static void *
	send_progress_thread(void *arg)
	{
	progress_arg_t *pa = arg;
	zfs_handle_t *zhp = pa->pa_zhp;
	uint64_t bytes;
	uint64_t blocks;
	char buf[16];
	time_t t;
	struct tm *tm;
	boolean_t firstloop = B_TRUE;

	/*
	* Print the progress from ZFS_IOC_SEND_PROGRESS every second.
	*/
	for (;;) {
	int err;
	(void) sleep(1);
	if ((err = zfs_send_progress(zhp, pa->pa_fd, &bytes,
	&blocks)) != 0) {
	if (err == EINTR \|\| err == ENOENT)
	return ((void *)0);
	return ((void *)(uintptr_t)err);
	}

	if (firstloop && !pa->pa_parsable) {
	(void) fprintf(stderr,
	"TIME %s %sSNAPSHOT %s\n",
	pa->pa_estimate ? "BYTES" : " SENT",
	pa->pa_verbosity >= 2 ? " BLOCKS " : "",
	zhp->zfs_name);
	firstloop = B_FALSE;
	}

	(void) time(&t);
	tm = localtime(&t);

	if (pa->pa_verbosity >= 2 && pa->pa_parsable) {
	(void) fprintf(stderr,
	"%02d:%02d:%02d\t%llu\t%llu\t%s\n",
	tm->tm_hour, tm->tm_min, tm->tm_sec,
	(u_longlong_t)bytes, (u_longlong_t)blocks,
	zhp->zfs_name);
	} else if (pa->pa_verbosity >= 2) {
	zfs_nicenum(bytes, buf, sizeof (buf));
	(void) fprintf(stderr,
	"%02d:%02d:%02d %5s %8llu %s\n",
	tm->tm_hour, tm->tm_min, tm->tm_sec,
	buf, (u_longlong_t)blocks, zhp->zfs_name);
	} else if (pa->pa_parsable) {
	(void) fprintf(stderr, "%02d:%02d:%02d\t%llu\t%s\n",
	tm->tm_hour, tm->tm_min, tm->tm_sec,
	(u_longlong_t)bytes, zhp->zfs_name);
	} else {
	zfs_nicebytes(bytes, buf, sizeof (buf));
	(void) fprintf(stderr, "%02d:%02d:%02d %5s %s\n",
	tm->tm_hour, tm->tm_min, tm->tm_sec,
	buf, zhp->zfs_name);
	}
	}
	}

	static void
	send_print_verbose(FILE fout, const char tosnap, const char *fromsnap,
	uint64_t size, boolean_t parsable)
	{
	if (parsable) {
	if (fromsnap != NULL) {
	(void) fprintf(fout, "incremental\t%s\t%s",
	fromsnap, tosnap);
	} else {
	(void) fprintf(fout, "full\t%s",
	tosnap);
	}
	} else {
	if (fromsnap != NULL) {
	if (strchr(fromsnap, '@') == NULL &&
	strchr(fromsnap, '#') == NULL) {
	(void) fprintf(fout, dgettext(TEXT_DOMAIN,
	"send from @%s to %s"),
	fromsnap, tosnap);
	} else {
	(void) fprintf(fout, dgettext(TEXT_DOMAIN,
	"send from %s to %s"),
	fromsnap, tosnap);
	}
	} else {
	(void) fprintf(fout, dgettext(TEXT_DOMAIN,
	"full send of %s"),
	tosnap);
	}
	}

	if (parsable) {
	(void) fprintf(fout, "\t%llu",
	(longlong_t)size);
	} else if (size != 0) {
	char buf[16];
	zfs_nicebytes(size, buf, sizeof (buf));
	(void) fprintf(fout, dgettext(TEXT_DOMAIN,
	" estimated size is %s"), buf);
	}
	(void) fprintf(fout, "\n");
	}

	static int
	dump_snapshot(zfs_handle_t zhp, void arg)
	{
	send_dump_data_t *sdd = arg;
	progress_arg_t pa = { 0 };
	pthread_t tid;
	char *thissnap;
	enum lzc_send_flags flags = 0;
	int err;
	boolean_t isfromsnap, istosnap, fromorigin;
	boolean_t exclude = B_FALSE;
	FILE *fout = sdd->std_out ? stdout : stderr;

	err = 0;
	thissnap = strchr(zhp->zfs_name, '@') + 1;
	isfromsnap = (sdd->fromsnap != NULL &&
	strcmp(sdd->fromsnap, thissnap) == 0);

	if (!sdd->seenfrom && isfromsnap) {
	gather_holds(zhp, sdd);
	sdd->seenfrom = B_TRUE;
	(void) strlcpy(sdd->prevsnap, thissnap,
	sizeof (sdd->prevsnap));
	sdd->prevsnap_obj = zfs_prop_get_int(zhp, ZFS_PROP_OBJSETID);
	zfs_close(zhp);
	return (0);
	}

	if (sdd->seento \|\| !sdd->seenfrom) {
	zfs_close(zhp);
	return (0);
	}

	istosnap = (strcmp(sdd->tosnap, thissnap) == 0);
	if (istosnap)
	sdd->seento = B_TRUE;

	if (sdd->large_block)
	flags \|= LZC_SEND_FLAG_LARGE_BLOCK;
	if (sdd->embed_data)
	flags \|= LZC_SEND_FLAG_EMBED_DATA;
	if (sdd->compress)
	flags \|= LZC_SEND_FLAG_COMPRESS;
	if (sdd->raw)
	flags \|= LZC_SEND_FLAG_RAW;

	if (!sdd->doall && !isfromsnap && !istosnap) {
	if (sdd->replicate) {
	char *snapname;
	nvlist_t *snapprops;
	/*
	* Filter out all intermediate snapshots except origin
	* snapshots needed to replicate clones.
	*/
	nvlist_t *nvfs = fsavl_find(sdd->fsavl,
	zhp->zfs_dmustats.dds_guid, &snapname);

	VERIFY(0 == nvlist_lookup_nvlist(nvfs,
	"snapprops", &snapprops));
	VERIFY(0 == nvlist_lookup_nvlist(snapprops,
	thissnap, &snapprops));
	exclude = !nvlist_exists(snapprops, "is_clone_origin");
	} else {
	exclude = B_TRUE;
	}
	}

	/*
	* If a filter function exists, call it to determine whether
	* this snapshot will be sent.
	*/
	if (exclude \|\| (sdd->filter_cb != NULL &&
	sdd->filter_cb(zhp, sdd->filter_cb_arg) == B_FALSE)) {
	/*
	* This snapshot is filtered out. Don't send it, and don't
	* set prevsnap_obj, so it will be as if this snapshot didn't
	* exist, and the next accepted snapshot will be sent as
	* an incremental from the last accepted one, or as the
	* first (and full) snapshot in the case of a replication,
	* non-incremental send.
	*/
	zfs_close(zhp);
	return (0);
	}

	gather_holds(zhp, sdd);
	fromorigin = sdd->prevsnap[0] == '\0' &&
	(sdd->fromorigin \|\| sdd->replicate);

	if (sdd->verbosity != 0) {
	uint64_t size = 0;
	char fromds[ZFS_MAX_DATASET_NAME_LEN];

	if (sdd->prevsnap[0] != '\0') {
	(void) strlcpy(fromds, zhp->zfs_name, sizeof (fromds));
	*(strchr(fromds, '@') + 1) = '\0';
	(void) strlcat(fromds, sdd->prevsnap, sizeof (fromds));
	}
	if (zfs_send_space(zhp, zhp->zfs_name,
	sdd->prevsnap[0] ? fromds : NULL, flags, &size) != 0) {
	size = 0; /* cannot estimate send space */
	} else {
	send_print_verbose(fout, zhp->zfs_name,
	sdd->prevsnap[0] ? sdd->prevsnap : NULL,
	size, sdd->parsable);
	}
	sdd->size += size;
	}

	if (!sdd->dryrun) {
	/*
	* If progress reporting is requested, spawn a new thread to
	* poll ZFS_IOC_SEND_PROGRESS at a regular interval.
	*/
	if (sdd->progress) {
	pa.pa_zhp = zhp;
	pa.pa_fd = sdd->outfd;
	pa.pa_parsable = sdd->parsable;
	pa.pa_estimate = B_FALSE;
	pa.pa_verbosity = sdd->verbosity;

	if ((err = pthread_create(&tid, NULL,
	send_progress_thread, &pa)) != 0) {
	zfs_close(zhp);
	return (err);
	}
	}

	err = dump_ioctl(zhp, sdd->prevsnap, sdd->prevsnap_obj,
	fromorigin, sdd->outfd, flags, sdd->debugnv);

	if (sdd->progress) {
	void *status = NULL;
	(void) pthread_cancel(tid);
	(void) pthread_join(tid, &status);
	int error = (int)(uintptr_t)status;
	if (error != 0 && status != PTHREAD_CANCELED) {
	char errbuf[1024];
	(void) snprintf(errbuf, sizeof (errbuf),
	dgettext(TEXT_DOMAIN,
	"progress thread exited nonzero"));
	return (zfs_standard_error(zhp->zfs_hdl, error,
	errbuf));
	}
	}
	}

	(void) strcpy(sdd->prevsnap, thissnap);
	sdd->prevsnap_obj = zfs_prop_get_int(zhp, ZFS_PROP_OBJSETID);
	zfs_close(zhp);
	return (err);
	}

	static int
	dump_filesystem(zfs_handle_t zhp, void arg)
	{
	int rv = 0;
	send_dump_data_t *sdd = arg;
	boolean_t missingfrom = B_FALSE;
	zfs_cmd_t zc = {"\0"};
	uint64_t min_txg = 0, max_txg = 0;

	(void) snprintf(zc.zc_name, sizeof (zc.zc_name), "%s@%s",
	zhp->zfs_name, sdd->tosnap);
	if (zfs_ioctl(zhp->zfs_hdl, ZFS_IOC_OBJSET_STATS, &zc) != 0) {
	(void) fprintf(stderr, dgettext(TEXT_DOMAIN,
	"WARNING: could not send %s@%s: does not exist\n"),
	zhp->zfs_name, sdd->tosnap);
	sdd->err = B_TRUE;
	return (0);
	}

	if (sdd->replicate && sdd->fromsnap) {
	/*
	* If this fs does not have fromsnap, and we're doing
	* recursive, we need to send a full stream from the
	* beginning (or an incremental from the origin if this
	* is a clone). If we're doing non-recursive, then let
	* them get the error.
	*/
	(void) snprintf(zc.zc_name, sizeof (zc.zc_name), "%s@%s",
	zhp->zfs_name, sdd->fromsnap);
	if (zfs_ioctl(zhp->zfs_hdl,
	ZFS_IOC_OBJSET_STATS, &zc) != 0) {
	missingfrom = B_TRUE;
	}
	}

	sdd->seenfrom = sdd->seento = sdd->prevsnap[0] = 0;
	sdd->prevsnap_obj = 0;
	if (sdd->fromsnap == NULL \|\| missingfrom)
	sdd->seenfrom = B_TRUE;



	/*
	* Iterate through all snapshots and process the ones we will be
	* sending. If we only have a "from" and "to" snapshot to deal
	* with, we can avoid iterating through all the other snapshots.
	*/
	if (sdd->doall \|\| sdd->replicate \|\| sdd->tosnap == NULL) {
	if (!sdd->replicate && sdd->fromsnap != NULL)
	min_txg = get_snap_txg(zhp->zfs_hdl, zhp->zfs_name,
	sdd->fromsnap);
	if (!sdd->replicate && sdd->tosnap != NULL)
	max_txg = get_snap_txg(zhp->zfs_hdl, zhp->zfs_name,
	sdd->tosnap);
	rv = zfs_iter_snapshots_sorted(zhp, dump_snapshot, arg,
	min_txg, max_txg);
	} else {
	char snapname[MAXPATHLEN] = { 0 };
	zfs_handle_t *snap;

	if (!sdd->seenfrom) {
	(void) snprintf(snapname, sizeof (snapname),
	"%s@%s", zhp->zfs_name, sdd->fromsnap);
	snap = zfs_open(zhp->zfs_hdl, snapname,
	ZFS_TYPE_SNAPSHOT);
	if (snap != NULL)
	rv = dump_snapshot(snap, sdd);
	else
	rv = -1;
	}

	if (rv == 0) {
	(void) snprintf(snapname, sizeof (snapname),
	"%s@%s", zhp->zfs_name, sdd->tosnap);
	snap = zfs_open(zhp->zfs_hdl, snapname,
	ZFS_TYPE_SNAPSHOT);
	if (snap != NULL)
	rv = dump_snapshot(snap, sdd);
	else
	rv = -1;
	}
	}

	if (!sdd->seenfrom) {
	(void) fprintf(stderr, dgettext(TEXT_DOMAIN,
	"WARNING: could not send %s@%s:\n"
	"incremental source (%s@%s) does not exist\n"),
	zhp->zfs_name, sdd->tosnap,
	zhp->zfs_name, sdd->fromsnap);
	sdd->err = B_TRUE;
	} else if (!sdd->seento) {
	if (sdd->fromsnap) {
	(void) fprintf(stderr, dgettext(TEXT_DOMAIN,
	"WARNING: could not send %s@%s:\n"
	"incremental source (%s@%s) "
	"is not earlier than it\n"),
	zhp->zfs_name, sdd->tosnap,
	zhp->zfs_name, sdd->fromsnap);
	} else {
	(void) fprintf(stderr, dgettext(TEXT_DOMAIN,
	"WARNING: "
	"could not send %s@%s: does not exist\n"),
	zhp->zfs_name, sdd->tosnap);
	}
	sdd->err = B_TRUE;
	}

	return (rv);
	}

	static int
	dump_filesystems(zfs_handle_t rzhp, void arg)
	{
	send_dump_data_t *sdd = arg;
	nvpair_t *fspair;
	boolean_t needagain, progress;

	if (!sdd->replicate)
	return (dump_filesystem(rzhp, sdd));

	/* Mark the clone origin snapshots. */
	for (fspair = nvlist_next_nvpair(sdd->fss, NULL); fspair;
	fspair = nvlist_next_nvpair(sdd->fss, fspair)) {
	nvlist_t *nvfs;
	uint64_t origin_guid = 0;

	VERIFY(0 == nvpair_value_nvlist(fspair, &nvfs));
	(void) nvlist_lookup_uint64(nvfs, "origin", &origin_guid);
	if (origin_guid != 0) {
	char *snapname;
	nvlist_t *origin_nv = fsavl_find(sdd->fsavl,
	origin_guid, &snapname);
	if (origin_nv != NULL) {
	nvlist_t *snapprops;
	VERIFY(0 == nvlist_lookup_nvlist(origin_nv,
	"snapprops", &snapprops));
	VERIFY(0 == nvlist_lookup_nvlist(snapprops,
	snapname, &snapprops));
	VERIFY(0 == nvlist_add_boolean(
	snapprops, "is_clone_origin"));
	}
	}
	}
	again:
	needagain = progress = B_FALSE;
	for (fspair = nvlist_next_nvpair(sdd->fss, NULL); fspair;
	fspair = nvlist_next_nvpair(sdd->fss, fspair)) {
	nvlist_t fslist, parent_nv;
	char *fsname;
	zfs_handle_t *zhp;
	int err;
	uint64_t origin_guid = 0;
	uint64_t parent_guid = 0;

	VERIFY(nvpair_value_nvlist(fspair, &fslist) == 0);
	if (nvlist_lookup_boolean(fslist, "sent") == 0)
	continue;

	VERIFY(nvlist_lookup_string(fslist, "name", &fsname) == 0);
	(void) nvlist_lookup_uint64(fslist, "origin", &origin_guid);
	(void) nvlist_lookup_uint64(fslist, "parentfromsnap",
	&parent_guid);

	if (parent_guid != 0) {
	parent_nv = fsavl_find(sdd->fsavl, parent_guid, NULL);
	if (!nvlist_exists(parent_nv, "sent")) {
	/* parent has not been sent; skip this one */
	needagain = B_TRUE;
	continue;
	}
	}

	if (origin_guid != 0) {
	nvlist_t *origin_nv = fsavl_find(sdd->fsavl,
	origin_guid, NULL);
	if (origin_nv != NULL &&
	!nvlist_exists(origin_nv, "sent")) {
	/*
	* origin has not been sent yet;
	* skip this clone.
	*/
	needagain = B_TRUE;
	continue;
	}
	}

	zhp = zfs_open(rzhp->zfs_hdl, fsname, ZFS_TYPE_DATASET);
	if (zhp == NULL)
	return (-1);
	err = dump_filesystem(zhp, sdd);
	VERIFY(nvlist_add_boolean(fslist, "sent") == 0);
	progress = B_TRUE;
	zfs_close(zhp);
	if (err)
	return (err);
	}
	if (needagain) {
	assert(progress);
	goto again;
	}

	/* clean out the sent flags in case we reuse this fss */
	for (fspair = nvlist_next_nvpair(sdd->fss, NULL); fspair;
	fspair = nvlist_next_nvpair(sdd->fss, fspair)) {
	nvlist_t *fslist;

	VERIFY(nvpair_value_nvlist(fspair, &fslist) == 0);
	(void) nvlist_remove_all(fslist, "sent");
	}

	return (0);
	}

	nvlist_t *
	zfs_send_resume_token_to_nvlist(libzfs_handle_t hdl, const char token)
	{
	unsigned int version;
	int nread, i;
	unsigned long long checksum, packed_len;

	/*
	* Decode token header, which is:
	* <token version>-<checksum of payload>-<uncompressed payload length>
	* Note that the only supported token version is 1.
	*/
	nread = sscanf(token, "%u-%llx-%llx-",
	&version, &checksum, &packed_len);
	if (nread != 3) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"resume token is corrupt (invalid format)"));
	return (NULL);
	}

	if (version != ZFS_SEND_RESUME_TOKEN_VERSION) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"resume token is corrupt (invalid version %u)"),
	version);
	return (NULL);
	}

	/* convert hexadecimal representation to binary */
	token = strrchr(token, '-') + 1;
	int len = strlen(token) / 2;
	unsigned char *compressed = zfs_alloc(hdl, len);
	for (i = 0; i < len; i++) {
	nread = sscanf(token + i * 2, "%2hhx", compressed + i);
	if (nread != 1) {
	free(compressed);
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"resume token is corrupt "
	"(payload is not hex-encoded)"));
	return (NULL);
	}
	}

	/* verify checksum */
	zio_cksum_t cksum;
	fletcher_4_native_varsize(compressed, len, &cksum);
	if (cksum.zc_word[0] != checksum) {
	free(compressed);
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"resume token is corrupt (incorrect checksum)"));
	return (NULL);
	}

	/* uncompress */
	void *packed = zfs_alloc(hdl, packed_len);
	uLongf packed_len_long = packed_len;
	if (uncompress(packed, &packed_len_long, compressed, len) != Z_OK \|\|
	packed_len_long != packed_len) {
	free(packed);
	free(compressed);
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"resume token is corrupt (decompression failed)"));
	return (NULL);
	}

	/* unpack nvlist */
	nvlist_t *nv;
	int error = nvlist_unpack(packed, packed_len, &nv, KM_SLEEP);
	free(packed);
	free(compressed);
	if (error != 0) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"resume token is corrupt (nvlist_unpack failed)"));
	return (NULL);
	}
	return (nv);
	}
	static enum lzc_send_flags
	lzc_flags_from_sendflags(const sendflags_t *flags)
	{
	enum lzc_send_flags lzc_flags = 0;
	if (flags->largeblock)
	lzc_flags \|= LZC_SEND_FLAG_LARGE_BLOCK;
	if (flags->embed_data)
	lzc_flags \|= LZC_SEND_FLAG_EMBED_DATA;
	if (flags->compress)
	lzc_flags \|= LZC_SEND_FLAG_COMPRESS;
	if (flags->raw)
	lzc_flags \|= LZC_SEND_FLAG_RAW;
	if (flags->saved)
	lzc_flags \|= LZC_SEND_FLAG_SAVED;
	return (lzc_flags);
	}

	static int
	estimate_size(zfs_handle_t zhp, const char from, int fd, sendflags_t *flags,
	uint64_t resumeobj, uint64_t resumeoff, uint64_t bytes,
	const char redactbook, char errbuf)
	{
	uint64_t size;
	FILE *fout = flags->dryrun ? stdout : stderr;
	progress_arg_t pa = { 0 };
	int err = 0;
	pthread_t ptid;

	if (flags->progress) {
	pa.pa_zhp = zhp;
	pa.pa_fd = fd;
	pa.pa_parsable = flags->parsable;
	pa.pa_estimate = B_TRUE;
	pa.pa_verbosity = flags->verbosity;

	err = pthread_create(&ptid, NULL,
	send_progress_thread, &pa);
	if (err != 0) {
	zfs_error_aux(zhp->zfs_hdl, strerror(errno));
	return (zfs_error(zhp->zfs_hdl,
	EZFS_THREADCREATEFAILED, errbuf));
	}
	}

	err = lzc_send_space_resume_redacted(zhp->zfs_name, from,
	lzc_flags_from_sendflags(flags), resumeobj, resumeoff, bytes,
	redactbook, fd, &size);

	if (flags->progress) {
	void *status = NULL;
	(void) pthread_cancel(ptid);
	(void) pthread_join(ptid, &status);
	int error = (int)(uintptr_t)status;
	if (error != 0 && status != PTHREAD_CANCELED) {
	char errbuf[1024];
	(void) snprintf(errbuf, sizeof (errbuf),
	dgettext(TEXT_DOMAIN, "progress thread exited "
	"nonzero"));
	return (zfs_standard_error(zhp->zfs_hdl, error,
	errbuf));
	}
	}

	if (err != 0) {
	zfs_error_aux(zhp->zfs_hdl, strerror(err));
	return (zfs_error(zhp->zfs_hdl, EZFS_BADBACKUP,
	errbuf));
	}
	send_print_verbose(fout, zhp->zfs_name, from, size,
	flags->parsable);

	if (flags->parsable) {
	(void) fprintf(fout, "size\t%llu\n", (longlong_t)size);
	} else {
	char buf[16];
	zfs_nicenum(size, buf, sizeof (buf));
	(void) fprintf(fout, dgettext(TEXT_DOMAIN,
	"total estimated size is %s\n"), buf);
	}
	return (0);
	}

	static boolean_t
	redact_snaps_contains(const uint64_t *snaps, uint64_t num_snaps, uint64_t guid)
	{
	for (int i = 0; i < num_snaps; i++) {
	if (snaps[i] == guid)
	return (B_TRUE);
	}
	return (B_FALSE);
	}

	static boolean_t
	redact_snaps_equal(const uint64_t *snaps1, uint64_t num_snaps1,
	const uint64_t *snaps2, uint64_t num_snaps2)
	{
	if (num_snaps1 != num_snaps2)
	return (B_FALSE);
	for (int i = 0; i < num_snaps1; i++) {
	if (!redact_snaps_contains(snaps2, num_snaps2, snaps1[i]))
	return (B_FALSE);
	}
	return (B_TRUE);
	}

	/*
	* Check that the list of redaction snapshots in the bookmark matches the send
	* we're resuming, and return whether or not it's complete.
	*
	* Note that the caller needs to free the contents of *bookname with free() if
	* this function returns successfully.
	*/
	static int
	find_redact_book(libzfs_handle_t hdl, const char path,
	const uint64_t *redact_snap_guids, int num_redact_snaps,
	char **bookname)
	{
	char errbuf[1024];
	int error = 0;
	nvlist_t *props = fnvlist_alloc();
	nvlist_t *bmarks;

	(void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN,
	"cannot resume send"));

	fnvlist_add_boolean(props, "redact_complete");
	fnvlist_add_boolean(props, zfs_prop_to_name(ZFS_PROP_REDACT_SNAPS));
	error = lzc_get_bookmarks(path, props, &bmarks);
	nvlist_free(props);
	if (error != 0) {
	if (error == ESRCH) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"nonexistent redaction bookmark provided"));
	} else if (error == ENOENT) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"dataset to be sent no longer exists"));
	} else {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"unknown error: %s"), strerror(error));
	}
	return (zfs_error(hdl, EZFS_BADPROP, errbuf));
	}
	nvpair_t *pair;
	for (pair = nvlist_next_nvpair(bmarks, NULL); pair;
	pair = nvlist_next_nvpair(bmarks, pair)) {

	nvlist_t *bmark = fnvpair_value_nvlist(pair);
	nvlist_t *vallist = fnvlist_lookup_nvlist(bmark,
	zfs_prop_to_name(ZFS_PROP_REDACT_SNAPS));
	uint_t len = 0;
	uint64_t *bmarksnaps = fnvlist_lookup_uint64_array(vallist,
	ZPROP_VALUE, &len);
	if (redact_snaps_equal(redact_snap_guids,
	num_redact_snaps, bmarksnaps, len)) {
	break;
	}
	}
	if (pair == NULL) {
	fnvlist_free(bmarks);
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"no appropriate redaction bookmark exists"));
	return (zfs_error(hdl, EZFS_BADPROP, errbuf));
	}
	char *name = nvpair_name(pair);
	nvlist_t *bmark = fnvpair_value_nvlist(pair);
	nvlist_t *vallist = fnvlist_lookup_nvlist(bmark, "redact_complete");
	boolean_t complete = fnvlist_lookup_boolean_value(vallist,
	ZPROP_VALUE);
	if (!complete) {
	fnvlist_free(bmarks);
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"incomplete redaction bookmark provided"));
	return (zfs_error(hdl, EZFS_BADPROP, errbuf));
	}
	*bookname = strndup(name, ZFS_MAX_DATASET_NAME_LEN);
	ASSERT3P(*bookname, !=, NULL);
	fnvlist_free(bmarks);
	return (0);
	}

	static int
	zfs_send_resume_impl(libzfs_handle_t hdl, sendflags_t flags, int outfd,
	nvlist_t *resume_nvl)
	{
	char errbuf[1024];
	char *toname;
	char *fromname = NULL;
	uint64_t resumeobj, resumeoff, toguid, fromguid, bytes;
	zfs_handle_t *zhp;
	int error = 0;
	char name[ZFS_MAX_DATASET_NAME_LEN];
	enum lzc_send_flags lzc_flags = 0;
	FILE *fout = (flags->verbosity > 0 && flags->dryrun) ? stdout : stderr;
	uint64_t *redact_snap_guids = NULL;
	int num_redact_snaps = 0;
	char *redact_book = NULL;

	(void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN,
	"cannot resume send"));

	if (flags->verbosity != 0) {
	(void) fprintf(fout, dgettext(TEXT_DOMAIN,
	"resume token contents:\n"));
	nvlist_print(fout, resume_nvl);
	}

	if (nvlist_lookup_string(resume_nvl, "toname", &toname) != 0 \|\|
	nvlist_lookup_uint64(resume_nvl, "object", &resumeobj) != 0 \|\|
	nvlist_lookup_uint64(resume_nvl, "offset", &resumeoff) != 0 \|\|
	nvlist_lookup_uint64(resume_nvl, "bytes", &bytes) != 0 \|\|
	nvlist_lookup_uint64(resume_nvl, "toguid", &toguid) != 0) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"resume token is corrupt"));
	return (zfs_error(hdl, EZFS_FAULT, errbuf));
	}
	fromguid = 0;
	(void) nvlist_lookup_uint64(resume_nvl, "fromguid", &fromguid);

	if (flags->largeblock \|\| nvlist_exists(resume_nvl, "largeblockok"))
	lzc_flags \|= LZC_SEND_FLAG_LARGE_BLOCK;
	if (flags->embed_data \|\| nvlist_exists(resume_nvl, "embedok"))
	lzc_flags \|= LZC_SEND_FLAG_EMBED_DATA;
	if (flags->compress \|\| nvlist_exists(resume_nvl, "compressok"))
	lzc_flags \|= LZC_SEND_FLAG_COMPRESS;
	if (flags->raw \|\| nvlist_exists(resume_nvl, "rawok"))
	lzc_flags \|= LZC_SEND_FLAG_RAW;
	if (flags->saved \|\| nvlist_exists(resume_nvl, "savedok"))
	lzc_flags \|= LZC_SEND_FLAG_SAVED;

	if (flags->saved) {
	(void) strcpy(name, toname);
	} else {
	error = guid_to_name(hdl, toname, toguid, B_FALSE, name);
	if (error != 0) {
	if (zfs_dataset_exists(hdl, toname, ZFS_TYPE_DATASET)) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"'%s' is no longer the same snapshot "
	"used in the initial send"), toname);
	} else {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"'%s' used in the initial send no "
	"longer exists"), toname);
	}
	return (zfs_error(hdl, EZFS_BADPATH, errbuf));
	}
	}

	zhp = zfs_open(hdl, name, ZFS_TYPE_DATASET);
	if (zhp == NULL) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"unable to access '%s'"), name);
	return (zfs_error(hdl, EZFS_BADPATH, errbuf));
	}

	if (nvlist_lookup_uint64_array(resume_nvl, "book_redact_snaps",
	&redact_snap_guids, (uint_t *)&num_redact_snaps) != 0) {
	num_redact_snaps = -1;
	}

	if (fromguid != 0) {
	if (guid_to_name_redact_snaps(hdl, toname, fromguid, B_TRUE,
	redact_snap_guids, num_redact_snaps, name) != 0) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"incremental source %#llx no longer exists"),
	(longlong_t)fromguid);
	return (zfs_error(hdl, EZFS_BADPATH, errbuf));
	}
	fromname = name;
	}

	redact_snap_guids = NULL;

	if (nvlist_lookup_uint64_array(resume_nvl,
	zfs_prop_to_name(ZFS_PROP_REDACT_SNAPS), &redact_snap_guids,
	(uint_t *)&num_redact_snaps) == 0) {
	char path[ZFS_MAX_DATASET_NAME_LEN];

	(void) strlcpy(path, toname, sizeof (path));
	char *at = strchr(path, '@');
	ASSERT3P(at, !=, NULL);

	*at = '\0';

	if ((error = find_redact_book(hdl, path, redact_snap_guids,
	num_redact_snaps, &redact_book)) != 0) {
	return (error);
	}
	}

	if (flags->verbosity != 0) {
	/*
	* Some of these may have come from the resume token, set them
	* here for size estimate purposes.
	*/
	sendflags_t tmpflags = *flags;
	if (lzc_flags & LZC_SEND_FLAG_LARGE_BLOCK)
	tmpflags.largeblock = B_TRUE;
	if (lzc_flags & LZC_SEND_FLAG_COMPRESS)
	tmpflags.compress = B_TRUE;
	if (lzc_flags & LZC_SEND_FLAG_EMBED_DATA)
	tmpflags.embed_data = B_TRUE;
	error = estimate_size(zhp, fromname, outfd, &tmpflags,
	resumeobj, resumeoff, bytes, redact_book, errbuf);
	}

	if (!flags->dryrun) {
	progress_arg_t pa = { 0 };
	pthread_t tid;
	/*
	* If progress reporting is requested, spawn a new thread to
	* poll ZFS_IOC_SEND_PROGRESS at a regular interval.
	*/
	if (flags->progress) {
	pa.pa_zhp = zhp;
	pa.pa_fd = outfd;
	pa.pa_parsable = flags->parsable;
	pa.pa_estimate = B_FALSE;
	pa.pa_verbosity = flags->verbosity;

	error = pthread_create(&tid, NULL,
	send_progress_thread, &pa);
	if (error != 0) {
	if (redact_book != NULL)
	free(redact_book);
	zfs_close(zhp);
	return (error);
	}
	}

	error = lzc_send_resume_redacted(zhp->zfs_name, fromname, outfd,
	lzc_flags, resumeobj, resumeoff, redact_book);
	if (redact_book != NULL)
	free(redact_book);

	if (flags->progress) {
	void *status = NULL;
	(void) pthread_cancel(tid);
	(void) pthread_join(tid, &status);
	int error = (int)(uintptr_t)status;
	if (error != 0 && status != PTHREAD_CANCELED) {
	char errbuf[1024];
	(void) snprintf(errbuf, sizeof (errbuf),
	dgettext(TEXT_DOMAIN,
	"progress thread exited nonzero"));
	return (zfs_standard_error(hdl, error, errbuf));
	}
	}

	char errbuf[1024];
	(void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN,
	"warning: cannot send '%s'"), zhp->zfs_name);

	zfs_close(zhp);

	switch (error) {
	case 0:
	return (0);
	case EACCES:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"source key must be loaded"));
	return (zfs_error(hdl, EZFS_CRYPTOFAILED, errbuf));
	case ESRCH:
	if (lzc_exists(zhp->zfs_name)) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"incremental source could not be found"));
	}
	return (zfs_error(hdl, EZFS_NOENT, errbuf));

	case EXDEV:
	case ENOENT:
	case EDQUOT:
	case EFBIG:
	case EIO:
	case ENOLINK:
	case ENOSPC:
	case ENOSTR:
	case ENXIO:
	case EPIPE:
	case ERANGE:
	case EFAULT:
	case EROFS:
	zfs_error_aux(hdl, strerror(errno));
	return (zfs_error(hdl, EZFS_BADBACKUP, errbuf));

	default:
	return (zfs_standard_error(hdl, errno, errbuf));
	}
	} else {
	if (redact_book != NULL)
	free(redact_book);
	}

	zfs_close(zhp);

	return (error);
	}

	int
	zfs_send_resume(libzfs_handle_t hdl, sendflags_t flags, int outfd,
	const char *resume_token)
	{
	int ret;
	char errbuf[1024];
	nvlist_t *resume_nvl;

	(void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN,
	"cannot resume send"));

	resume_nvl = zfs_send_resume_token_to_nvlist(hdl, resume_token);
	if (resume_nvl == NULL) {
	/*
	* zfs_error_aux has already been set by
	* zfs_send_resume_token_to_nvlist()
	*/
	return (zfs_error(hdl, EZFS_FAULT, errbuf));
	}

	ret = zfs_send_resume_impl(hdl, flags, outfd, resume_nvl);
	nvlist_free(resume_nvl);

	return (ret);
	}

	int
	zfs_send_saved(zfs_handle_t zhp, sendflags_t flags, int outfd,
	const char *resume_token)
	{
	int ret;
	libzfs_handle_t *hdl = zhp->zfs_hdl;
	nvlist_t saved_nvl = NULL, resume_nvl = NULL;
	uint64_t saved_guid = 0, resume_guid = 0;
	uint64_t obj = 0, off = 0, bytes = 0;
	char token_buf[ZFS_MAXPROPLEN];
	char errbuf[1024];

	(void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN,
	"saved send failed"));

	ret = zfs_prop_get(zhp, ZFS_PROP_RECEIVE_RESUME_TOKEN,
	token_buf, sizeof (token_buf), NULL, NULL, 0, B_TRUE);
	if (ret != 0)
	goto out;

	saved_nvl = zfs_send_resume_token_to_nvlist(hdl, token_buf);
	if (saved_nvl == NULL) {
	/*
	* zfs_error_aux has already been set by
	* zfs_send_resume_token_to_nvlist()
	*/
	ret = zfs_error(hdl, EZFS_FAULT, errbuf);
	goto out;
	}

	/*
	* If a resume token is provided we use the object and offset
	* from that instead of the default, which starts from the
	* beginning.
	*/
	if (resume_token != NULL) {
	resume_nvl = zfs_send_resume_token_to_nvlist(hdl,
	resume_token);
	if (resume_nvl == NULL) {
	ret = zfs_error(hdl, EZFS_FAULT, errbuf);
	goto out;
	}

	if (nvlist_lookup_uint64(resume_nvl, "object", &obj) != 0 \|\|
	nvlist_lookup_uint64(resume_nvl, "offset", &off) != 0 \|\|
	nvlist_lookup_uint64(resume_nvl, "bytes", &bytes) != 0 \|\|
	nvlist_lookup_uint64(resume_nvl, "toguid",
	&resume_guid) != 0) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"provided resume token is corrupt"));
	ret = zfs_error(hdl, EZFS_FAULT, errbuf);
	goto out;
	}

	if (nvlist_lookup_uint64(saved_nvl, "toguid",
	&saved_guid)) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"dataset's resume token is corrupt"));
	ret = zfs_error(hdl, EZFS_FAULT, errbuf);
	goto out;
	}

	if (resume_guid != saved_guid) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"provided resume token does not match dataset"));
	ret = zfs_error(hdl, EZFS_BADBACKUP, errbuf);
	goto out;
	}
	}

	(void) nvlist_remove_all(saved_nvl, "object");
	fnvlist_add_uint64(saved_nvl, "object", obj);

	(void) nvlist_remove_all(saved_nvl, "offset");
	fnvlist_add_uint64(saved_nvl, "offset", off);

	(void) nvlist_remove_all(saved_nvl, "bytes");
	fnvlist_add_uint64(saved_nvl, "bytes", bytes);

	(void) nvlist_remove_all(saved_nvl, "toname");
	fnvlist_add_string(saved_nvl, "toname", zhp->zfs_name);

	ret = zfs_send_resume_impl(hdl, flags, outfd, saved_nvl);

	out:
	nvlist_free(saved_nvl);
	nvlist_free(resume_nvl);
	return (ret);
	}

	/*
	* This function informs the target system that the recursive send is complete.
	* The record is also expected in the case of a send -p.
	*/
	static int
	send_conclusion_record(int fd, zio_cksum_t *zc)
	{
	dmu_replay_record_t drr = { 0 };
	drr.drr_type = DRR_END;
	if (zc != NULL)
	drr.drr_u.drr_end.drr_checksum = *zc;
	if (write(fd, &drr, sizeof (drr)) == -1) {
	return (errno);
	}
	return (0);
	}

	/*
	* This function is responsible for sending the records that contain the
	* necessary information for the target system's libzfs to be able to set the
	* properties of the filesystem being received, or to be able to prepare for
	* a recursive receive.
	*
	* The "zhp" argument is the handle of the snapshot we are sending
	* (the "tosnap"). The "from" argument is the short snapshot name (the part
	* after the @) of the incremental source.
	*/
	static int
	send_prelim_records(zfs_handle_t zhp, const char from, int fd,
	boolean_t gather_props, boolean_t recursive, boolean_t verbose,
	boolean_t dryrun, boolean_t raw, boolean_t replicate, boolean_t backup,
	boolean_t holds, boolean_t props, boolean_t doall,
	nvlist_t fssp, avl_tree_t fsavlp)
	{
	int err = 0;
	char *packbuf = NULL;
	size_t buflen = 0;
	zio_cksum_t zc = { {0} };
	int featureflags = 0;
	/* name of filesystem/volume that contains snapshot we are sending */
	char tofs[ZFS_MAX_DATASET_NAME_LEN];
	/* short name of snap we are sending */
	char *tosnap = "";

	char errbuf[1024];
	(void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN,
	"warning: cannot send '%s'"), zhp->zfs_name);
	if (zhp->zfs_type == ZFS_TYPE_FILESYSTEM && zfs_prop_get_int(zhp,
	ZFS_PROP_VERSION) >= ZPL_VERSION_SA) {
	featureflags \|= DMU_BACKUP_FEATURE_SA_SPILL;
	}

	if (holds)
	featureflags \|= DMU_BACKUP_FEATURE_HOLDS;

	(void) strlcpy(tofs, zhp->zfs_name, ZFS_MAX_DATASET_NAME_LEN);
	char *at = strchr(tofs, '@');
	if (at != NULL) {
	*at = '\0';
	tosnap = at + 1;
	}

	if (gather_props) {
	nvlist_t *hdrnv = fnvlist_alloc();
	nvlist_t *fss = NULL;

	if (from != NULL)
	fnvlist_add_string(hdrnv, "fromsnap", from);
	fnvlist_add_string(hdrnv, "tosnap", tosnap);
	if (!recursive)
	fnvlist_add_boolean(hdrnv, "not_recursive");

	if (raw) {
	VERIFY0(nvlist_add_boolean(hdrnv, "raw"));
	}

	if ((err = gather_nvlist(zhp->zfs_hdl, tofs,
	from, tosnap, recursive, raw, doall, replicate, verbose,
	backup, holds, props, &fss, fsavlp)) != 0) {
	return (zfs_error(zhp->zfs_hdl, EZFS_BADBACKUP,
	errbuf));
	}
	/*
	* Do not allow the size of the properties list to exceed
	* the limit
	*/
	if ((fnvlist_size(fss) + fnvlist_size(hdrnv)) >
	zhp->zfs_hdl->libzfs_max_nvlist) {
	(void) snprintf(errbuf, sizeof (errbuf),
	dgettext(TEXT_DOMAIN, "warning: cannot send '%s': "
	"the size of the list of snapshots and properties "
	"is too large to be received successfully.\n"
	"Select a smaller number of snapshots to send.\n"),
	zhp->zfs_name);
	return (zfs_error(zhp->zfs_hdl, EZFS_NOSPC,
	errbuf));
	}
	fnvlist_add_nvlist(hdrnv, "fss", fss);
	VERIFY0(nvlist_pack(hdrnv, &packbuf, &buflen, NV_ENCODE_XDR,
	0));
	if (fssp != NULL) {
	*fssp = fss;
	} else {
	nvlist_free(fss);
	}
	nvlist_free(hdrnv);
	}

	if (!dryrun) {
	dmu_replay_record_t drr = { 0 };
	/* write first begin record */
	drr.drr_type = DRR_BEGIN;
	drr.drr_u.drr_begin.drr_magic = DMU_BACKUP_MAGIC;
	DMU_SET_STREAM_HDRTYPE(drr.drr_u.drr_begin.
	drr_versioninfo, DMU_COMPOUNDSTREAM);
	DMU_SET_FEATUREFLAGS(drr.drr_u.drr_begin.
	drr_versioninfo, featureflags);
	if (snprintf(drr.drr_u.drr_begin.drr_toname,
	sizeof (drr.drr_u.drr_begin.drr_toname), "%s@%s", tofs,
	tosnap) >= sizeof (drr.drr_u.drr_begin.drr_toname)) {
	return (zfs_error(zhp->zfs_hdl, EZFS_BADBACKUP,
	errbuf));
	}
	drr.drr_payloadlen = buflen;

	err = dump_record(&drr, packbuf, buflen, &zc, fd);
	free(packbuf);
	if (err != 0) {
	zfs_error_aux(zhp->zfs_hdl, strerror(err));
	return (zfs_error(zhp->zfs_hdl, EZFS_BADBACKUP,
	errbuf));
	}
	err = send_conclusion_record(fd, &zc);
	if (err != 0) {
	zfs_error_aux(zhp->zfs_hdl, strerror(err));
	return (zfs_error(zhp->zfs_hdl, EZFS_BADBACKUP,
	errbuf));
	}
	}
	return (0);
	}

	/*
	* Generate a send stream. The "zhp" argument is the filesystem/volume
	* that contains the snapshot to send. The "fromsnap" argument is the
	* short name (the part after the '@') of the snapshot that is the
	* incremental source to send from (if non-NULL). The "tosnap" argument
	* is the short name of the snapshot to send.
	*
	* The content of the send stream is the snapshot identified by
	* 'tosnap'. Incremental streams are requested in two ways:
	* - from the snapshot identified by "fromsnap" (if non-null) or
	* - from the origin of the dataset identified by zhp, which must
	* be a clone. In this case, "fromsnap" is null and "fromorigin"
	* is TRUE.
	*
	* The send stream is recursive (i.e. dumps a hierarchy of snapshots) and
	* uses a special header (with a hdrtype field of DMU_COMPOUNDSTREAM)
	* if "replicate" is set. If "doall" is set, dump all the intermediate
	* snapshots. The DMU_COMPOUNDSTREAM header is used in the "doall"
	* case too. If "props" is set, send properties.
	*/
	int
	zfs_send(zfs_handle_t zhp, const char fromsnap, const char *tosnap,
	sendflags_t *flags, int outfd, snapfilter_cb_t filter_func,
	void cb_arg, nvlist_t *debugnvp)
	{
	char errbuf[1024];
	send_dump_data_t sdd = { 0 };
	int err = 0;
	nvlist_t *fss = NULL;
	avl_tree_t *fsavl = NULL;
	static uint64_t holdseq;
	int spa_version;
	- pthread_t tid = 0;
	- int pipefd[2];
	int featureflags = 0;
	FILE *fout;

	(void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN,
	"cannot send '%s'"), zhp->zfs_name);

	if (fromsnap && fromsnap[0] == '\0') {
	zfs_error_aux(zhp->zfs_hdl, dgettext(TEXT_DOMAIN,
	"zero-length incremental source"));
	return (zfs_error(zhp->zfs_hdl, EZFS_NOENT, errbuf));
	}

	if (zhp->zfs_type == ZFS_TYPE_FILESYSTEM) {
	uint64_t version;
	version = zfs_prop_get_int(zhp, ZFS_PROP_VERSION);
	if (version >= ZPL_VERSION_SA) {
	featureflags \|= DMU_BACKUP_FEATURE_SA_SPILL;
	}
	}

	if (flags->holds)
	featureflags \|= DMU_BACKUP_FEATURE_HOLDS;

	if (flags->replicate \|\| flags->doall \|\| flags->props \|\|
	flags->holds \|\| flags->backup) {
	char full_tosnap_name[ZFS_MAX_DATASET_NAME_LEN];
	if (snprintf(full_tosnap_name, sizeof (full_tosnap_name),
	"%s@%s", zhp->zfs_name, tosnap) >=
	sizeof (full_tosnap_name)) {
	err = EINVAL;
	goto stderr_out;
	}
	zfs_handle_t *tosnap = zfs_open(zhp->zfs_hdl,
	full_tosnap_name, ZFS_TYPE_SNAPSHOT);
	if (tosnap == NULL) {
	err = -1;
	goto err_out;
	}
	err = send_prelim_records(tosnap, fromsnap, outfd,
	flags->replicate \|\| flags->props \|\| flags->holds,
	flags->replicate, flags->verbosity > 0, flags->dryrun,
	flags->raw, flags->replicate, flags->backup, flags->holds,
	flags->props, flags->doall, &fss, &fsavl);
	zfs_close(tosnap);
	if (err != 0)
	goto err_out;
	}

	/* dump each stream */
	sdd.fromsnap = fromsnap;
	sdd.tosnap = tosnap;
	- if (tid != 0)
	- sdd.outfd = pipefd[0];
	- else
	- sdd.outfd = outfd;
	+ sdd.outfd = outfd;
	sdd.replicate = flags->replicate;
	sdd.doall = flags->doall;
	sdd.fromorigin = flags->fromorigin;
	sdd.fss = fss;
	sdd.fsavl = fsavl;
	sdd.verbosity = flags->verbosity;
	sdd.parsable = flags->parsable;
	sdd.progress = flags->progress;
	sdd.dryrun = flags->dryrun;
	sdd.large_block = flags->largeblock;
	sdd.embed_data = flags->embed_data;
	sdd.compress = flags->compress;
	sdd.raw = flags->raw;
	sdd.holds = flags->holds;
	sdd.filter_cb = filter_func;
	sdd.filter_cb_arg = cb_arg;
	if (debugnvp)
	sdd.debugnv = *debugnvp;
	if (sdd.verbosity != 0 && sdd.dryrun)
	sdd.std_out = B_TRUE;
	fout = sdd.std_out ? stdout : stderr;

	/*
	* Some flags require that we place user holds on the datasets that are
	* being sent so they don't get destroyed during the send. We can skip
	* this step if the pool is imported read-only since the datasets cannot
	* be destroyed.
	*/
	if (!flags->dryrun && !zpool_get_prop_int(zfs_get_pool_handle(zhp),
	ZPOOL_PROP_READONLY, NULL) &&
	zfs_spa_version(zhp, &spa_version) == 0 &&
	spa_version >= SPA_VERSION_USERREFS &&
	(flags->doall \|\| flags->replicate)) {
	++holdseq;
	(void) snprintf(sdd.holdtag, sizeof (sdd.holdtag),
	".send-%d-%llu", getpid(), (u_longlong_t)holdseq);
	sdd.cleanup_fd = open(ZFS_DEV, O_RDWR);
	if (sdd.cleanup_fd < 0) {
	err = errno;
	goto stderr_out;
	}
	sdd.snapholds = fnvlist_alloc();
	} else {
	sdd.cleanup_fd = -1;
	sdd.snapholds = NULL;
	}

	if (flags->verbosity != 0 \|\| sdd.snapholds != NULL) {
	/*
	* Do a verbose no-op dry run to get all the verbose output
	* or to gather snapshot hold's before generating any data,
	* then do a non-verbose real run to generate the streams.
	*/
	sdd.dryrun = B_TRUE;
	err = dump_filesystems(zhp, &sdd);

	if (err != 0)
	goto stderr_out;

	if (flags->verbosity != 0) {
	if (flags->parsable) {
	(void) fprintf(fout, "size\t%llu\n",
	(longlong_t)sdd.size);
	} else {
	char buf[16];
	zfs_nicebytes(sdd.size, buf, sizeof (buf));
	(void) fprintf(fout, dgettext(TEXT_DOMAIN,
	"total estimated size is %s\n"), buf);
	}
	}

	/* Ensure no snaps found is treated as an error. */
	if (!sdd.seento) {
	err = ENOENT;
	goto err_out;
	}

	/* Skip the second run if dryrun was requested. */
	if (flags->dryrun)
	goto err_out;

	if (sdd.snapholds != NULL) {
	err = zfs_hold_nvl(zhp, sdd.cleanup_fd, sdd.snapholds);
	if (err != 0)
	goto stderr_out;

	fnvlist_free(sdd.snapholds);
	sdd.snapholds = NULL;
	}

	sdd.dryrun = B_FALSE;
	sdd.verbosity = 0;
	}

	err = dump_filesystems(zhp, &sdd);
	fsavl_destroy(fsavl);
	nvlist_free(fss);

	/* Ensure no snaps found is treated as an error. */
	if (err == 0 && !sdd.seento)
	err = ENOENT;

	- if (tid != 0) {
	- if (err != 0)
	- (void) pthread_cancel(tid);
	- (void) close(pipefd[0]);
	- (void) pthread_join(tid, NULL);
	- }
	-
	if (sdd.cleanup_fd != -1) {
	VERIFY(0 == close(sdd.cleanup_fd));
	sdd.cleanup_fd = -1;
	}

	if (!flags->dryrun && (flags->replicate \|\| flags->doall \|\|
	flags->props \|\| flags->backup \|\| flags->holds)) {
	/*
	* write final end record. NB: want to do this even if
	* there was some error, because it might not be totally
	* failed.
	*/
	err = send_conclusion_record(outfd, NULL);
	if (err != 0)
	return (zfs_standard_error(zhp->zfs_hdl, err, errbuf));
	}

	return (err \|\| sdd.err);

	stderr_out:
	err = zfs_standard_error(zhp->zfs_hdl, err, errbuf);
	err_out:
	fsavl_destroy(fsavl);
	nvlist_free(fss);
	fnvlist_free(sdd.snapholds);

	if (sdd.cleanup_fd != -1)
	VERIFY(0 == close(sdd.cleanup_fd));
	- if (tid != 0) {
	- (void) pthread_cancel(tid);
	- (void) close(pipefd[0]);
	- (void) pthread_join(tid, NULL);
	- }
	return (err);
	}

	static zfs_handle_t *
	name_to_dir_handle(libzfs_handle_t hdl, const char snapname)
	{
	char dirname[ZFS_MAX_DATASET_NAME_LEN];
	(void) strlcpy(dirname, snapname, ZFS_MAX_DATASET_NAME_LEN);
	char *c = strchr(dirname, '@');
	if (c != NULL)
	*c = '\0';
	return (zfs_open(hdl, dirname, ZFS_TYPE_DATASET));
	}

	/*
	* Returns B_TRUE if earlier is an earlier snapshot in later's timeline; either
	* an earlier snapshot in the same filesystem, or a snapshot before later's
	* origin, or it's origin's origin, etc.
	*/
	static boolean_t
	snapshot_is_before(zfs_handle_t earlier, zfs_handle_t later)
	{
	boolean_t ret;
	uint64_t later_txg =
	(later->zfs_type == ZFS_TYPE_FILESYSTEM \|\|
	later->zfs_type == ZFS_TYPE_VOLUME ?
	UINT64_MAX : zfs_prop_get_int(later, ZFS_PROP_CREATETXG));
	uint64_t earlier_txg = zfs_prop_get_int(earlier, ZFS_PROP_CREATETXG);

	if (earlier_txg >= later_txg)
	return (B_FALSE);

	zfs_handle_t *earlier_dir = name_to_dir_handle(earlier->zfs_hdl,
	earlier->zfs_name);
	zfs_handle_t *later_dir = name_to_dir_handle(later->zfs_hdl,
	later->zfs_name);

	if (strcmp(earlier_dir->zfs_name, later_dir->zfs_name) == 0) {
	zfs_close(earlier_dir);
	zfs_close(later_dir);
	return (B_TRUE);
	}

	char clonename[ZFS_MAX_DATASET_NAME_LEN];
	if (zfs_prop_get(later_dir, ZFS_PROP_ORIGIN, clonename,
	ZFS_MAX_DATASET_NAME_LEN, NULL, NULL, 0, B_TRUE) != 0) {
	zfs_close(earlier_dir);
	zfs_close(later_dir);
	return (B_FALSE);
	}

	zfs_handle_t *origin = zfs_open(earlier->zfs_hdl, clonename,
	ZFS_TYPE_DATASET);
	uint64_t origin_txg = zfs_prop_get_int(origin, ZFS_PROP_CREATETXG);

	/*
	* If "earlier" is exactly the origin, then
	* snapshot_is_before(earlier, origin) will return false (because
	* they're the same).
	*/
	if (origin_txg == earlier_txg &&
	strcmp(origin->zfs_name, earlier->zfs_name) == 0) {
	zfs_close(earlier_dir);
	zfs_close(later_dir);
	zfs_close(origin);
	return (B_TRUE);
	}
	zfs_close(earlier_dir);
	zfs_close(later_dir);

	ret = snapshot_is_before(earlier, origin);
	zfs_close(origin);
	return (ret);
	}

	/*
	* The "zhp" argument is the handle of the dataset to send (typically a
	* snapshot). The "from" argument is the full name of the snapshot or
	* bookmark that is the incremental source.
	*/
	int
	zfs_send_one(zfs_handle_t zhp, const char from, int fd, sendflags_t *flags,
	const char *redactbook)
	{
	int err;
	libzfs_handle_t *hdl = zhp->zfs_hdl;
	char *name = zhp->zfs_name;
	int orig_fd = fd;
	pthread_t ptid;
	progress_arg_t pa = { 0 };

	char errbuf[1024];
	(void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN,
	"warning: cannot send '%s'"), name);

	if (from != NULL && strchr(from, '@')) {
	zfs_handle_t *from_zhp = zfs_open(hdl, from,
	ZFS_TYPE_DATASET);
	if (from_zhp == NULL)
	return (-1);
	if (!snapshot_is_before(from_zhp, zhp)) {
	zfs_close(from_zhp);
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"not an earlier snapshot from the same fs"));
	return (zfs_error(hdl, EZFS_CROSSTARGET, errbuf));
	}
	zfs_close(from_zhp);
	}

	if (redactbook != NULL) {
	char bookname[ZFS_MAX_DATASET_NAME_LEN];
	nvlist_t *redact_snaps;
	zfs_handle_t *book_zhp;
	char at, pound;
	int dsnamelen;

	pound = strchr(redactbook, '#');
	if (pound != NULL)
	redactbook = pound + 1;
	at = strchr(name, '@');
	if (at == NULL) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"cannot do a redacted send to a filesystem"));
	return (zfs_error(hdl, EZFS_BADTYPE, errbuf));
	}
	dsnamelen = at - name;
	if (snprintf(bookname, sizeof (bookname), "%.*s#%s",
	dsnamelen, name, redactbook)
	>= sizeof (bookname)) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"invalid bookmark name"));
	return (zfs_error(hdl, EZFS_INVALIDNAME, errbuf));
	}
	book_zhp = zfs_open(hdl, bookname, ZFS_TYPE_BOOKMARK);
	if (book_zhp == NULL)
	return (-1);
	if (nvlist_lookup_nvlist(book_zhp->zfs_props,
	zfs_prop_to_name(ZFS_PROP_REDACT_SNAPS),
	&redact_snaps) != 0 \|\| redact_snaps == NULL) {
	zfs_close(book_zhp);
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"not a redaction bookmark"));
	return (zfs_error(hdl, EZFS_BADTYPE, errbuf));
	}
	zfs_close(book_zhp);
	}

	/*
	* Send fs properties
	*/
	if (flags->props \|\| flags->holds \|\| flags->backup) {
	/*
	* Note: the header generated by send_prelim_records()
	* assumes that the incremental source is in the same
	* filesystem/volume as the target (which is a requirement
	* when doing "zfs send -R"). But that isn't always the
	* case here (e.g. send from snap in origin, or send from
	* bookmark). We pass from=NULL, which will omit this
	* information from the prelim records; it isn't used
	* when receiving this type of stream.
	*/
	err = send_prelim_records(zhp, NULL, fd, B_TRUE, B_FALSE,
	flags->verbosity > 0, flags->dryrun, flags->raw,
	flags->replicate, flags->backup, flags->holds,
	flags->props, flags->doall, NULL, NULL);
	if (err != 0)
	return (err);
	}

	/*
	* Perform size estimate if verbose was specified.
	*/
	if (flags->verbosity != 0) {
	err = estimate_size(zhp, from, fd, flags, 0, 0, 0, redactbook,
	errbuf);
	if (err != 0)
	return (err);
	}

	if (flags->dryrun)
	return (0);

	/*
	* If progress reporting is requested, spawn a new thread to poll
	* ZFS_IOC_SEND_PROGRESS at a regular interval.
	*/
	if (flags->progress) {
	pa.pa_zhp = zhp;
	pa.pa_fd = fd;
	pa.pa_parsable = flags->parsable;
	pa.pa_estimate = B_FALSE;
	pa.pa_verbosity = flags->verbosity;

	err = pthread_create(&ptid, NULL,
	send_progress_thread, &pa);
	if (err != 0) {
	zfs_error_aux(zhp->zfs_hdl, strerror(errno));
	return (zfs_error(zhp->zfs_hdl,
	EZFS_THREADCREATEFAILED, errbuf));
	}
	}

	err = lzc_send_redacted(name, from, fd,
	lzc_flags_from_sendflags(flags), redactbook);

	if (flags->progress) {
	void *status = NULL;
	if (err != 0)
	(void) pthread_cancel(ptid);
	(void) pthread_join(ptid, &status);
	int error = (int)(uintptr_t)status;
	if (error != 0 && status != PTHREAD_CANCELED) {
	char errbuf[1024];
	(void) snprintf(errbuf, sizeof (errbuf),
	dgettext(TEXT_DOMAIN, "progress thread exited "
	"nonzero"));
	return (zfs_standard_error(hdl, error, errbuf));
	}
	}

	if (flags->props \|\| flags->holds \|\| flags->backup) {
	/* Write the final end record. */
	err = send_conclusion_record(orig_fd, NULL);
	if (err != 0)
	return (zfs_standard_error(hdl, err, errbuf));
	}
	if (err != 0) {
	switch (errno) {
	case EXDEV:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"not an earlier snapshot from the same fs"));
	return (zfs_error(hdl, EZFS_CROSSTARGET, errbuf));

	case ENOENT:
	case ESRCH:
	if (lzc_exists(name)) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"incremental source (%s) does not exist"),
	from);
	}
	return (zfs_error(hdl, EZFS_NOENT, errbuf));

	case EACCES:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"dataset key must be loaded"));
	return (zfs_error(hdl, EZFS_CRYPTOFAILED, errbuf));

	case EBUSY:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"target is busy; if a filesystem, "
	"it must not be mounted"));
	return (zfs_error(hdl, EZFS_BUSY, errbuf));

	case EDQUOT:
	case EFAULT:
	case EFBIG:
	case EINVAL:
	case EIO:
	case ENOLINK:
	case ENOSPC:
	case ENOSTR:
	case ENXIO:
	case EPIPE:
	case ERANGE:
	case EROFS:
	zfs_error_aux(hdl, strerror(errno));
	return (zfs_error(hdl, EZFS_BADBACKUP, errbuf));

	default:
	return (zfs_standard_error(hdl, errno, errbuf));
	}
	}
	return (err != 0);
	}

	/*
	* Routines specific to "zfs recv"
	*/

	static int
	recv_read(libzfs_handle_t hdl, int fd, void buf, int ilen,
	boolean_t byteswap, zio_cksum_t *zc)
	{
	char *cp = buf;
	int rv;
	int len = ilen;

	do {
	rv = read(fd, cp, len);
	cp += rv;
	len -= rv;
	} while (rv > 0);

	if (rv < 0 \|\| len != 0) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"failed to read from stream"));
	return (zfs_error(hdl, EZFS_BADSTREAM, dgettext(TEXT_DOMAIN,
	"cannot receive")));
	}

	if (zc) {
	if (byteswap)
	fletcher_4_incremental_byteswap(buf, ilen, zc);
	else
	fletcher_4_incremental_native(buf, ilen, zc);
	}
	return (0);
	}

	static int
	recv_read_nvlist(libzfs_handle_t hdl, int fd, int len, nvlist_t *nvp,
	boolean_t byteswap, zio_cksum_t *zc)
	{
	char *buf;
	int err;

	buf = zfs_alloc(hdl, len);
	if (buf == NULL)
	return (ENOMEM);

	if (len > hdl->libzfs_max_nvlist) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "nvlist too large"));
	free(buf);
	return (ENOMEM);
	}

	err = recv_read(hdl, fd, buf, len, byteswap, zc);
	if (err != 0) {
	free(buf);
	return (err);
	}

	err = nvlist_unpack(buf, len, nvp, 0);
	free(buf);
	if (err != 0) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "invalid "
	"stream (malformed nvlist)"));
	return (EINVAL);
	}
	return (0);
	}

	/*
	* Returns the grand origin (origin of origin of origin...) of a given handle.
	* If this dataset is not a clone, it simply returns a copy of the original
	* handle.
	*/
	static zfs_handle_t *
	recv_open_grand_origin(zfs_handle_t *zhp)
	{
	char origin[ZFS_MAX_DATASET_NAME_LEN];
	zprop_source_t src;
	zfs_handle_t *ozhp = zfs_handle_dup(zhp);

	while (ozhp != NULL) {
	if (zfs_prop_get(ozhp, ZFS_PROP_ORIGIN, origin,
	sizeof (origin), &src, NULL, 0, B_FALSE) != 0)
	break;

	(void) zfs_close(ozhp);
	ozhp = zfs_open(zhp->zfs_hdl, origin, ZFS_TYPE_FILESYSTEM);
	}

	return (ozhp);
	}

	static int
	recv_rename_impl(zfs_handle_t zhp, const char name, const char *newname)
	{
	int err;
	zfs_handle_t *ozhp = NULL;

	/*
	* Attempt to rename the dataset. If it fails with EACCES we have
	* attempted to rename the dataset outside of its encryption root.
	* Force the dataset to become an encryption root and try again.
	*/
	err = lzc_rename(name, newname);
	if (err == EACCES) {
	ozhp = recv_open_grand_origin(zhp);
	if (ozhp == NULL) {
	err = ENOENT;
	goto out;
	}

	err = lzc_change_key(ozhp->zfs_name, DCP_CMD_FORCE_NEW_KEY,
	NULL, NULL, 0);
	if (err != 0)
	goto out;

	err = lzc_rename(name, newname);
	}

	out:
	if (ozhp != NULL)
	zfs_close(ozhp);
	return (err);
	}

	static int
	recv_rename(libzfs_handle_t hdl, const char name, const char *tryname,
	int baselen, char newname, recvflags_t flags)
	{
	static int seq;
	int err;
	prop_changelist_t *clp = NULL;
	zfs_handle_t *zhp = NULL;

	zhp = zfs_open(hdl, name, ZFS_TYPE_DATASET);
	if (zhp == NULL) {
	err = -1;
	goto out;
	}
	clp = changelist_gather(zhp, ZFS_PROP_NAME, 0,
	flags->force ? MS_FORCE : 0);
	if (clp == NULL) {
	err = -1;
	goto out;
	}
	err = changelist_prefix(clp);
	if (err)
	goto out;

	if (tryname) {
	(void) strcpy(newname, tryname);
	if (flags->verbose) {
	(void) printf("attempting rename %s to %s\n",
	name, newname);
	}
	err = recv_rename_impl(zhp, name, newname);
	if (err == 0)
	changelist_rename(clp, name, tryname);
	} else {
	err = ENOENT;
	}

	if (err != 0 && strncmp(name + baselen, "recv-", 5) != 0) {
	seq++;

	(void) snprintf(newname, ZFS_MAX_DATASET_NAME_LEN,
	"%.*srecv-%u-%u", baselen, name, getpid(), seq);

	if (flags->verbose) {
	(void) printf("failed - trying rename %s to %s\n",
	name, newname);
	}
	err = recv_rename_impl(zhp, name, newname);
	if (err == 0)
	changelist_rename(clp, name, newname);
	if (err && flags->verbose) {
	(void) printf("failed (%u) - "
	"will try again on next pass\n", errno);
	}
	err = EAGAIN;
	} else if (flags->verbose) {
	if (err == 0)
	(void) printf("success\n");
	else
	(void) printf("failed (%u)\n", errno);
	}

	(void) changelist_postfix(clp);

	out:
	if (clp != NULL)
	changelist_free(clp);
	if (zhp != NULL)
	zfs_close(zhp);

	return (err);
	}

	static int
	recv_promote(libzfs_handle_t hdl, const char fsname,
	const char origin_fsname, recvflags_t flags)
	{
	int err;
	zfs_cmd_t zc = {"\0"};
	zfs_handle_t zhp = NULL, ozhp = NULL;

	if (flags->verbose)
	(void) printf("promoting %s\n", fsname);

	(void) strlcpy(zc.zc_value, origin_fsname, sizeof (zc.zc_value));
	(void) strlcpy(zc.zc_name, fsname, sizeof (zc.zc_name));

	/*
	* Attempt to promote the dataset. If it fails with EACCES the
	* promotion would cause this dataset to leave its encryption root.
	* Force the origin to become an encryption root and try again.
	*/
	err = zfs_ioctl(hdl, ZFS_IOC_PROMOTE, &zc);
	if (err == EACCES) {
	zhp = zfs_open(hdl, fsname, ZFS_TYPE_DATASET);
	if (zhp == NULL) {
	err = -1;
	goto out;
	}

	ozhp = recv_open_grand_origin(zhp);
	if (ozhp == NULL) {
	err = -1;
	goto out;
	}

	err = lzc_change_key(ozhp->zfs_name, DCP_CMD_FORCE_NEW_KEY,
	NULL, NULL, 0);
	if (err != 0)
	goto out;

	err = zfs_ioctl(hdl, ZFS_IOC_PROMOTE, &zc);
	}

	out:
	if (zhp != NULL)
	zfs_close(zhp);
	if (ozhp != NULL)
	zfs_close(ozhp);

	return (err);
	}

	static int
	recv_destroy(libzfs_handle_t hdl, const char name, int baselen,
	char newname, recvflags_t flags)
	{
	int err = 0;
	prop_changelist_t *clp;
	zfs_handle_t *zhp;
	boolean_t defer = B_FALSE;
	int spa_version;

	zhp = zfs_open(hdl, name, ZFS_TYPE_DATASET);
	if (zhp == NULL)
	return (-1);
	clp = changelist_gather(zhp, ZFS_PROP_NAME, 0,
	flags->force ? MS_FORCE : 0);
	if (zfs_get_type(zhp) == ZFS_TYPE_SNAPSHOT &&
	zfs_spa_version(zhp, &spa_version) == 0 &&
	spa_version >= SPA_VERSION_USERREFS)
	defer = B_TRUE;
	zfs_close(zhp);
	if (clp == NULL)
	return (-1);
	err = changelist_prefix(clp);
	if (err)
	return (err);

	if (flags->verbose)
	(void) printf("attempting destroy %s\n", name);
	if (zhp->zfs_type == ZFS_TYPE_SNAPSHOT) {
	nvlist_t *nv = fnvlist_alloc();
	fnvlist_add_boolean(nv, name);
	err = lzc_destroy_snaps(nv, defer, NULL);
	fnvlist_free(nv);
	} else {
	err = lzc_destroy(name);
	}
	if (err == 0) {
	if (flags->verbose)
	(void) printf("success\n");
	changelist_remove(clp, name);
	}

	(void) changelist_postfix(clp);
	changelist_free(clp);

	/*
	* Deferred destroy might destroy the snapshot or only mark it to be
	* destroyed later, and it returns success in either case.
	*/
	if (err != 0 \|\| (defer && zfs_dataset_exists(hdl, name,
	ZFS_TYPE_SNAPSHOT))) {
	err = recv_rename(hdl, name, NULL, baselen, newname, flags);
	}

	return (err);
	}

	typedef struct guid_to_name_data {
	uint64_t guid;
	boolean_t bookmark_ok;
	char *name;
	char *skip;
	uint64_t *redact_snap_guids;
	uint64_t num_redact_snaps;
	} guid_to_name_data_t;

	static boolean_t
	redact_snaps_match(zfs_handle_t zhp, guid_to_name_data_t gtnd)
	{
	uint64_t *bmark_snaps;
	uint_t bmark_num_snaps;
	nvlist_t *nvl;
	if (zhp->zfs_type != ZFS_TYPE_BOOKMARK)
	return (B_FALSE);

	nvl = fnvlist_lookup_nvlist(zhp->zfs_props,
	zfs_prop_to_name(ZFS_PROP_REDACT_SNAPS));
	bmark_snaps = fnvlist_lookup_uint64_array(nvl, ZPROP_VALUE,
	&bmark_num_snaps);
	if (bmark_num_snaps != gtnd->num_redact_snaps)
	return (B_FALSE);
	int i = 0;
	for (; i < bmark_num_snaps; i++) {
	int j = 0;
	for (; j < bmark_num_snaps; j++) {
	if (bmark_snaps[i] == gtnd->redact_snap_guids[j])
	break;
	}
	if (j == bmark_num_snaps)
	break;
	}
	return (i == bmark_num_snaps);
	}

	static int
	guid_to_name_cb(zfs_handle_t zhp, void arg)
	{
	guid_to_name_data_t *gtnd = arg;
	const char *slash;
	int err;

	if (gtnd->skip != NULL &&
	(slash = strrchr(zhp->zfs_name, '/')) != NULL &&
	strcmp(slash + 1, gtnd->skip) == 0) {
	zfs_close(zhp);
	return (0);
	}

	if (zfs_prop_get_int(zhp, ZFS_PROP_GUID) == gtnd->guid &&
	(gtnd->num_redact_snaps == -1 \|\| redact_snaps_match(zhp, gtnd))) {
	(void) strcpy(gtnd->name, zhp->zfs_name);
	zfs_close(zhp);
	return (EEXIST);
	}

	err = zfs_iter_children(zhp, guid_to_name_cb, gtnd);
	if (err != EEXIST && gtnd->bookmark_ok)
	err = zfs_iter_bookmarks(zhp, guid_to_name_cb, gtnd);
	zfs_close(zhp);
	return (err);
	}

	/*
	* Attempt to find the local dataset associated with this guid. In the case of
	* multiple matches, we attempt to find the "best" match by searching
	* progressively larger portions of the hierarchy. This allows one to send a
	* tree of datasets individually and guarantee that we will find the source
	* guid within that hierarchy, even if there are multiple matches elsewhere.
	*
	* If num_redact_snaps is not -1, we attempt to find a redaction bookmark with
	* the specified number of redaction snapshots. If num_redact_snaps isn't 0 or
	* -1, then redact_snap_guids will be an array of the guids of the snapshots the
	* redaction bookmark was created with. If num_redact_snaps is -1, then we will
	* attempt to find a snapshot or bookmark (if bookmark_ok is passed) with the
	* given guid. Note that a redaction bookmark can be returned if
	* num_redact_snaps == -1.
	*/
	static int
	guid_to_name_redact_snaps(libzfs_handle_t hdl, const char parent,
	uint64_t guid, boolean_t bookmark_ok, uint64_t *redact_snap_guids,
	uint64_t num_redact_snaps, char *name)
	{
	char pname[ZFS_MAX_DATASET_NAME_LEN];
	guid_to_name_data_t gtnd;

	gtnd.guid = guid;
	gtnd.bookmark_ok = bookmark_ok;
	gtnd.name = name;
	gtnd.skip = NULL;
	gtnd.redact_snap_guids = redact_snap_guids;
	gtnd.num_redact_snaps = num_redact_snaps;

	/*
	* Search progressively larger portions of the hierarchy, starting
	* with the filesystem specified by 'parent'. This will
	* select the "most local" version of the origin snapshot in the case
	* that there are multiple matching snapshots in the system.
	*/
	(void) strlcpy(pname, parent, sizeof (pname));
	char *cp = strrchr(pname, '@');
	if (cp == NULL)
	cp = strchr(pname, '\0');
	for (; cp != NULL; cp = strrchr(pname, '/')) {
	/* Chop off the last component and open the parent */
	*cp = '\0';
	zfs_handle_t *zhp = make_dataset_handle(hdl, pname);

	if (zhp == NULL)
	continue;
	int err = guid_to_name_cb(zfs_handle_dup(zhp), &gtnd);
	if (err != EEXIST)
	err = zfs_iter_children(zhp, guid_to_name_cb, &gtnd);
	if (err != EEXIST && bookmark_ok)
	err = zfs_iter_bookmarks(zhp, guid_to_name_cb, &gtnd);
	zfs_close(zhp);
	if (err == EEXIST)
	return (0);

	/*
	* Remember the last portion of the dataset so we skip it next
	* time through (as we've already searched that portion of the
	* hierarchy).
	*/
	gtnd.skip = strrchr(pname, '/') + 1;
	}

	return (ENOENT);
	}

	static int
	guid_to_name(libzfs_handle_t hdl, const char parent, uint64_t guid,
	boolean_t bookmark_ok, char *name)
	{
	return (guid_to_name_redact_snaps(hdl, parent, guid, bookmark_ok, NULL,
	-1, name));
	}

	/*
	* Return +1 if guid1 is before guid2, 0 if they are the same, and -1 if
	* guid1 is after guid2.
	*/
	static int
	created_before(libzfs_handle_t hdl, avl_tree_t avl,
	uint64_t guid1, uint64_t guid2)
	{
	nvlist_t *nvfs;
	char fsname = NULL, snapname = NULL;
	char buf[ZFS_MAX_DATASET_NAME_LEN];
	int rv;
	zfs_handle_t guid1hdl, guid2hdl;
	uint64_t create1, create2;

	if (guid2 == 0)
	return (0);
	if (guid1 == 0)
	return (1);

	nvfs = fsavl_find(avl, guid1, &snapname);
	VERIFY(0 == nvlist_lookup_string(nvfs, "name", &fsname));
	(void) snprintf(buf, sizeof (buf), "%s@%s", fsname, snapname);
	guid1hdl = zfs_open(hdl, buf, ZFS_TYPE_SNAPSHOT);
	if (guid1hdl == NULL)
	return (-1);

	nvfs = fsavl_find(avl, guid2, &snapname);
	VERIFY(0 == nvlist_lookup_string(nvfs, "name", &fsname));
	(void) snprintf(buf, sizeof (buf), "%s@%s", fsname, snapname);
	guid2hdl = zfs_open(hdl, buf, ZFS_TYPE_SNAPSHOT);
	if (guid2hdl == NULL) {
	zfs_close(guid1hdl);
	return (-1);
	}

	create1 = zfs_prop_get_int(guid1hdl, ZFS_PROP_CREATETXG);
	create2 = zfs_prop_get_int(guid2hdl, ZFS_PROP_CREATETXG);

	if (create1 < create2)
	rv = -1;
	else if (create1 > create2)
	rv = +1;
	else
	rv = 0;

	zfs_close(guid1hdl);
	zfs_close(guid2hdl);

	return (rv);
	}

	/*
	* This function reestablishes the hierarchy of encryption roots after a
	* recursive incremental receive has completed. This must be done after the
	* second call to recv_incremental_replication() has renamed and promoted all
	* sent datasets to their final locations in the dataset hierarchy.
	*/
	static int
	recv_fix_encryption_hierarchy(libzfs_handle_t hdl, const char top_zfs,
	nvlist_t stream_nv, avl_tree_t stream_avl)
	{
	int err;
	nvpair_t *fselem = NULL;
	nvlist_t *stream_fss;

	VERIFY(0 == nvlist_lookup_nvlist(stream_nv, "fss", &stream_fss));

	while ((fselem = nvlist_next_nvpair(stream_fss, fselem)) != NULL) {
	zfs_handle_t *zhp = NULL;
	uint64_t crypt;
	nvlist_t snaps, props, *stream_nvfs = NULL;
	nvpair_t *snapel = NULL;
	boolean_t is_encroot, is_clone, stream_encroot;
	char *cp;
	char *stream_keylocation = NULL;
	char keylocation[MAXNAMELEN];
	char fsname[ZFS_MAX_DATASET_NAME_LEN];

	keylocation[0] = '\0';
	VERIFY(0 == nvpair_value_nvlist(fselem, &stream_nvfs));
	VERIFY(0 == nvlist_lookup_nvlist(stream_nvfs, "snaps", &snaps));
	VERIFY(0 == nvlist_lookup_nvlist(stream_nvfs, "props", &props));
	stream_encroot = nvlist_exists(stream_nvfs, "is_encroot");

	/* find a snapshot from the stream that exists locally */
	err = ENOENT;
	while ((snapel = nvlist_next_nvpair(snaps, snapel)) != NULL) {
	uint64_t guid;

	VERIFY(0 == nvpair_value_uint64(snapel, &guid));
	err = guid_to_name(hdl, top_zfs, guid, B_FALSE,
	fsname);
	if (err == 0)
	break;
	}

	if (err != 0)
	continue;

	cp = strchr(fsname, '@');
	if (cp != NULL)
	*cp = '\0';

	zhp = zfs_open(hdl, fsname, ZFS_TYPE_DATASET);
	if (zhp == NULL) {
	err = ENOENT;
	goto error;
	}

	crypt = zfs_prop_get_int(zhp, ZFS_PROP_ENCRYPTION);
	is_clone = zhp->zfs_dmustats.dds_origin[0] != '\0';
	(void) zfs_crypto_get_encryption_root(zhp, &is_encroot, NULL);

	/* we don't need to do anything for unencrypted datasets */
	if (crypt == ZIO_CRYPT_OFF) {
	zfs_close(zhp);
	continue;
	}

	/*
	* If the dataset is flagged as an encryption root, was not
	* received as a clone and is not currently an encryption root,
	* force it to become one. Fixup the keylocation if necessary.
	*/
	if (stream_encroot) {
	if (!is_clone && !is_encroot) {
	err = lzc_change_key(fsname,
	DCP_CMD_FORCE_NEW_KEY, NULL, NULL, 0);
	if (err != 0) {
	zfs_close(zhp);
	goto error;
	}
	}

	VERIFY(0 == nvlist_lookup_string(props,
	zfs_prop_to_name(ZFS_PROP_KEYLOCATION),
	&stream_keylocation));

	/*
	* Refresh the properties in case the call to
	* lzc_change_key() changed the value.
	*/
	zfs_refresh_properties(zhp);
	err = zfs_prop_get(zhp, ZFS_PROP_KEYLOCATION,
	keylocation, sizeof (keylocation), NULL, NULL,
	0, B_TRUE);
	if (err != 0) {
	zfs_close(zhp);
	goto error;
	}

	if (strcmp(keylocation, stream_keylocation) != 0) {
	err = zfs_prop_set(zhp,
	zfs_prop_to_name(ZFS_PROP_KEYLOCATION),
	stream_keylocation);
	if (err != 0) {
	zfs_close(zhp);
	goto error;
	}
	}
	}

	/*
	* If the dataset is not flagged as an encryption root and is
	* currently an encryption root, force it to inherit from its
	* parent. The root of a raw send should never be
	* force-inherited.
	*/
	if (!stream_encroot && is_encroot &&
	strcmp(top_zfs, fsname) != 0) {
	err = lzc_change_key(fsname, DCP_CMD_FORCE_INHERIT,
	NULL, NULL, 0);
	if (err != 0) {
	zfs_close(zhp);
	goto error;
	}
	}

	zfs_close(zhp);
	}

	return (0);

	error:
	return (err);
	}

	static int
	recv_incremental_replication(libzfs_handle_t hdl, const char tofs,
	recvflags_t flags, nvlist_t stream_nv, avl_tree_t *stream_avl,
	nvlist_t *renamed)
	{
	nvlist_t local_nv, deleted = NULL;
	avl_tree_t *local_avl;
	nvpair_t fselem, nextfselem;
	char *fromsnap;
	char newname[ZFS_MAX_DATASET_NAME_LEN];
	char guidname[32];
	int error;
	boolean_t needagain, progress, recursive;
	char s1, s2;

	VERIFY(0 == nvlist_lookup_string(stream_nv, "fromsnap", &fromsnap));

	recursive = (nvlist_lookup_boolean(stream_nv, "not_recursive") ==
	ENOENT);

	if (flags->dryrun)
	return (0);

	again:
	needagain = progress = B_FALSE;

	VERIFY(0 == nvlist_alloc(&deleted, NV_UNIQUE_NAME, 0));

	if ((error = gather_nvlist(hdl, tofs, fromsnap, NULL,
	recursive, B_TRUE, B_FALSE, recursive, B_FALSE, B_FALSE,
	B_FALSE, B_TRUE, &local_nv, &local_avl)) != 0)
	return (error);

	/*
	* Process deletes and renames
	*/
	for (fselem = nvlist_next_nvpair(local_nv, NULL);
	fselem; fselem = nextfselem) {
	nvlist_t nvfs, snaps;
	nvlist_t *stream_nvfs = NULL;
	nvpair_t snapelem, nextsnapelem;
	uint64_t fromguid = 0;
	uint64_t originguid = 0;
	uint64_t stream_originguid = 0;
	uint64_t parent_fromsnap_guid, stream_parent_fromsnap_guid;
	char fsname, stream_fsname;

	nextfselem = nvlist_next_nvpair(local_nv, fselem);

	VERIFY(0 == nvpair_value_nvlist(fselem, &nvfs));
	VERIFY(0 == nvlist_lookup_nvlist(nvfs, "snaps", &snaps));
	VERIFY(0 == nvlist_lookup_string(nvfs, "name", &fsname));
	VERIFY(0 == nvlist_lookup_uint64(nvfs, "parentfromsnap",
	&parent_fromsnap_guid));
	(void) nvlist_lookup_uint64(nvfs, "origin", &originguid);

	/*
	* First find the stream's fs, so we can check for
	* a different origin (due to "zfs promote")
	*/
	for (snapelem = nvlist_next_nvpair(snaps, NULL);
	snapelem; snapelem = nvlist_next_nvpair(snaps, snapelem)) {
	uint64_t thisguid;

	VERIFY(0 == nvpair_value_uint64(snapelem, &thisguid));
	stream_nvfs = fsavl_find(stream_avl, thisguid, NULL);

	if (stream_nvfs != NULL)
	break;
	}

	/* check for promote */
	(void) nvlist_lookup_uint64(stream_nvfs, "origin",
	&stream_originguid);
	if (stream_nvfs && originguid != stream_originguid) {
	switch (created_before(hdl, local_avl,
	stream_originguid, originguid)) {
	case 1: {
	/* promote it! */
	nvlist_t *origin_nvfs;
	char *origin_fsname;

	origin_nvfs = fsavl_find(local_avl, originguid,
	NULL);
	VERIFY(0 == nvlist_lookup_string(origin_nvfs,
	"name", &origin_fsname));
	error = recv_promote(hdl, fsname, origin_fsname,
	flags);
	if (error == 0)
	progress = B_TRUE;
	break;
	}
	default:
	break;
	case -1:
	fsavl_destroy(local_avl);
	nvlist_free(local_nv);
	return (-1);
	}
	/*
	* We had/have the wrong origin, therefore our
	* list of snapshots is wrong. Need to handle
	* them on the next pass.
	*/
	needagain = B_TRUE;
	continue;
	}

	for (snapelem = nvlist_next_nvpair(snaps, NULL);
	snapelem; snapelem = nextsnapelem) {
	uint64_t thisguid;
	char *stream_snapname;
	nvlist_t found, props;

	nextsnapelem = nvlist_next_nvpair(snaps, snapelem);

	VERIFY(0 == nvpair_value_uint64(snapelem, &thisguid));
	found = fsavl_find(stream_avl, thisguid,
	&stream_snapname);

	/* check for delete */
	if (found == NULL) {
	char name[ZFS_MAX_DATASET_NAME_LEN];

	if (!flags->force)
	continue;

	(void) snprintf(name, sizeof (name), "%s@%s",
	fsname, nvpair_name(snapelem));

	error = recv_destroy(hdl, name,
	strlen(fsname)+1, newname, flags);
	if (error)
	needagain = B_TRUE;
	else
	progress = B_TRUE;
	sprintf(guidname, "%llu",
	(u_longlong_t)thisguid);
	nvlist_add_boolean(deleted, guidname);
	continue;
	}

	stream_nvfs = found;

	if (0 == nvlist_lookup_nvlist(stream_nvfs, "snapprops",
	&props) && 0 == nvlist_lookup_nvlist(props,
	stream_snapname, &props)) {
	zfs_cmd_t zc = {"\0"};

	zc.zc_cookie = B_TRUE; /* received */
	(void) snprintf(zc.zc_name, sizeof (zc.zc_name),
	"%s@%s", fsname, nvpair_name(snapelem));
	if (zcmd_write_src_nvlist(hdl, &zc,
	props) == 0) {
	(void) zfs_ioctl(hdl,
	ZFS_IOC_SET_PROP, &zc);
	zcmd_free_nvlists(&zc);
	}
	}

	/* check for different snapname */
	if (strcmp(nvpair_name(snapelem),
	stream_snapname) != 0) {
	char name[ZFS_MAX_DATASET_NAME_LEN];
	char tryname[ZFS_MAX_DATASET_NAME_LEN];

	(void) snprintf(name, sizeof (name), "%s@%s",
	fsname, nvpair_name(snapelem));
	(void) snprintf(tryname, sizeof (name), "%s@%s",
	fsname, stream_snapname);

	error = recv_rename(hdl, name, tryname,
	strlen(fsname)+1, newname, flags);
	if (error)
	needagain = B_TRUE;
	else
	progress = B_TRUE;
	}

	if (strcmp(stream_snapname, fromsnap) == 0)
	fromguid = thisguid;
	}

	/* check for delete */
	if (stream_nvfs == NULL) {
	if (!flags->force)
	continue;

	error = recv_destroy(hdl, fsname, strlen(tofs)+1,
	newname, flags);
	if (error)
	needagain = B_TRUE;
	else
	progress = B_TRUE;
	sprintf(guidname, "%llu",
	(u_longlong_t)parent_fromsnap_guid);
	nvlist_add_boolean(deleted, guidname);
	continue;
	}

	if (fromguid == 0) {
	if (flags->verbose) {
	(void) printf("local fs %s does not have "
	"fromsnap (%s in stream); must have "
	"been deleted locally; ignoring\n",
	fsname, fromsnap);
	}
	continue;
	}

	VERIFY(0 == nvlist_lookup_string(stream_nvfs,
	"name", &stream_fsname));
	VERIFY(0 == nvlist_lookup_uint64(stream_nvfs,
	"parentfromsnap", &stream_parent_fromsnap_guid));

	s1 = strrchr(fsname, '/');
	s2 = strrchr(stream_fsname, '/');

	/*
	* Check if we're going to rename based on parent guid change
	* and the current parent guid was also deleted. If it was then
	* rename will fail and is likely unneeded, so avoid this and
	* force an early retry to determine the new
	* parent_fromsnap_guid.
	*/
	if (stream_parent_fromsnap_guid != 0 &&
	parent_fromsnap_guid != 0 &&
	stream_parent_fromsnap_guid != parent_fromsnap_guid) {
	sprintf(guidname, "%llu",
	(u_longlong_t)parent_fromsnap_guid);
	if (nvlist_exists(deleted, guidname)) {
	progress = B_TRUE;
	needagain = B_TRUE;
	goto doagain;
	}
	}

	/*
	* Check for rename. If the exact receive path is specified, it
	* does not count as a rename, but we still need to check the
	* datasets beneath it.
	*/
	if ((stream_parent_fromsnap_guid != 0 &&
	parent_fromsnap_guid != 0 &&
	stream_parent_fromsnap_guid != parent_fromsnap_guid) \|\|
	((flags->isprefix \|\| strcmp(tofs, fsname) != 0) &&
	(s1 != NULL) && (s2 != NULL) && strcmp(s1, s2) != 0)) {
	nvlist_t *parent;
	char tryname[ZFS_MAX_DATASET_NAME_LEN];

	parent = fsavl_find(local_avl,
	stream_parent_fromsnap_guid, NULL);
	/*
	* NB: parent might not be found if we used the
	* tosnap for stream_parent_fromsnap_guid,
	* because the parent is a newly-created fs;
	* we'll be able to rename it after we recv the
	* new fs.
	*/
	if (parent != NULL) {
	char *pname;

	VERIFY(0 == nvlist_lookup_string(parent, "name",
	&pname));
	(void) snprintf(tryname, sizeof (tryname),
	"%s%s", pname, strrchr(stream_fsname, '/'));
	} else {
	tryname[0] = '\0';
	if (flags->verbose) {
	(void) printf("local fs %s new parent "
	"not found\n", fsname);
	}
	}

	newname[0] = '\0';

	error = recv_rename(hdl, fsname, tryname,
	strlen(tofs)+1, newname, flags);

	if (renamed != NULL && newname[0] != '\0') {
	VERIFY(0 == nvlist_add_boolean(renamed,
	newname));
	}

	if (error)
	needagain = B_TRUE;
	else
	progress = B_TRUE;
	}
	}

	doagain:
	fsavl_destroy(local_avl);
	nvlist_free(local_nv);
	nvlist_free(deleted);

	if (needagain && progress) {
	/* do another pass to fix up temporary names */
	if (flags->verbose)
	(void) printf("another pass:\n");
	goto again;
	}

	return (needagain \|\| error != 0);
	}

	static int
	zfs_receive_package(libzfs_handle_t hdl, int fd, const char destname,
	recvflags_t flags, dmu_replay_record_t drr, zio_cksum_t *zc,
	char *top_zfs, nvlist_t cmdprops)
	{
	nvlist_t *stream_nv = NULL;
	avl_tree_t *stream_avl = NULL;
	char *fromsnap = NULL;
	char *sendsnap = NULL;
	char *cp;
	char tofs[ZFS_MAX_DATASET_NAME_LEN];
	char sendfs[ZFS_MAX_DATASET_NAME_LEN];
	char errbuf[1024];
	dmu_replay_record_t drre;
	int error;
	boolean_t anyerr = B_FALSE;
	boolean_t softerr = B_FALSE;
	boolean_t recursive, raw;

	(void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN,
	"cannot receive"));

	assert(drr->drr_type == DRR_BEGIN);
	assert(drr->drr_u.drr_begin.drr_magic == DMU_BACKUP_MAGIC);
	assert(DMU_GET_STREAM_HDRTYPE(drr->drr_u.drr_begin.drr_versioninfo) ==
	DMU_COMPOUNDSTREAM);

	/*
	* Read in the nvlist from the stream.
	*/
	if (drr->drr_payloadlen != 0) {
	error = recv_read_nvlist(hdl, fd, drr->drr_payloadlen,
	&stream_nv, flags->byteswap, zc);
	if (error) {
	error = zfs_error(hdl, EZFS_BADSTREAM, errbuf);
	goto out;
	}
	}

	recursive = (nvlist_lookup_boolean(stream_nv, "not_recursive") ==
	ENOENT);
	raw = (nvlist_lookup_boolean(stream_nv, "raw") == 0);

	if (recursive && strchr(destname, '@')) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"cannot specify snapshot name for multi-snapshot stream"));
	error = zfs_error(hdl, EZFS_BADSTREAM, errbuf);
	goto out;
	}

	/*
	* Read in the end record and verify checksum.
	*/
	if (0 != (error = recv_read(hdl, fd, &drre, sizeof (drre),
	flags->byteswap, NULL)))
	goto out;
	if (flags->byteswap) {
	drre.drr_type = BSWAP_32(drre.drr_type);
	drre.drr_u.drr_end.drr_checksum.zc_word[0] =
	BSWAP_64(drre.drr_u.drr_end.drr_checksum.zc_word[0]);
	drre.drr_u.drr_end.drr_checksum.zc_word[1] =
	BSWAP_64(drre.drr_u.drr_end.drr_checksum.zc_word[1]);
	drre.drr_u.drr_end.drr_checksum.zc_word[2] =
	BSWAP_64(drre.drr_u.drr_end.drr_checksum.zc_word[2]);
	drre.drr_u.drr_end.drr_checksum.zc_word[3] =
	BSWAP_64(drre.drr_u.drr_end.drr_checksum.zc_word[3]);
	}
	if (drre.drr_type != DRR_END) {
	error = zfs_error(hdl, EZFS_BADSTREAM, errbuf);
	goto out;
	}
	if (!ZIO_CHECKSUM_EQUAL(drre.drr_u.drr_end.drr_checksum, *zc)) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"incorrect header checksum"));
	error = zfs_error(hdl, EZFS_BADSTREAM, errbuf);
	goto out;
	}

	(void) nvlist_lookup_string(stream_nv, "fromsnap", &fromsnap);

	if (drr->drr_payloadlen != 0) {
	nvlist_t *stream_fss;

	VERIFY(0 == nvlist_lookup_nvlist(stream_nv, "fss",
	&stream_fss));
	if ((stream_avl = fsavl_create(stream_fss)) == NULL) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"couldn't allocate avl tree"));
	error = zfs_error(hdl, EZFS_NOMEM, errbuf);
	goto out;
	}

	if (fromsnap != NULL && recursive) {
	nvlist_t *renamed = NULL;
	nvpair_t *pair = NULL;

	(void) strlcpy(tofs, destname, sizeof (tofs));
	if (flags->isprefix) {
	struct drr_begin *drrb = &drr->drr_u.drr_begin;
	int i;

	if (flags->istail) {
	cp = strrchr(drrb->drr_toname, '/');
	if (cp == NULL) {
	(void) strlcat(tofs, "/",
	sizeof (tofs));
	i = 0;
	} else {
	i = (cp - drrb->drr_toname);
	}
	} else {
	i = strcspn(drrb->drr_toname, "/@");
	}
	/* zfs_receive_one() will create_parents() */
	(void) strlcat(tofs, &drrb->drr_toname[i],
	sizeof (tofs));
	*strchr(tofs, '@') = '\0';
	}

	if (!flags->dryrun && !flags->nomount) {
	VERIFY(0 == nvlist_alloc(&renamed,
	NV_UNIQUE_NAME, 0));
	}

	softerr = recv_incremental_replication(hdl, tofs, flags,
	stream_nv, stream_avl, renamed);

	/* Unmount renamed filesystems before receiving. */
	while ((pair = nvlist_next_nvpair(renamed,
	pair)) != NULL) {
	zfs_handle_t *zhp;
	prop_changelist_t *clp = NULL;

	zhp = zfs_open(hdl, nvpair_name(pair),
	ZFS_TYPE_FILESYSTEM);
	if (zhp != NULL) {
	clp = changelist_gather(zhp,
	ZFS_PROP_MOUNTPOINT, 0,
	flags->forceunmount ? MS_FORCE : 0);
	zfs_close(zhp);
	if (clp != NULL) {
	softerr \|=
	changelist_prefix(clp);
	changelist_free(clp);
	}
	}
	}

	nvlist_free(renamed);
	}
	}

	/*
	* Get the fs specified by the first path in the stream (the top level
	* specified by 'zfs send') and pass it to each invocation of
	* zfs_receive_one().
	*/
	(void) strlcpy(sendfs, drr->drr_u.drr_begin.drr_toname,
	sizeof (sendfs));
	if ((cp = strchr(sendfs, '@')) != NULL) {
	*cp = '\0';
	/*
	* Find the "sendsnap", the final snapshot in a replication
	* stream. zfs_receive_one() handles certain errors
	* differently, depending on if the contained stream is the
	* last one or not.
	*/
	sendsnap = (cp + 1);
	}

	/* Finally, receive each contained stream */
	do {
	/*
	* we should figure out if it has a recoverable
	* error, in which case do a recv_skip() and drive on.
	* Note, if we fail due to already having this guid,
	* zfs_receive_one() will take care of it (ie,
	* recv_skip() and return 0).
	*/
	error = zfs_receive_impl(hdl, destname, NULL, flags, fd,
	sendfs, stream_nv, stream_avl, top_zfs, sendsnap, cmdprops);
	if (error == ENODATA) {
	error = 0;
	break;
	}
	anyerr \|= error;
	} while (error == 0);

	if (drr->drr_payloadlen != 0 && recursive && fromsnap != NULL) {
	/*
	* Now that we have the fs's they sent us, try the
	* renames again.
	*/
	softerr = recv_incremental_replication(hdl, tofs, flags,
	stream_nv, stream_avl, NULL);
	}

	if (raw && softerr == 0 && *top_zfs != NULL) {
	softerr = recv_fix_encryption_hierarchy(hdl, *top_zfs,
	stream_nv, stream_avl);
	}

	out:
	fsavl_destroy(stream_avl);
	nvlist_free(stream_nv);
	if (softerr)
	error = -2;
	if (anyerr)
	error = -1;
	return (error);
	}

	static void
	trunc_prop_errs(int truncated)
	{
	ASSERT(truncated != 0);

	if (truncated == 1)
	(void) fprintf(stderr, dgettext(TEXT_DOMAIN,
	"1 more property could not be set\n"));
	else
	(void) fprintf(stderr, dgettext(TEXT_DOMAIN,
	"%d more properties could not be set\n"), truncated);
	}

	static int
	recv_skip(libzfs_handle_t *hdl, int fd, boolean_t byteswap)
	{
	dmu_replay_record_t *drr;
	void *buf = zfs_alloc(hdl, SPA_MAXBLOCKSIZE);
	uint64_t payload_size;
	char errbuf[1024];

	(void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN,
	"cannot receive"));

	/* XXX would be great to use lseek if possible... */
	drr = buf;

	while (recv_read(hdl, fd, drr, sizeof (dmu_replay_record_t),
	byteswap, NULL) == 0) {
	if (byteswap)
	drr->drr_type = BSWAP_32(drr->drr_type);

	switch (drr->drr_type) {
	case DRR_BEGIN:
	if (drr->drr_payloadlen != 0) {
	(void) recv_read(hdl, fd, buf,
	drr->drr_payloadlen, B_FALSE, NULL);
	}
	break;

	case DRR_END:
	free(buf);
	return (0);

	case DRR_OBJECT:
	if (byteswap) {
	drr->drr_u.drr_object.drr_bonuslen =
	BSWAP_32(drr->drr_u.drr_object.
	drr_bonuslen);
	drr->drr_u.drr_object.drr_raw_bonuslen =
	BSWAP_32(drr->drr_u.drr_object.
	drr_raw_bonuslen);
	}

	payload_size =
	DRR_OBJECT_PAYLOAD_SIZE(&drr->drr_u.drr_object);
	(void) recv_read(hdl, fd, buf, payload_size,
	B_FALSE, NULL);
	break;

	case DRR_WRITE:
	if (byteswap) {
	drr->drr_u.drr_write.drr_logical_size =
	BSWAP_64(
	drr->drr_u.drr_write.drr_logical_size);
	drr->drr_u.drr_write.drr_compressed_size =
	BSWAP_64(
	drr->drr_u.drr_write.drr_compressed_size);
	}
	payload_size =
	DRR_WRITE_PAYLOAD_SIZE(&drr->drr_u.drr_write);
	assert(payload_size <= SPA_MAXBLOCKSIZE);
	(void) recv_read(hdl, fd, buf,
	payload_size, B_FALSE, NULL);
	break;
	case DRR_SPILL:
	if (byteswap) {
	drr->drr_u.drr_spill.drr_length =
	BSWAP_64(drr->drr_u.drr_spill.drr_length);
	drr->drr_u.drr_spill.drr_compressed_size =
	BSWAP_64(drr->drr_u.drr_spill.
	drr_compressed_size);
	}

	payload_size =
	DRR_SPILL_PAYLOAD_SIZE(&drr->drr_u.drr_spill);
	(void) recv_read(hdl, fd, buf, payload_size,
	B_FALSE, NULL);
	break;
	case DRR_WRITE_EMBEDDED:
	if (byteswap) {
	drr->drr_u.drr_write_embedded.drr_psize =
	BSWAP_32(drr->drr_u.drr_write_embedded.
	drr_psize);
	}
	(void) recv_read(hdl, fd, buf,
	P2ROUNDUP(drr->drr_u.drr_write_embedded.drr_psize,
	8), B_FALSE, NULL);
	break;
	case DRR_OBJECT_RANGE:
	case DRR_WRITE_BYREF:
	case DRR_FREEOBJECTS:
	case DRR_FREE:
	break;

	default:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"invalid record type"));
	free(buf);
	return (zfs_error(hdl, EZFS_BADSTREAM, errbuf));
	}
	}

	free(buf);
	return (-1);
	}

	static void
	recv_ecksum_set_aux(libzfs_handle_t hdl, const char target_snap,
	boolean_t resumable, boolean_t checksum)
	{
	char target_fs[ZFS_MAX_DATASET_NAME_LEN];

	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, (checksum ?
	"checksum mismatch" : "incomplete stream")));

	if (!resumable)
	return;
	(void) strlcpy(target_fs, target_snap, sizeof (target_fs));
	*strchr(target_fs, '@') = '\0';
	zfs_handle_t *zhp = zfs_open(hdl, target_fs,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME);
	if (zhp == NULL)
	return;

	char token_buf[ZFS_MAXPROPLEN];
	int error = zfs_prop_get(zhp, ZFS_PROP_RECEIVE_RESUME_TOKEN,
	token_buf, sizeof (token_buf),
	NULL, NULL, 0, B_TRUE);
	if (error == 0) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"checksum mismatch or incomplete stream.\n"
	"Partially received snapshot is saved.\n"
	"A resuming stream can be generated on the sending "
	"system by running:\n"
	" zfs send -t %s"),
	token_buf);
	}
	zfs_close(zhp);
	}

	/*
	* Prepare a new nvlist of properties that are to override (-o) or be excluded
	* (-x) from the received dataset
	* recvprops: received properties from the send stream
	* cmdprops: raw input properties from command line
	* origprops: properties, both locally-set and received, currently set on the
	* target dataset if it exists, NULL otherwise.
	* oxprops: valid output override (-o) and excluded (-x) properties
	*/
	static int
	zfs_setup_cmdline_props(libzfs_handle_t *hdl, zfs_type_t type,
	char *fsname, boolean_t zoned, boolean_t recursive, boolean_t newfs,
	boolean_t raw, boolean_t toplevel, nvlist_t recvprops, nvlist_t cmdprops,
	nvlist_t origprops, nvlist_t oxprops, uint8_t *wkeydata_out,
	uint_t wkeylen_out, const char errbuf)
	{
	nvpair_t *nvp;
	nvlist_t oprops, voprops;
	zfs_handle_t *zhp = NULL;
	zpool_handle_t *zpool_hdl = NULL;
	char *cp;
	int ret = 0;
	char namebuf[ZFS_MAX_DATASET_NAME_LEN];

	if (nvlist_empty(cmdprops))
	return (0); /* No properties to override or exclude */

	*oxprops = fnvlist_alloc();
	oprops = fnvlist_alloc();

	strlcpy(namebuf, fsname, ZFS_MAX_DATASET_NAME_LEN);

	/*
	* Get our dataset handle. The target dataset may not exist yet.
	*/
	if (zfs_dataset_exists(hdl, namebuf, ZFS_TYPE_DATASET)) {
	zhp = zfs_open(hdl, namebuf, ZFS_TYPE_DATASET);
	if (zhp == NULL) {
	ret = -1;
	goto error;
	}
	}

	/* open the zpool handle */
	cp = strchr(namebuf, '/');
	if (cp != NULL)
	*cp = '\0';
	zpool_hdl = zpool_open(hdl, namebuf);
	if (zpool_hdl == NULL) {
	ret = -1;
	goto error;
	}

	/* restore namebuf to match fsname for later use */
	if (cp != NULL)
	*cp = '/';

	/*
	* first iteration: process excluded (-x) properties now and gather
	* added (-o) properties to be later processed by zfs_valid_proplist()
	*/
	nvp = NULL;
	while ((nvp = nvlist_next_nvpair(cmdprops, nvp)) != NULL) {
	const char *name = nvpair_name(nvp);
	zfs_prop_t prop = zfs_name_to_prop(name);

	/* "origin" is processed separately, don't handle it here */
	if (prop == ZFS_PROP_ORIGIN)
	continue;

	/*
	* we're trying to override or exclude a property that does not
	* make sense for this type of dataset, but we don't want to
	* fail if the receive is recursive: this comes in handy when
	* the send stream contains, for instance, a child ZVOL and
	* we're trying to receive it with "-o atime=on"
	*/
	if (!zfs_prop_valid_for_type(prop, type, B_FALSE) &&
	!zfs_prop_user(name)) {
	if (recursive)
	continue;
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"property '%s' does not apply to datasets of this "
	"type"), name);
	ret = zfs_error(hdl, EZFS_BADPROP, errbuf);
	goto error;
	}

	/* raw streams can't override encryption properties */
	if ((zfs_prop_encryption_key_param(prop) \|\|
	prop == ZFS_PROP_ENCRYPTION) && raw) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"encryption property '%s' cannot "
	"be set or excluded for raw streams."), name);
	ret = zfs_error(hdl, EZFS_BADPROP, errbuf);
	goto error;
	}

	/* incremental streams can only exclude encryption properties */
	if ((zfs_prop_encryption_key_param(prop) \|\|
	prop == ZFS_PROP_ENCRYPTION) && !newfs &&
	nvpair_type(nvp) != DATA_TYPE_BOOLEAN) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"encryption property '%s' cannot "
	"be set for incremental streams."), name);
	ret = zfs_error(hdl, EZFS_BADPROP, errbuf);
	goto error;
	}

	switch (nvpair_type(nvp)) {
	case DATA_TYPE_BOOLEAN: /* -x property */
	/*
	* DATA_TYPE_BOOLEAN is the way we're asked to "exclude"
	* a property: this is done by forcing an explicit
	* inherit on the destination so the effective value is
	* not the one we received from the send stream.
	* We do this only if the property is not already
	* locally-set, in which case its value will take
	* priority over the received anyway.
	*/
	if (nvlist_exists(origprops, name)) {
	nvlist_t *attrs;
	char *source = NULL;

	attrs = fnvlist_lookup_nvlist(origprops, name);
	if (nvlist_lookup_string(attrs,
	ZPROP_SOURCE, &source) == 0 &&
	strcmp(source, ZPROP_SOURCE_VAL_RECVD) != 0)
	continue;
	}
	/*
	* We can't force an explicit inherit on non-inheritable
	* properties: if we're asked to exclude this kind of
	* values we remove them from "recvprops" input nvlist.
	*/
	if (!zfs_prop_inheritable(prop) &&
	!zfs_prop_user(name) && /* can be inherited too */
	nvlist_exists(recvprops, name))
	fnvlist_remove(recvprops, name);
	else
	fnvlist_add_nvpair(*oxprops, nvp);
	break;
	case DATA_TYPE_STRING: /* -o property=value */
	fnvlist_add_nvpair(oprops, nvp);
	break;
	default:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"property '%s' must be a string or boolean"), name);
	ret = zfs_error(hdl, EZFS_BADPROP, errbuf);
	goto error;
	}
	}

	if (toplevel) {
	/* convert override strings properties to native */
	if ((voprops = zfs_valid_proplist(hdl, ZFS_TYPE_DATASET,
	oprops, zoned, zhp, zpool_hdl, B_FALSE, errbuf)) == NULL) {
	ret = zfs_error(hdl, EZFS_BADPROP, errbuf);
	goto error;
	}

	/*
	* zfs_crypto_create() requires the parent name. Get it
	* by truncating the fsname copy stored in namebuf.
	*/
	cp = strrchr(namebuf, '/');
	if (cp != NULL)
	*cp = '\0';

	if (!raw && zfs_crypto_create(hdl, namebuf, voprops, NULL,
	B_FALSE, wkeydata_out, wkeylen_out) != 0) {
	fnvlist_free(voprops);
	ret = zfs_error(hdl, EZFS_CRYPTOFAILED, errbuf);
	goto error;
	}

	/* second pass: process "-o" properties */
	fnvlist_merge(*oxprops, voprops);
	fnvlist_free(voprops);
	} else {
	/* override props on child dataset are inherited */
	nvp = NULL;
	while ((nvp = nvlist_next_nvpair(oprops, nvp)) != NULL) {
	const char *name = nvpair_name(nvp);
	fnvlist_add_boolean(*oxprops, name);
	}
	}

	error:
	if (zhp != NULL)
	zfs_close(zhp);
	if (zpool_hdl != NULL)
	zpool_close(zpool_hdl);
	fnvlist_free(oprops);
	return (ret);
	}

	/*
	* Restores a backup of tosnap from the file descriptor specified by infd.
	*/
	static int
	zfs_receive_one(libzfs_handle_t hdl, int infd, const char tosnap,
	const char originsnap, recvflags_t flags, dmu_replay_record_t *drr,
	dmu_replay_record_t drr_noswap, const char sendfs, nvlist_t *stream_nv,
	avl_tree_t stream_avl, char *top_zfs,
	const char finalsnap, nvlist_t cmdprops)
	{
	time_t begin_time;
	int ioctl_err, ioctl_errno, err;
	char *cp;
	struct drr_begin *drrb = &drr->drr_u.drr_begin;
	char errbuf[1024];
	const char *chopprefix;
	boolean_t newfs = B_FALSE;
	- boolean_t stream_wantsnewfs;
	+ boolean_t stream_wantsnewfs, stream_resumingnewfs;
	boolean_t newprops = B_FALSE;
	uint64_t read_bytes = 0;
	uint64_t errflags = 0;
	uint64_t parent_snapguid = 0;
	prop_changelist_t *clp = NULL;
	nvlist_t *snapprops_nvlist = NULL;
	nvlist_t *snapholds_nvlist = NULL;
	zprop_errflags_t prop_errflags;
	nvlist_t *prop_errors = NULL;
	boolean_t recursive;
	char *snapname = NULL;
	char destsnap[MAXPATHLEN * 2];
	char origin[MAXNAMELEN];
	char name[MAXPATHLEN];
	char tmp_keylocation[MAXNAMELEN];
	nvlist_t rcvprops = NULL; / props received from the send stream */
	nvlist_t oxprops = NULL; / override (-o) and exclude (-x) props */
	nvlist_t origprops = NULL; / original props (if destination exists) */
	zfs_type_t type;
	boolean_t toplevel = B_FALSE;
	boolean_t zoned = B_FALSE;
	boolean_t hastoken = B_FALSE;
	boolean_t redacted;
	uint8_t *wkeydata = NULL;
	uint_t wkeylen = 0;

	begin_time = time(NULL);
	bzero(origin, MAXNAMELEN);
	bzero(tmp_keylocation, MAXNAMELEN);

	(void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN,
	"cannot receive"));

	recursive = (nvlist_lookup_boolean(stream_nv, "not_recursive") ==
	ENOENT);

	/* Did the user request holds be skipped via zfs recv -k? */
	boolean_t holds = flags->holds && !flags->skipholds;

	if (stream_avl != NULL) {
	char *keylocation = NULL;
	nvlist_t *lookup = NULL;
	nvlist_t *fs = fsavl_find(stream_avl, drrb->drr_toguid,
	&snapname);

	(void) nvlist_lookup_uint64(fs, "parentfromsnap",
	&parent_snapguid);
	err = nvlist_lookup_nvlist(fs, "props", &rcvprops);
	if (err) {
	VERIFY(0 == nvlist_alloc(&rcvprops, NV_UNIQUE_NAME, 0));
	newprops = B_TRUE;
	}

	/*
	* The keylocation property may only be set on encryption roots,
	* but this dataset might not become an encryption root until
	* recv_fix_encryption_hierarchy() is called. That function
	* will fixup the keylocation anyway, so we temporarily unset
	* the keylocation for now to avoid any errors from the receive
	* ioctl.
	*/
	err = nvlist_lookup_string(rcvprops,
	zfs_prop_to_name(ZFS_PROP_KEYLOCATION), &keylocation);
	if (err == 0) {
	strcpy(tmp_keylocation, keylocation);
	(void) nvlist_remove_all(rcvprops,
	zfs_prop_to_name(ZFS_PROP_KEYLOCATION));
	}

	if (flags->canmountoff) {
	VERIFY(0 == nvlist_add_uint64(rcvprops,
	zfs_prop_to_name(ZFS_PROP_CANMOUNT), 0));
	} else if (newprops) { /* nothing in rcvprops, eliminate it */
	nvlist_free(rcvprops);
	rcvprops = NULL;
	newprops = B_FALSE;
	}
	if (0 == nvlist_lookup_nvlist(fs, "snapprops", &lookup)) {
	VERIFY(0 == nvlist_lookup_nvlist(lookup,
	snapname, &snapprops_nvlist));
	}
	if (holds) {
	if (0 == nvlist_lookup_nvlist(fs, "snapholds",
	&lookup)) {
	VERIFY(0 == nvlist_lookup_nvlist(lookup,
	snapname, &snapholds_nvlist));
	}
	}
	}

	cp = NULL;

	/*
	* Determine how much of the snapshot name stored in the stream
	* we are going to tack on to the name they specified on the
	* command line, and how much we are going to chop off.
	*
	* If they specified a snapshot, chop the entire name stored in
	* the stream.
	*/
	if (flags->istail) {
	/*
	* A filesystem was specified with -e. We want to tack on only
	* the tail of the sent snapshot path.
	*/
	if (strchr(tosnap, '@')) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "invalid "
	"argument - snapshot not allowed with -e"));
	err = zfs_error(hdl, EZFS_INVALIDNAME, errbuf);
	goto out;
	}

	chopprefix = strrchr(sendfs, '/');

	if (chopprefix == NULL) {
	/*
	* The tail is the poolname, so we need to
	* prepend a path separator.
	*/
	int len = strlen(drrb->drr_toname);
	cp = malloc(len + 2);
	cp[0] = '/';
	(void) strcpy(&cp[1], drrb->drr_toname);
	chopprefix = cp;
	} else {
	chopprefix = drrb->drr_toname + (chopprefix - sendfs);
	}
	} else if (flags->isprefix) {
	/*
	* A filesystem was specified with -d. We want to tack on
	* everything but the first element of the sent snapshot path
	* (all but the pool name).
	*/
	if (strchr(tosnap, '@')) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "invalid "
	"argument - snapshot not allowed with -d"));
	err = zfs_error(hdl, EZFS_INVALIDNAME, errbuf);
	goto out;
	}

	chopprefix = strchr(drrb->drr_toname, '/');
	if (chopprefix == NULL)
	chopprefix = strchr(drrb->drr_toname, '@');
	} else if (strchr(tosnap, '@') == NULL) {
	/*
	* If a filesystem was specified without -d or -e, we want to
	* tack on everything after the fs specified by 'zfs send'.
	*/
	chopprefix = drrb->drr_toname + strlen(sendfs);
	} else {
	/* A snapshot was specified as an exact path (no -d or -e). */
	if (recursive) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"cannot specify snapshot name for multi-snapshot "
	"stream"));
	err = zfs_error(hdl, EZFS_BADSTREAM, errbuf);
	goto out;
	}
	chopprefix = drrb->drr_toname + strlen(drrb->drr_toname);
	}

	ASSERT(strstr(drrb->drr_toname, sendfs) == drrb->drr_toname);
	ASSERT(chopprefix > drrb->drr_toname \|\| strchr(sendfs, '/') == NULL);
	ASSERT(chopprefix <= drrb->drr_toname + strlen(drrb->drr_toname) \|\|
	strchr(sendfs, '/') == NULL);
	ASSERT(chopprefix[0] == '/' \|\| chopprefix[0] == '@' \|\|
	chopprefix[0] == '\0');

	/*
	* Determine name of destination snapshot.
	*/
	(void) strlcpy(destsnap, tosnap, sizeof (destsnap));
	(void) strlcat(destsnap, chopprefix, sizeof (destsnap));
	free(cp);
	if (!zfs_name_valid(destsnap, ZFS_TYPE_SNAPSHOT)) {
	err = zfs_error(hdl, EZFS_INVALIDNAME, errbuf);
	goto out;
	}

	/*
	* Determine the name of the origin snapshot.
	*/
	if (originsnap) {
	(void) strlcpy(origin, originsnap, sizeof (origin));
	if (flags->verbose)
	(void) printf("using provided clone origin %s\n",
	origin);
	} else if (drrb->drr_flags & DRR_FLAG_CLONE) {
	if (guid_to_name(hdl, destsnap,
	drrb->drr_fromguid, B_FALSE, origin) != 0) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"local origin for clone %s does not exist"),
	destsnap);
	err = zfs_error(hdl, EZFS_NOENT, errbuf);
	goto out;
	}
	if (flags->verbose)
	(void) printf("found clone origin %s\n", origin);
	}

	if ((DMU_GET_FEATUREFLAGS(drrb->drr_versioninfo) &
	DMU_BACKUP_FEATURE_DEDUP)) {
	(void) fprintf(stderr,
	gettext("ERROR: \"zfs receive\" no longer supports "
	"deduplicated send streams. Use\n"
	"the \"zstream redup\" command to convert this stream "
	"to a regular,\n"
	"non-deduplicated stream.\n"));
	err = zfs_error(hdl, EZFS_NOTSUP, errbuf);
	goto out;
	}

	boolean_t resuming = DMU_GET_FEATUREFLAGS(drrb->drr_versioninfo) &
	DMU_BACKUP_FEATURE_RESUMING;
	boolean_t raw = DMU_GET_FEATUREFLAGS(drrb->drr_versioninfo) &
	DMU_BACKUP_FEATURE_RAW;
	boolean_t embedded = DMU_GET_FEATUREFLAGS(drrb->drr_versioninfo) &
	DMU_BACKUP_FEATURE_EMBED_DATA;
	stream_wantsnewfs = (drrb->drr_fromguid == 0 \|\|
	(drrb->drr_flags & DRR_FLAG_CLONE) \|\| originsnap) && !resuming;
	+ stream_resumingnewfs = (drrb->drr_fromguid == 0 \|\|
	+ (drrb->drr_flags & DRR_FLAG_CLONE) \|\| originsnap) && resuming;

	if (stream_wantsnewfs) {
	/*
	* if the parent fs does not exist, look for it based on
	* the parent snap GUID
	*/
	(void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN,
	"cannot receive new filesystem stream"));

	(void) strcpy(name, destsnap);
	cp = strrchr(name, '/');
	if (cp)
	*cp = '\0';
	if (cp &&
	!zfs_dataset_exists(hdl, name, ZFS_TYPE_DATASET)) {
	char suffix[ZFS_MAX_DATASET_NAME_LEN];
	(void) strcpy(suffix, strrchr(destsnap, '/'));
	if (guid_to_name(hdl, name, parent_snapguid,
	B_FALSE, destsnap) == 0) {
	*strchr(destsnap, '@') = '\0';
	(void) strcat(destsnap, suffix);
	}
	}
	} else {
	/*
	* If the fs does not exist, look for it based on the
	* fromsnap GUID.
	*/
	if (resuming) {
	(void) snprintf(errbuf, sizeof (errbuf),
	dgettext(TEXT_DOMAIN,
	"cannot receive resume stream"));
	} else {
	(void) snprintf(errbuf, sizeof (errbuf),
	dgettext(TEXT_DOMAIN,
	"cannot receive incremental stream"));
	}

	(void) strcpy(name, destsnap);
	*strchr(name, '@') = '\0';

	/*
	* If the exact receive path was specified and this is the
	* topmost path in the stream, then if the fs does not exist we
	* should look no further.
	*/
	if ((flags->isprefix \|\| (*(chopprefix = drrb->drr_toname +
	strlen(sendfs)) != '\0' && *chopprefix != '@')) &&
	!zfs_dataset_exists(hdl, name, ZFS_TYPE_DATASET)) {
	char snap[ZFS_MAX_DATASET_NAME_LEN];
	(void) strcpy(snap, strchr(destsnap, '@'));
	if (guid_to_name(hdl, name, drrb->drr_fromguid,
	B_FALSE, destsnap) == 0) {
	*strchr(destsnap, '@') = '\0';
	(void) strcat(destsnap, snap);
	}
	}
	}

	(void) strcpy(name, destsnap);
	*strchr(name, '@') = '\0';

	redacted = DMU_GET_FEATUREFLAGS(drrb->drr_versioninfo) &
	DMU_BACKUP_FEATURE_REDACTED;

	if (zfs_dataset_exists(hdl, name, ZFS_TYPE_DATASET)) {
	zfs_cmd_t zc = {"\0"};
	zfs_handle_t *zhp;
	boolean_t encrypted;

	(void) strcpy(zc.zc_name, name);

	/*
	* Destination fs exists. It must be one of these cases:
	* - an incremental send stream
	* - the stream specifies a new fs (full stream or clone)
	* and they want us to blow away the existing fs (and
	* have therefore specified -F and removed any snapshots)
	* - we are resuming a failed receive.
	*/
	if (stream_wantsnewfs) {
	boolean_t is_volume = drrb->drr_type == DMU_OST_ZVOL;
	if (!flags->force) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"destination '%s' exists\n"
	"must specify -F to overwrite it"), name);
	err = zfs_error(hdl, EZFS_EXISTS, errbuf);
	goto out;
	}
	if (zfs_ioctl(hdl, ZFS_IOC_SNAPSHOT_LIST_NEXT,
	&zc) == 0) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"destination has snapshots (eg. %s)\n"
	"must destroy them to overwrite it"),
	zc.zc_name);
	err = zfs_error(hdl, EZFS_EXISTS, errbuf);
	goto out;
	}
	if (is_volume && strrchr(name, '/') == NULL) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"destination %s is the root dataset\n"
	"cannot overwrite with a ZVOL"),
	name);
	err = zfs_error(hdl, EZFS_EXISTS, errbuf);
	goto out;
	}
	if (is_volume &&
	zfs_ioctl(hdl, ZFS_IOC_DATASET_LIST_NEXT,
	&zc) == 0) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"destination has children (eg. %s)\n"
	"cannot overwrite with a ZVOL"),
	zc.zc_name);
	err = zfs_error(hdl, EZFS_WRONG_PARENT, errbuf);
	goto out;
	}
	}

	if ((zhp = zfs_open(hdl, name,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME)) == NULL) {
	err = -1;
	goto out;
	}

	if (stream_wantsnewfs &&
	zhp->zfs_dmustats.dds_origin[0]) {
	zfs_close(zhp);
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"destination '%s' is a clone\n"
	"must destroy it to overwrite it"), name);
	err = zfs_error(hdl, EZFS_EXISTS, errbuf);
	goto out;
	}

	/*
	* Raw sends can not be performed as an incremental on top
	* of existing unencrypted datasets. zfs recv -F can't be
	* used to blow away an existing encrypted filesystem. This
	* is because it would require the dsl dir to point to the
	* new key (or lack of a key) and the old key at the same
	* time. The -F flag may still be used for deleting
	* intermediate snapshots that would otherwise prevent the
	* receive from working.
	*/
	encrypted = zfs_prop_get_int(zhp, ZFS_PROP_ENCRYPTION) !=
	ZIO_CRYPT_OFF;
	if (!stream_wantsnewfs && !encrypted && raw) {
	zfs_close(zhp);
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"cannot perform raw receive on top of "
	"existing unencrypted dataset"));
	err = zfs_error(hdl, EZFS_BADRESTORE, errbuf);
	goto out;
	}

	if (stream_wantsnewfs && flags->force &&
	((raw && !encrypted) \|\| encrypted)) {
	zfs_close(zhp);
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"zfs receive -F cannot be used to destroy an "
	"encrypted filesystem or overwrite an "
	"unencrypted one with an encrypted one"));
	err = zfs_error(hdl, EZFS_BADRESTORE, errbuf);
	goto out;
	}

	if (!flags->dryrun && zhp->zfs_type == ZFS_TYPE_FILESYSTEM &&
	- stream_wantsnewfs) {
	+ (stream_wantsnewfs \|\| stream_resumingnewfs)) {
	/* We can't do online recv in this case */
	clp = changelist_gather(zhp, ZFS_PROP_NAME, 0,
	flags->forceunmount ? MS_FORCE : 0);
	if (clp == NULL) {
	zfs_close(zhp);
	err = -1;
	goto out;
	}
	if (changelist_prefix(clp) != 0) {
	changelist_free(clp);
	zfs_close(zhp);
	err = -1;
	goto out;
	}
	}

	/*
	* If we are resuming a newfs, set newfs here so that we will
	* mount it if the recv succeeds this time. We can tell
	* that it was a newfs on the first recv because the fs
	* itself will be inconsistent (if the fs existed when we
	* did the first recv, we would have received it into
	* .../%recv).
	*/
	if (resuming && zfs_prop_get_int(zhp, ZFS_PROP_INCONSISTENT))
	newfs = B_TRUE;

	/* we want to know if we're zoned when validating -o\|-x props */
	zoned = zfs_prop_get_int(zhp, ZFS_PROP_ZONED);

	/* may need this info later, get it now we have zhp around */
	if (zfs_prop_get(zhp, ZFS_PROP_RECEIVE_RESUME_TOKEN, NULL, 0,
	NULL, NULL, 0, B_TRUE) == 0)
	hastoken = B_TRUE;

	/* gather existing properties on destination */
	origprops = fnvlist_alloc();
	fnvlist_merge(origprops, zhp->zfs_props);
	fnvlist_merge(origprops, zhp->zfs_user_props);

	zfs_close(zhp);
	} else {
	zfs_handle_t *zhp;

	/*
	* Destination filesystem does not exist. Therefore we better
	* be creating a new filesystem (either from a full backup, or
	* a clone). It would therefore be invalid if the user
	* specified only the pool name (i.e. if the destination name
	* contained no slash character).
	*/
	cp = strrchr(name, '/');

	if (!stream_wantsnewfs \|\| cp == NULL) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"destination '%s' does not exist"), name);
	err = zfs_error(hdl, EZFS_NOENT, errbuf);
	goto out;
	}

	/*
	* Trim off the final dataset component so we perform the
	* recvbackup ioctl to the filesystems's parent.
	*/
	*cp = '\0';

	if (flags->isprefix && !flags->istail && !flags->dryrun &&
	create_parents(hdl, destsnap, strlen(tosnap)) != 0) {
	err = zfs_error(hdl, EZFS_BADRESTORE, errbuf);
	goto out;
	}

	/* validate parent */
	zhp = zfs_open(hdl, name, ZFS_TYPE_DATASET);
	if (zhp == NULL) {
	err = zfs_error(hdl, EZFS_BADRESTORE, errbuf);
	goto out;
	}
	if (zfs_get_type(zhp) != ZFS_TYPE_FILESYSTEM) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"parent '%s' is not a filesystem"), name);
	err = zfs_error(hdl, EZFS_WRONG_PARENT, errbuf);
	zfs_close(zhp);
	goto out;
	}

	zfs_close(zhp);

	newfs = B_TRUE;
	*cp = '/';
	}

	if (flags->verbose) {
	(void) printf("%s %s stream of %s into %s\n",
	flags->dryrun ? "would receive" : "receiving",
	drrb->drr_fromguid ? "incremental" : "full",
	drrb->drr_toname, destsnap);
	(void) fflush(stdout);
	}

	if (flags->dryrun) {
	void *buf = zfs_alloc(hdl, SPA_MAXBLOCKSIZE);

	/*
	* We have read the DRR_BEGIN record, but we have
	* not yet read the payload. For non-dryrun sends
	* this will be done by the kernel, so we must
	* emulate that here, before attempting to read
	* more records.
	*/
	err = recv_read(hdl, infd, buf, drr->drr_payloadlen,
	flags->byteswap, NULL);
	free(buf);
	if (err != 0)
	goto out;

	err = recv_skip(hdl, infd, flags->byteswap);
	goto out;
	}

	/*
	* If this is the top-level dataset, record it so we can use it
	* for recursive operations later.
	*/
	if (top_zfs != NULL &&
	(top_zfs == NULL \|\| strcmp(top_zfs, name) == 0)) {
	toplevel = B_TRUE;
	if (*top_zfs == NULL)
	*top_zfs = zfs_strdup(hdl, name);
	}

	if (drrb->drr_type == DMU_OST_ZVOL) {
	type = ZFS_TYPE_VOLUME;
	} else if (drrb->drr_type == DMU_OST_ZFS) {
	type = ZFS_TYPE_FILESYSTEM;
	} else {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"invalid record type: 0x%d"), drrb->drr_type);
	err = zfs_error(hdl, EZFS_BADSTREAM, errbuf);
	goto out;
	}
	if ((err = zfs_setup_cmdline_props(hdl, type, name, zoned, recursive,
	stream_wantsnewfs, raw, toplevel, rcvprops, cmdprops, origprops,
	&oxprops, &wkeydata, &wkeylen, errbuf)) != 0)
	goto out;

	/*
	* When sending with properties (zfs send -p), the encryption property
	* is not included because it is a SETONCE property and therefore
	* treated as read only. However, we are always able to determine its
	* value because raw sends will include it in the DRR_BDEGIN payload
	* and non-raw sends with properties are not allowed for encrypted
	* datasets. Therefore, if this is a non-raw properties stream, we can
	* infer that the value should be ZIO_CRYPT_OFF and manually add that
	* to the received properties.
	*/
	if (stream_wantsnewfs && !raw && rcvprops != NULL &&
	!nvlist_exists(cmdprops, zfs_prop_to_name(ZFS_PROP_ENCRYPTION))) {
	if (oxprops == NULL)
	oxprops = fnvlist_alloc();
	fnvlist_add_uint64(oxprops,
	zfs_prop_to_name(ZFS_PROP_ENCRYPTION), ZIO_CRYPT_OFF);
	}

	err = ioctl_err = lzc_receive_with_cmdprops(destsnap, rcvprops,
	oxprops, wkeydata, wkeylen, origin, flags->force, flags->resumable,
	raw, infd, drr_noswap, -1, &read_bytes, &errflags,
	NULL, &prop_errors);
	ioctl_errno = ioctl_err;
	prop_errflags = errflags;

	if (err == 0) {
	nvpair_t *prop_err = NULL;

	while ((prop_err = nvlist_next_nvpair(prop_errors,
	prop_err)) != NULL) {
	char tbuf[1024];
	zfs_prop_t prop;
	int intval;

	prop = zfs_name_to_prop(nvpair_name(prop_err));
	(void) nvpair_value_int32(prop_err, &intval);
	if (strcmp(nvpair_name(prop_err),
	ZPROP_N_MORE_ERRORS) == 0) {
	trunc_prop_errs(intval);
	break;
	} else if (snapname == NULL \|\| finalsnap == NULL \|\|
	strcmp(finalsnap, snapname) == 0 \|\|
	strcmp(nvpair_name(prop_err),
	zfs_prop_to_name(ZFS_PROP_REFQUOTA)) != 0) {
	/*
	* Skip the special case of, for example,
	* "refquota", errors on intermediate
	* snapshots leading up to a final one.
	* That's why we have all of the checks above.
	*
	* See zfs_ioctl.c's extract_delay_props() for
	* a list of props which can fail on
	* intermediate snapshots, but shouldn't
	* affect the overall receive.
	*/
	(void) snprintf(tbuf, sizeof (tbuf),
	dgettext(TEXT_DOMAIN,
	"cannot receive %s property on %s"),
	nvpair_name(prop_err), name);
	zfs_setprop_error(hdl, prop, intval, tbuf);
	}
	}
	}

	if (err == 0 && snapprops_nvlist) {
	zfs_cmd_t zc = {"\0"};

	(void) strcpy(zc.zc_name, destsnap);
	zc.zc_cookie = B_TRUE; /* received */
	if (zcmd_write_src_nvlist(hdl, &zc, snapprops_nvlist) == 0) {
	(void) zfs_ioctl(hdl, ZFS_IOC_SET_PROP, &zc);
	zcmd_free_nvlists(&zc);
	}
	}
	if (err == 0 && snapholds_nvlist) {
	nvpair_t *pair;
	nvlist_t holds, errors = NULL;
	int cleanup_fd = -1;

	VERIFY(0 == nvlist_alloc(&holds, 0, KM_SLEEP));
	for (pair = nvlist_next_nvpair(snapholds_nvlist, NULL);
	pair != NULL;
	pair = nvlist_next_nvpair(snapholds_nvlist, pair)) {
	VERIFY(0 == nvlist_add_string(holds, destsnap,
	nvpair_name(pair)));
	}
	(void) lzc_hold(holds, cleanup_fd, &errors);
	nvlist_free(snapholds_nvlist);
	nvlist_free(holds);
	}

	if (err && (ioctl_errno == ENOENT \|\| ioctl_errno == EEXIST)) {
	/*
	* It may be that this snapshot already exists,
	* in which case we want to consume & ignore it
	* rather than failing.
	*/
	avl_tree_t *local_avl;
	nvlist_t local_nv, fs;
	cp = strchr(destsnap, '@');

	/*
	* XXX Do this faster by just iterating over snaps in
	* this fs. Also if zc_value does not exist, we will
	* get a strange "does not exist" error message.
	*/
	*cp = '\0';
	if (gather_nvlist(hdl, destsnap, NULL, NULL, B_FALSE, B_TRUE,
	B_FALSE, B_FALSE, B_FALSE, B_FALSE, B_FALSE, B_TRUE,
	&local_nv, &local_avl) == 0) {
	*cp = '@';
	fs = fsavl_find(local_avl, drrb->drr_toguid, NULL);
	fsavl_destroy(local_avl);
	nvlist_free(local_nv);

	if (fs != NULL) {
	if (flags->verbose) {
	(void) printf("snap %s already exists; "
	"ignoring\n", destsnap);
	}
	err = ioctl_err = recv_skip(hdl, infd,
	flags->byteswap);
	}
	}
	*cp = '@';
	}

	if (ioctl_err != 0) {
	switch (ioctl_errno) {
	case ENODEV:
	cp = strchr(destsnap, '@');
	*cp = '\0';
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"most recent snapshot of %s does not\n"
	"match incremental source"), destsnap);
	(void) zfs_error(hdl, EZFS_BADRESTORE, errbuf);
	*cp = '@';
	break;
	case ETXTBSY:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"destination %s has been modified\n"
	"since most recent snapshot"), name);
	(void) zfs_error(hdl, EZFS_BADRESTORE, errbuf);
	break;
	case EACCES:
	if (raw && stream_wantsnewfs) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"failed to create encryption key"));
	} else if (raw && !stream_wantsnewfs) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"encryption key does not match "
	"existing key"));
	} else {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"inherited key must be loaded"));
	}
	(void) zfs_error(hdl, EZFS_CRYPTOFAILED, errbuf);
	break;
	case EEXIST:
	cp = strchr(destsnap, '@');
	if (newfs) {
	/* it's the containing fs that exists */
	*cp = '\0';
	}
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"destination already exists"));
	(void) zfs_error_fmt(hdl, EZFS_EXISTS,
	dgettext(TEXT_DOMAIN, "cannot restore to %s"),
	destsnap);
	*cp = '@';
	break;
	case EINVAL:
	if (flags->resumable) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"kernel modules must be upgraded to "
	"receive this stream."));
	} else if (embedded && !raw) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"incompatible embedded data stream "
	"feature with encrypted receive."));
	}
	(void) zfs_error(hdl, EZFS_BADSTREAM, errbuf);
	break;
	case ECKSUM:
	case ZFS_ERR_STREAM_TRUNCATED:
	recv_ecksum_set_aux(hdl, destsnap, flags->resumable,
	ioctl_err == ECKSUM);
	(void) zfs_error(hdl, EZFS_BADSTREAM, errbuf);
	break;
	case ZFS_ERR_STREAM_LARGE_BLOCK_MISMATCH:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"incremental send stream requires -L "
	"(--large-block), to match previous receive."));
	(void) zfs_error(hdl, EZFS_BADSTREAM, errbuf);
	break;
	case ENOTSUP:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"pool must be upgraded to receive this stream."));
	(void) zfs_error(hdl, EZFS_BADVERSION, errbuf);
	break;
	case EDQUOT:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"destination %s space quota exceeded."), name);
	(void) zfs_error(hdl, EZFS_NOSPC, errbuf);
	break;
	case ZFS_ERR_FROM_IVSET_GUID_MISSING:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"IV set guid missing. See errata %u at "
	"https://openzfs.github.io/openzfs-docs/msg/"
	"ZFS-8000-ER."),
	ZPOOL_ERRATA_ZOL_8308_ENCRYPTION);
	(void) zfs_error(hdl, EZFS_BADSTREAM, errbuf);
	break;
	case ZFS_ERR_FROM_IVSET_GUID_MISMATCH:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"IV set guid mismatch. See the 'zfs receive' "
	"man page section\n discussing the limitations "
	"of raw encrypted send streams."));
	(void) zfs_error(hdl, EZFS_BADSTREAM, errbuf);
	break;
	case ZFS_ERR_SPILL_BLOCK_FLAG_MISSING:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"Spill block flag missing for raw send.\n"
	"The zfs software on the sending system must "
	"be updated."));
	(void) zfs_error(hdl, EZFS_BADSTREAM, errbuf);
	break;
	case EBUSY:
	if (hastoken) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"destination %s contains "
	"partially-complete state from "
	"\"zfs receive -s\"."), name);
	(void) zfs_error(hdl, EZFS_BUSY, errbuf);
	break;
	}
	/* fallthru */
	default:
	(void) zfs_standard_error(hdl, ioctl_errno, errbuf);
	}
	}

	/*
	* Mount the target filesystem (if created). Also mount any
	* children of the target filesystem if we did a replication
	* receive (indicated by stream_avl being non-NULL).
	*/
	if (clp) {
	if (!flags->nomount)
	err \|= changelist_postfix(clp);
	changelist_free(clp);
	}

	if ((newfs \|\| stream_avl) && type == ZFS_TYPE_FILESYSTEM && !redacted)
	flags->domount = B_TRUE;

	if (prop_errflags & ZPROP_ERR_NOCLEAR) {
	(void) fprintf(stderr, dgettext(TEXT_DOMAIN, "Warning: "
	"failed to clear unreceived properties on %s"), name);
	(void) fprintf(stderr, "\n");
	}
	if (prop_errflags & ZPROP_ERR_NORESTORE) {
	(void) fprintf(stderr, dgettext(TEXT_DOMAIN, "Warning: "
	"failed to restore original properties on %s"), name);
	(void) fprintf(stderr, "\n");
	}

	if (err \|\| ioctl_err) {
	err = -1;
	goto out;
	}

	if (flags->verbose) {
	char buf1[64];
	char buf2[64];
	uint64_t bytes = read_bytes;
	time_t delta = time(NULL) - begin_time;
	if (delta == 0)
	delta = 1;
	zfs_nicebytes(bytes, buf1, sizeof (buf1));
	zfs_nicebytes(bytes/delta, buf2, sizeof (buf1));

	(void) printf("received %s stream in %lld seconds (%s/sec)\n",
	buf1, (longlong_t)delta, buf2);
	}

	err = 0;
	out:
	if (prop_errors != NULL)
	nvlist_free(prop_errors);

	if (tmp_keylocation[0] != '\0') {
	VERIFY(0 == nvlist_add_string(rcvprops,
	zfs_prop_to_name(ZFS_PROP_KEYLOCATION), tmp_keylocation));
	}

	if (newprops)
	nvlist_free(rcvprops);

	nvlist_free(oxprops);
	nvlist_free(origprops);

	return (err);
	}

	/*
	* Check properties we were asked to override (both -o\|-x)
	*/
	static boolean_t
	zfs_receive_checkprops(libzfs_handle_t hdl, nvlist_t props,
	const char *errbuf)
	{
	nvpair_t *nvp;
	zfs_prop_t prop;
	const char *name;

	nvp = NULL;
	while ((nvp = nvlist_next_nvpair(props, nvp)) != NULL) {
	name = nvpair_name(nvp);
	prop = zfs_name_to_prop(name);

	if (prop == ZPROP_INVAL) {
	if (!zfs_prop_user(name)) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"invalid property '%s'"), name);
	return (B_FALSE);
	}
	continue;
	}
	/*
	* "origin" is readonly but is used to receive datasets as
	* clones so we don't raise an error here
	*/
	if (prop == ZFS_PROP_ORIGIN)
	continue;

	/* encryption params have their own verification later */
	if (prop == ZFS_PROP_ENCRYPTION \|\|
	zfs_prop_encryption_key_param(prop))
	continue;

	/*
	* cannot override readonly, set-once and other specific
	* settable properties
	*/
	if (zfs_prop_readonly(prop) \|\| prop == ZFS_PROP_VERSION \|\|
	prop == ZFS_PROP_VOLSIZE) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"invalid property '%s'"), name);
	return (B_FALSE);
	}
	}

	return (B_TRUE);
	}

	static int
	zfs_receive_impl(libzfs_handle_t hdl, const char tosnap,
	const char originsnap, recvflags_t flags, int infd, const char *sendfs,
	nvlist_t stream_nv, avl_tree_t stream_avl, char **top_zfs,
	const char finalsnap, nvlist_t cmdprops)
	{
	int err;
	dmu_replay_record_t drr, drr_noswap;
	struct drr_begin *drrb = &drr.drr_u.drr_begin;
	char errbuf[1024];
	zio_cksum_t zcksum = { { 0 } };
	uint64_t featureflags;
	int hdrtype;

	(void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN,
	"cannot receive"));

	/* check cmdline props, raise an error if they cannot be received */
	if (!zfs_receive_checkprops(hdl, cmdprops, errbuf)) {
	return (zfs_error(hdl, EZFS_BADPROP, errbuf));
	}

	if (flags->isprefix &&
	!zfs_dataset_exists(hdl, tosnap, ZFS_TYPE_DATASET)) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "specified fs "
	"(%s) does not exist"), tosnap);
	return (zfs_error(hdl, EZFS_NOENT, errbuf));
	}
	if (originsnap &&
	!zfs_dataset_exists(hdl, originsnap, ZFS_TYPE_DATASET)) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "specified origin fs "
	"(%s) does not exist"), originsnap);
	return (zfs_error(hdl, EZFS_NOENT, errbuf));
	}

	/* read in the BEGIN record */
	if (0 != (err = recv_read(hdl, infd, &drr, sizeof (drr), B_FALSE,
	&zcksum)))
	return (err);

	if (drr.drr_type == DRR_END \|\| drr.drr_type == BSWAP_32(DRR_END)) {
	/* It's the double end record at the end of a package */
	return (ENODATA);
	}

	/* the kernel needs the non-byteswapped begin record */
	drr_noswap = drr;

	flags->byteswap = B_FALSE;
	if (drrb->drr_magic == BSWAP_64(DMU_BACKUP_MAGIC)) {
	/*
	* We computed the checksum in the wrong byteorder in
	* recv_read() above; do it again correctly.
	*/
	bzero(&zcksum, sizeof (zio_cksum_t));
	fletcher_4_incremental_byteswap(&drr, sizeof (drr), &zcksum);
	flags->byteswap = B_TRUE;

	drr.drr_type = BSWAP_32(drr.drr_type);
	drr.drr_payloadlen = BSWAP_32(drr.drr_payloadlen);
	drrb->drr_magic = BSWAP_64(drrb->drr_magic);
	drrb->drr_versioninfo = BSWAP_64(drrb->drr_versioninfo);
	drrb->drr_creation_time = BSWAP_64(drrb->drr_creation_time);
	drrb->drr_type = BSWAP_32(drrb->drr_type);
	drrb->drr_flags = BSWAP_32(drrb->drr_flags);
	drrb->drr_toguid = BSWAP_64(drrb->drr_toguid);
	drrb->drr_fromguid = BSWAP_64(drrb->drr_fromguid);
	}

	if (drrb->drr_magic != DMU_BACKUP_MAGIC \|\| drr.drr_type != DRR_BEGIN) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "invalid "
	"stream (bad magic number)"));
	return (zfs_error(hdl, EZFS_BADSTREAM, errbuf));
	}

	featureflags = DMU_GET_FEATUREFLAGS(drrb->drr_versioninfo);
	hdrtype = DMU_GET_STREAM_HDRTYPE(drrb->drr_versioninfo);

	if (!DMU_STREAM_SUPPORTED(featureflags) \|\|
	(hdrtype != DMU_SUBSTREAM && hdrtype != DMU_COMPOUNDSTREAM)) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"stream has unsupported feature, feature flags = %lx"),
	featureflags);
	return (zfs_error(hdl, EZFS_BADSTREAM, errbuf));
	}

	/* Holds feature is set once in the compound stream header. */
	if (featureflags & DMU_BACKUP_FEATURE_HOLDS)
	flags->holds = B_TRUE;

	if (strchr(drrb->drr_toname, '@') == NULL) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "invalid "
	"stream (bad snapshot name)"));
	return (zfs_error(hdl, EZFS_BADSTREAM, errbuf));
	}

	if (DMU_GET_STREAM_HDRTYPE(drrb->drr_versioninfo) == DMU_SUBSTREAM) {
	char nonpackage_sendfs[ZFS_MAX_DATASET_NAME_LEN];
	if (sendfs == NULL) {
	/*
	* We were not called from zfs_receive_package(). Get
	* the fs specified by 'zfs send'.
	*/
	char *cp;
	(void) strlcpy(nonpackage_sendfs,
	drr.drr_u.drr_begin.drr_toname,
	sizeof (nonpackage_sendfs));
	if ((cp = strchr(nonpackage_sendfs, '@')) != NULL)
	*cp = '\0';
	sendfs = nonpackage_sendfs;
	VERIFY(finalsnap == NULL);
	}
	return (zfs_receive_one(hdl, infd, tosnap, originsnap, flags,
	&drr, &drr_noswap, sendfs, stream_nv, stream_avl, top_zfs,
	finalsnap, cmdprops));
	} else {
	assert(DMU_GET_STREAM_HDRTYPE(drrb->drr_versioninfo) ==
	DMU_COMPOUNDSTREAM);
	return (zfs_receive_package(hdl, infd, tosnap, flags, &drr,
	&zcksum, top_zfs, cmdprops));
	}
	}

	/*
	* Restores a backup of tosnap from the file descriptor specified by infd.
	* Return 0 on total success, -2 if some things couldn't be
	* destroyed/renamed/promoted, -1 if some things couldn't be received.
	* (-1 will override -2, if -1 and the resumable flag was specified the
	* transfer can be resumed if the sending side supports it).
	*/
	int
	zfs_receive(libzfs_handle_t hdl, const char tosnap, nvlist_t *props,
	recvflags_t flags, int infd, avl_tree_t stream_avl)
	{
	char *top_zfs = NULL;
	int err;
	struct stat sb;
	char *originsnap = NULL;

	/*
	* The only way fstat can fail is if we do not have a valid file
	* descriptor.
	*/
	if (fstat(infd, &sb) == -1) {
	perror("fstat");
	return (-2);
	}

	/*
	* It is not uncommon for gigabytes to be processed in zfs receive.
	* Speculatively increase the buffer size if supported by the platform.
	*/
	if (S_ISFIFO(sb.st_mode))
	libzfs_set_pipe_max(infd);

	if (props) {
	err = nvlist_lookup_string(props, "origin", &originsnap);
	if (err && err != ENOENT)
	return (err);
	}

	err = zfs_receive_impl(hdl, tosnap, originsnap, flags, infd, NULL, NULL,
	stream_avl, &top_zfs, NULL, props);

	if (err == 0 && !flags->nomount && flags->domount && top_zfs) {
	zfs_handle_t *zhp = NULL;
	prop_changelist_t *clp = NULL;

	zhp = zfs_open(hdl, top_zfs,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME);
	if (zhp == NULL) {
	err = -1;
	goto out;
	} else {
	if (zhp->zfs_type == ZFS_TYPE_VOLUME) {
	zfs_close(zhp);
	goto out;
	}

	clp = changelist_gather(zhp, ZFS_PROP_MOUNTPOINT,
	CL_GATHER_MOUNT_ALWAYS,
	flags->forceunmount ? MS_FORCE : 0);
	zfs_close(zhp);
	if (clp == NULL) {
	err = -1;
	goto out;
	}

	/* mount and share received datasets */
	err = changelist_postfix(clp);
	changelist_free(clp);
	if (err != 0)
	err = -1;
	}
	}

	out:
	if (top_zfs)
	free(top_zfs);

	return (err);
	}
	Index: vendor-sys/openzfs/dist/lib/libzfs/libzfs_util.c
	===================================================================
	--- vendor-sys/openzfs/dist/lib/libzfs/libzfs_util.c (revision 366773)
	+++ vendor-sys/openzfs/dist/lib/libzfs/libzfs_util.c (revision 366774)
	@@ -1,2105 +1,2104 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/

	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright 2020 Joyent, Inc. All rights reserved.
	* Copyright (c) 2011, 2020 by Delphix. All rights reserved.
	* Copyright 2016 Igor Kozhukhov <ikozhukhov@gmail.com>
	* Copyright (c) 2017 Datto Inc.
	* Copyright (c) 2020 The FreeBSD Foundation
	*
	* Portions of this software were developed by Allan Jude
	* under sponsorship from the FreeBSD Foundation.
	*/

	/*
	* Internal utility routines for the ZFS library.
	*/

	#include <errno.h>
	#include <fcntl.h>
	#include <libintl.h>
	#include <stdarg.h>
	#include <stdio.h>
	#include <stdlib.h>
	#include <strings.h>
	#include <unistd.h>
	#include <math.h>
	#include <sys/stat.h>
	#include <sys/mnttab.h>
	#include <sys/mntent.h>
	#include <sys/types.h>
	#include <sys/wait.h>

	#include <libzfs.h>
	#include <libzfs_core.h>

	#include "libzfs_impl.h"
	#include "zfs_prop.h"
	#include "zfeature_common.h"
	#include <zfs_fletcher.h>
	#include <libzutil.h>

	/*
	* We only care about the scheme in order to match the scheme
	* with the handler. Each handler should validate the full URI
	* as necessary.
	*/
	#define URI_REGEX "^\$[A-Za-z][A-Za-z0-9+.\\-]*\$:"

	int
	libzfs_errno(libzfs_handle_t *hdl)
	{
	return (hdl->libzfs_error);
	}

	const char *
	libzfs_error_action(libzfs_handle_t *hdl)
	{
	return (hdl->libzfs_action);
	}

	const char *
	libzfs_error_description(libzfs_handle_t *hdl)
	{
	if (hdl->libzfs_desc[0] != '\0')
	return (hdl->libzfs_desc);

	switch (hdl->libzfs_error) {
	case EZFS_NOMEM:
	return (dgettext(TEXT_DOMAIN, "out of memory"));
	case EZFS_BADPROP:
	return (dgettext(TEXT_DOMAIN, "invalid property value"));
	case EZFS_PROPREADONLY:
	return (dgettext(TEXT_DOMAIN, "read-only property"));
	case EZFS_PROPTYPE:
	return (dgettext(TEXT_DOMAIN, "property doesn't apply to "
	"datasets of this type"));
	case EZFS_PROPNONINHERIT:
	return (dgettext(TEXT_DOMAIN, "property cannot be inherited"));
	case EZFS_PROPSPACE:
	return (dgettext(TEXT_DOMAIN, "invalid quota or reservation"));
	case EZFS_BADTYPE:
	return (dgettext(TEXT_DOMAIN, "operation not applicable to "
	"datasets of this type"));
	case EZFS_BUSY:
	return (dgettext(TEXT_DOMAIN, "pool or dataset is busy"));
	case EZFS_EXISTS:
	return (dgettext(TEXT_DOMAIN, "pool or dataset exists"));
	case EZFS_NOENT:
	return (dgettext(TEXT_DOMAIN, "no such pool or dataset"));
	case EZFS_BADSTREAM:
	return (dgettext(TEXT_DOMAIN, "invalid backup stream"));
	case EZFS_DSREADONLY:
	return (dgettext(TEXT_DOMAIN, "dataset is read-only"));
	case EZFS_VOLTOOBIG:
	return (dgettext(TEXT_DOMAIN, "volume size exceeds limit for "
	"this system"));
	case EZFS_INVALIDNAME:
	return (dgettext(TEXT_DOMAIN, "invalid name"));
	case EZFS_BADRESTORE:
	return (dgettext(TEXT_DOMAIN, "unable to restore to "
	"destination"));
	case EZFS_BADBACKUP:
	return (dgettext(TEXT_DOMAIN, "backup failed"));
	case EZFS_BADTARGET:
	return (dgettext(TEXT_DOMAIN, "invalid target vdev"));
	case EZFS_NODEVICE:
	return (dgettext(TEXT_DOMAIN, "no such device in pool"));
	case EZFS_BADDEV:
	return (dgettext(TEXT_DOMAIN, "invalid device"));
	case EZFS_NOREPLICAS:
	return (dgettext(TEXT_DOMAIN, "no valid replicas"));
	case EZFS_RESILVERING:
	return (dgettext(TEXT_DOMAIN, "currently resilvering"));
	case EZFS_BADVERSION:
	return (dgettext(TEXT_DOMAIN, "unsupported version or "
	"feature"));
	case EZFS_POOLUNAVAIL:
	return (dgettext(TEXT_DOMAIN, "pool is unavailable"));
	case EZFS_DEVOVERFLOW:
	return (dgettext(TEXT_DOMAIN, "too many devices in one vdev"));
	case EZFS_BADPATH:
	return (dgettext(TEXT_DOMAIN, "must be an absolute path"));
	case EZFS_CROSSTARGET:
	return (dgettext(TEXT_DOMAIN, "operation crosses datasets or "
	"pools"));
	case EZFS_ZONED:
	return (dgettext(TEXT_DOMAIN, "dataset in use by local zone"));
	case EZFS_MOUNTFAILED:
	return (dgettext(TEXT_DOMAIN, "mount failed"));
	case EZFS_UMOUNTFAILED:
	return (dgettext(TEXT_DOMAIN, "umount failed"));
	case EZFS_UNSHARENFSFAILED:
	return (dgettext(TEXT_DOMAIN, "unshare(1M) failed"));
	case EZFS_SHARENFSFAILED:
	return (dgettext(TEXT_DOMAIN, "share(1M) failed"));
	case EZFS_UNSHARESMBFAILED:
	return (dgettext(TEXT_DOMAIN, "smb remove share failed"));
	case EZFS_SHARESMBFAILED:
	return (dgettext(TEXT_DOMAIN, "smb add share failed"));
	case EZFS_PERM:
	return (dgettext(TEXT_DOMAIN, "permission denied"));
	case EZFS_NOSPC:
	return (dgettext(TEXT_DOMAIN, "out of space"));
	case EZFS_FAULT:
	return (dgettext(TEXT_DOMAIN, "bad address"));
	case EZFS_IO:
	return (dgettext(TEXT_DOMAIN, "I/O error"));
	case EZFS_INTR:
	return (dgettext(TEXT_DOMAIN, "signal received"));
	case EZFS_ISSPARE:
	return (dgettext(TEXT_DOMAIN, "device is reserved as a hot "
	"spare"));
	case EZFS_INVALCONFIG:
	return (dgettext(TEXT_DOMAIN, "invalid vdev configuration"));
	case EZFS_RECURSIVE:
	return (dgettext(TEXT_DOMAIN, "recursive dataset dependency"));
	case EZFS_NOHISTORY:
	return (dgettext(TEXT_DOMAIN, "no history available"));
	case EZFS_POOLPROPS:
	return (dgettext(TEXT_DOMAIN, "failed to retrieve "
	"pool properties"));
	case EZFS_POOL_NOTSUP:
	return (dgettext(TEXT_DOMAIN, "operation not supported "
	"on this type of pool"));
	case EZFS_POOL_INVALARG:
	return (dgettext(TEXT_DOMAIN, "invalid argument for "
	"this pool operation"));
	case EZFS_NAMETOOLONG:
	return (dgettext(TEXT_DOMAIN, "dataset name is too long"));
	case EZFS_OPENFAILED:
	return (dgettext(TEXT_DOMAIN, "open failed"));
	case EZFS_NOCAP:
	return (dgettext(TEXT_DOMAIN,
	"disk capacity information could not be retrieved"));
	case EZFS_LABELFAILED:
	return (dgettext(TEXT_DOMAIN, "write of label failed"));
	case EZFS_BADWHO:
	return (dgettext(TEXT_DOMAIN, "invalid user/group"));
	case EZFS_BADPERM:
	return (dgettext(TEXT_DOMAIN, "invalid permission"));
	case EZFS_BADPERMSET:
	return (dgettext(TEXT_DOMAIN, "invalid permission set name"));
	case EZFS_NODELEGATION:
	return (dgettext(TEXT_DOMAIN, "delegated administration is "
	"disabled on pool"));
	case EZFS_BADCACHE:
	return (dgettext(TEXT_DOMAIN, "invalid or missing cache file"));
	case EZFS_ISL2CACHE:
	return (dgettext(TEXT_DOMAIN, "device is in use as a cache"));
	case EZFS_VDEVNOTSUP:
	return (dgettext(TEXT_DOMAIN, "vdev specification is not "
	"supported"));
	case EZFS_NOTSUP:
	return (dgettext(TEXT_DOMAIN, "operation not supported "
	"on this dataset"));
	case EZFS_IOC_NOTSUPPORTED:
	return (dgettext(TEXT_DOMAIN, "operation not supported by "
	"zfs kernel module"));
	case EZFS_ACTIVE_SPARE:
	return (dgettext(TEXT_DOMAIN, "pool has active shared spare "
	"device"));
	case EZFS_UNPLAYED_LOGS:
	return (dgettext(TEXT_DOMAIN, "log device has unplayed intent "
	"logs"));
	case EZFS_REFTAG_RELE:
	return (dgettext(TEXT_DOMAIN, "no such tag on this dataset"));
	case EZFS_REFTAG_HOLD:
	return (dgettext(TEXT_DOMAIN, "tag already exists on this "
	"dataset"));
	case EZFS_TAGTOOLONG:
	return (dgettext(TEXT_DOMAIN, "tag too long"));
	case EZFS_PIPEFAILED:
	return (dgettext(TEXT_DOMAIN, "pipe create failed"));
	case EZFS_THREADCREATEFAILED:
	return (dgettext(TEXT_DOMAIN, "thread create failed"));
	case EZFS_POSTSPLIT_ONLINE:
	return (dgettext(TEXT_DOMAIN, "disk was split from this pool "
	"into a new one"));
	case EZFS_SCRUB_PAUSED:
	return (dgettext(TEXT_DOMAIN, "scrub is paused; "
	"use 'zpool scrub' to resume"));
	case EZFS_SCRUBBING:
	return (dgettext(TEXT_DOMAIN, "currently scrubbing; "
	"use 'zpool scrub -s' to cancel current scrub"));
	case EZFS_NO_SCRUB:
	return (dgettext(TEXT_DOMAIN, "there is no active scrub"));
	case EZFS_DIFF:
	return (dgettext(TEXT_DOMAIN, "unable to generate diffs"));
	case EZFS_DIFFDATA:
	return (dgettext(TEXT_DOMAIN, "invalid diff data"));
	case EZFS_POOLREADONLY:
	return (dgettext(TEXT_DOMAIN, "pool is read-only"));
	case EZFS_NO_PENDING:
	return (dgettext(TEXT_DOMAIN, "operation is not "
	"in progress"));
	case EZFS_CHECKPOINT_EXISTS:
	return (dgettext(TEXT_DOMAIN, "checkpoint exists"));
	case EZFS_DISCARDING_CHECKPOINT:
	return (dgettext(TEXT_DOMAIN, "currently discarding "
	"checkpoint"));
	case EZFS_NO_CHECKPOINT:
	return (dgettext(TEXT_DOMAIN, "checkpoint does not exist"));
	case EZFS_DEVRM_IN_PROGRESS:
	return (dgettext(TEXT_DOMAIN, "device removal in progress"));
	case EZFS_VDEV_TOO_BIG:
	return (dgettext(TEXT_DOMAIN, "device exceeds supported size"));
	case EZFS_ACTIVE_POOL:
	return (dgettext(TEXT_DOMAIN, "pool is imported on a "
	"different host"));
	case EZFS_CRYPTOFAILED:
	return (dgettext(TEXT_DOMAIN, "encryption failure"));
	case EZFS_TOOMANY:
	return (dgettext(TEXT_DOMAIN, "argument list too long"));
	case EZFS_INITIALIZING:
	return (dgettext(TEXT_DOMAIN, "currently initializing"));
	case EZFS_NO_INITIALIZE:
	return (dgettext(TEXT_DOMAIN, "there is no active "
	"initialization"));
	case EZFS_WRONG_PARENT:
	return (dgettext(TEXT_DOMAIN, "invalid parent dataset"));
	case EZFS_TRIMMING:
	return (dgettext(TEXT_DOMAIN, "currently trimming"));
	case EZFS_NO_TRIM:
	return (dgettext(TEXT_DOMAIN, "there is no active trim"));
	case EZFS_TRIM_NOTSUP:
	return (dgettext(TEXT_DOMAIN, "trim operations are not "
	"supported by this device"));
	case EZFS_NO_RESILVER_DEFER:
	return (dgettext(TEXT_DOMAIN, "this action requires the "
	"resilver_defer feature"));
	case EZFS_EXPORT_IN_PROGRESS:
	return (dgettext(TEXT_DOMAIN, "pool export in progress"));
	case EZFS_REBUILDING:
	return (dgettext(TEXT_DOMAIN, "currently sequentially "
	"resilvering"));
	case EZFS_UNKNOWN:
	return (dgettext(TEXT_DOMAIN, "unknown error"));
	default:
	assert(hdl->libzfs_error == 0);
	return (dgettext(TEXT_DOMAIN, "no error"));
	}
	}

	/PRINTFLIKE2/
	void
	zfs_error_aux(libzfs_handle_t hdl, const char fmt, ...)
	{
	va_list ap;

	va_start(ap, fmt);

	(void) vsnprintf(hdl->libzfs_desc, sizeof (hdl->libzfs_desc),
	fmt, ap);
	hdl->libzfs_desc_active = 1;

	va_end(ap);
	}

	static void
	zfs_verror(libzfs_handle_t hdl, int error, const char fmt, va_list ap)
	{
	(void) vsnprintf(hdl->libzfs_action, sizeof (hdl->libzfs_action),
	fmt, ap);
	hdl->libzfs_error = error;

	if (hdl->libzfs_desc_active)
	hdl->libzfs_desc_active = 0;
	else
	hdl->libzfs_desc[0] = '\0';

	if (hdl->libzfs_printerr) {
	if (error == EZFS_UNKNOWN) {
	(void) fprintf(stderr, dgettext(TEXT_DOMAIN, "internal "
	"error: %s: %s\n"), hdl->libzfs_action,
	libzfs_error_description(hdl));
	abort();
	}

	(void) fprintf(stderr, "%s: %s\n", hdl->libzfs_action,
	libzfs_error_description(hdl));
	if (error == EZFS_NOMEM)
	exit(1);
	}
	}

	int
	zfs_error(libzfs_handle_t hdl, int error, const char msg)
	{
	return (zfs_error_fmt(hdl, error, "%s", msg));
	}

	/PRINTFLIKE3/
	int
	zfs_error_fmt(libzfs_handle_t hdl, int error, const char fmt, ...)
	{
	va_list ap;

	va_start(ap, fmt);

	zfs_verror(hdl, error, fmt, ap);

	va_end(ap);

	return (-1);
	}

	static int
	zfs_common_error(libzfs_handle_t hdl, int error, const char fmt,
	va_list ap)
	{
	switch (error) {
	case EPERM:
	case EACCES:
	zfs_verror(hdl, EZFS_PERM, fmt, ap);
	return (-1);

	case ECANCELED:
	zfs_verror(hdl, EZFS_NODELEGATION, fmt, ap);
	return (-1);

	case EIO:
	zfs_verror(hdl, EZFS_IO, fmt, ap);
	return (-1);

	case EFAULT:
	zfs_verror(hdl, EZFS_FAULT, fmt, ap);
	return (-1);

	case EINTR:
	zfs_verror(hdl, EZFS_INTR, fmt, ap);
	return (-1);
	}

	return (0);
	}

	int
	zfs_standard_error(libzfs_handle_t hdl, int error, const char msg)
	{
	return (zfs_standard_error_fmt(hdl, error, "%s", msg));
	}

	/PRINTFLIKE3/
	int
	zfs_standard_error_fmt(libzfs_handle_t hdl, int error, const char fmt, ...)
	{
	va_list ap;

	va_start(ap, fmt);

	if (zfs_common_error(hdl, error, fmt, ap) != 0) {
	va_end(ap);
	return (-1);
	}

	switch (error) {
	case ENXIO:
	case ENODEV:
	case EPIPE:
	zfs_verror(hdl, EZFS_IO, fmt, ap);
	break;

	case ENOENT:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"dataset does not exist"));
	zfs_verror(hdl, EZFS_NOENT, fmt, ap);
	break;

	case ENOSPC:
	case EDQUOT:
	zfs_verror(hdl, EZFS_NOSPC, fmt, ap);
	break;

	case EEXIST:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"dataset already exists"));
	zfs_verror(hdl, EZFS_EXISTS, fmt, ap);
	break;

	case EBUSY:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"dataset is busy"));
	zfs_verror(hdl, EZFS_BUSY, fmt, ap);
	break;
	case EROFS:
	zfs_verror(hdl, EZFS_POOLREADONLY, fmt, ap);
	break;
	case ENAMETOOLONG:
	zfs_verror(hdl, EZFS_NAMETOOLONG, fmt, ap);
	break;
	case ENOTSUP:
	zfs_verror(hdl, EZFS_BADVERSION, fmt, ap);
	break;
	case EAGAIN:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"pool I/O is currently suspended"));
	zfs_verror(hdl, EZFS_POOLUNAVAIL, fmt, ap);
	break;
	case EREMOTEIO:
	zfs_verror(hdl, EZFS_ACTIVE_POOL, fmt, ap);
	break;
	case ZFS_ERR_UNKNOWN_SEND_STREAM_FEATURE:
	case ZFS_ERR_IOC_CMD_UNAVAIL:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "the loaded zfs "
	"module does not support this operation. A reboot may "
	"be required to enable this operation."));
	zfs_verror(hdl, EZFS_IOC_NOTSUPPORTED, fmt, ap);
	break;
	case ZFS_ERR_IOC_ARG_UNAVAIL:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "the loaded zfs "
	"module does not support an option for this operation. "
	"A reboot may be required to enable this option."));
	zfs_verror(hdl, EZFS_IOC_NOTSUPPORTED, fmt, ap);
	break;
	case ZFS_ERR_IOC_ARG_REQUIRED:
	case ZFS_ERR_IOC_ARG_BADTYPE:
	zfs_verror(hdl, EZFS_IOC_NOTSUPPORTED, fmt, ap);
	break;
	case ZFS_ERR_WRONG_PARENT:
	zfs_verror(hdl, EZFS_WRONG_PARENT, fmt, ap);
	break;
	case ZFS_ERR_BADPROP:
	zfs_verror(hdl, EZFS_BADPROP, fmt, ap);
	break;
	default:
	zfs_error_aux(hdl, strerror(error));
	zfs_verror(hdl, EZFS_UNKNOWN, fmt, ap);
	break;
	}

	va_end(ap);
	return (-1);
	}

	void
	zfs_setprop_error(libzfs_handle_t *hdl, zfs_prop_t prop, int err,
	char *errbuf)
	{
	switch (err) {

	case ENOSPC:
	/*
	* For quotas and reservations, ENOSPC indicates
	* something different; setting a quota or reservation
	* doesn't use any disk space.
	*/
	switch (prop) {
	case ZFS_PROP_QUOTA:
	case ZFS_PROP_REFQUOTA:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"size is less than current used or "
	"reserved space"));
	(void) zfs_error(hdl, EZFS_PROPSPACE, errbuf);
	break;

	case ZFS_PROP_RESERVATION:
	case ZFS_PROP_REFRESERVATION:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"size is greater than available space"));
	(void) zfs_error(hdl, EZFS_PROPSPACE, errbuf);
	break;

	default:
	(void) zfs_standard_error(hdl, err, errbuf);
	break;
	}
	break;

	case EBUSY:
	(void) zfs_standard_error(hdl, EBUSY, errbuf);
	break;

	case EROFS:
	(void) zfs_error(hdl, EZFS_DSREADONLY, errbuf);
	break;

	case E2BIG:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"property value too long"));
	(void) zfs_error(hdl, EZFS_BADPROP, errbuf);
	break;

	case ENOTSUP:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"pool and or dataset must be upgraded to set this "
	"property or value"));
	(void) zfs_error(hdl, EZFS_BADVERSION, errbuf);
	break;

	case ERANGE:
	if (prop == ZFS_PROP_COMPRESSION \|\|
	prop == ZFS_PROP_DNODESIZE \|\|
	prop == ZFS_PROP_RECORDSIZE) {
	(void) zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"property setting is not allowed on "
	"bootable datasets"));
	(void) zfs_error(hdl, EZFS_NOTSUP, errbuf);
	} else if (prop == ZFS_PROP_CHECKSUM \|\|
	prop == ZFS_PROP_DEDUP) {
	(void) zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"property setting is not allowed on "
	"root pools"));
	(void) zfs_error(hdl, EZFS_NOTSUP, errbuf);
	} else {
	(void) zfs_standard_error(hdl, err, errbuf);
	}
	break;

	case EINVAL:
	if (prop == ZPROP_INVAL) {
	(void) zfs_error(hdl, EZFS_BADPROP, errbuf);
	} else {
	(void) zfs_standard_error(hdl, err, errbuf);
	}
	break;

	case ZFS_ERR_BADPROP:
	(void) zfs_error(hdl, EZFS_BADPROP, errbuf);
	break;

	case EACCES:
	if (prop == ZFS_PROP_KEYLOCATION) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"keylocation may only be set on encryption roots"));
	(void) zfs_error(hdl, EZFS_BADPROP, errbuf);
	} else {
	(void) zfs_standard_error(hdl, err, errbuf);
	}
	break;

	case EOVERFLOW:
	/*
	* This platform can't address a volume this big.
	*/
	#ifdef _ILP32
	if (prop == ZFS_PROP_VOLSIZE) {
	(void) zfs_error(hdl, EZFS_VOLTOOBIG, errbuf);
	break;
	}
	#endif
	/* FALLTHROUGH */
	default:
	(void) zfs_standard_error(hdl, err, errbuf);
	}
	}

	int
	zpool_standard_error(libzfs_handle_t hdl, int error, const char msg)
	{
	return (zpool_standard_error_fmt(hdl, error, "%s", msg));
	}

	/PRINTFLIKE3/
	int
	zpool_standard_error_fmt(libzfs_handle_t hdl, int error, const char fmt, ...)
	{
	va_list ap;

	va_start(ap, fmt);

	if (zfs_common_error(hdl, error, fmt, ap) != 0) {
	va_end(ap);
	return (-1);
	}

	switch (error) {
	case ENODEV:
	zfs_verror(hdl, EZFS_NODEVICE, fmt, ap);
	break;

	case ENOENT:
	zfs_error_aux(hdl,
	dgettext(TEXT_DOMAIN, "no such pool or dataset"));
	zfs_verror(hdl, EZFS_NOENT, fmt, ap);
	break;

	case EEXIST:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"pool already exists"));
	zfs_verror(hdl, EZFS_EXISTS, fmt, ap);
	break;

	case EBUSY:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "pool is busy"));
	zfs_verror(hdl, EZFS_BUSY, fmt, ap);
	break;

	/* There is no pending operation to cancel */
	case ENOTACTIVE:
	zfs_verror(hdl, EZFS_NO_PENDING, fmt, ap);
	break;

	case ENXIO:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"one or more devices is currently unavailable"));
	zfs_verror(hdl, EZFS_BADDEV, fmt, ap);
	break;

	case ENAMETOOLONG:
	zfs_verror(hdl, EZFS_DEVOVERFLOW, fmt, ap);
	break;

	case ENOTSUP:
	zfs_verror(hdl, EZFS_POOL_NOTSUP, fmt, ap);
	break;

	case EINVAL:
	zfs_verror(hdl, EZFS_POOL_INVALARG, fmt, ap);
	break;

	case ENOSPC:
	case EDQUOT:
	zfs_verror(hdl, EZFS_NOSPC, fmt, ap);
	return (-1);

	case EAGAIN:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"pool I/O is currently suspended"));
	zfs_verror(hdl, EZFS_POOLUNAVAIL, fmt, ap);
	break;

	case EROFS:
	zfs_verror(hdl, EZFS_POOLREADONLY, fmt, ap);
	break;
	case EDOM:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"block size out of range or does not match"));
	zfs_verror(hdl, EZFS_BADPROP, fmt, ap);
	break;
	case EREMOTEIO:
	zfs_verror(hdl, EZFS_ACTIVE_POOL, fmt, ap);
	break;
	case ZFS_ERR_CHECKPOINT_EXISTS:
	zfs_verror(hdl, EZFS_CHECKPOINT_EXISTS, fmt, ap);
	break;
	case ZFS_ERR_DISCARDING_CHECKPOINT:
	zfs_verror(hdl, EZFS_DISCARDING_CHECKPOINT, fmt, ap);
	break;
	case ZFS_ERR_NO_CHECKPOINT:
	zfs_verror(hdl, EZFS_NO_CHECKPOINT, fmt, ap);
	break;
	case ZFS_ERR_DEVRM_IN_PROGRESS:
	zfs_verror(hdl, EZFS_DEVRM_IN_PROGRESS, fmt, ap);
	break;
	case ZFS_ERR_VDEV_TOO_BIG:
	zfs_verror(hdl, EZFS_VDEV_TOO_BIG, fmt, ap);
	break;
	case ZFS_ERR_EXPORT_IN_PROGRESS:
	zfs_verror(hdl, EZFS_EXPORT_IN_PROGRESS, fmt, ap);
	break;
	case ZFS_ERR_RESILVER_IN_PROGRESS:
	zfs_verror(hdl, EZFS_RESILVERING, fmt, ap);
	break;
	case ZFS_ERR_REBUILD_IN_PROGRESS:
	zfs_verror(hdl, EZFS_REBUILDING, fmt, ap);
	break;
	case ZFS_ERR_BADPROP:
	zfs_verror(hdl, EZFS_BADPROP, fmt, ap);
	break;
	case ZFS_ERR_IOC_CMD_UNAVAIL:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "the loaded zfs "
	"module does not support this operation. A reboot may "
	"be required to enable this operation."));
	zfs_verror(hdl, EZFS_IOC_NOTSUPPORTED, fmt, ap);
	break;
	case ZFS_ERR_IOC_ARG_UNAVAIL:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "the loaded zfs "
	"module does not support an option for this operation. "
	"A reboot may be required to enable this option."));
	zfs_verror(hdl, EZFS_IOC_NOTSUPPORTED, fmt, ap);
	break;
	case ZFS_ERR_IOC_ARG_REQUIRED:
	case ZFS_ERR_IOC_ARG_BADTYPE:
	zfs_verror(hdl, EZFS_IOC_NOTSUPPORTED, fmt, ap);
	break;
	default:
	zfs_error_aux(hdl, strerror(error));
	zfs_verror(hdl, EZFS_UNKNOWN, fmt, ap);
	}

	va_end(ap);
	return (-1);
	}

	/*
	* Display an out of memory error message and abort the current program.
	*/
	int
	no_memory(libzfs_handle_t *hdl)
	{
	return (zfs_error(hdl, EZFS_NOMEM, "internal error"));
	}

	/*
	* A safe form of malloc() which will die if the allocation fails.
	*/
	void *
	zfs_alloc(libzfs_handle_t *hdl, size_t size)
	{
	void *data;

	if ((data = calloc(1, size)) == NULL)
	(void) no_memory(hdl);

	return (data);
	}

	/*
	* A safe form of asprintf() which will die if the allocation fails.
	*/
	/PRINTFLIKE2/
	char *
	zfs_asprintf(libzfs_handle_t hdl, const char fmt, ...)
	{
	va_list ap;
	char *ret;
	int err;

	va_start(ap, fmt);

	err = vasprintf(&ret, fmt, ap);

	va_end(ap);

	if (err < 0)
	(void) no_memory(hdl);

	return (ret);
	}

	/*
	* A safe form of realloc(), which also zeroes newly allocated space.
	*/
	void *
	zfs_realloc(libzfs_handle_t hdl, void ptr, size_t oldsize, size_t newsize)
	{
	void *ret;

	if ((ret = realloc(ptr, newsize)) == NULL) {
	(void) no_memory(hdl);
	return (NULL);
	}

	bzero((char *)ret + oldsize, (newsize - oldsize));
	return (ret);
	}

	/*
	* A safe form of strdup() which will die if the allocation fails.
	*/
	char *
	zfs_strdup(libzfs_handle_t hdl, const char str)
	{
	char *ret;

	if ((ret = strdup(str)) == NULL)
	(void) no_memory(hdl);

	return (ret);
	}

	void
	libzfs_print_on_error(libzfs_handle_t *hdl, boolean_t printerr)
	{
	hdl->libzfs_printerr = printerr;
	}

	/*
	* Read lines from an open file descriptor and store them in an array of
	* strings until EOF. lines[] will be allocated and populated with all the
	* lines read. All newlines are replaced with NULL terminators for
	* convenience. lines[] must be freed after use with libzfs_free_str_array().
	*
	* Returns the number of lines read.
	*/
	static int
	libzfs_read_stdout_from_fd(int fd, char **lines[])
	{

	FILE *fp;
	int lines_cnt = 0;
	size_t len = 0;
	char *line = NULL;
	char tmp_lines = NULL, tmp;
	char *nl = NULL;
	int rc;

	fp = fdopen(fd, "r");
	if (fp == NULL)
	return (0);
	while (1) {
	rc = getline(&line, &len, fp);
	if (rc == -1)
	break;

	tmp = realloc(tmp_lines, sizeof (tmp_lines) (lines_cnt + 1));
	if (tmp == NULL) {
	/* Return the lines we were able to process */
	break;
	}
	tmp_lines = tmp;

	/* Terminate newlines */
	if ((nl = strchr(line, '\n')) != NULL)
	*nl = '\0';
	tmp_lines[lines_cnt] = line;
	lines_cnt++;
	line = NULL;
	}
	fclose(fp);
	*lines = tmp_lines;
	return (lines_cnt);
	}

	static int
	libzfs_run_process_impl(const char path, char argv[], char *env[], int flags,
	char *lines[], int lines_cnt)
	{
	pid_t pid;
	int error, devnull_fd;
	int link[2];

	/*
	* Setup a pipe between our child and parent process if we're
	* reading stdout.
	*/
	if ((lines != NULL) && pipe(link) == -1)
	return (-EPIPE);

	pid = vfork();
	if (pid == 0) {
	/* Child process */
	devnull_fd = open("/dev/null", O_WRONLY);

	if (devnull_fd < 0)
	_exit(-1);

	if (!(flags & STDOUT_VERBOSE) && (lines == NULL))
	(void) dup2(devnull_fd, STDOUT_FILENO);
	else if (lines != NULL) {
	/* Save the output to lines[] */
	dup2(link[1], STDOUT_FILENO);
	close(link[0]);
	close(link[1]);
	}

	if (!(flags & STDERR_VERBOSE))
	(void) dup2(devnull_fd, STDERR_FILENO);

	close(devnull_fd);

	if (flags & NO_DEFAULT_PATH) {
	if (env == NULL)
	execv(path, argv);
	else
	execve(path, argv, env);
	} else {
	if (env == NULL)
	execvp(path, argv);
	else
	execvpe(path, argv, env);
	}

	_exit(-1);
	} else if (pid > 0) {
	/* Parent process */
	int status;

	while ((error = waitpid(pid, &status, 0)) == -1 &&
	errno == EINTR) { }
	if (error < 0 \|\| !WIFEXITED(status))
	return (-1);

	if (lines != NULL) {
	close(link[1]);
	*lines_cnt = libzfs_read_stdout_from_fd(link[0], lines);
	}
	return (WEXITSTATUS(status));
	}

	return (-1);
	}

	int
	libzfs_run_process(const char path, char argv[], int flags)
	{
	return (libzfs_run_process_impl(path, argv, NULL, flags, NULL, NULL));
	}

	/*
	* Run a command and store its stdout lines in an array of strings (lines[]).
	* lines[] is allocated and populated for you, and the number of lines is set in
	* lines_cnt. lines[] must be freed after use with libzfs_free_str_array().
	* All newlines (\n) in lines[] are terminated for convenience.
	*/
	int
	libzfs_run_process_get_stdout(const char path, char argv[], char *env[],
	char *lines[], int lines_cnt)
	{
	return (libzfs_run_process_impl(path, argv, env, 0, lines, lines_cnt));
	}

	/*
	* Same as libzfs_run_process_get_stdout(), but run without $PATH set. This
	* means that *path needs to be the full path to the executable.
	*/
	int
	libzfs_run_process_get_stdout_nopath(const char path, char argv[],
	char env[], char lines[], int lines_cnt)
	{
	return (libzfs_run_process_impl(path, argv, env, NO_DEFAULT_PATH,
	lines, lines_cnt));
	}

	/*
	* Free an array of strings. Free both the strings contained in the array and
	* the array itself.
	*/
	void
	libzfs_free_str_array(char **strs, int count)
	{
	while (--count >= 0)
	free(strs[count]);

	free(strs);
	}

	/*
	* Returns 1 if environment variable is set to "YES", "yes", "ON", "on", or
	* a non-zero number.
	*
	* Returns 0 otherwise.
	*/
	int
	libzfs_envvar_is_set(char *envvar)
	{
	char *env = getenv(envvar);
	if (env && (strtoul(env, NULL, 0) > 0 \|\|
	(!strncasecmp(env, "YES", 3) && strnlen(env, 4) == 3) \|\|
	(!strncasecmp(env, "ON", 2) && strnlen(env, 3) == 2)))
	return (1);

	return (0);
	}

	libzfs_handle_t *
	libzfs_init(void)
	{
	libzfs_handle_t *hdl;
	int error;
	char *env;

	- error = libzfs_load_module();
	- if (error) {
	+ if ((error = libzfs_load_module()) != 0) {
	errno = error;
	return (NULL);
	}

	if ((hdl = calloc(1, sizeof (libzfs_handle_t))) == NULL) {
	return (NULL);
	}

	if (regcomp(&hdl->libzfs_urire, URI_REGEX, 0) != 0) {
	free(hdl);
	return (NULL);
	}

	if ((hdl->libzfs_fd = open(ZFS_DEV, O_RDWR\|O_EXCL)) < 0) {
	free(hdl);
	return (NULL);
	}

	#ifdef HAVE_SETMNTENT
	if ((hdl->libzfs_mnttab = setmntent(MNTTAB, "r")) == NULL) {
	#else
	if ((hdl->libzfs_mnttab = fopen(MNTTAB, "r")) == NULL) {
	#endif
	(void) close(hdl->libzfs_fd);
	free(hdl);
	return (NULL);
	}

	if (libzfs_core_init() != 0) {
	(void) close(hdl->libzfs_fd);
	(void) fclose(hdl->libzfs_mnttab);
	free(hdl);
	return (NULL);
	}

	zfs_prop_init();
	zpool_prop_init();
	zpool_feature_init();
	libzfs_mnttab_init(hdl);
	fletcher_4_init();

	if (getenv("ZFS_PROP_DEBUG") != NULL) {
	hdl->libzfs_prop_debug = B_TRUE;
	}
	if ((env = getenv("ZFS_SENDRECV_MAX_NVLIST")) != NULL) {
	if ((error = zfs_nicestrtonum(hdl, env,
	&hdl->libzfs_max_nvlist))) {
	errno = error;
	(void) close(hdl->libzfs_fd);
	(void) fclose(hdl->libzfs_mnttab);
	free(hdl);
	return (NULL);
	}
	} else {
	hdl->libzfs_max_nvlist = (SPA_MAXBLOCKSIZE * 4);
	}

	/*
	* For testing, remove some settable properties and features
	*/
	if (libzfs_envvar_is_set("ZFS_SYSFS_PROP_SUPPORT_TEST")) {
	zprop_desc_t *proptbl;

	proptbl = zpool_prop_get_table();
	proptbl[ZPOOL_PROP_COMMENT].pd_zfs_mod_supported = B_FALSE;

	proptbl = zfs_prop_get_table();
	proptbl[ZFS_PROP_DNODESIZE].pd_zfs_mod_supported = B_FALSE;

	zfeature_info_t *ftbl = spa_feature_table;
	ftbl[SPA_FEATURE_LARGE_BLOCKS].fi_zfs_mod_supported = B_FALSE;
	}

	return (hdl);
	}

	void
	libzfs_fini(libzfs_handle_t *hdl)
	{
	(void) close(hdl->libzfs_fd);
	if (hdl->libzfs_mnttab)
	#ifdef HAVE_SETMNTENT
	(void) endmntent(hdl->libzfs_mnttab);
	#else
	(void) fclose(hdl->libzfs_mnttab);
	#endif
	zpool_free_handles(hdl);
	namespace_clear(hdl);
	libzfs_mnttab_fini(hdl);
	libzfs_core_fini();
	regfree(&hdl->libzfs_urire);
	fletcher_4_fini();
	free(hdl);
	}

	libzfs_handle_t *
	zpool_get_handle(zpool_handle_t *zhp)
	{
	return (zhp->zpool_hdl);
	}

	libzfs_handle_t *
	zfs_get_handle(zfs_handle_t *zhp)
	{
	return (zhp->zfs_hdl);
	}

	zpool_handle_t *
	zfs_get_pool_handle(const zfs_handle_t *zhp)
	{
	return (zhp->zpool_hdl);
	}

	/*
	* Given a name, determine whether or not it's a valid path
	* (starts with '/' or "./"). If so, walk the mnttab trying
	* to match the device number. If not, treat the path as an
	* fs/vol/snap/bkmark name.
	*/
	zfs_handle_t *
	zfs_path_to_zhandle(libzfs_handle_t hdl, const char path, zfs_type_t argtype)
	{
	struct stat64 statbuf;
	struct extmnttab entry;

	if (path[0] != '/' && strncmp(path, "./", strlen("./")) != 0) {
	/*
	* It's not a valid path, assume it's a name of type 'argtype'.
	*/
	return (zfs_open(hdl, path, argtype));
	}

	/* Reopen MNTTAB to prevent reading stale data from open file */
	if (freopen(MNTTAB, "r", hdl->libzfs_mnttab) == NULL)
	return (NULL);

	if (getextmntent(path, &entry, &statbuf) != 0)
	return (NULL);

	if (strcmp(entry.mnt_fstype, MNTTYPE_ZFS) != 0) {
	(void) fprintf(stderr, gettext("'%s': not a ZFS filesystem\n"),
	path);
	return (NULL);
	}

	return (zfs_open(hdl, entry.mnt_special, ZFS_TYPE_FILESYSTEM));
	}

	/*
	* Initialize the zc_nvlist_dst member to prepare for receiving an nvlist from
	* an ioctl().
	*/
	int
	zcmd_alloc_dst_nvlist(libzfs_handle_t hdl, zfs_cmd_t zc, size_t len)
	{
	if (len == 0)
	len = 256 * 1024;
	zc->zc_nvlist_dst_size = len;
	zc->zc_nvlist_dst =
	(uint64_t)(uintptr_t)zfs_alloc(hdl, zc->zc_nvlist_dst_size);
	if (zc->zc_nvlist_dst == 0)
	return (-1);

	return (0);
	}

	/*
	* Called when an ioctl() which returns an nvlist fails with ENOMEM. This will
	* expand the nvlist to the size specified in 'zc_nvlist_dst_size', which was
	* filled in by the kernel to indicate the actual required size.
	*/
	int
	zcmd_expand_dst_nvlist(libzfs_handle_t hdl, zfs_cmd_t zc)
	{
	free((void *)(uintptr_t)zc->zc_nvlist_dst);
	zc->zc_nvlist_dst =
	(uint64_t)(uintptr_t)zfs_alloc(hdl, zc->zc_nvlist_dst_size);
	if (zc->zc_nvlist_dst == 0)
	return (-1);

	return (0);
	}

	/*
	* Called to free the src and dst nvlists stored in the command structure.
	*/
	void
	zcmd_free_nvlists(zfs_cmd_t *zc)
	{
	free((void *)(uintptr_t)zc->zc_nvlist_conf);
	free((void *)(uintptr_t)zc->zc_nvlist_src);
	free((void *)(uintptr_t)zc->zc_nvlist_dst);
	zc->zc_nvlist_conf = 0;
	zc->zc_nvlist_src = 0;
	zc->zc_nvlist_dst = 0;
	}

	static int
	zcmd_write_nvlist_com(libzfs_handle_t hdl, uint64_t outnv, uint64_t *outlen,
	nvlist_t *nvl)
	{
	char *packed;
	size_t len;

	verify(nvlist_size(nvl, &len, NV_ENCODE_NATIVE) == 0);

	if ((packed = zfs_alloc(hdl, len)) == NULL)
	return (-1);

	verify(nvlist_pack(nvl, &packed, &len, NV_ENCODE_NATIVE, 0) == 0);

	*outnv = (uint64_t)(uintptr_t)packed;
	*outlen = len;

	return (0);
	}

	int
	zcmd_write_conf_nvlist(libzfs_handle_t hdl, zfs_cmd_t zc, nvlist_t *nvl)
	{
	return (zcmd_write_nvlist_com(hdl, &zc->zc_nvlist_conf,
	&zc->zc_nvlist_conf_size, nvl));
	}

	int
	zcmd_write_src_nvlist(libzfs_handle_t hdl, zfs_cmd_t zc, nvlist_t *nvl)
	{
	return (zcmd_write_nvlist_com(hdl, &zc->zc_nvlist_src,
	&zc->zc_nvlist_src_size, nvl));
	}

	/*
	* Unpacks an nvlist from the ZFS ioctl command structure.
	*/
	int
	zcmd_read_dst_nvlist(libzfs_handle_t hdl, zfs_cmd_t zc, nvlist_t **nvlp)
	{
	if (nvlist_unpack((void *)(uintptr_t)zc->zc_nvlist_dst,
	zc->zc_nvlist_dst_size, nvlp, 0) != 0)
	return (no_memory(hdl));

	return (0);
	}

	/*
	* ================================================================
	* API shared by zfs and zpool property management
	* ================================================================
	*/

	static void
	zprop_print_headers(zprop_get_cbdata_t *cbp, zfs_type_t type)
	{
	zprop_list_t *pl = cbp->cb_proplist;
	int i;
	char *title;
	size_t len;

	cbp->cb_first = B_FALSE;
	if (cbp->cb_scripted)
	return;

	/*
	* Start with the length of the column headers.
	*/
	cbp->cb_colwidths[GET_COL_NAME] = strlen(dgettext(TEXT_DOMAIN, "NAME"));
	cbp->cb_colwidths[GET_COL_PROPERTY] = strlen(dgettext(TEXT_DOMAIN,
	"PROPERTY"));
	cbp->cb_colwidths[GET_COL_VALUE] = strlen(dgettext(TEXT_DOMAIN,
	"VALUE"));
	cbp->cb_colwidths[GET_COL_RECVD] = strlen(dgettext(TEXT_DOMAIN,
	"RECEIVED"));
	cbp->cb_colwidths[GET_COL_SOURCE] = strlen(dgettext(TEXT_DOMAIN,
	"SOURCE"));

	/* first property is always NAME */
	assert(cbp->cb_proplist->pl_prop ==
	((type == ZFS_TYPE_POOL) ? ZPOOL_PROP_NAME : ZFS_PROP_NAME));

	/*
	* Go through and calculate the widths for each column. For the
	* 'source' column, we kludge it up by taking the worst-case scenario of
	* inheriting from the longest name. This is acceptable because in the
	* majority of cases 'SOURCE' is the last column displayed, and we don't
	* use the width anyway. Note that the 'VALUE' column can be oversized,
	* if the name of the property is much longer than any values we find.
	*/
	for (pl = cbp->cb_proplist; pl != NULL; pl = pl->pl_next) {
	/*
	* 'PROPERTY' column
	*/
	if (pl->pl_prop != ZPROP_INVAL) {
	const char *propname = (type == ZFS_TYPE_POOL) ?
	zpool_prop_to_name(pl->pl_prop) :
	zfs_prop_to_name(pl->pl_prop);

	len = strlen(propname);
	if (len > cbp->cb_colwidths[GET_COL_PROPERTY])
	cbp->cb_colwidths[GET_COL_PROPERTY] = len;
	} else {
	len = strlen(pl->pl_user_prop);
	if (len > cbp->cb_colwidths[GET_COL_PROPERTY])
	cbp->cb_colwidths[GET_COL_PROPERTY] = len;
	}

	/*
	* 'VALUE' column. The first property is always the 'name'
	* property that was tacked on either by /sbin/zfs's
	* zfs_do_get() or when calling zprop_expand_list(), so we
	* ignore its width. If the user specified the name property
	* to display, then it will be later in the list in any case.
	*/
	if (pl != cbp->cb_proplist &&
	pl->pl_width > cbp->cb_colwidths[GET_COL_VALUE])
	cbp->cb_colwidths[GET_COL_VALUE] = pl->pl_width;

	/* 'RECEIVED' column. */
	if (pl != cbp->cb_proplist &&
	pl->pl_recvd_width > cbp->cb_colwidths[GET_COL_RECVD])
	cbp->cb_colwidths[GET_COL_RECVD] = pl->pl_recvd_width;

	/*
	* 'NAME' and 'SOURCE' columns
	*/
	if (pl->pl_prop == (type == ZFS_TYPE_POOL ? ZPOOL_PROP_NAME :
	ZFS_PROP_NAME) &&
	pl->pl_width > cbp->cb_colwidths[GET_COL_NAME]) {
	cbp->cb_colwidths[GET_COL_NAME] = pl->pl_width;
	cbp->cb_colwidths[GET_COL_SOURCE] = pl->pl_width +
	strlen(dgettext(TEXT_DOMAIN, "inherited from"));
	}
	}

	/*
	* Now go through and print the headers.
	*/
	for (i = 0; i < ZFS_GET_NCOLS; i++) {
	switch (cbp->cb_columns[i]) {
	case GET_COL_NAME:
	title = dgettext(TEXT_DOMAIN, "NAME");
	break;
	case GET_COL_PROPERTY:
	title = dgettext(TEXT_DOMAIN, "PROPERTY");
	break;
	case GET_COL_VALUE:
	title = dgettext(TEXT_DOMAIN, "VALUE");
	break;
	case GET_COL_RECVD:
	title = dgettext(TEXT_DOMAIN, "RECEIVED");
	break;
	case GET_COL_SOURCE:
	title = dgettext(TEXT_DOMAIN, "SOURCE");
	break;
	default:
	title = NULL;
	}

	if (title != NULL) {
	if (i == (ZFS_GET_NCOLS - 1) \|\|
	cbp->cb_columns[i + 1] == GET_COL_NONE)
	(void) printf("%s", title);
	else
	(void) printf("%-*s ",
	cbp->cb_colwidths[cbp->cb_columns[i]],
	title);
	}
	}
	(void) printf("\n");
	}

	/*
	* Display a single line of output, according to the settings in the callback
	* structure.
	*/
	void
	zprop_print_one_property(const char name, zprop_get_cbdata_t cbp,
	const char propname, const char value, zprop_source_t sourcetype,
	const char source, const char recvd_value)
	{
	int i;
	const char *str = NULL;
	char buf[128];

	/*
	* Ignore those source types that the user has chosen to ignore.
	*/
	if ((sourcetype & cbp->cb_sources) == 0)
	return;

	if (cbp->cb_first)
	zprop_print_headers(cbp, cbp->cb_type);

	for (i = 0; i < ZFS_GET_NCOLS; i++) {
	switch (cbp->cb_columns[i]) {
	case GET_COL_NAME:
	str = name;
	break;

	case GET_COL_PROPERTY:
	str = propname;
	break;

	case GET_COL_VALUE:
	str = value;
	break;

	case GET_COL_SOURCE:
	switch (sourcetype) {
	case ZPROP_SRC_NONE:
	str = "-";
	break;

	case ZPROP_SRC_DEFAULT:
	str = "default";
	break;

	case ZPROP_SRC_LOCAL:
	str = "local";
	break;

	case ZPROP_SRC_TEMPORARY:
	str = "temporary";
	break;

	case ZPROP_SRC_INHERITED:
	(void) snprintf(buf, sizeof (buf),
	"inherited from %s", source);
	str = buf;
	break;
	case ZPROP_SRC_RECEIVED:
	str = "received";
	break;

	default:
	str = NULL;
	assert(!"unhandled zprop_source_t");
	}
	break;

	case GET_COL_RECVD:
	str = (recvd_value == NULL ? "-" : recvd_value);
	break;

	default:
	continue;
	}

	if (i == (ZFS_GET_NCOLS - 1) \|\|
	cbp->cb_columns[i + 1] == GET_COL_NONE)
	(void) printf("%s", str);
	else if (cbp->cb_scripted)
	(void) printf("%s\t", str);
	else
	(void) printf("%-*s ",
	cbp->cb_colwidths[cbp->cb_columns[i]],
	str);
	}

	(void) printf("\n");
	}

	/*
	* Given a numeric suffix, convert the value into a number of bits that the
	* resulting value must be shifted.
	*/
	static int
	str2shift(libzfs_handle_t hdl, const char buf)
	{
	const char *ends = "BKMGTPEZ";
	int i;

	if (buf[0] == '\0')
	return (0);
	for (i = 0; i < strlen(ends); i++) {
	if (toupper(buf[0]) == ends[i])
	break;
	}
	if (i == strlen(ends)) {
	if (hdl)
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"invalid numeric suffix '%s'"), buf);
	return (-1);
	}

	/*
	* Allow 'G' = 'GB' = 'GiB', case-insensitively.
	* However, 'BB' and 'BiB' are disallowed.
	*/
	if (buf[1] == '\0' \|\|
	(toupper(buf[0]) != 'B' &&
	((toupper(buf[1]) == 'B' && buf[2] == '\0') \|\|
	(toupper(buf[1]) == 'I' && toupper(buf[2]) == 'B' &&
	buf[3] == '\0'))))
	return (10 * i);

	if (hdl)
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"invalid numeric suffix '%s'"), buf);
	return (-1);
	}

	/*
	* Convert a string of the form '100G' into a real number. Used when setting
	* properties or creating a volume. 'buf' is used to place an extended error
	* message for the caller to use.
	*/
	int
	zfs_nicestrtonum(libzfs_handle_t hdl, const char value, uint64_t *num)
	{
	char *end;
	int shift;

	*num = 0;

	/* Check to see if this looks like a number. */
	if ((value[0] < '0' \|\| value[0] > '9') && value[0] != '.') {
	if (hdl)
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"bad numeric value '%s'"), value);
	return (-1);
	}

	/* Rely on strtoull() to process the numeric portion. */
	errno = 0;
	*num = strtoull(value, &end, 10);

	/*
	* Check for ERANGE, which indicates that the value is too large to fit
	* in a 64-bit value.
	*/
	if (errno == ERANGE) {
	if (hdl)
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"numeric value is too large"));
	return (-1);
	}

	/*
	* If we have a decimal value, then do the computation with floating
	* point arithmetic. Otherwise, use standard arithmetic.
	*/
	if (*end == '.') {
	double fval = strtod(value, &end);

	if ((shift = str2shift(hdl, end)) == -1)
	return (-1);

	fval *= pow(2, shift);

	/*
	* UINT64_MAX is not exactly representable as a double.
	* The closest representation is UINT64_MAX + 1, so we
	* use a >= comparison instead of > for the bounds check.
	*/
	if (fval >= (double)UINT64_MAX) {
	if (hdl)
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"numeric value is too large"));
	return (-1);
	}

	*num = (uint64_t)fval;
	} else {
	if ((shift = str2shift(hdl, end)) == -1)
	return (-1);

	/* Check for overflow */
	if (shift >= 64 \|\| (num << shift) >> shift != num) {
	if (hdl)
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"numeric value is too large"));
	return (-1);
	}

	*num <<= shift;
	}

	return (0);
	}

	/*
	* Given a propname=value nvpair to set, parse any numeric properties
	* (index, boolean, etc) if they are specified as strings and add the
	* resulting nvpair to the returned nvlist.
	*
	* At the DSL layer, all properties are either 64-bit numbers or strings.
	* We want the user to be able to ignore this fact and specify properties
	* as native values (numbers, for example) or as strings (to simplify
	* command line utilities). This also handles converting index types
	* (compression, checksum, etc) from strings to their on-disk index.
	*/
	int
	zprop_parse_value(libzfs_handle_t hdl, nvpair_t elem, int prop,
	zfs_type_t type, nvlist_t ret, char svalp, uint64_t ivalp,
	const char *errbuf)
	{
	data_type_t datatype = nvpair_type(elem);
	zprop_type_t proptype;
	const char *propname;
	char *value;
	boolean_t isnone = B_FALSE;
	boolean_t isauto = B_FALSE;
	int err = 0;

	if (type == ZFS_TYPE_POOL) {
	proptype = zpool_prop_get_type(prop);
	propname = zpool_prop_to_name(prop);
	} else {
	proptype = zfs_prop_get_type(prop);
	propname = zfs_prop_to_name(prop);
	}

	/*
	* Convert any properties to the internal DSL value types.
	*/
	*svalp = NULL;
	*ivalp = 0;

	switch (proptype) {
	case PROP_TYPE_STRING:
	if (datatype != DATA_TYPE_STRING) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"'%s' must be a string"), nvpair_name(elem));
	goto error;
	}
	err = nvpair_value_string(elem, svalp);
	if (err != 0) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"'%s' is invalid"), nvpair_name(elem));
	goto error;
	}
	if (strlen(*svalp) >= ZFS_MAXPROPLEN) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"'%s' is too long"), nvpair_name(elem));
	goto error;
	}
	break;

	case PROP_TYPE_NUMBER:
	if (datatype == DATA_TYPE_STRING) {
	(void) nvpair_value_string(elem, &value);
	if (strcmp(value, "none") == 0) {
	isnone = B_TRUE;
	} else if (strcmp(value, "auto") == 0) {
	isauto = B_TRUE;
	} else if (zfs_nicestrtonum(hdl, value, ivalp) != 0) {
	goto error;
	}
	} else if (datatype == DATA_TYPE_UINT64) {
	(void) nvpair_value_uint64(elem, ivalp);
	} else {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"'%s' must be a number"), nvpair_name(elem));
	goto error;
	}

	/*
	* Quota special: force 'none' and don't allow 0.
	*/
	if ((type & ZFS_TYPE_DATASET) && *ivalp == 0 && !isnone &&
	(prop == ZFS_PROP_QUOTA \|\| prop == ZFS_PROP_REFQUOTA)) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"use 'none' to disable quota/refquota"));
	goto error;
	}

	/*
	* Special handling for "*_limit=none". In this case it's not
	* 0 but UINT64_MAX.
	*/
	if ((type & ZFS_TYPE_DATASET) && isnone &&
	(prop == ZFS_PROP_FILESYSTEM_LIMIT \|\|
	prop == ZFS_PROP_SNAPSHOT_LIMIT)) {
	*ivalp = UINT64_MAX;
	}

	/*
	* Special handling for setting 'refreservation' to 'auto'. Use
	* UINT64_MAX to tell the caller to use zfs_fix_auto_resv().
	* 'auto' is only allowed on volumes.
	*/
	if (isauto) {
	switch (prop) {
	case ZFS_PROP_REFRESERVATION:
	if ((type & ZFS_TYPE_VOLUME) == 0) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"'%s=auto' only allowed on "
	"volumes"), nvpair_name(elem));
	goto error;
	}
	*ivalp = UINT64_MAX;
	break;
	default:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"'auto' is invalid value for '%s'"),
	nvpair_name(elem));
	goto error;
	}
	}

	break;

	case PROP_TYPE_INDEX:
	if (datatype != DATA_TYPE_STRING) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"'%s' must be a string"), nvpair_name(elem));
	goto error;
	}

	(void) nvpair_value_string(elem, &value);

	if (zprop_string_to_index(prop, value, ivalp, type) != 0) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"'%s' must be one of '%s'"), propname,
	zprop_values(prop, type));
	goto error;
	}
	break;

	default:
	abort();
	}

	/*
	* Add the result to our return set of properties.
	*/
	if (*svalp != NULL) {
	if (nvlist_add_string(ret, propname, *svalp) != 0) {
	(void) no_memory(hdl);
	return (-1);
	}
	} else {
	if (nvlist_add_uint64(ret, propname, *ivalp) != 0) {
	(void) no_memory(hdl);
	return (-1);
	}
	}

	return (0);
	error:
	(void) zfs_error(hdl, EZFS_BADPROP, errbuf);
	return (-1);
	}

	static int
	addlist(libzfs_handle_t hdl, char propname, zprop_list_t **listp,
	zfs_type_t type)
	{
	int prop;
	zprop_list_t *entry;

	prop = zprop_name_to_prop(propname, type);

	if (prop != ZPROP_INVAL && !zprop_valid_for_type(prop, type, B_FALSE))
	prop = ZPROP_INVAL;

	/*
	* When no property table entry can be found, return failure if
	* this is a pool property or if this isn't a user-defined
	* dataset property,
	*/
	if (prop == ZPROP_INVAL && ((type == ZFS_TYPE_POOL &&
	!zpool_prop_feature(propname) &&
	!zpool_prop_unsupported(propname)) \|\|
	(type == ZFS_TYPE_DATASET && !zfs_prop_user(propname) &&
	!zfs_prop_userquota(propname) && !zfs_prop_written(propname)))) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"invalid property '%s'"), propname);
	return (zfs_error(hdl, EZFS_BADPROP,
	dgettext(TEXT_DOMAIN, "bad property list")));
	}

	if ((entry = zfs_alloc(hdl, sizeof (zprop_list_t))) == NULL)
	return (-1);

	entry->pl_prop = prop;
	if (prop == ZPROP_INVAL) {
	if ((entry->pl_user_prop = zfs_strdup(hdl, propname)) ==
	NULL) {
	free(entry);
	return (-1);
	}
	entry->pl_width = strlen(propname);
	} else {
	entry->pl_width = zprop_width(prop, &entry->pl_fixed,
	type);
	}

	*listp = entry;

	return (0);
	}

	/*
	* Given a comma-separated list of properties, construct a property list
	* containing both user-defined and native properties. This function will
	* return a NULL list if 'all' is specified, which can later be expanded
	* by zprop_expand_list().
	*/
	int
	zprop_get_list(libzfs_handle_t hdl, char props, zprop_list_t **listp,
	zfs_type_t type)
	{
	*listp = NULL;

	/*
	* If 'all' is specified, return a NULL list.
	*/
	if (strcmp(props, "all") == 0)
	return (0);

	/*
	* If no props were specified, return an error.
	*/
	if (props[0] == '\0') {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"no properties specified"));
	return (zfs_error(hdl, EZFS_BADPROP, dgettext(TEXT_DOMAIN,
	"bad property list")));
	}

	/*
	* It would be nice to use getsubopt() here, but the inclusion of column
	* aliases makes this more effort than it's worth.
	*/
	while (*props != '\0') {
	size_t len;
	char *p;
	char c;

	if ((p = strchr(props, ',')) == NULL) {
	len = strlen(props);
	p = props + len;
	} else {
	len = p - props;
	}

	/*
	* Check for empty options.
	*/
	if (len == 0) {
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"empty property name"));
	return (zfs_error(hdl, EZFS_BADPROP,
	dgettext(TEXT_DOMAIN, "bad property list")));
	}

	/*
	* Check all regular property names.
	*/
	c = props[len];
	props[len] = '\0';

	if (strcmp(props, "space") == 0) {
	static char *spaceprops[] = {
	"name", "avail", "used", "usedbysnapshots",
	"usedbydataset", "usedbyrefreservation",
	"usedbychildren", NULL
	};
	int i;

	for (i = 0; spaceprops[i]; i++) {
	if (addlist(hdl, spaceprops[i], listp, type))
	return (-1);
	listp = &(*listp)->pl_next;
	}
	} else {
	if (addlist(hdl, props, listp, type))
	return (-1);
	listp = &(*listp)->pl_next;
	}

	props = p;
	if (c == ',')
	props++;
	}

	return (0);
	}

	void
	zprop_free_list(zprop_list_t *pl)
	{
	zprop_list_t *next;

	while (pl != NULL) {
	next = pl->pl_next;
	free(pl->pl_user_prop);
	free(pl);
	pl = next;
	}
	}

	typedef struct expand_data {
	zprop_list_t **last;
	libzfs_handle_t *hdl;
	zfs_type_t type;
	} expand_data_t;

	static int
	zprop_expand_list_cb(int prop, void *cb)
	{
	zprop_list_t *entry;
	expand_data_t *edp = cb;

	if ((entry = zfs_alloc(edp->hdl, sizeof (zprop_list_t))) == NULL)
	return (ZPROP_INVAL);

	entry->pl_prop = prop;
	entry->pl_width = zprop_width(prop, &entry->pl_fixed, edp->type);
	entry->pl_all = B_TRUE;

	*(edp->last) = entry;
	edp->last = &entry->pl_next;

	return (ZPROP_CONT);
	}

	int
	zprop_expand_list(libzfs_handle_t hdl, zprop_list_t *plp, zfs_type_t type)
	{
	zprop_list_t *entry;
	zprop_list_t **last;
	expand_data_t exp;

	if (*plp == NULL) {
	/*
	* If this is the very first time we've been called for an 'all'
	* specification, expand the list to include all native
	* properties.
	*/
	last = plp;

	exp.last = last;
	exp.hdl = hdl;
	exp.type = type;

	if (zprop_iter_common(zprop_expand_list_cb, &exp, B_FALSE,
	B_FALSE, type) == ZPROP_INVAL)
	return (-1);

	/*
	* Add 'name' to the beginning of the list, which is handled
	* specially.
	*/
	if ((entry = zfs_alloc(hdl, sizeof (zprop_list_t))) == NULL)
	return (-1);

	entry->pl_prop = (type == ZFS_TYPE_POOL) ? ZPOOL_PROP_NAME :
	ZFS_PROP_NAME;
	entry->pl_width = zprop_width(entry->pl_prop,
	&entry->pl_fixed, type);
	entry->pl_all = B_TRUE;
	entry->pl_next = *plp;
	*plp = entry;
	}
	return (0);
	}

	int
	zprop_iter(zprop_func func, void *cb, boolean_t show_all, boolean_t ordered,
	zfs_type_t type)
	{
	return (zprop_iter_common(func, cb, show_all, ordered, type));
	}

	/*
	* Fill given version buffer with zfs userland version
	*/
	void
	zfs_version_userland(char *version, int len)
	{
	(void) strlcpy(version, ZFS_META_ALIAS, len);
	}

	/*
	* Prints both zfs userland and kernel versions
	* Returns 0 on success, and -1 on error (with errno set)
	*/
	int
	zfs_version_print(void)
	{
	char zver_userland[128];
	char zver_kernel[128];

	zfs_version_userland(zver_userland, sizeof (zver_userland));

	(void) printf("%s\n", zver_userland);

	if (zfs_version_kernel(zver_kernel, sizeof (zver_kernel)) == -1) {
	fprintf(stderr, "zfs_version_kernel() failed: %s\n",
	strerror(errno));
	return (-1);
	}

	(void) printf("zfs-kmod-%s\n", zver_kernel);

	return (0);
	}

	/*
	* Return 1 if the user requested ANSI color output, and our terminal supports
	* it. Return 0 for no color.
	*/
	static int
	use_color(void)
	{
	static int use_color = -1;
	char *term;

	/*
	* Optimization:
	*
	* For each zpool invocation, we do a single check to see if we should
	* be using color or not, and cache that value for the lifetime of the
	* the zpool command. That makes it cheap to call use_color() when
	* we're printing with color. We assume that the settings are not going
	* to change during the invocation of a zpool command (the user isn't
	* going to change the ZFS_COLOR value while zpool is running, for
	* example).
	*/
	if (use_color != -1) {
	/*
	* We've already figured out if we should be using color or
	* not. Return the cached value.
	*/
	return (use_color);
	}

	term = getenv("TERM");
	/*
	* The user sets the ZFS_COLOR env var set to enable zpool ANSI color
	* output. However if NO_COLOR is set (https://no-color.org/) then
	* don't use it. Also, don't use color if terminal doesn't support
	* it.
	*/
	if (libzfs_envvar_is_set("ZFS_COLOR") &&
	!libzfs_envvar_is_set("NO_COLOR") &&
	isatty(STDOUT_FILENO) && term && strcmp("dumb", term) != 0 &&
	strcmp("unknown", term) != 0) {
	/* Color supported */
	use_color = 1;
	} else {
	use_color = 0;
	}

	return (use_color);
	}

	/*
	* color_start() and color_end() are used for when you want to colorize a block
	* of text. For example:
	*
	* color_start(ANSI_RED_FG)
	* printf("hello");
	* printf("world");
	* color_end();
	*/
	void
	color_start(char *color)
	{
	if (use_color())
	printf("%s", color);
	}

	void
	color_end(void)
	{
	if (use_color())
	printf(ANSI_RESET);
	}

	/* printf() with a color. If color is NULL, then do a normal printf. */
	int
	printf_color(char color, char format, ...)
	{
	va_list aptr;
	int rc;

	if (color)
	color_start(color);

	va_start(aptr, format);
	rc = vprintf(format, aptr);
	va_end(aptr);

	if (color)
	color_end();

	return (rc);
	}
	Index: vendor-sys/openzfs/dist/lib/libzfs/os/freebsd/libzfs_compat.c
	===================================================================
	--- vendor-sys/openzfs/dist/lib/libzfs/os/freebsd/libzfs_compat.c (revision 366773)
	+++ vendor-sys/openzfs/dist/lib/libzfs/os/freebsd/libzfs_compat.c (revision 366774)
	@@ -1,321 +1,332 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/

	/*
	* Copyright (c) 2013 Martin Matuska <mm@FreeBSD.org>. All rights reserved.
	*/
	#include <os/freebsd/zfs/sys/zfs_ioctl_compat.h>
	#include <libzfs_impl.h>
	#include <libzfs.h>
	#include <libzutil.h>
	#include <sys/sysctl.h>
	#include <libintl.h>
	#include <sys/linker.h>
	#include <sys/module.h>
	#include <sys/stat.h>
	#include <sys/param.h>

	#ifdef IN_BASE
	#define ZFS_KMOD "zfs"
	#else
	#define ZFS_KMOD "openzfs"
	#endif

	void
	libzfs_set_pipe_max(int infd)
	{
	/* FreeBSD automatically resizes */
	}

	static int
	execvPe(const char name, const char path, char * const *argv,
	char * const *envp)
	{
	const char **memp;
	size_t cnt, lp, ln;
	int eacces, save_errno;
	char *cur, buf[MAXPATHLEN];
	const char p, bp;
	struct stat sb;

	eacces = 0;

	/* If it's an absolute or relative path name, it's easy. */
	if (strchr(name, '/')) {
	bp = name;
	cur = NULL;
	goto retry;
	}
	bp = buf;

	/* If it's an empty path name, fail in the usual POSIX way. */
	if (*name == '\0') {
	errno = ENOENT;
	return (-1);
	}

	cur = alloca(strlen(path) + 1);
	if (cur == NULL) {
	errno = ENOMEM;
	return (-1);
	}
	strcpy(cur, path);
	while ((p = strsep(&cur, ":")) != NULL) {
	/*
	* It's a SHELL path -- double, leading and trailing colons
	* mean the current directory.
	*/
	if (*p == '\0') {
	p = ".";
	lp = 1;
	} else
	lp = strlen(p);
	ln = strlen(name);

	/*
	* If the path is too long complain. This is a possible
	* security issue; given a way to make the path too long
	* the user may execute the wrong program.
	*/
	if (lp + ln + 2 > sizeof (buf)) {
	(void) write(STDERR_FILENO, "execvP: ", 8);
	(void) write(STDERR_FILENO, p, lp);
	(void) write(STDERR_FILENO, ": path too long\n",
	16);
	continue;
	}
	bcopy(p, buf, lp);
	buf[lp] = '/';
	bcopy(name, buf + lp + 1, ln);
	buf[lp + ln + 1] = '\0';

	retry: (void) execve(bp, argv, envp);
	switch (errno) {
	case E2BIG:
	goto done;
	case ELOOP:
	case ENAMETOOLONG:
	case ENOENT:
	break;
	case ENOEXEC:
	for (cnt = 0; argv[cnt]; ++cnt)
	;
	memp = alloca((cnt + 2) * sizeof (char *));
	if (memp == NULL) {
	/* errno = ENOMEM; XXX override ENOEXEC? */
	goto done;
	}
	memp[0] = "sh";
	memp[1] = bp;
	bcopy(argv + 1, memp + 2, cnt * sizeof (char *));
	execve(_PATH_BSHELL, __DECONST(char **, memp), envp);
	goto done;
	case ENOMEM:
	goto done;
	case ENOTDIR:
	break;
	case ETXTBSY:
	/*
	* We used to retry here, but sh(1) doesn't.
	*/
	goto done;
	default:
	/*
	* EACCES may be for an inaccessible directory or
	* a non-executable file. Call stat() to decide
	* which. This also handles ambiguities for EFAULT
	* and EIO, and undocumented errors like ESTALE.
	* We hope that the race for a stat() is unimportant.
	*/
	save_errno = errno;
	if (stat(bp, &sb) != 0)
	break;
	if (save_errno == EACCES) {
	eacces = 1;
	continue;
	}
	errno = save_errno;
	goto done;
	}
	}
	if (eacces)
	errno = EACCES;
	else
	errno = ENOENT;
	done:
	return (-1);
	}

	int
	execvpe(const char name, char const argv[], char * const envp[])
	{
	const char *path;

	/* Get the path we're searching. */
	if ((path = getenv("PATH")) == NULL)
	path = _PATH_DEFPATH;

	return (execvPe(name, path, argv, envp));
	}

	+#define ERRBUFLEN 256
	+
	+__thread static char errbuf[ERRBUFLEN];
	+
	const char *
	libzfs_error_init(int error)
	{
	+ char *msg = errbuf;
	+ size_t len, msglen = ERRBUFLEN;

	- return (strerror(error));
	+ if (modfind("zfs") < 0) {
	+ len = snprintf(msg, msglen, dgettext(TEXT_DOMAIN,
	+ "Failed to load %s module: "), ZFS_KMOD);
	+ msg += len;
	+ msglen -= len;
	+ }
	+
	+ (void) snprintf(msg, msglen, "%s", strerror(error));
	+
	+ return (errbuf);
	}

	int
	zfs_ioctl(libzfs_handle_t hdl, int request, zfs_cmd_t zc)
	{
	return (zfs_ioctl_fd(hdl->libzfs_fd, request, zc));
	}

	/*
	* Verify the required ZFS_DEV device is available and optionally attempt
	* to load the ZFS modules. Under normal circumstances the modules
	* should already have been loaded by some external mechanism.
	- *
	- * Environment variables:
	- * - ZFS_MODULE_LOADING="YES\|yes\|ON\|on" - Attempt to load modules.
	- * - ZFS_MODULE_TIMEOUT="<seconds>" - Seconds to wait for ZFS_DEV
	*/
	int
	libzfs_load_module(void)
	{
	/*
	* XXX: kldfind(ZFS_KMOD) would be nice here, but we retain
	* modfind("zfs") so out-of-base openzfs userland works with the
	* in-base module.
	*/
	if (modfind("zfs") < 0) {
	/* Not present in kernel, try loading it. */
	if (kldload(ZFS_KMOD) < 0 && errno != EEXIST) {
	return (errno);
	}
	}
	return (0);
	}

	int
	zpool_relabel_disk(libzfs_handle_t hdl, const char path, const char *msg)
	{
	return (0);
	}

	int
	zpool_label_disk(libzfs_handle_t hdl, zpool_handle_t zhp, const char *name)
	{
	return (0);
	}

	int
	find_shares_object(differ_info_t *di)
	{
	return (0);
	}

	/*
	* Attach/detach the given filesystem to/from the given jail.
	*/
	int
	zfs_jail(zfs_handle_t *zhp, int jailid, int attach)
	{
	libzfs_handle_t *hdl = zhp->zfs_hdl;
	zfs_cmd_t zc = {"\0"};
	char errbuf[1024];
	unsigned long cmd;
	int ret;

	if (attach) {
	(void) snprintf(errbuf, sizeof (errbuf),
	dgettext(TEXT_DOMAIN, "cannot jail '%s'"), zhp->zfs_name);
	} else {
	(void) snprintf(errbuf, sizeof (errbuf),
	dgettext(TEXT_DOMAIN, "cannot unjail '%s'"), zhp->zfs_name);
	}

	switch (zhp->zfs_type) {
	case ZFS_TYPE_VOLUME:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"volumes can not be jailed"));
	return (zfs_error(hdl, EZFS_BADTYPE, errbuf));
	case ZFS_TYPE_SNAPSHOT:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"snapshots can not be jailed"));
	return (zfs_error(hdl, EZFS_BADTYPE, errbuf));
	case ZFS_TYPE_BOOKMARK:
	zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
	"bookmarks can not be jailed"));
	return (zfs_error(hdl, EZFS_BADTYPE, errbuf));
	case ZFS_TYPE_POOL:
	case ZFS_TYPE_FILESYSTEM:
	/* OK */
	;
	}
	assert(zhp->zfs_type == ZFS_TYPE_FILESYSTEM);

	(void) strlcpy(zc.zc_name, zhp->zfs_name, sizeof (zc.zc_name));
	zc.zc_objset_type = DMU_OST_ZFS;
	zc.zc_zoneid = jailid;

	cmd = attach ? ZFS_IOC_JAIL : ZFS_IOC_UNJAIL;
	if ((ret = zfs_ioctl(hdl, cmd, &zc)) != 0)
	zfs_standard_error(hdl, errno, errbuf);

	return (ret);
	}

	/*
	* Set loader options for next boot.
	*/
	int
	zpool_nextboot(libzfs_handle_t *hdl, uint64_t pool_guid, uint64_t dev_guid,
	const char *command)
	{
	zfs_cmd_t zc = {"\0"};
	nvlist_t *args;
	int error;

	args = fnvlist_alloc();
	fnvlist_add_uint64(args, ZPOOL_CONFIG_POOL_GUID, pool_guid);
	fnvlist_add_uint64(args, ZPOOL_CONFIG_GUID, dev_guid);
	fnvlist_add_string(args, "command", command);
	error = zcmd_write_src_nvlist(hdl, &zc, args);
	if (error == 0)
	error = zfs_ioctl(hdl, ZFS_IOC_NEXTBOOT, &zc);
	zcmd_free_nvlists(&zc);
	nvlist_free(args);
	return (error);
	}

	/*
	* Fill given version buffer with zfs kernel version.
	* Returns 0 on success, and -1 on error (with errno set)
	*/
	int
	zfs_version_kernel(char *version, int len)
	{
	size_t l = len;

	return (sysctlbyname("vfs.zfs.version.module",
	version, &l, NULL, 0));
	}
	Index: vendor-sys/openzfs/dist/lib/libzfsbootenv/libzfsbootenv.pc.in
	===================================================================
	--- vendor-sys/openzfs/dist/lib/libzfsbootenv/libzfsbootenv.pc.in (revision 366773)
	+++ vendor-sys/openzfs/dist/lib/libzfsbootenv/libzfsbootenv.pc.in (revision 366774)
	@@ -1,12 +1,12 @@
	prefix=@prefix@
	exec_prefix=@exec_prefix@
	libdir=@libdir@
	includedir=@includedir@

	Name: libzfsbootenv
	Description: LibZFSBootENV library
	Version: @VERSION@
	-URL: https://zfsonlinux.org
	+URL: https://github.com/openzfs/zfs
	Requires: libzfs libnvpair
	Cflags: -I${includedir}
	Libs: -L${libdir} -lzfsbootenv
	Index: vendor-sys/openzfs/dist/lib/libzutil/os/linux/zutil_import_os.c
	===================================================================
	--- vendor-sys/openzfs/dist/lib/libzutil/os/linux/zutil_import_os.c (revision 366773)
	+++ vendor-sys/openzfs/dist/lib/libzutil/os/linux/zutil_import_os.c (revision 366774)
	@@ -1,871 +1,871 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright 2015 Nexenta Systems, Inc. All rights reserved.
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2012, 2018 by Delphix. All rights reserved.
	* Copyright 2015 RackTop Systems.
	* Copyright (c) 2016, Intel Corporation.
	*/

	/*
	* Pool import support functions.
	*
	* Used by zpool, ztest, zdb, and zhack to locate importable configs. Since
	* these commands are expected to run in the global zone, we can assume
	* that the devices are all readable when called.
	*
	* To import a pool, we rely on reading the configuration information from the
	* ZFS label of each device. If we successfully read the label, then we
	* organize the configuration information in the following hierarchy:
	*
	* pool guid -> toplevel vdev guid -> label txg
	*
	* Duplicate entries matching this same tuple will be discarded. Once we have
	* examined every device, we pick the best label txg config for each toplevel
	* vdev. We then arrange these toplevel vdevs into a complete pool config, and
	* update any paths that have changed. Finally, we attempt to import the pool
	* using our derived config, and record the results.
	*/

	#include <ctype.h>
	#include <dirent.h>
	#include <errno.h>
	#include <libintl.h>
	#include <libgen.h>
	#include <stddef.h>
	#include <stdlib.h>
	#include <stdio.h>
	#include <string.h>
	#include <sys/stat.h>
	#include <unistd.h>
	#include <fcntl.h>
	#include <sys/dktp/fdisk.h>
	#include <sys/vdev_impl.h>
	#include <sys/fs/zfs.h>
	#include <sys/vdev_impl.h>

	#include <thread_pool.h>
	#include <libzutil.h>
	#include <libnvpair.h>

	#include "zutil_import.h"

	#ifdef HAVE_LIBUDEV
	#include <libudev.h>
	#include <sched.h>
	#endif
	#include <blkid/blkid.h>

	#define DEFAULT_IMPORT_PATH_SIZE 9
	#define DEV_BYID_PATH "/dev/disk/by-id/"

	static boolean_t
	is_watchdog_dev(char *dev)
	{
	/* For 'watchdog' dev */
	if (strcmp(dev, "watchdog") == 0)
	return (B_TRUE);

	/* For 'watchdog<digit><whatever> */
	if (strstr(dev, "watchdog") == dev && isdigit(dev[8]))
	return (B_TRUE);

	return (B_FALSE);
	}

	int
	zfs_dev_flush(int fd)
	{
	return (ioctl(fd, BLKFLSBUF));
	}

	void
	zpool_open_func(void *arg)
	{
	rdsk_node_t *rn = arg;
	libpc_handle_t *hdl = rn->rn_hdl;
	struct stat64 statbuf;
	nvlist_t *config;
	char bname, dupname;
	uint64_t vdev_guid = 0;
	int error;
	int num_labels = 0;
	int fd;

	/*
	* Skip devices with well known prefixes there can be side effects
	* when opening devices which need to be avoided.
	*
	* hpet - High Precision Event Timer
	* watchdog - Watchdog must be closed in a special way.
	*/
	dupname = zutil_strdup(hdl, rn->rn_name);
	bname = basename(dupname);
	error = ((strcmp(bname, "hpet") == 0) \|\| is_watchdog_dev(bname));
	free(dupname);
	if (error)
	return;

	/*
	* Ignore failed stats. We only want regular files and block devices.
	*/
	if (stat64(rn->rn_name, &statbuf) != 0 \|\|
	(!S_ISREG(statbuf.st_mode) && !S_ISBLK(statbuf.st_mode)))
	return;

	/*
	* Preferentially open using O_DIRECT to bypass the block device
	* cache which may be stale for multipath devices. An EINVAL errno
	* indicates O_DIRECT is unsupported so fallback to just O_RDONLY.
	*/
	fd = open(rn->rn_name, O_RDONLY \| O_DIRECT);
	if ((fd < 0) && (errno == EINVAL))
	fd = open(rn->rn_name, O_RDONLY);
	if ((fd < 0) && (errno == EACCES))
	hdl->lpc_open_access_error = B_TRUE;
	if (fd < 0)
	return;

	/*
	* This file is too small to hold a zpool
	*/
	if (S_ISREG(statbuf.st_mode) && statbuf.st_size < SPA_MINDEVSIZE) {
	(void) close(fd);
	return;
	}

	error = zpool_read_label(fd, &config, &num_labels);
	if (error != 0) {
	(void) close(fd);
	return;
	}

	if (num_labels == 0) {
	(void) close(fd);
	nvlist_free(config);
	return;
	}

	/*
	* Check that the vdev is for the expected guid. Additional entries
	* are speculatively added based on the paths stored in the labels.
	* Entries with valid paths but incorrect guids must be removed.
	*/
	error = nvlist_lookup_uint64(config, ZPOOL_CONFIG_GUID, &vdev_guid);
	if (error \|\| (rn->rn_vdev_guid && rn->rn_vdev_guid != vdev_guid)) {
	(void) close(fd);
	nvlist_free(config);
	return;
	}

	(void) close(fd);

	rn->rn_config = config;
	rn->rn_num_labels = num_labels;

	/*
	* Add additional entries for paths described by this label.
	*/
	if (rn->rn_labelpaths) {
	char *path = NULL;
	char *devid = NULL;
	char *env = NULL;
	rdsk_node_t *slice;
	avl_index_t where;
	int timeout;
	int error;

	if (label_paths(rn->rn_hdl, rn->rn_config, &path, &devid))
	return;

	env = getenv("ZPOOL_IMPORT_UDEV_TIMEOUT_MS");
	if ((env == NULL) \|\| sscanf(env, "%d", &timeout) != 1 \|\|
	timeout < 0) {
	timeout = DISK_LABEL_WAIT;
	}

	/*
	* Allow devlinks to stabilize so all paths are available.
	*/
	zpool_label_disk_wait(rn->rn_name, timeout);

	if (path != NULL) {
	slice = zutil_alloc(hdl, sizeof (rdsk_node_t));
	slice->rn_name = zutil_strdup(hdl, path);
	slice->rn_vdev_guid = vdev_guid;
	slice->rn_avl = rn->rn_avl;
	slice->rn_hdl = hdl;
	slice->rn_order = IMPORT_ORDER_PREFERRED_1;
	slice->rn_labelpaths = B_FALSE;
	pthread_mutex_lock(rn->rn_lock);
	if (avl_find(rn->rn_avl, slice, &where)) {
	pthread_mutex_unlock(rn->rn_lock);
	free(slice->rn_name);
	free(slice);
	} else {
	avl_insert(rn->rn_avl, slice, where);
	pthread_mutex_unlock(rn->rn_lock);
	zpool_open_func(slice);
	}
	}

	if (devid != NULL) {
	slice = zutil_alloc(hdl, sizeof (rdsk_node_t));
	error = asprintf(&slice->rn_name, "%s%s",
	DEV_BYID_PATH, devid);
	if (error == -1) {
	free(slice);
	return;
	}

	slice->rn_vdev_guid = vdev_guid;
	slice->rn_avl = rn->rn_avl;
	slice->rn_hdl = hdl;
	slice->rn_order = IMPORT_ORDER_PREFERRED_2;
	slice->rn_labelpaths = B_FALSE;
	pthread_mutex_lock(rn->rn_lock);
	if (avl_find(rn->rn_avl, slice, &where)) {
	pthread_mutex_unlock(rn->rn_lock);
	free(slice->rn_name);
	free(slice);
	} else {
	avl_insert(rn->rn_avl, slice, where);
	pthread_mutex_unlock(rn->rn_lock);
	zpool_open_func(slice);
	}
	}
	}
	}

	static char *
	zpool_default_import_path[DEFAULT_IMPORT_PATH_SIZE] = {
	"/dev/disk/by-vdev", /* Custom rules, use first if they exist */
	"/dev/mapper", /* Use multipath devices before components */
	"/dev/disk/by-partlabel", /* Single unique entry set by user */
	"/dev/disk/by-partuuid", /* Generated partition uuid */
	"/dev/disk/by-label", /* Custom persistent labels */
	"/dev/disk/by-uuid", /* Single unique entry and persistent */
	"/dev/disk/by-id", /* May be multiple entries and persistent */
	"/dev/disk/by-path", /* Encodes physical location and persistent */
	"/dev" /* UNSAFE device names will change */
	};

	const char * const *
	zpool_default_search_paths(size_t *count)
	{
	*count = DEFAULT_IMPORT_PATH_SIZE;
	return ((const char * const *)zpool_default_import_path);
	}

	/*
	* Given a full path to a device determine if that device appears in the
	* import search path. If it does return the first match and store the
	* index in the passed 'order' variable, otherwise return an error.
	*/
	static int
	zfs_path_order(char name, int order)
	{
	int i = 0, error = ENOENT;
	char dir, env, *envdup;

	env = getenv("ZPOOL_IMPORT_PATH");
	if (env) {
	envdup = strdup(env);
	dir = strtok(envdup, ":");
	while (dir) {
	if (strncmp(name, dir, strlen(dir)) == 0) {
	*order = i;
	error = 0;
	break;
	}
	dir = strtok(NULL, ":");
	i++;
	}
	free(envdup);
	} else {
	for (i = 0; i < DEFAULT_IMPORT_PATH_SIZE; i++) {
	if (strncmp(name, zpool_default_import_path[i],
	strlen(zpool_default_import_path[i])) == 0) {
	*order = i;
	error = 0;
	break;
	}
	}
	}

	return (error);
	}

	/*
	* Use libblkid to quickly enumerate all known zfs devices.
	*/
	int
	zpool_find_import_blkid(libpc_handle_t hdl, pthread_mutex_t lock,
	avl_tree_t **slice_cache)
	{
	rdsk_node_t *slice;
	blkid_cache cache;
	blkid_dev_iterate iter;
	blkid_dev dev;
	avl_index_t where;
	int error;

	*slice_cache = NULL;

	error = blkid_get_cache(&cache, NULL);
	if (error != 0)
	return (error);

	error = blkid_probe_all_new(cache);
	if (error != 0) {
	blkid_put_cache(cache);
	return (error);
	}

	iter = blkid_dev_iterate_begin(cache);
	if (iter == NULL) {
	blkid_put_cache(cache);
	return (EINVAL);
	}

	error = blkid_dev_set_search(iter, "TYPE", "zfs_member");
	if (error != 0) {
	blkid_dev_iterate_end(iter);
	blkid_put_cache(cache);
	return (error);
	}

	*slice_cache = zutil_alloc(hdl, sizeof (avl_tree_t));
	avl_create(*slice_cache, slice_cache_compare, sizeof (rdsk_node_t),
	offsetof(rdsk_node_t, rn_node));

	while (blkid_dev_next(iter, &dev) == 0) {
	slice = zutil_alloc(hdl, sizeof (rdsk_node_t));
	slice->rn_name = zutil_strdup(hdl, blkid_dev_devname(dev));
	slice->rn_vdev_guid = 0;
	slice->rn_lock = lock;
	slice->rn_avl = *slice_cache;
	slice->rn_hdl = hdl;
	slice->rn_labelpaths = B_TRUE;

	error = zfs_path_order(slice->rn_name, &slice->rn_order);
	if (error == 0)
	slice->rn_order += IMPORT_ORDER_SCAN_OFFSET;
	else
	slice->rn_order = IMPORT_ORDER_DEFAULT;

	pthread_mutex_lock(lock);
	if (avl_find(*slice_cache, slice, &where)) {
	free(slice->rn_name);
	free(slice);
	} else {
	avl_insert(*slice_cache, slice, where);
	}
	pthread_mutex_unlock(lock);
	}

	blkid_dev_iterate_end(iter);
	blkid_put_cache(cache);

	return (0);
	}

	/*
	* Linux persistent device strings for vdev labels
	*
	* based on libudev for consistency with libudev disk add/remove events
	*/

	typedef struct vdev_dev_strs {
	char vds_devid[128];
	char vds_devphys[128];
	} vdev_dev_strs_t;

	#ifdef HAVE_LIBUDEV

	/*
	* Obtain the persistent device id string (describes what)
	*
	* used by ZED vdev matching for auto-{online,expand,replace}
	*/
	int
	zfs_device_get_devid(struct udev_device dev, char bufptr, size_t buflen)
	{
	struct udev_list_entry *entry;
	const char *bus;
	char devbyid[MAXPATHLEN];

	/* The bus based by-id path is preferred */
	bus = udev_device_get_property_value(dev, "ID_BUS");

	if (bus == NULL) {
	const char *dm_uuid;

	/*
	* For multipath nodes use the persistent uuid based identifier
	*
	* Example: /dev/disk/by-id/dm-uuid-mpath-35000c5006304de3f
	*/
	dm_uuid = udev_device_get_property_value(dev, "DM_UUID");
	if (dm_uuid != NULL) {
	(void) snprintf(bufptr, buflen, "dm-uuid-%s", dm_uuid);
	return (0);
	}

	/*
	* For volumes use the persistent /dev/zvol/dataset identifier
	*/
	entry = udev_device_get_devlinks_list_entry(dev);
	while (entry != NULL) {
	const char *name;

	name = udev_list_entry_get_name(entry);
	if (strncmp(name, ZVOL_ROOT, strlen(ZVOL_ROOT)) == 0) {
	(void) strlcpy(bufptr, name, buflen);
	return (0);
	}
	entry = udev_list_entry_get_next(entry);
	}

	/*
	* NVME 'by-id' symlinks are similar to bus case
	*/
	struct udev_device *parent;

	parent = udev_device_get_parent_with_subsystem_devtype(dev,
	"nvme", NULL);
	if (parent != NULL)
	bus = "nvme"; /* continue with bus symlink search */
	else
	return (ENODATA);
	}

	/*
	* locate the bus specific by-id link
	*/
	(void) snprintf(devbyid, sizeof (devbyid), "%s%s-", DEV_BYID_PATH, bus);
	entry = udev_device_get_devlinks_list_entry(dev);
	while (entry != NULL) {
	const char *name;

	name = udev_list_entry_get_name(entry);
	if (strncmp(name, devbyid, strlen(devbyid)) == 0) {
	name += strlen(DEV_BYID_PATH);
	(void) strlcpy(bufptr, name, buflen);
	return (0);
	}
	entry = udev_list_entry_get_next(entry);
	}

	return (ENODATA);
	}

	/*
	* Obtain the persistent physical location string (describes where)
	*
	* used by ZED vdev matching for auto-{online,expand,replace}
	*/
	int
	zfs_device_get_physical(struct udev_device dev, char bufptr, size_t buflen)
	{
	const char *physpath = NULL;
	struct udev_list_entry *entry;

	/*
	* Normal disks use ID_PATH for their physical path.
	*/
	physpath = udev_device_get_property_value(dev, "ID_PATH");
	if (physpath != NULL && strlen(physpath) > 0) {
	(void) strlcpy(bufptr, physpath, buflen);
	return (0);
	}

	/*
	* Device mapper devices are virtual and don't have a physical
	* path. For them we use ID_VDEV instead, which is setup via the
	* /etc/vdev_id.conf file. ID_VDEV provides a persistent path
	* to a virtual device. If you don't have vdev_id.conf setup,
	* you cannot use multipath autoreplace with device mapper.
	*/
	physpath = udev_device_get_property_value(dev, "ID_VDEV");
	if (physpath != NULL && strlen(physpath) > 0) {
	(void) strlcpy(bufptr, physpath, buflen);
	return (0);
	}

	/*
	* For ZFS volumes use the persistent /dev/zvol/dataset identifier
	*/
	entry = udev_device_get_devlinks_list_entry(dev);
	while (entry != NULL) {
	physpath = udev_list_entry_get_name(entry);
	if (strncmp(physpath, ZVOL_ROOT, strlen(ZVOL_ROOT)) == 0) {
	(void) strlcpy(bufptr, physpath, buflen);
	return (0);
	}
	entry = udev_list_entry_get_next(entry);
	}

	/*
	* For all other devices fallback to using the by-uuid name.
	*/
	entry = udev_device_get_devlinks_list_entry(dev);
	while (entry != NULL) {
	physpath = udev_list_entry_get_name(entry);
	if (strncmp(physpath, "/dev/disk/by-uuid", 17) == 0) {
	(void) strlcpy(bufptr, physpath, buflen);
	return (0);
	}
	entry = udev_list_entry_get_next(entry);
	}

	return (ENODATA);
	}

	/*
	* A disk is considered a multipath whole disk when:
	* DEVNAME key value has "dm-"
	* DM_NAME key value has "mpath" prefix
	* DM_UUID key exists
	* ID_PART_TABLE_TYPE key does not exist or is not gpt
	*/
	static boolean_t
	udev_mpath_whole_disk(struct udev_device *dev)
	{
	const char devname, type, *uuid;

	devname = udev_device_get_property_value(dev, "DEVNAME");
	type = udev_device_get_property_value(dev, "ID_PART_TABLE_TYPE");
	uuid = udev_device_get_property_value(dev, "DM_UUID");

	if ((devname != NULL && strncmp(devname, "/dev/dm-", 8) == 0) &&
	((type == NULL) \|\| (strcmp(type, "gpt") != 0)) &&
	(uuid != NULL)) {
	return (B_TRUE);
	}

	return (B_FALSE);
	}

	static int
	udev_device_is_ready(struct udev_device *dev)
	{
	#ifdef HAVE_LIBUDEV_UDEV_DEVICE_GET_IS_INITIALIZED
	return (udev_device_get_is_initialized(dev));
	#else
	/* wait for DEVLINKS property to be initialized */
	return (udev_device_get_property_value(dev, "DEVLINKS") != NULL);
	#endif
	}

	#else

	/* ARGSUSED */
	int
	zfs_device_get_devid(struct udev_device dev, char bufptr, size_t buflen)
	{
	return (ENODATA);
	}

	/* ARGSUSED */
	int
	zfs_device_get_physical(struct udev_device dev, char bufptr, size_t buflen)
	{
	return (ENODATA);
	}

	#endif /* HAVE_LIBUDEV */

	/*
	* Wait up to timeout_ms for udev to set up the device node. The device is
	* considered ready when libudev determines it has been initialized, all of
	* the device links have been verified to exist, and it has been allowed to
	* settle. At this point the device the device can be accessed reliably.
	* Depending on the complexity of the udev rules this process could take
	* several seconds.
	*/
	int
	zpool_label_disk_wait(const char *path, int timeout_ms)
	{
	#ifdef HAVE_LIBUDEV
	struct udev *udev;
	struct udev_device *dev = NULL;
	char nodepath[MAXPATHLEN];
	char *sysname = NULL;
	int ret = ENODEV;
	int settle_ms = 50;
	long sleep_ms = 10;
	hrtime_t start, settle;

	if ((udev = udev_new()) == NULL)
	return (ENXIO);

	start = gethrtime();
	settle = 0;

	do {
	if (sysname == NULL) {
	if (realpath(path, nodepath) != NULL) {
	sysname = strrchr(nodepath, '/') + 1;
	} else {
	(void) usleep(sleep_ms * MILLISEC);
	continue;
	}
	}

	dev = udev_device_new_from_subsystem_sysname(udev,
	"block", sysname);
	if ((dev != NULL) && udev_device_is_ready(dev)) {
	struct udev_list_entry links, link = NULL;

	ret = 0;
	links = udev_device_get_devlinks_list_entry(dev);

	udev_list_entry_foreach(link, links) {
	struct stat64 statbuf;
	const char *name;

	name = udev_list_entry_get_name(link);
	errno = 0;
	if (stat64(name, &statbuf) == 0 && errno == 0)
	continue;

	settle = 0;
	ret = ENODEV;
	break;
	}

	if (ret == 0) {
	if (settle == 0) {
	settle = gethrtime();
	} else if (NSEC2MSEC(gethrtime() - settle) >=
	settle_ms) {
	udev_device_unref(dev);
	break;
	}
	}
	}

	udev_device_unref(dev);
	(void) usleep(sleep_ms * MILLISEC);

	} while (NSEC2MSEC(gethrtime() - start) < timeout_ms);

	udev_unref(udev);

	return (ret);
	#else
	int settle_ms = 50;
	long sleep_ms = 10;
	hrtime_t start, settle;
	struct stat64 statbuf;

	start = gethrtime();
	settle = 0;

	do {
	errno = 0;
	if ((stat64(path, &statbuf) == 0) && (errno == 0)) {
	if (settle == 0)
	settle = gethrtime();
	else if (NSEC2MSEC(gethrtime() - settle) >= settle_ms)
	return (0);
	} else if (errno != ENOENT) {
	return (errno);
	}

	usleep(sleep_ms * MILLISEC);
	} while (NSEC2MSEC(gethrtime() - start) < timeout_ms);

	return (ENODEV);
	#endif /* HAVE_LIBUDEV */
	}

	/*
	* Encode the persistent devices strings
	* used for the vdev disk label
	*/
	static int
	encode_device_strings(const char path, vdev_dev_strs_t ds,
	boolean_t wholedisk)
	{
	#ifdef HAVE_LIBUDEV
	struct udev *udev;
	struct udev_device *dev = NULL;
	char nodepath[MAXPATHLEN];
	char *sysname;
	int ret = ENODEV;
	hrtime_t start;

	if ((udev = udev_new()) == NULL)
	return (ENXIO);

	/* resolve path to a runtime device node instance */
	if (realpath(path, nodepath) == NULL)
	goto no_dev;

	sysname = strrchr(nodepath, '/') + 1;

	/*
	* Wait up to 3 seconds for udev to set up the device node context
	*/
	start = gethrtime();
	do {
	dev = udev_device_new_from_subsystem_sysname(udev, "block",
	sysname);
	if (dev == NULL)
	goto no_dev;
	if (udev_device_is_ready(dev))
	break; /* udev ready */

	udev_device_unref(dev);
	dev = NULL;

	if (NSEC2MSEC(gethrtime() - start) < 10)
	(void) sched_yield(); /* yield/busy wait up to 10ms */
	else
	(void) usleep(10 * MILLISEC);

	} while (NSEC2MSEC(gethrtime() - start) < (3 * MILLISEC));

	if (dev == NULL)
	goto no_dev;

	/*
	* Only whole disks require extra device strings
	*/
	if (!wholedisk && !udev_mpath_whole_disk(dev))
	goto no_dev;

	ret = zfs_device_get_devid(dev, ds->vds_devid, sizeof (ds->vds_devid));
	if (ret != 0)
	goto no_dev_ref;

	/* physical location string (optional) */
	if (zfs_device_get_physical(dev, ds->vds_devphys,
	sizeof (ds->vds_devphys)) != 0) {
	ds->vds_devphys[0] = '\0'; /* empty string --> not available */
	}

	no_dev_ref:
	udev_device_unref(dev);
	no_dev:
	udev_unref(udev);

	return (ret);
	#else
	return (ENOENT);
	#endif
	}

	/*
	* Update a leaf vdev's persistent device strings
	*
	* - only applies for a dedicated leaf vdev (aka whole disk)
	* - updated during pool create\|add\|attach\|import
	* - used for matching device matching during auto-{online,expand,replace}
	* - stored in a leaf disk config label (i.e. alongside 'path' NVP)
	* - these strings are currently not used in kernel (i.e. for vdev_disk_open)
	*
	* single device node example:
	* devid: 'scsi-MG03SCA300_350000494a8cb3d67-part1'
	* phys_path: 'pci-0000:04:00.0-sas-0x50000394a8cb3d67-lun-0'
	*
	* multipath device node example:
	* devid: 'dm-uuid-mpath-35000c5006304de3f'
	*
	* We also store the enclosure sysfs path for turning on enclosure LEDs
	* (if applicable):
	* vdev_enc_sysfs_path: '/sys/class/enclosure/11:0:1:0/SLOT 4'
	*/
	void
	update_vdev_config_dev_strs(nvlist_t *nv)
	{
	vdev_dev_strs_t vds;
	char env, type, *path;
	uint64_t wholedisk = 0;
	char upath, spath;

	/*
	* For the benefit of legacy ZFS implementations, allow
	* for opting out of devid strings in the vdev label.
	*
	* example use:
	* env ZFS_VDEV_DEVID_OPT_OUT=YES zpool import dozer
	*
	* explanation:
	- * Older ZFS on Linux implementations had issues when attempting to
	- * display pool config VDEV names if a "devid" NVP value is present
	- * in the pool's config.
	+ * Older OpenZFS implementations had issues when attempting to
	+ * display pool config VDEV names if a "devid" NVP value is
	+ * present in the pool's config.
	*
	* For example, a pool that originated on illumos platform would
	* have a devid value in the config and "zpool status" would fail
	* when listing the config.
	*
	* A pool can be stripped of any "devid" values on import or
	* prevented from adding them on zpool create\|add by setting
	* ZFS_VDEV_DEVID_OPT_OUT.
	*/
	env = getenv("ZFS_VDEV_DEVID_OPT_OUT");
	if (env && (strtoul(env, NULL, 0) > 0 \|\|
	!strncasecmp(env, "YES", 3) \|\| !strncasecmp(env, "ON", 2))) {
	(void) nvlist_remove_all(nv, ZPOOL_CONFIG_DEVID);
	(void) nvlist_remove_all(nv, ZPOOL_CONFIG_PHYS_PATH);
	return;
	}

	if (nvlist_lookup_string(nv, ZPOOL_CONFIG_TYPE, &type) != 0 \|\|
	strcmp(type, VDEV_TYPE_DISK) != 0) {
	return;
	}
	if (nvlist_lookup_string(nv, ZPOOL_CONFIG_PATH, &path) != 0)
	return;
	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_WHOLE_DISK, &wholedisk);

	/*
	* Update device string values in the config nvlist.
	*/
	if (encode_device_strings(path, &vds, (boolean_t)wholedisk) == 0) {
	(void) nvlist_add_string(nv, ZPOOL_CONFIG_DEVID, vds.vds_devid);
	if (vds.vds_devphys[0] != '\0') {
	(void) nvlist_add_string(nv, ZPOOL_CONFIG_PHYS_PATH,
	vds.vds_devphys);
	}

	/* Add enclosure sysfs path (if disk is in an enclosure). */
	upath = zfs_get_underlying_path(path);
	spath = zfs_get_enclosure_sysfs_path(upath);
	if (spath)
	nvlist_add_string(nv, ZPOOL_CONFIG_VDEV_ENC_SYSFS_PATH,
	spath);
	else
	nvlist_remove_all(nv, ZPOOL_CONFIG_VDEV_ENC_SYSFS_PATH);

	free(upath);
	free(spath);
	} else {
	/* Clear out any stale entries. */
	(void) nvlist_remove_all(nv, ZPOOL_CONFIG_DEVID);
	(void) nvlist_remove_all(nv, ZPOOL_CONFIG_PHYS_PATH);
	(void) nvlist_remove_all(nv, ZPOOL_CONFIG_VDEV_ENC_SYSFS_PATH);
	}
	}
	Index: vendor-sys/openzfs/dist/man/man1/raidz_test.1
	===================================================================
	--- vendor-sys/openzfs/dist/man/man1/raidz_test.1 (revision 366773)
	+++ vendor-sys/openzfs/dist/man/man1/raidz_test.1 (revision 366774)
	@@ -1,97 +1,97 @@
	'\" t
	.\"
	.\" CDDL HEADER START
	.\"
	.\" The contents of this file are subject to the terms of the
	.\" Common Development and Distribution License (the "License").
	.\" You may not use this file except in compliance with the License.
	.\"
	.\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	.\" or http://www.opensolaris.org/os/licensing.
	.\" See the License for the specific language governing permissions
	.\" and limitations under the License.
	.\"
	.\" When distributing Covered Code, include this CDDL HEADER in each
	.\" file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	.\" If applicable, add the following below this CDDL HEADER, with the
	.\" fields enclosed by brackets "[]" replaced with your own identifying
	.\" information: Portions Copyright [yyyy] [name of copyright owner]
	.\"
	.\" CDDL HEADER END
	.\"
	.\"
	.\" Copyright (c) 2016 Gvozden Nešković. All rights reserved.
	.\"
	.TH RAIDZ_TEST 1 "Aug 24, 2020" OpenZFS

	.SH NAME
	\fBraidz_test\fR \- raidz implementation verification and benchmarking tool
	.SH SYNOPSIS
	.LP
	.BI "raidz_test <options>"
	.SH DESCRIPTION
	.LP
	This manual page documents briefly the \fBraidz_test\fR command.
	.LP
	Purpose of this tool is to run all supported raidz implementation and verify
	results of all methods. Tool also contains a parameter sweep option where all
	parameters affecting RAIDZ block are verified (like ashift size, data offset,
	data size, etc...).
	The tool also supports a benchmarking mode using -B option.
	.SH OPTION
	.HP
	.BI "\-h" ""
	.IP
	Print a help summary.
	.HP
	.BI "\-a" " ashift (default: 9)"
	.IP
	Ashift value.
	.HP
	.BI "\-o" " zio_off_shift" " (default: 0)"
	.IP
	Zio offset for raidz block. Offset value is 1 << (zio_off_shift)
	.HP
	.BI "\-d" " raidz_data_disks" " (default: 8)"
	.IP
	Number of raidz data disks to use. Additional disks for parity will be used
	during testing.
	.HP
	.BI "\-s" " zio_size_shift" " (default: 19)"
	.IP
	Size of data for raidz block. Size is 1 << (zio_size_shift).
	.HP
	.BI "\-S(weep)"
	.IP
	Sweep parameter space while verifying the raidz implementations. This option
	will exhaust all most of valid values for -a -o -d -s options. Runtime using
	this option will be long.
	.HP
	.BI "\-t(imeout)"
	.IP
	Wall time for sweep test in seconds. The actual runtime could be longer.
	.HP
	.BI "\-B(enchmark)"
	.IP
	This options starts the benchmark mode. All implementations are benchmarked
	using increasing per disk data size. Results are given as throughput per disk,
	measured in MiB/s.
	.HP
	.BI "\-v(erbose)"
	.IP
	Increase verbosity.
	.HP
	.BI "\-T(est the test)"
	.IP
	Debugging option. When this option is specified tool is supposed to fail
	all tests. This is to check if tests would properly verify bit-exactness.
	.HP
	.BI "\-D(ebug)"
	.IP
	Debugging option. Specify to attach gdb when SIGSEGV or SIGABRT are received.
	.HP

	.SH "SEE ALSO"
	.BR "ztest (1)"
	.SH "AUTHORS"
	-vdev_raidz, created for ZFS on Linux by Gvozden Nešković <neskovic@gmail.com>
	+vdev_raidz, created for OpenZFS by Gvozden Nešković <neskovic@gmail.com>
	Index: vendor-sys/openzfs/dist/man/man5/zfs-module-parameters.5
	===================================================================
	--- vendor-sys/openzfs/dist/man/man5/zfs-module-parameters.5 (revision 366773)
	+++ vendor-sys/openzfs/dist/man/man5/zfs-module-parameters.5 (revision 366774)
	@@ -1,4157 +1,4168 @@
	'\" te
	.\" Copyright (c) 2013 by Turbo Fredriksson <turbo@bayour.com>. All rights reserved.
	.\" Copyright (c) 2019, 2020 by Delphix. All rights reserved.
	.\" Copyright (c) 2019 Datto Inc.
	.\" The contents of this file are subject to the terms of the Common Development
	.\" and Distribution License (the "License"). You may not use this file except
	.\" in compliance with the License. You can obtain a copy of the license at
	.\" usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing.
	.\"
	.\" See the License for the specific language governing permissions and
	.\" limitations under the License. When distributing Covered Code, include this
	.\" CDDL HEADER in each file and include the License file at
	.\" usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this
	.\" CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your
	.\" own identifying information:
	.\" Portions Copyright [yyyy] [name of copyright owner]
	.TH ZFS-MODULE-PARAMETERS 5 "Aug 24, 2020" OpenZFS
	.SH NAME
	zfs\-module\-parameters \- ZFS module parameters
	.SH DESCRIPTION
	.sp
	.LP
	Description of the different parameters to the ZFS module.

	.SS "Module parameters"
	.sp
	.LP

	.sp
	.ne 2
	.na
	\fBdbuf_cache_max_bytes\fR (ulong)
	.ad
	.RS 12n
	Maximum size in bytes of the dbuf cache. The target size is determined by the
	MIN versus \fB1/2^dbuf_cache_shift\fR (1/32) of the target ARC size. The
	behavior of the dbuf cache and its associated settings can be observed via the
	\fB/proc/spl/kstat/zfs/dbufstats\fR kstat.
	.sp
	Default value: \fBULONG_MAX\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBdbuf_metadata_cache_max_bytes\fR (ulong)
	.ad
	.RS 12n
	Maximum size in bytes of the metadata dbuf cache. The target size is
	determined by the MIN versus \fB1/2^dbuf_metadata_cache_shift\fR (1/64) of the
	target ARC size. The behavior of the metadata dbuf cache and its associated
	settings can be observed via the \fB/proc/spl/kstat/zfs/dbufstats\fR kstat.
	.sp
	Default value: \fBULONG_MAX\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBdbuf_cache_hiwater_pct\fR (uint)
	.ad
	.RS 12n
	The percentage over \fBdbuf_cache_max_bytes\fR when dbufs must be evicted
	directly.
	.sp
	Default value: \fB10\fR%.
	.RE

	.sp
	.ne 2
	.na
	\fBdbuf_cache_lowater_pct\fR (uint)
	.ad
	.RS 12n
	The percentage below \fBdbuf_cache_max_bytes\fR when the evict thread stops
	evicting dbufs.
	.sp
	Default value: \fB10\fR%.
	.RE

	.sp
	.ne 2
	.na
	\fBdbuf_cache_shift\fR (int)
	.ad
	.RS 12n
	Set the size of the dbuf cache, \fBdbuf_cache_max_bytes\fR, to a log2 fraction
	of the target ARC size.
	.sp
	Default value: \fB5\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBdbuf_metadata_cache_shift\fR (int)
	.ad
	.RS 12n
	Set the size of the dbuf metadata cache, \fBdbuf_metadata_cache_max_bytes\fR,
	to a log2 fraction of the target ARC size.
	.sp
	Default value: \fB6\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBdmu_object_alloc_chunk_shift\fR (int)
	.ad
	.RS 12n
	dnode slots allocated in a single operation as a power of 2. The default value
	minimizes lock contention for the bulk operation performed.
	.sp
	Default value: \fB7\fR (128).
	.RE

	.sp
	.ne 2
	.na
	\fBdmu_prefetch_max\fR (int)
	.ad
	.RS 12n
	Limit the amount we can prefetch with one call to this amount (in bytes).
	This helps to limit the amount of memory that can be used by prefetching.
	.sp
	Default value: \fB134,217,728\fR (128MB).
	.RE

	.sp
	.ne 2
	.na
	\fBignore_hole_birth\fR (int)
	.ad
	.RS 12n
	This is an alias for \fBsend_holes_without_birth_time\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBl2arc_feed_again\fR (int)
	.ad
	.RS 12n
	Turbo L2ARC warm-up. When the L2ARC is cold the fill interval will be set as
	fast as possible.
	.sp
	Use \fB1\fR for yes (default) and \fB0\fR to disable.
	.RE

	.sp
	.ne 2
	.na
	\fBl2arc_feed_min_ms\fR (ulong)
	.ad
	.RS 12n
	Min feed interval in milliseconds. Requires \fBl2arc_feed_again=1\fR and only
	applicable in related situations.
	.sp
	Default value: \fB200\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBl2arc_feed_secs\fR (ulong)
	.ad
	.RS 12n
	Seconds between L2ARC writing
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBl2arc_headroom\fR (ulong)
	.ad
	.RS 12n
	How far through the ARC lists to search for L2ARC cacheable content, expressed
	as a multiplier of \fBl2arc_write_max\fR.
	ARC persistence across reboots can be achieved with persistent L2ARC by setting
	this parameter to \fB0\fR allowing the full length of ARC lists to be searched
	for cacheable content.
	.sp
	Default value: \fB2\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBl2arc_headroom_boost\fR (ulong)
	.ad
	.RS 12n
	Scales \fBl2arc_headroom\fR by this percentage when L2ARC contents are being
	successfully compressed before writing. A value of \fB100\fR disables this
	feature.
	.sp
	Default value: \fB200\fR%.
	.RE

	.sp
	.ne 2
	.na
	\fBl2arc_mfuonly\fR (int)
	.ad
	.RS 12n
	Controls whether only MFU metadata and data are cached from ARC into L2ARC.
	This may be desired to avoid wasting space on L2ARC when reading/writing large
	amounts of data that are not expected to be accessed more than once. The
	default is \fB0\fR, meaning both MRU and MFU data and metadata are cached.
	When turning off (\fB0\fR) this feature some MRU buffers will still be present
	in ARC and eventually cached on L2ARC.
	.sp
	Use \fB0\fR for no (default) and \fB1\fR for yes.
	.RE

	.sp
	.ne 2
	.na
	\fBl2arc_meta_percent\fR (int)
	.ad
	.RS 12n
	Percent of ARC size allowed for L2ARC-only headers.
	Since L2ARC buffers are not evicted on memory pressure, too large amount of
	headers on system with irrationaly large L2ARC can render it slow or unusable.
	This parameter limits L2ARC writes and rebuild to achieve it.
	.sp
	Default value: \fB33\fR%.
	.RE

	.sp
	.ne 2
	.na
	\fBl2arc_trim_ahead\fR (ulong)
	.ad
	.RS 12n
	Trims ahead of the current write size (\fBl2arc_write_max\fR) on L2ARC devices
	by this percentage of write size if we have filled the device. If set to
	\fB100\fR we TRIM twice the space required to accommodate upcoming writes. A
	minimum of 64MB will be trimmed. It also enables TRIM of the whole L2ARC device
	upon creation or addition to an existing pool or if the header of the device is
	invalid upon importing a pool or onlining a cache device. A value of \fB0\fR
	disables TRIM on L2ARC altogether and is the default as it can put significant
	stress on the underlying storage devices. This will vary depending of how well
	the specific device handles these commands.
	.sp
	Default value: \fB0\fR%.
	.RE

	.sp
	.ne 2
	.na
	\fBl2arc_noprefetch\fR (int)
	.ad
	.RS 12n
	Do not write buffers to L2ARC if they were prefetched but not used by
	applications.
	.sp
	Use \fB1\fR for yes (default) and \fB0\fR to disable.
	.RE

	.sp
	.ne 2
	.na
	\fBl2arc_norw\fR (int)
	.ad
	.RS 12n
	No reads during writes.
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBl2arc_write_boost\fR (ulong)
	.ad
	.RS 12n
	Cold L2ARC devices will have \fBl2arc_write_max\fR increased by this amount
	while they remain cold.
	.sp
	Default value: \fB8,388,608\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBl2arc_write_max\fR (ulong)
	.ad
	.RS 12n
	Max write bytes per interval.
	.sp
	Default value: \fB8,388,608\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBl2arc_rebuild_enabled\fR (int)
	.ad
	.RS 12n
	Rebuild the L2ARC when importing a pool (persistent L2ARC). This can be
	disabled if there are problems importing a pool or attaching an L2ARC device
	(e.g. the L2ARC device is slow in reading stored log metadata, or the metadata
	has become somehow fragmented/unusable).
	.sp
	Use \fB1\fR for yes (default) and \fB0\fR for no.
	.RE

	.sp
	.ne 2
	.na
	\fBl2arc_rebuild_blocks_min_l2size\fR (ulong)
	.ad
	.RS 12n
	Min size (in bytes) of an L2ARC device required in order to write log blocks
	in it. The log blocks are used upon importing the pool to rebuild
	the L2ARC (persistent L2ARC). Rationale: for L2ARC devices less than 1GB, the
	amount of data l2arc_evict() evicts is significant compared to the amount of
	restored L2ARC data. In this case do not write log blocks in L2ARC in order not
	to waste space.
	.sp
	Default value: \fB1,073,741,824\fR (1GB).
	.RE

	.sp
	.ne 2
	.na
	\fBmetaslab_aliquot\fR (ulong)
	.ad
	.RS 12n
	Metaslab granularity, in bytes. This is roughly similar to what would be
	referred to as the "stripe size" in traditional RAID arrays. In normal
	operation, ZFS will try to write this amount of data to a top-level vdev
	before moving on to the next one.
	.sp
	Default value: \fB524,288\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBmetaslab_bias_enabled\fR (int)
	.ad
	.RS 12n
	Enable metaslab group biasing based on its vdev's over- or under-utilization
	relative to the pool.
	.sp
	Use \fB1\fR for yes (default) and \fB0\fR for no.
	.RE

	.sp
	.ne 2
	.na
	\fBmetaslab_force_ganging\fR (ulong)
	.ad
	.RS 12n
	Make some blocks above a certain size be gang blocks. This option is used
	by the test suite to facilitate testing.
	.sp
	Default value: \fB16,777,217\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_keep_log_spacemaps_at_export\fR (int)
	.ad
	.RS 12n
	Prevent log spacemaps from being destroyed during pool exports and destroys.
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_metaslab_segment_weight_enabled\fR (int)
	.ad
	.RS 12n
	Enable/disable segment-based metaslab selection.
	.sp
	Use \fB1\fR for yes (default) and \fB0\fR for no.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_metaslab_switch_threshold\fR (int)
	.ad
	.RS 12n
	When using segment-based metaslab selection, continue allocating
	from the active metaslab until \fBzfs_metaslab_switch_threshold\fR
	worth of buckets have been exhausted.
	.sp
	Default value: \fB2\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBmetaslab_debug_load\fR (int)
	.ad
	.RS 12n
	Load all metaslabs during pool import.
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBmetaslab_debug_unload\fR (int)
	.ad
	.RS 12n
	Prevent metaslabs from being unloaded.
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBmetaslab_fragmentation_factor_enabled\fR (int)
	.ad
	.RS 12n
	Enable use of the fragmentation metric in computing metaslab weights.
	.sp
	Use \fB1\fR for yes (default) and \fB0\fR for no.
	.RE

	.sp
	.ne 2
	.na
	\fBmetaslab_df_max_search\fR (int)
	.ad
	.RS 12n
	Maximum distance to search forward from the last offset. Without this limit,
	fragmented pools can see >100,000 iterations and metaslab_block_picker()
	becomes the performance limiting factor on high-performance storage.

	With the default setting of 16MB, we typically see less than 500 iterations,
	even with very fragmented, ashift=9 pools. The maximum number of iterations
	possible is: \fBmetaslab_df_max_search / (2 * (1<<ashift))\fR.
	With the default setting of 16MB this is 16*1024 (with ashift=9) or 2048
	(with ashift=12).
	.sp
	Default value: \fB16,777,216\fR (16MB)
	.RE

	.sp
	.ne 2
	.na
	\fBmetaslab_df_use_largest_segment\fR (int)
	.ad
	.RS 12n
	If we are not searching forward (due to metaslab_df_max_search,
	metaslab_df_free_pct, or metaslab_df_alloc_threshold), this tunable controls
	what segment is used. If it is set, we will use the largest free segment.
	If it is not set, we will use a segment of exactly the requested size (or
	larger).
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_metaslab_max_size_cache_sec\fR (ulong)
	.ad
	.RS 12n
	When we unload a metaslab, we cache the size of the largest free chunk. We use
	that cached size to determine whether or not to load a metaslab for a given
	allocation. As more frees accumulate in that metaslab while it's unloaded, the
	cached max size becomes less and less accurate. After a number of seconds
	controlled by this tunable, we stop considering the cached max size and start
	considering only the histogram instead.
	.sp
	Default value: \fB3600 seconds\fR (one hour)
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_metaslab_mem_limit\fR (int)
	.ad
	.RS 12n
	When we are loading a new metaslab, we check the amount of memory being used
	to store metaslab range trees. If it is over a threshold, we attempt to unload
	the least recently used metaslab to prevent the system from clogging all of
	its memory with range trees. This tunable sets the percentage of total system
	memory that is the threshold.
	.sp
	Default value: \fB25 percent\fR
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_default_ms_count\fR (int)
	.ad
	.RS 12n
	When a vdev is added target this number of metaslabs per top-level vdev.
	.sp
	Default value: \fB200\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_default_ms_shift\fR (int)
	.ad
	.RS 12n
	Default limit for metaslab size.
	.sp
	Default value: \fB29\fR [meaning (1 << 29) = 512MB].
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_max_auto_ashift\fR (ulong)
	.ad
	.RS 12n
	Maximum ashift used when optimizing for logical -> physical sector size on new
	top-level vdevs.
	.sp
	Default value: \fBASHIFT_MAX\fR (16).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_min_auto_ashift\fR (ulong)
	.ad
	.RS 12n
	Minimum ashift used when creating new top-level vdevs.
	.sp
	Default value: \fBASHIFT_MIN\fR (9).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_min_ms_count\fR (int)
	.ad
	.RS 12n
	Minimum number of metaslabs to create in a top-level vdev.
	.sp
	Default value: \fB16\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBvdev_validate_skip\fR (int)
	.ad
	.RS 12n
	Skip label validation steps during pool import. Changing is not recommended
	unless you know what you are doing and are recovering a damaged label.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_ms_count_limit\fR (int)
	.ad
	.RS 12n
	Practical upper limit of total metaslabs per top-level vdev.
	.sp
	Default value: \fB131,072\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBmetaslab_preload_enabled\fR (int)
	.ad
	.RS 12n
	Enable metaslab group preloading.
	.sp
	Use \fB1\fR for yes (default) and \fB0\fR for no.
	.RE

	.sp
	.ne 2
	.na
	\fBmetaslab_lba_weighting_enabled\fR (int)
	.ad
	.RS 12n
	Give more weight to metaslabs with lower LBAs, assuming they have
	greater bandwidth as is typically the case on a modern constant
	angular velocity disk drive.
	.sp
	Use \fB1\fR for yes (default) and \fB0\fR for no.
	.RE

	.sp
	.ne 2
	.na
	\fBmetaslab_unload_delay\fR (int)
	.ad
	.RS 12n
	After a metaslab is used, we keep it loaded for this many txgs, to attempt to
	reduce unnecessary reloading. Note that both this many txgs and
	\fBmetaslab_unload_delay_ms\fR milliseconds must pass before unloading will
	occur.
	.sp
	Default value: \fB32\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBmetaslab_unload_delay_ms\fR (int)
	.ad
	.RS 12n
	After a metaslab is used, we keep it loaded for this many milliseconds, to
	attempt to reduce unnecessary reloading. Note that both this many
	milliseconds and \fBmetaslab_unload_delay\fR txgs must pass before unloading
	will occur.
	.sp
	Default value: \fB600000\fR (ten minutes).
	.RE

	.sp
	.ne 2
	.na
	\fBsend_holes_without_birth_time\fR (int)
	.ad
	.RS 12n
	When set, the hole_birth optimization will not be used, and all holes will
	always be sent on zfs send. This is useful if you suspect your datasets are
	affected by a bug in hole_birth.
	.sp
	Use \fB1\fR for on (default) and \fB0\fR for off.
	.RE

	.sp
	.ne 2
	.na
	\fBspa_config_path\fR (charp)
	.ad
	.RS 12n
	SPA config file
	.sp
	Default value: \fB/etc/zfs/zpool.cache\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBspa_asize_inflation\fR (int)
	.ad
	.RS 12n
	Multiplication factor used to estimate actual disk consumption from the
	size of data being written. The default value is a worst case estimate,
	but lower values may be valid for a given pool depending on its
	configuration. Pool administrators who understand the factors involved
	may wish to specify a more realistic inflation factor, particularly if
	they operate close to quota or capacity limits.
	.sp
	Default value: \fB24\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBspa_load_print_vdev_tree\fR (int)
	.ad
	.RS 12n
	Whether to print the vdev tree in the debugging message buffer during pool import.
	Use 0 to disable and 1 to enable.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBspa_load_verify_data\fR (int)
	.ad
	.RS 12n
	Whether to traverse data blocks during an "extreme rewind" (\fB-X\fR)
	import. Use 0 to disable and 1 to enable.

	An extreme rewind import normally performs a full traversal of all
	blocks in the pool for verification. If this parameter is set to 0,
	the traversal skips non-metadata blocks. It can be toggled once the
	import has started to stop or start the traversal of non-metadata blocks.
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBspa_load_verify_metadata\fR (int)
	.ad
	.RS 12n
	Whether to traverse blocks during an "extreme rewind" (\fB-X\fR)
	pool import. Use 0 to disable and 1 to enable.

	An extreme rewind import normally performs a full traversal of all
	blocks in the pool for verification. If this parameter is set to 0,
	the traversal is not performed. It can be toggled once the import has
	started to stop or start the traversal.
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBspa_load_verify_shift\fR (int)
	.ad
	.RS 12n
	Sets the maximum number of bytes to consume during pool import to the log2
	fraction of the target ARC size.
	.sp
	Default value: \fB4\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBspa_slop_shift\fR (int)
	.ad
	.RS 12n
	Normally, we don't allow the last 3.2% (1/(2^spa_slop_shift)) of space
	in the pool to be consumed. This ensures that we don't run the pool
	completely out of space, due to unaccounted changes (e.g. to the MOS).
	It also limits the worst-case time to allocate space. If we have
	less than this amount of free space, most ZPL operations (e.g. write,
	create) will return ENOSPC.
	.sp
	Default value: \fB5\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBvdev_removal_max_span\fR (int)
	.ad
	.RS 12n
	During top-level vdev removal, chunks of data are copied from the vdev
	which may include free space in order to trade bandwidth for IOPS.
	This parameter determines the maximum span of free space (in bytes)
	which will be included as "unnecessary" data in a chunk of copied data.

	The default value here was chosen to align with
	\fBzfs_vdev_read_gap_limit\fR, which is a similar concept when doing
	regular reads (but there's no reason it has to be the same).
	.sp
	Default value: \fB32,768\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBvdev_file_logical_ashift\fR (ulong)
	.ad
	.RS 12n
	Logical ashift for file-based devices.
	.sp
	Default value: \fB9\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBvdev_file_physical_ashift\fR (ulong)
	.ad
	.RS 12n
	Physical ashift for file-based devices.
	.sp
	Default value: \fB9\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzap_iterate_prefetch\fR (int)
	.ad
	.RS 12n
	If this is set, when we start iterating over a ZAP object, zfs will prefetch
	the entire object (all leaf blocks). However, this is limited by
	\fBdmu_prefetch_max\fR.
	.sp
	Use \fB1\fR for on (default) and \fB0\fR for off.
	.RE

	.sp
	.ne 2
	.na
	\fBzfetch_array_rd_sz\fR (ulong)
	.ad
	.RS 12n
	If prefetching is enabled, disable prefetching for reads larger than this size.
	.sp
	Default value: \fB1,048,576\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfetch_max_distance\fR (uint)
	.ad
	.RS 12n
	-Max bytes to prefetch per stream (default 8MB).
	+Max bytes to prefetch per stream.
	.sp
	-Default value: \fB8,388,608\fR.
	+Default value: \fB8,388,608\fR (8MB).
	.RE

	.sp
	.ne 2
	.na
	+\fBzfetch_max_idistance\fR (uint)
	+.ad
	+.RS 12n
	+Max bytes to prefetch indirects for per stream.
	+.sp
	+Default vaule: \fB67,108,864\fR (64MB).
	+.RE
	+
	+.sp
	+.ne 2
	+.na
	\fBzfetch_max_streams\fR (uint)
	.ad
	.RS 12n
	Max number of streams per zfetch (prefetch streams per file).
	.sp
	Default value: \fB8\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfetch_min_sec_reap\fR (uint)
	.ad
	.RS 12n
	Min time before an active prefetch stream can be reclaimed
	.sp
	Default value: \fB2\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_abd_scatter_enabled\fR (int)
	.ad
	.RS 12n
	Enables ARC from using scatter/gather lists and forces all allocations to be
	linear in kernel memory. Disabling can improve performance in some code paths
	at the expense of fragmented kernel memory.
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_abd_scatter_max_order\fR (iunt)
	.ad
	.RS 12n
	Maximum number of consecutive memory pages allocated in a single block for
	scatter/gather lists. Default value is specified by the kernel itself.
	.sp
	Default value: \fB10\fR at the time of this writing.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_abd_scatter_min_size\fR (uint)
	.ad
	.RS 12n
	This is the minimum allocation size that will use scatter (page-based)
	ABD's. Smaller allocations will use linear ABD's.
	.sp
	Default value: \fB1536\fR (512B and 1KB allocations will be linear).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_dnode_limit\fR (ulong)
	.ad
	.RS 12n
	When the number of bytes consumed by dnodes in the ARC exceeds this number of
	bytes, try to unpin some of it in response to demand for non-metadata. This
	value acts as a ceiling to the amount of dnode metadata, and defaults to 0 which
	indicates that a percent which is based on \fBzfs_arc_dnode_limit_percent\fR of
	the ARC meta buffers that may be used for dnodes.

	See also \fBzfs_arc_meta_prune\fR which serves a similar purpose but is used
	when the amount of metadata in the ARC exceeds \fBzfs_arc_meta_limit\fR rather
	than in response to overall demand for non-metadata.

	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_dnode_limit_percent\fR (ulong)
	.ad
	.RS 12n
	Percentage that can be consumed by dnodes of ARC meta buffers.
	.sp
	See also \fBzfs_arc_dnode_limit\fR which serves a similar purpose but has a
	higher priority if set to nonzero value.
	.sp
	Default value: \fB10\fR%.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_dnode_reduce_percent\fR (ulong)
	.ad
	.RS 12n
	Percentage of ARC dnodes to try to scan in response to demand for non-metadata
	when the number of bytes consumed by dnodes exceeds \fBzfs_arc_dnode_limit\fR.

	.sp
	Default value: \fB10\fR% of the number of dnodes in the ARC.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_average_blocksize\fR (int)
	.ad
	.RS 12n
	The ARC's buffer hash table is sized based on the assumption of an average
	block size of \fBzfs_arc_average_blocksize\fR (default 8K). This works out
	to roughly 1MB of hash table per 1GB of physical memory with 8-byte pointers.
	For configurations with a known larger average block size this value can be
	increased to reduce the memory footprint.

	.sp
	Default value: \fB8192\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_eviction_pct\fR (int)
	.ad
	.RS 12n
	When \fBarc_is_overflowing()\fR, \fBarc_get_data_impl()\fR waits for this
	percent of the requested amount of data to be evicted. For example, by
	default for every 2KB that's evicted, 1KB of it may be "reused" by a new
	allocation. Since this is above 100%, it ensures that progress is made
	towards getting \fBarc_size\fR under \fBarc_c\fR. Since this is finite, it
	ensures that allocations can still happen, even during the potentially long
	time that \fBarc_size\fR is more than \fBarc_c\fR.
	.sp
	Default value: \fB200\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_evict_batch_limit\fR (int)
	.ad
	.RS 12n
	Number ARC headers to evict per sub-list before proceeding to another sub-list.
	This batch-style operation prevents entire sub-lists from being evicted at once
	but comes at a cost of additional unlocking and locking.
	.sp
	Default value: \fB10\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_grow_retry\fR (int)
	.ad
	.RS 12n
	If set to a non zero value, it will replace the arc_grow_retry value with this value.
	The arc_grow_retry value (default 5) is the number of seconds the ARC will wait before
	trying to resume growth after a memory pressure event.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_lotsfree_percent\fR (int)
	.ad
	.RS 12n
	Throttle I/O when free system memory drops below this percentage of total
	system memory. Setting this value to 0 will disable the throttle.
	.sp
	Default value: \fB10\fR%.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_max\fR (ulong)
	.ad
	.RS 12n
	Max size of ARC in bytes. If set to 0 then the max size of ARC is determined
	by the amount of system memory installed. For Linux, 1/2 of system memory will
	be used as the limit. For FreeBSD, the larger of all system memory - 1GB or
	5/8 of system memory will be used as the limit. This value must be at least
	67108864 (64 megabytes).
	.sp
	This value can be changed dynamically with some caveats. It cannot be set back
	to 0 while running and reducing it below the current ARC size will not cause
	the ARC to shrink without memory pressure to induce shrinking.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_meta_adjust_restarts\fR (ulong)
	.ad
	.RS 12n
	The number of restart passes to make while scanning the ARC attempting
	the free buffers in order to stay below the \fBzfs_arc_meta_limit\fR.
	This value should not need to be tuned but is available to facilitate
	performance analysis.
	.sp
	Default value: \fB4096\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_meta_limit\fR (ulong)
	.ad
	.RS 12n
	The maximum allowed size in bytes that meta data buffers are allowed to
	consume in the ARC. When this limit is reached meta data buffers will
	be reclaimed even if the overall arc_c_max has not been reached. This
	value defaults to 0 which indicates that a percent which is based on
	\fBzfs_arc_meta_limit_percent\fR of the ARC may be used for meta data.
	.sp
	This value my be changed dynamically except that it cannot be set back to 0
	for a specific percent of the ARC; it must be set to an explicit value.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_meta_limit_percent\fR (ulong)
	.ad
	.RS 12n
	Percentage of ARC buffers that can be used for meta data.

	See also \fBzfs_arc_meta_limit\fR which serves a similar purpose but has a
	higher priority if set to nonzero value.

	.sp
	Default value: \fB75\fR%.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_meta_min\fR (ulong)
	.ad
	.RS 12n
	The minimum allowed size in bytes that meta data buffers may consume in
	the ARC. This value defaults to 0 which disables a floor on the amount
	of the ARC devoted meta data.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_meta_prune\fR (int)
	.ad
	.RS 12n
	The number of dentries and inodes to be scanned looking for entries
	which can be dropped. This may be required when the ARC reaches the
	\fBzfs_arc_meta_limit\fR because dentries and inodes can pin buffers
	in the ARC. Increasing this value will cause to dentry and inode caches
	to be pruned more aggressively. Setting this value to 0 will disable
	pruning the inode and dentry caches.
	.sp
	Default value: \fB10,000\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_meta_strategy\fR (int)
	.ad
	.RS 12n
	Define the strategy for ARC meta data buffer eviction (meta reclaim strategy).
	A value of 0 (META_ONLY) will evict only the ARC meta data buffers.
	A value of 1 (BALANCED) indicates that additional data buffers may be evicted if
	that is required to in order to evict the required number of meta data buffers.
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_min\fR (ulong)
	.ad
	.RS 12n
	Min size of ARC in bytes. If set to 0 then arc_c_min will default to
	consuming the larger of 32M or 1/32 of total system memory.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_min_prefetch_ms\fR (int)
	.ad
	.RS 12n
	Minimum time prefetched blocks are locked in the ARC, specified in ms.
	A value of \fB0\fR will default to 1000 ms.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_min_prescient_prefetch_ms\fR (int)
	.ad
	.RS 12n
	Minimum time "prescient prefetched" blocks are locked in the ARC, specified
	in ms. These blocks are meant to be prefetched fairly aggressively ahead of
	the code that may use them. A value of \fB0\fR will default to 6000 ms.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_max_missing_tvds\fR (int)
	.ad
	.RS 12n
	Number of missing top-level vdevs which will be allowed during
	pool import (only in read-only mode).
	.sp
	Default value: \fB0\fR
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_max_nvlist_src_size\fR (ulong)
	.ad
	.RS 12n
	Maximum size in bytes allowed to be passed as zc_nvlist_src_size for ioctls on
	/dev/zfs. This prevents a user from causing the kernel to allocate an excessive
	amount of memory. When the limit is exceeded, the ioctl fails with EINVAL and a
	description of the error is sent to the zfs-dbgmsg log. This parameter should
	not need to be touched under normal circumstances. On FreeBSD, the default is
	based on the system limit on user wired memory. On Linux, the default is
	\fBKMALLOC_MAX_SIZE\fR .
	.sp
	Default value: \fB0\fR (kernel decides)
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_multilist_num_sublists\fR (int)
	.ad
	.RS 12n
	To allow more fine-grained locking, each ARC state contains a series
	of lists for both data and meta data objects. Locking is performed at
	the level of these "sub-lists". This parameters controls the number of
	sub-lists per ARC state, and also applies to other uses of the
	multilist data structure.
	.sp
	Default value: \fB4\fR or the number of online CPUs, whichever is greater
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_overflow_shift\fR (int)
	.ad
	.RS 12n
	The ARC size is considered to be overflowing if it exceeds the current
	ARC target size (arc_c) by a threshold determined by this parameter.
	The threshold is calculated as a fraction of arc_c using the formula
	"arc_c >> \fBzfs_arc_overflow_shift\fR".

	The default value of 8 causes the ARC to be considered to be overflowing
	if it exceeds the target size by 1/256th (0.3%) of the target size.

	When the ARC is overflowing, new buffer allocations are stalled until
	the reclaim thread catches up and the overflow condition no longer exists.
	.sp
	Default value: \fB8\fR.
	.RE

	.sp
	.ne 2
	.na

	\fBzfs_arc_p_min_shift\fR (int)
	.ad
	.RS 12n
	If set to a non zero value, this will update arc_p_min_shift (default 4)
	with the new value.
	arc_p_min_shift is used to shift of arc_c for calculating both min and max
	max arc_p
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_p_dampener_disable\fR (int)
	.ad
	.RS 12n
	Disable arc_p adapt dampener
	.sp
	Use \fB1\fR for yes (default) and \fB0\fR to disable.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_shrink_shift\fR (int)
	.ad
	.RS 12n
	If set to a non zero value, this will update arc_shrink_shift (default 7)
	with the new value.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_pc_percent\fR (uint)
	.ad
	.RS 12n
	Percent of pagecache to reclaim arc to

	This tunable allows ZFS arc to play more nicely with the kernel's LRU
	pagecache. It can guarantee that the ARC size won't collapse under scanning
	pressure on the pagecache, yet still allows arc to be reclaimed down to
	zfs_arc_min if necessary. This value is specified as percent of pagecache
	size (as measured by NR_FILE_PAGES) where that percent may exceed 100. This
	only operates during memory pressure/reclaim.
	.sp
	Default value: \fB0\fR% (disabled).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_shrinker_limit\fR (int)
	.ad
	.RS 12n
	This is a limit on how many pages the ARC shrinker makes available for
	eviction in response to one page allocation attempt. Note that in
	practice, the kernel's shrinker can ask us to evict up to about 4x this
	for one allocation attempt.
	.sp
	The default limit of 10,000 (in practice, 160MB per allocation attempt with
	4K pages) limits the amount of time spent attempting to reclaim ARC memory to
	less than 100ms per allocation attempt, even with a small average compressed
	block size of ~8KB.
	.sp
	The parameter can be set to 0 (zero) to disable the limit.
	.sp
	This parameter only applies on Linux.
	.sp
	Default value: \fB10,000\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_arc_sys_free\fR (ulong)
	.ad
	.RS 12n
	The target number of bytes the ARC should leave as free memory on the system.
	Defaults to the larger of 1/64 of physical memory or 512K. Setting this
	option to a non-zero value will override the default.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_autoimport_disable\fR (int)
	.ad
	.RS 12n
	Disable pool import at module load by ignoring the cache file (typically \fB/etc/zfs/zpool.cache\fR).
	.sp
	Use \fB1\fR for yes (default) and \fB0\fR for no.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_checksum_events_per_second\fR (uint)
	.ad
	.RS 12n
	Rate limit checksum events to this many per second. Note that this should
	not be set below the zed thresholds (currently 10 checksums over 10 sec)
	or else zed may not trigger any action.
	.sp
	Default value: 20
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_commit_timeout_pct\fR (int)
	.ad
	.RS 12n
	This controls the amount of time that a ZIL block (lwb) will remain "open"
	when it isn't "full", and it has a thread waiting for it to be committed to
	stable storage. The timeout is scaled based on a percentage of the last lwb
	latency to avoid significantly impacting the latency of each individual
	transaction record (itx).
	.sp
	Default value: \fB5\fR%.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_condense_indirect_commit_entry_delay_ms\fR (int)
	.ad
	.RS 12n
	Vdev indirection layer (used for device removal) sleeps for this many
	milliseconds during mapping generation. Intended for use with the test suite
	to throttle vdev removal speed.
	.sp
	Default value: \fB0\fR (no throttle).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_condense_indirect_vdevs_enable\fR (int)
	.ad
	.RS 12n
	Enable condensing indirect vdev mappings. When set to a non-zero value,
	attempt to condense indirect vdev mappings if the mapping uses more than
	\fBzfs_condense_min_mapping_bytes\fR bytes of memory and if the obsolete
	space map object uses more than \fBzfs_condense_max_obsolete_bytes\fR
	bytes on-disk. The condensing process is an attempt to save memory by
	removing obsolete mappings.
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_condense_max_obsolete_bytes\fR (ulong)
	.ad
	.RS 12n
	Only attempt to condense indirect vdev mappings if the on-disk size
	of the obsolete space map object is greater than this number of bytes
	(see \fBfBzfs_condense_indirect_vdevs_enable\fR).
	.sp
	Default value: \fB1,073,741,824\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_condense_min_mapping_bytes\fR (ulong)
	.ad
	.RS 12n
	Minimum size vdev mapping to attempt to condense (see
	\fBzfs_condense_indirect_vdevs_enable\fR).
	.sp
	Default value: \fB131,072\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_dbgmsg_enable\fR (int)
	.ad
	.RS 12n
	Internally ZFS keeps a small log to facilitate debugging. By default the log
	is disabled, to enable it set this option to 1. The contents of the log can
	be accessed by reading the /proc/spl/kstat/zfs/dbgmsg file. Writing 0 to
	this proc file clears the log.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_dbgmsg_maxsize\fR (int)
	.ad
	.RS 12n
	The maximum size in bytes of the internal ZFS debug log.
	.sp
	Default value: \fB4M\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_dbuf_state_index\fR (int)
	.ad
	.RS 12n
	This feature is currently unused. It is normally used for controlling what
	reporting is available under /proc/spl/kstat/zfs.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_deadman_enabled\fR (int)
	.ad
	.RS 12n
	When a pool sync operation takes longer than \fBzfs_deadman_synctime_ms\fR
	milliseconds, or when an individual I/O takes longer than
	\fBzfs_deadman_ziotime_ms\fR milliseconds, then the operation is considered to
	be "hung". If \fBzfs_deadman_enabled\fR is set then the deadman behavior is
	invoked as described by the \fBzfs_deadman_failmode\fR module option.
	By default the deadman is enabled and configured to \fBwait\fR which results
	in "hung" I/Os only being logged. The deadman is automatically disabled
	when a pool gets suspended.
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_deadman_failmode\fR (charp)
	.ad
	.RS 12n
	Controls the failure behavior when the deadman detects a "hung" I/O. Valid
	values are \fBwait\fR, \fBcontinue\fR, and \fBpanic\fR.
	.sp
	\fBwait\fR - Wait for a "hung" I/O to complete. For each "hung" I/O a
	"deadman" event will be posted describing that I/O.
	.sp
	\fBcontinue\fR - Attempt to recover from a "hung" I/O by re-dispatching it
	to the I/O pipeline if possible.
	.sp
	\fBpanic\fR - Panic the system. This can be used to facilitate an automatic
	fail-over to a properly configured fail-over partner.
	.sp
	Default value: \fBwait\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_deadman_checktime_ms\fR (int)
	.ad
	.RS 12n
	Check time in milliseconds. This defines the frequency at which we check
	for hung I/O and potentially invoke the \fBzfs_deadman_failmode\fR behavior.
	.sp
	Default value: \fB60,000\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_deadman_synctime_ms\fR (ulong)
	.ad
	.RS 12n
	Interval in milliseconds after which the deadman is triggered and also
	the interval after which a pool sync operation is considered to be "hung".
	Once this limit is exceeded the deadman will be invoked every
	\fBzfs_deadman_checktime_ms\fR milliseconds until the pool sync completes.
	.sp
	Default value: \fB600,000\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_deadman_ziotime_ms\fR (ulong)
	.ad
	.RS 12n
	Interval in milliseconds after which the deadman is triggered and an
	individual I/O operation is considered to be "hung". As long as the I/O
	remains "hung" the deadman will be invoked every \fBzfs_deadman_checktime_ms\fR
	milliseconds until the I/O completes.
	.sp
	Default value: \fB300,000\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_dedup_prefetch\fR (int)
	.ad
	.RS 12n
	Enable prefetching dedup-ed blks
	.sp
	Use \fB1\fR for yes and \fB0\fR to disable (default).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_delay_min_dirty_percent\fR (int)
	.ad
	.RS 12n
	Start to delay each transaction once there is this amount of dirty data,
	expressed as a percentage of \fBzfs_dirty_data_max\fR.
	This value should be >= zfs_vdev_async_write_active_max_dirty_percent.
	See the section "ZFS TRANSACTION DELAY".
	.sp
	Default value: \fB60\fR%.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_delay_scale\fR (int)
	.ad
	.RS 12n
	This controls how quickly the transaction delay approaches infinity.
	Larger values cause longer delays for a given amount of dirty data.
	.sp
	For the smoothest delay, this value should be about 1 billion divided
	by the maximum number of operations per second. This will smoothly
	handle between 10x and 1/10th this number.
	.sp
	See the section "ZFS TRANSACTION DELAY".
	.sp
	Note: \fBzfs_delay_scale\fR * \fBzfs_dirty_data_max\fR must be < 2^64.
	.sp
	Default value: \fB500,000\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_disable_ivset_guid_check\fR (int)
	.ad
	.RS 12n
	Disables requirement for IVset guids to be present and match when doing a raw
	receive of encrypted datasets. Intended for users whose pools were created with
	-ZFS on Linux pre-release versions and now have compatibility issues.
	+OpenZFS pre-release versions and now have compatibility issues.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_key_max_salt_uses\fR (ulong)
	.ad
	.RS 12n
	Maximum number of uses of a single salt value before generating a new one for
	encrypted datasets. The default value is also the maximum that will be
	accepted.
	.sp
	Default value: \fB400,000,000\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_object_mutex_size\fR (uint)
	.ad
	.RS 12n
	Size of the znode hashtable used for holds.

	Due to the need to hold locks on objects that may not exist yet, kernel mutexes
	are not created per-object and instead a hashtable is used where collisions
	will result in objects waiting when there is not actually contention on the
	same object.
	.sp
	Default value: \fB64\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_slow_io_events_per_second\fR (int)
	.ad
	.RS 12n
	Rate limit delay zevents (which report slow I/Os) to this many per second.
	.sp
	Default value: 20
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_unflushed_max_mem_amt\fR (ulong)
	.ad
	.RS 12n
	Upper-bound limit for unflushed metadata changes to be held by the
	log spacemap in memory (in bytes).
	.sp
	Default value: \fB1,073,741,824\fR (1GB).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_unflushed_max_mem_ppm\fR (ulong)
	.ad
	.RS 12n
	Percentage of the overall system memory that ZFS allows to be used
	for unflushed metadata changes by the log spacemap.
	(value is calculated over 1000000 for finer granularity).
	.sp
	Default value: \fB1000\fR (which is divided by 1000000, resulting in
	the limit to be \fB0.1\fR% of memory)
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_unflushed_log_block_max\fR (ulong)
	.ad
	.RS 12n
	Describes the maximum number of log spacemap blocks allowed for each pool.
	The default value of 262144 means that the space in all the log spacemaps
	can add up to no more than 262144 blocks (which means 32GB of logical
	space before compression and ditto blocks, assuming that blocksize is
	128k).
	.sp
	This tunable is important because it involves a trade-off between import
	time after an unclean export and the frequency of flushing metaslabs.
	The higher this number is, the more log blocks we allow when the pool is
	active which means that we flush metaslabs less often and thus decrease
	the number of I/Os for spacemap updates per TXG.
	At the same time though, that means that in the event of an unclean export,
	there will be more log spacemap blocks for us to read, inducing overhead
	in the import time of the pool.
	The lower the number, the amount of flushing increases destroying log
	blocks quicker as they become obsolete faster, which leaves less blocks
	to be read during import time after a crash.
	.sp
	Each log spacemap block existing during pool import leads to approximately
	one extra logical I/O issued.
	This is the reason why this tunable is exposed in terms of blocks rather
	than space used.
	.sp
	Default value: \fB262144\fR (256K).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_unflushed_log_block_min\fR (ulong)
	.ad
	.RS 12n
	If the number of metaslabs is small and our incoming rate is high, we
	could get into a situation that we are flushing all our metaslabs every
	TXG.
	Thus we always allow at least this many log blocks.
	.sp
	Default value: \fB1000\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_unflushed_log_block_pct\fR (ulong)
	.ad
	.RS 12n
	Tunable used to determine the number of blocks that can be used for
	the spacemap log, expressed as a percentage of the total number of
	metaslabs in the pool.
	.sp
	Default value: \fB400\fR (read as \fB400\fR% - meaning that the number
	of log spacemap blocks are capped at 4 times the number of
	metaslabs in the pool).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_unlink_suspend_progress\fR (uint)
	.ad
	.RS 12n
	When enabled, files will not be asynchronously removed from the list of pending
	unlinks and the space they consume will be leaked. Once this option has been
	disabled and the dataset is remounted, the pending unlinks will be processed
	and the freed space returned to the pool.
	This option is used by the test suite to facilitate testing.
	.sp
	Uses \fB0\fR (default) to allow progress and \fB1\fR to pause progress.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_delete_blocks\fR (ulong)
	.ad
	.RS 12n
	This is the used to define a large file for the purposes of delete. Files
	containing more than \fBzfs_delete_blocks\fR will be deleted asynchronously
	while smaller files are deleted synchronously. Decreasing this value will
	reduce the time spent in an unlink(2) system call at the expense of a longer
	delay before the freed space is available.
	.sp
	Default value: \fB20,480\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_dirty_data_max\fR (int)
	.ad
	.RS 12n
	Determines the dirty space limit in bytes. Once this limit is exceeded, new
	writes are halted until space frees up. This parameter takes precedence
	over \fBzfs_dirty_data_max_percent\fR.
	See the section "ZFS TRANSACTION DELAY".
	.sp
	Default value: \fB10\fR% of physical RAM, capped at \fBzfs_dirty_data_max_max\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_dirty_data_max_max\fR (int)
	.ad
	.RS 12n
	Maximum allowable value of \fBzfs_dirty_data_max\fR, expressed in bytes.
	This limit is only enforced at module load time, and will be ignored if
	\fBzfs_dirty_data_max\fR is later changed. This parameter takes
	precedence over \fBzfs_dirty_data_max_max_percent\fR. See the section
	"ZFS TRANSACTION DELAY".
	.sp
	Default value: \fB25\fR% of physical RAM.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_dirty_data_max_max_percent\fR (int)
	.ad
	.RS 12n
	Maximum allowable value of \fBzfs_dirty_data_max\fR, expressed as a
	percentage of physical RAM. This limit is only enforced at module load
	time, and will be ignored if \fBzfs_dirty_data_max\fR is later changed.
	The parameter \fBzfs_dirty_data_max_max\fR takes precedence over this
	one. See the section "ZFS TRANSACTION DELAY".
	.sp
	Default value: \fB25\fR%.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_dirty_data_max_percent\fR (int)
	.ad
	.RS 12n
	Determines the dirty space limit, expressed as a percentage of all
	memory. Once this limit is exceeded, new writes are halted until space frees
	up. The parameter \fBzfs_dirty_data_max\fR takes precedence over this
	one. See the section "ZFS TRANSACTION DELAY".
	.sp
	Default value: \fB10\fR%, subject to \fBzfs_dirty_data_max_max\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_dirty_data_sync_percent\fR (int)
	.ad
	.RS 12n
	Start syncing out a transaction group if there's at least this much dirty data
	as a percentage of \fBzfs_dirty_data_max\fR. This should be less than
	\fBzfs_vdev_async_write_active_min_dirty_percent\fR.
	.sp
	Default value: \fB20\fR% of \fBzfs_dirty_data_max\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_fallocate_reserve_percent\fR (uint)
	.ad
	.RS 12n
	Since ZFS is a copy-on-write filesystem with snapshots, blocks cannot be
	preallocated for a file in order to guarantee that later writes will not
	run out of space. Instead, fallocate() space preallocation only checks
	that sufficient space is currently available in the pool or the user's
	project quota allocation, and then creates a sparse file of the requested
	size. The requested space is multiplied by \fBzfs_fallocate_reserve_percent\fR
	to allow additional space for indirect blocks and other internal metadata.
	Setting this value to 0 disables support for fallocate(2) and returns
	EOPNOTSUPP for fallocate() space preallocation again.
	.sp
	Default value: \fB110\fR%
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_fletcher_4_impl\fR (string)
	.ad
	.RS 12n
	Select a fletcher 4 implementation.
	.sp
	Supported selectors are: \fBfastest\fR, \fBscalar\fR, \fBsse2\fR, \fBssse3\fR,
	\fBavx2\fR, \fBavx512f\fR, \fBavx512bw\fR, and \fBaarch64_neon\fR.
	All of the selectors except \fBfastest\fR and \fBscalar\fR require instruction
	set extensions to be available and will only appear if ZFS detects that they are
	present at runtime. If multiple implementations of fletcher 4 are available,
	the \fBfastest\fR will be chosen using a micro benchmark. Selecting \fBscalar\fR
	results in the original, CPU based calculation, being used. Selecting any option
	other than \fBfastest\fR and \fBscalar\fR results in vector instructions from
	the respective CPU instruction set being used.
	.sp
	Default value: \fBfastest\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_free_bpobj_enabled\fR (int)
	.ad
	.RS 12n
	Enable/disable the processing of the free_bpobj object.
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_async_block_max_blocks\fR (ulong)
	.ad
	.RS 12n
	Maximum number of blocks freed in a single txg.
	.sp
	Default value: \fBULONG_MAX\fR (unlimited).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_max_async_dedup_frees\fR (ulong)
	.ad
	.RS 12n
	Maximum number of dedup blocks freed in a single txg.
	.sp
	Default value: \fB100,000\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_override_estimate_recordsize\fR (ulong)
	.ad
	.RS 12n
	Record size calculation override for zfs send estimates.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_async_read_max_active\fR (int)
	.ad
	.RS 12n
	Maximum asynchronous read I/Os active to each device.
	See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB3\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_async_read_min_active\fR (int)
	.ad
	.RS 12n
	Minimum asynchronous read I/Os active to each device.
	See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_async_write_active_max_dirty_percent\fR (int)
	.ad
	.RS 12n
	When the pool has more than
	\fBzfs_vdev_async_write_active_max_dirty_percent\fR dirty data, use
	\fBzfs_vdev_async_write_max_active\fR to limit active async writes. If
	the dirty data is between min and max, the active I/O limit is linearly
	interpolated. See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB60\fR%.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_async_write_active_min_dirty_percent\fR (int)
	.ad
	.RS 12n
	When the pool has less than
	\fBzfs_vdev_async_write_active_min_dirty_percent\fR dirty data, use
	\fBzfs_vdev_async_write_min_active\fR to limit active async writes. If
	the dirty data is between min and max, the active I/O limit is linearly
	interpolated. See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB30\fR%.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_async_write_max_active\fR (int)
	.ad
	.RS 12n
	Maximum asynchronous write I/Os active to each device.
	See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB10\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_async_write_min_active\fR (int)
	.ad
	.RS 12n
	Minimum asynchronous write I/Os active to each device.
	See the section "ZFS I/O SCHEDULER".
	.sp
	Lower values are associated with better latency on rotational media but poorer
	resilver performance. The default value of 2 was chosen as a compromise. A
	value of 3 has been shown to improve resilver performance further at a cost of
	further increasing latency.
	.sp
	Default value: \fB2\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_initializing_max_active\fR (int)
	.ad
	.RS 12n
	Maximum initializing I/Os active to each device.
	See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_initializing_min_active\fR (int)
	.ad
	.RS 12n
	Minimum initializing I/Os active to each device.
	See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_max_active\fR (int)
	.ad
	.RS 12n
	The maximum number of I/Os active to each device. Ideally, this will be >=
	the sum of each queue's max_active. It must be at least the sum of each
	queue's min_active. See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB1,000\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_rebuild_max_active\fR (int)
	.ad
	.RS 12n
	Maximum sequential resilver I/Os active to each device.
	See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB3\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_rebuild_min_active\fR (int)
	.ad
	.RS 12n
	Minimum sequential resilver I/Os active to each device.
	See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_removal_max_active\fR (int)
	.ad
	.RS 12n
	Maximum removal I/Os active to each device.
	See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB2\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_removal_min_active\fR (int)
	.ad
	.RS 12n
	Minimum removal I/Os active to each device.
	See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_scrub_max_active\fR (int)
	.ad
	.RS 12n
	Maximum scrub I/Os active to each device.
	See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB2\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_scrub_min_active\fR (int)
	.ad
	.RS 12n
	Minimum scrub I/Os active to each device.
	See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_sync_read_max_active\fR (int)
	.ad
	.RS 12n
	Maximum synchronous read I/Os active to each device.
	See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB10\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_sync_read_min_active\fR (int)
	.ad
	.RS 12n
	Minimum synchronous read I/Os active to each device.
	See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB10\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_sync_write_max_active\fR (int)
	.ad
	.RS 12n
	Maximum synchronous write I/Os active to each device.
	See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB10\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_sync_write_min_active\fR (int)
	.ad
	.RS 12n
	Minimum synchronous write I/Os active to each device.
	See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB10\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_trim_max_active\fR (int)
	.ad
	.RS 12n
	Maximum trim/discard I/Os active to each device.
	See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB2\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_trim_min_active\fR (int)
	.ad
	.RS 12n
	Minimum trim/discard I/Os active to each device.
	See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_queue_depth_pct\fR (int)
	.ad
	.RS 12n
	Maximum number of queued allocations per top-level vdev expressed as
	a percentage of \fBzfs_vdev_async_write_max_active\fR which allows the
	system to detect devices that are more capable of handling allocations
	and to allocate more blocks to those devices. It allows for dynamic
	allocation distribution when devices are imbalanced as fuller devices
	will tend to be slower than empty devices.

	See also \fBzio_dva_throttle_enabled\fR.
	.sp
	Default value: \fB1000\fR%.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_expire_snapshot\fR (int)
	.ad
	.RS 12n
	Seconds to expire .zfs/snapshot
	.sp
	Default value: \fB300\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_admin_snapshot\fR (int)
	.ad
	.RS 12n
	Allow the creation, removal, or renaming of entries in the .zfs/snapshot
	directory to cause the creation, destruction, or renaming of snapshots.
	When enabled this functionality works both locally and over NFS exports
	which have the 'no_root_squash' option set. This functionality is disabled
	by default.
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_flags\fR (int)
	.ad
	.RS 12n
	Set additional debugging flags. The following flags may be bitwise-or'd
	together.
	.sp
	.TS
	box;
	rB lB
	lB lB
	r l.
	Value Symbolic Name
	Description
	_
	1 ZFS_DEBUG_DPRINTF
	Enable dprintf entries in the debug log.
	_
	2 ZFS_DEBUG_DBUF_VERIFY *
	Enable extra dbuf verifications.
	_
	4 ZFS_DEBUG_DNODE_VERIFY *
	Enable extra dnode verifications.
	_
	8 ZFS_DEBUG_SNAPNAMES
	Enable snapshot name verification.
	_
	16 ZFS_DEBUG_MODIFY
	Check for illegally modified ARC buffers.
	_
	64 ZFS_DEBUG_ZIO_FREE
	Enable verification of block frees.
	_
	128 ZFS_DEBUG_HISTOGRAM_VERIFY
	Enable extra spacemap histogram verifications.
	_
	256 ZFS_DEBUG_METASLAB_VERIFY
	Verify space accounting on disk matches in-core range_trees.
	_
	512 ZFS_DEBUG_SET_ERROR
	Enable SET_ERROR and dprintf entries in the debug log.
	_
	1024 ZFS_DEBUG_INDIRECT_REMAP
	Verify split blocks created by device removal.
	_
	2048 ZFS_DEBUG_TRIM
	Verify TRIM ranges are always within the allocatable range tree.
	_
	4096 ZFS_DEBUG_LOG_SPACEMAP
	Verify that the log summary is consistent with the spacemap log
	and enable zfs_dbgmsgs for metaslab loading and flushing.
	.TE
	.sp
	* Requires debug build.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_free_leak_on_eio\fR (int)
	.ad
	.RS 12n
	If destroy encounters an EIO while reading metadata (e.g. indirect
	blocks), space referenced by the missing metadata can not be freed.
	Normally this causes the background destroy to become "stalled", as
	it is unable to make forward progress. While in this stalled state,
	all remaining space to free from the error-encountering filesystem is
	"temporarily leaked". Set this flag to cause it to ignore the EIO,
	permanently leak the space from indirect blocks that can not be read,
	and continue to free everything else that it can.

	The default, "stalling" behavior is useful if the storage partially
	fails (i.e. some but not all i/os fail), and then later recovers. In
	this case, we will be able to continue pool operations while it is
	partially failed, and when it recovers, we can continue to free the
	space, with no leaks. However, note that this case is actually
	fairly rare.

	Typically pools either (a) fail completely (but perhaps temporarily,
	e.g. a top-level vdev going offline), or (b) have localized,
	permanent errors (e.g. disk returns the wrong data due to bit flip or
	firmware bug). In case (a), this setting does not matter because the
	pool will be suspended and the sync thread will not be able to make
	forward progress regardless. In case (b), because the error is
	permanent, the best we can do is leak the minimum amount of space,
	which is what setting this flag will do. Therefore, it is reasonable
	for this flag to normally be set, but we chose the more conservative
	approach of not setting it, so that there is no possibility of
	leaking space in the "partial temporary" failure case.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_free_min_time_ms\fR (int)
	.ad
	.RS 12n
	During a \fBzfs destroy\fR operation using \fBfeature@async_destroy\fR a minimum
	of this much time will be spent working on freeing blocks per txg.
	.sp
	Default value: \fB1,000\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_obsolete_min_time_ms\fR (int)
	.ad
	.RS 12n
	Similar to \fBzfs_free_min_time_ms\fR but for cleanup of old indirection records
	for removed vdevs.
	.sp
	Default value: \fB500\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_immediate_write_sz\fR (long)
	.ad
	.RS 12n
	Largest data block to write to zil. Larger blocks will be treated as if the
	dataset being written to had the property setting \fBlogbias=throughput\fR.
	.sp
	Default value: \fB32,768\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_initialize_value\fR (ulong)
	.ad
	.RS 12n
	Pattern written to vdev free space by \fBzpool initialize\fR.
	.sp
	Default value: \fB16,045,690,984,833,335,022\fR (0xdeadbeefdeadbeee).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_initialize_chunk_size\fR (ulong)
	.ad
	.RS 12n
	Size of writes used by \fBzpool initialize\fR.
	This option is used by the test suite to facilitate testing.
	.sp
	Default value: \fB1,048,576\fR
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_livelist_max_entries\fR (ulong)
	.ad
	.RS 12n
	The threshold size (in block pointers) at which we create a new sub-livelist.
	Larger sublists are more costly from a memory perspective but the fewer
	sublists there are, the lower the cost of insertion.
	.sp
	Default value: \fB500,000\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_livelist_min_percent_shared\fR (int)
	.ad
	.RS 12n
	If the amount of shared space between a snapshot and its clone drops below
	this threshold, the clone turns off the livelist and reverts to the old deletion
	method. This is in place because once a clone has been overwritten enough
	livelists no long give us a benefit.
	.sp
	Default value: \fB75\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_livelist_condense_new_alloc\fR (int)
	.ad
	.RS 12n
	Incremented each time an extra ALLOC blkptr is added to a livelist entry while
	it is being condensed.
	This option is used by the test suite to track race conditions.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_livelist_condense_sync_cancel\fR (int)
	.ad
	.RS 12n
	Incremented each time livelist condensing is canceled while in
	spa_livelist_condense_sync.
	This option is used by the test suite to track race conditions.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_livelist_condense_sync_pause\fR (int)
	.ad
	.RS 12n
	When set, the livelist condense process pauses indefinitely before
	executing the synctask - spa_livelist_condense_sync.
	This option is used by the test suite to trigger race conditions.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_livelist_condense_zthr_cancel\fR (int)
	.ad
	.RS 12n
	Incremented each time livelist condensing is canceled while in
	spa_livelist_condense_cb.
	This option is used by the test suite to track race conditions.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_livelist_condense_zthr_pause\fR (int)
	.ad
	.RS 12n
	When set, the livelist condense process pauses indefinitely before
	executing the open context condensing work in spa_livelist_condense_cb.
	This option is used by the test suite to trigger race conditions.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_lua_max_instrlimit\fR (ulong)
	.ad
	.RS 12n
	The maximum execution time limit that can be set for a ZFS channel program,
	specified as a number of Lua instructions.
	.sp
	Default value: \fB100,000,000\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_lua_max_memlimit\fR (ulong)
	.ad
	.RS 12n
	The maximum memory limit that can be set for a ZFS channel program, specified
	in bytes.
	.sp
	Default value: \fB104,857,600\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_max_dataset_nesting\fR (int)
	.ad
	.RS 12n
	The maximum depth of nested datasets. This value can be tuned temporarily to
	fix existing datasets that exceed the predefined limit.
	.sp
	Default value: \fB50\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_max_log_walking\fR (ulong)
	.ad
	.RS 12n
	The number of past TXGs that the flushing algorithm of the log spacemap
	feature uses to estimate incoming log blocks.
	.sp
	Default value: \fB5\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_max_logsm_summary_length\fR (ulong)
	.ad
	.RS 12n
	Maximum number of rows allowed in the summary of the spacemap log.
	.sp
	Default value: \fB10\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_max_recordsize\fR (int)
	.ad
	.RS 12n
	We currently support block sizes from 512 bytes to 16MB. The benefits of
	larger blocks, and thus larger I/O, need to be weighed against the cost of
	COWing a giant block to modify one byte. Additionally, very large blocks
	can have an impact on i/o latency, and also potentially on the memory
	allocator. Therefore, we do not allow the recordsize to be set larger than
	zfs_max_recordsize (default 1MB). Larger blocks can be created by changing
	this tunable, and pools with larger blocks can always be imported and used,
	regardless of this setting.
	.sp
	Default value: \fB1,048,576\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_allow_redacted_dataset_mount\fR (int)
	.ad
	.RS 12n
	Allow datasets received with redacted send/receive to be mounted. Normally
	disabled because these datasets may be missing key data.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_min_metaslabs_to_flush\fR (ulong)
	.ad
	.RS 12n
	Minimum number of metaslabs to flush per dirty TXG
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_metaslab_fragmentation_threshold\fR (int)
	.ad
	.RS 12n
	Allow metaslabs to keep their active state as long as their fragmentation
	percentage is less than or equal to this value. An active metaslab that
	exceeds this threshold will no longer keep its active status allowing
	better metaslabs to be selected.
	.sp
	Default value: \fB70\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_mg_fragmentation_threshold\fR (int)
	.ad
	.RS 12n
	Metaslab groups are considered eligible for allocations if their
	fragmentation metric (measured as a percentage) is less than or equal to
	this value. If a metaslab group exceeds this threshold then it will be
	skipped unless all metaslab groups within the metaslab class have also
	crossed this threshold.
	.sp
	Default value: \fB95\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_mg_noalloc_threshold\fR (int)
	.ad
	.RS 12n
	Defines a threshold at which metaslab groups should be eligible for
	allocations. The value is expressed as a percentage of free space
	beyond which a metaslab group is always eligible for allocations.
	If a metaslab group's free space is less than or equal to the
	threshold, the allocator will avoid allocating to that group
	unless all groups in the pool have reached the threshold. Once all
	groups have reached the threshold, all groups are allowed to accept
	allocations. The default value of 0 disables the feature and causes
	all metaslab groups to be eligible for allocations.

	This parameter allows one to deal with pools having heavily imbalanced
	vdevs such as would be the case when a new vdev has been added.
	Setting the threshold to a non-zero percentage will stop allocations
	from being made to vdevs that aren't filled to the specified percentage
	and allow lesser filled vdevs to acquire more allocations than they
	otherwise would under the old \fBzfs_mg_alloc_failures\fR facility.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_ddt_data_is_special\fR (int)
	.ad
	.RS 12n
	If enabled, ZFS will place DDT data into the special allocation class.
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_user_indirect_is_special\fR (int)
	.ad
	.RS 12n
	If enabled, ZFS will place user data (both file and zvol) indirect blocks
	into the special allocation class.
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_multihost_history\fR (int)
	.ad
	.RS 12n
	Historical statistics for the last N multihost updates will be available in
	\fB/proc/spl/kstat/zfs/<pool>/multihost\fR
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_multihost_interval\fR (ulong)
	.ad
	.RS 12n
	Used to control the frequency of multihost writes which are performed when the
	\fBmultihost\fR pool property is on. This is one factor used to determine the
	length of the activity check during import.
	.sp
	The multihost write period is \fBzfs_multihost_interval / leaf-vdevs\fR
	milliseconds. On average a multihost write will be issued for each leaf vdev
	every \fBzfs_multihost_interval\fR milliseconds. In practice, the observed
	period can vary with the I/O load and this observed value is the delay which is
	stored in the uberblock.
	.sp
	Default value: \fB1000\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_multihost_import_intervals\fR (uint)
	.ad
	.RS 12n
	Used to control the duration of the activity test on import. Smaller values of
	\fBzfs_multihost_import_intervals\fR will reduce the import time but increase
	the risk of failing to detect an active pool. The total activity check time is
	never allowed to drop below one second.
	.sp
	On import the activity check waits a minimum amount of time determined by
	\fBzfs_multihost_interval * zfs_multihost_import_intervals\fR, or the same
	product computed on the host which last had the pool imported (whichever is
	greater). The activity check time may be further extended if the value of mmp
	delay found in the best uberblock indicates actual multihost updates happened
	at longer intervals than \fBzfs_multihost_interval\fR. A minimum value of
	\fB100ms\fR is enforced.
	.sp
	A value of 0 is ignored and treated as if it was set to 1.
	.sp
	Default value: \fB20\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_multihost_fail_intervals\fR (uint)
	.ad
	.RS 12n
	Controls the behavior of the pool when multihost write failures or delays are
	detected.
	.sp
	When \fBzfs_multihost_fail_intervals = 0\fR, multihost write failures or delays
	are ignored. The failures will still be reported to the ZED which depending on
	its configuration may take action such as suspending the pool or offlining a
	device.

	.sp
	When \fBzfs_multihost_fail_intervals > 0\fR, the pool will be suspended if
	\fBzfs_multihost_fail_intervals * zfs_multihost_interval\fR milliseconds pass
	without a successful mmp write. This guarantees the activity test will see
	mmp writes if the pool is imported. A value of 1 is ignored and treated as
	if it was set to 2. This is necessary to prevent the pool from being suspended
	due to normal, small I/O latency variations.

	.sp
	Default value: \fB10\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_no_scrub_io\fR (int)
	.ad
	.RS 12n
	Set for no scrub I/O. This results in scrubs not actually scrubbing data and
	simply doing a metadata crawl of the pool instead.
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_no_scrub_prefetch\fR (int)
	.ad
	.RS 12n
	Set to disable block prefetching for scrubs.
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_nocacheflush\fR (int)
	.ad
	.RS 12n
	Disable cache flush operations on disks when writing. Setting this will
	cause pool corruption on power loss if a volatile out-of-order write cache
	is enabled.
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_nopwrite_enabled\fR (int)
	.ad
	.RS 12n
	Enable NOP writes
	.sp
	Use \fB1\fR for yes (default) and \fB0\fR to disable.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_dmu_offset_next_sync\fR (int)
	.ad
	.RS 12n
	Enable forcing txg sync to find holes. When enabled forces ZFS to act
	like prior versions when SEEK_HOLE or SEEK_DATA flags are used, which
	when a dnode is dirty causes txg's to be synced so that this data can be
	found.
	.sp
	Use \fB1\fR for yes and \fB0\fR to disable (default).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_pd_bytes_max\fR (int)
	.ad
	.RS 12n
	The number of bytes which should be prefetched during a pool traversal
	(eg: \fBzfs send\fR or other data crawling operations)
	.sp
	Default value: \fB52,428,800\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_per_txg_dirty_frees_percent \fR (ulong)
	.ad
	.RS 12n
	Tunable to control percentage of dirtied indirect blocks from frees allowed
	into one TXG. After this threshold is crossed, additional frees will wait until
	the next TXG.
	A value of zero will disable this throttle.
	.sp
	Default value: \fB5\fR, set to \fB0\fR to disable.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_prefetch_disable\fR (int)
	.ad
	.RS 12n
	This tunable disables predictive prefetch. Note that it leaves "prescient"
	prefetch (e.g. prefetch for zfs send) intact. Unlike predictive prefetch,
	prescient prefetch never issues i/os that end up not being needed, so it
	can't hurt performance.
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_qat_checksum_disable\fR (int)
	.ad
	.RS 12n
	This tunable disables qat hardware acceleration for sha256 checksums. It
	may be set after the zfs modules have been loaded to initialize the qat
	hardware as long as support is compiled in and the qat driver is present.
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_qat_compress_disable\fR (int)
	.ad
	.RS 12n
	This tunable disables qat hardware acceleration for gzip compression. It
	may be set after the zfs modules have been loaded to initialize the qat
	hardware as long as support is compiled in and the qat driver is present.
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_qat_encrypt_disable\fR (int)
	.ad
	.RS 12n
	This tunable disables qat hardware acceleration for AES-GCM encryption. It
	may be set after the zfs modules have been loaded to initialize the qat
	hardware as long as support is compiled in and the qat driver is present.
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_read_chunk_size\fR (long)
	.ad
	.RS 12n
	Bytes to read per chunk
	.sp
	Default value: \fB1,048,576\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_read_history\fR (int)
	.ad
	.RS 12n
	Historical statistics for the last N reads will be available in
	\fB/proc/spl/kstat/zfs/<pool>/reads\fR
	.sp
	Default value: \fB0\fR (no data is kept).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_read_history_hits\fR (int)
	.ad
	.RS 12n
	Include cache hits in read history
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_rebuild_max_segment\fR (ulong)
	.ad
	.RS 12n
	Maximum read segment size to issue when sequentially resilvering a
	top-level vdev.
	.sp
	Default value: \fB1,048,576\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_reconstruct_indirect_combinations_max\fR (int)
	.ad
	.RS 12na
	If an indirect split block contains more than this many possible unique
	combinations when being reconstructed, consider it too computationally
	expensive to check them all. Instead, try at most
	\fBzfs_reconstruct_indirect_combinations_max\fR randomly-selected
	combinations each time the block is accessed. This allows all segment
	copies to participate fairly in the reconstruction when all combinations
	cannot be checked and prevents repeated use of one bad copy.
	.sp
	Default value: \fB4096\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_recover\fR (int)
	.ad
	.RS 12n
	Set to attempt to recover from fatal errors. This should only be used as a
	last resort, as it typically results in leaked space, or worse.
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_removal_ignore_errors\fR (int)
	.ad
	.RS 12n
	.sp
	Ignore hard IO errors during device removal. When set, if a device encounters
	a hard IO error during the removal process the removal will not be cancelled.
	This can result in a normally recoverable block becoming permanently damaged
	and is not recommended. This should only be used as a last resort when the
	pool cannot be returned to a healthy state prior to removing the device.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_removal_suspend_progress\fR (int)
	.ad
	.RS 12n
	.sp
	This is used by the test suite so that it can ensure that certain actions
	happen while in the middle of a removal.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_remove_max_segment\fR (int)
	.ad
	.RS 12n
	.sp
	The largest contiguous segment that we will attempt to allocate when removing
	a device. This can be no larger than 16MB. If there is a performance
	problem with attempting to allocate large blocks, consider decreasing this.
	.sp
	Default value: \fB16,777,216\fR (16MB).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_resilver_disable_defer\fR (int)
	.ad
	.RS 12n
	Disables the \fBresilver_defer\fR feature, causing an operation that would
	start a resilver to restart one in progress immediately.
	.sp
	Default value: \fB0\fR (feature enabled).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_resilver_min_time_ms\fR (int)
	.ad
	.RS 12n
	Resilvers are processed by the sync thread. While resilvering it will spend
	at least this much time working on a resilver between txg flushes.
	.sp
	Default value: \fB3,000\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_scan_ignore_errors\fR (int)
	.ad
	.RS 12n
	If set to a nonzero value, remove the DTL (dirty time list) upon
	completion of a pool scan (scrub) even if there were unrepairable
	errors. It is intended to be used during pool repair or recovery to
	stop resilvering when the pool is next imported.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_scrub_min_time_ms\fR (int)
	.ad
	.RS 12n
	Scrubs are processed by the sync thread. While scrubbing it will spend
	at least this much time working on a scrub between txg flushes.
	.sp
	Default value: \fB1,000\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_scan_checkpoint_intval\fR (int)
	.ad
	.RS 12n
	To preserve progress across reboots the sequential scan algorithm periodically
	needs to stop metadata scanning and issue all the verifications I/Os to disk.
	The frequency of this flushing is determined by the
	\fBzfs_scan_checkpoint_intval\fR tunable.
	.sp
	Default value: \fB7200\fR seconds (every 2 hours).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_scan_fill_weight\fR (int)
	.ad
	.RS 12n
	This tunable affects how scrub and resilver I/O segments are ordered. A higher
	number indicates that we care more about how filled in a segment is, while a
	lower number indicates we care more about the size of the extent without
	considering the gaps within a segment. This value is only tunable upon module
	insertion. Changing the value afterwards will have no affect on scrub or
	resilver performance.
	.sp
	Default value: \fB3\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_scan_issue_strategy\fR (int)
	.ad
	.RS 12n
	Determines the order that data will be verified while scrubbing or resilvering.
	If set to \fB1\fR, data will be verified as sequentially as possible, given the
	amount of memory reserved for scrubbing (see \fBzfs_scan_mem_lim_fact\fR). This
	may improve scrub performance if the pool's data is very fragmented. If set to
	\fB2\fR, the largest mostly-contiguous chunk of found data will be verified
	first. By deferring scrubbing of small segments, we may later find adjacent data
	to coalesce and increase the segment size. If set to \fB0\fR, zfs will use
	strategy \fB1\fR during normal verification and strategy \fB2\fR while taking a
	checkpoint.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_scan_legacy\fR (int)
	.ad
	.RS 12n
	A value of 0 indicates that scrubs and resilvers will gather metadata in
	memory before issuing sequential I/O. A value of 1 indicates that the legacy
	algorithm will be used where I/O is initiated as soon as it is discovered.
	Changing this value to 0 will not affect scrubs or resilvers that are already
	in progress.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_scan_max_ext_gap\fR (int)
	.ad
	.RS 12n
	Indicates the largest gap in bytes between scrub / resilver I/Os that will still
	be considered sequential for sorting purposes. Changing this value will not
	affect scrubs or resilvers that are already in progress.
	.sp
	Default value: \fB2097152 (2 MB)\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_scan_mem_lim_fact\fR (int)
	.ad
	.RS 12n
	Maximum fraction of RAM used for I/O sorting by sequential scan algorithm.
	This tunable determines the hard limit for I/O sorting memory usage.
	When the hard limit is reached we stop scanning metadata and start issuing
	data verification I/O. This is done until we get below the soft limit.
	.sp
	Default value: \fB20\fR which is 5% of RAM (1/20).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_scan_mem_lim_soft_fact\fR (int)
	.ad
	.RS 12n
	The fraction of the hard limit used to determined the soft limit for I/O sorting
	by the sequential scan algorithm. When we cross this limit from below no action
	is taken. When we cross this limit from above it is because we are issuing
	verification I/O. In this case (unless the metadata scan is done) we stop
	issuing verification I/O and start scanning metadata again until we get to the
	hard limit.
	.sp
	Default value: \fB20\fR which is 5% of the hard limit (1/20).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_scan_strict_mem_lim\fR (int)
	.ad
	.RS 12n
	Enforces tight memory limits on pool scans when a sequential scan is in
	progress. When disabled the memory limit may be exceeded by fast disks.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_scan_suspend_progress\fR (int)
	.ad
	.RS 12n
	Freezes a scrub/resilver in progress without actually pausing it. Intended for
	testing/debugging.
	.sp
	Default value: \fB0\fR.
	.RE


	.sp
	.ne 2
	.na
	\fBzfs_scan_vdev_limit\fR (int)
	.ad
	.RS 12n
	Maximum amount of data that can be concurrently issued at once for scrubs and
	resilvers per leaf device, given in bytes.
	.sp
	Default value: \fB41943040\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_send_corrupt_data\fR (int)
	.ad
	.RS 12n
	Allow sending of corrupt data (ignore read/checksum errors when sending data)
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_send_unmodified_spill_blocks\fR (int)
	.ad
	.RS 12n
	Include unmodified spill blocks in the send stream. Under certain circumstances
	previous versions of ZFS could incorrectly remove the spill block from an
	existing object. Including unmodified copies of the spill blocks creates a
	backwards compatible stream which will recreate a spill block if it was
	incorrectly removed.
	.sp
	Use \fB1\fR for yes (default) and \fB0\fR for no.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_send_no_prefetch_queue_ff\fR (int)
	.ad
	.RS 12n
	The fill fraction of the \fBzfs send\fR internal queues. The fill fraction
	controls the timing with which internal threads are woken up.
	.sp
	Default value: \fB20\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_send_no_prefetch_queue_length\fR (int)
	.ad
	.RS 12n
	The maximum number of bytes allowed in \fBzfs send\fR's internal queues.
	.sp
	Default value: \fB1,048,576\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_send_queue_ff\fR (int)
	.ad
	.RS 12n
	The fill fraction of the \fBzfs send\fR prefetch queue. The fill fraction
	controls the timing with which internal threads are woken up.
	.sp
	Default value: \fB20\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_send_queue_length\fR (int)
	.ad
	.RS 12n
	The maximum number of bytes allowed that will be prefetched by \fBzfs send\fR.
	This value must be at least twice the maximum block size in use.
	.sp
	Default value: \fB16,777,216\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_recv_queue_ff\fR (int)
	.ad
	.RS 12n
	The fill fraction of the \fBzfs receive\fR queue. The fill fraction
	controls the timing with which internal threads are woken up.
	.sp
	Default value: \fB20\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_recv_queue_length\fR (int)
	.ad
	.RS 12n
	The maximum number of bytes allowed in the \fBzfs receive\fR queue. This value
	must be at least twice the maximum block size in use.
	.sp
	Default value: \fB16,777,216\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_recv_write_batch_size\fR (int)
	.ad
	.RS 12n
	The maximum amount of data (in bytes) that \fBzfs receive\fR will write in
	one DMU transaction. This is the uncompressed size, even when receiving a
	compressed send stream. This setting will not reduce the write size below
	a single block. Capped at a maximum of 32MB
	.sp
	Default value: \fB1MB\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_override_estimate_recordsize\fR (ulong)
	.ad
	.RS 12n
	Setting this variable overrides the default logic for estimating block
	sizes when doing a zfs send. The default heuristic is that the average
	block size will be the current recordsize. Override this value if most data
	in your dataset is not of that size and you require accurate zfs send size
	estimates.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_sync_pass_deferred_free\fR (int)
	.ad
	.RS 12n
	Flushing of data to disk is done in passes. Defer frees starting in this pass
	.sp
	Default value: \fB2\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_spa_discard_memory_limit\fR (int)
	.ad
	.RS 12n
	Maximum memory used for prefetching a checkpoint's space map on each
	vdev while discarding the checkpoint.
	.sp
	Default value: \fB16,777,216\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_special_class_metadata_reserve_pct\fR (int)
	.ad
	.RS 12n
	Only allow small data blocks to be allocated on the special and dedup vdev
	types when the available free space percentage on these vdevs exceeds this
	value. This ensures reserved space is available for pool meta data as the
	special vdevs approach capacity.
	.sp
	Default value: \fB25\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_sync_pass_dont_compress\fR (int)
	.ad
	.RS 12n
	Starting in this sync pass, we disable compression (including of metadata).
	With the default setting, in practice, we don't have this many sync passes,
	so this has no effect.
	.sp
	The original intent was that disabling compression would help the sync passes
	to converge. However, in practice disabling compression increases the average
	number of sync passes, because when we turn compression off, a lot of block's
	size will change and thus we have to re-allocate (not overwrite) them. It
	also increases the number of 128KB allocations (e.g. for indirect blocks and
	spacemaps) because these will not be compressed. The 128K allocations are
	especially detrimental to performance on highly fragmented systems, which may
	have very few free segments of this size, and may need to load new metaslabs
	to satisfy 128K allocations.
	.sp
	Default value: \fB8\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_sync_pass_rewrite\fR (int)
	.ad
	.RS 12n
	Rewrite new block pointers starting in this pass
	.sp
	Default value: \fB2\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_sync_taskq_batch_pct\fR (int)
	.ad
	.RS 12n
	This controls the number of threads used by the dp_sync_taskq. The default
	value of 75% will create a maximum of one thread per cpu.
	.sp
	Default value: \fB75\fR%.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_trim_extent_bytes_max\fR (uint)
	.ad
	.RS 12n
	Maximum size of TRIM command. Ranges larger than this will be split in to
	chunks no larger than \fBzfs_trim_extent_bytes_max\fR bytes before being
	issued to the device.
	.sp
	Default value: \fB134,217,728\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_trim_extent_bytes_min\fR (uint)
	.ad
	.RS 12n
	Minimum size of TRIM commands. TRIM ranges smaller than this will be skipped
	unless they're part of a larger range which was broken in to chunks. This is
	done because it's common for these small TRIMs to negatively impact overall
	performance. This value can be set to 0 to TRIM all unallocated space.
	.sp
	Default value: \fB32,768\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_trim_metaslab_skip\fR (uint)
	.ad
	.RS 12n
	Skip uninitialized metaslabs during the TRIM process. This option is useful
	for pools constructed from large thinly-provisioned devices where TRIM
	operations are slow. As a pool ages an increasing fraction of the pools
	metaslabs will be initialized progressively degrading the usefulness of
	this option. This setting is stored when starting a manual TRIM and will
	persist for the duration of the requested TRIM.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_trim_queue_limit\fR (uint)
	.ad
	.RS 12n
	Maximum number of queued TRIMs outstanding per leaf vdev. The number of
	concurrent TRIM commands issued to the device is controlled by the
	\fBzfs_vdev_trim_min_active\fR and \fBzfs_vdev_trim_max_active\fR module
	options.
	.sp
	Default value: \fB10\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_trim_txg_batch\fR (uint)
	.ad
	.RS 12n
	The number of transaction groups worth of frees which should be aggregated
	before TRIM operations are issued to the device. This setting represents a
	trade-off between issuing larger, more efficient TRIM operations and the
	delay before the recently trimmed space is available for use by the device.
	.sp
	Increasing this value will allow frees to be aggregated for a longer time.
	This will result is larger TRIM operations and potentially increased memory
	usage. Decreasing this value will have the opposite effect. The default
	value of 32 was determined to be a reasonable compromise.
	.sp
	Default value: \fB32\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_txg_history\fR (int)
	.ad
	.RS 12n
	Historical statistics for the last N txgs will be available in
	\fB/proc/spl/kstat/zfs/<pool>/txgs\fR
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_txg_timeout\fR (int)
	.ad
	.RS 12n
	Flush dirty data to disk at least every N seconds (maximum txg duration)
	.sp
	Default value: \fB5\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_aggregate_trim\fR (int)
	.ad
	.RS 12n
	Allow TRIM I/Os to be aggregated. This is normally not helpful because
	the extents to be trimmed will have been already been aggregated by the
	metaslab. This option is provided for debugging and performance analysis.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_aggregation_limit\fR (int)
	.ad
	.RS 12n
	Max vdev I/O aggregation size
	.sp
	Default value: \fB1,048,576\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_aggregation_limit_non_rotating\fR (int)
	.ad
	.RS 12n
	Max vdev I/O aggregation size for non-rotating media
	.sp
	Default value: \fB131,072\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_cache_bshift\fR (int)
	.ad
	.RS 12n
	Shift size to inflate reads too
	.sp
	Default value: \fB16\fR (effectively 65536).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_cache_max\fR (int)
	.ad
	.RS 12n
	Inflate reads smaller than this value to meet the \fBzfs_vdev_cache_bshift\fR
	size (default 64k).
	.sp
	Default value: \fB16384\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_cache_size\fR (int)
	.ad
	.RS 12n
	Total size of the per-disk cache in bytes.
	.sp
	Currently this feature is disabled as it has been found to not be helpful
	for performance and in some cases harmful.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_mirror_rotating_inc\fR (int)
	.ad
	.RS 12n
	A number by which the balancing algorithm increments the load calculation for
	the purpose of selecting the least busy mirror member when an I/O immediately
	follows its predecessor on rotational vdevs for the purpose of making decisions
	based on load.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_mirror_rotating_seek_inc\fR (int)
	.ad
	.RS 12n
	A number by which the balancing algorithm increments the load calculation for
	the purpose of selecting the least busy mirror member when an I/O lacks
	locality as defined by the zfs_vdev_mirror_rotating_seek_offset. I/Os within
	this that are not immediately following the previous I/O are incremented by
	half.
	.sp
	Default value: \fB5\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_mirror_rotating_seek_offset\fR (int)
	.ad
	.RS 12n
	The maximum distance for the last queued I/O in which the balancing algorithm
	considers an I/O to have locality.
	See the section "ZFS I/O SCHEDULER".
	.sp
	Default value: \fB1048576\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_mirror_non_rotating_inc\fR (int)
	.ad
	.RS 12n
	A number by which the balancing algorithm increments the load calculation for
	the purpose of selecting the least busy mirror member on non-rotational vdevs
	when I/Os do not immediately follow one another.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_mirror_non_rotating_seek_inc\fR (int)
	.ad
	.RS 12n
	A number by which the balancing algorithm increments the load calculation for
	the purpose of selecting the least busy mirror member when an I/O lacks
	locality as defined by the zfs_vdev_mirror_rotating_seek_offset. I/Os within
	this that are not immediately following the previous I/O are incremented by
	half.
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_read_gap_limit\fR (int)
	.ad
	.RS 12n
	Aggregate read I/O operations if the gap on-disk between them is within this
	threshold.
	.sp
	Default value: \fB32,768\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_write_gap_limit\fR (int)
	.ad
	.RS 12n
	Aggregate write I/O over gap
	.sp
	Default value: \fB4,096\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_raidz_impl\fR (string)
	.ad
	.RS 12n
	Parameter for selecting raidz parity implementation to use.

	Options marked (always) below may be selected on module load as they are
	supported on all systems.
	The remaining options may only be set after the module is loaded, as they
	are available only if the implementations are compiled in and supported
	on the running system.

	Once the module is loaded, the content of
	/sys/module/zfs/parameters/zfs_vdev_raidz_impl will show available options
	with the currently selected one enclosed in [].
	Possible options are:
	fastest - (always) implementation selected using built-in benchmark
	original - (always) original raidz implementation
	scalar - (always) scalar raidz implementation
	sse2 - implementation using SSE2 instruction set (64bit x86 only)
	ssse3 - implementation using SSSE3 instruction set (64bit x86 only)
	avx2 - implementation using AVX2 instruction set (64bit x86 only)
	avx512f - implementation using AVX512F instruction set (64bit x86 only)
	avx512bw - implementation using AVX512F & AVX512BW instruction sets (64bit x86 only)
	aarch64_neon - implementation using NEON (Aarch64/64 bit ARMv8 only)
	aarch64_neonx2 - implementation using NEON with more unrolling (Aarch64/64 bit ARMv8 only)
	powerpc_altivec - implementation using Altivec (PowerPC only)
	.sp
	Default value: \fBfastest\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_vdev_scheduler\fR (charp)
	.ad
	.RS 12n
	\fBDEPRECATED\fR: This option exists for compatibility with older user
	configurations. It does nothing except print a warning to the kernel log if
	set.
	.sp
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_zevent_cols\fR (int)
	.ad
	.RS 12n
	When zevents are logged to the console use this as the word wrap width.
	.sp
	Default value: \fB80\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_zevent_console\fR (int)
	.ad
	.RS 12n
	Log events to the console
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_zevent_len_max\fR (int)
	.ad
	.RS 12n
	Max event queue length. A value of 0 will result in a calculated value which
	increases with the number of CPUs in the system (minimum 64 events). Events
	in the queue can be viewed with the \fBzpool events\fR command.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_zevent_retain_max\fR (int)
	.ad
	.RS 12n
	Maximum recent zevent records to retain for duplicate checking. Setting
	this value to zero disables duplicate detection.
	.sp
	Default value: \fB2000\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_zevent_retain_expire_secs\fR (int)
	.ad
	.RS 12n
	Lifespan for a recent ereport that was retained for duplicate checking.
	.sp
	Default value: \fB900\fR.
	.RE

	.na
	\fBzfs_zil_clean_taskq_maxalloc\fR (int)
	.ad
	.RS 12n
	The maximum number of taskq entries that are allowed to be cached. When this
	limit is exceeded transaction records (itxs) will be cleaned synchronously.
	.sp
	Default value: \fB1048576\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_zil_clean_taskq_minalloc\fR (int)
	.ad
	.RS 12n
	The number of taskq entries that are pre-populated when the taskq is first
	created and are immediately available for use.
	.sp
	Default value: \fB1024\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzfs_zil_clean_taskq_nthr_pct\fR (int)
	.ad
	.RS 12n
	This controls the number of threads used by the dp_zil_clean_taskq. The default
	value of 100% will create a maximum of one thread per cpu.
	.sp
	Default value: \fB100\fR%.
	.RE

	.sp
	.ne 2
	.na
	\fBzil_maxblocksize\fR (int)
	.ad
	.RS 12n
	This sets the maximum block size used by the ZIL. On very fragmented pools,
	lowering this (typically to 36KB) can improve performance.
	.sp
	Default value: \fB131072\fR (128KB).
	.RE

	.sp
	.ne 2
	.na
	\fBzil_nocacheflush\fR (int)
	.ad
	.RS 12n
	Disable the cache flush commands that are normally sent to the disk(s) by
	the ZIL after an LWB write has completed. Setting this will cause ZIL
	corruption on power loss if a volatile out-of-order write cache is enabled.
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBzil_replay_disable\fR (int)
	.ad
	.RS 12n
	Disable intent logging replay. Can be disabled for recovery from corrupted
	ZIL
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBzil_slog_bulk\fR (ulong)
	.ad
	.RS 12n
	Limit SLOG write size per commit executed with synchronous priority.
	Any writes above that will be executed with lower (asynchronous) priority
	to limit potential SLOG device abuse by single active ZIL writer.
	.sp
	Default value: \fB786,432\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzio_deadman_log_all\fR (int)
	.ad
	.RS 12n
	If non-zero, the zio deadman will produce debugging messages (see
	\fBzfs_dbgmsg_enable\fR) for all zios, rather than only for leaf
	zios possessing a vdev. This is meant to be used by developers to gain
	diagnostic information for hang conditions which don't involve a mutex
	or other locking primitive; typically conditions in which a thread in
	the zio pipeline is looping indefinitely.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzio_decompress_fail_fraction\fR (int)
	.ad
	.RS 12n
	If non-zero, this value represents the denominator of the probability that zfs
	should induce a decompression failure. For instance, for a 5% decompression
	failure rate, this value should be set to 20.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzio_slow_io_ms\fR (int)
	.ad
	.RS 12n
	When an I/O operation takes more than \fBzio_slow_io_ms\fR milliseconds to
	complete is marked as a slow I/O. Each slow I/O causes a delay zevent. Slow
	I/O counters can be seen with "zpool status -s".

	.sp
	Default value: \fB30,000\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzio_dva_throttle_enabled\fR (int)
	.ad
	.RS 12n
	Throttle block allocations in the I/O pipeline. This allows for
	dynamic allocation distribution when devices are imbalanced.
	When enabled, the maximum number of pending allocations per top-level vdev
	is limited by \fBzfs_vdev_queue_depth_pct\fR.
	.sp
	Default value: \fB1\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzio_requeue_io_start_cut_in_line\fR (int)
	.ad
	.RS 12n
	Prioritize requeued I/O
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzio_taskq_batch_pct\fR (uint)
	.ad
	.RS 12n
	Percentage of online CPUs (or CPU cores, etc) which will run a worker thread
	for I/O. These workers are responsible for I/O work such as compression and
	checksum calculations. Fractional number of CPUs will be rounded down.
	.sp
	The default value of 75 was chosen to avoid using all CPUs which can result in
	latency issues and inconsistent application performance, especially when high
	compression is enabled.
	.sp
	Default value: \fB75\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzvol_inhibit_dev\fR (uint)
	.ad
	.RS 12n
	Do not create zvol device nodes. This may slightly improve startup time on
	systems with a very large number of zvols.
	.sp
	Use \fB1\fR for yes and \fB0\fR for no (default).
	.RE

	.sp
	.ne 2
	.na
	\fBzvol_major\fR (uint)
	.ad
	.RS 12n
	Major number for zvol block devices
	.sp
	Default value: \fB230\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzvol_max_discard_blocks\fR (ulong)
	.ad
	.RS 12n
	Discard (aka TRIM) operations done on zvols will be done in batches of this
	many blocks, where block size is determined by the \fBvolblocksize\fR property
	of a zvol.
	.sp
	Default value: \fB16,384\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzvol_prefetch_bytes\fR (uint)
	.ad
	.RS 12n
	When adding a zvol to the system prefetch \fBzvol_prefetch_bytes\fR
	from the start and end of the volume. Prefetching these regions
	of the volume is desirable because they are likely to be accessed
	immediately by \fBblkid(8)\fR or by the kernel scanning for a partition
	table.
	.sp
	Default value: \fB131,072\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzvol_request_sync\fR (uint)
	.ad
	.RS 12n
	When processing I/O requests for a zvol submit them synchronously. This
	effectively limits the queue depth to 1 for each I/O submitter. When set
	to 0 requests are handled asynchronously by a thread pool. The number of
	requests which can be handled concurrently is controller by \fBzvol_threads\fR.
	.sp
	Default value: \fB0\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzvol_threads\fR (uint)
	.ad
	.RS 12n
	Max number of threads which can handle zvol I/O requests concurrently.
	.sp
	Default value: \fB32\fR.
	.RE

	.sp
	.ne 2
	.na
	\fBzvol_volmode\fR (uint)
	.ad
	.RS 12n
	Defines zvol block devices behaviour when \fBvolmode\fR is set to \fBdefault\fR.
	Valid values are \fB1\fR (full), \fB2\fR (dev) and \fB3\fR (none).
	.sp
	Default value: \fB1\fR.
	.RE

	.SH ZFS I/O SCHEDULER
	ZFS issues I/O operations to leaf vdevs to satisfy and complete I/Os.
	The I/O scheduler determines when and in what order those operations are
	issued. The I/O scheduler divides operations into five I/O classes
	prioritized in the following order: sync read, sync write, async read,
	async write, and scrub/resilver. Each queue defines the minimum and
	maximum number of concurrent operations that may be issued to the
	device. In addition, the device has an aggregate maximum,
	\fBzfs_vdev_max_active\fR. Note that the sum of the per-queue minimums
	must not exceed the aggregate maximum. If the sum of the per-queue
	maximums exceeds the aggregate maximum, then the number of active I/Os
	may reach \fBzfs_vdev_max_active\fR, in which case no further I/Os will
	be issued regardless of whether all per-queue minimums have been met.
	.sp
	For many physical devices, throughput increases with the number of
	concurrent operations, but latency typically suffers. Further, physical
	devices typically have a limit at which more concurrent operations have no
	effect on throughput or can actually cause it to decrease.
	.sp
	The scheduler selects the next operation to issue by first looking for an
	I/O class whose minimum has not been satisfied. Once all are satisfied and
	the aggregate maximum has not been hit, the scheduler looks for classes
	whose maximum has not been satisfied. Iteration through the I/O classes is
	done in the order specified above. No further operations are issued if the
	aggregate maximum number of concurrent operations has been hit or if there
	are no operations queued for an I/O class that has not hit its maximum.
	Every time an I/O is queued or an operation completes, the I/O scheduler
	looks for new operations to issue.
	.sp
	In general, smaller max_active's will lead to lower latency of synchronous
	operations. Larger max_active's may lead to higher overall throughput,
	depending on underlying storage.
	.sp
	The ratio of the queues' max_actives determines the balance of performance
	between reads, writes, and scrubs. E.g., increasing
	\fBzfs_vdev_scrub_max_active\fR will cause the scrub or resilver to complete
	more quickly, but reads and writes to have higher latency and lower throughput.
	.sp
	All I/O classes have a fixed maximum number of outstanding operations
	except for the async write class. Asynchronous writes represent the data
	that is committed to stable storage during the syncing stage for
	transaction groups. Transaction groups enter the syncing state
	periodically so the number of queued async writes will quickly burst up
	and then bleed down to zero. Rather than servicing them as quickly as
	possible, the I/O scheduler changes the maximum number of active async
	write I/Os according to the amount of dirty data in the pool. Since
	both throughput and latency typically increase with the number of
	concurrent operations issued to physical devices, reducing the
	burstiness in the number of concurrent operations also stabilizes the
	response time of operations from other -- and in particular synchronous
	-- queues. In broad strokes, the I/O scheduler will issue more
	concurrent operations from the async write queue as there's more dirty
	data in the pool.
	.sp
	Async Writes
	.sp
	The number of concurrent operations issued for the async write I/O class
	follows a piece-wise linear function defined by a few adjustable points.
	.nf

	\| o---------\| <-- zfs_vdev_async_write_max_active
	^ \| /^ \|
	\| \| / \| \|
	active \| / \| \|
	I/O \| / \| \|
	count \| / \| \|
	\| / \| \|
	\|-------o \| \| <-- zfs_vdev_async_write_min_active
	0\|_______^______\|_________\|
	0% \| \| 100% of zfs_dirty_data_max
	\| \|
	\| `-- zfs_vdev_async_write_active_max_dirty_percent
	`--------- zfs_vdev_async_write_active_min_dirty_percent

	.fi
	Until the amount of dirty data exceeds a minimum percentage of the dirty
	data allowed in the pool, the I/O scheduler will limit the number of
	concurrent operations to the minimum. As that threshold is crossed, the
	number of concurrent operations issued increases linearly to the maximum at
	the specified maximum percentage of the dirty data allowed in the pool.
	.sp
	Ideally, the amount of dirty data on a busy pool will stay in the sloped
	part of the function between \fBzfs_vdev_async_write_active_min_dirty_percent\fR
	and \fBzfs_vdev_async_write_active_max_dirty_percent\fR. If it exceeds the
	maximum percentage, this indicates that the rate of incoming data is
	greater than the rate that the backend storage can handle. In this case, we
	must further throttle incoming writes, as described in the next section.

	.SH ZFS TRANSACTION DELAY
	We delay transactions when we've determined that the backend storage
	isn't able to accommodate the rate of incoming writes.
	.sp
	If there is already a transaction waiting, we delay relative to when
	that transaction will finish waiting. This way the calculated delay time
	is independent of the number of threads concurrently executing
	transactions.
	.sp
	If we are the only waiter, wait relative to when the transaction
	started, rather than the current time. This credits the transaction for
	"time already served", e.g. reading indirect blocks.
	.sp
	The minimum time for a transaction to take is calculated as:
	.nf
	min_time = zfs_delay_scale * (dirty - min) / (max - dirty)
	min_time is then capped at 100 milliseconds.
	.fi
	.sp
	The delay has two degrees of freedom that can be adjusted via tunables. The
	percentage of dirty data at which we start to delay is defined by
	\fBzfs_delay_min_dirty_percent\fR. This should typically be at or above
	\fBzfs_vdev_async_write_active_max_dirty_percent\fR so that we only start to
	delay after writing at full speed has failed to keep up with the incoming write
	rate. The scale of the curve is defined by \fBzfs_delay_scale\fR. Roughly speaking,
	this variable determines the amount of delay at the midpoint of the curve.
	.sp
	.nf
	delay
	10ms +-------------------------------------------------------------*+
	\| *\|
	9ms + *+
	\| *\|
	8ms + *+
	\| * \|
	7ms + * +
	\| * \|
	6ms + * +
	\| * \|
	5ms + * +
	\| * \|
	4ms + * +
	\| * \|
	3ms + * +
	\| * \|
	2ms + (midpoint) * +
	\| \| ** \|
	1ms + v *** +
	\| zfs_delay_scale ----------> ******** \|
	0 +-------------------------------------*********----------------+
	0% <- zfs_dirty_data_max -> 100%
	.fi
	.sp
	Note that since the delay is added to the outstanding time remaining on the
	most recent transaction, the delay is effectively the inverse of IOPS.
	Here the midpoint of 500us translates to 2000 IOPS. The shape of the curve
	was chosen such that small changes in the amount of accumulated dirty data
	in the first 3/4 of the curve yield relatively small differences in the
	amount of delay.
	.sp
	The effects can be easier to understand when the amount of delay is
	represented on a log scale:
	.sp
	.nf
	delay
	100ms +-------------------------------------------------------------++
	+ +
	\| \|
	+ *+
	10ms + *+
	+ ** +
	\| (midpoint) ** \|
	+ \| ** +
	1ms + v **** +
	+ zfs_delay_scale ----------> ***** +
	\| **** \|
	+ **** +
	100us + ** +
	+ * +
	\| * \|
	+ * +
	10us + * +
	+ +
	\| \|
	+ +
	+--------------------------------------------------------------+
	0% <- zfs_dirty_data_max -> 100%
	.fi
	.sp
	Note here that only as the amount of dirty data approaches its limit does
	the delay start to increase rapidly. The goal of a properly tuned system
	should be to keep the amount of dirty data out of that range by first
	ensuring that the appropriate limits are set for the I/O scheduler to reach
	optimal throughput on the backend storage, and then by changing the value
	of \fBzfs_delay_scale\fR to increase the steepness of the curve.
	Index: vendor-sys/openzfs/dist/man/man8/zed.8.in
	===================================================================
	--- vendor-sys/openzfs/dist/man/man8/zed.8.in (revision 366773)
	+++ vendor-sys/openzfs/dist/man/man8/zed.8.in (revision 366774)
	@@ -1,268 +1,267 @@
	.\"
	-.\" This file is part of the ZFS Event Daemon (ZED)
	-.\" for ZFS on Linux (ZoL) <https://zfsonlinux.org/>.
	+.\" This file is part of the ZFS Event Daemon (ZED).
	.\" Developed at Lawrence Livermore National Laboratory (LLNL-CODE-403049).
	.\" Copyright (C) 2013-2014 Lawrence Livermore National Security, LLC.
	.\" Refer to the ZoL git commit log for authoritative copyright attribution.
	.\"
	.\" The contents of this file are subject to the terms of the
	.\" Common Development and Distribution License Version 1.0 (CDDL-1.0).
	.\" You can obtain a copy of the license from the top-level file
	.\" "OPENSOLARIS.LICENSE" or at <http://opensource.org/licenses/CDDL-1.0>.
	.\" You may not use this file except in compliance with the license.
	.\"
	.TH ZED 8 "Aug 24, 2020" OpenZFS

	.SH NAME
	ZED \- ZFS Event Daemon

	.SH SYNOPSIS
	.HP
	.B zed
	.\" [\fB\-c\fR \fIconfigfile\fR]
	[\fB\-d\fR \fIzedletdir\fR]
	[\fB\-f\fR]
	[\fB\-F\fR]
	[\fB\-h\fR]
	[\fB\-I\fR]
	[\fB\-L\fR]
	[\fB\-M\fR]
	[\fB\-p\fR \fIpidfile\fR]
	[\fB\-P\fR \fIpath\fR]
	[\fB\-s\fR \fIstatefile\fR]
	[\fB\-v\fR]
	[\fB\-V\fR]
	[\fB\-Z\fR]

	.SH DESCRIPTION
	.PP
	\fBZED\fR (ZFS Event Daemon) monitors events generated by the ZFS kernel
	module. When a zevent (ZFS Event) is posted, \fBZED\fR will run any ZEDLETs
	(ZFS Event Daemon Linkage for Executable Tasks) that have been enabled for the
	corresponding zevent class.

	.SH OPTIONS
	.TP
	.BI \-h
	Display a summary of the command-line options.
	.TP
	.BI \-L
	Display license information.
	.TP
	.BI \-V
	Display version information.
	.TP
	.BI \-v
	Be verbose.
	.TP
	.BI \-f
	Force the daemon to run if at all possible, disabling security checks and
	throwing caution to the wind. Not recommended for use in production.
	.TP
	.BI \-F
	Run the daemon in the foreground.
	.TP
	.BI \-M
	Lock all current and future pages in the virtual memory address space.
	This may help the daemon remain responsive when the system is under heavy
	memory pressure.
	.TP
	.BI \-I
	Request that the daemon idle rather than exit when the kernel modules are
	not loaded. Processing of events will start, or resume, when the kernel
	modules are (re)loaded. Under Linux the kernel modules cannot be unloaded
	while the daemon is running.
	.TP
	.BI \-Z
	Zero the daemon's state, thereby allowing zevents still within the kernel
	to be reprocessed.
	.\" .TP
	.\" .BI \-c\ configfile
	.\" Read the configuration from the specified file.
	.TP
	.BI \-d\ zedletdir
	Read the enabled ZEDLETs from the specified directory.
	.TP
	.BI \-p\ pidfile
	Write the daemon's process ID to the specified file.
	.TP
	.BI \-P\ path
	Custom $PATH for zedlets to use. Normally zedlets run in a locked-down
	environment, with hardcoded paths to the ZFS commands ($ZFS, $ZPOOL, $ZED, ...),
	and a hardcoded $PATH. This is done for security reasons. However, the
	ZFS test suite uses a custom PATH for its ZFS commands, and passes it to zed
	with -P. In short, -P is only to be used by the ZFS test suite; never use
	it in production!
	.TP
	.BI \-s\ statefile
	Write the daemon's state to the specified file.
	.SH ZEVENTS
	.PP
	A zevent is comprised of a list of nvpairs (name/value pairs). Each zevent
	contains an EID (Event IDentifier) that uniquely identifies it throughout
	the lifetime of the loaded ZFS kernel module; this EID is a monotonically
	increasing integer that resets to 1 each time the kernel module is loaded.
	Each zevent also contains a class string that identifies the type of event.
	For brevity, a subclass string is defined that omits the leading components
	of the class string. Additional nvpairs exist to provide event details.
	.PP
	The kernel maintains a list of recent zevents that can be viewed (along with
	their associated lists of nvpairs) using the "\fBzpool events \-v\fR" command.

	.SH CONFIGURATION
	.PP
	ZEDLETs to be invoked in response to zevents are located in the
	\fIenabled-zedlets\fR directory. These can be symlinked or copied from the
	\fIinstalled-zedlets\fR directory; symlinks allow for automatic updates
	from the installed ZEDLETs, whereas copies preserve local modifications.
	As a security measure, ZEDLETs must be owned by root. They must have
	execute permissions for the user, but they must not have write permissions
	for group or other. Dotfiles are ignored.
	.PP
	ZEDLETs are named after the zevent class for which they should be invoked.
	In particular, a ZEDLET will be invoked for a given zevent if either its
	class or subclass string is a prefix of its filename (and is followed by
	a non-alphabetic character). As a special case, the prefix "all" matches
	all zevents. Multiple ZEDLETs may be invoked for a given zevent.

	.SH ZEDLETS
	.PP
	ZEDLETs are executables invoked by the ZED in response to a given zevent.
	They should be written under the presumption they can be invoked concurrently,
	and they should use appropriate locking to access any shared resources.
	Common variables used by ZEDLETs can be stored in the default rc file which
	is sourced by scripts; these variables should be prefixed with "ZED_".
	.PP
	The zevent nvpairs are passed to ZEDLETs as environment variables.
	Each nvpair name is converted to an environment variable in the following
	manner: 1) it is prefixed with "ZEVENT_", 2) it is converted to uppercase,
	and 3) each non-alphanumeric character is converted to an underscore.
	Some additional environment variables have been defined to present certain
	nvpair values in a more convenient form. An incomplete list of zevent
	environment variables is as follows:
	.TP
	.B
	ZEVENT_EID
	The Event IDentifier.
	.TP
	.B
	ZEVENT_CLASS
	The zevent class string.
	.TP
	.B
	ZEVENT_SUBCLASS
	The zevent subclass string.
	.TP
	.B
	ZEVENT_TIME
	The time at which the zevent was posted as
	"\fIseconds\fR\ \fInanoseconds\fR" since the Epoch.
	.TP
	.B
	ZEVENT_TIME_SECS
	The \fIseconds\fR component of ZEVENT_TIME.
	.TP
	.B
	ZEVENT_TIME_NSECS
	The \fInanoseconds\fR component of ZEVENT_TIME.
	.TP
	.B
	ZEVENT_TIME_STRING
	An almost-RFC3339-compliant string for ZEVENT_TIME.
	.PP
	Additionally, the following ZED & ZFS variables are defined:
	.TP
	.B
	ZED_PID
	The daemon's process ID.
	.TP
	.B
	ZED_ZEDLET_DIR
	The daemon's current \fIenabled-zedlets\fR directory.
	.TP
	.B
	ZFS_ALIAS
	The ZFS alias (\fIname-version-release\fR) string used to build the daemon.
	.TP
	.B
	ZFS_VERSION
	The ZFS version used to build the daemon.
	.TP
	.B
	ZFS_RELEASE
	The ZFS release used to build the daemon.
	.PP
	ZEDLETs may need to call other ZFS commands. The installation paths of
	the following executables are defined: \fBZDB\fR, \fBZED\fR, \fBZFS\fR,
	\fBZINJECT\fR, and \fBZPOOL\fR. These variables can be overridden in the
	rc file if needed.

	.SH FILES
	.\" .TP
	.\" @sysconfdir@/zfs/zed.conf
	.\" The default configuration file for the daemon.
	.TP
	.I @sysconfdir@/zfs/zed.d
	The default directory for enabled ZEDLETs.
	.TP
	.I @sysconfdir@/zfs/zed.d/zed.rc
	The default rc file for common variables used by ZEDLETs.
	.TP
	.I @zfsexecdir@/zed.d
	The default directory for installed ZEDLETs.
	.TP
	.I @runstatedir@/zed.pid
	The default file containing the daemon's process ID.
	.TP
	.I @runstatedir@/zed.state
	The default file containing the daemon's state.

	.SH SIGNALS
	.TP
	.B HUP
	Reconfigure the daemon and rescan the directory for enabled ZEDLETs.
	.TP
	.B TERM
	Terminate the daemon.

	.SH NOTES
	.PP
	\fBZED\fR requires root privileges.
	.\" Do not taunt zed.

	.SH BUGS
	.PP
	Events are processed synchronously by a single thread. This can delay the
	processing of simultaneous zevents.
	.PP
	There is no maximum timeout for ZEDLET execution. Consequently, a misbehaving
	ZEDLET can delay the processing of subsequent zevents.
	.PP
	The ownership and permissions of the \fIenabled-zedlets\fR directory (along
	with all parent directories) are not checked. If any of these directories
	are improperly owned or permissioned, an unprivileged user could insert a
	ZEDLET to be executed as root. The requirement that ZEDLETs be owned by
	root mitigates this to some extent.
	.PP
	ZEDLETs are unable to return state/status information to the kernel.
	.PP
	Some zevent nvpair types are not handled. These are denoted by zevent
	environment variables having a "_NOT_IMPLEMENTED_" value.
	.PP
	Internationalization support via gettext has not been added.
	.PP
	The configuration file is not yet implemented.
	.PP
	The diagnosis engine is not yet implemented.

	.SH LICENSE
	.PP
	\fBZED\fR (ZFS Event Daemon) is distributed under the terms of the
	Common Development and Distribution License Version 1.0 (CDDL\-1.0).
	.PP
	Developed at Lawrence Livermore National Laboratory (LLNL\-CODE\-403049).

	.SH SEE ALSO
	.BR zfs (8),
	.BR zpool (8)
	.BR zpool-events (8)
	Index: vendor-sys/openzfs/dist/man/man8/zfsprops.8
	===================================================================
	--- vendor-sys/openzfs/dist/man/man8/zfsprops.8 (revision 366773)
	+++ vendor-sys/openzfs/dist/man/man8/zfsprops.8 (revision 366774)
	@@ -1,1992 +1,2003 @@
	.\"
	.\" CDDL HEADER START
	.\"
	.\" The contents of this file are subject to the terms of the
	.\" Common Development and Distribution License (the "License").
	.\" You may not use this file except in compliance with the License.
	.\"
	.\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	.\" or http://www.opensolaris.org/os/licensing.
	.\" See the License for the specific language governing permissions
	.\" and limitations under the License.
	.\"
	.\" When distributing Covered Code, include this CDDL HEADER in each
	.\" file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	.\" If applicable, add the following below this CDDL HEADER, with the
	.\" fields enclosed by brackets "[]" replaced with your own identifying
	.\" information: Portions Copyright [yyyy] [name of copyright owner]
	.\"
	.\" CDDL HEADER END
	.\"
	.\"
	.\" Copyright (c) 2009 Sun Microsystems, Inc. All Rights Reserved.
	.\" Copyright 2011 Joshua M. Clulow <josh@sysmgr.org>
	.\" Copyright (c) 2011, 2019 by Delphix. All rights reserved.
	.\" Copyright (c) 2011, Pawel Jakub Dawidek <pjd@FreeBSD.org>
	.\" Copyright (c) 2012, Glen Barber <gjb@FreeBSD.org>
	.\" Copyright (c) 2012, Bryan Drewery <bdrewery@FreeBSD.org>
	.\" Copyright (c) 2013, Steven Hartland <smh@FreeBSD.org>
	.\" Copyright (c) 2013 by Saso Kiselkov. All rights reserved.
	.\" Copyright (c) 2014, Joyent, Inc. All rights reserved.
	.\" Copyright (c) 2014 by Adam Stevko. All rights reserved.
	.\" Copyright (c) 2014 Integros [integros.com]
	.\" Copyright (c) 2016 Nexenta Systems, Inc. All Rights Reserved.
	.\" Copyright (c) 2014, Xin LI <delphij@FreeBSD.org>
	.\" Copyright (c) 2014-2015, The FreeBSD Foundation, All Rights Reserved.
	.\" Copyright 2019 Richard Laager. All rights reserved.
	.\" Copyright 2018 Nexenta Systems, Inc.
	.\" Copyright 2019 Joyent, Inc.
	.\" Copyright (c) 2019, Kjeld Schouten-Lebbing
	.\"
	.Dd September 1, 2020
	.Dt ZFSPROPS 8
	.Os
	.Sh NAME
	.Nm zfsprops
	.Nd Native properties and user-defined of ZFS datasets.
	.Sh DESCRIPTION
	Properties are divided into two types, native properties and user-defined
	.Po or
	.Qq user
	.Pc
	properties.
	Native properties either export internal statistics or control ZFS behavior.
	In addition, native properties are either editable or read-only.
	User properties have no effect on ZFS behavior, but you can use them to annotate
	datasets in a way that is meaningful in your environment.
	For more information about user properties, see the
	.Sx User Properties
	section, below.
	.Ss Native Properties
	Every dataset has a set of properties that export statistics about the dataset
	as well as control various behaviors.
	Properties are inherited from the parent unless overridden by the child.
	Some properties apply only to certain types of datasets
	.Pq file systems, volumes, or snapshots .
	.Pp
	The values of numeric properties can be specified using human-readable suffixes
	.Po for example,
	.Sy k ,
	.Sy KB ,
	.Sy M ,
	.Sy Gb ,
	and so forth, up to
	.Sy Z
	for zettabyte
	.Pc .
	The following are all valid
	.Pq and equal
	specifications:
	.Li 1536M, 1.5g, 1.50GB .
	.Pp
	The values of non-numeric properties are case sensitive and must be lowercase,
	except for
	.Sy mountpoint ,
	.Sy sharenfs ,
	and
	.Sy sharesmb .
	.Pp
	The following native properties consist of read-only statistics about the
	dataset.
	These properties can be neither set, nor inherited.
	Native properties apply to all dataset types unless otherwise noted.
	.Bl -tag -width "usedbyrefreservation"
	.It Sy available
	The amount of space available to the dataset and all its children, assuming that
	there is no other activity in the pool.
	Because space is shared within a pool, availability can be limited by any number
	of factors, including physical pool size, quotas, reservations, or other
	datasets within the pool.
	.Pp
	This property can also be referred to by its shortened column name,
	.Sy avail .
	.It Sy compressratio
	For non-snapshots, the compression ratio achieved for the
	.Sy used
	space of this dataset, expressed as a multiplier.
	The
	.Sy used
	property includes descendant datasets, and, for clones, does not include the
	space shared with the origin snapshot.
	For snapshots, the
	.Sy compressratio
	is the same as the
	.Sy refcompressratio
	property.
	Compression can be turned on by running:
	.Nm zfs Cm set Sy compression Ns = Ns Sy on Ar dataset .
	The default value is
	.Sy off .
	.It Sy createtxg
	The transaction group (txg) in which the dataset was created. Bookmarks have
	the same
	.Sy createtxg
	as the snapshot they are initially tied to. This property is suitable for
	ordering a list of snapshots, e.g. for incremental send and receive.
	.It Sy creation
	The time this dataset was created.
	.It Sy clones
	For snapshots, this property is a comma-separated list of filesystems or volumes
	which are clones of this snapshot.
	The clones'
	.Sy origin
	property is this snapshot.
	If the
	.Sy clones
	property is not empty, then this snapshot can not be destroyed
	.Po even with the
	.Fl r
	or
	.Fl f
	options
	.Pc .
	The roles of origin and clone can be swapped by promoting the clone with the
	.Nm zfs Cm promote
	command.
	.It Sy defer_destroy
	This property is
	.Sy on
	if the snapshot has been marked for deferred destroy by using the
	.Nm zfs Cm destroy Fl d
	command.
	Otherwise, the property is
	.Sy off .
	.It Sy encryptionroot
	For encrypted datasets, indicates where the dataset is currently inheriting its
	encryption key from. Loading or unloading a key for the
	.Sy encryptionroot
	will implicitly load / unload the key for any inheriting datasets (see
	.Nm zfs Cm load-key
	and
	.Nm zfs Cm unload-key
	for details).
	Clones will always share an
	encryption key with their origin. See the
	.Em Encryption
	section of
	.Xr zfs-load-key 8
	for details.
	.It Sy filesystem_count
	The total number of filesystems and volumes that exist under this location in
	the dataset tree.
	This value is only available when a
	.Sy filesystem_limit
	has been set somewhere in the tree under which the dataset resides.
	.It Sy keystatus
	Indicates if an encryption key is currently loaded into ZFS. The possible
	values are
	.Sy none ,
	.Sy available ,
	and
	.Sy unavailable .
	See
	.Nm zfs Cm load-key
	and
	.Nm zfs Cm unload-key .
	.It Sy guid
	The 64 bit GUID of this dataset or bookmark which does not change over its
	entire lifetime. When a snapshot is sent to another pool, the received
	snapshot has the same GUID. Thus, the
	.Sy guid
	is suitable to identify a snapshot across pools.
	.It Sy logicalreferenced
	The amount of space that is
	.Qq logically
	accessible by this dataset.
	See the
	.Sy referenced
	property.
	The logical space ignores the effect of the
	.Sy compression
	and
	.Sy copies
	properties, giving a quantity closer to the amount of data that applications
	see.
	However, it does include space consumed by metadata.
	.Pp
	This property can also be referred to by its shortened column name,
	.Sy lrefer .
	.It Sy logicalused
	The amount of space that is
	.Qq logically
	consumed by this dataset and all its descendents.
	See the
	.Sy used
	property.
	The logical space ignores the effect of the
	.Sy compression
	and
	.Sy copies
	properties, giving a quantity closer to the amount of data that applications
	see.
	However, it does include space consumed by metadata.
	.Pp
	This property can also be referred to by its shortened column name,
	.Sy lused .
	.It Sy mounted
	For file systems, indicates whether the file system is currently mounted.
	This property can be either
	.Sy yes
	or
	.Sy no .
	.It Sy objsetid
	A unique identifier for this dataset within the pool. Unlike the dataset's
	.Sy guid
	, the
	.Sy objsetid
	of a dataset is not transferred to other pools when the snapshot is copied
	with a send/receive operation.
	The
	.Sy objsetid
	can be reused (for a new dataset) after the dataset is deleted.
	.It Sy origin
	For cloned file systems or volumes, the snapshot from which the clone was
	created.
	See also the
	.Sy clones
	property.
	.It Sy receive_resume_token
	For filesystems or volumes which have saved partially-completed state from
	.Sy zfs receive -s ,
	this opaque token can be provided to
	.Sy zfs send -t
	to resume and complete the
	.Sy zfs receive .
	.It Sy redact_snaps
	For bookmarks, this is the list of snapshot guids the bookmark contains a redaction
	list for.
	For snapshots, this is the list of snapshot guids the snapshot is redacted with
	respect to.
	.It Sy referenced
	The amount of data that is accessible by this dataset, which may or may not be
	shared with other datasets in the pool.
	When a snapshot or clone is created, it initially references the same amount of
	space as the file system or snapshot it was created from, since its contents are
	identical.
	.Pp
	This property can also be referred to by its shortened column name,
	.Sy refer .
	.It Sy refcompressratio
	The compression ratio achieved for the
	.Sy referenced
	space of this dataset, expressed as a multiplier.
	See also the
	.Sy compressratio
	property.
	.It Sy snapshot_count
	The total number of snapshots that exist under this location in the dataset
	tree.
	This value is only available when a
	.Sy snapshot_limit
	has been set somewhere in the tree under which the dataset resides.
	.It Sy type
	The type of dataset:
	.Sy filesystem ,
	.Sy volume ,
	.Sy snapshot ,
	or
	.Sy bookmark .
	.It Sy used
	The amount of space consumed by this dataset and all its descendents.
	This is the value that is checked against this dataset's quota and reservation.
	The space used does not include this dataset's reservation, but does take into
	account the reservations of any descendent datasets.
	The amount of space that a dataset consumes from its parent, as well as the
	amount of space that is freed if this dataset is recursively destroyed, is the
	greater of its space used and its reservation.
	.Pp
	The used space of a snapshot
	.Po see the
	.Em Snapshots
	section of
	.Xr zfsconcepts 8
	.Pc
	is space that is referenced exclusively by this snapshot.
	If this snapshot is destroyed, the amount of
	.Sy used
	space will be freed.
	Space that is shared by multiple snapshots isn't accounted for in this metric.
	When a snapshot is destroyed, space that was previously shared with this
	snapshot can become unique to snapshots adjacent to it, thus changing the used
	space of those snapshots.
	The used space of the latest snapshot can also be affected by changes in the
	file system.
	Note that the
	.Sy used
	space of a snapshot is a subset of the
	.Sy written
	space of the snapshot.
	.Pp
	The amount of space used, available, or referenced does not take into account
	pending changes.
	Pending changes are generally accounted for within a few seconds.
	Committing a change to a disk using
	.Xr fsync 2
	or
	.Dv O_SYNC
	does not necessarily guarantee that the space usage information is updated
	immediately.
	.It Sy usedby*
	The
	.Sy usedby*
	properties decompose the
	.Sy used
	properties into the various reasons that space is used.
	Specifically,
	.Sy used No =
	.Sy usedbychildren No +
	.Sy usedbydataset No +
	.Sy usedbyrefreservation No +
	.Sy usedbysnapshots .
	These properties are only available for datasets created on
	.Nm zpool
	.Qo version 13 Qc
	pools.
	.It Sy usedbychildren
	The amount of space used by children of this dataset, which would be freed if
	all the dataset's children were destroyed.
	.It Sy usedbydataset
	The amount of space used by this dataset itself, which would be freed if the
	dataset were destroyed
	.Po after first removing any
	.Sy refreservation
	and destroying any necessary snapshots or descendents
	.Pc .
	.It Sy usedbyrefreservation
	The amount of space used by a
	.Sy refreservation
	set on this dataset, which would be freed if the
	.Sy refreservation
	was removed.
	.It Sy usedbysnapshots
	The amount of space consumed by snapshots of this dataset.
	In particular, it is the amount of space that would be freed if all of this
	dataset's snapshots were destroyed.
	Note that this is not simply the sum of the snapshots'
	.Sy used
	properties because space can be shared by multiple snapshots.
	.It Sy userused Ns @ Ns Em user
	The amount of space consumed by the specified user in this dataset.
	Space is charged to the owner of each file, as displayed by
	.Nm ls Fl l .
	The amount of space charged is displayed by
	.Nm du
	and
	.Nm ls Fl s .
	See the
	.Nm zfs Cm userspace
	subcommand for more information.
	.Pp
	Unprivileged users can access only their own space usage.
	The root user, or a user who has been granted the
	.Sy userused
	privilege with
	.Nm zfs Cm allow ,
	can access everyone's usage.
	.Pp
	The
	.Sy userused Ns @ Ns Em ...
	properties are not displayed by
	.Nm zfs Cm get Sy all .
	The user's name must be appended after the @ symbol, using one of the following
	forms:
	.Bl -bullet -width ""
	.It
	.Em POSIX name
	.Po for example,
	.Sy joe
	.Pc
	.It
	.Em POSIX numeric ID
	.Po for example,
	.Sy 789
	.Pc
	.It
	.Em SID name
	.Po for example,
	.Sy joe.smith@mydomain
	.Pc
	.It
	.Em SID numeric ID
	.Po for example,
	.Sy S-1-123-456-789
	.Pc
	.El
	.Pp
	Files created on Linux always have POSIX owners.
	.It Sy userobjused Ns @ Ns Em user
	The
	.Sy userobjused
	property is similar to
	.Sy userused
	but instead it counts the number of objects consumed by a user. This property
	counts all objects allocated on behalf of the user, it may differ from the
	results of system tools such as
	.Nm df Fl i .
	.Pp
	When the property
	.Sy xattr=on
	is set on a file system additional objects will be created per-file to store
	extended attributes. These additional objects are reflected in the
	.Sy userobjused
	value and are counted against the user's
	.Sy userobjquota .
	When a file system is configured to use
	.Sy xattr=sa
	no additional internal objects are normally required.
	.It Sy userrefs
	This property is set to the number of user holds on this snapshot.
	User holds are set by using the
	.Nm zfs Cm hold
	command.
	.It Sy groupused Ns @ Ns Em group
	The amount of space consumed by the specified group in this dataset.
	Space is charged to the group of each file, as displayed by
	.Nm ls Fl l .
	See the
	.Sy userused Ns @ Ns Em user
	property for more information.
	.Pp
	Unprivileged users can only access their own groups' space usage.
	The root user, or a user who has been granted the
	.Sy groupused
	privilege with
	.Nm zfs Cm allow ,
	can access all groups' usage.
	.It Sy groupobjused Ns @ Ns Em group
	The number of objects consumed by the specified group in this dataset.
	Multiple objects may be charged to the group for each file when extended
	attributes are in use. See the
	.Sy userobjused Ns @ Ns Em user
	property for more information.
	.Pp
	Unprivileged users can only access their own groups' space usage.
	The root user, or a user who has been granted the
	.Sy groupobjused
	privilege with
	.Nm zfs Cm allow ,
	can access all groups' usage.
	.It Sy projectused Ns @ Ns Em project
	The amount of space consumed by the specified project in this dataset. Project
	is identified via the project identifier (ID) that is object-based numeral
	attribute. An object can inherit the project ID from its parent object (if the
	parent has the flag of inherit project ID that can be set and changed via
	.Nm chattr Fl /+P
	or
	.Nm zfs project Fl s )
	when being created. The privileged user can set and change object's project
	ID via
	.Nm chattr Fl p
	or
	.Nm zfs project Fl s
	anytime. Space is charged to the project of each file, as displayed by
	.Nm lsattr Fl p
	or
	.Nm zfs project .
	See the
	.Sy userused Ns @ Ns Em user
	property for more information.
	.Pp
	The root user, or a user who has been granted the
	.Sy projectused
	privilege with
	.Nm zfs allow ,
	can access all projects' usage.
	.It Sy projectobjused Ns @ Ns Em project
	The
	.Sy projectobjused
	is similar to
	.Sy projectused
	but instead it counts the number of objects consumed by project. When the
	property
	.Sy xattr=on
	is set on a fileset, ZFS will create additional objects per-file to store
	extended attributes. These additional objects are reflected in the
	.Sy projectobjused
	value and are counted against the project's
	.Sy projectobjquota .
	When a filesystem is configured to use
	.Sy xattr=sa
	no additional internal objects are required. See the
	.Sy userobjused Ns @ Ns Em user
	property for more information.
	.Pp
	The root user, or a user who has been granted the
	.Sy projectobjused
	privilege with
	.Nm zfs allow ,
	can access all projects' objects usage.
	.It Sy volblocksize
	For volumes, specifies the block size of the volume.
	The
	.Sy blocksize
	cannot be changed once the volume has been written, so it should be set at
	volume creation time.
	The default
	.Sy blocksize
	for volumes is 8 Kbytes.
	Any power of 2 from 512 bytes to 128 Kbytes is valid.
	.Pp
	This property can also be referred to by its shortened column name,
	.Sy volblock .
	.It Sy written
	The amount of space
	.Sy referenced
	by this dataset, that was written since the previous snapshot
	.Pq i.e. that is not referenced by the previous snapshot .
	.It Sy written Ns @ Ns Em snapshot
	The amount of
	.Sy referenced
	space written to this dataset since the specified snapshot.
	This is the space that is referenced by this dataset but was not referenced by
	the specified snapshot.
	.Pp
	The
	.Em snapshot
	may be specified as a short snapshot name
	.Po just the part after the
	.Sy @
	.Pc ,
	in which case it will be interpreted as a snapshot in the same filesystem as
	this dataset.
	The
	.Em snapshot
	may be a full snapshot name
	.Po Em filesystem Ns @ Ns Em snapshot Pc ,
	which for clones may be a snapshot in the origin's filesystem
	.Pq or the origin of the origin's filesystem, etc.
	.El
	.Pp
	The following native properties can be used to change the behavior of a ZFS
	dataset.
	.Bl -tag -width ""
	.It Xo
	.Sy aclinherit Ns = Ns Sy discard Ns \| Ns Sy noallow Ns \| Ns
	.Sy restricted Ns \| Ns Sy passthrough Ns \| Ns Sy passthrough-x
	.Xc
	Controls how ACEs are inherited when files and directories are created.
	.Bl -tag -width "passthrough-x"
	.It Sy discard
	does not inherit any ACEs.
	.It Sy noallow
	only inherits inheritable ACEs that specify
	.Qq deny
	permissions.
	.It Sy restricted
	default, removes the
	.Sy write_acl
	and
	.Sy write_owner
	permissions when the ACE is inherited.
	.It Sy passthrough
	inherits all inheritable ACEs without any modifications.
	.It Sy passthrough-x
	same meaning as
	.Sy passthrough ,
	except that the
	.Sy owner@ ,
	.Sy group@ ,
	and
	.Sy everyone@
	ACEs inherit the execute permission only if the file creation mode also requests
	the execute bit.
	.El
	.Pp
	When the property value is set to
	.Sy passthrough ,
	files are created with a mode determined by the inheritable ACEs.
	If no inheritable ACEs exist that affect the mode, then the mode is set in
	accordance to the requested mode from the application.
	.Pp
	The
	.Sy aclinherit
	property does not apply to POSIX ACLs.
	.It Xo
	.Sy aclmode Ns = Ns Sy discard Ns \| Ns Sy groupmask Ns \| Ns
	.Sy passthrough Ns \| Ns Sy restricted Ns
	.Xc
	Controls how an ACL is modified during chmod(2) and how inherited ACEs
	are modified by the file creation mode.
	.Bl -tag -width "passthrough"
	.It Sy discard
	default, deletes all
	.Sy ACEs
	except for those representing
	the mode of the file or directory requested by
	.Xr chmod 2 .
	.It Sy groupmask
	reduces permissions granted in all
	.Sy ALLOW
	entries found in the
	.Sy ACL
	such that they are no greater than the group permissions specified by
	.Xr chmod 2 .
	.It Sy passthrough
	indicates that no changes are made to the
	.Tn ACL
	other than creating or updating the necessary
	.Tn ACL
	entries to represent the new mode of the file or directory.
	.It Sy restricted
	will cause the
	.Xr chmod 2
	operation to return an error when used on any file or directory which has
	a non-trivial
	.Tn ACL
	whose entries can not be represented by a mode.
	.Xr chmod 2
	is required to change the set user ID, set group ID, or sticky bits on a file
	or directory, as they do not have equivalent
	.Tn ACL
	entries.
	In order to use
	.Xr chmod 2
	on a file or directory with a non-trivial
	.Tn ACL
	when
	.Sy aclmode
	is set to
	.Sy restricted ,
	you must first remove all
	.Tn ACL
	entries which do not represent the current mode.
	.El
	-.It Sy acltype Ns = Ns Sy off Ns \| Ns Sy posix
	+.It Sy acltype Ns = Ns Sy off Ns \| Ns Sy nfsv4 Ns \| Ns Sy posix
	Controls whether ACLs are enabled and if so what type of ACL to use.
	-This property is not visible on FreeBSD yet.
	+When this property is set to a type of ACL not supported by the current
	+platform, the behavior is the same as if it were set to
	+.Sy off .
	.Bl -tag -width "posixacl"
	.It Sy off
	-default, when a file system has the
	+default on Linux, when a file system has the
	.Sy acltype
	property set to off then ACLs are disabled.
	.It Sy noacl
	an alias for
	.Sy off
	+.It Sy nfsv4
	+default on FreeBSD, indicates that NFSv4-style ZFS ACLs should be used.
	+These ACLs can be managed with the
	+.Xr getfacl 1
	+and
	+.Xr setfacl 1
	+commands on FreeBSD. The
	+.Sy nfsv4
	+ZFS ACL type is not yet supported on Linux.
	.It Sy posix
	indicates POSIX ACLs should be used. POSIX ACLs are specific to Linux and are
	not functional on other platforms. POSIX ACLs are stored as an extended
	attribute and therefore will not overwrite any existing NFSv4 ACLs which
	may be set.
	.It Sy posixacl
	an alias for
	.Sy posix
	.El
	.Pp
	To obtain the best performance when setting
	.Sy posix
	users are strongly encouraged to set the
	.Sy xattr=sa
	property. This will result in the POSIX ACL being stored more efficiently on
	disk. But as a consequence, all new extended attributes will only be
	accessible from OpenZFS implementations which support the
	.Sy xattr=sa
	property. See the
	.Sy xattr
	property for more details.
	.It Sy atime Ns = Ns Sy on Ns \| Ns Sy off
	Controls whether the access time for files is updated when they are read.
	Turning this property off avoids producing write traffic when reading files and
	can result in significant performance gains, though it might confuse mailers
	and other similar utilities. The values
	.Sy on
	and
	.Sy off
	are equivalent to the
	.Sy atime
	and
	.Sy noatime
	mount options. The default value is
	.Sy on .
	See also
	.Sy relatime
	below.
	.It Sy canmount Ns = Ns Sy on Ns \| Ns Sy off Ns \| Ns Sy noauto
	If this property is set to
	.Sy off ,
	the file system cannot be mounted, and is ignored by
	.Nm zfs Cm mount Fl a .
	Setting this property to
	.Sy off
	is similar to setting the
	.Sy mountpoint
	property to
	.Sy none ,
	except that the dataset still has a normal
	.Sy mountpoint
	property, which can be inherited.
	Setting this property to
	.Sy off
	allows datasets to be used solely as a mechanism to inherit properties.
	One example of setting
	.Sy canmount Ns = Ns Sy off
	is to have two datasets with the same
	.Sy mountpoint ,
	so that the children of both datasets appear in the same directory, but might
	have different inherited characteristics.
	.Pp
	When set to
	.Sy noauto ,
	a dataset can only be mounted and unmounted explicitly.
	The dataset is not mounted automatically when the dataset is created or
	imported, nor is it mounted by the
	.Nm zfs Cm mount Fl a
	command or unmounted by the
	.Nm zfs Cm unmount Fl a
	command.
	.Pp
	This property is not inherited.
	.It Xo
	.Sy checksum Ns = Ns Sy on Ns \| Ns Sy off Ns \| Ns Sy fletcher2 Ns \| Ns
	.Sy fletcher4 Ns \| Ns Sy sha256 Ns \| Ns Sy noparity Ns \| Ns
	.Sy sha512 Ns \| Ns Sy skein Ns \| Ns Sy edonr
	.Xc
	Controls the checksum used to verify data integrity.
	The default value is
	.Sy on ,
	which automatically selects an appropriate algorithm
	.Po currently,
	.Sy fletcher4 ,
	but this may change in future releases
	.Pc .
	The value
	.Sy off
	disables integrity checking on user data.
	The value
	.Sy noparity
	not only disables integrity but also disables maintaining parity for user data.
	This setting is used internally by a dump device residing on a RAID-Z pool and
	should not be used by any other dataset.
	Disabling checksums is
	.Sy NOT
	a recommended practice.
	.Pp
	The
	.Sy sha512 ,
	.Sy skein ,
	and
	.Sy edonr
	checksum algorithms require enabling the appropriate features on the pool.
	FreeBSD does not support the
	.Sy edonr
	algorithm.
	.Pp
	Please see
	.Xr zpool-features 5
	for more information on these algorithms.
	.Pp
	Changing this property affects only newly-written data.
	.It Xo
	.Sy compression Ns = Ns Sy on Ns \| Ns Sy off Ns \| Ns Sy gzip Ns \| Ns
	.Sy gzip- Ns Em N Ns \| Ns Sy lz4 Ns \| Ns Sy lzjb Ns \| Ns Sy zle Ns \| Ns Sy zstd Ns \| Ns
	.Sy zstd- Ns Em N Ns \| Ns Sy zstd-fast Ns \| Ns Sy zstd-fast- Ns Em N
	.Xc
	Controls the compression algorithm used for this dataset.
	.Pp
	Setting compression to
	.Sy on
	indicates that the current default compression algorithm should be used.
	The default balances compression and decompression speed, with compression ratio
	and is expected to work well on a wide variety of workloads.
	Unlike all other settings for this property,
	.Sy on
	does not select a fixed compression type.
	As new compression algorithms are added to ZFS and enabled on a pool, the
	default compression algorithm may change.
	The current default compression algorithm is either
	.Sy lzjb
	or, if the
	.Sy lz4_compress
	feature is enabled,
	.Sy lz4 .
	.Pp
	The
	.Sy lz4
	compression algorithm is a high-performance replacement for the
	.Sy lzjb
	algorithm.
	It features significantly faster compression and decompression, as well as a
	moderately higher compression ratio than
	.Sy lzjb ,
	but can only be used on pools with the
	.Sy lz4_compress
	feature set to
	.Sy enabled .
	See
	.Xr zpool-features 5
	for details on ZFS feature flags and the
	.Sy lz4_compress
	feature.
	.Pp
	The
	.Sy lzjb
	compression algorithm is optimized for performance while providing decent data
	compression.
	.Pp
	The
	.Sy gzip
	compression algorithm uses the same compression as the
	.Xr gzip 1
	command.
	You can specify the
	.Sy gzip
	level by using the value
	.Sy gzip- Ns Em N ,
	where
	.Em N
	is an integer from 1
	.Pq fastest
	to 9
	.Pq best compression ratio .
	Currently,
	.Sy gzip
	is equivalent to
	.Sy gzip-6
	.Po which is also the default for
	.Xr gzip 1
	.Pc .
	.Pp
	The
	.Sy zstd
	compression algorithm provides both high compression ratios and good
	performance. You can specify the
	.Sy zstd
	level by using the value
	.Sy zstd- Ns Em N ,
	where
	.Em N
	is an integer from 1
	.Pq fastest
	to 19
	.Pq best compression ratio .
	.Sy zstd
	is equivalent to
	.Sy zstd-3 .
	.Pp
	Faster speeds at the cost of the compression ratio can be requested by
	setting a negative
	.Sy zstd
	level. This is done using
	.Sy zstd-fast- Ns Em N ,
	where
	.Em N
	is an integer in [1-9,10,20,30,...,100,500,1000] which maps to a negative
	.Sy zstd
	level. The lower the level the faster the compression - 1000 provides
	the fastest compression and lowest compression ratio.
	.Sy zstd-fast
	is equivalent to
	.Sy zstd-fast-1 .
	.Pp
	The
	.Sy zle
	compression algorithm compresses runs of zeros.
	.Pp
	This property can also be referred to by its shortened column name
	.Sy compress .
	Changing this property affects only newly-written data.
	.Pp
	When any setting except
	.Sy off
	is selected, compression will explicitly check for blocks consisting of only
	zeroes (the NUL byte). When a zero-filled block is detected, it is stored as
	a hole and not compressed using the indicated compression algorithm.
	.Pp
	Any block being compressed must be no larger than 7/8 of its original size
	after compression, otherwise the compression will not be considered worthwhile
	and the block saved uncompressed. Note that when the logical block is less than
	8 times the disk sector size this effectively reduces the necessary compression
	ratio; for example 8k blocks on disks with 4k disk sectors must compress to 1/2
	or less of their original size.
	.It Xo
	.Sy context Ns = Ns Sy none Ns \| Ns
	.Em SELinux_User:SElinux_Role:Selinux_Type:Sensitivity_Level
	.Xc
	This flag sets the SELinux context for all files in the file system under
	a mount point for that file system. See
	.Xr selinux 8
	for more information.
	.It Xo
	.Sy fscontext Ns = Ns Sy none Ns \| Ns
	.Em SELinux_User:SElinux_Role:Selinux_Type:Sensitivity_Level
	.Xc
	This flag sets the SELinux context for the file system file system being
	mounted. See
	.Xr selinux 8
	for more information.
	.It Xo
	.Sy defcontext Ns = Ns Sy none Ns \| Ns
	.Em SELinux_User:SElinux_Role:Selinux_Type:Sensitivity_Level
	.Xc
	This flag sets the SELinux default context for unlabeled files. See
	.Xr selinux 8
	for more information.
	.It Xo
	.Sy rootcontext Ns = Ns Sy none Ns \| Ns
	.Em SELinux_User:SElinux_Role:Selinux_Type:Sensitivity_Level
	.Xc
	This flag sets the SELinux context for the root inode of the file system. See
	.Xr selinux 8
	for more information.
	.It Sy copies Ns = Ns Sy 1 Ns \| Ns Sy 2 Ns \| Ns Sy 3
	Controls the number of copies of data stored for this dataset.
	These copies are in addition to any redundancy provided by the pool, for
	example, mirroring or RAID-Z.
	The copies are stored on different disks, if possible.
	The space used by multiple copies is charged to the associated file and dataset,
	changing the
	.Sy used
	property and counting against quotas and reservations.
	.Pp
	Changing this property only affects newly-written data.
	Therefore, set this property at file system creation time by using the
	.Fl o Sy copies Ns = Ns Ar N
	option.
	.Pp
	Remember that ZFS will not import a pool with a missing top-level vdev. Do
	.Sy NOT
	create, for example a two-disk striped pool and set
	.Sy copies=2
	on some datasets thinking you have setup redundancy for them. When a disk
	fails you will not be able to import the pool and will have lost all of your
	data.
	.Pp
	Encrypted datasets may not have
	.Sy copies Ns = Ns Em 3
	since the implementation stores some encryption metadata where the third copy
	would normally be.
	.It Sy devices Ns = Ns Sy on Ns \| Ns Sy off
	Controls whether device nodes can be opened on this file system.
	The default value is
	.Sy on .
	The values
	.Sy on
	and
	.Sy off
	are equivalent to the
	.Sy dev
	and
	.Sy nodev
	mount options.
	.It Xo
	.Sy dedup Ns = Ns Sy off Ns \| Ns Sy on Ns \| Ns Sy verify Ns \| Ns
	.Sy sha256[,verify] Ns \| Ns Sy sha512[,verify] Ns \| Ns Sy skein[,verify] Ns \| Ns
	.Sy edonr,verify
	.Xc
	Configures deduplication for a dataset. The default value is
	.Sy off .
	The default deduplication checksum is
	.Sy sha256
	(this may change in the future). When
	.Sy dedup
	is enabled, the checksum defined here overrides the
	.Sy checksum
	property. Setting the value to
	.Sy verify
	has the same effect as the setting
	.Sy sha256,verify.
	.Pp
	If set to
	.Sy verify ,
	ZFS will do a byte-to-byte comparison in case of two blocks having the same
	signature to make sure the block contents are identical. Specifying
	.Sy verify
	is mandatory for the
	.Sy edonr
	algorithm.
	.Pp
	Unless necessary, deduplication should NOT be enabled on a system. See the
	.Em Deduplication
	section of
	.Xr zfsconcepts 8 .
	.It Xo
	.Sy dnodesize Ns = Ns Sy legacy Ns \| Ns Sy auto Ns \| Ns Sy 1k Ns \| Ns
	.Sy 2k Ns \| Ns Sy 4k Ns \| Ns Sy 8k Ns \| Ns Sy 16k
	.Xc
	Specifies a compatibility mode or literal value for the size of dnodes in the
	file system. The default value is
	.Sy legacy .
	Setting this property to a value other than
	.Sy legacy
	requires the large_dnode pool feature to be enabled.
	.Pp
	Consider setting
	.Sy dnodesize
	to
	.Sy auto
	if the dataset uses the
	.Sy xattr=sa
	property setting and the workload makes heavy use of extended attributes. This
	may be applicable to SELinux-enabled systems, Lustre servers, and Samba
	servers, for example. Literal values are supported for cases where the optimal
	size is known in advance and for performance testing.
	.Pp
	Leave
	.Sy dnodesize
	set to
	.Sy legacy
	if you need to receive a send stream of this dataset on a pool that doesn't
	enable the large_dnode feature, or if you need to import this pool on a system
	that doesn't support the large_dnode feature.
	.Pp
	This property can also be referred to by its shortened column name,
	.Sy dnsize .
	.It Xo
	.Sy encryption Ns = Ns Sy off Ns \| Ns Sy on Ns \| Ns Sy aes-128-ccm Ns \| Ns
	.Sy aes-192-ccm Ns \| Ns Sy aes-256-ccm Ns \| Ns Sy aes-128-gcm Ns \| Ns
	.Sy aes-192-gcm Ns \| Ns Sy aes-256-gcm
	.Xc
	Controls the encryption cipher suite (block cipher, key length, and mode) used
	for this dataset. Requires the
	.Sy encryption
	feature to be enabled on the pool.
	Requires a
	.Sy keyformat
	to be set at dataset creation time.
	.Pp
	Selecting
	.Sy encryption Ns = Ns Sy on
	when creating a dataset indicates that the default encryption suite will be
	selected, which is currently
	.Sy aes-256-gcm .
	In order to provide consistent data protection, encryption must be specified at
	dataset creation time and it cannot be changed afterwards.
	.Pp
	For more details and caveats about encryption see the
	.Em Encryption
	section of
	.Xr zfs-load-key 8 .
	.It Sy keyformat Ns = Ns Sy raw Ns \| Ns Sy hex Ns \| Ns Sy passphrase
	Controls what format the user's encryption key will be provided as. This
	property is only set when the dataset is encrypted.
	.Pp
	Raw keys and hex keys must be 32 bytes long (regardless of the chosen
	encryption suite) and must be randomly generated. A raw key can be generated
	with the following command:
	.Bd -literal
	# dd if=/dev/urandom of=/path/to/output/key bs=32 count=1
	.Ed
	.Pp
	Passphrases must be between 8 and 512 bytes long and will be processed through
	PBKDF2 before being used (see the
	.Sy pbkdf2iters
	property). Even though the
	encryption suite cannot be changed after dataset creation, the keyformat can be
	with
	.Nm zfs Cm change-key .
	.It Xo
	.Sy keylocation Ns = Ns Sy prompt Ns \| Ns Sy file:// Ns Em </absolute/file/path>
	.Xc
	Controls where the user's encryption key will be loaded from by default for
	commands such as
	.Nm zfs Cm load-key
	and
	.Nm zfs Cm mount Cm -l .
	This property is only set for encrypted datasets which are encryption roots. If
	unspecified, the default is
	.Sy prompt.
	.Pp
	Even though the encryption suite cannot be changed after dataset creation, the
	keylocation can be with either
	.Nm zfs Cm set
	or
	.Nm zfs Cm change-key .
	If
	.Sy prompt
	is selected ZFS will ask for the key at the command prompt when it is required
	to access the encrypted data (see
	.Nm zfs Cm load-key
	for details). This setting will also allow the key to be passed in via STDIN,
	but users should be careful not to place keys which should be kept secret on
	the command line. If a file URI is selected, the key will be loaded from the
	specified absolute file path.
	.It Sy pbkdf2iters Ns = Ns Ar iterations
	Controls the number of PBKDF2 iterations that a
	.Sy passphrase
	encryption key should be run through when processing it into an encryption key.
	This property is only defined when encryption is enabled and a keyformat of
	.Sy passphrase
	is selected. The goal of PBKDF2 is to significantly increase the
	computational difficulty needed to brute force a user's passphrase. This is
	accomplished by forcing the attacker to run each passphrase through a
	computationally expensive hashing function many times before they arrive at the
	resulting key. A user who actually knows the passphrase will only have to pay
	this cost once. As CPUs become better at processing, this number should be
	raised to ensure that a brute force attack is still not possible. The current
	default is
	.Sy 350000
	and the minimum is
	.Sy 100000 .
	This property may be changed with
	.Nm zfs Cm change-key .
	.It Sy exec Ns = Ns Sy on Ns \| Ns Sy off
	Controls whether processes can be executed from within this file system.
	The default value is
	.Sy on .
	The values
	.Sy on
	and
	.Sy off
	are equivalent to the
	.Sy exec
	and
	.Sy noexec
	mount options.
	.It Sy filesystem_limit Ns = Ns Em count Ns \| Ns Sy none
	Limits the number of filesystems and volumes that can exist under this point in
	the dataset tree.
	The limit is not enforced if the user is allowed to change the limit.
	Setting a
	.Sy filesystem_limit
	to
	.Sy on
	a descendent of a filesystem that already has a
	.Sy filesystem_limit
	does not override the ancestor's
	.Sy filesystem_limit ,
	but rather imposes an additional limit.
	This feature must be enabled to be used
	.Po see
	.Xr zpool-features 5
	.Pc .
	.It Sy special_small_blocks Ns = Ns Em size
	This value represents the threshold block size for including small file
	blocks into the special allocation class. Blocks smaller than or equal to this
	value will be assigned to the special allocation class while greater blocks
	will be assigned to the regular class. Valid values are zero or a power of two
	from 512B up to 1M. The default size is 0 which means no small file blocks
	will be allocated in the special class.
	.Pp
	Before setting this property, a special class vdev must be added to the
	pool. See
	.Xr zpool 8
	for more details on the special allocation class.
	.It Sy mountpoint Ns = Ns Pa path Ns \| Ns Sy none Ns \| Ns Sy legacy
	Controls the mount point used for this file system.
	See the
	.Em Mount Points
	section of
	.Xr zfsconcepts 8
	for more information on how this property is used.
	.Pp
	When the
	.Sy mountpoint
	property is changed for a file system, the file system and any children that
	inherit the mount point are unmounted.
	If the new value is
	.Sy legacy ,
	then they remain unmounted.
	Otherwise, they are automatically remounted in the new location if the property
	was previously
	.Sy legacy
	or
	.Sy none ,
	or if they were mounted before the property was changed.
	In addition, any shared file systems are unshared and shared in the new
	location.
	.It Sy nbmand Ns = Ns Sy on Ns \| Ns Sy off
	Controls whether the file system should be mounted with
	.Sy nbmand
	.Pq Non Blocking mandatory locks .
	This is used for SMB clients.
	Changes to this property only take effect when the file system is umounted and
	remounted.
	See
	.Xr mount 8
	for more information on
	.Sy nbmand
	mounts. This property is not used on Linux.
	.It Sy overlay Ns = Ns Sy on Ns \| Ns Sy off
	Allow mounting on a busy directory or a directory which already contains
	files or directories.
	This is the default mount behavior for Linux and FreeBSD file systems.
	On these platforms the property is
	.Sy on
	by default.
	Set to
	.Sy off
	to disable overlay mounts for consistency with OpenZFS on other platforms.
	.It Sy primarycache Ns = Ns Sy all Ns \| Ns Sy none Ns \| Ns Sy metadata
	Controls what is cached in the primary cache
	.Pq ARC .
	If this property is set to
	.Sy all ,
	then both user data and metadata is cached.
	If this property is set to
	.Sy none ,
	then neither user data nor metadata is cached.
	If this property is set to
	.Sy metadata ,
	then only metadata is cached.
	The default value is
	.Sy all .
	.It Sy quota Ns = Ns Em size Ns \| Ns Sy none
	Limits the amount of space a dataset and its descendents can consume.
	This property enforces a hard limit on the amount of space used.
	This includes all space consumed by descendents, including file systems and
	snapshots.
	Setting a quota on a descendent of a dataset that already has a quota does not
	override the ancestor's quota, but rather imposes an additional limit.
	.Pp
	Quotas cannot be set on volumes, as the
	.Sy volsize
	property acts as an implicit quota.
	.It Sy snapshot_limit Ns = Ns Em count Ns \| Ns Sy none
	Limits the number of snapshots that can be created on a dataset and its
	descendents.
	Setting a
	.Sy snapshot_limit
	on a descendent of a dataset that already has a
	.Sy snapshot_limit
	does not override the ancestor's
	.Sy snapshot_limit ,
	but rather imposes an additional limit.
	The limit is not enforced if the user is allowed to change the limit.
	For example, this means that recursive snapshots taken from the global zone are
	counted against each delegated dataset within a zone.
	This feature must be enabled to be used
	.Po see
	.Xr zpool-features 5
	.Pc .
	.It Sy userquota@ Ns Em user Ns = Ns Em size Ns \| Ns Sy none
	Limits the amount of space consumed by the specified user.
	User space consumption is identified by the
	.Sy userspace@ Ns Em user
	property.
	.Pp
	Enforcement of user quotas may be delayed by several seconds.
	This delay means that a user might exceed their quota before the system notices
	that they are over quota and begins to refuse additional writes with the
	.Er EDQUOT
	error message.
	See the
	.Nm zfs Cm userspace
	subcommand for more information.
	.Pp
	Unprivileged users can only access their own groups' space usage.
	The root user, or a user who has been granted the
	.Sy userquota
	privilege with
	.Nm zfs Cm allow ,
	can get and set everyone's quota.
	.Pp
	This property is not available on volumes, on file systems before version 4, or
	on pools before version 15.
	The
	.Sy userquota@ Ns Em ...
	properties are not displayed by
	.Nm zfs Cm get Sy all .
	The user's name must be appended after the
	.Sy @
	symbol, using one of the following forms:
	.Bl -bullet
	.It
	.Em POSIX name
	.Po for example,
	.Sy joe
	.Pc
	.It
	.Em POSIX numeric ID
	.Po for example,
	.Sy 789
	.Pc
	.It
	.Em SID name
	.Po for example,
	.Sy joe.smith@mydomain
	.Pc
	.It
	.Em SID numeric ID
	.Po for example,
	.Sy S-1-123-456-789
	.Pc
	.El
	.Pp
	Files created on Linux always have POSIX owners.
	.It Sy userobjquota@ Ns Em user Ns = Ns Em size Ns \| Ns Sy none
	The
	.Sy userobjquota
	is similar to
	.Sy userquota
	but it limits the number of objects a user can create. Please refer to
	.Sy userobjused
	for more information about how objects are counted.
	.It Sy groupquota@ Ns Em group Ns = Ns Em size Ns \| Ns Sy none
	Limits the amount of space consumed by the specified group.
	Group space consumption is identified by the
	.Sy groupused@ Ns Em group
	property.
	.Pp
	Unprivileged users can access only their own groups' space usage.
	The root user, or a user who has been granted the
	.Sy groupquota
	privilege with
	.Nm zfs Cm allow ,
	can get and set all groups' quotas.
	.It Sy groupobjquota@ Ns Em group Ns = Ns Em size Ns \| Ns Sy none
	The
	.Sy groupobjquota
	is similar to
	.Sy groupquota
	but it limits number of objects a group can consume. Please refer to
	.Sy userobjused
	for more information about how objects are counted.
	.It Sy projectquota@ Ns Em project Ns = Ns Em size Ns \| Ns Sy none
	Limits the amount of space consumed by the specified project. Project
	space consumption is identified by the
	.Sy projectused@ Ns Em project
	property. Please refer to
	.Sy projectused
	for more information about how project is identified and set/changed.
	.Pp
	The root user, or a user who has been granted the
	.Sy projectquota
	privilege with
	.Nm zfs allow ,
	can access all projects' quota.
	.It Sy projectobjquota@ Ns Em project Ns = Ns Em size Ns \| Ns Sy none
	The
	.Sy projectobjquota
	is similar to
	.Sy projectquota
	but it limits number of objects a project can consume. Please refer to
	.Sy userobjused
	for more information about how objects are counted.
	.It Sy readonly Ns = Ns Sy on Ns \| Ns Sy off
	Controls whether this dataset can be modified.
	The default value is
	.Sy off .
	The values
	.Sy on
	and
	.Sy off
	are equivalent to the
	.Sy ro
	and
	.Sy rw
	mount options.
	.Pp
	This property can also be referred to by its shortened column name,
	.Sy rdonly .
	.It Sy recordsize Ns = Ns Em size
	Specifies a suggested block size for files in the file system.
	This property is designed solely for use with database workloads that access
	files in fixed-size records.
	ZFS automatically tunes block sizes according to internal algorithms optimized
	for typical access patterns.
	.Pp
	For databases that create very large files but access them in small random
	chunks, these algorithms may be suboptimal.
	Specifying a
	.Sy recordsize
	greater than or equal to the record size of the database can result in
	significant performance gains.
	Use of this property for general purpose file systems is strongly discouraged,
	and may adversely affect performance.
	.Pp
	The size specified must be a power of two greater than or equal to 512 and less
	than or equal to 128 Kbytes.
	If the
	.Sy large_blocks
	feature is enabled on the pool, the size may be up to 1 Mbyte.
	See
	.Xr zpool-features 5
	for details on ZFS feature flags.
	.Pp
	Changing the file system's
	.Sy recordsize
	affects only files created afterward; existing files are unaffected.
	.Pp
	This property can also be referred to by its shortened column name,
	.Sy recsize .
	.It Sy redundant_metadata Ns = Ns Sy all Ns \| Ns Sy most
	Controls what types of metadata are stored redundantly.
	ZFS stores an extra copy of metadata, so that if a single block is corrupted,
	the amount of user data lost is limited.
	This extra copy is in addition to any redundancy provided at the pool level
	.Pq e.g. by mirroring or RAID-Z ,
	and is in addition to an extra copy specified by the
	.Sy copies
	property
	.Pq up to a total of 3 copies .
	For example if the pool is mirrored,
	.Sy copies Ns = Ns 2 ,
	and
	.Sy redundant_metadata Ns = Ns Sy most ,
	then ZFS stores 6 copies of most metadata, and 4 copies of data and some
	metadata.
	.Pp
	When set to
	.Sy all ,
	ZFS stores an extra copy of all metadata.
	If a single on-disk block is corrupt, at worst a single block of user data
	.Po which is
	.Sy recordsize
	bytes long
	.Pc
	can be lost.
	.Pp
	When set to
	.Sy most ,
	ZFS stores an extra copy of most types of metadata.
	This can improve performance of random writes, because less metadata must be
	written.
	In practice, at worst about 100 blocks
	.Po of
	.Sy recordsize
	bytes each
	.Pc
	of user data can be lost if a single on-disk block is corrupt.
	The exact behavior of which metadata blocks are stored redundantly may change in
	future releases.
	.Pp
	The default value is
	.Sy all .
	.It Sy refquota Ns = Ns Em size Ns \| Ns Sy none
	Limits the amount of space a dataset can consume.
	This property enforces a hard limit on the amount of space used.
	This hard limit does not include space used by descendents, including file
	systems and snapshots.
	.It Sy refreservation Ns = Ns Em size Ns \| Ns Sy none Ns \| Ns Sy auto
	The minimum amount of space guaranteed to a dataset, not including its
	descendents.
	When the amount of space used is below this value, the dataset is treated as if
	it were taking up the amount of space specified by
	.Sy refreservation .
	The
	.Sy refreservation
	reservation is accounted for in the parent datasets' space used, and counts
	against the parent datasets' quotas and reservations.
	.Pp
	If
	.Sy refreservation
	is set, a snapshot is only allowed if there is enough free pool space outside of
	this reservation to accommodate the current number of
	.Qq referenced
	bytes in the dataset.
	.Pp
	If
	.Sy refreservation
	is set to
	.Sy auto ,
	a volume is thick provisioned
	.Po or
	.Qq not sparse
	.Pc .
	.Sy refreservation Ns = Ns Sy auto
	is only supported on volumes.
	See
	.Sy volsize
	in the
	.Sx Native Properties
	section for more information about sparse volumes.
	.Pp
	This property can also be referred to by its shortened column name,
	.Sy refreserv .
	.It Sy relatime Ns = Ns Sy on Ns \| Ns Sy off
	Controls the manner in which the access time is updated when
	.Sy atime=on
	is set. Turning this property on causes the access time to be updated relative
	to the modify or change time. Access time is only updated if the previous
	access time was earlier than the current modify or change time or if the
	existing access time hasn't been updated within the past 24 hours. The default
	value is
	.Sy off .
	The values
	.Sy on
	and
	.Sy off
	are equivalent to the
	.Sy relatime
	and
	.Sy norelatime
	mount options.
	.It Sy reservation Ns = Ns Em size Ns \| Ns Sy none
	The minimum amount of space guaranteed to a dataset and its descendants.
	When the amount of space used is below this value, the dataset is treated as if
	it were taking up the amount of space specified by its reservation.
	Reservations are accounted for in the parent datasets' space used, and count
	against the parent datasets' quotas and reservations.
	.Pp
	This property can also be referred to by its shortened column name,
	.Sy reserv .
	.It Sy secondarycache Ns = Ns Sy all Ns \| Ns Sy none Ns \| Ns Sy metadata
	Controls what is cached in the secondary cache
	.Pq L2ARC .
	If this property is set to
	.Sy all ,
	then both user data and metadata is cached.
	If this property is set to
	.Sy none ,
	then neither user data nor metadata is cached.
	If this property is set to
	.Sy metadata ,
	then only metadata is cached.
	The default value is
	.Sy all .
	.It Sy setuid Ns = Ns Sy on Ns \| Ns Sy off
	Controls whether the setuid bit is respected for the file system.
	The default value is
	.Sy on .
	The values
	.Sy on
	and
	.Sy off
	are equivalent to the
	.Sy suid
	and
	.Sy nosuid
	mount options.
	.It Sy sharesmb Ns = Ns Sy on Ns \| Ns Sy off Ns \| Ns Em opts
	Controls whether the file system is shared by using
	.Sy Samba USERSHARES
	and what options are to be used. Otherwise, the file system is automatically
	shared and unshared with the
	.Nm zfs Cm share
	and
	.Nm zfs Cm unshare
	commands. If the property is set to on, the
	.Xr net 8
	command is invoked to create a
	.Sy USERSHARE .
	.Pp
	Because SMB shares requires a resource name, a unique resource name is
	constructed from the dataset name. The constructed name is a copy of the
	dataset name except that the characters in the dataset name, which would be
	invalid in the resource name, are replaced with underscore (_) characters.
	Linux does not currently support additional options which might be available
	on Solaris.
	.Pp
	If the
	.Sy sharesmb
	property is set to
	.Sy off ,
	the file systems are unshared.
	.Pp
	The share is created with the ACL (Access Control List) "Everyone:F" ("F"
	stands for "full permissions", ie. read and write permissions) and no guest
	access (which means Samba must be able to authenticate a real user, system
	passwd/shadow, LDAP or smbpasswd based) by default. This means that any
	additional access control (disallow specific user specific access etc) must
	be done on the underlying file system.
	.It Sy sharenfs Ns = Ns Sy on Ns \| Ns Sy off Ns \| Ns Em opts
	Controls whether the file system is shared via NFS, and what options are to be
	used.
	A file system with a
	.Sy sharenfs
	property of
	.Sy off
	is managed with the
	.Xr exportfs 8
	command and entries in the
	.Em /etc/exports
	file.
	Otherwise, the file system is automatically shared and unshared with the
	.Nm zfs Cm share
	and
	.Nm zfs Cm unshare
	commands.
	If the property is set to
	.Sy on ,
	the dataset is shared using the default options:
	.Pp
	.Em sec=sys,rw,crossmnt,no_subtree_check
	.Pp
	See
	.Xr exports 5
	for the meaning of the default options. Otherwise, the
	.Xr exportfs 8
	command is invoked with options equivalent to the contents of this property.
	.Pp
	When the
	.Sy sharenfs
	property is changed for a dataset, the dataset and any children inheriting the
	property are re-shared with the new options, only if the property was previously
	.Sy off ,
	or if they were shared before the property was changed.
	If the new property is
	.Sy off ,
	the file systems are unshared.
	.It Sy logbias Ns = Ns Sy latency Ns \| Ns Sy throughput
	Provide a hint to ZFS about handling of synchronous requests in this dataset.
	If
	.Sy logbias
	is set to
	.Sy latency
	.Pq the default ,
	ZFS will use pool log devices
	.Pq if configured
	to handle the requests at low latency.
	If
	.Sy logbias
	is set to
	.Sy throughput ,
	ZFS will not use configured pool log devices.
	ZFS will instead optimize synchronous operations for global pool throughput and
	efficient use of resources.
	.It Sy snapdev Ns = Ns Sy hidden Ns \| Ns Sy visible
	Controls whether the volume snapshot devices under
	.Em /dev/zvol/<pool>
	are hidden or visible. The default value is
	.Sy hidden .
	.It Sy snapdir Ns = Ns Sy hidden Ns \| Ns Sy visible
	Controls whether the
	.Pa .zfs
	directory is hidden or visible in the root of the file system as discussed in
	the
	.Em Snapshots
	section of
	.Xr zfsconcepts 8 .
	The default value is
	.Sy hidden .
	.It Sy sync Ns = Ns Sy standard Ns \| Ns Sy always Ns \| Ns Sy disabled
	Controls the behavior of synchronous requests
	.Pq e.g. fsync, O_DSYNC .
	.Sy standard
	is the
	.Tn POSIX
	specified behavior of ensuring all synchronous requests are written to stable
	storage and all devices are flushed to ensure data is not cached by device
	controllers
	.Pq this is the default .
	.Sy always
	causes every file system transaction to be written and flushed before its
	system call returns.
	This has a large performance penalty.
	.Sy disabled
	disables synchronous requests.
	File system transactions are only committed to stable storage periodically.
	This option will give the highest performance.
	However, it is very dangerous as ZFS would be ignoring the synchronous
	transaction demands of applications such as databases or NFS.
	Administrators should only use this option when the risks are understood.
	.It Sy version Ns = Ns Em N Ns \| Ns Sy current
	The on-disk version of this file system, which is independent of the pool
	version.
	This property can only be set to later supported versions.
	See the
	.Nm zfs Cm upgrade
	command.
	.It Sy volsize Ns = Ns Em size
	For volumes, specifies the logical size of the volume.
	By default, creating a volume establishes a reservation of equal size.
	For storage pools with a version number of 9 or higher, a
	.Sy refreservation
	is set instead.
	Any changes to
	.Sy volsize
	are reflected in an equivalent change to the reservation
	.Po or
	.Sy refreservation
	.Pc .
	The
	.Sy volsize
	can only be set to a multiple of
	.Sy volblocksize ,
	and cannot be zero.
	.Pp
	The reservation is kept equal to the volume's logical size to prevent unexpected
	behavior for consumers.
	Without the reservation, the volume could run out of space, resulting in
	undefined behavior or data corruption, depending on how the volume is used.
	These effects can also occur when the volume size is changed while it is in use
	.Pq particularly when shrinking the size .
	Extreme care should be used when adjusting the volume size.
	.Pp
	Though not recommended, a
	.Qq sparse volume
	.Po also known as
	.Qq thin provisioned
	.Pc
	can be created by specifying the
	.Fl s
	option to the
	.Nm zfs Cm create Fl V
	command, or by changing the value of the
	.Sy refreservation
	property
	.Po or
	.Sy reservation
	property on pool version 8 or earlier
	.Pc
	after the volume has been created.
	A
	.Qq sparse volume
	is a volume where the value of
	.Sy refreservation
	is less than the size of the volume plus the space required to store its
	metadata.
	Consequently, writes to a sparse volume can fail with
	.Er ENOSPC
	when the pool is low on space.
	For a sparse volume, changes to
	.Sy volsize
	are not reflected in the
	.Sy refreservation.
	A volume that is not sparse is said to be
	.Qq thick provisioned .
	A sparse volume can become thick provisioned by setting
	.Sy refreservation
	to
	.Sy auto .
	.It Sy volmode Ns = Ns Cm default \| full \| geom \| dev \| none
	This property specifies how volumes should be exposed to the OS.
	Setting it to
	.Sy full
	exposes volumes as fully fledged block devices, providing maximal
	functionality. The value
	.Sy geom
	is just an alias for
	.Sy full
	and is kept for compatibility.
	Setting it to
	.Sy dev
	hides its partitions.
	Volumes with property set to
	.Sy none
	are not exposed outside ZFS, but can be snapshotted, cloned, replicated, etc,
	that can be suitable for backup purposes.
	Value
	.Sy default
	means that volumes exposition is controlled by system-wide tunable
	.Va zvol_volmode ,
	where
	.Sy full ,
	.Sy dev
	and
	.Sy none
	are encoded as 1, 2 and 3 respectively.
	The default values is
	.Sy full .
	.It Sy vscan Ns = Ns Sy on Ns \| Ns Sy off
	Controls whether regular files should be scanned for viruses when a file is
	opened and closed.
	In addition to enabling this property, the virus scan service must also be
	enabled for virus scanning to occur.
	The default value is
	.Sy off .
	This property is not used on Linux.
	.It Sy xattr Ns = Ns Sy on Ns \| Ns Sy off Ns \| Ns Sy sa
	Controls whether extended attributes are enabled for this file system. Two
	styles of extended attributes are supported either directory based or system
	attribute based.
	.Pp
	The default value of
	.Sy on
	enables directory based extended attributes. This style of extended attribute
	imposes no practical limit on either the size or number of attributes which
	can be set on a file. Although under Linux the
	.Xr getxattr 2
	and
	.Xr setxattr 2
	system calls limit the maximum size to 64K. This is the most compatible
	style of extended attribute and is supported by all OpenZFS implementations.
	.Pp
	System attribute based xattrs can be enabled by setting the value to
	.Sy sa .
	The key advantage of this type of xattr is improved performance. Storing
	extended attributes as system attributes significantly decreases the amount of
	disk IO required. Up to 64K of data may be stored per-file in the space
	reserved for system attributes. If there is not enough space available for
	an extended attribute then it will be automatically written as a directory
	based xattr. System attribute based extended attributes are not accessible
	on platforms which do not support the
	.Sy xattr=sa
	feature.
	.Pp
	The use of system attribute based xattrs is strongly encouraged for users of
	-SELinux or POSIX ACLs. Both of these features heavily rely of extended
	+SELinux or POSIX ACLs. Both of these features heavily rely on extended
	attributes and benefit significantly from the reduced access time.
	.Pp
	The values
	.Sy on
	and
	.Sy off
	are equivalent to the
	.Sy xattr
	and
	.Sy noxattr
	mount options.
	.It Sy jailed Ns = Ns Cm off \| on
	Controls whether the dataset is managed from a jail. See the
	.Qq Sx Jails
	section in
	.Xr zfs 8
	for more information. Jails are a FreeBSD feature and are not relevant on
	other platforms. The default value is
	.Cm off .
	.It Sy zoned Ns = Ns Sy on Ns \| Ns Sy off
	Controls whether the dataset is managed from a non-global zone. Zones are a
	Solaris feature and are not relevant on other platforms. The default value is
	.Sy off .
	.El
	.Pp
	The following three properties cannot be changed after the file system is
	created, and therefore, should be set when the file system is created.
	If the properties are not set with the
	.Nm zfs Cm create
	or
	.Nm zpool Cm create
	commands, these properties are inherited from the parent dataset.
	If the parent dataset lacks these properties due to having been created prior to
	these features being supported, the new file system will have the default values
	for these properties.
	.Bl -tag -width ""
	.It Xo
	.Sy casesensitivity Ns = Ns Sy sensitive Ns \| Ns
	.Sy insensitive Ns \| Ns Sy mixed
	.Xc
	Indicates whether the file name matching algorithm used by the file system
	should be case-sensitive, case-insensitive, or allow a combination of both
	styles of matching.
	The default value for the
	.Sy casesensitivity
	property is
	.Sy sensitive .
	Traditionally,
	.Ux
	and
	.Tn POSIX
	file systems have case-sensitive file names.
	.Pp
	The
	.Sy mixed
	value for the
	.Sy casesensitivity
	property indicates that the file system can support requests for both
	case-sensitive and case-insensitive matching behavior.
	Currently, case-insensitive matching behavior on a file system that supports
	mixed behavior is limited to the SMB server product.
	For more information about the
	.Sy mixed
	value behavior, see the "ZFS Administration Guide".
	.It Xo
	.Sy normalization Ns = Ns Sy none Ns \| Ns Sy formC Ns \| Ns
	.Sy formD Ns \| Ns Sy formKC Ns \| Ns Sy formKD
	.Xc
	Indicates whether the file system should perform a
	.Sy unicode
	normalization of file names whenever two file names are compared, and which
	normalization algorithm should be used.
	File names are always stored unmodified, names are normalized as part of any
	comparison process.
	If this property is set to a legal value other than
	.Sy none ,
	and the
	.Sy utf8only
	property was left unspecified, the
	.Sy utf8only
	property is automatically set to
	.Sy on .
	The default value of the
	.Sy normalization
	property is
	.Sy none .
	This property cannot be changed after the file system is created.
	.It Sy utf8only Ns = Ns Sy on Ns \| Ns Sy off
	Indicates whether the file system should reject file names that include
	characters that are not present in the
	.Sy UTF-8
	character code set.
	If this property is explicitly set to
	.Sy off ,
	the normalization property must either not be explicitly set or be set to
	.Sy none .
	The default value for the
	.Sy utf8only
	property is
	.Sy off .
	This property cannot be changed after the file system is created.
	.El
	.Pp
	The
	.Sy casesensitivity ,
	.Sy normalization ,
	and
	.Sy utf8only
	properties are also new permissions that can be assigned to non-privileged users
	by using the ZFS delegated administration feature.
	.Ss "Temporary Mount Point Properties"
	When a file system is mounted, either through
	.Xr mount 8
	for legacy mounts or the
	.Nm zfs Cm mount
	command for normal file systems, its mount options are set according to its
	properties.
	The correlation between properties and mount options is as follows:
	.Bd -literal
	PROPERTY MOUNT OPTION
	atime atime/noatime
	canmount auto/noauto
	devices dev/nodev
	exec exec/noexec
	readonly ro/rw
	relatime relatime/norelatime
	setuid suid/nosuid
	xattr xattr/noxattr
	.Ed
	.Pp
	In addition, these options can be set on a per-mount basis using the
	.Fl o
	option, without affecting the property that is stored on disk.
	The values specified on the command line override the values stored in the
	dataset.
	The
	.Sy nosuid
	option is an alias for
	.Sy nodevices Ns \&, Ns Sy nosetuid .
	These properties are reported as
	.Qq temporary
	by the
	.Nm zfs Cm get
	command.
	If the properties are changed while the dataset is mounted, the new setting
	overrides any temporary settings.
	.Ss "User Properties"
	In addition to the standard native properties, ZFS supports arbitrary user
	properties.
	User properties have no effect on ZFS behavior, but applications or
	administrators can use them to annotate datasets
	.Pq file systems, volumes, and snapshots .
	.Pp
	User property names must contain a colon
	.Pq Qq Sy \&:
	character to distinguish them from native properties.
	They may contain lowercase letters, numbers, and the following punctuation
	characters: colon
	.Pq Qq Sy \&: ,
	dash
	.Pq Qq Sy - ,
	period
	.Pq Qq Sy \&. ,
	and underscore
	.Pq Qq Sy _ .
	The expected convention is that the property name is divided into two portions
	such as
	.Em module Ns \&: Ns Em property ,
	but this namespace is not enforced by ZFS.
	User property names can be at most 256 characters, and cannot begin with a dash
	.Pq Qq Sy - .
	.Pp
	When making programmatic use of user properties, it is strongly suggested to use
	a reversed
	.Sy DNS
	domain name for the
	.Em module
	component of property names to reduce the chance that two
	independently-developed packages use the same property name for different
	purposes.
	.Pp
	The values of user properties are arbitrary strings, are always inherited, and
	are never validated.
	All of the commands that operate on properties
	.Po Nm zfs Cm list ,
	.Nm zfs Cm get ,
	.Nm zfs Cm set ,
	and so forth
	.Pc
	can be used to manipulate both native properties and user properties.
	Use the
	.Nm zfs Cm inherit
	command to clear a user property.
	If the property is not defined in any parent dataset, it is removed entirely.
	Property values are limited to 8192 bytes.
	Index: vendor-sys/openzfs/dist/man/man8/zpool.8
	===================================================================
	--- vendor-sys/openzfs/dist/man/man8/zpool.8 (revision 366773)
	+++ vendor-sys/openzfs/dist/man/man8/zpool.8 (revision 366774)
	@@ -1,599 +1,599 @@
	.\"
	.\" CDDL HEADER START
	.\"
	.\" The contents of this file are subject to the terms of the
	.\" Common Development and Distribution License (the "License").
	.\" You may not use this file except in compliance with the License.
	.\"
	.\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	.\" or http://www.opensolaris.org/os/licensing.
	.\" See the License for the specific language governing permissions
	.\" and limitations under the License.
	.\"
	.\" When distributing Covered Code, include this CDDL HEADER in each
	.\" file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	.\" If applicable, add the following below this CDDL HEADER, with the
	.\" fields enclosed by brackets "[]" replaced with your own identifying
	.\" information: Portions Copyright [yyyy] [name of copyright owner]
	.\"
	.\" CDDL HEADER END
	.\"
	.\"
	.\" Copyright (c) 2007, Sun Microsystems, Inc. All Rights Reserved.
	.\" Copyright (c) 2012, 2018 by Delphix. All rights reserved.
	.\" Copyright (c) 2012 Cyril Plisko. All Rights Reserved.
	.\" Copyright (c) 2017 Datto Inc.
	.\" Copyright (c) 2018 George Melikov. All Rights Reserved.
	.\" Copyright 2017 Nexenta Systems, Inc.
	.\" Copyright (c) 2017 Open-E, Inc. All Rights Reserved.
	.\"
	.Dd August 9, 2019
	.Dt ZPOOL 8
	.Os
	.Sh NAME
	.Nm zpool
	.Nd configure ZFS storage pools
	.Sh SYNOPSIS
	.Nm
	.Fl ?V
	.Nm
	.Cm version
	.Nm
	.Cm <subcommand>
	.Op Ar <args>
	.Sh DESCRIPTION
	The
	.Nm
	command configures ZFS storage pools.
	A storage pool is a collection of devices that provides physical storage and
	data replication for ZFS datasets.
	All datasets within a storage pool share the same space.
	See
	.Xr zfs 8
	for information on managing datasets.
	.Pp
	For an overview of creating and managing ZFS storage pools see the
	.Xr zpoolconcepts 8
	manual page.
	.Sh SUBCOMMANDS
	All subcommands that modify state are logged persistently to the pool in their
	original form.
	.Pp
	The
	.Nm
	command provides subcommands to create and destroy storage pools, add capacity
	to storage pools, and provide information about the storage pools.
	The following subcommands are supported:
	.Bl -tag -width Ds
	.It Xo
	.Nm
	.Fl ?
	.Xc
	Displays a help message.
	.It Xo
	.Nm
	.Fl V, -version
	.Xc
	An alias for the
	.Nm zpool Cm version
	subcommand.
	.It Xo
	.Nm
	.Cm version
	.Xc
	Displays the software version of the
	.Nm
	userland utility and the zfs kernel module.
	.El
	.Ss Creation
	.Bl -tag -width Ds
	.It Xr zpool-create 8
	Creates a new storage pool containing the virtual devices specified on the
	command line.
	.It Xr zpool-initialize 8
	Begins initializing by writing to all unallocated regions on the specified
	devices, or all eligible devices in the pool if no individual devices are
	specified.
	.El
	.Ss Destruction
	.Bl -tag -width Ds
	.It Xr zpool-destroy 8
	Destroys the given pool, freeing up any devices for other use.
	.It Xr zpool-labelclear 8
	Removes ZFS label information from the specified
	.Ar device .
	.El
	.Ss Virtual Devices
	.Bl -tag -width Ds
	.It Xo
	.Xr zpool-attach 8 /
	.Xr zpool-detach 8
	.Xc
	Increases or decreases redundancy by
	.Cm attach Ns -ing or
	.Cm detach Ns -ing a device on an existing vdev (virtual device).
	.It Xo
	.Xr zpool-add 8 /
	.Xr zpool-remove 8
	.Xc
	Adds the specified virtual devices to the given pool,
	or removes the specified device from the pool.
	.It Xr zpool-replace 8
	Replaces an existing device (which may be faulted) with a new one.
	.It Xr zpool-split 8
	Creates a new pool by splitting all mirrors in an existing pool (which decreases its redundancy).
	.El
	.Ss Properties
	Available pool properties listed in the
	.Xr zpoolprops 8
	manual page.
	.Bl -tag -width Ds
	.It Xr zpool-list 8
	Lists the given pools along with a health status and space usage.
	.It Xo
	.Xr zpool-get 8 /
	.Xr zpool-set 8
	.Xc
	Retrieves the given list of properties
	.Po
	or all properties if
	.Sy all
	is used
	.Pc
	for the specified storage pool(s).
	.El
	.Ss Monitoring
	.Bl -tag -width Ds
	.It Xr zpool-status 8
	Displays the detailed health status for the given pools.
	.It Xr zpool-iostat 8
	Displays logical I/O statistics for the given pools/vdevs. Physical I/Os may
	be observed via
	.Xr iostat 1 .
	.It Xr zpool-events 8
	Lists all recent events generated by the ZFS kernel modules. These events
	are consumed by the
	.Xr zed 8
	and used to automate administrative tasks such as replacing a failed device
	with a hot spare. For more information about the subclasses and event payloads
	that can be generated see the
	.Xr zfs-events 5
	man page.
	.It Xr zpool-history 8
	Displays the command history of the specified pool(s) or all pools if no pool is
	specified.
	.El
	.Ss Maintenance
	.Bl -tag -width Ds
	.It Xr zpool-scrub 8
	Begins a scrub or resumes a paused scrub.
	.It Xr zpool-checkpoint 8
	Checkpoints the current state of
	.Ar pool
	, which can be later restored by
	.Nm zpool Cm import --rewind-to-checkpoint .
	.It Xr zpool-trim 8
	Initiates an immediate on-demand TRIM operation for all of the free space in
	a pool. This operation informs the underlying storage devices of all blocks
	in the pool which are no longer allocated and allows thinly provisioned
	devices to reclaim the space.
	.It Xr zpool-sync 8
	This command forces all in-core dirty data to be written to the primary
	pool storage and not the ZIL. It will also update administrative
	information including quota reporting. Without arguments,
	.Sy zpool sync
	will sync all pools on the system. Otherwise, it will sync only the
	specified pool(s).
	.It Xr zpool-upgrade 8
	Manage the on-disk format version of storage pools.
	.It Xr zpool-wait 8
	Waits until all background activity of the given types has ceased in the given
	pool.
	.El
	.Ss Fault Resolution
	.Bl -tag -width Ds
	.It Xo
	.Xr zpool-offline 8
	.Xr zpool-online 8
	.Xc
	Takes the specified physical device offline or brings it online.
	.It Xr zpool-resilver 8
	Starts a resilver. If an existing resilver is already running it will be
	restarted from the beginning.
	.It Xr zpool-reopen 8
	Reopen all the vdevs associated with the pool.
	.It Xr zpool-clear 8
	Clears device errors in a pool.
	.El
	.Ss Import & Export
	.Bl -tag -width Ds
	.It Xr zpool-import 8
	Make disks containing ZFS storage pools available for use on the system.
	.It Xr zpool-export 8
	Exports the given pools from the system.
	.It Xr zpool-reguid 8
	Generates a new unique identifier for the pool.
	.El
	.Sh EXIT STATUS
	The following exit values are returned:
	.Bl -tag -width Ds
	.It Sy 0
	Successful completion.
	.It Sy 1
	An error occurred.
	.It Sy 2
	Invalid command line options were specified.
	.El
	.Sh EXAMPLES
	.Bl -tag -width Ds
	.It Sy Example 1 No Creating a RAID-Z Storage Pool
	The following command creates a pool with a single raidz root vdev that
	consists of six disks.
	.Bd -literal
	# zpool create tank raidz sda sdb sdc sdd sde sdf
	.Ed
	.It Sy Example 2 No Creating a Mirrored Storage Pool
	The following command creates a pool with two mirrors, where each mirror
	contains two disks.
	.Bd -literal
	# zpool create tank mirror sda sdb mirror sdc sdd
	.Ed
	.It Sy Example 3 No Creating a ZFS Storage Pool by Using Partitions
	The following command creates an unmirrored pool using two disk partitions.
	.Bd -literal
	# zpool create tank sda1 sdb2
	.Ed
	.It Sy Example 4 No Creating a ZFS Storage Pool by Using Files
	The following command creates an unmirrored pool using files.
	While not recommended, a pool based on files can be useful for experimental
	purposes.
	.Bd -literal
	# zpool create tank /path/to/file/a /path/to/file/b
	.Ed
	.It Sy Example 5 No Adding a Mirror to a ZFS Storage Pool
	The following command adds two mirrored disks to the pool
	.Em tank ,
	assuming the pool is already made up of two-way mirrors.
	The additional space is immediately available to any datasets within the pool.
	.Bd -literal
	# zpool add tank mirror sda sdb
	.Ed
	.It Sy Example 6 No Listing Available ZFS Storage Pools
	The following command lists all available pools on the system.
	In this case, the pool
	.Em zion
	is faulted due to a missing device.
	The results from this command are similar to the following:
	.Bd -literal
	# zpool list
	NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
	rpool 19.9G 8.43G 11.4G - 33% 42% 1.00x ONLINE -
	tank 61.5G 20.0G 41.5G - 48% 32% 1.00x ONLINE -
	zion - - - - - - - FAULTED -
	.Ed
	.It Sy Example 7 No Destroying a ZFS Storage Pool
	The following command destroys the pool
	.Em tank
	and any datasets contained within.
	.Bd -literal
	# zpool destroy -f tank
	.Ed
	.It Sy Example 8 No Exporting a ZFS Storage Pool
	The following command exports the devices in pool
	.Em tank
	so that they can be relocated or later imported.
	.Bd -literal
	# zpool export tank
	.Ed
	.It Sy Example 9 No Importing a ZFS Storage Pool
	The following command displays available pools, and then imports the pool
	.Em tank
	for use on the system.
	The results from this command are similar to the following:
	.Bd -literal
	# zpool import
	pool: tank
	id: 15451357997522795478
	state: ONLINE
	action: The pool can be imported using its name or numeric identifier.
	config:

	tank ONLINE
	mirror ONLINE
	sda ONLINE
	sdb ONLINE

	# zpool import tank
	.Ed
	.It Sy Example 10 No Upgrading All ZFS Storage Pools to the Current Version
	The following command upgrades all ZFS Storage pools to the current version of
	the software.
	.Bd -literal
	# zpool upgrade -a
	This system is currently running ZFS version 2.
	.Ed
	.It Sy Example 11 No Managing Hot Spares
	The following command creates a new pool with an available hot spare:
	.Bd -literal
	# zpool create tank mirror sda sdb spare sdc
	.Ed
	.Pp
	If one of the disks were to fail, the pool would be reduced to the degraded
	state.
	The failed device can be replaced using the following command:
	.Bd -literal
	# zpool replace tank sda sdd
	.Ed
	.Pp
	Once the data has been resilvered, the spare is automatically removed and is
	made available for use should another device fail.
	The hot spare can be permanently removed from the pool using the following
	command:
	.Bd -literal
	# zpool remove tank sdc
	.Ed
	.It Sy Example 12 No Creating a ZFS Pool with Mirrored Separate Intent Logs
	The following command creates a ZFS storage pool consisting of two, two-way
	mirrors and mirrored log devices:
	.Bd -literal
	# zpool create pool mirror sda sdb mirror sdc sdd log mirror \\
	sde sdf
	.Ed
	.It Sy Example 13 No Adding Cache Devices to a ZFS Pool
	The following command adds two disks for use as cache devices to a ZFS storage
	pool:
	.Bd -literal
	# zpool add pool cache sdc sdd
	.Ed
	.Pp
	Once added, the cache devices gradually fill with content from main memory.
	Depending on the size of your cache devices, it could take over an hour for
	them to fill.
	Capacity and reads can be monitored using the
	.Cm iostat
	option as follows:
	.Bd -literal
	# zpool iostat -v pool 5
	.Ed
	.It Sy Example 14 No Removing a Mirrored top-level (Log or Data) Device
	The following commands remove the mirrored log device
	.Sy mirror-2
	and mirrored top-level data device
	.Sy mirror-1 .
	.Pp
	Given this configuration:
	.Bd -literal
	pool: tank
	state: ONLINE
	scrub: none requested
	config:

	NAME STATE READ WRITE CKSUM
	tank ONLINE 0 0 0
	mirror-0 ONLINE 0 0 0
	sda ONLINE 0 0 0
	sdb ONLINE 0 0 0
	mirror-1 ONLINE 0 0 0
	sdc ONLINE 0 0 0
	sdd ONLINE 0 0 0
	logs
	mirror-2 ONLINE 0 0 0
	sde ONLINE 0 0 0
	sdf ONLINE 0 0 0
	.Ed
	.Pp
	The command to remove the mirrored log
	.Sy mirror-2
	is:
	.Bd -literal
	# zpool remove tank mirror-2
	.Ed
	.Pp
	The command to remove the mirrored data
	.Sy mirror-1
	is:
	.Bd -literal
	# zpool remove tank mirror-1
	.Ed
	.It Sy Example 15 No Displaying expanded space on a device
	The following command displays the detailed information for the pool
	.Em data .
	This pool is comprised of a single raidz vdev where one of its devices
	increased its capacity by 10GB.
	In this example, the pool will not be able to utilize this extra capacity until
	all the devices under the raidz vdev have been expanded.
	.Bd -literal
	# zpool list -v data
	NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
	data 23.9G 14.6G 9.30G - 48% 61% 1.00x ONLINE -
	raidz1 23.9G 14.6G 9.30G - 48%
	sda - - - - -
	sdb - - - 10G -
	sdc - - - - -
	.Ed
	.It Sy Example 16 No Adding output columns
	Additional columns can be added to the
	.Nm zpool Cm status
	and
	.Nm zpool Cm iostat
	output with
	.Fl c
	option.
	.Bd -literal
	# zpool status -c vendor,model,size
	NAME STATE READ WRITE CKSUM vendor model size
	tank ONLINE 0 0 0
	mirror-0 ONLINE 0 0 0
	U1 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T
	U10 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T
	U11 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T
	U12 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T
	U13 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T
	U14 ONLINE 0 0 0 SEAGATE ST8000NM0075 7.3T

	# zpool iostat -vc size
	capacity operations bandwidth
	pool alloc free read write read write size
	---------- ----- ----- ----- ----- ----- ----- ----
	rpool 14.6G 54.9G 4 55 250K 2.69M
	sda1 14.6G 54.9G 4 55 250K 2.69M 70G
	---------- ----- ----- ----- ----- ----- ----- ----
	.Ed
	.El
	.Sh ENVIRONMENT VARIABLES
	.Bl -tag -width "ZFS_ABORT"
	.It Ev ZFS_ABORT
	Cause
	.Nm zpool
	to dump core on exit for the purposes of running
	.Sy ::findleaks .
	.El
	.Bl -tag -width "ZFS_COLOR"
	.It Ev ZFS_COLOR
	Use ANSI color in
	.Nm zpool status
	output.
	.El
	.Bl -tag -width "ZPOOL_IMPORT_PATH"
	.It Ev ZPOOL_IMPORT_PATH
	The search path for devices or files to use with the pool. This is a colon-separated list of directories in which
	.Nm zpool
	looks for device nodes and files.
	Similar to the
	.Fl d
	option in
	.Nm zpool import .
	.El
	.Bl -tag -width "ZPOOL_IMPORT_UDEV_TIMEOUT_MS"
	.It Ev ZPOOL_IMPORT_UDEV_TIMEOUT_MS
	The maximum time in milliseconds that
	.Nm zpool import
	will wait for an expected device to be available.
	.El
	.Bl -tag -width "ZPOOL_STATUS_NON_NATIVE_ASHIFT_IGNORE"
	.It Ev ZPOOL_STATUS_NON_NATIVE_ASHIFT_IGNORE
	If set, suppress warning about non-native vdev ashift in
	.Nm zpool status .
	The value is not used, only the presence or absence of the variable matters.
	.El
	.Bl -tag -width "ZPOOL_VDEV_NAME_GUID"
	.It Ev ZPOOL_VDEV_NAME_GUID
	Cause
	.Nm zpool
	subcommands to output vdev guids by default. This behavior is identical to the
	.Nm zpool status -g
	command line option.
	.El
	.Bl -tag -width "ZPOOL_VDEV_NAME_FOLLOW_LINKS"
	.It Ev ZPOOL_VDEV_NAME_FOLLOW_LINKS
	Cause
	.Nm zpool
	subcommands to follow links for vdev names by default. This behavior is identical to the
	.Nm zpool status -L
	command line option.
	.El
	.Bl -tag -width "ZPOOL_VDEV_NAME_PATH"
	.It Ev ZPOOL_VDEV_NAME_PATH
	Cause
	.Nm zpool
	subcommands to output full vdev path names by default. This
	behavior is identical to the
	.Nm zpool status -P
	command line option.
	.El
	.Bl -tag -width "ZFS_VDEV_DEVID_OPT_OUT"
	.It Ev ZFS_VDEV_DEVID_OPT_OUT
	-Older ZFS on Linux implementations had issues when attempting to display pool
	+Older OpenZFS implementations had issues when attempting to display pool
	config VDEV names if a
	.Sy devid
	NVP value is present in the pool's config.
	.Pp
	For example, a pool that originated on illumos platform would have a devid
	value in the config and
	.Nm zpool status
	would fail when listing the config.
	This would also be true for future Linux based pools.
	.Pp
	A pool can be stripped of any
	.Sy devid
	values on import or prevented from adding
	them on
	.Nm zpool create
	or
	.Nm zpool add
	by setting
	.Sy ZFS_VDEV_DEVID_OPT_OUT .
	.El
	.Bl -tag -width "ZPOOL_SCRIPTS_AS_ROOT"
	.It Ev ZPOOL_SCRIPTS_AS_ROOT
	Allow a privileged user to run the
	.Nm zpool status/iostat
	with the
	.Fl c
	option. Normally, only unprivileged users are allowed to run
	.Fl c .
	.El
	.Bl -tag -width "ZPOOL_SCRIPTS_PATH"
	.It Ev ZPOOL_SCRIPTS_PATH
	The search path for scripts when running
	.Nm zpool status/iostat
	with the
	.Fl c
	option. This is a colon-separated list of directories and overrides the default
	.Pa ~/.zpool.d
	and
	.Pa /etc/zfs/zpool.d
	search paths.
	.El
	.Bl -tag -width "ZPOOL_SCRIPTS_ENABLED"
	.It Ev ZPOOL_SCRIPTS_ENABLED
	Allow a user to run
	.Nm zpool status/iostat
	with the
	.Fl c
	option. If
	.Sy ZPOOL_SCRIPTS_ENABLED
	is not set, it is assumed that the user is allowed to run
	.Nm zpool status/iostat -c .
	.El
	.Sh INTERFACE STABILITY
	.Sy Evolving
	.Sh SEE ALSO
	.Xr zfs-events 5 ,
	.Xr zfs-module-parameters 5 ,
	.Xr zpool-features 5 ,
	.Xr zed 8 ,
	.Xr zfs 8 ,
	.Xr zpool-add 8 ,
	.Xr zpool-attach 8 ,
	.Xr zpool-checkpoint 8 ,
	.Xr zpool-clear 8 ,
	.Xr zpool-create 8 ,
	.Xr zpool-destroy 8 ,
	.Xr zpool-detach 8 ,
	.Xr zpool-events 8 ,
	.Xr zpool-export 8 ,
	.Xr zpool-get 8 ,
	.Xr zpool-history 8 ,
	.Xr zpool-import 8 ,
	.Xr zpool-initialize 8 ,
	.Xr zpool-iostat 8 ,
	.Xr zpool-labelclear 8 ,
	.Xr zpool-list 8 ,
	.Xr zpool-offline 8 ,
	.Xr zpool-online 8 ,
	.Xr zpool-reguid 8 ,
	.Xr zpool-remove 8 ,
	.Xr zpool-reopen 8 ,
	.Xr zpool-replace 8 ,
	.Xr zpool-resilver 8 ,
	.Xr zpool-scrub 8 ,
	.Xr zpool-set 8 ,
	.Xr zpool-split 8 ,
	.Xr zpool-status 8 ,
	.Xr zpool-sync 8 ,
	.Xr zpool-trim 8 ,
	.Xr zpool-upgrade 8 ,
	.Xr zpool-wait 8 ,
	.Xr zpoolconcepts 8 ,
	.Xr zpoolprops 8
	Index: vendor-sys/openzfs/dist/module/Makefile.in
	===================================================================
	--- vendor-sys/openzfs/dist/module/Makefile.in (revision 366773)
	+++ vendor-sys/openzfs/dist/module/Makefile.in (revision 366774)
	@@ -1,115 +1,115 @@
	include Kbuild

	INSTALL_MOD_DIR ?= extra

	SUBDIR_TARGETS = icp lua zstd

	all: modules
	distclean maintainer-clean: clean
	install: modules_install
	uninstall: modules_uninstall
	check:

	.PHONY: all distclean maintainer-clean install uninstall check distdir \
	modules modules-Linux modules-FreeBSD modules-unknown \
	clean clean-Linux clean-FreeBSD \
	modules_install modules_install-Linux modules_install-FreeBSD \
	modules_uninstall modules_uninstall-Linux modules_uninstall-FreeBSD

	# Filter out options that FreeBSD make doesn't understand
	getflags = ( \
	set -- \
	$(filter-out --%,$(firstword $(MFLAGS))) \
	$(filter -I%,$(MFLAGS)) \
	$(filter -j%,$(MFLAGS)); \
	fmakeflags=""; \
	while getopts :deiI:j:knqrstw flag; do \
	case $$flag in \
	\?) :;; \
	:) if [ $$OPTARG = "j" ]; then \
	ncpus=$$(sysctl -n kern.smp.cpus 2>/dev/null \|\| :); \
	if [ -n "$$ncpus" ]; then fmakeflags="$$fmakeflags -j$$ncpus"; fi; \
	fi;; \
	d) fmakeflags="$$fmakeflags -dA";; \
	*) fmakeflags="$$fmakeflags -$$flag$$OPTARG";; \
	esac; \
	done; \
	echo $$fmakeflags \
	)
	FMAKEFLAGS = -C @abs_srcdir@ -f Makefile.bsd $(shell $(getflags))

	ifneq (@abs_srcdir@,@abs_builddir@)
	FMAKEFLAGS += MAKEOBJDIR=@abs_builddir@
	endif
	FMAKE = env -u MAKEFLAGS make $(FMAKEFLAGS)

	modules-Linux:
	list='$(SUBDIR_TARGETS)'; for targetdir in $$list; do \
	$(MAKE) -C $$targetdir; \
	done
	$(MAKE) -C @LINUX_OBJ@ M=`pwd` @KERNEL_MAKE@ CONFIG_ZFS=m modules

	modules-FreeBSD:
	+$(FMAKE)

	modules-unknown:
	@true

	modules: modules-@ac_system@

	clean-Linux:
	@# Only cleanup the kernel build directories when CONFIG_KERNEL
	@# is defined. This indicates that kernel modules should be built.
	@CONFIG_KERNEL_TRUE@ $(MAKE) -C @LINUX_OBJ@ M=`pwd` @KERNEL_MAKE@ clean

	if [ -f @LINUX_SYMBOLS@ ]; then $(RM) @LINUX_SYMBOLS@; fi
	if [ -f Module.markers ]; then $(RM) Module.markers; fi

	find . -name '*.ur-safe' -type f -print \| xargs $(RM)

	clean-FreeBSD:
	+$(FMAKE) clean

	clean: clean-@ac_system@

	modules_install-Linux:
	@# Install the kernel modules
	$(MAKE) -C @LINUX_OBJ@ M=`pwd` modules_install \
	INSTALL_MOD_PATH=$(DESTDIR)$(INSTALL_MOD_PATH) \
	INSTALL_MOD_DIR=$(INSTALL_MOD_DIR) \
	KERNELRELEASE=@LINUX_VERSION@
	@# Remove extraneous build products when packaging
	kmoddir=$(DESTDIR)$(INSTALL_MOD_PATH)/lib/modules/@LINUX_VERSION@; \
	if [ -n "$(DESTDIR)" ]; then \
	find $$kmoddir -name 'modules.*' \| xargs $(RM); \
	fi
	sysmap=$(DESTDIR)$(INSTALL_MOD_PATH)/boot/System.map-@LINUX_VERSION@; \
	if [ -f $$sysmap ]; then \
	depmod -ae -F $$sysmap @LINUX_VERSION@; \
	fi

	modules_install-FreeBSD:
	@# Install the kernel modules
	+$(FMAKE) install

	modules_install: modules_install-@ac_system@

	modules_uninstall-Linux:
	@# Uninstall the kernel modules
	- kmoddir=$(DESTDIR)$(INSTALL_MOD_PATH)/lib/modules/@LINUX_VERSION@ \
	+ kmoddir=$(DESTDIR)$(INSTALL_MOD_PATH)/lib/modules/@LINUX_VERSION@; \
	for objdir in $(ZFS_MODULES); do \
	$(RM) -R $$kmoddir/$(INSTALL_MOD_DIR)/$$objdir; \
	done

	modules_uninstall-FreeBSD:
	@false

	modules_uninstall: modules_uninstall-@ac_system@

	distdir:
	(cd @srcdir@ && find $(ZFS_MODULES) os -name '*.[chS]') \| \
	while read path; do \
	mkdir -p $$distdir/$${path%/*}; \
	cp @srcdir@/$$path $$distdir/$$path; \
	done; \
	cp @srcdir@/Makefile.bsd $$distdir/Makefile.bsd
	Index: vendor-sys/openzfs/dist/module/os/freebsd/spl/list.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/freebsd/spl/list.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/freebsd/spl/list.c (revision 366774)
	@@ -1,244 +1,244 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright 2008 Sun Microsystems, Inc. All rights reserved.
	* Use is subject to license terms.
	*/

	/*
	* Generic doubly-linked list implementation
	*/

	+#include <sys/param.h>
	#include <sys/list.h>
	#include <sys/list_impl.h>
	#include <sys/types.h>
	-#include <sys/sysmacros.h>
	#include <sys/debug.h>

	#define list_d2l(a, obj) ((list_node_t )(((char )obj) + (a)->list_offset))
	#define list_object(a, node) ((void )(((char )node) - (a)->list_offset))
	#define list_empty(a) ((a)->list_head.list_next == &(a)->list_head)

	#define list_insert_after_node(list, node, object) { \
	list_node_t *lnew = list_d2l(list, object); \
	lnew->list_prev = (node); \
	lnew->list_next = (node)->list_next; \
	(node)->list_next->list_prev = lnew; \
	(node)->list_next = lnew; \
	}

	#define list_insert_before_node(list, node, object) { \
	list_node_t *lnew = list_d2l(list, object); \
	lnew->list_next = (node); \
	lnew->list_prev = (node)->list_prev; \
	(node)->list_prev->list_next = lnew; \
	(node)->list_prev = lnew; \
	}

	#define list_remove_node(node) \
	(node)->list_prev->list_next = (node)->list_next; \
	(node)->list_next->list_prev = (node)->list_prev; \
	(node)->list_next = (node)->list_prev = NULL

	void
	list_create(list_t *list, size_t size, size_t offset)
	{
	ASSERT(list);
	ASSERT(size > 0);
	ASSERT(size >= offset + sizeof (list_node_t));

	list->list_size = size;
	list->list_offset = offset;
	list->list_head.list_next = list->list_head.list_prev =
	&list->list_head;
	}

	void
	list_destroy(list_t *list)
	{
	list_node_t *node = &list->list_head;

	ASSERT(list);
	ASSERT(list->list_head.list_next == node);
	ASSERT(list->list_head.list_prev == node);

	node->list_next = node->list_prev = NULL;
	}

	void
	list_insert_after(list_t list, void object, void *nobject)
	{
	if (object == NULL) {
	list_insert_head(list, nobject);
	} else {
	list_node_t *lold = list_d2l(list, object);
	list_insert_after_node(list, lold, nobject);
	}
	}

	void
	list_insert_before(list_t list, void object, void *nobject)
	{
	if (object == NULL) {
	list_insert_tail(list, nobject);
	} else {
	list_node_t *lold = list_d2l(list, object);
	list_insert_before_node(list, lold, nobject);
	}
	}

	void
	list_insert_head(list_t list, void object)
	{
	list_node_t *lold = &list->list_head;
	list_insert_after_node(list, lold, object);
	}

	void
	list_insert_tail(list_t list, void object)
	{
	list_node_t *lold = &list->list_head;
	list_insert_before_node(list, lold, object);
	}

	void
	list_remove(list_t list, void object)
	{
	list_node_t *lold = list_d2l(list, object);
	ASSERT(!list_empty(list));
	ASSERT(lold->list_next != NULL);
	list_remove_node(lold);
	}

	void *
	list_remove_head(list_t *list)
	{
	list_node_t *head = list->list_head.list_next;
	if (head == &list->list_head)
	return (NULL);
	list_remove_node(head);
	return (list_object(list, head));
	}

	void *
	list_remove_tail(list_t *list)
	{
	list_node_t *tail = list->list_head.list_prev;
	if (tail == &list->list_head)
	return (NULL);
	list_remove_node(tail);
	return (list_object(list, tail));
	}

	void *
	list_head(list_t *list)
	{
	if (list_empty(list))
	return (NULL);
	return (list_object(list, list->list_head.list_next));
	}

	void *
	list_tail(list_t *list)
	{
	if (list_empty(list))
	return (NULL);
	return (list_object(list, list->list_head.list_prev));
	}

	void *
	list_next(list_t list, void object)
	{
	list_node_t *node = list_d2l(list, object);

	if (node->list_next != &list->list_head)
	return (list_object(list, node->list_next));

	return (NULL);
	}

	void *
	list_prev(list_t list, void object)
	{
	list_node_t *node = list_d2l(list, object);

	if (node->list_prev != &list->list_head)
	return (list_object(list, node->list_prev));

	return (NULL);
	}

	/*
	* Insert src list after dst list. Empty src list thereafter.
	*/
	void
	list_move_tail(list_t dst, list_t src)
	{
	list_node_t *dstnode = &dst->list_head;
	list_node_t *srcnode = &src->list_head;

	ASSERT(dst->list_size == src->list_size);
	ASSERT(dst->list_offset == src->list_offset);

	if (list_empty(src))
	return;

	dstnode->list_prev->list_next = srcnode->list_next;
	srcnode->list_next->list_prev = dstnode->list_prev;
	dstnode->list_prev = srcnode->list_prev;
	srcnode->list_prev->list_next = dstnode;

	/* empty src list */
	srcnode->list_next = srcnode->list_prev = srcnode;
	}

	void
	list_link_replace(list_node_t lold, list_node_t lnew)
	{
	ASSERT(list_link_active(lold));
	ASSERT(!list_link_active(lnew));

	lnew->list_next = lold->list_next;
	lnew->list_prev = lold->list_prev;
	lold->list_prev->list_next = lnew;
	lold->list_next->list_prev = lnew;
	lold->list_next = lold->list_prev = NULL;
	}

	void
	list_link_init(list_node_t *link)
	{
	link->list_next = NULL;
	link->list_prev = NULL;
	}

	int
	list_link_active(list_node_t *link)
	{
	EQUIV(link->list_next == NULL, link->list_prev == NULL);
	return (link->list_next != NULL);
	}

	int
	list_is_empty(list_t *list)
	{
	return (list_empty(list));
	}
	Index: vendor-sys/openzfs/dist/module/os/freebsd/spl/spl_kstat.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/freebsd/spl/spl_kstat.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/freebsd/spl/spl_kstat.c (revision 366774)
	@@ -1,574 +1,575 @@
	/*
	* Copyright (c) 2007 Pawel Jakub Dawidek <pjd@FreeBSD.org>
	* All rights reserved.
	*
	* Redistribution and use in source and binary forms, with or without
	* modification, are permitted provided that the following conditions
	* are met:
	* 1. Redistributions of source code must retain the above copyright
	* notice, this list of conditions and the following disclaimer.
	* 2. Redistributions in binary form must reproduce the above copyright
	* notice, this list of conditions and the following disclaimer in the
	* documentation and/or other materials provided with the distribution.
	*
	* THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
	* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
	* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
	* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
	* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
	* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
	* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
	* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
	* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
	* SUCH DAMAGE.
	*
	* Links to Illumos.org for more information on kstat function:
	* [1] https://illumos.org/man/1M/kstat
	* [2] https://illumos.org/man/9f/kstat_create
	*/

	#include <sys/cdefs.h>
	__FBSDID("$FreeBSD$");

	#include <sys/types.h>
	#include <sys/param.h>
	#include <sys/kernel.h>
	#include <sys/systm.h>
	#include <sys/malloc.h>
	#include <sys/sysctl.h>
	#include <sys/kstat.h>
	#include <sys/sbuf.h>

	static MALLOC_DEFINE(M_KSTAT, "kstat_data", "Kernel statistics");

	SYSCTL_ROOT_NODE(OID_AUTO, kstat, CTLFLAG_RW, 0, "Kernel statistics");

	void
	__kstat_set_raw_ops(kstat_t *ksp,
	int (headers)(char buf, size_t size),
	int (data)(char buf, size_t size, void *data),
	void (addr)(kstat_t *ksp, loff_t index))
	{
	ksp->ks_raw_ops.headers = headers;
	ksp->ks_raw_ops.data = data;
	ksp->ks_raw_ops.addr = addr;
	}

	void
	__kstat_set_seq_raw_ops(kstat_t *ksp,
	int (headers)(struct seq_file f),
	int (data)(char buf, size_t size, void *data),
	void (addr)(kstat_t *ksp, loff_t index))
	{
	ksp->ks_raw_ops.seq_headers = headers;
	ksp->ks_raw_ops.data = data;
	ksp->ks_raw_ops.addr = addr;
	}

	static int
	kstat_default_update(kstat_t *ksp, int rw)
	{
	ASSERT(ksp != NULL);

	if (rw == KSTAT_WRITE)
	return (EACCES);

	return (0);
	}

	static int
	kstat_resize_raw(kstat_t *ksp)
	{
	if (ksp->ks_raw_bufsize == KSTAT_RAW_MAX)
	return (ENOMEM);

	free(ksp->ks_raw_buf, M_TEMP);
	ksp->ks_raw_bufsize = MIN(ksp->ks_raw_bufsize * 2, KSTAT_RAW_MAX);
	ksp->ks_raw_buf = malloc(ksp->ks_raw_bufsize, M_TEMP, M_WAITOK);

	return (0);
	}

	static void *
	kstat_raw_default_addr(kstat_t *ksp, loff_t n)
	{
	if (n == 0)
	return (ksp->ks_data);
	return (NULL);
	}

	static int
	kstat_sysctl(SYSCTL_HANDLER_ARGS)
	{
	kstat_t *ksp = arg1;
	kstat_named_t *ksent;
	uint64_t val;

	ksent = ksp->ks_data;
	/* Select the correct element */
	ksent += arg2;
	/* Update the aggsums before reading */
	(void) ksp->ks_update(ksp, KSTAT_READ);
	val = ksent->value.ui64;

	return (sysctl_handle_64(oidp, &val, 0, req));
	}

	static int
	kstat_sysctl_string(SYSCTL_HANDLER_ARGS)
	{
	kstat_t *ksp = arg1;
	kstat_named_t *ksent = ksp->ks_data;
	char *val;
	uint32_t len = 0;

	/* Select the correct element */
	ksent += arg2;
	/* Update the aggsums before reading */
	(void) ksp->ks_update(ksp, KSTAT_READ);
	val = KSTAT_NAMED_STR_PTR(ksent);
	len = KSTAT_NAMED_STR_BUFLEN(ksent);
	val[len-1] = '\0';

	return (sysctl_handle_string(oidp, val, len, req));
	}

	static int
	kstat_sysctl_io(SYSCTL_HANDLER_ARGS)
	{
	struct sbuf *sb;
	kstat_t *ksp = arg1;
	kstat_io_t *kip = ksp->ks_data;
	int rc;

	sb = sbuf_new_auto();
	if (sb == NULL)
	return (ENOMEM);
	/* Update the aggsums before reading */
	(void) ksp->ks_update(ksp, KSTAT_READ);

	/* though wlentime & friends are signed, they will never be negative */
	sbuf_printf(sb,
	"%-8llu %-8llu %-8u %-8u %-8llu %-8llu "
	"%-8llu %-8llu %-8llu %-8llu %-8u %-8u\n",
	kip->nread, kip->nwritten,
	kip->reads, kip->writes,
	kip->wtime, kip->wlentime, kip->wlastupdate,
	kip->rtime, kip->rlentime, kip->rlastupdate,
	kip->wcnt, kip->rcnt);
	rc = sbuf_finish(sb);
	if (rc == 0)
	rc = SYSCTL_OUT(req, sbuf_data(sb), sbuf_len(sb));
	sbuf_delete(sb);
	return (rc);
	}

	static int
	kstat_sysctl_raw(SYSCTL_HANDLER_ARGS)
	{
	struct sbuf *sb;
	void *data;
	kstat_t *ksp = arg1;
	void (addr_op)(kstat_t *ksp, loff_t index);
	int n, has_header, rc = 0;

	sb = sbuf_new_auto();
	if (sb == NULL)
	return (ENOMEM);

	if (ksp->ks_raw_ops.addr)
	addr_op = ksp->ks_raw_ops.addr;
	else
	addr_op = kstat_raw_default_addr;

	mutex_enter(ksp->ks_lock);

	/* Update the aggsums before reading */
	(void) ksp->ks_update(ksp, KSTAT_READ);

	ksp->ks_raw_bufsize = PAGE_SIZE;
	ksp->ks_raw_buf = malloc(PAGE_SIZE, M_TEMP, M_WAITOK);

	n = 0;
	has_header = (ksp->ks_raw_ops.headers \|\|
	ksp->ks_raw_ops.seq_headers);

	restart_headers:
	if (ksp->ks_raw_ops.headers) {
	rc = ksp->ks_raw_ops.headers(
	ksp->ks_raw_buf, ksp->ks_raw_bufsize);
	} else if (ksp->ks_raw_ops.seq_headers) {
	struct seq_file f;

	f.sf_buf = ksp->ks_raw_buf;
	f.sf_size = ksp->ks_raw_bufsize;
	rc = ksp->ks_raw_ops.seq_headers(&f);
	}
	if (has_header) {
	if (rc == ENOMEM && !kstat_resize_raw(ksp))
	goto restart_headers;
	if (rc == 0)
	sbuf_printf(sb, "\n%s", ksp->ks_raw_buf);
	}

	while ((data = addr_op(ksp, n)) != NULL) {
	restart:
	if (ksp->ks_raw_ops.data) {
	rc = ksp->ks_raw_ops.data(ksp->ks_raw_buf,
	ksp->ks_raw_bufsize, data);
	if (rc == ENOMEM && !kstat_resize_raw(ksp))
	goto restart;
	if (rc == 0)
	sbuf_printf(sb, "%s", ksp->ks_raw_buf);

	} else {
	ASSERT(ksp->ks_ndata == 1);
	sbuf_hexdump(sb, ksp->ks_data,
	ksp->ks_data_size, NULL, 0);
	}
	n++;
	}
	free(ksp->ks_raw_buf, M_TEMP);
	mutex_exit(ksp->ks_lock);
	+ sbuf_trim(sb);
	rc = sbuf_finish(sb);
	if (rc == 0)
	rc = SYSCTL_OUT(req, sbuf_data(sb), sbuf_len(sb));
	sbuf_delete(sb);
	return (rc);
	}

	kstat_t *
	__kstat_create(const char module, int instance, const char name,
	const char *class, uchar_t ks_type, uint_t ks_ndata, uchar_t flags)
	{
	char buf[KSTAT_STRLEN];
	struct sysctl_oid *root;
	kstat_t *ksp;
	char *pool;

	KASSERT(instance == 0, ("instance=%d", instance));
	if ((ks_type == KSTAT_TYPE_INTR) \|\| (ks_type == KSTAT_TYPE_IO))
	ASSERT(ks_ndata == 1);

	if (class == NULL)
	class = "misc";

	/*
	* Allocate the main structure. We don't need to keep a copy of
	* module in here, because it is only used for sysctl node creation
	* done in this function.
	*/
	ksp = malloc(sizeof (*ksp), M_KSTAT, M_WAITOK\|M_ZERO);

	ksp->ks_crtime = gethrtime();
	ksp->ks_snaptime = ksp->ks_crtime;
	ksp->ks_instance = instance;
	(void) strlcpy(ksp->ks_name, name, KSTAT_STRLEN);
	(void) strlcpy(ksp->ks_class, class, KSTAT_STRLEN);
	ksp->ks_type = ks_type;
	ksp->ks_flags = flags;
	ksp->ks_update = kstat_default_update;

	mutex_init(&ksp->ks_private_lock, NULL, MUTEX_DEFAULT, NULL);
	ksp->ks_lock = &ksp->ks_private_lock;

	switch (ksp->ks_type) {
	case KSTAT_TYPE_RAW:
	ksp->ks_ndata = 1;
	ksp->ks_data_size = ks_ndata;
	break;
	case KSTAT_TYPE_NAMED:
	ksp->ks_ndata = ks_ndata;
	ksp->ks_data_size = ks_ndata * sizeof (kstat_named_t);
	break;
	case KSTAT_TYPE_INTR:
	ksp->ks_ndata = ks_ndata;
	ksp->ks_data_size = ks_ndata * sizeof (kstat_intr_t);
	break;
	case KSTAT_TYPE_IO:
	ksp->ks_ndata = ks_ndata;
	ksp->ks_data_size = ks_ndata * sizeof (kstat_io_t);
	break;
	case KSTAT_TYPE_TIMER:
	ksp->ks_ndata = ks_ndata;
	ksp->ks_data_size = ks_ndata * sizeof (kstat_timer_t);
	break;
	default:
	panic("Undefined kstat type %d\n", ksp->ks_type);
	}

	if (ksp->ks_flags & KSTAT_FLAG_VIRTUAL) {
	ksp->ks_data = NULL;
	} else {
	ksp->ks_data = kmem_zalloc(ksp->ks_data_size, KM_SLEEP);
	if (ksp->ks_data == NULL) {
	kmem_free(ksp, sizeof (*ksp));
	ksp = NULL;
	}
	}

	/*
	* Some kstats use a module name like "zfs/poolname" to distinguish a
	* set of kstats belonging to a specific pool. Split on '/' to add an
	* extra node for the pool name if needed.
	*/
	(void) strlcpy(buf, module, KSTAT_STRLEN);
	module = buf;
	pool = strchr(module, '/');
	if (pool != NULL)
	*pool++ = '\0';

	/*
	* Create sysctl tree for those statistics:
	*
	* kstat.<module>[.<pool>].<class>.<name>
	*/
	sysctl_ctx_init(&ksp->ks_sysctl_ctx);
	root = SYSCTL_ADD_NODE(&ksp->ks_sysctl_ctx,
	SYSCTL_STATIC_CHILDREN(_kstat), OID_AUTO, module, CTLFLAG_RW, 0,
	"");
	if (root == NULL) {
	printf("%s: Cannot create kstat.%s tree!\n", __func__, module);
	sysctl_ctx_free(&ksp->ks_sysctl_ctx);
	free(ksp, M_KSTAT);
	return (NULL);
	}
	if (pool != NULL) {
	root = SYSCTL_ADD_NODE(&ksp->ks_sysctl_ctx,
	SYSCTL_CHILDREN(root), OID_AUTO, pool, CTLFLAG_RW, 0, "");
	if (root == NULL) {
	printf("%s: Cannot create kstat.%s.%s tree!\n",
	__func__, module, pool);
	sysctl_ctx_free(&ksp->ks_sysctl_ctx);
	free(ksp, M_KSTAT);
	return (NULL);
	}
	}
	root = SYSCTL_ADD_NODE(&ksp->ks_sysctl_ctx, SYSCTL_CHILDREN(root),
	OID_AUTO, class, CTLFLAG_RW, 0, "");
	if (root == NULL) {
	if (pool != NULL)
	printf("%s: Cannot create kstat.%s.%s.%s tree!\n",
	__func__, module, pool, class);
	else
	printf("%s: Cannot create kstat.%s.%s tree!\n",
	__func__, module, class);
	sysctl_ctx_free(&ksp->ks_sysctl_ctx);
	free(ksp, M_KSTAT);
	return (NULL);
	}
	if (ksp->ks_type == KSTAT_TYPE_NAMED) {
	root = SYSCTL_ADD_NODE(&ksp->ks_sysctl_ctx,
	SYSCTL_CHILDREN(root),
	OID_AUTO, name, CTLFLAG_RW, 0, "");
	if (root == NULL) {
	if (pool != NULL)
	printf("%s: Cannot create kstat.%s.%s.%s.%s "
	"tree!\n", __func__, module, pool, class,
	name);
	else
	printf("%s: Cannot create kstat.%s.%s.%s "
	"tree!\n", __func__, module, class, name);
	sysctl_ctx_free(&ksp->ks_sysctl_ctx);
	free(ksp, M_KSTAT);
	return (NULL);
	}

	}
	ksp->ks_sysctl_root = root;

	return (ksp);
	}

	static void
	kstat_install_named(kstat_t *ksp)
	{
	kstat_named_t *ksent;
	char *namelast;
	int typelast;

	ksent = ksp->ks_data;

	VERIFY((ksp->ks_flags & KSTAT_FLAG_VIRTUAL) \|\| ksent != NULL);

	typelast = 0;
	namelast = NULL;

	for (int i = 0; i < ksp->ks_ndata; i++, ksent++) {
	if (ksent->data_type != 0) {
	typelast = ksent->data_type;
	namelast = ksent->name;
	}
	switch (typelast) {
	case KSTAT_DATA_CHAR:
	/* Not Implemented */
	break;
	case KSTAT_DATA_INT32:
	SYSCTL_ADD_PROC(&ksp->ks_sysctl_ctx,
	SYSCTL_CHILDREN(ksp->ks_sysctl_root),
	OID_AUTO, namelast,
	CTLTYPE_S32 \| CTLFLAG_RD \| CTLFLAG_MPSAFE,
	ksp, i, kstat_sysctl, "I", namelast);
	break;
	case KSTAT_DATA_UINT32:
	SYSCTL_ADD_PROC(&ksp->ks_sysctl_ctx,
	SYSCTL_CHILDREN(ksp->ks_sysctl_root),
	OID_AUTO, namelast,
	CTLTYPE_U32 \| CTLFLAG_RD \| CTLFLAG_MPSAFE,
	ksp, i, kstat_sysctl, "IU", namelast);
	break;
	case KSTAT_DATA_INT64:
	SYSCTL_ADD_PROC(&ksp->ks_sysctl_ctx,
	SYSCTL_CHILDREN(ksp->ks_sysctl_root),
	OID_AUTO, namelast,
	CTLTYPE_S64 \| CTLFLAG_RD \| CTLFLAG_MPSAFE,
	ksp, i, kstat_sysctl, "Q", namelast);
	break;
	case KSTAT_DATA_UINT64:
	SYSCTL_ADD_PROC(&ksp->ks_sysctl_ctx,
	SYSCTL_CHILDREN(ksp->ks_sysctl_root),
	OID_AUTO, namelast,
	CTLTYPE_U64 \| CTLFLAG_RD \| CTLFLAG_MPSAFE,
	ksp, i, kstat_sysctl, "QU", namelast);
	break;
	case KSTAT_DATA_LONG:
	SYSCTL_ADD_PROC(&ksp->ks_sysctl_ctx,
	SYSCTL_CHILDREN(ksp->ks_sysctl_root),
	OID_AUTO, namelast,
	CTLTYPE_LONG \| CTLFLAG_RD \| CTLFLAG_MPSAFE,
	ksp, i, kstat_sysctl, "L", namelast);
	break;
	case KSTAT_DATA_ULONG:
	SYSCTL_ADD_PROC(&ksp->ks_sysctl_ctx,
	SYSCTL_CHILDREN(ksp->ks_sysctl_root),
	OID_AUTO, namelast,
	CTLTYPE_ULONG \| CTLFLAG_RD \| CTLFLAG_MPSAFE,
	ksp, i, kstat_sysctl, "LU", namelast);
	break;
	case KSTAT_DATA_STRING:
	SYSCTL_ADD_PROC(&ksp->ks_sysctl_ctx,
	SYSCTL_CHILDREN(ksp->ks_sysctl_root),
	OID_AUTO, namelast,
	CTLTYPE_STRING \| CTLFLAG_RD \| CTLFLAG_MPSAFE,
	ksp, i, kstat_sysctl_string, "A", namelast);
	break;
	default:
	panic("unsupported type: %d", typelast);
	}
	}
	}

	void
	kstat_install(kstat_t *ksp)
	{
	struct sysctl_oid *root;

	if (ksp->ks_ndata == UINT32_MAX)
	VERIFY(ksp->ks_type == KSTAT_TYPE_RAW);

	switch (ksp->ks_type) {
	case KSTAT_TYPE_NAMED:
	return (kstat_install_named(ksp));
	case KSTAT_TYPE_RAW:
	if (ksp->ks_raw_ops.data) {
	root = SYSCTL_ADD_PROC(&ksp->ks_sysctl_ctx,
	SYSCTL_CHILDREN(ksp->ks_sysctl_root),
	OID_AUTO, ksp->ks_name,
	CTLTYPE_STRING \| CTLFLAG_RD \| CTLFLAG_MPSAFE,
	ksp, 0, kstat_sysctl_raw, "A", ksp->ks_name);
	} else {
	root = SYSCTL_ADD_PROC(&ksp->ks_sysctl_ctx,
	SYSCTL_CHILDREN(ksp->ks_sysctl_root),
	OID_AUTO, ksp->ks_name,
	CTLTYPE_OPAQUE \| CTLFLAG_RD \| CTLFLAG_MPSAFE,
	ksp, 0, kstat_sysctl_raw, "", ksp->ks_name);
	}
	break;
	case KSTAT_TYPE_IO:
	root = SYSCTL_ADD_PROC(&ksp->ks_sysctl_ctx,
	SYSCTL_CHILDREN(ksp->ks_sysctl_root),
	OID_AUTO, ksp->ks_name,
	CTLTYPE_STRING \| CTLFLAG_RD \| CTLFLAG_MPSAFE,
	ksp, 0, kstat_sysctl_io, "A", ksp->ks_name);
	break;
	case KSTAT_TYPE_TIMER:
	case KSTAT_TYPE_INTR:
	default:
	panic("unsupported kstat type %d\n", ksp->ks_type);
	}
	VERIFY(root != NULL);
	ksp->ks_sysctl_root = root;
	}

	void
	kstat_delete(kstat_t *ksp)
	{

	sysctl_ctx_free(&ksp->ks_sysctl_ctx);
	ksp->ks_lock = NULL;
	mutex_destroy(&ksp->ks_private_lock);
	free(ksp, M_KSTAT);
	}

	void
	kstat_waitq_enter(kstat_io_t *kiop)
	{
	hrtime_t new, delta;
	ulong_t wcnt;

	new = gethrtime();
	delta = new - kiop->wlastupdate;
	kiop->wlastupdate = new;
	wcnt = kiop->wcnt++;
	if (wcnt != 0) {
	kiop->wlentime += delta * wcnt;
	kiop->wtime += delta;
	}
	}

	void
	kstat_waitq_exit(kstat_io_t *kiop)
	{
	hrtime_t new, delta;
	ulong_t wcnt;

	new = gethrtime();
	delta = new - kiop->wlastupdate;
	kiop->wlastupdate = new;
	wcnt = kiop->wcnt--;
	ASSERT((int)wcnt > 0);
	kiop->wlentime += delta * wcnt;
	kiop->wtime += delta;
	}

	void
	kstat_runq_enter(kstat_io_t *kiop)
	{
	hrtime_t new, delta;
	ulong_t rcnt;

	new = gethrtime();
	delta = new - kiop->rlastupdate;
	kiop->rlastupdate = new;
	rcnt = kiop->rcnt++;
	if (rcnt != 0) {
	kiop->rlentime += delta * rcnt;
	kiop->rtime += delta;
	}
	}

	void
	kstat_runq_exit(kstat_io_t *kiop)
	{
	hrtime_t new, delta;
	ulong_t rcnt;

	new = gethrtime();
	delta = new - kiop->rlastupdate;
	kiop->rlastupdate = new;
	rcnt = kiop->rcnt--;
	ASSERT((int)rcnt > 0);
	kiop->rlentime += delta * rcnt;
	kiop->rtime += delta;
	}
	Index: vendor-sys/openzfs/dist/module/os/freebsd/spl/spl_taskq.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/freebsd/spl/spl_taskq.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/freebsd/spl/spl_taskq.c (revision 366774)
	@@ -1,413 +1,444 @@
	/*
	* Copyright (c) 2009 Pawel Jakub Dawidek <pjd@FreeBSD.org>
	* All rights reserved.
	*
	* Copyright (c) 2012 Spectra Logic Corporation. All rights reserved.
	*
	* Redistribution and use in source and binary forms, with or without
	* modification, are permitted provided that the following conditions
	* are met:
	* 1. Redistributions of source code must retain the above copyright
	* notice, this list of conditions and the following disclaimer.
	* 2. Redistributions in binary form must reproduce the above copyright
	* notice, this list of conditions and the following disclaimer in the
	* documentation and/or other materials provided with the distribution.
	*
	* THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
	* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
	* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
	* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
	* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
	* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
	* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
	* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
	* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
	* SUCH DAMAGE.
	*/

	#include <sys/cdefs.h>
	__FBSDID("$FreeBSD$");

	-#include <sys/types.h>
	#include <sys/param.h>
	+#include <sys/ck.h>
	+#include <sys/epoch.h>
	#include <sys/kernel.h>
	#include <sys/kmem.h>
	#include <sys/lock.h>
	#include <sys/mutex.h>
	#include <sys/queue.h>
	-#include <sys/taskqueue.h>
	#include <sys/taskq.h>
	+#include <sys/taskqueue.h>
	#include <sys/zfs_context.h>
	-#include <sys/ck.h>
	-#include <sys/epoch.h>

	+#if defined(__i386__) \|\| defined(__amd64__) \|\| defined(__aarch64__)
	+#include <machine/pcb.h>
	+#endif
	+
	#include <vm/uma.h>

	#if __FreeBSD_version < 1201522
	#define taskqueue_start_threads_in_proc(tqp, count, pri, proc, name, ...) \
	taskqueue_start_threads(tqp, count, pri, name, __VA_ARGS__)
	#endif

	static uint_t taskq_tsd;
	static uma_zone_t taskq_zone;

	taskq_t *system_taskq = NULL;
	taskq_t *system_delay_taskq = NULL;
	taskq_t *dynamic_taskq = NULL;

	proc_t *system_proc;

	extern int uma_align_cache;

	static MALLOC_DEFINE(M_TASKQ, "taskq", "taskq structures");

	static CK_LIST_HEAD(tqenthashhead, taskq_ent) *tqenthashtbl;
	static unsigned long tqenthash;
	static unsigned long tqenthashlock;
	static struct sx *tqenthashtbl_lock;

	-static uint32_t tqidnext = 1;
	+static taskqid_t tqidnext;

	#define TQIDHASH(tqid) (&tqenthashtbl[(tqid) & tqenthash])
	#define TQIDHASHLOCK(tqid) (&tqenthashtbl_lock[((tqid) & tqenthashlock)])

	#define TIMEOUT_TASK 1
	#define NORMAL_TASK 2

	static void
	system_taskq_init(void *arg)
	{
	int i;

	tsd_create(&taskq_tsd, NULL);
	tqenthashtbl = hashinit(mp_ncpus * 8, M_TASKQ, &tqenthash);
	tqenthashlock = (tqenthash + 1) / 8;
	if (tqenthashlock > 0)
	tqenthashlock--;
	tqenthashtbl_lock =
	malloc(sizeof (tqenthashtbl_lock) (tqenthashlock + 1),
	M_TASKQ, M_WAITOK \| M_ZERO);
	for (i = 0; i < tqenthashlock + 1; i++)
	sx_init_flags(&tqenthashtbl_lock[i], "tqenthash", SX_DUPOK);
	- tqidnext = 1;
	taskq_zone = uma_zcreate("taskq_zone", sizeof (taskq_ent_t),
	NULL, NULL, NULL, NULL,
	UMA_ALIGN_CACHE, 0);
	system_taskq = taskq_create("system_taskq", mp_ncpus, minclsyspri,
	0, 0, 0);
	system_delay_taskq = taskq_create("system_delay_taskq", mp_ncpus,
	minclsyspri, 0, 0, 0);
	}
	SYSINIT(system_taskq_init, SI_SUB_CONFIGURE, SI_ORDER_ANY, system_taskq_init,
	NULL);

	static void
	system_taskq_fini(void *arg)
	{
	int i;

	taskq_destroy(system_delay_taskq);
	taskq_destroy(system_taskq);
	uma_zdestroy(taskq_zone);
	tsd_destroy(&taskq_tsd);
	for (i = 0; i < tqenthashlock + 1; i++)
	sx_destroy(&tqenthashtbl_lock[i]);
	for (i = 0; i < tqenthash + 1; i++)
	VERIFY(CK_LIST_EMPTY(&tqenthashtbl[i]));
	free(tqenthashtbl_lock, M_TASKQ);
	free(tqenthashtbl, M_TASKQ);
	}
	SYSUNINIT(system_taskq_fini, SI_SUB_CONFIGURE, SI_ORDER_ANY, system_taskq_fini,
	NULL);

	+#ifdef __LP64__
	+static taskqid_t
	+__taskq_genid(void)
	+{
	+ taskqid_t tqid;
	+
	+ /*
	+ * Assume a 64-bit counter will not wrap in practice.
	+ */
	+ tqid = atomic_add_64_nv(&tqidnext, 1);
	+ VERIFY(tqid);
	+ return (tqid);
	+}
	+#else
	+static taskqid_t
	+__taskq_genid(void)
	+{
	+ taskqid_t tqid;
	+
	+ for (;;) {
	+ tqid = atomic_add_32_nv(&tqidnext, 1);
	+ if (__predict_true(tqid != 0))
	+ break;
	+ }
	+ VERIFY(tqid);
	+ return (tqid);
	+}
	+#endif
	+
	static taskq_ent_t *
	taskq_lookup(taskqid_t tqid)
	{
	taskq_ent_t *ent = NULL;

	sx_xlock(TQIDHASHLOCK(tqid));
	CK_LIST_FOREACH(ent, TQIDHASH(tqid), tqent_hash) {
	if (ent->tqent_id == tqid)
	break;
	}
	if (ent != NULL)
	refcount_acquire(&ent->tqent_rc);
	sx_xunlock(TQIDHASHLOCK(tqid));
	return (ent);
	}

	static taskqid_t
	taskq_insert(taskq_ent_t *ent)
	{
	- taskqid_t tqid = atomic_fetchadd_int(&tqidnext, 1);
	+ taskqid_t tqid;

	+ tqid = __taskq_genid();
	ent->tqent_id = tqid;
	ent->tqent_registered = B_TRUE;
	sx_xlock(TQIDHASHLOCK(tqid));
	CK_LIST_INSERT_HEAD(TQIDHASH(tqid), ent, tqent_hash);
	sx_xunlock(TQIDHASHLOCK(tqid));
	return (tqid);
	}

	static void
	taskq_remove(taskq_ent_t *ent)
	{
	taskqid_t tqid = ent->tqent_id;

	if (!ent->tqent_registered)
	return;

	sx_xlock(TQIDHASHLOCK(tqid));
	CK_LIST_REMOVE(ent, tqent_hash);
	sx_xunlock(TQIDHASHLOCK(tqid));
	ent->tqent_registered = B_FALSE;
	}

	static void
	taskq_tsd_set(void *context)
	{
	taskq_t *tq = context;

	#if defined(__amd64__) \|\| defined(__i386__) \|\| defined(__aarch64__)
	if (context != NULL && tsd_get(taskq_tsd) == NULL)
	fpu_kern_thread(FPU_KERN_NORMAL);
	#endif
	tsd_set(taskq_tsd, tq);
	}

	static taskq_t *
	taskq_create_impl(const char *name, int nthreads, pri_t pri,
	proc_t *proc __maybe_unused, uint_t flags)
	{
	taskq_t *tq;

	if ((flags & TASKQ_THREADS_CPU_PCT) != 0)
	nthreads = MAX((mp_ncpus * nthreads) / 100, 1);

	tq = kmem_alloc(sizeof (*tq), KM_SLEEP);
	tq->tq_queue = taskqueue_create(name, M_WAITOK,
	taskqueue_thread_enqueue, &tq->tq_queue);
	taskqueue_set_callback(tq->tq_queue, TASKQUEUE_CALLBACK_TYPE_INIT,
	taskq_tsd_set, tq);
	taskqueue_set_callback(tq->tq_queue, TASKQUEUE_CALLBACK_TYPE_SHUTDOWN,
	taskq_tsd_set, NULL);
	(void) taskqueue_start_threads_in_proc(&tq->tq_queue, nthreads, pri,
	proc, "%s", name);

	return ((taskq_t *)tq);
	}

	taskq_t *
	taskq_create(const char *name, int nthreads, pri_t pri, int minalloc __unused,
	int maxalloc __unused, uint_t flags)
	{
	return (taskq_create_impl(name, nthreads, pri, system_proc, flags));
	}

	taskq_t *
	taskq_create_proc(const char *name, int nthreads, pri_t pri,
	int minalloc __unused, int maxalloc __unused, proc_t *proc, uint_t flags)
	{
	return (taskq_create_impl(name, nthreads, pri, proc, flags));
	}

	void
	taskq_destroy(taskq_t *tq)
	{

	taskqueue_free(tq->tq_queue);
	kmem_free(tq, sizeof (*tq));
	}

	int
	taskq_member(taskq_t tq, kthread_t thread)
	{

	return (taskqueue_member(tq->tq_queue, thread));
	}

	taskq_t *
	taskq_of_curthread(void)
	{
	return (tsd_get(taskq_tsd));
	}

	static void
	taskq_free(taskq_ent_t *task)
	{
	taskq_remove(task);
	if (refcount_release(&task->tqent_rc))
	uma_zfree(taskq_zone, task);
	}

	int
	taskq_cancel_id(taskq_t *tq, taskqid_t tid)
	{
	uint32_t pend;
	int rc;
	taskq_ent_t *ent;

	if (tid == 0)
	return (0);

	if ((ent = taskq_lookup(tid)) == NULL)
	return (0);

	ent->tqent_cancelled = B_TRUE;
	if (ent->tqent_type == TIMEOUT_TASK) {
	rc = taskqueue_cancel_timeout(tq->tq_queue,
	&ent->tqent_timeout_task, &pend);
	} else
	rc = taskqueue_cancel(tq->tq_queue, &ent->tqent_task, &pend);
	if (rc == EBUSY) {
	taskqueue_drain(tq->tq_queue, &ent->tqent_task);
	} else if (pend) {
	/*
	* Tasks normally free themselves when run, but here the task
	* was cancelled so it did not free itself.
	*/
	taskq_free(ent);
	}
	/* Free the extra reference we added with taskq_lookup. */
	taskq_free(ent);
	return (rc);
	}

	static void
	taskq_run(void *arg, int pending __unused)
	{
	taskq_ent_t *task = arg;

	if (!task->tqent_cancelled)
	task->tqent_func(task->tqent_arg);
	taskq_free(task);
	}

	taskqid_t
	taskq_dispatch_delay(taskq_t tq, task_func_t func, void arg,
	uint_t flags, clock_t expire_time)
	{
	taskq_ent_t *task;
	- taskqid_t tid;
	+ taskqid_t tqid;
	clock_t timo;
	int mflag;

	timo = expire_time - ddi_get_lbolt();
	if (timo <= 0)
	return (taskq_dispatch(tq, func, arg, flags));

	if ((flags & (TQ_SLEEP \| TQ_NOQUEUE)) == TQ_SLEEP)
	mflag = M_WAITOK;
	else
	mflag = M_NOWAIT;

	task = uma_zalloc(taskq_zone, mflag);
	if (task == NULL)
	return (0);
	task->tqent_func = func;
	task->tqent_arg = arg;
	task->tqent_type = TIMEOUT_TASK;
	task->tqent_cancelled = B_FALSE;
	refcount_init(&task->tqent_rc, 1);
	- tid = taskq_insert(task);
	+ tqid = taskq_insert(task);
	TIMEOUT_TASK_INIT(tq->tq_queue, &task->tqent_timeout_task, 0,
	taskq_run, task);

	taskqueue_enqueue_timeout(tq->tq_queue, &task->tqent_timeout_task,
	timo);
	- return (tid);
	+ return (tqid);
	}

	taskqid_t
	taskq_dispatch(taskq_t tq, task_func_t func, void arg, uint_t flags)
	{
	taskq_ent_t *task;
	int mflag, prio;
	- taskqid_t tid;
	+ taskqid_t tqid;

	if ((flags & (TQ_SLEEP \| TQ_NOQUEUE)) == TQ_SLEEP)
	mflag = M_WAITOK;
	else
	mflag = M_NOWAIT;
	/*
	* If TQ_FRONT is given, we want higher priority for this task, so it
	* can go at the front of the queue.
	*/
	prio = !!(flags & TQ_FRONT);

	task = uma_zalloc(taskq_zone, mflag);
	if (task == NULL)
	return (0);
	refcount_init(&task->tqent_rc, 1);
	task->tqent_func = func;
	task->tqent_arg = arg;
	task->tqent_cancelled = B_FALSE;
	task->tqent_type = NORMAL_TASK;
	- tid = taskq_insert(task);
	+ tqid = taskq_insert(task);
	TASK_INIT(&task->tqent_task, prio, taskq_run, task);
	taskqueue_enqueue(tq->tq_queue, &task->tqent_task);
	- VERIFY(tid);
	- return (tid);
	+ return (tqid);
	}

	static void
	taskq_run_ent(void *arg, int pending __unused)
	{
	taskq_ent_t *task = arg;

	task->tqent_func(task->tqent_arg);
	}

	void
	taskq_dispatch_ent(taskq_t tq, task_func_t func, void arg, uint32_t flags,
	taskq_ent_t *task)
	{
	int prio;

	/*
	* If TQ_FRONT is given, we want higher priority for this task, so it
	* can go at the front of the queue.
	*/
	prio = !!(flags & TQ_FRONT);
	task->tqent_cancelled = B_FALSE;
	task->tqent_registered = B_FALSE;
	task->tqent_id = 0;
	task->tqent_func = func;
	task->tqent_arg = arg;

	TASK_INIT(&task->tqent_task, prio, taskq_run_ent, task);
	taskqueue_enqueue(tq->tq_queue, &task->tqent_task);
	}

	void
	taskq_wait(taskq_t *tq)
	{
	taskqueue_quiesce(tq->tq_queue);
	}

	void
	taskq_wait_id(taskq_t *tq, taskqid_t tid)
	{
	taskq_ent_t *ent;

	if (tid == 0)
	return;
	if ((ent = taskq_lookup(tid)) == NULL)
	return;

	taskqueue_drain(tq->tq_queue, &ent->tqent_task);
	taskq_free(ent);
	}

	void
	taskq_wait_outstanding(taskq_t *tq, taskqid_t id __unused)
	{
	taskqueue_drain_all(tq->tq_queue);
	}

	int
	taskq_empty_ent(taskq_ent_t *t)
	{
	return (t->tqent_task.ta_pending == 0);
	}
	Index: vendor-sys/openzfs/dist/module/os/freebsd/zfs/kmod_core.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/freebsd/zfs/kmod_core.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/freebsd/zfs/kmod_core.c (revision 366774)
	@@ -1,382 +1,375 @@
	/*
	* Copyright (c) 2020 iXsystems, Inc.
	* All rights reserved.
	*
	* Redistribution and use in source and binary forms, with or without
	* modification, are permitted provided that the following conditions
	* are met:
	* 1. Redistributions of source code must retain the above copyright
	* notice, this list of conditions and the following disclaimer.
	* 2. Redistributions in binary form must reproduce the above copyright
	* notice, this list of conditions and the following disclaimer in the
	* documentation and/or other materials provided with the distribution.
	*
	* THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
	* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
	* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
	* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
	* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
	* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
	* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
	* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
	* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
	* SUCH DAMAGE.
	*
	*/

	#include <sys/cdefs.h>
	__FBSDID("$FreeBSD$");

	-#include <sys/types.h>
	#include <sys/param.h>
	-#include <sys/systm.h>
	+#include <sys/buf.h>
	+#include <sys/cmn_err.h>
	#include <sys/conf.h>
	+#include <sys/dmu.h>
	+#include <sys/dmu_impl.h>
	+#include <sys/dmu_objset.h>
	+#include <sys/dmu_send.h>
	+#include <sys/dmu_tx.h>
	+#include <sys/dsl_bookmark.h>
	+#include <sys/dsl_crypt.h>
	+#include <sys/dsl_dataset.h>
	+#include <sys/dsl_deleg.h>
	+#include <sys/dsl_destroy.h>
	+#include <sys/dsl_dir.h>
	+#include <sys/dsl_prop.h>
	+#include <sys/dsl_scan.h>
	+#include <sys/dsl_userhold.h>
	+#include <sys/errno.h>
	#include <sys/eventhandler.h>
	+#include <sys/file.h>
	+#include <sys/fm/util.h>
	+#include <sys/fs/zfs.h>
	#include <sys/kernel.h>
	+#include <sys/kmem.h>
	#include <sys/lock.h>
	#include <sys/malloc.h>
	+#include <sys/mount.h>
	#include <sys/mutex.h>
	+#include <sys/nvpair.h>
	+#include <sys/policy.h>
	#include <sys/proc.h>
	-#include <sys/errno.h>
	-#include <sys/uio.h>
	-#include <sys/buf.h>
	-#include <sys/file.h>
	-#include <sys/kmem.h>
	-#include <sys/conf.h>
	-#include <sys/eventhandler.h>
	-#include <sys/cmn_err.h>
	-#include <sys/stat.h>
	-#include <sys/zfs_ioctl.h>
	-#include <sys/zfs_vfsops.h>
	-#include <sys/zfs_znode.h>
	-#include <sys/zap.h>
	+#include <sys/sdt.h>
	#include <sys/spa.h>
	#include <sys/spa_impl.h>
	-#include <sys/vdev.h>
	-#include <sys/dmu.h>
	-#include <sys/dsl_dir.h>
	-#include <sys/dsl_dataset.h>
	-#include <sys/dsl_prop.h>
	-#include <sys/dsl_deleg.h>
	-#include <sys/dmu_objset.h>
	-#include <sys/dmu_impl.h>
	-#include <sys/dmu_tx.h>
	-#include <sys/fm/util.h>
	+#include <sys/stat.h>
	#include <sys/sunddi.h>
	-#include <sys/policy.h>
	-#include <sys/zone.h>
	-#include <sys/nvpair.h>
	-#include <sys/mount.h>
	+#include <sys/systm.h>
	#include <sys/taskqueue.h>
	-#include <sys/sdt.h>
	-#include <sys/fs/zfs.h>
	+#include <sys/uio.h>
	+#include <sys/vdev.h>
	+#include <sys/vdev_removal.h>
	+#include <sys/zap.h>
	+#include <sys/zcp.h>
	+#include <sys/zfeature.h>
	+#include <sys/zfs_context.h>
	#include <sys/zfs_ctldir.h>
	#include <sys/zfs_dir.h>
	+#include <sys/zfs_ioctl.h>
	+#include <sys/zfs_ioctl_compat.h>
	+#include <sys/zfs_ioctl_impl.h>
	#include <sys/zfs_onexit.h>
	-#include <sys/zvol.h>
	-#include <sys/dsl_scan.h>
	-#include <sys/dmu_objset.h>
	-#include <sys/dmu_send.h>
	-#include <sys/dsl_destroy.h>
	-#include <sys/dsl_bookmark.h>
	-#include <sys/dsl_userhold.h>
	-#include <sys/zfeature.h>
	-#include <sys/zcp.h>
	+#include <sys/zfs_vfsops.h>
	+#include <sys/zfs_znode.h>
	#include <sys/zio_checksum.h>
	-#include <sys/vdev_removal.h>
	-#include <sys/dsl_crypt.h>
	-#include <sys/zfs_context.h>
	+#include <sys/zone.h>
	+#include <sys/zvol.h>

	-#include <sys/zfs_ioctl_compat.h>
	-#include <sys/zfs_ioctl_impl.h>
	-
	+#include "zfs_comutil.h"
	+#include "zfs_deleg.h"
	#include "zfs_namecheck.h"
	#include "zfs_prop.h"
	-#include "zfs_deleg.h"
	-#include "zfs_comutil.h"

	SYSCTL_DECL(_vfs_zfs);
	SYSCTL_DECL(_vfs_zfs_vdev);

	extern uint_t rrw_tsd_key;
	static int zfs_version_ioctl = ZFS_IOCVER_OZFS;
	SYSCTL_DECL(_vfs_zfs_version);
	SYSCTL_INT(_vfs_zfs_version, OID_AUTO, ioctl, CTLFLAG_RD, &zfs_version_ioctl,
	0, "ZFS_IOCTL_VERSION");

	static struct cdev *zfsdev;

	static struct root_hold_token *zfs_root_token;

	extern uint_t rrw_tsd_key;
	extern uint_t zfs_allow_log_key;
	extern uint_t zfs_geom_probe_vdev_key;

	static int zfs__init(void);
	static int zfs__fini(void);
	static void zfs_shutdown(void *, int);

	static eventhandler_tag zfs_shutdown_event_tag;
	extern zfsdev_state_t *zfsdev_state_list;

	#define ZFS_MIN_KSTACK_PAGES 4

	-
	static int
	zfsdev_ioctl(struct cdev *dev, ulong_t zcmd, caddr_t arg, int flag,
	struct thread *td)
	{
	uint_t len;
	int vecnum;
	zfs_iocparm_t *zp;
	zfs_cmd_t *zc;
	zfs_cmd_legacy_t *zcl;
	int rc, error;
	void *uaddr;

	len = IOCPARM_LEN(zcmd);
	vecnum = zcmd & 0xff;
	zp = (void *)arg;
	uaddr = (void *)zp->zfs_cmd;
	error = 0;
	zcl = NULL;

	if (len != sizeof (zfs_iocparm_t)) {
	printf("len %d vecnum: %d sizeof (zfs_cmd_t) %ju\n",
	len, vecnum, (uintmax_t)sizeof (zfs_cmd_t));
	return (EINVAL);
	}

	zc = kmem_zalloc(sizeof (zfs_cmd_t), KM_SLEEP);
	/*
	* Remap ioctl code for legacy user binaries
	*/
	if (zp->zfs_ioctl_version == ZFS_IOCVER_LEGACY) {
	vecnum = zfs_ioctl_legacy_to_ozfs(vecnum);
	if (vecnum < 0) {
	kmem_free(zc, sizeof (zfs_cmd_t));
	return (ENOTSUP);
	}
	zcl = kmem_zalloc(sizeof (zfs_cmd_legacy_t), KM_SLEEP);
	if (copyin(uaddr, zcl, sizeof (zfs_cmd_legacy_t))) {
	error = SET_ERROR(EFAULT);
	goto out;
	}
	zfs_cmd_legacy_to_ozfs(zcl, zc);
	} else if (copyin(uaddr, zc, sizeof (zfs_cmd_t))) {
	error = SET_ERROR(EFAULT);
	goto out;
	}
	error = zfsdev_ioctl_common(vecnum, zc, 0);
	if (zcl) {
	zfs_cmd_ozfs_to_legacy(zc, zcl);
	rc = copyout(zcl, uaddr, sizeof (*zcl));
	} else {
	rc = copyout(zc, uaddr, sizeof (*zc));
	}
	if (error == 0 && rc != 0)
	error = SET_ERROR(EFAULT);
	out:
	if (zcl)
	kmem_free(zcl, sizeof (zfs_cmd_legacy_t));
	kmem_free(zc, sizeof (zfs_cmd_t));
	MPASS(tsd_get(rrw_tsd_key) == NULL);
	return (error);
	}

	static void
	zfsdev_close(void *data)
	{
	zfsdev_state_t zs, zsp = data;

	mutex_enter(&zfsdev_state_lock);
	for (zs = zfsdev_state_list; zs != NULL; zs = zs->zs_next) {
	if (zs == zsp)
	break;
	}
	if (zs == NULL \|\| zs->zs_minor <= 0) {
	mutex_exit(&zfsdev_state_lock);
	return;
	}
	zs->zs_minor = -1;
	zfs_onexit_destroy(zs->zs_onexit);
	zfs_zevent_destroy(zs->zs_zevent);
	mutex_exit(&zfsdev_state_lock);
	zs->zs_onexit = NULL;
	zs->zs_zevent = NULL;
	}

	static int
	zfs_ctldev_init(struct cdev *devp)
	{
	boolean_t newzs = B_FALSE;
	minor_t minor;
	zfsdev_state_t zs, zsprev = NULL;

	ASSERT(MUTEX_HELD(&zfsdev_state_lock));

	minor = zfsdev_minor_alloc();
	if (minor == 0)
	return (SET_ERROR(ENXIO));

	for (zs = zfsdev_state_list; zs != NULL; zs = zs->zs_next) {
	if (zs->zs_minor == -1)
	break;
	zsprev = zs;
	}

	if (!zs) {
	zs = kmem_zalloc(sizeof (zfsdev_state_t), KM_SLEEP);
	newzs = B_TRUE;
	}

	devfs_set_cdevpriv(zs, zfsdev_close);

	zfs_onexit_init((zfs_onexit_t **)&zs->zs_onexit);
	zfs_zevent_init((zfs_zevent_t **)&zs->zs_zevent);

	if (newzs) {
	zs->zs_minor = minor;
	wmb();
	zsprev->zs_next = zs;
	} else {
	wmb();
	zs->zs_minor = minor;
	}
	return (0);
	}

	static int
	zfsdev_open(struct cdev devp, int flag, int mode, struct thread td)
	{
	int error;

	mutex_enter(&zfsdev_state_lock);
	error = zfs_ctldev_init(devp);
	mutex_exit(&zfsdev_state_lock);

	return (error);
	}

	static struct cdevsw zfs_cdevsw = {
	.d_version = D_VERSION,
	.d_open = zfsdev_open,
	.d_ioctl = zfsdev_ioctl,
	.d_name = ZFS_DRIVER
	};

	int
	zfsdev_attach(void)
	{
	zfsdev = make_dev(&zfs_cdevsw, 0x0, UID_ROOT, GID_OPERATOR, 0666,
	ZFS_DRIVER);
	return (0);
	}

	void
	zfsdev_detach(void)
	{
	if (zfsdev != NULL)
	destroy_dev(zfsdev);
	}

	int
	zfs__init(void)
	{
	int error;

	#if KSTACK_PAGES < ZFS_MIN_KSTACK_PAGES
	printf("ZFS NOTICE: KSTACK_PAGES is %d which could result in stack "
	"overflow panic!\nPlease consider adding "
	"'options KSTACK_PAGES=%d' to your kernel config\n", KSTACK_PAGES,
	ZFS_MIN_KSTACK_PAGES);
	#endif
	zfs_root_token = root_mount_hold("ZFS");
	if ((error = zfs_kmod_init()) != 0) {
	printf("ZFS: Failed to Load ZFS Filesystem"
	", rc = %d\n", error);
	root_mount_rel(zfs_root_token);
	return (error);
	}


	tsd_create(&zfs_geom_probe_vdev_key, NULL);

	printf("ZFS storage pool version: features support ("
	SPA_VERSION_STRING ")\n");
	root_mount_rel(zfs_root_token);
	ddi_sysevent_init();
	return (0);
	}

	int
	zfs__fini(void)
	{
	if (zfs_busy() \|\| zvol_busy() \|\|
	zio_injection_enabled) {
	return (EBUSY);
	}
	zfs_kmod_fini();
	tsd_destroy(&zfs_geom_probe_vdev_key);
	return (0);
	}

	static void
	zfs_shutdown(void *arg __unused, int howto __unused)
	{

	/*
	* ZFS fini routines can not properly work in a panic-ed system.
	*/
	if (panicstr == NULL)
	zfs__fini();
	}
	-

	static int
	zfs_modevent(module_t mod, int type, void *unused __unused)
	{
	int err;

	switch (type) {
	case MOD_LOAD:
	err = zfs__init();
	if (err == 0)
	zfs_shutdown_event_tag = EVENTHANDLER_REGISTER(
	shutdown_post_sync, zfs_shutdown, NULL,
	SHUTDOWN_PRI_FIRST);
	return (err);
	case MOD_UNLOAD:
	err = zfs__fini();
	if (err == 0 && zfs_shutdown_event_tag != NULL)
	EVENTHANDLER_DEREGISTER(shutdown_post_sync,
	zfs_shutdown_event_tag);
	return (err);
	case MOD_SHUTDOWN:
	return (0);
	default:
	break;
	}
	return (EOPNOTSUPP);
	}

	static moduledata_t zfs_mod = {
	"zfsctrl",
	zfs_modevent,
	0
	};

	#ifdef _KERNEL
	EVENTHANDLER_DEFINE(mountroot, spa_boot_init, NULL, 0);
	#endif

	DECLARE_MODULE(zfsctrl, zfs_mod, SI_SUB_CLOCKS, SI_ORDER_ANY);
	MODULE_VERSION(zfsctrl, 1);
	#if __FreeBSD_version > 1300092
	MODULE_DEPEND(zfsctrl, xdr, 1, 1, 1);
	#else
	MODULE_DEPEND(zfsctrl, krpc, 1, 1, 1);
	#endif
	MODULE_DEPEND(zfsctrl, acl_nfs4, 1, 1, 1);
	MODULE_DEPEND(zfsctrl, crypto, 1, 1, 1);
	Index: vendor-sys/openzfs/dist/module/os/freebsd/zfs/sysctl_os.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/freebsd/zfs/sysctl_os.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/freebsd/zfs/sysctl_os.c (revision 366774)
	@@ -1,699 +1,700 @@
	/*
	* Copyright (c) 2020 iXsystems, Inc.
	* All rights reserved.
	*
	* Redistribution and use in source and binary forms, with or without
	* modification, are permitted provided that the following conditions
	* are met:
	* 1. Redistributions of source code must retain the above copyright
	* notice, this list of conditions and the following disclaimer.
	* 2. Redistributions in binary form must reproduce the above copyright
	* notice, this list of conditions and the following disclaimer in the
	* documentation and/or other materials provided with the distribution.
	*
	* THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
	* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
	* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
	* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
	* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
	* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
	* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
	* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
	* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
	* SUCH DAMAGE.
	*
	*/

	#include <sys/cdefs.h>
	__FBSDID("$FreeBSD$");

	#include <sys/types.h>
	#include <sys/param.h>
	#include <sys/systm.h>
	#include <sys/conf.h>
	#include <sys/kernel.h>
	#include <sys/lock.h>
	#include <sys/malloc.h>
	#include <sys/mutex.h>
	#include <sys/proc.h>
	#include <sys/errno.h>
	#include <sys/uio.h>
	#include <sys/buf.h>
	#include <sys/file.h>
	#include <sys/kmem.h>
	#include <sys/conf.h>
	#include <sys/cmn_err.h>
	#include <sys/stat.h>
	#include <sys/zfs_ioctl.h>
	#include <sys/zfs_vfsops.h>
	#include <sys/zfs_znode.h>
	#include <sys/zap.h>
	#include <sys/spa.h>
	#include <sys/spa_impl.h>
	#include <sys/vdev.h>
	#include <sys/vdev_impl.h>
	#include <sys/dmu.h>
	#include <sys/dsl_dir.h>
	#include <sys/dsl_dataset.h>
	#include <sys/dsl_prop.h>
	#include <sys/dsl_deleg.h>
	#include <sys/dmu_objset.h>
	#include <sys/dmu_impl.h>
	#include <sys/dmu_tx.h>
	#include <sys/sunddi.h>
	#include <sys/policy.h>
	#include <sys/zone.h>
	#include <sys/nvpair.h>
	#include <sys/mount.h>
	#include <sys/taskqueue.h>
	#include <sys/sdt.h>
	#include <sys/fs/zfs.h>
	#include <sys/zfs_ctldir.h>
	#include <sys/zfs_dir.h>
	#include <sys/zfs_onexit.h>
	#include <sys/zvol.h>
	#include <sys/dsl_scan.h>
	#include <sys/dmu_objset.h>
	#include <sys/dmu_send.h>
	#include <sys/dsl_destroy.h>
	#include <sys/dsl_bookmark.h>
	#include <sys/dsl_userhold.h>
	#include <sys/zfeature.h>
	#include <sys/zcp.h>
	#include <sys/zio_checksum.h>
	#include <sys/vdev_removal.h>
	#include <sys/dsl_crypt.h>

	#include <sys/zfs_ioctl_compat.h>
	#include <sys/zfs_context.h>

	#include <sys/arc_impl.h>
	#include <sys/dsl_pool.h>


	/* BEGIN CSTYLED */
	SYSCTL_DECL(_vfs_zfs);
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, arc, CTLFLAG_RW, 0, "ZFS adaptive replacement cache");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, condense, CTLFLAG_RW, 0, "ZFS condense");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, dbuf, CTLFLAG_RW, 0, "ZFS disk buf cache");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, dbuf_cache, CTLFLAG_RW, 0, "ZFS disk buf cache");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, deadman, CTLFLAG_RW, 0, "ZFS deadman");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, dedup, CTLFLAG_RW, 0, "ZFS dedup");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, l2arc, CTLFLAG_RW, 0, "ZFS l2arc");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, livelist, CTLFLAG_RW, 0, "ZFS livelist");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, lua, CTLFLAG_RW, 0, "ZFS lua");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, metaslab, CTLFLAG_RW, 0, "ZFS metaslab");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, mg, CTLFLAG_RW, 0, "ZFS metaslab group");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, multihost, CTLFLAG_RW, 0, "ZFS multihost protection");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, prefetch, CTLFLAG_RW, 0, "ZFS prefetch");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, reconstruct, CTLFLAG_RW, 0, "ZFS reconstruct");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, recv, CTLFLAG_RW, 0, "ZFS receive");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, send, CTLFLAG_RW, 0, "ZFS send");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, spa, CTLFLAG_RW, 0, "ZFS space allocation");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, trim, CTLFLAG_RW, 0, "ZFS TRIM");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, txg, CTLFLAG_RW, 0, "ZFS transaction group");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, vdev, CTLFLAG_RW, 0, "ZFS VDEV");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, zevent, CTLFLAG_RW, 0, "ZFS event");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, zil, CTLFLAG_RW, 0, "ZFS ZIL");
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, zio, CTLFLAG_RW, 0, "ZFS ZIO");

	SYSCTL_NODE(_vfs_zfs_livelist, OID_AUTO, condense, CTLFLAG_RW, 0,
	"ZFS livelist condense");
	SYSCTL_NODE(_vfs_zfs_vdev, OID_AUTO, cache, CTLFLAG_RW, 0, "ZFS VDEV Cache");
	SYSCTL_NODE(_vfs_zfs_vdev, OID_AUTO, file, CTLFLAG_RW, 0, "ZFS VDEV file");
	SYSCTL_NODE(_vfs_zfs_vdev, OID_AUTO, mirror, CTLFLAG_RD, 0,
	"ZFS VDEV mirror");

	SYSCTL_DECL(_vfs_zfs_version);
	SYSCTL_CONST_STRING(_vfs_zfs_version, OID_AUTO, module, CTLFLAG_RD,
	(ZFS_META_VERSION "-" ZFS_META_RELEASE), "OpenZFS module version");

	extern arc_state_t ARC_anon;
	extern arc_state_t ARC_mru;
	extern arc_state_t ARC_mru_ghost;
	extern arc_state_t ARC_mfu;
	extern arc_state_t ARC_mfu_ghost;
	extern arc_state_t ARC_l2c_only;

	/*
	* minimum lifespan of a prefetch block in clock ticks
	* (initialized in arc_init())
	*/

	/* arc.c */

	/* legacy compat */
	extern uint64_t l2arc_write_max; /* def max write size */
	extern uint64_t l2arc_write_boost; /* extra warmup write */
	extern uint64_t l2arc_headroom; /* # of dev writes */
	extern uint64_t l2arc_headroom_boost;
	extern uint64_t l2arc_feed_secs; /* interval seconds */
	extern uint64_t l2arc_feed_min_ms; /* min interval msecs */
	extern int l2arc_noprefetch; /* don't cache prefetch bufs */
	extern int l2arc_feed_again; /* turbo warmup */
	extern int l2arc_norw; /* no reads during writes */

	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, l2arc_write_max, CTLFLAG_RW,
	&l2arc_write_max, 0, "max write size (LEGACY)");
	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, l2arc_write_boost, CTLFLAG_RW,
	&l2arc_write_boost, 0, "extra write during warmup (LEGACY)");
	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, l2arc_headroom, CTLFLAG_RW,
	&l2arc_headroom, 0, "number of dev writes (LEGACY)");
	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, l2arc_feed_secs, CTLFLAG_RW,
	&l2arc_feed_secs, 0, "interval seconds (LEGACY)");
	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, l2arc_feed_min_ms, CTLFLAG_RW,
	&l2arc_feed_min_ms, 0, "min interval milliseconds (LEGACY)");

	SYSCTL_INT(_vfs_zfs, OID_AUTO, l2arc_noprefetch, CTLFLAG_RW,
	&l2arc_noprefetch, 0, "don't cache prefetch bufs (LEGACY)");
	SYSCTL_INT(_vfs_zfs, OID_AUTO, l2arc_feed_again, CTLFLAG_RW,
	&l2arc_feed_again, 0, "turbo warmup (LEGACY)");
	SYSCTL_INT(_vfs_zfs, OID_AUTO, l2arc_norw, CTLFLAG_RW,
	&l2arc_norw, 0, "no reads during writes (LEGACY)");
	#if 0
	extern int zfs_compressed_arc_enabled;
	SYSCTL_INT(_vfs_zfs, OID_AUTO, compressed_arc_enabled, CTLFLAG_RW,
	&zfs_compressed_arc_enabled, 1, "compressed arc buffers (LEGACY)");
	#endif

	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, anon_size, CTLFLAG_RD,
	&ARC_anon.arcs_size.rc_count, 0, "size of anonymous state");
	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, anon_metadata_esize, CTLFLAG_RD,
	&ARC_anon.arcs_esize[ARC_BUFC_METADATA].rc_count, 0,
	"size of anonymous state");
	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, anon_data_esize, CTLFLAG_RD,
	&ARC_anon.arcs_esize[ARC_BUFC_DATA].rc_count, 0,
	"size of anonymous state");

	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mru_size, CTLFLAG_RD,
	&ARC_mru.arcs_size.rc_count, 0, "size of mru state");
	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mru_metadata_esize, CTLFLAG_RD,
	&ARC_mru.arcs_esize[ARC_BUFC_METADATA].rc_count, 0,
	"size of metadata in mru state");
	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mru_data_esize, CTLFLAG_RD,
	&ARC_mru.arcs_esize[ARC_BUFC_DATA].rc_count, 0,
	"size of data in mru state");

	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mru_ghost_size, CTLFLAG_RD,
	&ARC_mru_ghost.arcs_size.rc_count, 0, "size of mru ghost state");
	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mru_ghost_metadata_esize, CTLFLAG_RD,
	&ARC_mru_ghost.arcs_esize[ARC_BUFC_METADATA].rc_count, 0,
	"size of metadata in mru ghost state");
	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mru_ghost_data_esize, CTLFLAG_RD,
	&ARC_mru_ghost.arcs_esize[ARC_BUFC_DATA].rc_count, 0,
	"size of data in mru ghost state");

	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mfu_size, CTLFLAG_RD,
	&ARC_mfu.arcs_size.rc_count, 0, "size of mfu state");
	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mfu_metadata_esize, CTLFLAG_RD,
	&ARC_mfu.arcs_esize[ARC_BUFC_METADATA].rc_count, 0,
	"size of metadata in mfu state");
	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mfu_data_esize, CTLFLAG_RD,
	&ARC_mfu.arcs_esize[ARC_BUFC_DATA].rc_count, 0,
	"size of data in mfu state");

	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mfu_ghost_size, CTLFLAG_RD,
	&ARC_mfu_ghost.arcs_size.rc_count, 0, "size of mfu ghost state");
	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mfu_ghost_metadata_esize, CTLFLAG_RD,
	&ARC_mfu_ghost.arcs_esize[ARC_BUFC_METADATA].rc_count, 0,
	"size of metadata in mfu ghost state");
	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mfu_ghost_data_esize, CTLFLAG_RD,
	&ARC_mfu_ghost.arcs_esize[ARC_BUFC_DATA].rc_count, 0,
	"size of data in mfu ghost state");

	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, l2c_only_size, CTLFLAG_RD,
	&ARC_l2c_only.arcs_size.rc_count, 0, "size of mru state");

	static int
	sysctl_vfs_zfs_arc_no_grow_shift(SYSCTL_HANDLER_ARGS)
	{
	uint32_t val;
	int err;

	val = arc_no_grow_shift;
	err = sysctl_handle_32(oidp, &val, 0, req);
	if (err != 0 \|\| req->newptr == NULL)
	return (err);

	if (val >= arc_shrink_shift)
	return (EINVAL);

	arc_no_grow_shift = val;
	return (0);
	}

	SYSCTL_PROC(_vfs_zfs, OID_AUTO, arc_no_grow_shift,
	CTLTYPE_U32 \| CTLFLAG_RWTUN \| CTLFLAG_MPSAFE, 0, sizeof (uint32_t),
	sysctl_vfs_zfs_arc_no_grow_shift, "U",
	"log2(fraction of ARC which must be free to allow growing)");

	int
	param_set_arc_long(SYSCTL_HANDLER_ARGS)
	{
	int err;

	err = sysctl_handle_long(oidp, arg1, 0, req);
	if (err != 0 \|\| req->newptr == NULL)
	return (err);

	arc_tuning_update(B_TRUE);

	return (0);
	}

	int
	param_set_arc_int(SYSCTL_HANDLER_ARGS)
	{
	int err;

	err = sysctl_handle_int(oidp, arg1, 0, req);
	if (err != 0 \|\| req->newptr == NULL)
	return (err);

	arc_tuning_update(B_TRUE);

	return (0);
	}

	SYSCTL_PROC(_vfs_zfs, OID_AUTO, arc_min,
	CTLTYPE_ULONG \| CTLFLAG_RWTUN \| CTLFLAG_MPSAFE,
	&zfs_arc_min, sizeof (zfs_arc_min), param_set_arc_long, "LU",
	"min arc size (LEGACY)");
	SYSCTL_PROC(_vfs_zfs, OID_AUTO, arc_max,
	CTLTYPE_ULONG \| CTLFLAG_RWTUN \| CTLFLAG_MPSAFE,
	&zfs_arc_max, sizeof (zfs_arc_max), param_set_arc_long, "LU",
	"max arc size (LEGACY)");

	/* dbuf.c */


	/* dmu.c */

	/* dmu_zfetch.c */
	SYSCTL_NODE(_vfs_zfs, OID_AUTO, zfetch, CTLFLAG_RW, 0, "ZFS ZFETCH (LEGACY)");

	/* max bytes to prefetch per stream (default 8MB) */
	extern uint32_t zfetch_max_distance;
	SYSCTL_UINT(_vfs_zfs_zfetch, OID_AUTO, max_distance, CTLFLAG_RWTUN,
	&zfetch_max_distance, 0, "Max bytes to prefetch per stream (LEGACY)");

	/* max bytes to prefetch indirects for per stream (default 64MB) */
	extern uint32_t zfetch_max_idistance;
	-SYSCTL_UINT(_vfs_zfs_prefetch, OID_AUTO, max_idistance, CTLFLAG_RWTUN,
	- &zfetch_max_idistance, 0, "Max bytes to prefetch indirects for per stream");
	+SYSCTL_UINT(_vfs_zfs_zfetch, OID_AUTO, max_idistance, CTLFLAG_RWTUN,
	+ &zfetch_max_idistance, 0,
	+ "Max bytes to prefetch indirects for per stream (LEGACY)");

	/* dsl_pool.c */

	/* dnode.c */
	extern int zfs_default_bs;
	SYSCTL_INT(_vfs_zfs, OID_AUTO, default_bs, CTLFLAG_RWTUN,
	&zfs_default_bs, 0, "Default dnode block shift");

	extern int zfs_default_ibs;
	SYSCTL_INT(_vfs_zfs, OID_AUTO, default_ibs, CTLFLAG_RWTUN,
	&zfs_default_ibs, 0, "Default dnode indirect block shift");


	/* dsl_scan.c */

	/* metaslab.c */

	/*
	* In pools where the log space map feature is not enabled we touch
	* multiple metaslabs (and their respective space maps) with each
	* transaction group. Thus, we benefit from having a small space map
	* block size since it allows us to issue more I/O operations scattered
	* around the disk. So a sane default for the space map block size
	* is 8~16K.
	*/
	extern int zfs_metaslab_sm_blksz_no_log;
	SYSCTL_INT(_vfs_zfs_metaslab, OID_AUTO, sm_blksz_no_log, CTLFLAG_RDTUN,
	&zfs_metaslab_sm_blksz_no_log, 0,
	"Block size for space map in pools with log space map disabled. "
	"Power of 2 and greater than 4096.");

	/*
	* When the log space map feature is enabled, we accumulate a lot of
	* changes per metaslab that are flushed once in a while so we benefit
	* from a bigger block size like 128K for the metaslab space maps.
	*/
	extern int zfs_metaslab_sm_blksz_with_log;
	SYSCTL_INT(_vfs_zfs_metaslab, OID_AUTO, sm_blksz_with_log, CTLFLAG_RDTUN,
	&zfs_metaslab_sm_blksz_with_log, 0,
	"Block size for space map in pools with log space map enabled. "
	"Power of 2 and greater than 4096.");

	/*
	* The in-core space map representation is more compact than its on-disk form.
	* The zfs_condense_pct determines how much more compact the in-core
	* space map representation must be before we compact it on-disk.
	* Values should be greater than or equal to 100.
	*/
	extern int zfs_condense_pct;
	SYSCTL_INT(_vfs_zfs, OID_AUTO, condense_pct, CTLFLAG_RWTUN,
	&zfs_condense_pct, 0,
	"Condense on-disk spacemap when it is more than this many percents"
	" of in-memory counterpart");

	extern int zfs_remove_max_segment;
	SYSCTL_INT(_vfs_zfs, OID_AUTO, remove_max_segment, CTLFLAG_RWTUN,
	&zfs_remove_max_segment, 0, "Largest contiguous segment ZFS will attempt to"
	" allocate when removing a device");

	extern int zfs_removal_suspend_progress;
	SYSCTL_INT(_vfs_zfs, OID_AUTO, removal_suspend_progress, CTLFLAG_RWTUN,
	&zfs_removal_suspend_progress, 0, "Ensures certain actions can happen while"
	" in the middle of a removal");


	/*
	* Minimum size which forces the dynamic allocator to change
	* it's allocation strategy. Once the space map cannot satisfy
	* an allocation of this size then it switches to using more
	* aggressive strategy (i.e search by size rather than offset).
	*/
	extern uint64_t metaslab_df_alloc_threshold;
	SYSCTL_QUAD(_vfs_zfs_metaslab, OID_AUTO, df_alloc_threshold, CTLFLAG_RWTUN,
	&metaslab_df_alloc_threshold, 0,
	"Minimum size which forces the dynamic allocator to change it's allocation strategy");

	/*
	* The minimum free space, in percent, which must be available
	* in a space map to continue allocations in a first-fit fashion.
	* Once the space map's free space drops below this level we dynamically
	* switch to using best-fit allocations.
	*/
	extern int metaslab_df_free_pct;
	SYSCTL_INT(_vfs_zfs_metaslab, OID_AUTO, df_free_pct, CTLFLAG_RWTUN,
	&metaslab_df_free_pct, 0,
	"The minimum free space, in percent, which must be available in a "
	"space map to continue allocations in a first-fit fashion");

	/*
	* Percentage of all cpus that can be used by the metaslab taskq.
	*/
	extern int metaslab_load_pct;
	SYSCTL_INT(_vfs_zfs_metaslab, OID_AUTO, load_pct, CTLFLAG_RWTUN,
	&metaslab_load_pct, 0,
	"Percentage of cpus that can be used by the metaslab taskq");

	/*
	* Max number of metaslabs per group to preload.
	*/
	extern int metaslab_preload_limit;
	SYSCTL_INT(_vfs_zfs_metaslab, OID_AUTO, preload_limit, CTLFLAG_RWTUN,
	&metaslab_preload_limit, 0,
	"Max number of metaslabs per group to preload");

	/* refcount.c */
	extern int reference_tracking_enable;
	SYSCTL_INT(_vfs_zfs, OID_AUTO, reference_tracking_enable, CTLFLAG_RDTUN,
	&reference_tracking_enable, 0,
	"Track reference holders to refcount_t objects, used mostly by ZFS");

	/* spa.c */
	extern int zfs_ccw_retry_interval;
	SYSCTL_INT(_vfs_zfs, OID_AUTO, ccw_retry_interval, CTLFLAG_RWTUN,
	&zfs_ccw_retry_interval, 0,
	"Configuration cache file write, retry after failure, interval (seconds)");

	extern uint64_t zfs_max_missing_tvds_cachefile;
	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, max_missing_tvds_cachefile, CTLFLAG_RWTUN,
	&zfs_max_missing_tvds_cachefile, 0,
	"allow importing pools with missing top-level vdevs in cache file");

	extern uint64_t zfs_max_missing_tvds_scan;
	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, max_missing_tvds_scan, CTLFLAG_RWTUN,
	&zfs_max_missing_tvds_scan, 0,
	"allow importing pools with missing top-level vdevs during scan");

	/* spa_misc.c */
	extern int zfs_flags;
	static int
	sysctl_vfs_zfs_debug_flags(SYSCTL_HANDLER_ARGS)
	{
	int err, val;

	val = zfs_flags;
	err = sysctl_handle_int(oidp, &val, 0, req);
	if (err != 0 \|\| req->newptr == NULL)
	return (err);

	/*
	* ZFS_DEBUG_MODIFY must be enabled prior to boot so all
	* arc buffers in the system have the necessary additional
	* checksum data. However, it is safe to disable at any
	* time.
	*/
	if (!(zfs_flags & ZFS_DEBUG_MODIFY))
	val &= ~ZFS_DEBUG_MODIFY;
	zfs_flags = val;

	return (0);
	}

	SYSCTL_PROC(_vfs_zfs, OID_AUTO, debugflags,
	CTLTYPE_UINT \| CTLFLAG_MPSAFE \| CTLFLAG_RWTUN, NULL, 0,
	sysctl_vfs_zfs_debug_flags, "IU", "Debug flags for ZFS testing.");

	int
	param_set_deadman_synctime(SYSCTL_HANDLER_ARGS)
	{
	unsigned long val;
	int err;

	val = zfs_deadman_synctime_ms;
	err = sysctl_handle_long(oidp, &val, 0, req);
	if (err != 0 \|\| req->newptr == NULL)
	return (err);
	zfs_deadman_synctime_ms = val;

	spa_set_deadman_synctime(MSEC2NSEC(zfs_deadman_synctime_ms));

	return (0);
	}

	int
	param_set_deadman_ziotime(SYSCTL_HANDLER_ARGS)
	{
	unsigned long val;
	int err;

	val = zfs_deadman_ziotime_ms;
	err = sysctl_handle_long(oidp, &val, 0, req);
	if (err != 0 \|\| req->newptr == NULL)
	return (err);
	zfs_deadman_ziotime_ms = val;

	spa_set_deadman_ziotime(MSEC2NSEC(zfs_deadman_synctime_ms));

	return (0);
	}

	int
	param_set_deadman_failmode(SYSCTL_HANDLER_ARGS)
	{
	char buf[16];
	int rc;

	if (req->newptr == NULL)
	strlcpy(buf, zfs_deadman_failmode, sizeof (buf));

	rc = sysctl_handle_string(oidp, buf, sizeof (buf), req);
	if (rc \|\| req->newptr == NULL)
	return (rc);
	if (strcmp(buf, zfs_deadman_failmode) == 0)
	return (0);
	if (!strcmp(buf, "wait"))
	zfs_deadman_failmode = "wait";
	if (!strcmp(buf, "continue"))
	zfs_deadman_failmode = "continue";
	if (!strcmp(buf, "panic"))
	zfs_deadman_failmode = "panic";

	return (-param_set_deadman_failmode_common(buf));
	}


	/* spacemap.c */
	extern int space_map_ibs;
	SYSCTL_INT(_vfs_zfs, OID_AUTO, space_map_ibs, CTLFLAG_RWTUN,
	&space_map_ibs, 0, "Space map indirect block shift");


	/* vdev.c */
	int
	param_set_min_auto_ashift(SYSCTL_HANDLER_ARGS)
	{
	uint64_t val;
	int err;

	val = zfs_vdev_min_auto_ashift;
	err = sysctl_handle_64(oidp, &val, 0, req);
	if (err != 0 \|\| req->newptr == NULL)
	return (SET_ERROR(err));

	if (val < ASHIFT_MIN \|\| val > zfs_vdev_max_auto_ashift)
	return (SET_ERROR(EINVAL));

	zfs_vdev_min_auto_ashift = val;

	return (0);
	}

	int
	param_set_max_auto_ashift(SYSCTL_HANDLER_ARGS)
	{
	uint64_t val;
	int err;

	val = zfs_vdev_max_auto_ashift;
	err = sysctl_handle_64(oidp, &val, 0, req);
	if (err != 0 \|\| req->newptr == NULL)
	return (SET_ERROR(err));

	if (val > ASHIFT_MAX \|\| val < zfs_vdev_min_auto_ashift)
	return (SET_ERROR(EINVAL));

	zfs_vdev_max_auto_ashift = val;

	return (0);
	}

	SYSCTL_PROC(_vfs_zfs, OID_AUTO, min_auto_ashift,
	CTLTYPE_U64 \| CTLFLAG_RWTUN \| CTLFLAG_MPSAFE,
	&zfs_vdev_min_auto_ashift, sizeof (zfs_vdev_min_auto_ashift),
	param_set_min_auto_ashift, "QU",
	"Min ashift used when creating new top-level vdev. (LEGACY)");
	SYSCTL_PROC(_vfs_zfs, OID_AUTO, max_auto_ashift,
	CTLTYPE_U64 \| CTLFLAG_RWTUN \| CTLFLAG_MPSAFE,
	&zfs_vdev_max_auto_ashift, sizeof (zfs_vdev_max_auto_ashift),
	param_set_max_auto_ashift, "QU",
	"Max ashift used when optimizing for logical -> physical sector size on "
	"new top-level vdevs. (LEGACY)");

	/*
	* Since the DTL space map of a vdev is not expected to have a lot of
	* entries, we default its block size to 4K.
	*/
	extern int zfs_vdev_dtl_sm_blksz;
	SYSCTL_INT(_vfs_zfs, OID_AUTO, dtl_sm_blksz, CTLFLAG_RDTUN,
	&zfs_vdev_dtl_sm_blksz, 0,
	"Block size for DTL space map. Power of 2 and greater than 4096.");

	/*
	* vdev-wide space maps that have lots of entries written to them at
	* the end of each transaction can benefit from a higher I/O bandwidth
	* (e.g. vdev_obsolete_sm), thus we default their block size to 128K.
	*/
	extern int zfs_vdev_standard_sm_blksz;
	SYSCTL_INT(_vfs_zfs, OID_AUTO, standard_sm_blksz, CTLFLAG_RDTUN,
	&zfs_vdev_standard_sm_blksz, 0,
	"Block size for standard space map. Power of 2 and greater than 4096.");

	extern int vdev_validate_skip;
	SYSCTL_INT(_vfs_zfs, OID_AUTO, validate_skip, CTLFLAG_RDTUN,
	&vdev_validate_skip, 0,
	"Enable to bypass vdev_validate().");


	/* vdev_cache.c */

	/* vdev_mirror.c */
	/*
	* The load configuration settings below are tuned by default for
	* the case where all devices are of the same rotational type.
	*
	* If there is a mixture of rotating and non-rotating media, setting
	* non_rotating_seek_inc to 0 may well provide better results as it
	* will direct more reads to the non-rotating vdevs which are more
	* likely to have a higher performance.
	*/


	/* vdev_queue.c */
	#define ZFS_VDEV_QUEUE_KNOB_MIN(name) \
	extern uint32_t zfs_vdev_ ## name ## _min_active; \
	SYSCTL_UINT(_vfs_zfs_vdev, OID_AUTO, name ## _min_active, CTLFLAG_RWTUN,\
	&zfs_vdev_ ## name ## _min_active, 0, \
	"Initial number of I/O requests of type " #name \
	" active for each device");

	#define ZFS_VDEV_QUEUE_KNOB_MAX(name) \
	extern uint32_t zfs_vdev_ ## name ## _max_active; \
	SYSCTL_UINT(_vfs_zfs_vdev, OID_AUTO, name ## _max_active, CTLFLAG_RWTUN, \
	&zfs_vdev_ ## name ## _max_active, 0, \
	"Maximum number of I/O requests of type " #name \
	" active for each device");


	#undef ZFS_VDEV_QUEUE_KNOB

	extern uint32_t zfs_vdev_max_active;
	SYSCTL_UINT(_vfs_zfs, OID_AUTO, top_maxinflight, CTLFLAG_RWTUN,
	&zfs_vdev_max_active, 0,
	"The maximum number of I/Os of all types active for each device. (LEGACY)");

	extern int zfs_vdev_def_queue_depth;
	SYSCTL_INT(_vfs_zfs_vdev, OID_AUTO, def_queue_depth, CTLFLAG_RWTUN,
	&zfs_vdev_def_queue_depth, 0,
	"Default queue depth for each allocator");

	/*extern uint64_t zfs_multihost_history;
	SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, multihost_history, CTLFLAG_RWTUN,
	&zfs_multihost_history, 0,
	"Historical staticists for the last N multihost updates");*/

	#ifdef notyet
	SYSCTL_INT(_vfs_zfs_vdev, OID_AUTO, trim_on_init, CTLFLAG_RW,
	&vdev_trim_on_init, 0, "Enable/disable full vdev trim on initialisation");
	#endif


	/* zio.c */
	#if defined(__LP64__)
	int zio_use_uma = 1;
	#else
	int zio_use_uma = 0;
	#endif

	SYSCTL_INT(_vfs_zfs_zio, OID_AUTO, use_uma, CTLFLAG_RDTUN, &zio_use_uma, 0,
	"Use uma(9) for ZIO allocations");
	SYSCTL_INT(_vfs_zfs_zio, OID_AUTO, exclude_metadata, CTLFLAG_RDTUN, &zio_exclude_metadata, 0,
	"Exclude metadata buffers from dumps as well");

	int
	param_set_slop_shift(SYSCTL_HANDLER_ARGS)
	{
	int val;
	int err;

	val = (int )arg1;

	err = sysctl_handle_int(oidp, &val, 0, req);
	if (err != 0 \|\| req->newptr == NULL)
	return (err);

	if (val < 1 \|\| val > 31)
	return (EINVAL);

	(int )arg1 = val;

	return (0);
	}

	int
	param_set_multihost_interval(SYSCTL_HANDLER_ARGS)
	{
	int err;

	err = sysctl_handle_long(oidp, arg1, 0, req);
	if (err != 0 \|\| req->newptr == NULL)
	return (err);

	if (spa_mode_global != SPA_MODE_UNINIT)
	mmp_signal_all_threads();

	return (0);
	}
	Index: vendor-sys/openzfs/dist/module/os/freebsd/zfs/zfs_acl.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/freebsd/zfs/zfs_acl.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/freebsd/zfs/zfs_acl.c (revision 366774)
	@@ -1,2700 +1,2700 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2013 by Delphix. All rights reserved.
	* Copyright 2017 Nexenta Systems, Inc. All rights reserved.
	*/

	#include <sys/types.h>
	#include <sys/param.h>
	#include <sys/time.h>
	#include <sys/systm.h>
	#include <sys/sysmacros.h>
	#include <sys/resource.h>
	#include <sys/vfs.h>
	#include <sys/vnode.h>
	#include <sys/file.h>
	#include <sys/stat.h>
	#include <sys/kmem.h>
	#include <sys/cmn_err.h>
	#include <sys/errno.h>
	#include <sys/unistd.h>
	#include <sys/sdt.h>
	#include <sys/fs/zfs.h>
	#include <sys/policy.h>
	#include <sys/zfs_znode.h>
	#include <sys/zfs_fuid.h>
	#include <sys/zfs_acl.h>
	#include <sys/zfs_dir.h>
	#include <sys/zfs_quota.h>
	#include <sys/zfs_vfsops.h>
	#include <sys/dmu.h>
	#include <sys/dnode.h>
	#include <sys/zap.h>
	#include <sys/sa.h>
	#include <acl/acl_common.h>


	#define ALLOW ACE_ACCESS_ALLOWED_ACE_TYPE
	#define DENY ACE_ACCESS_DENIED_ACE_TYPE
	#define MAX_ACE_TYPE ACE_SYSTEM_ALARM_CALLBACK_OBJECT_ACE_TYPE
	#define MIN_ACE_TYPE ALLOW

	#define OWNING_GROUP (ACE_GROUP\|ACE_IDENTIFIER_GROUP)
	#define EVERYONE_ALLOW_MASK (ACE_READ_ACL\|ACE_READ_ATTRIBUTES \| \
	ACE_READ_NAMED_ATTRS\|ACE_SYNCHRONIZE)
	#define EVERYONE_DENY_MASK (ACE_WRITE_ACL\|ACE_WRITE_OWNER \| \
	ACE_WRITE_ATTRIBUTES\|ACE_WRITE_NAMED_ATTRS)
	#define OWNER_ALLOW_MASK (ACE_WRITE_ACL \| ACE_WRITE_OWNER \| \
	ACE_WRITE_ATTRIBUTES\|ACE_WRITE_NAMED_ATTRS)

	#define ZFS_CHECKED_MASKS (ACE_READ_ACL\|ACE_READ_ATTRIBUTES\|ACE_READ_DATA\| \
	ACE_READ_NAMED_ATTRS\|ACE_WRITE_DATA\|ACE_WRITE_ATTRIBUTES\| \
	ACE_WRITE_NAMED_ATTRS\|ACE_APPEND_DATA\|ACE_EXECUTE\|ACE_WRITE_OWNER\| \
	ACE_WRITE_ACL\|ACE_DELETE\|ACE_DELETE_CHILD\|ACE_SYNCHRONIZE)

	#define WRITE_MASK_DATA (ACE_WRITE_DATA\|ACE_APPEND_DATA\|ACE_WRITE_NAMED_ATTRS)
	#define WRITE_MASK_ATTRS (ACE_WRITE_ACL\|ACE_WRITE_OWNER\|ACE_WRITE_ATTRIBUTES\| \
	ACE_DELETE\|ACE_DELETE_CHILD)
	#define WRITE_MASK (WRITE_MASK_DATA\|WRITE_MASK_ATTRS)

	#define OGE_CLEAR (ACE_READ_DATA\|ACE_LIST_DIRECTORY\|ACE_WRITE_DATA\| \
	ACE_ADD_FILE\|ACE_APPEND_DATA\|ACE_ADD_SUBDIRECTORY\|ACE_EXECUTE)

	#define OKAY_MASK_BITS (ACE_READ_DATA\|ACE_LIST_DIRECTORY\|ACE_WRITE_DATA\| \
	ACE_ADD_FILE\|ACE_APPEND_DATA\|ACE_ADD_SUBDIRECTORY\|ACE_EXECUTE)

	#define ALL_INHERIT (ACE_FILE_INHERIT_ACE\|ACE_DIRECTORY_INHERIT_ACE \| \
	ACE_NO_PROPAGATE_INHERIT_ACE\|ACE_INHERIT_ONLY_ACE\|ACE_INHERITED_ACE)

	#define RESTRICTED_CLEAR (ACE_WRITE_ACL\|ACE_WRITE_OWNER)

	#define V4_ACL_WIDE_FLAGS (ZFS_ACL_AUTO_INHERIT\|ZFS_ACL_DEFAULTED\|\
	ZFS_ACL_PROTECTED)

	#define ZFS_ACL_WIDE_FLAGS (V4_ACL_WIDE_FLAGS\|ZFS_ACL_TRIVIAL\|ZFS_INHERIT_ACE\|\
	ZFS_ACL_OBJ_ACE)

	#define ALL_MODE_EXECS (S_IXUSR \| S_IXGRP \| S_IXOTH)

	static uint16_t
	zfs_ace_v0_get_type(void *acep)
	{
	return (((zfs_oldace_t *)acep)->z_type);
	}

	static uint16_t
	zfs_ace_v0_get_flags(void *acep)
	{
	return (((zfs_oldace_t *)acep)->z_flags);
	}

	static uint32_t
	zfs_ace_v0_get_mask(void *acep)
	{
	return (((zfs_oldace_t *)acep)->z_access_mask);
	}

	static uint64_t
	zfs_ace_v0_get_who(void *acep)
	{
	return (((zfs_oldace_t *)acep)->z_fuid);
	}

	static void
	zfs_ace_v0_set_type(void *acep, uint16_t type)
	{
	((zfs_oldace_t *)acep)->z_type = type;
	}

	static void
	zfs_ace_v0_set_flags(void *acep, uint16_t flags)
	{
	((zfs_oldace_t *)acep)->z_flags = flags;
	}

	static void
	zfs_ace_v0_set_mask(void *acep, uint32_t mask)
	{
	((zfs_oldace_t *)acep)->z_access_mask = mask;
	}

	static void
	zfs_ace_v0_set_who(void *acep, uint64_t who)
	{
	((zfs_oldace_t *)acep)->z_fuid = who;
	}

	/ARGSUSED/
	static size_t
	zfs_ace_v0_size(void *acep)
	{
	return (sizeof (zfs_oldace_t));
	}

	static size_t
	zfs_ace_v0_abstract_size(void)
	{
	return (sizeof (zfs_oldace_t));
	}

	static int
	zfs_ace_v0_mask_off(void)
	{
	return (offsetof(zfs_oldace_t, z_access_mask));
	}

	/ARGSUSED/
	static int
	zfs_ace_v0_data(void acep, void *datap)
	{
	*datap = NULL;
	return (0);
	}

	static acl_ops_t zfs_acl_v0_ops = {
	zfs_ace_v0_get_mask,
	zfs_ace_v0_set_mask,
	zfs_ace_v0_get_flags,
	zfs_ace_v0_set_flags,
	zfs_ace_v0_get_type,
	zfs_ace_v0_set_type,
	zfs_ace_v0_get_who,
	zfs_ace_v0_set_who,
	zfs_ace_v0_size,
	zfs_ace_v0_abstract_size,
	zfs_ace_v0_mask_off,
	zfs_ace_v0_data
	};

	static uint16_t
	zfs_ace_fuid_get_type(void *acep)
	{
	return (((zfs_ace_hdr_t *)acep)->z_type);
	}

	static uint16_t
	zfs_ace_fuid_get_flags(void *acep)
	{
	return (((zfs_ace_hdr_t *)acep)->z_flags);
	}

	static uint32_t
	zfs_ace_fuid_get_mask(void *acep)
	{
	return (((zfs_ace_hdr_t *)acep)->z_access_mask);
	}

	static uint64_t
	zfs_ace_fuid_get_who(void *args)
	{
	uint16_t entry_type;
	zfs_ace_t *acep = args;

	entry_type = acep->z_hdr.z_flags & ACE_TYPE_FLAGS;

	if (entry_type == ACE_OWNER \|\| entry_type == OWNING_GROUP \|\|
	entry_type == ACE_EVERYONE)
	return (-1);
	return (((zfs_ace_t *)acep)->z_fuid);
	}

	static void
	zfs_ace_fuid_set_type(void *acep, uint16_t type)
	{
	((zfs_ace_hdr_t *)acep)->z_type = type;
	}

	static void
	zfs_ace_fuid_set_flags(void *acep, uint16_t flags)
	{
	((zfs_ace_hdr_t *)acep)->z_flags = flags;
	}

	static void
	zfs_ace_fuid_set_mask(void *acep, uint32_t mask)
	{
	((zfs_ace_hdr_t *)acep)->z_access_mask = mask;
	}

	static void
	zfs_ace_fuid_set_who(void *arg, uint64_t who)
	{
	zfs_ace_t *acep = arg;

	uint16_t entry_type = acep->z_hdr.z_flags & ACE_TYPE_FLAGS;

	if (entry_type == ACE_OWNER \|\| entry_type == OWNING_GROUP \|\|
	entry_type == ACE_EVERYONE)
	return;
	acep->z_fuid = who;
	}

	static size_t
	zfs_ace_fuid_size(void *acep)
	{
	zfs_ace_hdr_t *zacep = acep;
	uint16_t entry_type;

	switch (zacep->z_type) {
	case ACE_ACCESS_ALLOWED_OBJECT_ACE_TYPE:
	case ACE_ACCESS_DENIED_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_AUDIT_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_ALARM_OBJECT_ACE_TYPE:
	return (sizeof (zfs_object_ace_t));
	case ALLOW:
	case DENY:
	entry_type =
	(((zfs_ace_hdr_t *)acep)->z_flags & ACE_TYPE_FLAGS);
	if (entry_type == ACE_OWNER \|\|
	entry_type == OWNING_GROUP \|\|
	entry_type == ACE_EVERYONE)
	return (sizeof (zfs_ace_hdr_t));
	/FALLTHROUGH/
	default:
	return (sizeof (zfs_ace_t));
	}
	}

	static size_t
	zfs_ace_fuid_abstract_size(void)
	{
	return (sizeof (zfs_ace_hdr_t));
	}

	static int
	zfs_ace_fuid_mask_off(void)
	{
	return (offsetof(zfs_ace_hdr_t, z_access_mask));
	}

	static int
	zfs_ace_fuid_data(void acep, void *datap)
	{
	zfs_ace_t *zacep = acep;
	zfs_object_ace_t *zobjp;

	switch (zacep->z_hdr.z_type) {
	case ACE_ACCESS_ALLOWED_OBJECT_ACE_TYPE:
	case ACE_ACCESS_DENIED_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_AUDIT_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_ALARM_OBJECT_ACE_TYPE:
	zobjp = acep;
	*datap = (caddr_t)zobjp + sizeof (zfs_ace_t);
	return (sizeof (zfs_object_ace_t) - sizeof (zfs_ace_t));
	default:
	*datap = NULL;
	return (0);
	}
	}

	static acl_ops_t zfs_acl_fuid_ops = {
	zfs_ace_fuid_get_mask,
	zfs_ace_fuid_set_mask,
	zfs_ace_fuid_get_flags,
	zfs_ace_fuid_set_flags,
	zfs_ace_fuid_get_type,
	zfs_ace_fuid_set_type,
	zfs_ace_fuid_get_who,
	zfs_ace_fuid_set_who,
	zfs_ace_fuid_size,
	zfs_ace_fuid_abstract_size,
	zfs_ace_fuid_mask_off,
	zfs_ace_fuid_data
	};

	/*
	* The following three functions are provided for compatibility with
	* older ZPL version in order to determine if the file use to have
	* an external ACL and what version of ACL previously existed on the
	* file. Would really be nice to not need this, sigh.
	*/
	uint64_t
	zfs_external_acl(znode_t *zp)
	{
	zfs_acl_phys_t acl_phys;
	int error;

	if (zp->z_is_sa)
	return (0);

	/*
	* Need to deal with a potential
	* race where zfs_sa_upgrade could cause
	* z_isa_sa to change.
	*
	* If the lookup fails then the state of z_is_sa should have
	* changed.
	*/

	if ((error = sa_lookup(zp->z_sa_hdl, SA_ZPL_ZNODE_ACL(zp->z_zfsvfs),
	&acl_phys, sizeof (acl_phys))) == 0)
	return (acl_phys.z_acl_extern_obj);
	else {
	/*
	* after upgrade the SA_ZPL_ZNODE_ACL should have been
	* removed
	*/
	VERIFY(zp->z_is_sa && error == ENOENT);
	return (0);
	}
	}

	/*
	* Determine size of ACL in bytes
	*
	* This is more complicated than it should be since we have to deal
	* with old external ACLs.
	*/
	static int
	zfs_acl_znode_info(znode_t zp, int aclsize, int *aclcount,
	zfs_acl_phys_t *aclphys)
	{
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;
	uint64_t acl_count;
	int size;
	int error;

	ASSERT(MUTEX_HELD(&zp->z_acl_lock));
	if (zp->z_is_sa) {
	if ((error = sa_size(zp->z_sa_hdl, SA_ZPL_DACL_ACES(zfsvfs),
	&size)) != 0)
	return (error);
	*aclsize = size;
	if ((error = sa_lookup(zp->z_sa_hdl, SA_ZPL_DACL_COUNT(zfsvfs),
	&acl_count, sizeof (acl_count))) != 0)
	return (error);
	*aclcount = acl_count;
	} else {
	if ((error = sa_lookup(zp->z_sa_hdl, SA_ZPL_ZNODE_ACL(zfsvfs),
	aclphys, sizeof (*aclphys))) != 0)
	return (error);

	if (aclphys->z_acl_version == ZFS_ACL_VERSION_INITIAL) {
	*aclsize = ZFS_ACL_SIZE(aclphys->z_acl_size);
	*aclcount = aclphys->z_acl_size;
	} else {
	*aclsize = aclphys->z_acl_size;
	*aclcount = aclphys->z_acl_count;
	}
	}
	return (0);
	}

	int
	zfs_znode_acl_version(znode_t *zp)
	{
	zfs_acl_phys_t acl_phys;

	if (zp->z_is_sa)
	return (ZFS_ACL_VERSION_FUID);
	else {
	int error;

	/*
	* Need to deal with a potential
	* race where zfs_sa_upgrade could cause
	* z_isa_sa to change.
	*
	* If the lookup fails then the state of z_is_sa should have
	* changed.
	*/
	if ((error = sa_lookup(zp->z_sa_hdl,
	SA_ZPL_ZNODE_ACL(zp->z_zfsvfs),
	&acl_phys, sizeof (acl_phys))) == 0)
	return (acl_phys.z_acl_version);
	else {
	/*
	* After upgrade SA_ZPL_ZNODE_ACL should have
	* been removed.
	*/
	VERIFY(zp->z_is_sa && error == ENOENT);
	return (ZFS_ACL_VERSION_FUID);
	}
	}
	}

	static int
	zfs_acl_version(int version)
	{
	if (version < ZPL_VERSION_FUID)
	return (ZFS_ACL_VERSION_INITIAL);
	else
	return (ZFS_ACL_VERSION_FUID);
	}

	static int
	zfs_acl_version_zp(znode_t *zp)
	{
	return (zfs_acl_version(zp->z_zfsvfs->z_version));
	}

	zfs_acl_t *
	zfs_acl_alloc(int vers)
	{
	zfs_acl_t *aclp;

	aclp = kmem_zalloc(sizeof (zfs_acl_t), KM_SLEEP);
	list_create(&aclp->z_acl, sizeof (zfs_acl_node_t),
	offsetof(zfs_acl_node_t, z_next));
	aclp->z_version = vers;
	if (vers == ZFS_ACL_VERSION_FUID)
	aclp->z_ops = &zfs_acl_fuid_ops;
	else
	aclp->z_ops = &zfs_acl_v0_ops;
	return (aclp);
	}

	zfs_acl_node_t *
	zfs_acl_node_alloc(size_t bytes)
	{
	zfs_acl_node_t *aclnode;

	aclnode = kmem_zalloc(sizeof (zfs_acl_node_t), KM_SLEEP);
	if (bytes) {
	aclnode->z_acldata = kmem_alloc(bytes, KM_SLEEP);
	aclnode->z_allocdata = aclnode->z_acldata;
	aclnode->z_allocsize = bytes;
	aclnode->z_size = bytes;
	}

	return (aclnode);
	}

	static void
	zfs_acl_node_free(zfs_acl_node_t *aclnode)
	{
	if (aclnode->z_allocsize)
	kmem_free(aclnode->z_allocdata, aclnode->z_allocsize);
	kmem_free(aclnode, sizeof (zfs_acl_node_t));
	}

	static void
	zfs_acl_release_nodes(zfs_acl_t *aclp)
	{
	zfs_acl_node_t *aclnode;

	while ((aclnode = list_head(&aclp->z_acl))) {
	list_remove(&aclp->z_acl, aclnode);
	zfs_acl_node_free(aclnode);
	}
	aclp->z_acl_count = 0;
	aclp->z_acl_bytes = 0;
	}

	void
	zfs_acl_free(zfs_acl_t *aclp)
	{
	zfs_acl_release_nodes(aclp);
	list_destroy(&aclp->z_acl);
	kmem_free(aclp, sizeof (zfs_acl_t));
	}

	static boolean_t
	zfs_acl_valid_ace_type(uint_t type, uint_t flags)
	{
	uint16_t entry_type;

	switch (type) {
	case ALLOW:
	case DENY:
	case ACE_SYSTEM_AUDIT_ACE_TYPE:
	case ACE_SYSTEM_ALARM_ACE_TYPE:
	entry_type = flags & ACE_TYPE_FLAGS;
	return (entry_type == ACE_OWNER \|\|
	entry_type == OWNING_GROUP \|\|
	entry_type == ACE_EVERYONE \|\| entry_type == 0 \|\|
	entry_type == ACE_IDENTIFIER_GROUP);
	default:
	if (type >= MIN_ACE_TYPE && type <= MAX_ACE_TYPE)
	return (B_TRUE);
	}
	return (B_FALSE);
	}

	static boolean_t
	zfs_ace_valid(vtype_t obj_type, zfs_acl_t *aclp, uint16_t type, uint16_t iflags)
	{
	/*
	* first check type of entry
	*/

	if (!zfs_acl_valid_ace_type(type, iflags))
	return (B_FALSE);

	switch (type) {
	case ACE_ACCESS_ALLOWED_OBJECT_ACE_TYPE:
	case ACE_ACCESS_DENIED_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_AUDIT_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_ALARM_OBJECT_ACE_TYPE:
	if (aclp->z_version < ZFS_ACL_VERSION_FUID)
	return (B_FALSE);
	aclp->z_hints \|= ZFS_ACL_OBJ_ACE;
	}

	/*
	* next check inheritance level flags
	*/

	if (obj_type == VDIR &&
	(iflags & (ACE_FILE_INHERIT_ACE\|ACE_DIRECTORY_INHERIT_ACE)))
	aclp->z_hints \|= ZFS_INHERIT_ACE;

	if (iflags & (ACE_INHERIT_ONLY_ACE\|ACE_NO_PROPAGATE_INHERIT_ACE)) {
	if ((iflags & (ACE_FILE_INHERIT_ACE\|
	ACE_DIRECTORY_INHERIT_ACE)) == 0) {
	return (B_FALSE);
	}
	}

	return (B_TRUE);
	}

	static void *
	zfs_acl_next_ace(zfs_acl_t aclp, void start, uint64_t *who,
	uint32_t access_mask, uint16_t iflags, uint16_t *type)
	{
	zfs_acl_node_t *aclnode;

	ASSERT(aclp);

	if (start == NULL) {
	aclnode = list_head(&aclp->z_acl);
	if (aclnode == NULL)
	return (NULL);

	aclp->z_next_ace = aclnode->z_acldata;
	aclp->z_curr_node = aclnode;
	aclnode->z_ace_idx = 0;
	}

	aclnode = aclp->z_curr_node;

	if (aclnode == NULL)
	return (NULL);

	if (aclnode->z_ace_idx >= aclnode->z_ace_count) {
	aclnode = list_next(&aclp->z_acl, aclnode);
	if (aclnode == NULL)
	return (NULL);
	else {
	aclp->z_curr_node = aclnode;
	aclnode->z_ace_idx = 0;
	aclp->z_next_ace = aclnode->z_acldata;
	}
	}

	if (aclnode->z_ace_idx < aclnode->z_ace_count) {
	void *acep = aclp->z_next_ace;
	size_t ace_size;

	/*
	* Make sure we don't overstep our bounds
	*/
	ace_size = aclp->z_ops->ace_size(acep);

	if (((caddr_t)acep + ace_size) >
	((caddr_t)aclnode->z_acldata + aclnode->z_size)) {
	return (NULL);
	}

	*iflags = aclp->z_ops->ace_flags_get(acep);
	*type = aclp->z_ops->ace_type_get(acep);
	*access_mask = aclp->z_ops->ace_mask_get(acep);
	*who = aclp->z_ops->ace_who_get(acep);
	aclp->z_next_ace = (caddr_t)aclp->z_next_ace + ace_size;
	aclnode->z_ace_idx++;

	return ((void *)acep);
	}
	return (NULL);
	}

	/ARGSUSED/
	static uint64_t
	zfs_ace_walk(void *datap, uint64_t cookie, int aclcnt,
	uint16_t flags, uint16_t type, uint32_t *mask)
	{
	zfs_acl_t *aclp = datap;
	zfs_ace_hdr_t acep = (zfs_ace_hdr_t )(uintptr_t)cookie;
	uint64_t who;

	acep = zfs_acl_next_ace(aclp, acep, &who, mask,
	flags, type);
	return ((uint64_t)(uintptr_t)acep);
	}

	/*
	* Copy ACE to internal ZFS format.
	* While processing the ACL each ACE will be validated for correctness.
	* ACE FUIDs will be created later.
	*/
	static int
	zfs_copy_ace_2_fuid(zfsvfs_t zfsvfs, vtype_t obj_type, zfs_acl_t aclp,
	void datap, zfs_ace_t z_acl, uint64_t aclcnt, size_t *size,
	zfs_fuid_info_t *fuidp, cred_t cr)
	{
	int i;
	uint16_t entry_type;
	zfs_ace_t *aceptr = z_acl;
	ace_t *acep = datap;
	zfs_object_ace_t *zobjacep;
	ace_object_t *aceobjp;

	for (i = 0; i != aclcnt; i++) {
	aceptr->z_hdr.z_access_mask = acep->a_access_mask;
	aceptr->z_hdr.z_flags = acep->a_flags;
	aceptr->z_hdr.z_type = acep->a_type;
	entry_type = aceptr->z_hdr.z_flags & ACE_TYPE_FLAGS;
	if (entry_type != ACE_OWNER && entry_type != OWNING_GROUP &&
	entry_type != ACE_EVERYONE) {
	aceptr->z_fuid = zfs_fuid_create(zfsvfs, acep->a_who,
	cr, (entry_type == 0) ?
	ZFS_ACE_USER : ZFS_ACE_GROUP, fuidp);
	}

	/*
	* Make sure ACE is valid
	*/
	if (zfs_ace_valid(obj_type, aclp, aceptr->z_hdr.z_type,
	aceptr->z_hdr.z_flags) != B_TRUE)
	return (SET_ERROR(EINVAL));

	switch (acep->a_type) {
	case ACE_ACCESS_ALLOWED_OBJECT_ACE_TYPE:
	case ACE_ACCESS_DENIED_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_AUDIT_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_ALARM_OBJECT_ACE_TYPE:
	zobjacep = (zfs_object_ace_t *)aceptr;
	aceobjp = (ace_object_t *)acep;

	bcopy(aceobjp->a_obj_type, zobjacep->z_object_type,
	sizeof (aceobjp->a_obj_type));
	bcopy(aceobjp->a_inherit_obj_type,
	zobjacep->z_inherit_type,
	sizeof (aceobjp->a_inherit_obj_type));
	acep = (ace_t *)((caddr_t)acep + sizeof (ace_object_t));
	break;
	default:
	acep = (ace_t *)((caddr_t)acep + sizeof (ace_t));
	}

	aceptr = (zfs_ace_t *)((caddr_t)aceptr +
	aclp->z_ops->ace_size(aceptr));
	}

	*size = (caddr_t)aceptr - (caddr_t)z_acl;

	return (0);
	}

	/*
	* Copy ZFS ACEs to fixed size ace_t layout
	*/
	static void
	zfs_copy_fuid_2_ace(zfsvfs_t zfsvfs, zfs_acl_t aclp, cred_t *cr,
	void *datap, int filter)
	{
	uint64_t who;
	uint32_t access_mask;
	uint16_t iflags, type;
	zfs_ace_hdr_t *zacep = NULL;
	ace_t *acep = datap;
	ace_object_t *objacep;
	zfs_object_ace_t *zobjacep;
	size_t ace_size;
	uint16_t entry_type;

	while ((zacep = zfs_acl_next_ace(aclp, zacep,
	&who, &access_mask, &iflags, &type))) {

	switch (type) {
	case ACE_ACCESS_ALLOWED_OBJECT_ACE_TYPE:
	case ACE_ACCESS_DENIED_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_AUDIT_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_ALARM_OBJECT_ACE_TYPE:
	if (filter) {
	continue;
	}
	zobjacep = (zfs_object_ace_t *)zacep;
	objacep = (ace_object_t *)acep;
	bcopy(zobjacep->z_object_type,
	objacep->a_obj_type,
	sizeof (zobjacep->z_object_type));
	bcopy(zobjacep->z_inherit_type,
	objacep->a_inherit_obj_type,
	sizeof (zobjacep->z_inherit_type));
	ace_size = sizeof (ace_object_t);
	break;
	default:
	ace_size = sizeof (ace_t);
	break;
	}

	entry_type = (iflags & ACE_TYPE_FLAGS);
	if ((entry_type != ACE_OWNER &&
	entry_type != OWNING_GROUP &&
	entry_type != ACE_EVERYONE)) {
	acep->a_who = zfs_fuid_map_id(zfsvfs, who,
	cr, (entry_type & ACE_IDENTIFIER_GROUP) ?
	ZFS_ACE_GROUP : ZFS_ACE_USER);
	} else {
	acep->a_who = (uid_t)(int64_t)who;
	}
	acep->a_access_mask = access_mask;
	acep->a_flags = iflags;
	acep->a_type = type;
	acep = (ace_t *)((caddr_t)acep + ace_size);
	}
	}

	static int
	zfs_copy_ace_2_oldace(vtype_t obj_type, zfs_acl_t aclp, ace_t acep,
	zfs_oldace_t z_acl, int aclcnt, size_t size)
	{
	int i;
	zfs_oldace_t *aceptr = z_acl;

	for (i = 0; i != aclcnt; i++, aceptr++) {
	aceptr->z_access_mask = acep[i].a_access_mask;
	aceptr->z_type = acep[i].a_type;
	aceptr->z_flags = acep[i].a_flags;
	aceptr->z_fuid = acep[i].a_who;
	/*
	* Make sure ACE is valid
	*/
	if (zfs_ace_valid(obj_type, aclp, aceptr->z_type,
	aceptr->z_flags) != B_TRUE)
	return (SET_ERROR(EINVAL));
	}
	*size = (caddr_t)aceptr - (caddr_t)z_acl;
	return (0);
	}

	/*
	* convert old ACL format to new
	*/
	void
	zfs_acl_xform(znode_t zp, zfs_acl_t aclp, cred_t *cr)
	{
	zfs_oldace_t *oldaclp;
	int i;
	uint16_t type, iflags;
	uint32_t access_mask;
	uint64_t who;
	void *cookie = NULL;
	zfs_acl_node_t *newaclnode;

	ASSERT(aclp->z_version == ZFS_ACL_VERSION_INITIAL);
	/*
	* First create the ACE in a contiguous piece of memory
	* for zfs_copy_ace_2_fuid().
	*
	* We only convert an ACL once, so this won't happen
	* everytime.
	*/
	oldaclp = kmem_alloc(sizeof (zfs_oldace_t) * aclp->z_acl_count,
	KM_SLEEP);
	i = 0;
	while ((cookie = zfs_acl_next_ace(aclp, cookie, &who,
	&access_mask, &iflags, &type))) {
	oldaclp[i].z_flags = iflags;
	oldaclp[i].z_type = type;
	oldaclp[i].z_fuid = who;
	oldaclp[i++].z_access_mask = access_mask;
	}

	newaclnode = zfs_acl_node_alloc(aclp->z_acl_count *
	sizeof (zfs_object_ace_t));
	aclp->z_ops = &zfs_acl_fuid_ops;
	VERIFY(zfs_copy_ace_2_fuid(zp->z_zfsvfs, ZTOV(zp)->v_type, aclp,
	oldaclp, newaclnode->z_acldata, aclp->z_acl_count,
	&newaclnode->z_size, NULL, cr) == 0);
	newaclnode->z_ace_count = aclp->z_acl_count;
	aclp->z_version = ZFS_ACL_VERSION;
	kmem_free(oldaclp, aclp->z_acl_count * sizeof (zfs_oldace_t));

	/*
	* Release all previous ACL nodes
	*/

	zfs_acl_release_nodes(aclp);

	list_insert_head(&aclp->z_acl, newaclnode);

	aclp->z_acl_bytes = newaclnode->z_size;
	aclp->z_acl_count = newaclnode->z_ace_count;

	}

	/*
	* Convert unix access mask to v4 access mask
	*/
	static uint32_t
	zfs_unix_to_v4(uint32_t access_mask)
	{
	uint32_t new_mask = 0;

	if (access_mask & S_IXOTH)
	new_mask \|= ACE_EXECUTE;
	if (access_mask & S_IWOTH)
	new_mask \|= ACE_WRITE_DATA;
	if (access_mask & S_IROTH)
	new_mask \|= ACE_READ_DATA;
	return (new_mask);
	}

	static void
	zfs_set_ace(zfs_acl_t aclp, void acep, uint32_t access_mask,
	uint16_t access_type, uint64_t fuid, uint16_t entry_type)
	{
	uint16_t type = entry_type & ACE_TYPE_FLAGS;

	aclp->z_ops->ace_mask_set(acep, access_mask);
	aclp->z_ops->ace_type_set(acep, access_type);
	aclp->z_ops->ace_flags_set(acep, entry_type);
	if ((type != ACE_OWNER && type != OWNING_GROUP &&
	type != ACE_EVERYONE))
	aclp->z_ops->ace_who_set(acep, fuid);
	}

	/*
	* Determine mode of file based on ACL.
	*/
	uint64_t
	zfs_mode_compute(uint64_t fmode, zfs_acl_t *aclp,
	uint64_t *pflags, uint64_t fuid, uint64_t fgid)
	{
	int entry_type;
	mode_t mode;
	mode_t seen = 0;
	zfs_ace_hdr_t *acep = NULL;
	uint64_t who;
	uint16_t iflags, type;
	uint32_t access_mask;
	boolean_t an_exec_denied = B_FALSE;

	mode = (fmode & (S_IFMT \| S_ISUID \| S_ISGID \| S_ISVTX));

	while ((acep = zfs_acl_next_ace(aclp, acep, &who,
	&access_mask, &iflags, &type))) {

	if (!zfs_acl_valid_ace_type(type, iflags))
	continue;

	entry_type = (iflags & ACE_TYPE_FLAGS);

	/*
	* Skip over any inherit_only ACEs
	*/
	if (iflags & ACE_INHERIT_ONLY_ACE)
	continue;

	if (entry_type == ACE_OWNER \|\| (entry_type == 0 &&
	who == fuid)) {
	if ((access_mask & ACE_READ_DATA) &&
	(!(seen & S_IRUSR))) {
	seen \|= S_IRUSR;
	if (type == ALLOW) {
	mode \|= S_IRUSR;
	}
	}
	if ((access_mask & ACE_WRITE_DATA) &&
	(!(seen & S_IWUSR))) {
	seen \|= S_IWUSR;
	if (type == ALLOW) {
	mode \|= S_IWUSR;
	}
	}
	if ((access_mask & ACE_EXECUTE) &&
	(!(seen & S_IXUSR))) {
	seen \|= S_IXUSR;
	if (type == ALLOW) {
	mode \|= S_IXUSR;
	}
	}
	} else if (entry_type == OWNING_GROUP \|\|
	(entry_type == ACE_IDENTIFIER_GROUP && who == fgid)) {
	if ((access_mask & ACE_READ_DATA) &&
	(!(seen & S_IRGRP))) {
	seen \|= S_IRGRP;
	if (type == ALLOW) {
	mode \|= S_IRGRP;
	}
	}
	if ((access_mask & ACE_WRITE_DATA) &&
	(!(seen & S_IWGRP))) {
	seen \|= S_IWGRP;
	if (type == ALLOW) {
	mode \|= S_IWGRP;
	}
	}
	if ((access_mask & ACE_EXECUTE) &&
	(!(seen & S_IXGRP))) {
	seen \|= S_IXGRP;
	if (type == ALLOW) {
	mode \|= S_IXGRP;
	}
	}
	} else if (entry_type == ACE_EVERYONE) {
	if ((access_mask & ACE_READ_DATA)) {
	if (!(seen & S_IRUSR)) {
	seen \|= S_IRUSR;
	if (type == ALLOW) {
	mode \|= S_IRUSR;
	}
	}
	if (!(seen & S_IRGRP)) {
	seen \|= S_IRGRP;
	if (type == ALLOW) {
	mode \|= S_IRGRP;
	}
	}
	if (!(seen & S_IROTH)) {
	seen \|= S_IROTH;
	if (type == ALLOW) {
	mode \|= S_IROTH;
	}
	}
	}
	if ((access_mask & ACE_WRITE_DATA)) {
	if (!(seen & S_IWUSR)) {
	seen \|= S_IWUSR;
	if (type == ALLOW) {
	mode \|= S_IWUSR;
	}
	}
	if (!(seen & S_IWGRP)) {
	seen \|= S_IWGRP;
	if (type == ALLOW) {
	mode \|= S_IWGRP;
	}
	}
	if (!(seen & S_IWOTH)) {
	seen \|= S_IWOTH;
	if (type == ALLOW) {
	mode \|= S_IWOTH;
	}
	}
	}
	if ((access_mask & ACE_EXECUTE)) {
	if (!(seen & S_IXUSR)) {
	seen \|= S_IXUSR;
	if (type == ALLOW) {
	mode \|= S_IXUSR;
	}
	}
	if (!(seen & S_IXGRP)) {
	seen \|= S_IXGRP;
	if (type == ALLOW) {
	mode \|= S_IXGRP;
	}
	}
	if (!(seen & S_IXOTH)) {
	seen \|= S_IXOTH;
	if (type == ALLOW) {
	mode \|= S_IXOTH;
	}
	}
	}
	} else {
	/*
	* Only care if this IDENTIFIER_GROUP or
	* USER ACE denies execute access to someone,
	* mode is not affected
	*/
	if ((access_mask & ACE_EXECUTE) && type == DENY)
	an_exec_denied = B_TRUE;
	}
	}

	/*
	* Failure to allow is effectively a deny, so execute permission
	* is denied if it was never mentioned or if we explicitly
	* weren't allowed it.
	*/
	if (!an_exec_denied &&
	((seen & ALL_MODE_EXECS) != ALL_MODE_EXECS \|\|
	(mode & ALL_MODE_EXECS) != ALL_MODE_EXECS))
	an_exec_denied = B_TRUE;

	if (an_exec_denied)
	*pflags &= ~ZFS_NO_EXECS_DENIED;
	else
	*pflags \|= ZFS_NO_EXECS_DENIED;

	return (mode);
	}

	/*
	* Read an external acl object. If the intent is to modify, always
	* create a new acl and leave any cached acl in place.
	*/
	int
	zfs_acl_node_read(znode_t zp, boolean_t have_lock, zfs_acl_t *aclpp,
	boolean_t will_modify)
	{
	zfs_acl_t *aclp;
	int aclsize;
	int acl_count;
	zfs_acl_node_t *aclnode;
	zfs_acl_phys_t znode_acl;
	int version;
	int error;

	ASSERT(MUTEX_HELD(&zp->z_acl_lock));
	if (zp->z_zfsvfs->z_replay == B_FALSE)
	ASSERT_VOP_LOCKED(ZTOV(zp), __func__);

	if (zp->z_acl_cached && !will_modify) {
	*aclpp = zp->z_acl_cached;
	return (0);
	}

	version = zfs_znode_acl_version(zp);

	if ((error = zfs_acl_znode_info(zp, &aclsize,
	&acl_count, &znode_acl)) != 0) {
	goto done;
	}

	aclp = zfs_acl_alloc(version);

	aclp->z_acl_count = acl_count;
	aclp->z_acl_bytes = aclsize;

	aclnode = zfs_acl_node_alloc(aclsize);
	aclnode->z_ace_count = aclp->z_acl_count;
	aclnode->z_size = aclsize;

	if (!zp->z_is_sa) {
	if (znode_acl.z_acl_extern_obj) {
	error = dmu_read(zp->z_zfsvfs->z_os,
	znode_acl.z_acl_extern_obj, 0, aclnode->z_size,
	aclnode->z_acldata, DMU_READ_PREFETCH);
	} else {
	bcopy(znode_acl.z_ace_data, aclnode->z_acldata,
	aclnode->z_size);
	}
	} else {
	error = sa_lookup(zp->z_sa_hdl, SA_ZPL_DACL_ACES(zp->z_zfsvfs),
	aclnode->z_acldata, aclnode->z_size);
	}

	if (error != 0) {
	zfs_acl_free(aclp);
	zfs_acl_node_free(aclnode);
	/* convert checksum errors into IO errors */
	if (error == ECKSUM)
	error = SET_ERROR(EIO);
	goto done;
	}

	list_insert_head(&aclp->z_acl, aclnode);

	*aclpp = aclp;
	if (!will_modify)
	zp->z_acl_cached = aclp;
	done:
	return (error);
	}

	/ARGSUSED/
	void
	zfs_acl_data_locator(void *dataptr, uint32_t length, uint32_t buflen,
	boolean_t start, void *userdata)
	{
	zfs_acl_locator_cb_t cb = (zfs_acl_locator_cb_t )userdata;

	if (start) {
	cb->cb_acl_node = list_head(&cb->cb_aclp->z_acl);
	} else {
	cb->cb_acl_node = list_next(&cb->cb_aclp->z_acl,
	cb->cb_acl_node);
	}
	*dataptr = cb->cb_acl_node->z_acldata;
	*length = cb->cb_acl_node->z_size;
	}

	int
	zfs_acl_chown_setattr(znode_t *zp)
	{
	int error;
	zfs_acl_t *aclp;

	if (zp->z_zfsvfs->z_replay == B_FALSE)
	ASSERT_VOP_ELOCKED(ZTOV(zp), __func__);
	ASSERT(MUTEX_HELD(&zp->z_acl_lock));
	ASSERT_VOP_IN_SEQC(ZTOV(zp));

	if ((error = zfs_acl_node_read(zp, B_TRUE, &aclp, B_FALSE)) == 0)
	zp->z_mode = zfs_mode_compute(zp->z_mode, aclp,
	&zp->z_pflags, zp->z_uid, zp->z_gid);
	return (error);
	}

	/*
	* common code for setting ACLs.
	*
	* This function is called from zfs_mode_update, zfs_perm_init, and zfs_setacl.
	* zfs_setacl passes a non-NULL inherit pointer (ihp) to indicate that it's
	* already checked the acl and knows whether to inherit.
	*/
	int
	zfs_aclset_common(znode_t zp, zfs_acl_t aclp, cred_t cr, dmu_tx_t tx)
	{
	int error;
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;
	dmu_object_type_t otype;
	zfs_acl_locator_cb_t locate = { 0 };
	uint64_t mode;
	sa_bulk_attr_t bulk[5];
	uint64_t ctime[2];
	int count = 0;
	zfs_acl_phys_t acl_phys;

	ASSERT_VOP_IN_SEQC(ZTOV(zp));

	mode = zp->z_mode;

	mode = zfs_mode_compute(mode, aclp, &zp->z_pflags,
	zp->z_uid, zp->z_gid);

	zp->z_mode = mode;
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_MODE(zfsvfs), NULL,
	&mode, sizeof (mode));
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_FLAGS(zfsvfs), NULL,
	&zp->z_pflags, sizeof (zp->z_pflags));
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_CTIME(zfsvfs), NULL,
	&ctime, sizeof (ctime));

	if (zp->z_acl_cached) {
	zfs_acl_free(zp->z_acl_cached);
	zp->z_acl_cached = NULL;
	}

	/*
	* Upgrade needed?
	*/
	if (!zfsvfs->z_use_fuids) {
	otype = DMU_OT_OLDACL;
	} else {
	if ((aclp->z_version == ZFS_ACL_VERSION_INITIAL) &&
	(zfsvfs->z_version >= ZPL_VERSION_FUID))
	zfs_acl_xform(zp, aclp, cr);
	ASSERT(aclp->z_version >= ZFS_ACL_VERSION_FUID);
	otype = DMU_OT_ACL;
	}

	/*
	* Arrgh, we have to handle old on disk format
	* as well as newer (preferred) SA format.
	*/

	if (zp->z_is_sa) { /* the easy case, just update the ACL attribute */
	locate.cb_aclp = aclp;
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_DACL_ACES(zfsvfs),
	zfs_acl_data_locator, &locate, aclp->z_acl_bytes);
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_DACL_COUNT(zfsvfs),
	NULL, &aclp->z_acl_count, sizeof (uint64_t));
	} else { /* Painful legacy way */
	zfs_acl_node_t *aclnode;
	uint64_t off = 0;
	uint64_t aoid;

	if ((error = sa_lookup(zp->z_sa_hdl, SA_ZPL_ZNODE_ACL(zfsvfs),
	&acl_phys, sizeof (acl_phys))) != 0)
	return (error);

	aoid = acl_phys.z_acl_extern_obj;

	if (aclp->z_acl_bytes > ZFS_ACE_SPACE) {
	/*
	* If ACL was previously external and we are now
	* converting to new ACL format then release old
	* ACL object and create a new one.
	*/
	if (aoid &&
	aclp->z_version != acl_phys.z_acl_version) {
	error = dmu_object_free(zfsvfs->z_os, aoid, tx);
	if (error)
	return (error);
	aoid = 0;
	}
	if (aoid == 0) {
	aoid = dmu_object_alloc(zfsvfs->z_os,
	otype, aclp->z_acl_bytes,
	otype == DMU_OT_ACL ?
	DMU_OT_SYSACL : DMU_OT_NONE,
	otype == DMU_OT_ACL ?
	DN_OLD_MAX_BONUSLEN : 0, tx);
	} else {
	(void) dmu_object_set_blocksize(zfsvfs->z_os,
	aoid, aclp->z_acl_bytes, 0, tx);
	}
	acl_phys.z_acl_extern_obj = aoid;
	for (aclnode = list_head(&aclp->z_acl); aclnode;
	aclnode = list_next(&aclp->z_acl, aclnode)) {
	if (aclnode->z_ace_count == 0)
	continue;
	dmu_write(zfsvfs->z_os, aoid, off,
	aclnode->z_size, aclnode->z_acldata, tx);
	off += aclnode->z_size;
	}
	} else {
	void *start = acl_phys.z_ace_data;
	/*
	* Migrating back embedded?
	*/
	if (acl_phys.z_acl_extern_obj) {
	error = dmu_object_free(zfsvfs->z_os,
	acl_phys.z_acl_extern_obj, tx);
	if (error)
	return (error);
	acl_phys.z_acl_extern_obj = 0;
	}

	for (aclnode = list_head(&aclp->z_acl); aclnode;
	aclnode = list_next(&aclp->z_acl, aclnode)) {
	if (aclnode->z_ace_count == 0)
	continue;
	bcopy(aclnode->z_acldata, start,
	aclnode->z_size);
	start = (caddr_t)start + aclnode->z_size;
	}
	}
	/*
	* If Old version then swap count/bytes to match old
	* layout of znode_acl_phys_t.
	*/
	if (aclp->z_version == ZFS_ACL_VERSION_INITIAL) {
	acl_phys.z_acl_size = aclp->z_acl_count;
	acl_phys.z_acl_count = aclp->z_acl_bytes;
	} else {
	acl_phys.z_acl_size = aclp->z_acl_bytes;
	acl_phys.z_acl_count = aclp->z_acl_count;
	}
	acl_phys.z_acl_version = aclp->z_version;

	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_ZNODE_ACL(zfsvfs), NULL,
	&acl_phys, sizeof (acl_phys));
	}

	/*
	* Replace ACL wide bits, but first clear them.
	*/
	zp->z_pflags &= ~ZFS_ACL_WIDE_FLAGS;

	zp->z_pflags \|= aclp->z_hints;

	if (ace_trivial_common(aclp, 0, zfs_ace_walk) == 0)
	zp->z_pflags \|= ZFS_ACL_TRIVIAL;

	zfs_tstamp_update_setup(zp, STATE_CHANGED, NULL, ctime);
	return (sa_bulk_update(zp->z_sa_hdl, bulk, count, tx));
	}

	static void
	zfs_acl_chmod(vtype_t vtype, uint64_t mode, boolean_t split, boolean_t trim,
	zfs_acl_t *aclp)
	{
	void *acep = NULL;
	uint64_t who;
	int new_count, new_bytes;
	int ace_size;
	int entry_type;
	uint16_t iflags, type;
	uint32_t access_mask;
	zfs_acl_node_t *newnode;
	size_t abstract_size = aclp->z_ops->ace_abstract_size();
	void *zacep;
	boolean_t isdir;
	trivial_acl_t masks;

	new_count = new_bytes = 0;

	isdir = (vtype == VDIR);

	acl_trivial_access_masks((mode_t)mode, isdir, &masks);

	newnode = zfs_acl_node_alloc((abstract_size * 6) + aclp->z_acl_bytes);

	zacep = newnode->z_acldata;
	if (masks.allow0) {
	zfs_set_ace(aclp, zacep, masks.allow0, ALLOW, -1, ACE_OWNER);
	zacep = (void *)((uintptr_t)zacep + abstract_size);
	new_count++;
	new_bytes += abstract_size;
	}
	if (masks.deny1) {
	zfs_set_ace(aclp, zacep, masks.deny1, DENY, -1, ACE_OWNER);
	zacep = (void *)((uintptr_t)zacep + abstract_size);
	new_count++;
	new_bytes += abstract_size;
	}
	if (masks.deny2) {
	zfs_set_ace(aclp, zacep, masks.deny2, DENY, -1, OWNING_GROUP);
	zacep = (void *)((uintptr_t)zacep + abstract_size);
	new_count++;
	new_bytes += abstract_size;
	}

	while ((acep = zfs_acl_next_ace(aclp, acep, &who, &access_mask,
	&iflags, &type))) {
	entry_type = (iflags & ACE_TYPE_FLAGS);
	/*
	* ACEs used to represent the file mode may be divided
	* into an equivalent pair of inherit-only and regular
	* ACEs, if they are inheritable.
	* Skip regular ACEs, which are replaced by the new mode.
	*/
	if (split && (entry_type == ACE_OWNER \|\|
	entry_type == OWNING_GROUP \|\|
	entry_type == ACE_EVERYONE)) {
	if (!isdir \|\| !(iflags &
	(ACE_FILE_INHERIT_ACE\|ACE_DIRECTORY_INHERIT_ACE)))
	continue;
	/*
	* We preserve owner@, group@, or @everyone
	* permissions, if they are inheritable, by
	* copying them to inherit_only ACEs. This
	* prevents inheritable permissions from being
	* altered along with the file mode.
	*/
	iflags \|= ACE_INHERIT_ONLY_ACE;
	}

	/*
	* If this ACL has any inheritable ACEs, mark that in
	* the hints (which are later masked into the pflags)
	* so create knows to do inheritance.
	*/
	if (isdir && (iflags &
	(ACE_FILE_INHERIT_ACE\|ACE_DIRECTORY_INHERIT_ACE)))
	aclp->z_hints \|= ZFS_INHERIT_ACE;

	if ((type != ALLOW && type != DENY) \|\|
	(iflags & ACE_INHERIT_ONLY_ACE)) {
	switch (type) {
	case ACE_ACCESS_ALLOWED_OBJECT_ACE_TYPE:
	case ACE_ACCESS_DENIED_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_AUDIT_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_ALARM_OBJECT_ACE_TYPE:
	aclp->z_hints \|= ZFS_ACL_OBJ_ACE;
	break;
	}
	} else {
	/*
	* Limit permissions granted by ACEs to be no greater
	* than permissions of the requested group mode.
	* Applies when the "aclmode" property is set to
	* "groupmask".
	*/
	if ((type == ALLOW) && trim)
	access_mask &= masks.group;
	}
	zfs_set_ace(aclp, zacep, access_mask, type, who, iflags);
	ace_size = aclp->z_ops->ace_size(acep);
	zacep = (void *)((uintptr_t)zacep + ace_size);
	new_count++;
	new_bytes += ace_size;
	}
	zfs_set_ace(aclp, zacep, masks.owner, ALLOW, -1, ACE_OWNER);
	zacep = (void *)((uintptr_t)zacep + abstract_size);
	zfs_set_ace(aclp, zacep, masks.group, ALLOW, -1, OWNING_GROUP);
	zacep = (void *)((uintptr_t)zacep + abstract_size);
	zfs_set_ace(aclp, zacep, masks.everyone, ALLOW, -1, ACE_EVERYONE);

	new_count += 3;
	new_bytes += abstract_size * 3;
	zfs_acl_release_nodes(aclp);
	aclp->z_acl_count = new_count;
	aclp->z_acl_bytes = new_bytes;
	newnode->z_ace_count = new_count;
	newnode->z_size = new_bytes;
	list_insert_tail(&aclp->z_acl, newnode);
	}

	int
	zfs_acl_chmod_setattr(znode_t zp, zfs_acl_t *aclp, uint64_t mode)
	{
	int error = 0;

	mutex_enter(&zp->z_acl_lock);
	if (zp->z_zfsvfs->z_replay == B_FALSE)
	ASSERT_VOP_ELOCKED(ZTOV(zp), __func__);
	if (zp->z_zfsvfs->z_acl_mode == ZFS_ACL_DISCARD)
	*aclp = zfs_acl_alloc(zfs_acl_version_zp(zp));
	else
	error = zfs_acl_node_read(zp, B_TRUE, aclp, B_TRUE);

	if (error == 0) {
	(*aclp)->z_hints = zp->z_pflags & V4_ACL_WIDE_FLAGS;
	zfs_acl_chmod(ZTOV(zp)->v_type, mode, B_TRUE,
	(zp->z_zfsvfs->z_acl_mode == ZFS_ACL_GROUPMASK), *aclp);
	}
	mutex_exit(&zp->z_acl_lock);

	return (error);
	}

	/*
	* Should ACE be inherited?
	*/
	static int
	zfs_ace_can_use(vtype_t vtype, uint16_t acep_flags)
	{
	int iflags = (acep_flags & 0xf);

	if ((vtype == VDIR) && (iflags & ACE_DIRECTORY_INHERIT_ACE))
	return (1);
	else if (iflags & ACE_FILE_INHERIT_ACE)
	return (!((vtype == VDIR) &&
	(iflags & ACE_NO_PROPAGATE_INHERIT_ACE)));
	return (0);
	}

	/*
	* inherit inheritable ACEs from parent
	*/
	static zfs_acl_t *
	zfs_acl_inherit(zfsvfs_t zfsvfs, vtype_t vtype, zfs_acl_t paclp,
	uint64_t mode, boolean_t *need_chmod)
	{
	void *pacep = NULL;
	void *acep;
	zfs_acl_node_t *aclnode;
	zfs_acl_t *aclp = NULL;
	uint64_t who;
	uint32_t access_mask;
	uint16_t iflags, newflags, type;
	size_t ace_size;
	void data1, data2;
	size_t data1sz, data2sz;
	uint_t aclinherit;
	boolean_t isdir = (vtype == VDIR);
	boolean_t isreg = (vtype == VREG);

	*need_chmod = B_TRUE;

	aclp = zfs_acl_alloc(paclp->z_version);
	aclinherit = zfsvfs->z_acl_inherit;
	if (aclinherit == ZFS_ACL_DISCARD \|\| vtype == VLNK)
	return (aclp);

	while ((pacep = zfs_acl_next_ace(paclp, pacep, &who,
	&access_mask, &iflags, &type))) {

	/*
	* don't inherit bogus ACEs
	*/
	if (!zfs_acl_valid_ace_type(type, iflags))
	continue;

	/*
	* Check if ACE is inheritable by this vnode
	*/
	if ((aclinherit == ZFS_ACL_NOALLOW && type == ALLOW) \|\|
	!zfs_ace_can_use(vtype, iflags))
	continue;

	/*
	* If owner@, group@, or everyone@ inheritable
	* then zfs_acl_chmod() isn't needed.
	*/
	if ((aclinherit == ZFS_ACL_PASSTHROUGH \|\|
	aclinherit == ZFS_ACL_PASSTHROUGH_X) &&
	((iflags & (ACE_OWNER\|ACE_EVERYONE)) \|\|
	((iflags & OWNING_GROUP) == OWNING_GROUP)) &&
	(isreg \|\| (isdir && (iflags & ACE_DIRECTORY_INHERIT_ACE))))
	*need_chmod = B_FALSE;

	/*
	* Strip inherited execute permission from file if
	* not in mode
	*/
	if (aclinherit == ZFS_ACL_PASSTHROUGH_X && type == ALLOW &&
	!isdir && ((mode & (S_IXUSR\|S_IXGRP\|S_IXOTH)) == 0)) {
	access_mask &= ~ACE_EXECUTE;
	}

	/*
	* Strip write_acl and write_owner from permissions
	* when inheriting an ACE
	*/
	if (aclinherit == ZFS_ACL_RESTRICTED && type == ALLOW) {
	access_mask &= ~RESTRICTED_CLEAR;
	}

	ace_size = aclp->z_ops->ace_size(pacep);
	aclnode = zfs_acl_node_alloc(ace_size);
	list_insert_tail(&aclp->z_acl, aclnode);
	acep = aclnode->z_acldata;

	zfs_set_ace(aclp, acep, access_mask, type,
	who, iflags\|ACE_INHERITED_ACE);

	/*
	* Copy special opaque data if any
	*/
	if ((data1sz = paclp->z_ops->ace_data(pacep, &data1)) != 0) {
	VERIFY((data2sz = aclp->z_ops->ace_data(acep,
	&data2)) == data1sz);
	bcopy(data1, data2, data2sz);
	}

	aclp->z_acl_count++;
	aclnode->z_ace_count++;
	aclp->z_acl_bytes += aclnode->z_size;
	newflags = aclp->z_ops->ace_flags_get(acep);

	/*
	* If ACE is not to be inherited further, or if the vnode is
	* not a directory, remove all inheritance flags
	*/
	if (!isdir \|\| (iflags & ACE_NO_PROPAGATE_INHERIT_ACE)) {
	newflags &= ~ALL_INHERIT;
	aclp->z_ops->ace_flags_set(acep,
	newflags\|ACE_INHERITED_ACE);
	continue;
	}

	/*
	* This directory has an inheritable ACE
	*/
	aclp->z_hints \|= ZFS_INHERIT_ACE;

	/*
	* If only FILE_INHERIT is set then turn on
	* inherit_only
	*/
	if ((iflags & (ACE_FILE_INHERIT_ACE \|
	ACE_DIRECTORY_INHERIT_ACE)) == ACE_FILE_INHERIT_ACE) {
	newflags \|= ACE_INHERIT_ONLY_ACE;
	aclp->z_ops->ace_flags_set(acep,
	newflags\|ACE_INHERITED_ACE);
	} else {
	newflags &= ~ACE_INHERIT_ONLY_ACE;
	aclp->z_ops->ace_flags_set(acep,
	newflags\|ACE_INHERITED_ACE);
	}
	}
	if (zfsvfs->z_acl_mode == ZFS_ACL_RESTRICTED &&
	aclp->z_acl_count != 0) {
	*need_chmod = B_FALSE;
	}

	return (aclp);
	}

	/*
	* Create file system object initial permissions
	* including inheritable ACEs.
	* Also, create FUIDs for owner and group.
	*/
	int
	zfs_acl_ids_create(znode_t dzp, int flag, vattr_t vap, cred_t *cr,
	vsecattr_t vsecp, zfs_acl_ids_t acl_ids)
	{
	int error;
	zfsvfs_t *zfsvfs = dzp->z_zfsvfs;
	zfs_acl_t *paclp;
	gid_t gid;
	boolean_t need_chmod = B_TRUE;
	boolean_t trim = B_FALSE;
	boolean_t inherited = B_FALSE;

	if ((flag & IS_ROOT_NODE) == 0) {
	if (zfsvfs->z_replay == B_FALSE)
	ASSERT_VOP_ELOCKED(ZTOV(dzp), __func__);
	} else
	ASSERT(dzp->z_vnode == NULL);
	bzero(acl_ids, sizeof (zfs_acl_ids_t));
	acl_ids->z_mode = MAKEIMODE(vap->va_type, vap->va_mode);

	if (vsecp)
	if ((error = zfs_vsec_2_aclp(zfsvfs, vap->va_type, vsecp, cr,
	&acl_ids->z_fuidp, &acl_ids->z_aclp)) != 0)
	return (error);
	/*
	* Determine uid and gid.
	*/
	if ((flag & IS_ROOT_NODE) \|\| zfsvfs->z_replay \|\|
	((flag & IS_XATTR) && (vap->va_type == VDIR))) {
	acl_ids->z_fuid = zfs_fuid_create(zfsvfs,
	(uint64_t)vap->va_uid, cr,
	ZFS_OWNER, &acl_ids->z_fuidp);
	acl_ids->z_fgid = zfs_fuid_create(zfsvfs,
	(uint64_t)vap->va_gid, cr,
	ZFS_GROUP, &acl_ids->z_fuidp);
	gid = vap->va_gid;
	} else {
	acl_ids->z_fuid = zfs_fuid_create_cred(zfsvfs, ZFS_OWNER,
	cr, &acl_ids->z_fuidp);
	acl_ids->z_fgid = 0;
	if (vap->va_mask & AT_GID) {
	acl_ids->z_fgid = zfs_fuid_create(zfsvfs,
	(uint64_t)vap->va_gid,
	cr, ZFS_GROUP, &acl_ids->z_fuidp);
	gid = vap->va_gid;
	if (acl_ids->z_fgid != dzp->z_gid &&
	!groupmember(vap->va_gid, cr) &&
	secpolicy_vnode_create_gid(cr) != 0)
	acl_ids->z_fgid = 0;
	}
	if (acl_ids->z_fgid == 0) {
	char *domain;
	uint32_t rid;

	acl_ids->z_fgid = dzp->z_gid;
	gid = zfs_fuid_map_id(zfsvfs, acl_ids->z_fgid,
	cr, ZFS_GROUP);

	if (zfsvfs->z_use_fuids &&
	IS_EPHEMERAL(acl_ids->z_fgid)) {
	domain =
	zfs_fuid_idx_domain(&zfsvfs->z_fuid_idx,
	FUID_INDEX(acl_ids->z_fgid));
	rid = FUID_RID(acl_ids->z_fgid);
	zfs_fuid_node_add(&acl_ids->z_fuidp,
	domain, rid, FUID_INDEX(acl_ids->z_fgid),
	acl_ids->z_fgid, ZFS_GROUP);
	}
	}
	}

	/*
	* If we're creating a directory, and the parent directory has the
	* set-GID bit set, set in on the new directory.
	* Otherwise, if the user is neither privileged nor a member of the
	* file's new group, clear the file's set-GID bit.
	*/

	if (!(flag & IS_ROOT_NODE) && (dzp->z_mode & S_ISGID) &&
	(vap->va_type == VDIR)) {
	acl_ids->z_mode \|= S_ISGID;
	} else {
	if ((acl_ids->z_mode & S_ISGID) &&
	secpolicy_vnode_setids_setgids(ZTOV(dzp), cr, gid) != 0)
	acl_ids->z_mode &= ~S_ISGID;
	}

	if (acl_ids->z_aclp == NULL) {
	mutex_enter(&dzp->z_acl_lock);
	if (!(flag & IS_ROOT_NODE) &&
	(dzp->z_pflags & ZFS_INHERIT_ACE) &&
	!(dzp->z_pflags & ZFS_XATTR)) {
	VERIFY0(zfs_acl_node_read(dzp, B_TRUE,
	&paclp, B_FALSE));
	acl_ids->z_aclp = zfs_acl_inherit(zfsvfs,
	vap->va_type, paclp, acl_ids->z_mode, &need_chmod);
	inherited = B_TRUE;
	} else {
	acl_ids->z_aclp =
	zfs_acl_alloc(zfs_acl_version_zp(dzp));
	acl_ids->z_aclp->z_hints \|= ZFS_ACL_TRIVIAL;
	}
	mutex_exit(&dzp->z_acl_lock);

	if (need_chmod) {
	if (vap->va_type == VDIR)
	acl_ids->z_aclp->z_hints \|=
	ZFS_ACL_AUTO_INHERIT;

	if (zfsvfs->z_acl_mode == ZFS_ACL_GROUPMASK &&
	zfsvfs->z_acl_inherit != ZFS_ACL_PASSTHROUGH &&
	zfsvfs->z_acl_inherit != ZFS_ACL_PASSTHROUGH_X)
	trim = B_TRUE;
	zfs_acl_chmod(vap->va_type, acl_ids->z_mode, B_FALSE,
	trim, acl_ids->z_aclp);
	}
	}

	if (inherited \|\| vsecp) {
	acl_ids->z_mode = zfs_mode_compute(acl_ids->z_mode,
	acl_ids->z_aclp, &acl_ids->z_aclp->z_hints,
	acl_ids->z_fuid, acl_ids->z_fgid);
	if (ace_trivial_common(acl_ids->z_aclp, 0, zfs_ace_walk) == 0)
	acl_ids->z_aclp->z_hints \|= ZFS_ACL_TRIVIAL;
	}

	return (0);
	}

	/*
	* Free ACL and fuid_infop, but not the acl_ids structure
	*/
	void
	zfs_acl_ids_free(zfs_acl_ids_t *acl_ids)
	{
	if (acl_ids->z_aclp)
	zfs_acl_free(acl_ids->z_aclp);
	if (acl_ids->z_fuidp)
	zfs_fuid_info_free(acl_ids->z_fuidp);
	acl_ids->z_aclp = NULL;
	acl_ids->z_fuidp = NULL;
	}

	boolean_t
	zfs_acl_ids_overquota(zfsvfs_t zv, zfs_acl_ids_t acl_ids, uint64_t projid)
	{
	return (zfs_id_overquota(zv, DMU_USERUSED_OBJECT, acl_ids->z_fuid) \|\|
	zfs_id_overquota(zv, DMU_GROUPUSED_OBJECT, acl_ids->z_fgid) \|\|
	(projid != ZFS_DEFAULT_PROJID && projid != ZFS_INVALID_PROJID &&
	zfs_id_overquota(zv, DMU_PROJECTUSED_OBJECT, projid)));
	}

	/*
	* Retrieve a file's ACL
	*/
	int
	zfs_getacl(znode_t zp, vsecattr_t vsecp, boolean_t skipaclchk, cred_t *cr)
	{
	zfs_acl_t *aclp;
	ulong_t mask;
	int error;
	int count = 0;
	int largeace = 0;

	mask = vsecp->vsa_mask & (VSA_ACE \| VSA_ACECNT \|
	VSA_ACE_ACLFLAGS \| VSA_ACE_ALLTYPES);

	if (mask == 0)
	return (SET_ERROR(ENOSYS));

	if ((error = zfs_zaccess(zp, ACE_READ_ACL, 0, skipaclchk, cr)))
	return (error);

	mutex_enter(&zp->z_acl_lock);

	if (zp->z_zfsvfs->z_replay == B_FALSE)
	ASSERT_VOP_LOCKED(ZTOV(zp), __func__);
	error = zfs_acl_node_read(zp, B_TRUE, &aclp, B_FALSE);
	if (error != 0) {
	mutex_exit(&zp->z_acl_lock);
	return (error);
	}

	/*
	* Scan ACL to determine number of ACEs
	*/
	if ((zp->z_pflags & ZFS_ACL_OBJ_ACE) && !(mask & VSA_ACE_ALLTYPES)) {
	void *zacep = NULL;
	uint64_t who;
	uint32_t access_mask;
	uint16_t type, iflags;

	while ((zacep = zfs_acl_next_ace(aclp, zacep,
	&who, &access_mask, &iflags, &type))) {
	switch (type) {
	case ACE_ACCESS_ALLOWED_OBJECT_ACE_TYPE:
	case ACE_ACCESS_DENIED_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_AUDIT_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_ALARM_OBJECT_ACE_TYPE:
	largeace++;
	continue;
	default:
	count++;
	}
	}
	vsecp->vsa_aclcnt = count;
	} else
	count = (int)aclp->z_acl_count;

	if (mask & VSA_ACECNT) {
	vsecp->vsa_aclcnt = count;
	}

	if (mask & VSA_ACE) {
	size_t aclsz;

	aclsz = count * sizeof (ace_t) +
	sizeof (ace_object_t) * largeace;

	vsecp->vsa_aclentp = kmem_alloc(aclsz, KM_SLEEP);
	vsecp->vsa_aclentsz = aclsz;

	if (aclp->z_version == ZFS_ACL_VERSION_FUID)
	zfs_copy_fuid_2_ace(zp->z_zfsvfs, aclp, cr,
	vsecp->vsa_aclentp, !(mask & VSA_ACE_ALLTYPES));
	else {
	zfs_acl_node_t *aclnode;
	void *start = vsecp->vsa_aclentp;

	for (aclnode = list_head(&aclp->z_acl); aclnode;
	aclnode = list_next(&aclp->z_acl, aclnode)) {
	bcopy(aclnode->z_acldata, start,
	aclnode->z_size);
	start = (caddr_t)start + aclnode->z_size;
	}
	ASSERT((caddr_t)start - (caddr_t)vsecp->vsa_aclentp ==
	aclp->z_acl_bytes);
	}
	}
	if (mask & VSA_ACE_ACLFLAGS) {
	vsecp->vsa_aclflags = 0;
	if (zp->z_pflags & ZFS_ACL_DEFAULTED)
	vsecp->vsa_aclflags \|= ACL_DEFAULTED;
	if (zp->z_pflags & ZFS_ACL_PROTECTED)
	vsecp->vsa_aclflags \|= ACL_PROTECTED;
	if (zp->z_pflags & ZFS_ACL_AUTO_INHERIT)
	vsecp->vsa_aclflags \|= ACL_AUTO_INHERIT;
	}

	mutex_exit(&zp->z_acl_lock);

	return (0);
	}

	int
	zfs_vsec_2_aclp(zfsvfs_t *zfsvfs, umode_t obj_type,
	vsecattr_t vsecp, cred_t cr, zfs_fuid_info_t fuidp, zfs_acl_t zaclp)
	{
	zfs_acl_t *aclp;
	zfs_acl_node_t *aclnode;
	int aclcnt = vsecp->vsa_aclcnt;
	int error;

	if (vsecp->vsa_aclcnt > MAX_ACL_ENTRIES \|\| vsecp->vsa_aclcnt <= 0)
	return (SET_ERROR(EINVAL));

	aclp = zfs_acl_alloc(zfs_acl_version(zfsvfs->z_version));

	aclp->z_hints = 0;
	aclnode = zfs_acl_node_alloc(aclcnt * sizeof (zfs_object_ace_t));
	if (aclp->z_version == ZFS_ACL_VERSION_INITIAL) {
	if ((error = zfs_copy_ace_2_oldace(obj_type, aclp,
	(ace_t *)vsecp->vsa_aclentp, aclnode->z_acldata,
	aclcnt, &aclnode->z_size)) != 0) {
	zfs_acl_free(aclp);
	zfs_acl_node_free(aclnode);
	return (error);
	}
	} else {
	if ((error = zfs_copy_ace_2_fuid(zfsvfs, obj_type, aclp,
	vsecp->vsa_aclentp, aclnode->z_acldata, aclcnt,
	&aclnode->z_size, fuidp, cr)) != 0) {
	zfs_acl_free(aclp);
	zfs_acl_node_free(aclnode);
	return (error);
	}
	}
	aclp->z_acl_bytes = aclnode->z_size;
	aclnode->z_ace_count = aclcnt;
	aclp->z_acl_count = aclcnt;
	list_insert_head(&aclp->z_acl, aclnode);

	/*
	* If flags are being set then add them to z_hints
	*/
	if (vsecp->vsa_mask & VSA_ACE_ACLFLAGS) {
	if (vsecp->vsa_aclflags & ACL_PROTECTED)
	aclp->z_hints \|= ZFS_ACL_PROTECTED;
	if (vsecp->vsa_aclflags & ACL_DEFAULTED)
	aclp->z_hints \|= ZFS_ACL_DEFAULTED;
	if (vsecp->vsa_aclflags & ACL_AUTO_INHERIT)
	aclp->z_hints \|= ZFS_ACL_AUTO_INHERIT;
	}

	*zaclp = aclp;

	return (0);
	}

	/*
	* Set a file's ACL
	*/
	int
	zfs_setacl(znode_t zp, vsecattr_t vsecp, boolean_t skipaclchk, cred_t *cr)
	{
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;
	zilog_t *zilog = zfsvfs->z_log;
	ulong_t mask = vsecp->vsa_mask & (VSA_ACE \| VSA_ACECNT);
	dmu_tx_t *tx;
	int error;
	zfs_acl_t *aclp;
	zfs_fuid_info_t *fuidp = NULL;
	boolean_t fuid_dirtied;
	uint64_t acl_obj;

	if (zp->z_zfsvfs->z_replay == B_FALSE)
	ASSERT_VOP_ELOCKED(ZTOV(zp), __func__);
	if (mask == 0)
	return (SET_ERROR(ENOSYS));

	if (zp->z_pflags & ZFS_IMMUTABLE)
	return (SET_ERROR(EPERM));

	if ((error = zfs_zaccess(zp, ACE_WRITE_ACL, 0, skipaclchk, cr)))
	return (error);

	error = zfs_vsec_2_aclp(zfsvfs, ZTOV(zp)->v_type, vsecp, cr, &fuidp,
	&aclp);
	if (error)
	return (error);

	/*
	* If ACL wide flags aren't being set then preserve any
	* existing flags.
	*/
	if (!(vsecp->vsa_mask & VSA_ACE_ACLFLAGS)) {
	aclp->z_hints \|=
	(zp->z_pflags & V4_ACL_WIDE_FLAGS);
	}
	top:
	mutex_enter(&zp->z_acl_lock);

	tx = dmu_tx_create(zfsvfs->z_os);

	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_TRUE);

	fuid_dirtied = zfsvfs->z_fuid_dirty;
	if (fuid_dirtied)
	zfs_fuid_txhold(zfsvfs, tx);

	/*
	* If old version and ACL won't fit in bonus and we aren't
	* upgrading then take out necessary DMU holds
	*/

	if ((acl_obj = zfs_external_acl(zp)) != 0) {
	if (zfsvfs->z_version >= ZPL_VERSION_FUID &&
	zfs_znode_acl_version(zp) <= ZFS_ACL_VERSION_INITIAL) {
	dmu_tx_hold_free(tx, acl_obj, 0,
	DMU_OBJECT_END);
	dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0,
	aclp->z_acl_bytes);
	} else {
	dmu_tx_hold_write(tx, acl_obj, 0, aclp->z_acl_bytes);
	}
	} else if (!zp->z_is_sa && aclp->z_acl_bytes > ZFS_ACE_SPACE) {
	dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0, aclp->z_acl_bytes);
	}

	zfs_sa_upgrade_txholds(tx, zp);
	error = dmu_tx_assign(tx, TXG_NOWAIT);
	if (error) {
	mutex_exit(&zp->z_acl_lock);

	if (error == ERESTART) {
	dmu_tx_wait(tx);
	dmu_tx_abort(tx);
	goto top;
	}
	dmu_tx_abort(tx);
	zfs_acl_free(aclp);
	return (error);
	}

	error = zfs_aclset_common(zp, aclp, cr, tx);
	ASSERT(error == 0);
	ASSERT(zp->z_acl_cached == NULL);
	zp->z_acl_cached = aclp;

	if (fuid_dirtied)
	zfs_fuid_sync(zfsvfs, tx);

	zfs_log_acl(zilog, tx, zp, vsecp, fuidp);

	if (fuidp)
	zfs_fuid_info_free(fuidp);
	dmu_tx_commit(tx);
	mutex_exit(&zp->z_acl_lock);

	return (error);
	}

	/*
	* Check accesses of interest (AoI) against attributes of the dataset
	* such as read-only. Returns zero if no AoI conflict with dataset
	* attributes, otherwise an appropriate errno is returned.
	*/
	static int
	zfs_zaccess_dataset_check(znode_t *zp, uint32_t v4_mode)
	{
	if ((v4_mode & WRITE_MASK) &&
	(zp->z_zfsvfs->z_vfs->vfs_flag & VFS_RDONLY) &&
	(!IS_DEVVP(ZTOV(zp)) \|\|
	(IS_DEVVP(ZTOV(zp)) && (v4_mode & WRITE_MASK_ATTRS)))) {
	return (SET_ERROR(EROFS));
	}

	/*
	* Intentionally allow ZFS_READONLY through here.
	* See zfs_zaccess_common().
	*/
	if ((v4_mode & WRITE_MASK_DATA) &&
	(zp->z_pflags & ZFS_IMMUTABLE)) {
	return (SET_ERROR(EPERM));
	}

	/*
	* In FreeBSD we allow to modify directory's content is ZFS_NOUNLINK
	* (sunlnk) is set. We just don't allow directory removal, which is
	* handled in zfs_zaccess_delete().
	*/
	if ((v4_mode & ACE_DELETE) &&
	(zp->z_pflags & ZFS_NOUNLINK)) {
	return (EPERM);
	}

	if (((v4_mode & (ACE_READ_DATA\|ACE_EXECUTE)) &&
	(zp->z_pflags & ZFS_AV_QUARANTINED))) {
	return (SET_ERROR(EACCES));
	}

	return (0);
	}

	/*
	* The primary usage of this function is to loop through all of the
	* ACEs in the znode, determining what accesses of interest (AoI) to
	* the caller are allowed or denied. The AoI are expressed as bits in
	* the working_mode parameter. As each ACE is processed, bits covered
	* by that ACE are removed from the working_mode. This removal
	* facilitates two things. The first is that when the working mode is
	* empty (= 0), we know we've looked at all the AoI. The second is
	* that the ACE interpretation rules don't allow a later ACE to undo
	* something granted or denied by an earlier ACE. Removing the
	* discovered access or denial enforces this rule. At the end of
	* processing the ACEs, all AoI that were found to be denied are
	* placed into the working_mode, giving the caller a mask of denied
	* accesses. Returns:
	* 0 if all AoI granted
	* EACCESS if the denied mask is non-zero
	* other error if abnormal failure (e.g., IO error)
	*
	* A secondary usage of the function is to determine if any of the
	* AoI are granted. If an ACE grants any access in
	* the working_mode, we immediately short circuit out of the function.
	* This mode is chosen by setting anyaccess to B_TRUE. The
	* working_mode is not a denied access mask upon exit if the function
	* is used in this manner.
	*/
	static int
	zfs_zaccess_aces_check(znode_t zp, uint32_t working_mode,
	boolean_t anyaccess, cred_t *cr)
	{
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;
	zfs_acl_t *aclp;
	int error;
	uid_t uid = crgetuid(cr);
	uint64_t who;
	uint16_t type, iflags;
	uint16_t entry_type;
	uint32_t access_mask;
	uint32_t deny_mask = 0;
	zfs_ace_hdr_t *acep = NULL;
	boolean_t checkit;
	uid_t gowner;
	uid_t fowner;

	zfs_fuid_map_ids(zp, cr, &fowner, &gowner);

	mutex_enter(&zp->z_acl_lock);

	if (zp->z_zfsvfs->z_replay == B_FALSE)
	ASSERT_VOP_LOCKED(ZTOV(zp), __func__);
	error = zfs_acl_node_read(zp, B_TRUE, &aclp, B_FALSE);
	if (error != 0) {
	mutex_exit(&zp->z_acl_lock);
	return (error);
	}

	ASSERT(zp->z_acl_cached);

	while ((acep = zfs_acl_next_ace(aclp, acep, &who, &access_mask,
	&iflags, &type))) {
	uint32_t mask_matched;

	if (!zfs_acl_valid_ace_type(type, iflags))
	continue;

	if (ZTOV(zp)->v_type == VDIR && (iflags & ACE_INHERIT_ONLY_ACE))
	continue;

	/* Skip ACE if it does not affect any AoI */
	mask_matched = (access_mask & *working_mode);
	if (!mask_matched)
	continue;

	entry_type = (iflags & ACE_TYPE_FLAGS);

	checkit = B_FALSE;

	switch (entry_type) {
	case ACE_OWNER:
	if (uid == fowner)
	checkit = B_TRUE;
	break;
	case OWNING_GROUP:
	who = gowner;
	/FALLTHROUGH/
	case ACE_IDENTIFIER_GROUP:
	checkit = zfs_groupmember(zfsvfs, who, cr);
	break;
	case ACE_EVERYONE:
	checkit = B_TRUE;
	break;

	/* USER Entry */
	default:
	if (entry_type == 0) {
	uid_t newid;

	newid = zfs_fuid_map_id(zfsvfs, who, cr,
	ZFS_ACE_USER);
	if (newid != UID_NOBODY &&
	uid == newid)
	checkit = B_TRUE;
	break;
	} else {
	mutex_exit(&zp->z_acl_lock);
	return (SET_ERROR(EIO));
	}
	}

	if (checkit) {
	if (type == DENY) {
	DTRACE_PROBE3(zfs__ace__denies,
	znode_t *, zp,
	zfs_ace_hdr_t *, acep,
	uint32_t, mask_matched);
	deny_mask \|= mask_matched;
	} else {
	DTRACE_PROBE3(zfs__ace__allows,
	znode_t *, zp,
	zfs_ace_hdr_t *, acep,
	uint32_t, mask_matched);
	if (anyaccess) {
	mutex_exit(&zp->z_acl_lock);
	return (0);
	}
	}
	*working_mode &= ~mask_matched;
	}

	/* Are we done? */
	if (*working_mode == 0)
	break;
	}

	mutex_exit(&zp->z_acl_lock);

	/* Put the found 'denies' back on the working mode */
	if (deny_mask) {
	*working_mode \|= deny_mask;
	return (SET_ERROR(EACCES));
	} else if (*working_mode) {
	return (-1);
	}

	return (0);
	}

	/*
	* Return true if any access whatsoever granted, we don't actually
	* care what access is granted.
	*/
	boolean_t
	zfs_has_access(znode_t zp, cred_t cr)
	{
	uint32_t have = ACE_ALL_PERMS;

	if (zfs_zaccess_aces_check(zp, &have, B_TRUE, cr) != 0) {
	uid_t owner;

	owner = zfs_fuid_map_id(zp->z_zfsvfs, zp->z_uid, cr, ZFS_OWNER);
	return (secpolicy_vnode_any_access(cr, ZTOV(zp), owner) == 0);
	}
	return (B_TRUE);
	}

	static int
	zfs_zaccess_common(znode_t zp, uint32_t v4_mode, uint32_t working_mode,
	boolean_t check_privs, boolean_t skipaclchk, cred_t cr)
	{
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;
	int err;

	*working_mode = v4_mode;
	*check_privs = B_TRUE;

	/*
	* Short circuit empty requests
	*/
	if (v4_mode == 0 \|\| zfsvfs->z_replay) {
	*working_mode = 0;
	return (0);
	}

	if ((err = zfs_zaccess_dataset_check(zp, v4_mode)) != 0) {
	*check_privs = B_FALSE;
	return (err);
	}

	/*
	* The caller requested that the ACL check be skipped. This
	* would only happen if the caller checked VOP_ACCESS() with a
	* 32 bit ACE mask and already had the appropriate permissions.
	*/
	if (skipaclchk) {
	*working_mode = 0;
	return (0);
	}

	/*
	* Note: ZFS_READONLY represents the "DOS R/O" attribute.
	* When that flag is set, we should behave as if write access
	* were not granted by anything in the ACL. In particular:
	* We _must_ allow writes after opening the file r/w, then
	* setting the DOS R/O attribute, and writing some more.
	* (Similar to how you can write after fchmod(fd, 0444).)
	*
	* Therefore ZFS_READONLY is ignored in the dataset check
	* above, and checked here as if part of the ACL check.
	* Also note: DOS R/O is ignored for directories.
	*/
	if ((v4_mode & WRITE_MASK_DATA) &&
	(ZTOV(zp)->v_type != VDIR) &&
	(zp->z_pflags & ZFS_READONLY)) {
	return (SET_ERROR(EPERM));
	}

	return (zfs_zaccess_aces_check(zp, working_mode, B_FALSE, cr));
	}

	static int
	zfs_zaccess_append(znode_t zp, uint32_t working_mode, boolean_t *check_privs,
	cred_t *cr)
	{
	if (*working_mode != ACE_WRITE_DATA)
	return (SET_ERROR(EACCES));

	return (zfs_zaccess_common(zp, ACE_APPEND_DATA, working_mode,
	check_privs, B_FALSE, cr));
	}

	/*
	* Check if VEXEC is allowed.
	*
	* This routine is based on zfs_fastaccesschk_execute which has slowpath
	* calling zfs_zaccess. This would be incorrect on FreeBSD (see
	* zfs_freebsd_access for the difference). Thus this variant let's the
	* caller handle the slowpath (if necessary).
	*
	* On top of that we perform a lockless check for ZFS_NO_EXECS_DENIED.
	*
	* Safe access to znode_t is provided by the vnode lock.
	*/
	int
	zfs_fastaccesschk_execute(znode_t zdp, cred_t cr)
	{
	boolean_t is_attr;

	if (zdp->z_pflags & ZFS_AV_QUARANTINED)
	return (1);

	is_attr = ((zdp->z_pflags & ZFS_XATTR) &&
	(ZTOV(zdp)->v_type == VDIR));
	if (is_attr)
	return (1);

	if (zdp->z_pflags & ZFS_NO_EXECS_DENIED)
	return (0);

	return (1);
	}


	/*
	* Determine whether Access should be granted/denied.
	*
	* The least priv subsystem is always consulted as a basic privilege
	* can define any form of access.
	*/
	int
	zfs_zaccess(znode_t zp, int mode, int flags, boolean_t skipaclchk, cred_t cr)
	{
	uint32_t working_mode;
	int error;
	int is_attr;
	boolean_t check_privs;
	znode_t *xzp = NULL;
	znode_t *check_zp = zp;
	mode_t needed_bits;
	uid_t owner;

	is_attr = ((zp->z_pflags & ZFS_XATTR) && (ZTOV(zp)->v_type == VDIR));

	#ifdef __FreeBSD_kernel__
	/*
	* In FreeBSD, we don't care about permissions of individual ADS.
	* Note that not checking them is not just an optimization - without
	* this shortcut, EA operations may bogusly fail with EACCES.
	*/
	if (zp->z_pflags & ZFS_XATTR)
	return (0);
	#else
	/*
	* If attribute then validate against base file
	*/
	if (is_attr) {
	uint64_t parent;

	if ((error = sa_lookup(zp->z_sa_hdl,
	SA_ZPL_PARENT(zp->z_zfsvfs), &parent,
	sizeof (parent))) != 0)
	return (error);

	if ((error = zfs_zget(zp->z_zfsvfs,
	parent, &xzp)) != 0) {
	return (error);
	}

	check_zp = xzp;

	/*
	* fixup mode to map to xattr perms
	*/

	if (mode & (ACE_WRITE_DATA\|ACE_APPEND_DATA)) {
	mode &= ~(ACE_WRITE_DATA\|ACE_APPEND_DATA);
	mode \|= ACE_WRITE_NAMED_ATTRS;
	}

	if (mode & (ACE_READ_DATA\|ACE_EXECUTE)) {
	mode &= ~(ACE_READ_DATA\|ACE_EXECUTE);
	mode \|= ACE_READ_NAMED_ATTRS;
	}
	}
	#endif

	owner = zfs_fuid_map_id(zp->z_zfsvfs, zp->z_uid, cr, ZFS_OWNER);
	/*
	* Map the bits required to the standard vnode flags VREAD\|VWRITE\|VEXEC
	* in needed_bits. Map the bits mapped by working_mode (currently
	* missing) in missing_bits.
	* Call secpolicy_vnode_access2() with (needed_bits & ~checkmode),
	* needed_bits.
	*/
	needed_bits = 0;

	working_mode = mode;
	if ((working_mode & (ACE_READ_ACL\|ACE_READ_ATTRIBUTES)) &&
	owner == crgetuid(cr))
	working_mode &= ~(ACE_READ_ACL\|ACE_READ_ATTRIBUTES);

	if (working_mode & (ACE_READ_DATA\|ACE_READ_NAMED_ATTRS\|
	ACE_READ_ACL\|ACE_READ_ATTRIBUTES\|ACE_SYNCHRONIZE))
	needed_bits \|= VREAD;
	if (working_mode & (ACE_WRITE_DATA\|ACE_WRITE_NAMED_ATTRS\|
	ACE_APPEND_DATA\|ACE_WRITE_ATTRIBUTES\|ACE_SYNCHRONIZE))
	needed_bits \|= VWRITE;
	if (working_mode & ACE_EXECUTE)
	needed_bits \|= VEXEC;

	if ((error = zfs_zaccess_common(check_zp, mode, &working_mode,
	&check_privs, skipaclchk, cr)) == 0) {
	if (is_attr)
	VN_RELE(ZTOV(xzp));
	return (secpolicy_vnode_access2(cr, ZTOV(zp), owner,
	needed_bits, needed_bits));
	}

	if (error && !check_privs) {
	if (is_attr)
	VN_RELE(ZTOV(xzp));
	return (error);
	}

	if (error && (flags & V_APPEND)) {
	error = zfs_zaccess_append(zp, &working_mode, &check_privs, cr);
	}

	if (error && check_privs) {
	mode_t checkmode = 0;
	vnode_t *check_vp = ZTOV(check_zp);

	/*
	* First check for implicit owner permission on
	* read_acl/read_attributes
	*/

	error = 0;
	ASSERT(working_mode != 0);

	if ((working_mode & (ACE_READ_ACL\|ACE_READ_ATTRIBUTES) &&
	owner == crgetuid(cr)))
	working_mode &= ~(ACE_READ_ACL\|ACE_READ_ATTRIBUTES);

	if (working_mode & (ACE_READ_DATA\|ACE_READ_NAMED_ATTRS\|
	ACE_READ_ACL\|ACE_READ_ATTRIBUTES\|ACE_SYNCHRONIZE))
	checkmode \|= VREAD;
	if (working_mode & (ACE_WRITE_DATA\|ACE_WRITE_NAMED_ATTRS\|
	ACE_APPEND_DATA\|ACE_WRITE_ATTRIBUTES\|ACE_SYNCHRONIZE))
	checkmode \|= VWRITE;
	if (working_mode & ACE_EXECUTE)
	checkmode \|= VEXEC;

	error = secpolicy_vnode_access2(cr, check_vp, owner,
	needed_bits & ~checkmode, needed_bits);

	if (error == 0 && (working_mode & ACE_WRITE_OWNER))
	error = secpolicy_vnode_chown(check_vp, cr, owner);
	if (error == 0 && (working_mode & ACE_WRITE_ACL))
	error = secpolicy_vnode_setdac(check_vp, cr, owner);

	if (error == 0 && (working_mode &
	(ACE_DELETE\|ACE_DELETE_CHILD)))
	error = secpolicy_vnode_remove(check_vp, cr);

	if (error == 0 && (working_mode & ACE_SYNCHRONIZE)) {
	error = secpolicy_vnode_chown(check_vp, cr, owner);
	}
	if (error == 0) {
	/*
	* See if any bits other than those already checked
	* for are still present. If so then return EACCES
	*/
	if (working_mode & ~(ZFS_CHECKED_MASKS)) {
	error = SET_ERROR(EACCES);
	}
	}
	} else if (error == 0) {
	error = secpolicy_vnode_access2(cr, ZTOV(zp), owner,
	needed_bits, needed_bits);
	}


	if (is_attr)
	VN_RELE(ZTOV(xzp));

	return (error);
	}

	/*
	* Translate traditional unix VREAD/VWRITE/VEXEC mode into
	- * native ACL format and call zfs_zaccess()
	+ * NFSv4-style ZFS ACL format and call zfs_zaccess()
	*/
	int
	zfs_zaccess_rwx(znode_t zp, mode_t mode, int flags, cred_t cr)
	{
	return (zfs_zaccess(zp, zfs_unix_to_v4(mode >> 6), flags, B_FALSE, cr));
	}

	/*
	* Access function for secpolicy_vnode_setattr
	*/
	int
	zfs_zaccess_unix(znode_t zp, mode_t mode, cred_t cr)
	{
	int v4_mode = zfs_unix_to_v4(mode >> 6);

	return (zfs_zaccess(zp, v4_mode, 0, B_FALSE, cr));
	}

	static int
	zfs_delete_final_check(znode_t zp, znode_t dzp,
	mode_t available_perms, cred_t *cr)
	{
	int error;
	uid_t downer;

	downer = zfs_fuid_map_id(dzp->z_zfsvfs, dzp->z_uid, cr, ZFS_OWNER);

	error = secpolicy_vnode_access2(cr, ZTOV(dzp),
	downer, available_perms, VWRITE\|VEXEC);

	if (error == 0)
	error = zfs_sticky_remove_access(dzp, zp, cr);

	return (error);
	}

	/*
	* Determine whether Access should be granted/deny, without
	* consulting least priv subsystem.
	*
	* The following chart is the recommended NFSv4 enforcement for
	* ability to delete an object.
	*
	* -------------------------------------------------------
	* \| Parent Dir \| Target Object Permissions \|
	* \| permissions \| \|
	* -------------------------------------------------------
	* \| \| ACL Allows \| ACL Denies\| Delete \|
	* \| \| Delete \| Delete \| unspecified\|
	* -------------------------------------------------------
	* \| ACL Allows \| Permit \| Permit \| Permit \|
	* \| DELETE_CHILD \| \|
	* -------------------------------------------------------
	* \| ACL Denies \| Permit \| Deny \| Deny \|
	* \| DELETE_CHILD \| \| \| \|
	* -------------------------------------------------------
	* \| ACL specifies \| \| \| \|
	* \| only allow \| Permit \| Permit \| Permit \|
	* \| write and \| \| \| \|
	* \| execute \| \| \| \|
	* -------------------------------------------------------
	* \| ACL denies \| \| \| \|
	* \| write and \| Permit \| Deny \| Deny \|
	* \| execute \| \| \| \|
	* -------------------------------------------------------
	* ^
	* \|
	* No search privilege, can't even look up file?
	*
	*/
	int
	zfs_zaccess_delete(znode_t dzp, znode_t zp, cred_t *cr)
	{
	uint32_t dzp_working_mode = 0;
	uint32_t zp_working_mode = 0;
	int dzp_error, zp_error;
	mode_t available_perms;
	boolean_t dzpcheck_privs = B_TRUE;
	boolean_t zpcheck_privs = B_TRUE;

	/*
	* We want specific DELETE permissions to
	* take precedence over WRITE/EXECUTE. We don't
	* want an ACL such as this to mess us up.
	* user:joe:write_data:deny,user:joe:delete:allow
	*
	* However, deny permissions may ultimately be overridden
	* by secpolicy_vnode_access().
	*
	* We will ask for all of the necessary permissions and then
	* look at the working modes from the directory and target object
	* to determine what was found.
	*/

	if (zp->z_pflags & (ZFS_IMMUTABLE \| ZFS_NOUNLINK))
	return (SET_ERROR(EPERM));

	/*
	* First row
	* If the directory permissions allow the delete, we are done.
	*/
	if ((dzp_error = zfs_zaccess_common(dzp, ACE_DELETE_CHILD,
	&dzp_working_mode, &dzpcheck_privs, B_FALSE, cr)) == 0)
	return (0);

	/*
	* If target object has delete permission then we are done
	*/
	if ((zp_error = zfs_zaccess_common(zp, ACE_DELETE, &zp_working_mode,
	&zpcheck_privs, B_FALSE, cr)) == 0)
	return (0);

	ASSERT(dzp_error && zp_error);

	if (!dzpcheck_privs)
	return (dzp_error);
	if (!zpcheck_privs)
	return (zp_error);

	/*
	* Second row
	*
	* If directory returns EACCES then delete_child was denied
	* due to deny delete_child. In this case send the request through
	* secpolicy_vnode_remove(). We don't use zfs_delete_final_check()
	* since that could allow the delete based on write/execute permission
	* and we want delete permissions to override write/execute.
	*/

	if (dzp_error == EACCES) {
	/* XXXPJD: s/dzp/zp/ ? */
	return (secpolicy_vnode_remove(ZTOV(dzp), cr));
	}
	/*
	* Third Row
	* only need to see if we have write/execute on directory.
	*/

	dzp_error = zfs_zaccess_common(dzp, ACE_EXECUTE\|ACE_WRITE_DATA,
	&dzp_working_mode, &dzpcheck_privs, B_FALSE, cr);

	if (dzp_error != 0 && !dzpcheck_privs)
	return (dzp_error);

	/*
	* Fourth row
	*/

	available_perms = (dzp_working_mode & ACE_WRITE_DATA) ? 0 : VWRITE;
	available_perms \|= (dzp_working_mode & ACE_EXECUTE) ? 0 : VEXEC;

	return (zfs_delete_final_check(zp, dzp, available_perms, cr));

	}

	int
	zfs_zaccess_rename(znode_t sdzp, znode_t szp, znode_t *tdzp,
	znode_t tzp, cred_t cr)
	{
	int add_perm;
	int error;

	if (szp->z_pflags & ZFS_AV_QUARANTINED)
	return (SET_ERROR(EACCES));

	add_perm = (ZTOV(szp)->v_type == VDIR) ?
	ACE_ADD_SUBDIRECTORY : ACE_ADD_FILE;

	/*
	* Rename permissions are combination of delete permission +
	* add file/subdir permission.
	*
	* BSD operating systems also require write permission
	* on the directory being moved from one parent directory
	* to another.
	*/
	if (ZTOV(szp)->v_type == VDIR && ZTOV(sdzp) != ZTOV(tdzp)) {
	if ((error = zfs_zaccess(szp, ACE_WRITE_DATA, 0, B_FALSE, cr)))
	return (error);
	}

	/*
	* first make sure we do the delete portion.
	*
	* If that succeeds then check for add_file/add_subdir permissions
	*/

	if ((error = zfs_zaccess_delete(sdzp, szp, cr)))
	return (error);

	/*
	* If we have a tzp, see if we can delete it?
	*/
	if (tzp && (error = zfs_zaccess_delete(tdzp, tzp, cr)))
	return (error);

	/*
	* Now check for add permissions
	*/
	error = zfs_zaccess(tdzp, add_perm, 0, B_FALSE, cr);

	return (error);
	}
	Index: vendor-sys/openzfs/dist/module/os/freebsd/zfs/zfs_ctldir.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/freebsd/zfs/zfs_ctldir.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/freebsd/zfs/zfs_ctldir.c (revision 366774)
	@@ -1,1350 +1,1357 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2012, 2015 by Delphix. All rights reserved.
	* Copyright 2015, OmniTI Computer Consulting, Inc. All rights reserved.
	*/

	/*
	* ZFS control directory (a.k.a. ".zfs")
	*
	* This directory provides a common location for all ZFS meta-objects.
	* Currently, this is only the 'snapshot' directory, but this may expand in the
	* future. The elements are built using the GFS primitives, as the hierarchy
	* does not actually exist on disk.
	*
	* For 'snapshot', we don't want to have all snapshots always mounted, because
	* this would take up a huge amount of space in /etc/mnttab. We have three
	* types of objects:
	*
	* ctldir ------> snapshotdir -------> snapshot
	* \|
	* \|
	* V
	* mounted fs
	*
	* The 'snapshot' node contains just enough information to lookup '..' and act
	* as a mountpoint for the snapshot. Whenever we lookup a specific snapshot, we
	* perform an automount of the underlying filesystem and return the
	* corresponding vnode.
	*
	* All mounts are handled automatically by the kernel, but unmounts are
	* (currently) handled from user land. The main reason is that there is no
	* reliable way to auto-unmount the filesystem when it's "no longer in use".
	* When the user unmounts a filesystem, we call zfsctl_unmount(), which
	* unmounts any snapshots within the snapshot directory.
	*
	* The '.zfs', '.zfs/snapshot', and all directories created under
	* '.zfs/snapshot' (ie: '.zfs/snapshot/<snapname>') are all GFS nodes and
	* share the same vfs_t as the head filesystem (what '.zfs' lives under).
	*
	* File systems mounted ontop of the GFS nodes '.zfs/snapshot/<snapname>'
	* (ie: snapshots) are ZFS nodes and have their own unique vfs_t.
	* However, vnodes within these mounted on file systems have their v_vfsp
	* fields set to the head filesystem to make NFS happy (see
	* zfsctl_snapdir_lookup()). We VFS_HOLD the head filesystem's vfs_t
	* so that it cannot be freed until all snapshots have been unmounted.
	*/

	#include <sys/types.h>
	#include <sys/param.h>
	#include <sys/libkern.h>
	#include <sys/dirent.h>
	#include <sys/zfs_context.h>
	#include <sys/zfs_ctldir.h>
	#include <sys/zfs_ioctl.h>
	#include <sys/zfs_vfsops.h>
	#include <sys/namei.h>
	#include <sys/stat.h>
	#include <sys/dmu.h>
	#include <sys/dsl_dataset.h>
	#include <sys/dsl_destroy.h>
	#include <sys/dsl_deleg.h>
	#include <sys/mount.h>
	#include <sys/zap.h>
	#include <sys/sysproto.h>

	#include "zfs_namecheck.h"

	#include <sys/kernel.h>
	#include <sys/ccompat.h>

	/* Common access mode for all virtual directories under the ctldir */
	const uint16_t zfsctl_ctldir_mode = S_IRUSR \| S_IXUSR \| S_IRGRP \| S_IXGRP \|
	S_IROTH \| S_IXOTH;

	/*
	* "Synthetic" filesystem implementation.
	*/

	/*
	* Assert that A implies B.
	*/
	#define KASSERT_IMPLY(A, B, msg) KASSERT(!(A) \|\| (B), (msg));

	static MALLOC_DEFINE(M_SFSNODES, "sfs_nodes", "synthetic-fs nodes");

	typedef struct sfs_node {
	char sn_name[ZFS_MAX_DATASET_NAME_LEN];
	uint64_t sn_parent_id;
	uint64_t sn_id;
	} sfs_node_t;

	/*
	* Check the parent's ID as well as the node's to account for a chance
	* that IDs originating from different domains (snapshot IDs, artificial
	* IDs, znode IDs) may clash.
	*/
	static int
	sfs_compare_ids(struct vnode vp, void arg)
	{
	sfs_node_t *n1 = vp->v_data;
	sfs_node_t *n2 = arg;
	bool equal;

	equal = n1->sn_id == n2->sn_id &&
	n1->sn_parent_id == n2->sn_parent_id;

	/* Zero means equality. */
	return (!equal);
	}

	static int
	sfs_vnode_get(const struct mount *mp, int flags, uint64_t parent_id,
	uint64_t id, struct vnode **vpp)
	{
	sfs_node_t search;
	int err;

	search.sn_id = id;
	search.sn_parent_id = parent_id;
	err = vfs_hash_get(mp, (uint32_t)id, flags, curthread, vpp,
	sfs_compare_ids, &search);
	return (err);
	}

	static int
	sfs_vnode_insert(struct vnode *vp, int flags, uint64_t parent_id,
	uint64_t id, struct vnode **vpp)
	{
	int err;

	KASSERT(vp->v_data != NULL, ("sfs_vnode_insert with NULL v_data"));
	err = vfs_hash_insert(vp, (uint32_t)id, flags, curthread, vpp,
	sfs_compare_ids, vp->v_data);
	return (err);
	}

	static void
	sfs_vnode_remove(struct vnode *vp)
	{
	vfs_hash_remove(vp);
	}

	typedef void sfs_vnode_setup_fn(vnode_t vp, void arg);

	static int
	sfs_vgetx(struct mount *mp, int flags, uint64_t parent_id, uint64_t id,
	const char tag, struct vop_vector vops,
	sfs_vnode_setup_fn setup, void *arg,
	struct vnode **vpp)
	{
	struct vnode *vp;
	int error;

	error = sfs_vnode_get(mp, flags, parent_id, id, vpp);
	if (error != 0 \|\| *vpp != NULL) {
	KASSERT_IMPLY(error == 0, (*vpp)->v_data != NULL,
	"sfs vnode with no data");
	return (error);
	}

	/* Allocate a new vnode/inode. */
	error = getnewvnode(tag, mp, vops, &vp);
	if (error != 0) {
	*vpp = NULL;
	return (error);
	}

	/*
	* Exclusively lock the vnode vnode while it's being constructed.
	*/
	lockmgr(vp->v_vnlock, LK_EXCLUSIVE, NULL);
	error = insmntque(vp, mp);
	if (error != 0) {
	*vpp = NULL;
	return (error);
	}

	setup(vp, arg);

	error = sfs_vnode_insert(vp, flags, parent_id, id, vpp);
	if (error != 0 \|\| *vpp != NULL) {
	KASSERT_IMPLY(error == 0, (*vpp)->v_data != NULL,
	"sfs vnode with no data");
	return (error);
	}

	*vpp = vp;
	return (0);
	}

	static void
	sfs_print_node(sfs_node_t *node)
	{
	printf("\tname = %s\n", node->sn_name);
	printf("\tparent_id = %ju\n", (uintmax_t)node->sn_parent_id);
	printf("\tid = %ju\n", (uintmax_t)node->sn_id);
	}

	static sfs_node_t *
	sfs_alloc_node(size_t size, const char *name, uint64_t parent_id, uint64_t id)
	{
	struct sfs_node *node;

	KASSERT(strlen(name) < sizeof (node->sn_name),
	("sfs node name is too long"));
	KASSERT(size >= sizeof (*node), ("sfs node size is too small"));
	node = malloc(size, M_SFSNODES, M_WAITOK \| M_ZERO);
	strlcpy(node->sn_name, name, sizeof (node->sn_name));
	node->sn_parent_id = parent_id;
	node->sn_id = id;

	return (node);
	}

	static void
	sfs_destroy_node(sfs_node_t *node)
	{
	free(node, M_SFSNODES);
	}

	static void *
	sfs_reclaim_vnode(vnode_t *vp)
	{
	void *data;

	sfs_vnode_remove(vp);
	data = vp->v_data;
	vp->v_data = NULL;
	return (data);
	}

	static int
	sfs_readdir_common(uint64_t parent_id, uint64_t id, struct vop_readdir_args *ap,
	uio_t uio, off_t offp)
	{
	struct dirent entry;
	int error;

	/* Reset ncookies for subsequent use of vfs_read_dirent. */
	if (ap->a_ncookies != NULL)
	*ap->a_ncookies = 0;

	if (uio->uio_resid < sizeof (entry))
	return (SET_ERROR(EINVAL));

	if (uio->uio_offset < 0)
	return (SET_ERROR(EINVAL));
	if (uio->uio_offset == 0) {
	entry.d_fileno = id;
	entry.d_type = DT_DIR;
	entry.d_name[0] = '.';
	entry.d_name[1] = '\0';
	entry.d_namlen = 1;
	entry.d_reclen = sizeof (entry);
	error = vfs_read_dirent(ap, &entry, uio->uio_offset);
	if (error != 0)
	return (SET_ERROR(error));
	}

	if (uio->uio_offset < sizeof (entry))
	return (SET_ERROR(EINVAL));
	if (uio->uio_offset == sizeof (entry)) {
	entry.d_fileno = parent_id;
	entry.d_type = DT_DIR;
	entry.d_name[0] = '.';
	entry.d_name[1] = '.';
	entry.d_name[2] = '\0';
	entry.d_namlen = 2;
	entry.d_reclen = sizeof (entry);
	error = vfs_read_dirent(ap, &entry, uio->uio_offset);
	if (error != 0)
	return (SET_ERROR(error));
	}

	if (offp != NULL)
	offp = 2 sizeof (entry);
	return (0);
	}


	/*
	* .zfs inode namespace
	*
	* We need to generate unique inode numbers for all files and directories
	* within the .zfs pseudo-filesystem. We use the following scheme:
	*
	* ENTRY ZFSCTL_INODE
	* .zfs 1
	* .zfs/snapshot 2
	* .zfs/snapshot/<snap> objectid(snap)
	*/
	#define ZFSCTL_INO_SNAP(id) (id)

	static struct vop_vector zfsctl_ops_root;
	static struct vop_vector zfsctl_ops_snapdir;
	static struct vop_vector zfsctl_ops_snapshot;
	-static struct vop_vector zfsctl_ops_shares_dir;

	void
	zfsctl_init(void)
	{
	}

	void
	zfsctl_fini(void)
	{
	}

	boolean_t
	zfsctl_is_node(vnode_t *vp)
	{
	return (vn_matchops(vp, zfsctl_ops_root) \|\|
	vn_matchops(vp, zfsctl_ops_snapdir) \|\|
	- vn_matchops(vp, zfsctl_ops_snapshot) \|\|
	- vn_matchops(vp, zfsctl_ops_shares_dir));
	+ vn_matchops(vp, zfsctl_ops_snapshot));

	}

	typedef struct zfsctl_root {
	sfs_node_t node;
	sfs_node_t *snapdir;
	timestruc_t cmtime;
	} zfsctl_root_t;


	/*
	* Create the '.zfs' directory.
	*/
	void
	zfsctl_create(zfsvfs_t *zfsvfs)
	{
	zfsctl_root_t *dot_zfs;
	sfs_node_t *snapdir;
	vnode_t *rvp;
	uint64_t crtime[2];

	ASSERT(zfsvfs->z_ctldir == NULL);

	snapdir = sfs_alloc_node(sizeof (*snapdir), "snapshot", ZFSCTL_INO_ROOT,
	ZFSCTL_INO_SNAPDIR);
	dot_zfs = (zfsctl_root_t )sfs_alloc_node(sizeof (dot_zfs), ".zfs", 0,
	ZFSCTL_INO_ROOT);
	dot_zfs->snapdir = snapdir;

	VERIFY(VFS_ROOT(zfsvfs->z_vfs, LK_EXCLUSIVE, &rvp) == 0);
	VERIFY(0 == sa_lookup(VTOZ(rvp)->z_sa_hdl, SA_ZPL_CRTIME(zfsvfs),
	&crtime, sizeof (crtime)));
	ZFS_TIME_DECODE(&dot_zfs->cmtime, crtime);
	vput(rvp);

	zfsvfs->z_ctldir = dot_zfs;
	}

	/*
	* Destroy the '.zfs' directory. Only called when the filesystem is unmounted.
	* The nodes must not have any associated vnodes by now as they should be
	* vflush-ed.
	*/
	void
	zfsctl_destroy(zfsvfs_t *zfsvfs)
	{
	sfs_destroy_node(zfsvfs->z_ctldir->snapdir);
	sfs_destroy_node((sfs_node_t *)zfsvfs->z_ctldir);
	zfsvfs->z_ctldir = NULL;
	}

	static int
	zfsctl_fs_root_vnode(struct mount mp, void arg __unused, int flags,
	struct vnode **vpp)
	{
	return (VFS_ROOT(mp, flags, vpp));
	}

	static void
	zfsctl_common_vnode_setup(vnode_t vp, void arg)
	{
	ASSERT_VOP_ELOCKED(vp, __func__);

	/* We support shared locking. */
	VN_LOCK_ASHARE(vp);
	vp->v_type = VDIR;
	vp->v_data = arg;
	}

	static int
	zfsctl_root_vnode(struct mount mp, void arg __unused, int flags,
	struct vnode **vpp)
	{
	void *node;
	int err;

	node = ((zfsvfs_t *)mp->mnt_data)->z_ctldir;
	err = sfs_vgetx(mp, flags, 0, ZFSCTL_INO_ROOT, "zfs", &zfsctl_ops_root,
	zfsctl_common_vnode_setup, node, vpp);
	return (err);
	}

	static int
	zfsctl_snapdir_vnode(struct mount mp, void arg __unused, int flags,
	struct vnode **vpp)
	{
	void *node;
	int err;

	node = ((zfsvfs_t *)mp->mnt_data)->z_ctldir->snapdir;
	err = sfs_vgetx(mp, flags, ZFSCTL_INO_ROOT, ZFSCTL_INO_SNAPDIR, "zfs",
	&zfsctl_ops_snapdir, zfsctl_common_vnode_setup, node, vpp);
	return (err);
	}

	/*
	* Given a root znode, retrieve the associated .zfs directory.
	* Add a hold to the vnode and return it.
	*/
	int
	zfsctl_root(zfsvfs_t zfsvfs, int flags, vnode_t *vpp)
	{
	int error;

	error = zfsctl_root_vnode(zfsvfs->z_vfs, NULL, flags, vpp);
	return (error);
	}

	/*
	* Common open routine. Disallow any write access.
	*/
	static int
	zfsctl_common_open(struct vop_open_args *ap)
	{
	int flags = ap->a_mode;

	if (flags & FWRITE)
	return (SET_ERROR(EACCES));

	return (0);
	}

	/*
	* Common close routine. Nothing to do here.
	*/
	/* ARGSUSED */
	static int
	zfsctl_common_close(struct vop_close_args *ap)
	{
	return (0);
	}

	/*
	* Common access routine. Disallow writes.
	*/
	static int
	zfsctl_common_access(struct vop_access_args *ap)
	{
	accmode_t accmode = ap->a_accmode;

	if (accmode & VWRITE)
	return (SET_ERROR(EACCES));
	return (0);
	}

	/*
	* Common getattr function. Fill in basic information.
	*/
	static void
	zfsctl_common_getattr(vnode_t vp, vattr_t vap)
	{
	timestruc_t now;
	sfs_node_t *node;

	node = vp->v_data;

	vap->va_uid = 0;
	vap->va_gid = 0;
	vap->va_rdev = 0;
	/*
	* We are a purely virtual object, so we have no
	* blocksize or allocated blocks.
	*/
	vap->va_blksize = 0;
	vap->va_nblocks = 0;
	vap->va_seq = 0;
	vn_fsid(vp, vap);
	vap->va_mode = zfsctl_ctldir_mode;
	vap->va_type = VDIR;
	/*
	* We live in the now (for atime).
	*/
	gethrestime(&now);
	vap->va_atime = now;
	/* FreeBSD: Reset chflags(2) flags. */
	vap->va_flags = 0;

	vap->va_nodeid = node->sn_id;

	/* At least '.' and '..'. */
	vap->va_nlink = 2;
	}

	#ifndef _OPENSOLARIS_SYS_VNODE_H_
	struct vop_fid_args {
	struct vnode *a_vp;
	struct fid *a_fid;
	};
	#endif

	static int
	zfsctl_common_fid(struct vop_fid_args *ap)
	{
	vnode_t *vp = ap->a_vp;
	fid_t fidp = (void )ap->a_fid;
	sfs_node_t *node = vp->v_data;
	uint64_t object = node->sn_id;
	zfid_short_t *zfid;
	int i;

	zfid = (zfid_short_t *)fidp;
	zfid->zf_len = SHORT_FID_LEN;

	for (i = 0; i < sizeof (zfid->zf_object); i++)
	zfid->zf_object[i] = (uint8_t)(object >> (8 * i));

	/* .zfs nodes always have a generation number of 0 */
	for (i = 0; i < sizeof (zfid->zf_gen); i++)
	zfid->zf_gen[i] = 0;

	return (0);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_reclaim_args {
	struct vnode *a_vp;
	struct thread *a_td;
	};
	#endif

	static int
	zfsctl_common_reclaim(struct vop_reclaim_args *ap)
	{
	vnode_t *vp = ap->a_vp;

	(void) sfs_reclaim_vnode(vp);
	return (0);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_print_args {
	struct vnode *a_vp;
	};
	#endif

	static int
	zfsctl_common_print(struct vop_print_args *ap)
	{
	sfs_print_node(ap->a_vp->v_data);
	return (0);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_getattr_args {
	struct vnode *a_vp;
	struct vattr *a_vap;
	struct ucred *a_cred;
	};
	#endif

	/*
	* Get root directory attributes.
	*/
	static int
	zfsctl_root_getattr(struct vop_getattr_args *ap)
	{
	struct vnode *vp = ap->a_vp;
	struct vattr *vap = ap->a_vap;
	zfsctl_root_t *node = vp->v_data;

	zfsctl_common_getattr(vp, vap);
	vap->va_ctime = node->cmtime;
	vap->va_mtime = vap->va_ctime;
	vap->va_birthtime = vap->va_ctime;
	vap->va_nlink += 1; /* snapdir */
	vap->va_size = vap->va_nlink;
	return (0);
	}

	/*
	* When we lookup "." we still can be asked to lock it
	* differently, can't we?
	*/
	static int
	zfsctl_relock_dot(vnode_t *dvp, int ltype)
	{
	vref(dvp);
	if (ltype != VOP_ISLOCKED(dvp)) {
	if (ltype == LK_EXCLUSIVE)
	vn_lock(dvp, LK_UPGRADE \| LK_RETRY);
	else /* if (ltype == LK_SHARED) */
	vn_lock(dvp, LK_DOWNGRADE \| LK_RETRY);

	/* Relock for the "." case may left us with reclaimed vnode. */
	if (VN_IS_DOOMED(dvp)) {
	vrele(dvp);
	return (SET_ERROR(ENOENT));
	}
	}
	return (0);
	}

	/*
	* Special case the handling of "..".
	*/
	static int
	zfsctl_root_lookup(struct vop_lookup_args *ap)
	{
	struct componentname *cnp = ap->a_cnp;
	vnode_t *dvp = ap->a_dvp;
	vnode_t **vpp = ap->a_vpp;
	int flags = ap->a_cnp->cn_flags;
	int lkflags = ap->a_cnp->cn_lkflags;
	int nameiop = ap->a_cnp->cn_nameiop;
	int err;

	ASSERT(dvp->v_type == VDIR);

	if ((flags & ISLASTCN) != 0 && nameiop != LOOKUP)
	return (SET_ERROR(ENOTSUP));

	if (cnp->cn_namelen == 1 && *cnp->cn_nameptr == '.') {
	err = zfsctl_relock_dot(dvp, lkflags & LK_TYPE_MASK);
	if (err == 0)
	*vpp = dvp;
	} else if ((flags & ISDOTDOT) != 0) {
	err = vn_vget_ino_gen(dvp, zfsctl_fs_root_vnode, NULL,
	lkflags, vpp);
	} else if (strncmp(cnp->cn_nameptr, "snapshot", cnp->cn_namelen) == 0) {
	err = zfsctl_snapdir_vnode(dvp->v_mount, NULL, lkflags, vpp);
	} else {
	err = SET_ERROR(ENOENT);
	}
	if (err != 0)
	*vpp = NULL;
	return (err);
	}

	static int
	zfsctl_root_readdir(struct vop_readdir_args *ap)
	{
	struct dirent entry;
	vnode_t *vp = ap->a_vp;
	zfsvfs_t *zfsvfs = vp->v_vfsp->vfs_data;
	zfsctl_root_t *node = vp->v_data;
	uio_t *uio = ap->a_uio;
	int *eofp = ap->a_eofflag;
	off_t dots_offset;
	int error;

	ASSERT(vp->v_type == VDIR);

	error = sfs_readdir_common(zfsvfs->z_root, ZFSCTL_INO_ROOT, ap, uio,
	&dots_offset);
	if (error != 0) {
	if (error == ENAMETOOLONG) /* ran out of destination space */
	error = 0;
	return (error);
	}
	if (uio->uio_offset != dots_offset)
	return (SET_ERROR(EINVAL));

	CTASSERT(sizeof (node->snapdir->sn_name) <= sizeof (entry.d_name));
	entry.d_fileno = node->snapdir->sn_id;
	entry.d_type = DT_DIR;
	strcpy(entry.d_name, node->snapdir->sn_name);
	entry.d_namlen = strlen(entry.d_name);
	entry.d_reclen = sizeof (entry);
	error = vfs_read_dirent(ap, &entry, uio->uio_offset);
	if (error != 0) {
	if (error == ENAMETOOLONG)
	error = 0;
	return (SET_ERROR(error));
	}
	if (eofp != NULL)
	*eofp = 1;
	return (0);
	}

	static int
	zfsctl_root_vptocnp(struct vop_vptocnp_args *ap)
	{
	static const char dotzfs_name[4] = ".zfs";
	vnode_t *dvp;
	int error;

	if (*ap->a_buflen < sizeof (dotzfs_name))
	return (SET_ERROR(ENOMEM));

	error = vn_vget_ino_gen(ap->a_vp, zfsctl_fs_root_vnode, NULL,
	LK_SHARED, &dvp);
	if (error != 0)
	return (SET_ERROR(error));

	VOP_UNLOCK1(dvp);
	*ap->a_vpp = dvp;
	*ap->a_buflen -= sizeof (dotzfs_name);
	bcopy(dotzfs_name, ap->a_buf + *ap->a_buflen, sizeof (dotzfs_name));
	return (0);
	}

	static int
	zfsctl_common_pathconf(struct vop_pathconf_args *ap)
	{
	/*
	* We care about ACL variables so that user land utilities like ls
	* can display them correctly. Since the ctldir's st_dev is set to be
	* the same as the parent dataset, we must support all variables that
	* it supports.
	*/
	switch (ap->a_name) {
	case _PC_LINK_MAX:
	*ap->a_retval = MIN(LONG_MAX, ZFS_LINK_MAX);
	return (0);

	case _PC_FILESIZEBITS:
	*ap->a_retval = 64;
	return (0);

	case _PC_MIN_HOLE_SIZE:
	*ap->a_retval = (int)SPA_MINBLOCKSIZE;
	return (0);

	case _PC_ACL_EXTENDED:
	*ap->a_retval = 0;
	return (0);

	case _PC_ACL_NFS4:
	*ap->a_retval = 1;
	return (0);

	case _PC_ACL_PATH_MAX:
	*ap->a_retval = ACL_MAX_ENTRIES;
	return (0);

	case _PC_NAME_MAX:
	*ap->a_retval = NAME_MAX;
	return (0);

	default:
	return (vop_stdpathconf(ap));
	}
	}

	/*
	* Returns a trivial ACL
	*/
	static int
	zfsctl_common_getacl(struct vop_getacl_args *ap)
	{
	int i;

	if (ap->a_type != ACL_TYPE_NFS4)
	return (EINVAL);

	acl_nfs4_sync_acl_from_mode(ap->a_aclp, zfsctl_ctldir_mode, 0);
	/*
	* acl_nfs4_sync_acl_from_mode assumes that the owner can always modify
	* attributes. That is not the case for the ctldir, so we must clear
	* those bits. We also must clear ACL_READ_NAMED_ATTRS, because xattrs
	* aren't supported by the ctldir.
	*/
	for (i = 0; i < ap->a_aclp->acl_cnt; i++) {
	struct acl_entry *entry;
	entry = &(ap->a_aclp->acl_entry[i]);
	entry->ae_perm &= ~(ACL_WRITE_ACL \| ACL_WRITE_OWNER \|
	ACL_WRITE_ATTRIBUTES \| ACL_WRITE_NAMED_ATTRS \|
	ACL_READ_NAMED_ATTRS);
	}

	return (0);
	}

	static struct vop_vector zfsctl_ops_root = {
	.vop_default = &default_vnodeops,
	+#if __FreeBSD_version >= 1300121
	+ .vop_fplookup_vexec = VOP_EAGAIN,
	+#endif
	.vop_open = zfsctl_common_open,
	.vop_close = zfsctl_common_close,
	.vop_ioctl = VOP_EINVAL,
	.vop_getattr = zfsctl_root_getattr,
	.vop_access = zfsctl_common_access,
	.vop_readdir = zfsctl_root_readdir,
	.vop_lookup = zfsctl_root_lookup,
	.vop_inactive = VOP_NULL,
	.vop_reclaim = zfsctl_common_reclaim,
	.vop_fid = zfsctl_common_fid,
	.vop_print = zfsctl_common_print,
	.vop_vptocnp = zfsctl_root_vptocnp,
	.vop_pathconf = zfsctl_common_pathconf,
	.vop_getacl = zfsctl_common_getacl,
	};
	VFS_VOP_VECTOR_REGISTER(zfsctl_ops_root);

	static int
	zfsctl_snapshot_zname(vnode_t vp, const char name, int len, char *zname)
	{
	objset_t os = ((zfsvfs_t )((vp)->v_vfsp->vfs_data))->z_os;

	dmu_objset_name(os, zname);
	if (strlen(zname) + 1 + strlen(name) >= len)
	return (SET_ERROR(ENAMETOOLONG));
	(void) strcat(zname, "@");
	(void) strcat(zname, name);
	return (0);
	}

	static int
	zfsctl_snapshot_lookup(vnode_t vp, const char name, uint64_t *id)
	{
	objset_t os = ((zfsvfs_t )((vp)->v_vfsp->vfs_data))->z_os;
	int err;

	err = dsl_dataset_snap_lookup(dmu_objset_ds(os), name, id);
	return (err);
	}

	/*
	* Given a vnode get a root vnode of a filesystem mounted on top of
	* the vnode, if any. The root vnode is referenced and locked.
	* If no filesystem is mounted then the orinal vnode remains referenced
	* and locked. If any error happens the orinal vnode is unlocked and
	* released.
	*/
	static int
	zfsctl_mounted_here(vnode_t **vpp, int flags)
	{
	struct mount *mp;
	int err;

	ASSERT_VOP_LOCKED(*vpp, __func__);
	ASSERT3S((*vpp)->v_type, ==, VDIR);

	if ((mp = (*vpp)->v_mountedhere) != NULL) {
	err = vfs_busy(mp, 0);
	KASSERT(err == 0, ("vfs_busy(mp, 0) failed with %d", err));
	KASSERT(vrefcnt(*vpp) > 1, ("unreferenced mountpoint"));
	vput(*vpp);
	err = VFS_ROOT(mp, flags, vpp);
	vfs_unbusy(mp);
	return (err);
	}
	return (EJUSTRETURN);
	}

	typedef struct {
	const char *snap_name;
	uint64_t snap_id;
	} snapshot_setup_arg_t;

	static void
	zfsctl_snapshot_vnode_setup(vnode_t vp, void arg)
	{
	snapshot_setup_arg_t *ssa = arg;
	sfs_node_t *node;

	ASSERT_VOP_ELOCKED(vp, __func__);

	node = sfs_alloc_node(sizeof (sfs_node_t),
	ssa->snap_name, ZFSCTL_INO_SNAPDIR, ssa->snap_id);
	zfsctl_common_vnode_setup(vp, node);

	/* We have to support recursive locking. */
	VN_LOCK_AREC(vp);
	}

	/*
	* Lookup entry point for the 'snapshot' directory. Try to open the
	* snapshot if it exist, creating the pseudo filesystem vnode as necessary.
	* Perform a mount of the associated dataset on top of the vnode.
	* There are four possibilities:
	* - the snapshot node and vnode do not exist
	* - the snapshot vnode is covered by the mounted snapshot
	* - the snapshot vnode is not covered yet, the mount operation is in progress
	* - the snapshot vnode is not covered, because the snapshot has been unmounted
	* The last two states are transient and should be relatively short-lived.
	*/
	static int
	zfsctl_snapdir_lookup(struct vop_lookup_args *ap)
	{
	vnode_t *dvp = ap->a_dvp;
	vnode_t **vpp = ap->a_vpp;
	struct componentname *cnp = ap->a_cnp;
	char name[NAME_MAX + 1];
	char fullname[ZFS_MAX_DATASET_NAME_LEN];
	char *mountpoint;
	size_t mountpoint_len;
	zfsvfs_t *zfsvfs = dvp->v_vfsp->vfs_data;
	uint64_t snap_id;
	int nameiop = cnp->cn_nameiop;
	int lkflags = cnp->cn_lkflags;
	int flags = cnp->cn_flags;
	int err;

	ASSERT(dvp->v_type == VDIR);

	if ((flags & ISLASTCN) != 0 && nameiop != LOOKUP)
	return (SET_ERROR(ENOTSUP));

	if (cnp->cn_namelen == 1 && *cnp->cn_nameptr == '.') {
	err = zfsctl_relock_dot(dvp, lkflags & LK_TYPE_MASK);
	if (err == 0)
	*vpp = dvp;
	return (err);
	}
	if (flags & ISDOTDOT) {
	err = vn_vget_ino_gen(dvp, zfsctl_root_vnode, NULL, lkflags,
	vpp);
	return (err);
	}

	if (cnp->cn_namelen >= sizeof (name))
	return (SET_ERROR(ENAMETOOLONG));

	strlcpy(name, ap->a_cnp->cn_nameptr, ap->a_cnp->cn_namelen + 1);
	err = zfsctl_snapshot_lookup(dvp, name, &snap_id);
	if (err != 0)
	return (SET_ERROR(ENOENT));

	for (;;) {
	snapshot_setup_arg_t ssa;

	ssa.snap_name = name;
	ssa.snap_id = snap_id;
	err = sfs_vgetx(dvp->v_mount, LK_SHARED, ZFSCTL_INO_SNAPDIR,
	snap_id, "zfs", &zfsctl_ops_snapshot,
	zfsctl_snapshot_vnode_setup, &ssa, vpp);
	if (err != 0)
	return (err);

	/* Check if a new vnode has just been created. */
	if (VOP_ISLOCKED(*vpp) == LK_EXCLUSIVE)
	break;

	/*
	* Check if a snapshot is already mounted on top of the vnode.
	*/
	err = zfsctl_mounted_here(vpp, lkflags);
	if (err != EJUSTRETURN)
	return (err);

	/*
	* If the vnode is not covered, then either the mount operation
	* is in progress or the snapshot has already been unmounted
	* but the vnode hasn't been inactivated and reclaimed yet.
	* We can try to re-use the vnode in the latter case.
	*/
	VI_LOCK(*vpp);
	if (((*vpp)->v_iflag & VI_MOUNT) == 0) {
	/*
	* Upgrade to exclusive lock in order to:
	* - avoid race conditions
	* - satisfy the contract of mount_snapshot()
	*/
	err = VOP_LOCK(*vpp, LK_TRYUPGRADE \| LK_INTERLOCK);
	if (err == 0)
	break;
	} else {
	VI_UNLOCK(*vpp);
	}

	/*
	* In this state we can loop on uncontested locks and starve
	* the thread doing the lengthy, non-trivial mount operation.
	* So, yield to prevent that from happening.
	*/
	vput(*vpp);
	kern_yield(PRI_USER);
	}

	VERIFY0(zfsctl_snapshot_zname(dvp, name, sizeof (fullname), fullname));

	mountpoint_len = strlen(dvp->v_vfsp->mnt_stat.f_mntonname) +
	strlen("/" ZFS_CTLDIR_NAME "/snapshot/") + strlen(name) + 1;
	mountpoint = kmem_alloc(mountpoint_len, KM_SLEEP);
	(void) snprintf(mountpoint, mountpoint_len,
	"%s/" ZFS_CTLDIR_NAME "/snapshot/%s",
	dvp->v_vfsp->mnt_stat.f_mntonname, name);

	err = mount_snapshot(curthread, vpp, "zfs", mountpoint, fullname, 0);
	kmem_free(mountpoint, mountpoint_len);
	if (err == 0) {
	/*
	* Fix up the root vnode mounted on .zfs/snapshot/<snapname>.
	*
	* This is where we lie about our v_vfsp in order to
	* make .zfs/snapshot/<snapname> accessible over NFS
	* without requiring manual mounts of <snapname>.
	*/
	ASSERT(VTOZ(*vpp)->z_zfsvfs != zfsvfs);
	VTOZ(*vpp)->z_zfsvfs->z_parent = zfsvfs;

	/* Clear the root flag (set via VFS_ROOT) as well. */
	(*vpp)->v_vflag &= ~VV_ROOT;
	}

	if (err != 0)
	*vpp = NULL;
	return (err);
	}

	static int
	zfsctl_snapdir_readdir(struct vop_readdir_args *ap)
	{
	char snapname[ZFS_MAX_DATASET_NAME_LEN];
	struct dirent entry;
	vnode_t *vp = ap->a_vp;
	zfsvfs_t *zfsvfs = vp->v_vfsp->vfs_data;
	uio_t *uio = ap->a_uio;
	int *eofp = ap->a_eofflag;
	off_t dots_offset;
	int error;

	ASSERT(vp->v_type == VDIR);

	error = sfs_readdir_common(ZFSCTL_INO_ROOT, ZFSCTL_INO_SNAPDIR, ap, uio,
	&dots_offset);
	if (error != 0) {
	if (error == ENAMETOOLONG) /* ran out of destination space */
	error = 0;
	return (error);
	}

	ZFS_ENTER(zfsvfs);
	for (;;) {
	uint64_t cookie;
	uint64_t id;

	cookie = uio->uio_offset - dots_offset;

	dsl_pool_config_enter(dmu_objset_pool(zfsvfs->z_os), FTAG);
	error = dmu_snapshot_list_next(zfsvfs->z_os, sizeof (snapname),
	snapname, &id, &cookie, NULL);
	dsl_pool_config_exit(dmu_objset_pool(zfsvfs->z_os), FTAG);
	if (error != 0) {
	if (error == ENOENT) {
	if (eofp != NULL)
	*eofp = 1;
	error = 0;
	}
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	entry.d_fileno = id;
	entry.d_type = DT_DIR;
	strcpy(entry.d_name, snapname);
	entry.d_namlen = strlen(entry.d_name);
	entry.d_reclen = sizeof (entry);
	error = vfs_read_dirent(ap, &entry, uio->uio_offset);
	if (error != 0) {
	if (error == ENAMETOOLONG)
	error = 0;
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(error));
	}
	uio->uio_offset = cookie + dots_offset;
	}
	/* NOTREACHED */
	}

	static int
	zfsctl_snapdir_getattr(struct vop_getattr_args *ap)
	{
	vnode_t *vp = ap->a_vp;
	vattr_t *vap = ap->a_vap;
	zfsvfs_t *zfsvfs = vp->v_vfsp->vfs_data;
	dsl_dataset_t *ds;
	uint64_t snap_count;
	int err;

	ZFS_ENTER(zfsvfs);
	ds = dmu_objset_ds(zfsvfs->z_os);
	zfsctl_common_getattr(vp, vap);
	vap->va_ctime = dmu_objset_snap_cmtime(zfsvfs->z_os);
	vap->va_mtime = vap->va_ctime;
	vap->va_birthtime = vap->va_ctime;
	if (dsl_dataset_phys(ds)->ds_snapnames_zapobj != 0) {
	err = zap_count(dmu_objset_pool(ds->ds_objset)->dp_meta_objset,
	dsl_dataset_phys(ds)->ds_snapnames_zapobj, &snap_count);
	if (err != 0) {
	ZFS_EXIT(zfsvfs);
	return (err);
	}
	vap->va_nlink += snap_count;
	}
	vap->va_size = vap->va_nlink;

	ZFS_EXIT(zfsvfs);
	return (0);
	}

	static struct vop_vector zfsctl_ops_snapdir = {
	.vop_default = &default_vnodeops,
	+#if __FreeBSD_version >= 1300121
	+ .vop_fplookup_vexec = VOP_EAGAIN,
	+#endif
	.vop_open = zfsctl_common_open,
	.vop_close = zfsctl_common_close,
	.vop_getattr = zfsctl_snapdir_getattr,
	.vop_access = zfsctl_common_access,
	.vop_readdir = zfsctl_snapdir_readdir,
	.vop_lookup = zfsctl_snapdir_lookup,
	.vop_reclaim = zfsctl_common_reclaim,
	.vop_fid = zfsctl_common_fid,
	.vop_print = zfsctl_common_print,
	.vop_pathconf = zfsctl_common_pathconf,
	.vop_getacl = zfsctl_common_getacl,
	};
	VFS_VOP_VECTOR_REGISTER(zfsctl_ops_snapdir);


	static int
	zfsctl_snapshot_inactive(struct vop_inactive_args *ap)
	{
	vnode_t *vp = ap->a_vp;

	VERIFY(vrecycle(vp) == 1);
	return (0);
	}

	static int
	zfsctl_snapshot_reclaim(struct vop_reclaim_args *ap)
	{
	vnode_t *vp = ap->a_vp;
	void *data = vp->v_data;

	sfs_reclaim_vnode(vp);
	sfs_destroy_node(data);
	return (0);
	}

	static int
	zfsctl_snapshot_vptocnp(struct vop_vptocnp_args *ap)
	{
	struct mount *mp;
	vnode_t *dvp;
	vnode_t *vp;
	sfs_node_t *node;
	size_t len;
	int locked;
	int error;

	vp = ap->a_vp;
	node = vp->v_data;
	len = strlen(node->sn_name);
	if (*ap->a_buflen < len)
	return (SET_ERROR(ENOMEM));

	/*
	* Prevent unmounting of the snapshot while the vnode lock
	* is not held. That is not strictly required, but allows
	* us to assert that an uncovered snapshot vnode is never
	* "leaked".
	*/
	mp = vp->v_mountedhere;
	if (mp == NULL)
	return (SET_ERROR(ENOENT));
	error = vfs_busy(mp, 0);
	KASSERT(error == 0, ("vfs_busy(mp, 0) failed with %d", error));

	/*
	* We can vput the vnode as we can now depend on the reference owned
	* by the busied mp. But we also need to hold the vnode, because
	* the reference may go after vfs_unbusy() which has to be called
	* before we can lock the vnode again.
	*/
	locked = VOP_ISLOCKED(vp);
	#if __FreeBSD_version >= 1300045
	enum vgetstate vs = vget_prep(vp);
	#else
	vhold(vp);
	#endif
	vput(vp);

	/* Look up .zfs/snapshot, our parent. */
	error = zfsctl_snapdir_vnode(vp->v_mount, NULL, LK_SHARED, &dvp);
	if (error == 0) {
	VOP_UNLOCK1(dvp);
	*ap->a_vpp = dvp;
	*ap->a_buflen -= len;
	bcopy(node->sn_name, ap->a_buf + *ap->a_buflen, len);
	}
	vfs_unbusy(mp);
	#if __FreeBSD_version >= 1300045
	vget_finish(vp, locked \| LK_RETRY, vs);
	#else
	vget(vp, locked \| LK_VNHELD \| LK_RETRY, curthread);
	#endif
	return (error);
	}

	/*
	* These VP's should never see the light of day. They should always
	* be covered.
	*/
	static struct vop_vector zfsctl_ops_snapshot = {
	.vop_default = NULL, /* ensure very restricted access */
	+#if __FreeBSD_version >= 1300121
	+ .vop_fplookup_vexec = VOP_EAGAIN,
	+#endif
	.vop_inactive = zfsctl_snapshot_inactive,
	#if __FreeBSD_version >= 1300045
	.vop_need_inactive = vop_stdneed_inactive,
	#endif
	.vop_reclaim = zfsctl_snapshot_reclaim,
	.vop_vptocnp = zfsctl_snapshot_vptocnp,
	.vop_lock1 = vop_stdlock,
	.vop_unlock = vop_stdunlock,
	.vop_islocked = vop_stdislocked,
	.vop_advlockpurge = vop_stdadvlockpurge, /* called by vgone */
	.vop_print = zfsctl_common_print,
	};
	VFS_VOP_VECTOR_REGISTER(zfsctl_ops_snapshot);

	int
	zfsctl_lookup_objset(vfs_t vfsp, uint64_t objsetid, zfsvfs_t *zfsvfsp)
	{
	zfsvfs_t *zfsvfs __unused = vfsp->vfs_data;
	vnode_t *vp;
	int error;

	ASSERT(zfsvfs->z_ctldir != NULL);
	*zfsvfsp = NULL;
	error = sfs_vnode_get(vfsp, LK_EXCLUSIVE,
	ZFSCTL_INO_SNAPDIR, objsetid, &vp);
	if (error == 0 && vp != NULL) {
	/*
	* XXX Probably need to at least reference, if not busy, the mp.
	*/
	if (vp->v_mountedhere != NULL)
	*zfsvfsp = vp->v_mountedhere->mnt_data;
	vput(vp);
	}
	if (*zfsvfsp == NULL)
	return (SET_ERROR(EINVAL));
	return (0);
	}

	/*
	* Unmount any snapshots for the given filesystem. This is called from
	* zfs_umount() - if we have a ctldir, then go through and unmount all the
	* snapshots.
	*/
	int
	zfsctl_umount_snapshots(vfs_t vfsp, int fflags, cred_t cr)
	{
	char snapname[ZFS_MAX_DATASET_NAME_LEN];
	zfsvfs_t *zfsvfs = vfsp->vfs_data;
	struct mount *mp;
	vnode_t *vp;
	uint64_t cookie;
	int error;

	ASSERT(zfsvfs->z_ctldir != NULL);

	cookie = 0;
	for (;;) {
	uint64_t id;

	dsl_pool_config_enter(dmu_objset_pool(zfsvfs->z_os), FTAG);
	error = dmu_snapshot_list_next(zfsvfs->z_os, sizeof (snapname),
	snapname, &id, &cookie, NULL);
	dsl_pool_config_exit(dmu_objset_pool(zfsvfs->z_os), FTAG);
	if (error != 0) {
	if (error == ENOENT)
	error = 0;
	break;
	}

	for (;;) {
	error = sfs_vnode_get(vfsp, LK_EXCLUSIVE,
	ZFSCTL_INO_SNAPDIR, id, &vp);
	if (error != 0 \|\| vp == NULL)
	break;

	mp = vp->v_mountedhere;

	/*
	* v_mountedhere being NULL means that the
	* (uncovered) vnode is in a transient state
	* (mounting or unmounting), so loop until it
	* settles down.
	*/
	if (mp != NULL)
	break;
	vput(vp);
	}
	if (error != 0)
	break;
	if (vp == NULL)
	continue; /* no mountpoint, nothing to do */

	/*
	* The mount-point vnode is kept locked to avoid spurious EBUSY
	* from a concurrent umount.
	* The vnode lock must have recursive locking enabled.
	*/
	vfs_ref(mp);
	error = dounmount(mp, fflags, curthread);
	KASSERT_IMPLY(error == 0, vrefcnt(vp) == 1,
	("extra references after unmount"));
	vput(vp);
	if (error != 0)
	break;
	}
	KASSERT_IMPLY((fflags & MS_FORCE) != 0, error == 0,
	("force unmounting failed"));
	return (error);
	}

	int
	-zfsctl_snapshot_unmount(char *snapname, int flags __unused)
	+zfsctl_snapshot_unmount(const char *snapname, int flags __unused)
	{
	vfs_t *vfsp = NULL;
	zfsvfs_t *zfsvfs = NULL;

	if (strchr(snapname, '@') == NULL)
	return (0);

	int err = getzfsvfs(snapname, &zfsvfs);
	if (err != 0) {
	ASSERT3P(zfsvfs, ==, NULL);
	return (0);
	}
	vfsp = zfsvfs->z_vfs;

	ASSERT(!dsl_pool_config_held(dmu_objset_pool(zfsvfs->z_os)));

	vfs_ref(vfsp);
	vfs_unbusy(vfsp);
	return (dounmount(vfsp, MS_FORCE, curthread));
	}
	Index: vendor-sys/openzfs/dist/module/os/freebsd/zfs/zfs_vfsops.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/freebsd/zfs/zfs_vfsops.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/freebsd/zfs/zfs_vfsops.c (revision 366774)
	@@ -1,2293 +1,2315 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2011 Pawel Jakub Dawidek <pawel@dawidek.net>.
	* All rights reserved.
	* Copyright (c) 2012, 2015 by Delphix. All rights reserved.
	* Copyright (c) 2014 Integros [integros.com]
	* Copyright 2016 Nexenta Systems, Inc. All rights reserved.
	*/

	/* Portions Copyright 2010 Robert Milkowski */

	#include <sys/types.h>
	#include <sys/param.h>
	#include <sys/systm.h>
	#include <sys/kernel.h>
	#include <sys/sysmacros.h>
	#include <sys/kmem.h>
	#include <sys/acl.h>
	#include <sys/vnode.h>
	#include <sys/vfs.h>
	#include <sys/mntent.h>
	#include <sys/mount.h>
	#include <sys/cmn_err.h>
	#include <sys/zfs_znode.h>
	#include <sys/zfs_dir.h>
	#include <sys/zil.h>
	#include <sys/fs/zfs.h>
	#include <sys/dmu.h>
	#include <sys/dsl_prop.h>
	#include <sys/dsl_dataset.h>
	#include <sys/dsl_deleg.h>
	#include <sys/spa.h>
	#include <sys/zap.h>
	#include <sys/sa.h>
	#include <sys/sa_impl.h>
	#include <sys/policy.h>
	#include <sys/atomic.h>
	#include <sys/zfs_ioctl.h>
	#include <sys/zfs_ctldir.h>
	#include <sys/zfs_fuid.h>
	#include <sys/sunddi.h>
	#include <sys/dmu_objset.h>
	#include <sys/dsl_dir.h>
	#include <sys/spa_boot.h>
	#include <sys/jail.h>
	#include <ufs/ufs/quota.h>
	#include <sys/zfs_quota.h>

	#include "zfs_comutil.h"

	#ifndef MNTK_VMSETSIZE_BUG
	#define MNTK_VMSETSIZE_BUG 0
	#endif
	#ifndef MNTK_NOMSYNC
	#define MNTK_NOMSYNC 8
	#endif

	/* BEGIN CSTYLED */
	struct mtx zfs_debug_mtx;
	MTX_SYSINIT(zfs_debug_mtx, &zfs_debug_mtx, "zfs_debug", MTX_DEF);

	SYSCTL_NODE(_vfs, OID_AUTO, zfs, CTLFLAG_RW, 0, "ZFS file system");

	int zfs_super_owner;
	SYSCTL_INT(_vfs_zfs, OID_AUTO, super_owner, CTLFLAG_RW, &zfs_super_owner, 0,
	"File system owner can perform privileged operation on his file systems");

	int zfs_debug_level;
	SYSCTL_INT(_vfs_zfs, OID_AUTO, debug, CTLFLAG_RWTUN, &zfs_debug_level, 0,
	"Debug level");

	SYSCTL_NODE(_vfs_zfs, OID_AUTO, version, CTLFLAG_RD, 0, "ZFS versions");
	static int zfs_version_acl = ZFS_ACL_VERSION;
	SYSCTL_INT(_vfs_zfs_version, OID_AUTO, acl, CTLFLAG_RD, &zfs_version_acl, 0,
	"ZFS_ACL_VERSION");
	static int zfs_version_spa = SPA_VERSION;
	SYSCTL_INT(_vfs_zfs_version, OID_AUTO, spa, CTLFLAG_RD, &zfs_version_spa, 0,
	"SPA_VERSION");
	static int zfs_version_zpl = ZPL_VERSION;
	SYSCTL_INT(_vfs_zfs_version, OID_AUTO, zpl, CTLFLAG_RD, &zfs_version_zpl, 0,
	"ZPL_VERSION");
	/* END CSTYLED */

	static int zfs_quotactl(vfs_t vfsp, int cmds, uid_t id, void arg);
	static int zfs_mount(vfs_t *vfsp);
	static int zfs_umount(vfs_t *vfsp, int fflag);
	static int zfs_root(vfs_t vfsp, int flags, vnode_t *vpp);
	static int zfs_statfs(vfs_t vfsp, struct statfs statp);
	static int zfs_vget(vfs_t vfsp, ino_t ino, int flags, vnode_t *vpp);
	static int zfs_sync(vfs_t *vfsp, int waitfor);
	#if __FreeBSD_version >= 1300098
	static int zfs_checkexp(vfs_t vfsp, struct sockaddr nam, uint64_t *extflagsp,
	struct ucred *credanonp, int numsecflavors, int *secflavors);
	#else
	static int zfs_checkexp(vfs_t vfsp, struct sockaddr nam, int *extflagsp,
	struct ucred *credanonp, int numsecflavors, int **secflavors);
	#endif
	static int zfs_fhtovp(vfs_t vfsp, fid_t fidp, int flags, vnode_t **vpp);
	static void zfs_freevfs(vfs_t *vfsp);

	struct vfsops zfs_vfsops = {
	.vfs_mount = zfs_mount,
	.vfs_unmount = zfs_umount,
	#if __FreeBSD_version >= 1300049
	.vfs_root = vfs_cache_root,
	.vfs_cachedroot = zfs_root,
	#else
	.vfs_root = zfs_root,
	#endif
	.vfs_statfs = zfs_statfs,
	.vfs_vget = zfs_vget,
	.vfs_sync = zfs_sync,
	.vfs_checkexp = zfs_checkexp,
	.vfs_fhtovp = zfs_fhtovp,
	.vfs_quotactl = zfs_quotactl,
	};

	VFS_SET(zfs_vfsops, zfs, VFCF_JAIL \| VFCF_DELEGADMIN);

	/*
	* We need to keep a count of active fs's.
	* This is necessary to prevent our module
	* from being unloaded after a umount -f
	*/
	static uint32_t zfs_active_fs_count = 0;

	int
	zfs_get_temporary_prop(dsl_dataset_t ds, zfs_prop_t zfs_prop, uint64_t val,
	char *setpoint)
	{
	int error;
	zfsvfs_t *zfvp;
	vfs_t *vfsp;
	objset_t *os;
	uint64_t tmp = *val;

	error = dmu_objset_from_ds(ds, &os);
	if (error != 0)
	return (error);

	error = getzfsvfs_impl(os, &zfvp);
	if (error != 0)
	return (error);
	if (zfvp == NULL)
	return (ENOENT);
	vfsp = zfvp->z_vfs;
	switch (zfs_prop) {
	case ZFS_PROP_ATIME:
	if (vfs_optionisset(vfsp, MNTOPT_NOATIME, NULL))
	tmp = 0;
	if (vfs_optionisset(vfsp, MNTOPT_ATIME, NULL))
	tmp = 1;
	break;
	case ZFS_PROP_DEVICES:
	if (vfs_optionisset(vfsp, MNTOPT_NODEVICES, NULL))
	tmp = 0;
	if (vfs_optionisset(vfsp, MNTOPT_DEVICES, NULL))
	tmp = 1;
	break;
	case ZFS_PROP_EXEC:
	if (vfs_optionisset(vfsp, MNTOPT_NOEXEC, NULL))
	tmp = 0;
	if (vfs_optionisset(vfsp, MNTOPT_EXEC, NULL))
	tmp = 1;
	break;
	case ZFS_PROP_SETUID:
	if (vfs_optionisset(vfsp, MNTOPT_NOSETUID, NULL))
	tmp = 0;
	if (vfs_optionisset(vfsp, MNTOPT_SETUID, NULL))
	tmp = 1;
	break;
	case ZFS_PROP_READONLY:
	if (vfs_optionisset(vfsp, MNTOPT_RW, NULL))
	tmp = 0;
	if (vfs_optionisset(vfsp, MNTOPT_RO, NULL))
	tmp = 1;
	break;
	case ZFS_PROP_XATTR:
	if (zfvp->z_flags & ZSB_XATTR)
	tmp = zfvp->z_xattr;
	break;
	case ZFS_PROP_NBMAND:
	if (vfs_optionisset(vfsp, MNTOPT_NONBMAND, NULL))
	tmp = 0;
	if (vfs_optionisset(vfsp, MNTOPT_NBMAND, NULL))
	tmp = 1;
	break;
	default:
	vfs_unbusy(vfsp);
	return (ENOENT);
	}

	vfs_unbusy(vfsp);
	if (tmp != *val) {
	(void) strcpy(setpoint, "temporary");
	*val = tmp;
	}
	return (0);
	}

	static int
	zfs_getquota(zfsvfs_t zfsvfs, uid_t id, int isgroup, struct dqblk64 dqp)
	{
	int error = 0;
	char buf[32];
	uint64_t usedobj, quotaobj;
	uint64_t quota, used = 0;
	timespec_t now;

	usedobj = isgroup ? DMU_GROUPUSED_OBJECT : DMU_USERUSED_OBJECT;
	quotaobj = isgroup ? zfsvfs->z_groupquota_obj : zfsvfs->z_userquota_obj;

	if (quotaobj == 0 \|\| zfsvfs->z_replay) {
	error = ENOENT;
	goto done;
	}
	(void) sprintf(buf, "%llx", (longlong_t)id);
	if ((error = zap_lookup(zfsvfs->z_os, quotaobj,
	buf, sizeof (quota), 1, &quota)) != 0) {
	dprintf("%s(%d): quotaobj lookup failed\n",
	__FUNCTION__, __LINE__);
	goto done;
	}
	/*
	* quota(8) uses bsoftlimit as "quoota", and hardlimit as "limit".
	* So we set them to be the same.
	*/
	dqp->dqb_bsoftlimit = dqp->dqb_bhardlimit = btodb(quota);
	error = zap_lookup(zfsvfs->z_os, usedobj, buf, sizeof (used), 1, &used);
	if (error && error != ENOENT) {
	dprintf("%s(%d): usedobj failed; %d\n",
	__FUNCTION__, __LINE__, error);
	goto done;
	}
	dqp->dqb_curblocks = btodb(used);
	dqp->dqb_ihardlimit = dqp->dqb_isoftlimit = 0;
	vfs_timestamp(&now);
	/*
	* Setting this to 0 causes FreeBSD quota(8) to print
	* the number of days since the epoch, which isn't
	* particularly useful.
	*/
	dqp->dqb_btime = dqp->dqb_itime = now.tv_sec;
	done:
	return (error);
	}

	static int
	zfs_quotactl(vfs_t vfsp, int cmds, uid_t id, void arg)
	{
	zfsvfs_t *zfsvfs = vfsp->vfs_data;
	struct thread *td;
	int cmd, type, error = 0;
	int bitsize;
	zfs_userquota_prop_t quota_type;
	struct dqblk64 dqblk = { 0 };

	td = curthread;
	cmd = cmds >> SUBCMDSHIFT;
	type = cmds & SUBCMDMASK;

	ZFS_ENTER(zfsvfs);
	if (id == -1) {
	switch (type) {
	case USRQUOTA:
	id = td->td_ucred->cr_ruid;
	break;
	case GRPQUOTA:
	id = td->td_ucred->cr_rgid;
	break;
	default:
	error = EINVAL;
	if (cmd == Q_QUOTAON \|\| cmd == Q_QUOTAOFF)
	vfs_unbusy(vfsp);
	goto done;
	}
	}
	/*
	* Map BSD type to:
	* ZFS_PROP_USERUSED,
	* ZFS_PROP_USERQUOTA,
	* ZFS_PROP_GROUPUSED,
	* ZFS_PROP_GROUPQUOTA
	*/
	switch (cmd) {
	case Q_SETQUOTA:
	case Q_SETQUOTA32:
	if (type == USRQUOTA)
	quota_type = ZFS_PROP_USERQUOTA;
	else if (type == GRPQUOTA)
	quota_type = ZFS_PROP_GROUPQUOTA;
	else
	error = EINVAL;
	break;
	case Q_GETQUOTA:
	case Q_GETQUOTA32:
	if (type == USRQUOTA)
	quota_type = ZFS_PROP_USERUSED;
	else if (type == GRPQUOTA)
	quota_type = ZFS_PROP_GROUPUSED;
	else
	error = EINVAL;
	break;
	}

	/*
	* Depending on the cmd, we may need to get
	* the ruid and domain (see fuidstr_to_sid?),
	* the fuid (how?), or other information.
	* Create fuid using zfs_fuid_create(zfsvfs, id,
	* ZFS_OWNER or ZFS_GROUP, cr, &fuidp)?
	* I think I can use just the id?
	*
	* Look at zfs_id_overquota() to look up a quota.
	* zap_lookup(something, quotaobj, fuidstring,
	* sizeof (long long), 1, &quota)
	*
	* See zfs_set_userquota() to set a quota.
	*/
	if ((uint32_t)type >= MAXQUOTAS) {
	error = EINVAL;
	goto done;
	}

	switch (cmd) {
	case Q_GETQUOTASIZE:
	bitsize = 64;
	error = copyout(&bitsize, arg, sizeof (int));
	break;
	case Q_QUOTAON:
	// As far as I can tell, you can't turn quotas on or off on zfs
	error = 0;
	vfs_unbusy(vfsp);
	break;
	case Q_QUOTAOFF:
	error = ENOTSUP;
	vfs_unbusy(vfsp);
	break;
	case Q_SETQUOTA:
	error = copyin(arg, &dqblk, sizeof (dqblk));
	if (error == 0)
	error = zfs_set_userquota(zfsvfs, quota_type,
	"", id, dbtob(dqblk.dqb_bhardlimit));
	break;
	case Q_GETQUOTA:
	error = zfs_getquota(zfsvfs, id, type == GRPQUOTA, &dqblk);
	if (error == 0)
	error = copyout(&dqblk, arg, sizeof (dqblk));
	break;
	default:
	error = EINVAL;
	break;
	}
	done:
	ZFS_EXIT(zfsvfs);
	return (error);
	}


	boolean_t
	zfs_is_readonly(zfsvfs_t *zfsvfs)
	{
	return (!!(zfsvfs->z_vfs->vfs_flag & VFS_RDONLY));
	}

	/ARGSUSED/
	static int
	zfs_sync(vfs_t *vfsp, int waitfor)
	{

	/*
	* Data integrity is job one. We don't want a compromised kernel
	* writing to the storage pool, so we never sync during panic.
	*/
	if (panicstr)
	return (0);

	/*
	* Ignore the system syncher. ZFS already commits async data
	* at zfs_txg_timeout intervals.
	*/
	if (waitfor == MNT_LAZY)
	return (0);

	if (vfsp != NULL) {
	/*
	* Sync a specific filesystem.
	*/
	zfsvfs_t *zfsvfs = vfsp->vfs_data;
	dsl_pool_t *dp;
	int error;

	error = vfs_stdsync(vfsp, waitfor);
	if (error != 0)
	return (error);

	ZFS_ENTER(zfsvfs);
	dp = dmu_objset_pool(zfsvfs->z_os);

	/*
	* If the system is shutting down, then skip any
	* filesystems which may exist on a suspended pool.
	*/
	if (rebooting && spa_suspended(dp->dp_spa)) {
	ZFS_EXIT(zfsvfs);
	return (0);
	}

	if (zfsvfs->z_log != NULL)
	zil_commit(zfsvfs->z_log, 0);

	ZFS_EXIT(zfsvfs);
	} else {
	/*
	* Sync all ZFS filesystems. This is what happens when you
	* run sync(1M). Unlike other filesystems, ZFS honors the
	* request by waiting for all pools to commit all dirty data.
	*/
	spa_sync_allpools();
	}

	return (0);
	}

	static void
	atime_changed_cb(void *arg, uint64_t newval)
	{
	zfsvfs_t *zfsvfs = arg;

	if (newval == TRUE) {
	zfsvfs->z_atime = TRUE;
	zfsvfs->z_vfs->vfs_flag &= ~MNT_NOATIME;
	vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_NOATIME);
	vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_ATIME, NULL, 0);
	} else {
	zfsvfs->z_atime = FALSE;
	zfsvfs->z_vfs->vfs_flag \|= MNT_NOATIME;
	vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_ATIME);
	vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_NOATIME, NULL, 0);
	}
	}

	static void
	xattr_changed_cb(void *arg, uint64_t newval)
	{
	zfsvfs_t *zfsvfs = arg;

	if (newval == ZFS_XATTR_OFF) {
	zfsvfs->z_flags &= ~ZSB_XATTR;
	} else {
	zfsvfs->z_flags \|= ZSB_XATTR;

	if (newval == ZFS_XATTR_SA)
	zfsvfs->z_xattr_sa = B_TRUE;
	else
	zfsvfs->z_xattr_sa = B_FALSE;
	}
	}

	static void
	blksz_changed_cb(void *arg, uint64_t newval)
	{
	zfsvfs_t *zfsvfs = arg;
	ASSERT3U(newval, <=, spa_maxblocksize(dmu_objset_spa(zfsvfs->z_os)));
	ASSERT3U(newval, >=, SPA_MINBLOCKSIZE);
	ASSERT(ISP2(newval));

	zfsvfs->z_max_blksz = newval;
	zfsvfs->z_vfs->mnt_stat.f_iosize = newval;
	}

	static void
	readonly_changed_cb(void *arg, uint64_t newval)
	{
	zfsvfs_t *zfsvfs = arg;

	if (newval) {
	/* XXX locking on vfs_flag? */
	zfsvfs->z_vfs->vfs_flag \|= VFS_RDONLY;
	vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_RW);
	vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_RO, NULL, 0);
	} else {
	/* XXX locking on vfs_flag? */
	zfsvfs->z_vfs->vfs_flag &= ~VFS_RDONLY;
	vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_RO);
	vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_RW, NULL, 0);
	}
	}

	static void
	setuid_changed_cb(void *arg, uint64_t newval)
	{
	zfsvfs_t *zfsvfs = arg;

	if (newval == FALSE) {
	zfsvfs->z_vfs->vfs_flag \|= VFS_NOSETUID;
	vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_SETUID);
	vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_NOSETUID, NULL, 0);
	} else {
	zfsvfs->z_vfs->vfs_flag &= ~VFS_NOSETUID;
	vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_NOSETUID);
	vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_SETUID, NULL, 0);
	}
	}

	static void
	exec_changed_cb(void *arg, uint64_t newval)
	{
	zfsvfs_t *zfsvfs = arg;

	if (newval == FALSE) {
	zfsvfs->z_vfs->vfs_flag \|= VFS_NOEXEC;
	vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_EXEC);
	vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_NOEXEC, NULL, 0);
	} else {
	zfsvfs->z_vfs->vfs_flag &= ~VFS_NOEXEC;
	vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_NOEXEC);
	vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_EXEC, NULL, 0);
	}
	}

	/*
	* The nbmand mount option can be changed at mount time.
	* We can't allow it to be toggled on live file systems or incorrect
	* behavior may be seen from cifs clients
	*
	* This property isn't registered via dsl_prop_register(), but this callback
	* will be called when a file system is first mounted
	*/
	static void
	nbmand_changed_cb(void *arg, uint64_t newval)
	{
	zfsvfs_t *zfsvfs = arg;
	if (newval == FALSE) {
	vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_NBMAND);
	vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_NONBMAND, NULL, 0);
	} else {
	vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_NONBMAND);
	vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_NBMAND, NULL, 0);
	}
	}

	static void
	snapdir_changed_cb(void *arg, uint64_t newval)
	{
	zfsvfs_t *zfsvfs = arg;

	zfsvfs->z_show_ctldir = newval;
	}

	static void
	vscan_changed_cb(void *arg, uint64_t newval)
	{
	zfsvfs_t *zfsvfs = arg;

	zfsvfs->z_vscan = newval;
	}

	static void
	acl_mode_changed_cb(void *arg, uint64_t newval)
	{
	zfsvfs_t *zfsvfs = arg;

	zfsvfs->z_acl_mode = newval;
	}

	static void
	acl_inherit_changed_cb(void *arg, uint64_t newval)
	{
	zfsvfs_t *zfsvfs = arg;

	zfsvfs->z_acl_inherit = newval;
	}

	+static void
	+acl_type_changed_cb(void *arg, uint64_t newval)
	+{
	+ zfsvfs_t *zfsvfs = arg;
	+
	+ zfsvfs->z_acl_type = newval;
	+}
	+
	static int
	zfs_register_callbacks(vfs_t *vfsp)
	{
	struct dsl_dataset *ds = NULL;
	objset_t *os = NULL;
	zfsvfs_t *zfsvfs = NULL;
	uint64_t nbmand;
	boolean_t readonly = B_FALSE;
	boolean_t do_readonly = B_FALSE;
	boolean_t setuid = B_FALSE;
	boolean_t do_setuid = B_FALSE;
	boolean_t exec = B_FALSE;
	boolean_t do_exec = B_FALSE;
	boolean_t xattr = B_FALSE;
	boolean_t atime = B_FALSE;
	boolean_t do_atime = B_FALSE;
	boolean_t do_xattr = B_FALSE;
	int error = 0;

	ASSERT(vfsp);
	zfsvfs = vfsp->vfs_data;
	ASSERT(zfsvfs);
	os = zfsvfs->z_os;

	/*
	* This function can be called for a snapshot when we update snapshot's
	* mount point, which isn't really supported.
	*/
	if (dmu_objset_is_snapshot(os))
	return (EOPNOTSUPP);

	/*
	* The act of registering our callbacks will destroy any mount
	* options we may have. In order to enable temporary overrides
	* of mount options, we stash away the current values and
	* restore them after we register the callbacks.
	*/
	if (vfs_optionisset(vfsp, MNTOPT_RO, NULL) \|\|
	!spa_writeable(dmu_objset_spa(os))) {
	readonly = B_TRUE;
	do_readonly = B_TRUE;
	} else if (vfs_optionisset(vfsp, MNTOPT_RW, NULL)) {
	readonly = B_FALSE;
	do_readonly = B_TRUE;
	}
	if (vfs_optionisset(vfsp, MNTOPT_NOSETUID, NULL)) {
	setuid = B_FALSE;
	do_setuid = B_TRUE;
	} else if (vfs_optionisset(vfsp, MNTOPT_SETUID, NULL)) {
	setuid = B_TRUE;
	do_setuid = B_TRUE;
	}
	if (vfs_optionisset(vfsp, MNTOPT_NOEXEC, NULL)) {
	exec = B_FALSE;
	do_exec = B_TRUE;
	} else if (vfs_optionisset(vfsp, MNTOPT_EXEC, NULL)) {
	exec = B_TRUE;
	do_exec = B_TRUE;
	}
	if (vfs_optionisset(vfsp, MNTOPT_NOXATTR, NULL)) {
	zfsvfs->z_xattr = xattr = ZFS_XATTR_OFF;
	do_xattr = B_TRUE;
	} else if (vfs_optionisset(vfsp, MNTOPT_XATTR, NULL)) {
	zfsvfs->z_xattr = xattr = ZFS_XATTR_DIR;
	do_xattr = B_TRUE;
	} else if (vfs_optionisset(vfsp, MNTOPT_DIRXATTR, NULL)) {
	zfsvfs->z_xattr = xattr = ZFS_XATTR_DIR;
	do_xattr = B_TRUE;
	} else if (vfs_optionisset(vfsp, MNTOPT_SAXATTR, NULL)) {
	zfsvfs->z_xattr = xattr = ZFS_XATTR_SA;
	do_xattr = B_TRUE;
	}
	if (vfs_optionisset(vfsp, MNTOPT_NOATIME, NULL)) {
	atime = B_FALSE;
	do_atime = B_TRUE;
	} else if (vfs_optionisset(vfsp, MNTOPT_ATIME, NULL)) {
	atime = B_TRUE;
	do_atime = B_TRUE;
	}

	/*
	* We need to enter pool configuration here, so that we can use
	* dsl_prop_get_int_ds() to handle the special nbmand property below.
	* dsl_prop_get_integer() can not be used, because it has to acquire
	* spa_namespace_lock and we can not do that because we already hold
	* z_teardown_lock. The problem is that spa_write_cachefile() is called
	* with spa_namespace_lock held and the function calls ZFS vnode
	* operations to write the cache file and thus z_teardown_lock is
	* acquired after spa_namespace_lock.
	*/
	ds = dmu_objset_ds(os);
	dsl_pool_config_enter(dmu_objset_pool(os), FTAG);

	/*
	* nbmand is a special property. It can only be changed at
	* mount time.
	*
	* This is weird, but it is documented to only be changeable
	* at mount time.
	*/
	if (vfs_optionisset(vfsp, MNTOPT_NONBMAND, NULL)) {
	nbmand = B_FALSE;
	} else if (vfs_optionisset(vfsp, MNTOPT_NBMAND, NULL)) {
	nbmand = B_TRUE;
	} else if ((error = dsl_prop_get_int_ds(ds, "nbmand", &nbmand) != 0)) {
	dsl_pool_config_exit(dmu_objset_pool(os), FTAG);
	return (error);
	}

	/*
	* Register property callbacks.
	*
	* It would probably be fine to just check for i/o error from
	* the first prop_register(), but I guess I like to go
	* overboard...
	*/
	error = dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_ATIME), atime_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_XATTR), xattr_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_RECORDSIZE), blksz_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_READONLY), readonly_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_SETUID), setuid_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_EXEC), exec_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_SNAPDIR), snapdir_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	+ zfs_prop_to_name(ZFS_PROP_ACLTYPE), acl_type_changed_cb, zfsvfs);
	+ error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_ACLMODE), acl_mode_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_ACLINHERIT), acl_inherit_changed_cb,
	zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_VSCAN), vscan_changed_cb, zfsvfs);
	dsl_pool_config_exit(dmu_objset_pool(os), FTAG);
	if (error)
	goto unregister;

	/*
	* Invoke our callbacks to restore temporary mount options.
	*/
	if (do_readonly)
	readonly_changed_cb(zfsvfs, readonly);
	if (do_setuid)
	setuid_changed_cb(zfsvfs, setuid);
	if (do_exec)
	exec_changed_cb(zfsvfs, exec);
	if (do_xattr)
	xattr_changed_cb(zfsvfs, xattr);
	if (do_atime)
	atime_changed_cb(zfsvfs, atime);

	nbmand_changed_cb(zfsvfs, nbmand);

	return (0);

	unregister:
	dsl_prop_unregister_all(ds, zfsvfs);
	return (error);
	}

	/*
	* Associate this zfsvfs with the given objset, which must be owned.
	* This will cache a bunch of on-disk state from the objset in the
	* zfsvfs.
	*/
	static int
	zfsvfs_init(zfsvfs_t zfsvfs, objset_t os)
	{
	int error;
	uint64_t val;

	zfsvfs->z_max_blksz = SPA_OLD_MAXBLOCKSIZE;
	zfsvfs->z_show_ctldir = ZFS_SNAPDIR_VISIBLE;
	zfsvfs->z_os = os;

	error = zfs_get_zplprop(os, ZFS_PROP_VERSION, &zfsvfs->z_version);
	if (error != 0)
	return (error);
	if (zfsvfs->z_version >
	zfs_zpl_version_map(spa_version(dmu_objset_spa(os)))) {
	(void) printf("Can't mount a version %lld file system "
	"on a version %lld pool\n. Pool must be upgraded to mount "
	"this file system.", (u_longlong_t)zfsvfs->z_version,
	(u_longlong_t)spa_version(dmu_objset_spa(os)));
	return (SET_ERROR(ENOTSUP));
	}
	error = zfs_get_zplprop(os, ZFS_PROP_NORMALIZE, &val);
	if (error != 0)
	return (error);
	zfsvfs->z_norm = (int)val;

	error = zfs_get_zplprop(os, ZFS_PROP_UTF8ONLY, &val);
	if (error != 0)
	return (error);
	zfsvfs->z_utf8 = (val != 0);

	error = zfs_get_zplprop(os, ZFS_PROP_CASE, &val);
	if (error != 0)
	return (error);
	zfsvfs->z_case = (uint_t)val;

	+ error = zfs_get_zplprop(os, ZFS_PROP_ACLTYPE, &val);
	+ if (error != 0)
	+ return (error);
	+ zfsvfs->z_acl_type = (uint_t)val;
	+
	/*
	* Fold case on file systems that are always or sometimes case
	* insensitive.
	*/
	if (zfsvfs->z_case == ZFS_CASE_INSENSITIVE \|\|
	zfsvfs->z_case == ZFS_CASE_MIXED)
	zfsvfs->z_norm \|= U8_TEXTPREP_TOUPPER;

	zfsvfs->z_use_fuids = USE_FUIDS(zfsvfs->z_version, zfsvfs->z_os);
	zfsvfs->z_use_sa = USE_SA(zfsvfs->z_version, zfsvfs->z_os);

	uint64_t sa_obj = 0;
	if (zfsvfs->z_use_sa) {
	/* should either have both of these objects or none */
	error = zap_lookup(os, MASTER_NODE_OBJ, ZFS_SA_ATTRS, 8, 1,
	&sa_obj);
	if (error != 0)
	return (error);
	}

	error = sa_setup(os, sa_obj, zfs_attr_table, ZPL_END,
	&zfsvfs->z_attr_table);
	if (error != 0)
	return (error);

	if (zfsvfs->z_version >= ZPL_VERSION_SA)
	sa_register_update_callback(os, zfs_sa_upgrade);

	error = zap_lookup(os, MASTER_NODE_OBJ, ZFS_ROOT_OBJ, 8, 1,
	&zfsvfs->z_root);
	if (error != 0)
	return (error);
	ASSERT(zfsvfs->z_root != 0);

	error = zap_lookup(os, MASTER_NODE_OBJ, ZFS_UNLINKED_SET, 8, 1,
	&zfsvfs->z_unlinkedobj);
	if (error != 0)
	return (error);

	error = zap_lookup(os, MASTER_NODE_OBJ,
	zfs_userquota_prop_prefixes[ZFS_PROP_USERQUOTA],
	8, 1, &zfsvfs->z_userquota_obj);
	if (error == ENOENT)
	zfsvfs->z_userquota_obj = 0;
	else if (error != 0)
	return (error);

	error = zap_lookup(os, MASTER_NODE_OBJ,
	zfs_userquota_prop_prefixes[ZFS_PROP_GROUPQUOTA],
	8, 1, &zfsvfs->z_groupquota_obj);
	if (error == ENOENT)
	zfsvfs->z_groupquota_obj = 0;
	else if (error != 0)
	return (error);

	error = zap_lookup(os, MASTER_NODE_OBJ,
	zfs_userquota_prop_prefixes[ZFS_PROP_PROJECTQUOTA],
	8, 1, &zfsvfs->z_projectquota_obj);
	if (error == ENOENT)
	zfsvfs->z_projectquota_obj = 0;
	else if (error != 0)
	return (error);

	error = zap_lookup(os, MASTER_NODE_OBJ,
	zfs_userquota_prop_prefixes[ZFS_PROP_USEROBJQUOTA],
	8, 1, &zfsvfs->z_userobjquota_obj);
	if (error == ENOENT)
	zfsvfs->z_userobjquota_obj = 0;
	else if (error != 0)
	return (error);

	error = zap_lookup(os, MASTER_NODE_OBJ,
	zfs_userquota_prop_prefixes[ZFS_PROP_GROUPOBJQUOTA],
	8, 1, &zfsvfs->z_groupobjquota_obj);
	if (error == ENOENT)
	zfsvfs->z_groupobjquota_obj = 0;
	else if (error != 0)
	return (error);

	error = zap_lookup(os, MASTER_NODE_OBJ,
	zfs_userquota_prop_prefixes[ZFS_PROP_PROJECTOBJQUOTA],
	8, 1, &zfsvfs->z_projectobjquota_obj);
	if (error == ENOENT)
	zfsvfs->z_projectobjquota_obj = 0;
	else if (error != 0)
	return (error);

	error = zap_lookup(os, MASTER_NODE_OBJ, ZFS_FUID_TABLES, 8, 1,
	&zfsvfs->z_fuid_obj);
	if (error == ENOENT)
	zfsvfs->z_fuid_obj = 0;
	else if (error != 0)
	return (error);

	error = zap_lookup(os, MASTER_NODE_OBJ, ZFS_SHARES_DIR, 8, 1,
	&zfsvfs->z_shares_dir);
	if (error == ENOENT)
	zfsvfs->z_shares_dir = 0;
	else if (error != 0)
	return (error);

	/*
	* Only use the name cache if we are looking for a
	* name on a file system that does not require normalization
	* or case folding. We can also look there if we happen to be
	* on a non-normalizing, mixed sensitivity file system IF we
	* are looking for the exact name (which is always the case on
	* FreeBSD).
	*/
	zfsvfs->z_use_namecache = !zfsvfs->z_norm \|\|
	((zfsvfs->z_case == ZFS_CASE_MIXED) &&
	!(zfsvfs->z_norm & ~U8_TEXTPREP_TOUPPER));

	return (0);
	}

	taskq_t *zfsvfs_taskq;

	static void
	zfsvfs_task_unlinked_drain(void *context, int pending __unused)
	{

	zfs_unlinked_drain((zfsvfs_t *)context);
	}

	int
	zfsvfs_create(const char osname, boolean_t readonly, zfsvfs_t *zfvp)
	{
	objset_t *os;
	zfsvfs_t *zfsvfs;
	int error;
	boolean_t ro = (readonly \|\| (strchr(osname, '@') != NULL));

	/*
	* XXX: Fix struct statfs so this isn't necessary!
	*
	* The 'osname' is used as the filesystem's special node, which means
	* it must fit in statfs.f_mntfromname, or else it can't be
	* enumerated, so libzfs_mnttab_find() returns NULL, which causes
	* 'zfs unmount' to think it's not mounted when it is.
	*/
	if (strlen(osname) >= MNAMELEN)
	return (SET_ERROR(ENAMETOOLONG));

	zfsvfs = kmem_zalloc(sizeof (zfsvfs_t), KM_SLEEP);

	error = dmu_objset_own(osname, DMU_OST_ZFS, ro, B_TRUE, zfsvfs,
	&os);
	if (error != 0) {
	kmem_free(zfsvfs, sizeof (zfsvfs_t));
	return (error);
	}

	error = zfsvfs_create_impl(zfvp, zfsvfs, os);

	return (error);
	}


	int
	zfsvfs_create_impl(zfsvfs_t *zfvp, zfsvfs_t zfsvfs, objset_t *os)
	{
	int error;

	zfsvfs->z_vfs = NULL;
	zfsvfs->z_parent = zfsvfs;

	mutex_init(&zfsvfs->z_znodes_lock, NULL, MUTEX_DEFAULT, NULL);
	mutex_init(&zfsvfs->z_lock, NULL, MUTEX_DEFAULT, NULL);
	list_create(&zfsvfs->z_all_znodes, sizeof (znode_t),
	offsetof(znode_t, z_link_node));
	TASK_INIT(&zfsvfs->z_unlinked_drain_task, 0,
	zfsvfs_task_unlinked_drain, zfsvfs);
	#ifdef DIAGNOSTIC
	rrm_init(&zfsvfs->z_teardown_lock, B_TRUE);
	#else
	rrm_init(&zfsvfs->z_teardown_lock, B_FALSE);
	#endif
	ZFS_INIT_TEARDOWN_INACTIVE(zfsvfs);
	rw_init(&zfsvfs->z_fuid_lock, NULL, RW_DEFAULT, NULL);
	for (int i = 0; i != ZFS_OBJ_MTX_SZ; i++)
	mutex_init(&zfsvfs->z_hold_mtx[i], NULL, MUTEX_DEFAULT, NULL);

	error = zfsvfs_init(zfsvfs, os);
	if (error != 0) {
	dmu_objset_disown(os, B_TRUE, zfsvfs);
	*zfvp = NULL;
	kmem_free(zfsvfs, sizeof (zfsvfs_t));
	return (error);
	}

	*zfvp = zfsvfs;
	return (0);
	}

	static int
	zfsvfs_setup(zfsvfs_t *zfsvfs, boolean_t mounting)
	{
	int error;

	/*
	* Check for a bad on-disk format version now since we
	* lied about owning the dataset readonly before.
	*/
	if (!(zfsvfs->z_vfs->vfs_flag & VFS_RDONLY) &&
	dmu_objset_incompatible_encryption_version(zfsvfs->z_os))
	return (SET_ERROR(EROFS));

	error = zfs_register_callbacks(zfsvfs->z_vfs);
	if (error)
	return (error);

	zfsvfs->z_log = zil_open(zfsvfs->z_os, zfs_get_data);

	/*
	* If we are not mounting (ie: online recv), then we don't
	* have to worry about replaying the log as we blocked all
	* operations out since we closed the ZIL.
	*/
	if (mounting) {
	boolean_t readonly;

	ASSERT3P(zfsvfs->z_kstat.dk_kstats, ==, NULL);
	dataset_kstats_create(&zfsvfs->z_kstat, zfsvfs->z_os);

	/*
	* During replay we remove the read only flag to
	* allow replays to succeed.
	*/
	readonly = zfsvfs->z_vfs->vfs_flag & VFS_RDONLY;
	if (readonly != 0) {
	zfsvfs->z_vfs->vfs_flag &= ~VFS_RDONLY;
	} else {
	dsl_dir_t *dd;
	zap_stats_t zs;

	if (zap_get_stats(zfsvfs->z_os, zfsvfs->z_unlinkedobj,
	&zs) == 0) {
	dataset_kstats_update_nunlinks_kstat(
	&zfsvfs->z_kstat, zs.zs_num_entries);
	dprintf_ds(zfsvfs->z_os->os_dsl_dataset,
	"num_entries in unlinked set: %llu",
	zs.zs_num_entries);
	}

	zfs_unlinked_drain(zfsvfs);
	dd = zfsvfs->z_os->os_dsl_dataset->ds_dir;
	dd->dd_activity_cancelled = B_FALSE;
	}

	/*
	* Parse and replay the intent log.
	*
	* Because of ziltest, this must be done after
	* zfs_unlinked_drain(). (Further note: ziltest
	* doesn't use readonly mounts, where
	* zfs_unlinked_drain() isn't called.) This is because
	* ziltest causes spa_sync() to think it's committed,
	* but actually it is not, so the intent log contains
	* many txg's worth of changes.
	*
	* In particular, if object N is in the unlinked set in
	* the last txg to actually sync, then it could be
	* actually freed in a later txg and then reallocated
	* in a yet later txg. This would write a "create
	* object N" record to the intent log. Normally, this
	* would be fine because the spa_sync() would have
	* written out the fact that object N is free, before
	* we could write the "create object N" intent log
	* record.
	*
	* But when we are in ziltest mode, we advance the "open
	* txg" without actually spa_sync()-ing the changes to
	* disk. So we would see that object N is still
	* allocated and in the unlinked set, and there is an
	* intent log record saying to allocate it.
	*/
	if (spa_writeable(dmu_objset_spa(zfsvfs->z_os))) {
	if (zil_replay_disable) {
	zil_destroy(zfsvfs->z_log, B_FALSE);
	} else {
	boolean_t use_nc = zfsvfs->z_use_namecache;
	zfsvfs->z_use_namecache = B_FALSE;
	zfsvfs->z_replay = B_TRUE;
	zil_replay(zfsvfs->z_os, zfsvfs,
	zfs_replay_vector);
	zfsvfs->z_replay = B_FALSE;
	zfsvfs->z_use_namecache = use_nc;
	}
	}

	/* restore readonly bit */
	if (readonly != 0)
	zfsvfs->z_vfs->vfs_flag \|= VFS_RDONLY;
	}

	/*
	* Set the objset user_ptr to track its zfsvfs.
	*/
	mutex_enter(&zfsvfs->z_os->os_user_ptr_lock);
	dmu_objset_set_user(zfsvfs->z_os, zfsvfs);
	mutex_exit(&zfsvfs->z_os->os_user_ptr_lock);

	return (0);
	}

	extern krwlock_t zfsvfs_lock; /* in zfs_znode.c */

	void
	zfsvfs_free(zfsvfs_t *zfsvfs)
	{
	int i;

	/*
	* This is a barrier to prevent the filesystem from going away in
	* zfs_znode_move() until we can safely ensure that the filesystem is
	* not unmounted. We consider the filesystem valid before the barrier
	* and invalid after the barrier.
	*/
	rw_enter(&zfsvfs_lock, RW_READER);
	rw_exit(&zfsvfs_lock);

	zfs_fuid_destroy(zfsvfs);

	mutex_destroy(&zfsvfs->z_znodes_lock);
	mutex_destroy(&zfsvfs->z_lock);
	ASSERT(zfsvfs->z_nr_znodes == 0);
	list_destroy(&zfsvfs->z_all_znodes);
	rrm_destroy(&zfsvfs->z_teardown_lock);
	ZFS_DESTROY_TEARDOWN_INACTIVE(zfsvfs);
	rw_destroy(&zfsvfs->z_fuid_lock);
	for (i = 0; i != ZFS_OBJ_MTX_SZ; i++)
	mutex_destroy(&zfsvfs->z_hold_mtx[i]);
	dataset_kstats_destroy(&zfsvfs->z_kstat);
	kmem_free(zfsvfs, sizeof (zfsvfs_t));
	}

	static void
	zfs_set_fuid_feature(zfsvfs_t *zfsvfs)
	{
	zfsvfs->z_use_fuids = USE_FUIDS(zfsvfs->z_version, zfsvfs->z_os);
	if (zfsvfs->z_vfs) {
	if (zfsvfs->z_use_fuids) {
	vfs_set_feature(zfsvfs->z_vfs, VFSFT_XVATTR);
	vfs_set_feature(zfsvfs->z_vfs, VFSFT_SYSATTR_VIEWS);
	vfs_set_feature(zfsvfs->z_vfs, VFSFT_ACEMASKONACCESS);
	vfs_set_feature(zfsvfs->z_vfs, VFSFT_ACLONCREATE);
	vfs_set_feature(zfsvfs->z_vfs, VFSFT_ACCESS_FILTER);
	vfs_set_feature(zfsvfs->z_vfs, VFSFT_REPARSE);
	} else {
	vfs_clear_feature(zfsvfs->z_vfs, VFSFT_XVATTR);
	vfs_clear_feature(zfsvfs->z_vfs, VFSFT_SYSATTR_VIEWS);
	vfs_clear_feature(zfsvfs->z_vfs, VFSFT_ACEMASKONACCESS);
	vfs_clear_feature(zfsvfs->z_vfs, VFSFT_ACLONCREATE);
	vfs_clear_feature(zfsvfs->z_vfs, VFSFT_ACCESS_FILTER);
	vfs_clear_feature(zfsvfs->z_vfs, VFSFT_REPARSE);
	}
	}
	zfsvfs->z_use_sa = USE_SA(zfsvfs->z_version, zfsvfs->z_os);
	}

	static int
	zfs_domount(vfs_t vfsp, char osname)
	{
	uint64_t recordsize, fsid_guid;
	int error = 0;
	zfsvfs_t *zfsvfs;

	ASSERT(vfsp);
	ASSERT(osname);

	error = zfsvfs_create(osname, vfsp->mnt_flag & MNT_RDONLY, &zfsvfs);
	if (error)
	return (error);
	zfsvfs->z_vfs = vfsp;

	if ((error = dsl_prop_get_integer(osname,
	"recordsize", &recordsize, NULL)))
	goto out;
	zfsvfs->z_vfs->vfs_bsize = SPA_MINBLOCKSIZE;
	zfsvfs->z_vfs->mnt_stat.f_iosize = recordsize;

	vfsp->vfs_data = zfsvfs;
	vfsp->mnt_flag \|= MNT_LOCAL;
	vfsp->mnt_kern_flag \|= MNTK_LOOKUP_SHARED;
	vfsp->mnt_kern_flag \|= MNTK_SHARED_WRITES;
	vfsp->mnt_kern_flag \|= MNTK_EXTENDED_SHARED;
	/*
	* This can cause a loss of coherence between ARC and page cache
	* on ZoF - unclear if the problem is in FreeBSD or ZoF
	*/
	vfsp->mnt_kern_flag \|= MNTK_NO_IOPF; /* vn_io_fault can be used */
	vfsp->mnt_kern_flag \|= MNTK_NOMSYNC;
	vfsp->mnt_kern_flag \|= MNTK_VMSETSIZE_BUG;

	#if defined(_KERNEL) && !defined(KMEM_DEBUG)
	vfsp->mnt_kern_flag \|= MNTK_FPLOOKUP;
	#endif
	/*
	* The fsid is 64 bits, composed of an 8-bit fs type, which
	* separates our fsid from any other filesystem types, and a
	* 56-bit objset unique ID. The objset unique ID is unique to
	* all objsets open on this system, provided by unique_create().
	* The 8-bit fs type must be put in the low bits of fsid[1]
	* because that's where other Solaris filesystems put it.
	*/
	fsid_guid = dmu_objset_fsid_guid(zfsvfs->z_os);
	ASSERT((fsid_guid & ~((1ULL<<56)-1)) == 0);
	vfsp->vfs_fsid.val[0] = fsid_guid;
	vfsp->vfs_fsid.val[1] = ((fsid_guid>>32) << 8) \|
	(vfsp->mnt_vfc->vfc_typenum & 0xFF);

	/*
	* Set features for file system.
	*/
	zfs_set_fuid_feature(zfsvfs);
	if (zfsvfs->z_case == ZFS_CASE_INSENSITIVE) {
	vfs_set_feature(vfsp, VFSFT_DIRENTFLAGS);
	vfs_set_feature(vfsp, VFSFT_CASEINSENSITIVE);
	vfs_set_feature(vfsp, VFSFT_NOCASESENSITIVE);
	} else if (zfsvfs->z_case == ZFS_CASE_MIXED) {
	vfs_set_feature(vfsp, VFSFT_DIRENTFLAGS);
	vfs_set_feature(vfsp, VFSFT_CASEINSENSITIVE);
	}
	vfs_set_feature(vfsp, VFSFT_ZEROCOPY_SUPPORTED);

	if (dmu_objset_is_snapshot(zfsvfs->z_os)) {
	uint64_t pval;

	atime_changed_cb(zfsvfs, B_FALSE);
	readonly_changed_cb(zfsvfs, B_TRUE);
	if ((error = dsl_prop_get_integer(osname,
	"xattr", &pval, NULL)))
	goto out;
	xattr_changed_cb(zfsvfs, pval);
	+ if ((error = dsl_prop_get_integer(osname,
	+ "acltype", &pval, NULL)))
	+ goto out;
	+ acl_type_changed_cb(zfsvfs, pval);
	zfsvfs->z_issnap = B_TRUE;
	zfsvfs->z_os->os_sync = ZFS_SYNC_DISABLED;

	mutex_enter(&zfsvfs->z_os->os_user_ptr_lock);
	dmu_objset_set_user(zfsvfs->z_os, zfsvfs);
	mutex_exit(&zfsvfs->z_os->os_user_ptr_lock);
	} else {
	if ((error = zfsvfs_setup(zfsvfs, B_TRUE)))
	goto out;
	}

	vfs_mountedfrom(vfsp, osname);

	if (!zfsvfs->z_issnap)
	zfsctl_create(zfsvfs);
	out:
	if (error) {
	dmu_objset_disown(zfsvfs->z_os, B_TRUE, zfsvfs);
	zfsvfs_free(zfsvfs);
	} else {
	atomic_inc_32(&zfs_active_fs_count);
	}

	return (error);
	}

	static void
	zfs_unregister_callbacks(zfsvfs_t *zfsvfs)
	{
	objset_t *os = zfsvfs->z_os;

	if (!dmu_objset_is_snapshot(os))
	dsl_prop_unregister_all(dmu_objset_ds(os), zfsvfs);
	}

	static int
	getpoolname(const char osname, char poolname)
	{
	char *p;

	p = strchr(osname, '/');
	if (p == NULL) {
	if (strlen(osname) >= MAXNAMELEN)
	return (ENAMETOOLONG);
	(void) strcpy(poolname, osname);
	} else {
	if (p - osname >= MAXNAMELEN)
	return (ENAMETOOLONG);
	(void) strncpy(poolname, osname, p - osname);
	poolname[p - osname] = '\0';
	}
	return (0);
	}

	/ARGSUSED/
	static int
	zfs_mount(vfs_t *vfsp)
	{
	kthread_t *td = curthread;
	vnode_t *mvp = vfsp->mnt_vnodecovered;
	cred_t *cr = td->td_ucred;
	char *osname;
	int error = 0;
	int canwrite;

	if (vfs_getopt(vfsp->mnt_optnew, "from", (void **)&osname, NULL))
	return (SET_ERROR(EINVAL));

	/*
	* If full-owner-access is enabled and delegated administration is
	* turned on, we must set nosuid.
	*/
	if (zfs_super_owner &&
	dsl_deleg_access(osname, ZFS_DELEG_PERM_MOUNT, cr) != ECANCELED) {
	secpolicy_fs_mount_clearopts(cr, vfsp);
	}

	/*
	* Check for mount privilege?
	*
	* If we don't have privilege then see if
	* we have local permission to allow it
	*/
	error = secpolicy_fs_mount(cr, mvp, vfsp);
	if (error) {
	if (dsl_deleg_access(osname, ZFS_DELEG_PERM_MOUNT, cr) != 0)
	goto out;

	if (!(vfsp->vfs_flag & MS_REMOUNT)) {
	vattr_t vattr;

	/*
	* Make sure user is the owner of the mount point
	* or has sufficient privileges.
	*/

	vattr.va_mask = AT_UID;

	vn_lock(mvp, LK_SHARED \| LK_RETRY);
	if (VOP_GETATTR(mvp, &vattr, cr)) {
	VOP_UNLOCK1(mvp);
	goto out;
	}

	if (secpolicy_vnode_owner(mvp, cr, vattr.va_uid) != 0 &&
	VOP_ACCESS(mvp, VWRITE, cr, td) != 0) {
	VOP_UNLOCK1(mvp);
	goto out;
	}
	VOP_UNLOCK1(mvp);
	}

	secpolicy_fs_mount_clearopts(cr, vfsp);
	}

	/*
	* Refuse to mount a filesystem if we are in a local zone and the
	* dataset is not visible.
	*/
	if (!INGLOBALZONE(curproc) &&
	(!zone_dataset_visible(osname, &canwrite) \|\| !canwrite)) {
	error = SET_ERROR(EPERM);
	goto out;
	}

	vfsp->vfs_flag \|= MNT_NFS4ACLS;

	/*
	* When doing a remount, we simply refresh our temporary properties
	* according to those options set in the current VFS options.
	*/
	if (vfsp->vfs_flag & MS_REMOUNT) {
	zfsvfs_t *zfsvfs = vfsp->vfs_data;

	/*
	* Refresh mount options with z_teardown_lock blocking I/O while
	* the filesystem is in an inconsistent state.
	* The lock also serializes this code with filesystem
	* manipulations between entry to zfs_suspend_fs() and return
	* from zfs_resume_fs().
	*/
	rrm_enter(&zfsvfs->z_teardown_lock, RW_WRITER, FTAG);
	zfs_unregister_callbacks(zfsvfs);
	error = zfs_register_callbacks(vfsp);
	rrm_exit(&zfsvfs->z_teardown_lock, FTAG);
	goto out;
	}

	/* Initial root mount: try hard to import the requested root pool. */
	if ((vfsp->vfs_flag & MNT_ROOTFS) != 0 &&
	(vfsp->vfs_flag & MNT_UPDATE) == 0) {
	char pname[MAXNAMELEN];

	error = getpoolname(osname, pname);
	if (error == 0)
	error = spa_import_rootpool(pname, false);
	if (error)
	goto out;
	}
	DROP_GIANT();
	error = zfs_domount(vfsp, osname);
	PICKUP_GIANT();

	out:
	return (error);
	}

	static int
	zfs_statfs(vfs_t vfsp, struct statfs statp)
	{
	zfsvfs_t *zfsvfs = vfsp->vfs_data;
	uint64_t refdbytes, availbytes, usedobjs, availobjs;

	statp->f_version = STATFS_VERSION;

	ZFS_ENTER(zfsvfs);

	dmu_objset_space(zfsvfs->z_os,
	&refdbytes, &availbytes, &usedobjs, &availobjs);

	/*
	* The underlying storage pool actually uses multiple block sizes.
	* We report the fragsize as the smallest block size we support,
	* and we report our blocksize as the filesystem's maximum blocksize.
	*/
	statp->f_bsize = SPA_MINBLOCKSIZE;
	statp->f_iosize = zfsvfs->z_vfs->mnt_stat.f_iosize;

	/*
	* The following report "total" blocks of various kinds in the
	* file system, but reported in terms of f_frsize - the
	* "fragment" size.
	*/

	statp->f_blocks = (refdbytes + availbytes) >> SPA_MINBLOCKSHIFT;
	statp->f_bfree = availbytes / statp->f_bsize;
	statp->f_bavail = statp->f_bfree; /* no root reservation */

	/*
	* statvfs() should really be called statufs(), because it assumes
	* static metadata. ZFS doesn't preallocate files, so the best
	* we can do is report the max that could possibly fit in f_files,
	* and that minus the number actually used in f_ffree.
	* For f_ffree, report the smaller of the number of object available
	* and the number of blocks (each object will take at least a block).
	*/
	statp->f_ffree = MIN(availobjs, statp->f_bfree);
	statp->f_files = statp->f_ffree + usedobjs;

	/*
	* We're a zfs filesystem.
	*/
	strlcpy(statp->f_fstypename, "zfs",
	sizeof (statp->f_fstypename));

	strlcpy(statp->f_mntfromname, vfsp->mnt_stat.f_mntfromname,
	sizeof (statp->f_mntfromname));
	strlcpy(statp->f_mntonname, vfsp->mnt_stat.f_mntonname,
	sizeof (statp->f_mntonname));

	statp->f_namemax = MAXNAMELEN - 1;

	ZFS_EXIT(zfsvfs);
	return (0);
	}

	static int
	zfs_root(vfs_t vfsp, int flags, vnode_t *vpp)
	{
	zfsvfs_t *zfsvfs = vfsp->vfs_data;
	znode_t *rootzp;
	int error;

	ZFS_ENTER(zfsvfs);

	error = zfs_zget(zfsvfs, zfsvfs->z_root, &rootzp);
	if (error == 0)
	*vpp = ZTOV(rootzp);

	ZFS_EXIT(zfsvfs);

	if (error == 0) {
	error = vn_lock(*vpp, flags);
	if (error != 0) {
	VN_RELE(*vpp);
	*vpp = NULL;
	}
	}
	return (error);
	}

	/*
	* Teardown the zfsvfs::z_os.
	*
	* Note, if 'unmounting' is FALSE, we return with the 'z_teardown_lock'
	* and 'z_teardown_inactive_lock' held.
	*/
	static int
	zfsvfs_teardown(zfsvfs_t *zfsvfs, boolean_t unmounting)
	{
	znode_t *zp;
	dsl_dir_t *dd;

	/*
	* If someone has not already unmounted this file system,
	* drain the zrele_taskq to ensure all active references to the
	* zfsvfs_t have been handled only then can it be safely destroyed.
	*/
	if (zfsvfs->z_os) {
	/*
	* If we're unmounting we have to wait for the list to
	* drain completely.
	*
	* If we're not unmounting there's no guarantee the list
	* will drain completely, but zreles run from the taskq
	* may add the parents of dir-based xattrs to the taskq
	* so we want to wait for these.
	*
	* We can safely read z_nr_znodes without locking because the
	* VFS has already blocked operations which add to the
	* z_all_znodes list and thus increment z_nr_znodes.
	*/
	int round = 0;
	while (zfsvfs->z_nr_znodes > 0) {
	taskq_wait_outstanding(dsl_pool_zrele_taskq(
	dmu_objset_pool(zfsvfs->z_os)), 0);
	if (++round > 1 && !unmounting)
	break;
	}
	}
	rrm_enter(&zfsvfs->z_teardown_lock, RW_WRITER, FTAG);

	if (!unmounting) {
	/*
	* We purge the parent filesystem's vfsp as the parent
	* filesystem and all of its snapshots have their vnode's
	* v_vfsp set to the parent's filesystem's vfsp. Note,
	* 'z_parent' is self referential for non-snapshots.
	*/
	#ifdef FREEBSD_NAMECACHE
	#if __FreeBSD_version >= 1300117
	cache_purgevfs(zfsvfs->z_parent->z_vfs);
	#else
	cache_purgevfs(zfsvfs->z_parent->z_vfs, true);
	#endif
	#endif
	}

	/*
	* Close the zil. NB: Can't close the zil while zfs_inactive
	* threads are blocked as zil_close can call zfs_inactive.
	*/
	if (zfsvfs->z_log) {
	zil_close(zfsvfs->z_log);
	zfsvfs->z_log = NULL;
	}

	ZFS_WLOCK_TEARDOWN_INACTIVE(zfsvfs);

	/*
	* If we are not unmounting (ie: online recv) and someone already
	* unmounted this file system while we were doing the switcheroo,
	* or a reopen of z_os failed then just bail out now.
	*/
	if (!unmounting && (zfsvfs->z_unmounted \|\| zfsvfs->z_os == NULL)) {
	ZFS_WUNLOCK_TEARDOWN_INACTIVE(zfsvfs);
	rrm_exit(&zfsvfs->z_teardown_lock, FTAG);
	return (SET_ERROR(EIO));
	}

	/*
	* At this point there are no vops active, and any new vops will
	* fail with EIO since we have z_teardown_lock for writer (only
	* relevant for forced unmount).
	*
	* Release all holds on dbufs.
	*/
	mutex_enter(&zfsvfs->z_znodes_lock);
	for (zp = list_head(&zfsvfs->z_all_znodes); zp != NULL;
	zp = list_next(&zfsvfs->z_all_znodes, zp))
	if (zp->z_sa_hdl) {
	ASSERT(ZTOV(zp)->v_count >= 0);
	zfs_znode_dmu_fini(zp);
	}
	mutex_exit(&zfsvfs->z_znodes_lock);

	/*
	* If we are unmounting, set the unmounted flag and let new vops
	* unblock. zfs_inactive will have the unmounted behavior, and all
	* other vops will fail with EIO.
	*/
	if (unmounting) {
	zfsvfs->z_unmounted = B_TRUE;
	ZFS_WUNLOCK_TEARDOWN_INACTIVE(zfsvfs);
	rrm_exit(&zfsvfs->z_teardown_lock, FTAG);
	}

	/*
	* z_os will be NULL if there was an error in attempting to reopen
	* zfsvfs, so just return as the properties had already been
	* unregistered and cached data had been evicted before.
	*/
	if (zfsvfs->z_os == NULL)
	return (0);

	/*
	* Unregister properties.
	*/
	zfs_unregister_callbacks(zfsvfs);

	/*
	* Evict cached data
	*/
	if (!zfs_is_readonly(zfsvfs))
	txg_wait_synced(dmu_objset_pool(zfsvfs->z_os), 0);
	dmu_objset_evict_dbufs(zfsvfs->z_os);
	dd = zfsvfs->z_os->os_dsl_dataset->ds_dir;
	dsl_dir_cancel_waiters(dd);

	return (0);
	}

	/ARGSUSED/
	static int
	zfs_umount(vfs_t *vfsp, int fflag)
	{
	kthread_t *td = curthread;
	zfsvfs_t *zfsvfs = vfsp->vfs_data;
	objset_t *os;
	cred_t *cr = td->td_ucred;
	int ret;

	ret = secpolicy_fs_unmount(cr, vfsp);
	if (ret) {
	if (dsl_deleg_access((char *)vfsp->vfs_resource,
	ZFS_DELEG_PERM_MOUNT, cr))
	return (ret);
	}

	/*
	* Unmount any snapshots mounted under .zfs before unmounting the
	* dataset itself.
	*/
	if (zfsvfs->z_ctldir != NULL) {
	if ((ret = zfsctl_umount_snapshots(vfsp, fflag, cr)) != 0)
	return (ret);
	}

	if (fflag & MS_FORCE) {
	/*
	* Mark file system as unmounted before calling
	* vflush(FORCECLOSE). This way we ensure no future vnops
	* will be called and risk operating on DOOMED vnodes.
	*/
	rrm_enter(&zfsvfs->z_teardown_lock, RW_WRITER, FTAG);
	zfsvfs->z_unmounted = B_TRUE;
	rrm_exit(&zfsvfs->z_teardown_lock, FTAG);
	}

	/*
	* Flush all the files.
	*/
	ret = vflush(vfsp, 0, (fflag & MS_FORCE) ? FORCECLOSE : 0, td);
	if (ret != 0)
	return (ret);
	while (taskqueue_cancel(zfsvfs_taskq->tq_queue,
	&zfsvfs->z_unlinked_drain_task, NULL) != 0)
	taskqueue_drain(zfsvfs_taskq->tq_queue,
	&zfsvfs->z_unlinked_drain_task);

	VERIFY(zfsvfs_teardown(zfsvfs, B_TRUE) == 0);
	os = zfsvfs->z_os;

	/*
	* z_os will be NULL if there was an error in
	* attempting to reopen zfsvfs.
	*/
	if (os != NULL) {
	/*
	* Unset the objset user_ptr.
	*/
	mutex_enter(&os->os_user_ptr_lock);
	dmu_objset_set_user(os, NULL);
	mutex_exit(&os->os_user_ptr_lock);

	/*
	* Finally release the objset
	*/
	dmu_objset_disown(os, B_TRUE, zfsvfs);
	}

	/*
	* We can now safely destroy the '.zfs' directory node.
	*/
	if (zfsvfs->z_ctldir != NULL)
	zfsctl_destroy(zfsvfs);
	zfs_freevfs(vfsp);

	return (0);
	}

	static int
	zfs_vget(vfs_t vfsp, ino_t ino, int flags, vnode_t *vpp)
	{
	zfsvfs_t *zfsvfs = vfsp->vfs_data;
	znode_t *zp;
	int err;

	/*
	* zfs_zget() can't operate on virtual entries like .zfs/ or
	* .zfs/snapshot/ directories, that's why we return EOPNOTSUPP.
	* This will make NFS to switch to LOOKUP instead of using VGET.
	*/
	if (ino == ZFSCTL_INO_ROOT \|\| ino == ZFSCTL_INO_SNAPDIR \|\|
	(zfsvfs->z_shares_dir != 0 && ino == zfsvfs->z_shares_dir))
	return (EOPNOTSUPP);

	ZFS_ENTER(zfsvfs);
	err = zfs_zget(zfsvfs, ino, &zp);
	if (err == 0 && zp->z_unlinked) {
	vrele(ZTOV(zp));
	err = EINVAL;
	}
	if (err == 0)
	*vpp = ZTOV(zp);
	ZFS_EXIT(zfsvfs);
	if (err == 0) {
	err = vn_lock(*vpp, flags);
	if (err != 0)
	vrele(*vpp);
	}
	if (err != 0)
	*vpp = NULL;
	return (err);
	}

	static int
	#if __FreeBSD_version >= 1300098
	zfs_checkexp(vfs_t vfsp, struct sockaddr nam, uint64_t *extflagsp,
	struct ucred *credanonp, int numsecflavors, int *secflavors)
	#else
	zfs_checkexp(vfs_t vfsp, struct sockaddr nam, int *extflagsp,
	struct ucred *credanonp, int numsecflavors, int **secflavors)
	#endif
	{
	zfsvfs_t *zfsvfs = vfsp->vfs_data;

	/*
	* If this is regular file system vfsp is the same as
	* zfsvfs->z_parent->z_vfs, but if it is snapshot,
	* zfsvfs->z_parent->z_vfs represents parent file system
	* which we have to use here, because only this file system
	* has mnt_export configured.
	*/
	return (vfs_stdcheckexp(zfsvfs->z_parent->z_vfs, nam, extflagsp,
	credanonp, numsecflavors, secflavors));
	}

	CTASSERT(SHORT_FID_LEN <= sizeof (struct fid));
	CTASSERT(LONG_FID_LEN <= sizeof (struct fid));

	static int
	zfs_fhtovp(vfs_t vfsp, fid_t fidp, int flags, vnode_t **vpp)
	{
	struct componentname cn;
	zfsvfs_t *zfsvfs = vfsp->vfs_data;
	znode_t *zp;
	vnode_t *dvp;
	uint64_t object = 0;
	uint64_t fid_gen = 0;
	uint64_t gen_mask;
	uint64_t zp_gen;
	int i, err;

	*vpp = NULL;

	ZFS_ENTER(zfsvfs);

	/*
	* On FreeBSD we can get snapshot's mount point or its parent file
	* system mount point depending if snapshot is already mounted or not.
	*/
	if (zfsvfs->z_parent == zfsvfs && fidp->fid_len == LONG_FID_LEN) {
	zfid_long_t zlfid = (zfid_long_t )fidp;
	uint64_t objsetid = 0;
	uint64_t setgen = 0;

	for (i = 0; i < sizeof (zlfid->zf_setid); i++)
	objsetid \|= ((uint64_t)zlfid->zf_setid[i]) << (8 * i);

	for (i = 0; i < sizeof (zlfid->zf_setgen); i++)
	setgen \|= ((uint64_t)zlfid->zf_setgen[i]) << (8 * i);

	ZFS_EXIT(zfsvfs);

	err = zfsctl_lookup_objset(vfsp, objsetid, &zfsvfs);
	if (err)
	return (SET_ERROR(EINVAL));
	ZFS_ENTER(zfsvfs);
	}

	if (fidp->fid_len == SHORT_FID_LEN \|\| fidp->fid_len == LONG_FID_LEN) {
	zfid_short_t zfid = (zfid_short_t )fidp;

	for (i = 0; i < sizeof (zfid->zf_object); i++)
	object \|= ((uint64_t)zfid->zf_object[i]) << (8 * i);

	for (i = 0; i < sizeof (zfid->zf_gen); i++)
	fid_gen \|= ((uint64_t)zfid->zf_gen[i]) << (8 * i);
	} else {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	/*
	* A zero fid_gen means we are in .zfs or the .zfs/snapshot
	* directory tree. If the object == zfsvfs->z_shares_dir, then
	* we are in the .zfs/shares directory tree.
	*/
	if ((fid_gen == 0 &&
	(object == ZFSCTL_INO_ROOT \|\| object == ZFSCTL_INO_SNAPDIR)) \|\|
	(zfsvfs->z_shares_dir != 0 && object == zfsvfs->z_shares_dir)) {
	ZFS_EXIT(zfsvfs);
	VERIFY0(zfsctl_root(zfsvfs, LK_SHARED, &dvp));
	if (object == ZFSCTL_INO_SNAPDIR) {
	cn.cn_nameptr = "snapshot";
	cn.cn_namelen = strlen(cn.cn_nameptr);
	cn.cn_nameiop = LOOKUP;
	cn.cn_flags = ISLASTCN \| LOCKLEAF;
	cn.cn_lkflags = flags;
	VERIFY0(VOP_LOOKUP(dvp, vpp, &cn));
	vput(dvp);
	} else if (object == zfsvfs->z_shares_dir) {
	/*
	* XXX This branch must not be taken,
	* if it is, then the lookup below will
	* explode.
	*/
	cn.cn_nameptr = "shares";
	cn.cn_namelen = strlen(cn.cn_nameptr);
	cn.cn_nameiop = LOOKUP;
	cn.cn_flags = ISLASTCN;
	cn.cn_lkflags = flags;
	VERIFY0(VOP_LOOKUP(dvp, vpp, &cn));
	vput(dvp);
	} else {
	*vpp = dvp;
	}
	return (err);
	}

	gen_mask = -1ULL >> (64 - 8 * i);

	dprintf("getting %llu [%u mask %llx]\n", object, fid_gen, gen_mask);
	if ((err = zfs_zget(zfsvfs, object, &zp))) {
	ZFS_EXIT(zfsvfs);
	return (err);
	}
	(void) sa_lookup(zp->z_sa_hdl, SA_ZPL_GEN(zfsvfs), &zp_gen,
	sizeof (uint64_t));
	zp_gen = zp_gen & gen_mask;
	if (zp_gen == 0)
	zp_gen = 1;
	if (zp->z_unlinked \|\| zp_gen != fid_gen) {
	dprintf("znode gen (%u) != fid gen (%u)\n", zp_gen, fid_gen);
	vrele(ZTOV(zp));
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	*vpp = ZTOV(zp);
	ZFS_EXIT(zfsvfs);
	err = vn_lock(*vpp, flags);
	if (err == 0)
	vnode_create_vobject(*vpp, zp->z_size, curthread);
	else
	*vpp = NULL;
	return (err);
	}

	/*
	* Block out VOPs and close zfsvfs_t::z_os
	*
	* Note, if successful, then we return with the 'z_teardown_lock' and
	* 'z_teardown_inactive_lock' write held. We leave ownership of the underlying
	* dataset and objset intact so that they can be atomically handed off during
	* a subsequent rollback or recv operation and the resume thereafter.
	*/
	int
	zfs_suspend_fs(zfsvfs_t *zfsvfs)
	{
	int error;

	if ((error = zfsvfs_teardown(zfsvfs, B_FALSE)) != 0)
	return (error);

	return (0);
	}

	/*
	* Rebuild SA and release VOPs. Note that ownership of the underlying dataset
	* is an invariant across any of the operations that can be performed while the
	* filesystem was suspended. Whether it succeeded or failed, the preconditions
	* are the same: the relevant objset and associated dataset are owned by
	* zfsvfs, held, and long held on entry.
	*/
	int
	zfs_resume_fs(zfsvfs_t zfsvfs, dsl_dataset_t ds)
	{
	int err;
	znode_t *zp;

	ASSERT(RRM_WRITE_HELD(&zfsvfs->z_teardown_lock));
	ASSERT(ZFS_TEARDOWN_INACTIVE_WLOCKED(zfsvfs));

	/*
	* We already own this, so just update the objset_t, as the one we
	* had before may have been evicted.
	*/
	objset_t *os;
	VERIFY3P(ds->ds_owner, ==, zfsvfs);
	VERIFY(dsl_dataset_long_held(ds));
	dsl_pool_t *dp = spa_get_dsl(dsl_dataset_get_spa(ds));
	dsl_pool_config_enter(dp, FTAG);
	VERIFY0(dmu_objset_from_ds(ds, &os));
	dsl_pool_config_exit(dp, FTAG);

	err = zfsvfs_init(zfsvfs, os);
	if (err != 0)
	goto bail;

	ds->ds_dir->dd_activity_cancelled = B_FALSE;
	VERIFY(zfsvfs_setup(zfsvfs, B_FALSE) == 0);

	zfs_set_fuid_feature(zfsvfs);

	/*
	* Attempt to re-establish all the active znodes with
	* their dbufs. If a zfs_rezget() fails, then we'll let
	* any potential callers discover that via ZFS_ENTER_VERIFY_VP
	* when they try to use their znode.
	*/
	mutex_enter(&zfsvfs->z_znodes_lock);
	for (zp = list_head(&zfsvfs->z_all_znodes); zp;
	zp = list_next(&zfsvfs->z_all_znodes, zp)) {
	(void) zfs_rezget(zp);
	}
	mutex_exit(&zfsvfs->z_znodes_lock);

	bail:
	/* release the VOPs */
	ZFS_WUNLOCK_TEARDOWN_INACTIVE(zfsvfs);
	rrm_exit(&zfsvfs->z_teardown_lock, FTAG);

	if (err) {
	/*
	* Since we couldn't setup the sa framework, try to force
	* unmount this file system.
	*/
	if (vn_vfswlock(zfsvfs->z_vfs->vfs_vnodecovered) == 0) {
	vfs_ref(zfsvfs->z_vfs);
	(void) dounmount(zfsvfs->z_vfs, MS_FORCE, curthread);
	}
	}
	return (err);
	}

	static void
	zfs_freevfs(vfs_t *vfsp)
	{
	zfsvfs_t *zfsvfs = vfsp->vfs_data;

	zfsvfs_free(zfsvfs);

	atomic_dec_32(&zfs_active_fs_count);
	}

	#ifdef __i386__
	static int desiredvnodes_backup;
	#include <sys/vmmeter.h>


	#include <vm/vm_page.h>
	#include <vm/vm_object.h>
	#include <vm/vm_kern.h>
	#include <vm/vm_map.h>
	#endif

	static void
	zfs_vnodes_adjust(void)
	{
	#ifdef __i386__
	int newdesiredvnodes;

	desiredvnodes_backup = desiredvnodes;

	/*
	* We calculate newdesiredvnodes the same way it is done in
	* vntblinit(). If it is equal to desiredvnodes, it means that
	* it wasn't tuned by the administrator and we can tune it down.
	*/
	newdesiredvnodes = min(maxproc + vm_cnt.v_page_count / 4, 2 *
	vm_kmem_size / (5 * (sizeof (struct vm_object) +
	sizeof (struct vnode))));
	if (newdesiredvnodes == desiredvnodes)
	desiredvnodes = (3 * newdesiredvnodes) / 4;
	#endif
	}

	static void
	zfs_vnodes_adjust_back(void)
	{

	#ifdef __i386__
	desiredvnodes = desiredvnodes_backup;
	#endif
	}

	void
	zfs_init(void)
	{

	printf("ZFS filesystem version: " ZPL_VERSION_STRING "\n");

	/*
	* Initialize .zfs directory structures
	*/
	zfsctl_init();

	/*
	* Initialize znode cache, vnode ops, etc...
	*/
	zfs_znode_init();

	/*
	* Reduce number of vnodes. Originally number of vnodes is calculated
	* with UFS inode in mind. We reduce it here, because it's too big for
	* ZFS/i386.
	*/
	zfs_vnodes_adjust();

	dmu_objset_register_type(DMU_OST_ZFS, zpl_get_file_info);

	zfsvfs_taskq = taskq_create("zfsvfs", 1, minclsyspri, 0, 0, 0);
	}

	void
	zfs_fini(void)
	{
	taskq_destroy(zfsvfs_taskq);
	zfsctl_fini();
	zfs_znode_fini();
	zfs_vnodes_adjust_back();
	}

	int
	zfs_busy(void)
	{
	return (zfs_active_fs_count != 0);
	}

	/*
	* Release VOPs and unmount a suspended filesystem.
	*/
	int
	zfs_end_fs(zfsvfs_t zfsvfs, dsl_dataset_t ds)
	{
	ASSERT(RRM_WRITE_HELD(&zfsvfs->z_teardown_lock));
	ASSERT(ZFS_TEARDOWN_INACTIVE_WLOCKED(zfsvfs));

	/*
	* We already own this, so just hold and rele it to update the
	* objset_t, as the one we had before may have been evicted.
	*/
	objset_t *os;
	VERIFY3P(ds->ds_owner, ==, zfsvfs);
	VERIFY(dsl_dataset_long_held(ds));
	dsl_pool_t *dp = spa_get_dsl(dsl_dataset_get_spa(ds));
	dsl_pool_config_enter(dp, FTAG);
	VERIFY0(dmu_objset_from_ds(ds, &os));
	dsl_pool_config_exit(dp, FTAG);
	zfsvfs->z_os = os;

	/* release the VOPs */
	ZFS_WUNLOCK_TEARDOWN_INACTIVE(zfsvfs);
	rrm_exit(&zfsvfs->z_teardown_lock, FTAG);

	/*
	* Try to force unmount this file system.
	*/
	(void) zfs_umount(zfsvfs->z_vfs, 0);
	zfsvfs->z_unmounted = B_TRUE;
	return (0);
	}

	int
	zfs_set_version(zfsvfs_t *zfsvfs, uint64_t newvers)
	{
	int error;
	objset_t *os = zfsvfs->z_os;
	dmu_tx_t *tx;

	if (newvers < ZPL_VERSION_INITIAL \|\| newvers > ZPL_VERSION)
	return (SET_ERROR(EINVAL));

	if (newvers < zfsvfs->z_version)
	return (SET_ERROR(EINVAL));

	if (zfs_spa_version_map(newvers) >
	spa_version(dmu_objset_spa(zfsvfs->z_os)))
	return (SET_ERROR(ENOTSUP));

	tx = dmu_tx_create(os);
	dmu_tx_hold_zap(tx, MASTER_NODE_OBJ, B_FALSE, ZPL_VERSION_STR);
	if (newvers >= ZPL_VERSION_SA && !zfsvfs->z_use_sa) {
	dmu_tx_hold_zap(tx, MASTER_NODE_OBJ, B_TRUE,
	ZFS_SA_ATTRS);
	dmu_tx_hold_zap(tx, DMU_NEW_OBJECT, FALSE, NULL);
	}
	error = dmu_tx_assign(tx, TXG_WAIT);
	if (error) {
	dmu_tx_abort(tx);
	return (error);
	}

	error = zap_update(os, MASTER_NODE_OBJ, ZPL_VERSION_STR,
	8, 1, &newvers, tx);

	if (error) {
	dmu_tx_commit(tx);
	return (error);
	}

	if (newvers >= ZPL_VERSION_SA && !zfsvfs->z_use_sa) {
	uint64_t sa_obj;

	ASSERT3U(spa_version(dmu_objset_spa(zfsvfs->z_os)), >=,
	SPA_VERSION_SA);
	sa_obj = zap_create(os, DMU_OT_SA_MASTER_NODE,
	DMU_OT_NONE, 0, tx);

	error = zap_add(os, MASTER_NODE_OBJ,
	ZFS_SA_ATTRS, 8, 1, &sa_obj, tx);
	ASSERT0(error);

	VERIFY(0 == sa_set_sa_object(os, sa_obj));
	sa_register_update_callback(os, zfs_sa_upgrade);
	}

	spa_history_log_internal_ds(dmu_objset_ds(os), "upgrade", tx,
	"from %ju to %ju", (uintmax_t)zfsvfs->z_version,
	(uintmax_t)newvers);
	dmu_tx_commit(tx);

	zfsvfs->z_version = newvers;
	os->os_version = newvers;

	zfs_set_fuid_feature(zfsvfs);

	return (0);
	}

	/*
	* Read a property stored within the master node.
	*/
	int
	zfs_get_zplprop(objset_t os, zfs_prop_t prop, uint64_t value)
	{
	uint64_t *cached_copy = NULL;

	/*
	* Figure out where in the objset_t the cached copy would live, if it
	* is available for the requested property.
	*/
	if (os != NULL) {
	switch (prop) {
	case ZFS_PROP_VERSION:
	cached_copy = &os->os_version;
	break;
	case ZFS_PROP_NORMALIZE:
	cached_copy = &os->os_normalization;
	break;
	case ZFS_PROP_UTF8ONLY:
	cached_copy = &os->os_utf8only;
	break;
	case ZFS_PROP_CASE:
	cached_copy = &os->os_casesensitivity;
	break;
	default:
	break;
	}
	}
	if (cached_copy != NULL && *cached_copy != OBJSET_PROP_UNINITIALIZED) {
	value = cached_copy;
	return (0);
	}

	/*
	* If the property wasn't cached, look up the file system's value for
	* the property. For the version property, we look up a slightly
	* different string.
	*/
	const char *pname;
	int error = ENOENT;
	if (prop == ZFS_PROP_VERSION) {
	pname = ZPL_VERSION_STR;
	} else {
	pname = zfs_prop_to_name(prop);
	}

	if (os != NULL) {
	ASSERT3U(os->os_phys->os_type, ==, DMU_OST_ZFS);
	error = zap_lookup(os, MASTER_NODE_OBJ, pname, 8, 1, value);
	}

	if (error == ENOENT) {
	/* No value set, use the default value */
	switch (prop) {
	case ZFS_PROP_VERSION:
	*value = ZPL_VERSION;
	break;
	case ZFS_PROP_NORMALIZE:
	case ZFS_PROP_UTF8ONLY:
	*value = 0;
	break;
	case ZFS_PROP_CASE:
	*value = ZFS_CASE_SENSITIVE;
	+ break;
	+ case ZFS_PROP_ACLTYPE:
	+ *value = ZFS_ACLTYPE_NFSV4;
	break;
	default:
	return (error);
	}
	error = 0;
	}

	/*
	* If one of the methods for getting the property value above worked,
	* copy it into the objset_t's cache.
	*/
	if (error == 0 && cached_copy != NULL) {
	cached_copy = value;
	}

	return (error);
	}

	/*
	* Return true if the corresponding vfs's unmounted flag is set.
	* Otherwise return false.
	* If this function returns true we know VFS unmount has been initiated.
	*/
	boolean_t
	zfs_get_vfs_flag_unmounted(objset_t *os)
	{
	zfsvfs_t *zfvp;
	boolean_t unmounted = B_FALSE;

	ASSERT(dmu_objset_type(os) == DMU_OST_ZFS);

	mutex_enter(&os->os_user_ptr_lock);
	zfvp = dmu_objset_get_user(os);
	if (zfvp != NULL && zfvp->z_vfs != NULL &&
	(zfvp->z_vfs->mnt_kern_flag & MNTK_UNMOUNT))
	unmounted = B_TRUE;
	mutex_exit(&os->os_user_ptr_lock);

	return (unmounted);
	}

	#ifdef _KERNEL
	void
	zfsvfs_update_fromname(const char oldname, const char newname)
	{
	char tmpbuf[MAXPATHLEN];
	struct mount *mp;
	char *fromname;
	size_t oldlen;

	oldlen = strlen(oldname);

	mtx_lock(&mountlist_mtx);
	TAILQ_FOREACH(mp, &mountlist, mnt_list) {
	fromname = mp->mnt_stat.f_mntfromname;
	if (strcmp(fromname, oldname) == 0) {
	(void) strlcpy(fromname, newname,
	sizeof (mp->mnt_stat.f_mntfromname));
	continue;
	}
	if (strncmp(fromname, oldname, oldlen) == 0 &&
	(fromname[oldlen] == '/' \|\| fromname[oldlen] == '@')) {
	(void) snprintf(tmpbuf, sizeof (tmpbuf), "%s%s",
	newname, fromname + oldlen);
	(void) strlcpy(fromname, tmpbuf,
	sizeof (mp->mnt_stat.f_mntfromname));
	continue;
	}
	}
	mtx_unlock(&mountlist_mtx);
	}
	#endif
	Index: vendor-sys/openzfs/dist/module/os/freebsd/zfs/zfs_vnops.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/freebsd/zfs/zfs_vnops.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/freebsd/zfs/zfs_vnops.c (revision 366774)
	@@ -1,6631 +1,6653 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/

	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2012, 2015 by Delphix. All rights reserved.
	* Copyright (c) 2014 Integros [integros.com]
	* Copyright 2017 Nexenta Systems, Inc.
	*/

	/* Portions Copyright 2007 Jeremy Teo */
	/* Portions Copyright 2010 Robert Milkowski */

	#include <sys/types.h>
	#include <sys/param.h>
	#include <sys/time.h>
	#include <sys/systm.h>
	#include <sys/sysmacros.h>
	#include <sys/resource.h>
	#include <sys/vfs.h>
	#include <sys/endian.h>
	#include <sys/vm.h>
	#include <sys/vnode.h>
	#if __FreeBSD_version >= 1300102
	#include <sys/smr.h>
	#endif
	#include <sys/dirent.h>
	#include <sys/file.h>
	#include <sys/stat.h>
	#include <sys/kmem.h>
	#include <sys/taskq.h>
	#include <sys/uio.h>
	#include <sys/atomic.h>
	#include <sys/namei.h>
	#include <sys/mman.h>
	#include <sys/cmn_err.h>
	#include <sys/kdb.h>
	#include <sys/sysproto.h>
	#include <sys/errno.h>
	#include <sys/unistd.h>
	#include <sys/zfs_dir.h>
	#include <sys/zfs_ioctl.h>
	#include <sys/fs/zfs.h>
	#include <sys/dmu.h>
	#include <sys/dmu_objset.h>
	#include <sys/spa.h>
	#include <sys/txg.h>
	#include <sys/dbuf.h>
	#include <sys/zap.h>
	#include <sys/sa.h>
	#include <sys/policy.h>
	#include <sys/sunddi.h>
	#include <sys/filio.h>
	#include <sys/sid.h>
	#include <sys/zfs_ctldir.h>
	#include <sys/zfs_fuid.h>
	#include <sys/zfs_quota.h>
	#include <sys/zfs_sa.h>
	#include <sys/zfs_rlock.h>
	#include <sys/extdirent.h>
	#include <sys/bio.h>
	#include <sys/buf.h>
	#include <sys/sched.h>
	#include <sys/acl.h>
	#include <sys/vmmeter.h>
	#include <vm/vm_param.h>
	#include <sys/zil.h>
	#include <sys/zfs_vnops.h>

	#include <vm/vm_object.h>

	#include <sys/extattr.h>
	#include <sys/priv.h>

	#ifndef VN_OPEN_INVFS
	#define VN_OPEN_INVFS 0x0
	#endif

	VFS_SMR_DECLARE;

	#if __FreeBSD_version >= 1300047
	#define vm_page_wire_lock(pp)
	#define vm_page_wire_unlock(pp)
	#else
	#define vm_page_wire_lock(pp) vm_page_lock(pp)
	#define vm_page_wire_unlock(pp) vm_page_unlock(pp)
	#endif

	-static int
	-zfs_u8_validate(const char u8str, size_t n, char list, int flag, int errnum)
	-{
	-
	- return (u8_validate(__DECONST(char *, u8str), n, list, flag, errnum));
	-}
	-#define u8_validate zfs_u8_validate
	-
	#ifdef DEBUG_VFS_LOCKS
	#define VNCHECKREF(vp) \
	VNASSERT((vp)->v_holdcnt > 0 && (vp)->v_usecount > 0, vp, \
	("%s: wrong ref counts", __func__));
	#else
	#define VNCHECKREF(vp)
	#endif

	/*
	* Programming rules.
	*
	* Each vnode op performs some logical unit of work. To do this, the ZPL must
	* properly lock its in-core state, create a DMU transaction, do the work,
	* record this work in the intent log (ZIL), commit the DMU transaction,
	* and wait for the intent log to commit if it is a synchronous operation.
	* Moreover, the vnode ops must work in both normal and log replay context.
	* The ordering of events is important to avoid deadlocks and references
	* to freed memory. The example below illustrates the following Big Rules:
	*
	* (1) A check must be made in each zfs thread for a mounted file system.
	* This is done avoiding races using ZFS_ENTER(zfsvfs).
	* A ZFS_EXIT(zfsvfs) is needed before all returns. Any znodes
	* must be checked with ZFS_VERIFY_ZP(zp). Both of these macros
	* can return EIO from the calling function.
	*
	* (2) VN_RELE() should always be the last thing except for zil_commit()
	* (if necessary) and ZFS_EXIT(). This is for 3 reasons:
	* First, if it's the last reference, the vnode/znode
	* can be freed, so the zp may point to freed memory. Second, the last
	* reference will call zfs_zinactive(), which may induce a lot of work --
	* pushing cached pages (which acquires range locks) and syncing out
	* cached atime changes. Third, zfs_zinactive() may require a new tx,
	* which could deadlock the system if you were already holding one.
	* If you must call VN_RELE() within a tx then use VN_RELE_ASYNC().
	*
	* (3) All range locks must be grabbed before calling dmu_tx_assign(),
	* as they can span dmu_tx_assign() calls.
	*
	* (4) If ZPL locks are held, pass TXG_NOWAIT as the second argument to
	* dmu_tx_assign(). This is critical because we don't want to block
	* while holding locks.
	*
	* If no ZPL locks are held (aside from ZFS_ENTER()), use TXG_WAIT. This
	* reduces lock contention and CPU usage when we must wait (note that if
	* throughput is constrained by the storage, nearly every transaction
	* must wait).
	*
	* Note, in particular, that if a lock is sometimes acquired before
	* the tx assigns, and sometimes after (e.g. z_lock), then failing
	* to use a non-blocking assign can deadlock the system. The scenario:
	*
	* Thread A has grabbed a lock before calling dmu_tx_assign().
	* Thread B is in an already-assigned tx, and blocks for this lock.
	* Thread A calls dmu_tx_assign(TXG_WAIT) and blocks in txg_wait_open()
	* forever, because the previous txg can't quiesce until B's tx commits.
	*
	* If dmu_tx_assign() returns ERESTART and zfsvfs->z_assign is TXG_NOWAIT,
	* then drop all locks, call dmu_tx_wait(), and try again. On subsequent
	* calls to dmu_tx_assign(), pass TXG_NOTHROTTLE in addition to TXG_NOWAIT,
	* to indicate that this operation has already called dmu_tx_wait().
	* This will ensure that we don't retry forever, waiting a short bit
	* each time.
	*
	* (5) If the operation succeeded, generate the intent log entry for it
	* before dropping locks. This ensures that the ordering of events
	* in the intent log matches the order in which they actually occurred.
	* During ZIL replay the zfs_log_* functions will update the sequence
	* number to indicate the zil transaction has replayed.
	*
	* (6) At the end of each vnode op, the DMU tx must always commit,
	* regardless of whether there were any errors.
	*
	* (7) After dropping all locks, invoke zil_commit(zilog, foid)
	* to ensure that synchronous semantics are provided when necessary.
	*
	* In general, this is how things should be ordered in each vnode op:
	*
	* ZFS_ENTER(zfsvfs); // exit if unmounted
	* top:
	* zfs_dirent_lookup(&dl, ...) // lock directory entry (may VN_HOLD())
	* rw_enter(...); // grab any other locks you need
	* tx = dmu_tx_create(...); // get DMU tx
	* dmu_tx_hold_*(); // hold each object you might modify
	* error = dmu_tx_assign(tx, (waited ? TXG_NOTHROTTLE : 0) \| TXG_NOWAIT);
	* if (error) {
	* rw_exit(...); // drop locks
	* zfs_dirent_unlock(dl); // unlock directory entry
	* VN_RELE(...); // release held vnodes
	* if (error == ERESTART) {
	* waited = B_TRUE;
	* dmu_tx_wait(tx);
	* dmu_tx_abort(tx);
	* goto top;
	* }
	* dmu_tx_abort(tx); // abort DMU tx
	* ZFS_EXIT(zfsvfs); // finished in zfs
	* return (error); // really out of space
	* }
	* error = do_real_work(); // do whatever this VOP does
	* if (error == 0)
	* zfs_log_*(...); // on success, make ZIL entry
	* dmu_tx_commit(tx); // commit DMU tx -- error or not
	* rw_exit(...); // drop locks
	* zfs_dirent_unlock(dl); // unlock directory entry
	* VN_RELE(...); // release held vnodes
	* zil_commit(zilog, foid); // synchronous when necessary
	* ZFS_EXIT(zfsvfs); // finished in zfs
	* return (error); // done, report error
	*/

	/* ARGSUSED */
	static int
	zfs_open(vnode_t *vpp, int flag, cred_t cr)
	{
	znode_t zp = VTOZ(vpp);
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	if ((flag & FWRITE) && (zp->z_pflags & ZFS_APPENDONLY) &&
	((flag & FAPPEND) == 0)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EPERM));
	}

	if (!zfs_has_ctldir(zp) && zp->z_zfsvfs->z_vscan &&
	ZTOV(zp)->v_type == VREG &&
	!(zp->z_pflags & ZFS_AV_QUARANTINED) && zp->z_size > 0) {
	if (fs_vscan(*vpp, cr, 0) != 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EACCES));
	}
	}

	/* Keep a count of the synchronous opens in the znode */
	if (flag & (FSYNC \| FDSYNC))
	atomic_inc_32(&zp->z_sync_cnt);

	ZFS_EXIT(zfsvfs);
	return (0);
	}

	/* ARGSUSED */
	static int
	zfs_close(vnode_t vp, int flag, int count, offset_t offset, cred_t cr)
	{
	znode_t *zp = VTOZ(vp);
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	/* Decrement the synchronous opens in the znode */
	if ((flag & (FSYNC \| FDSYNC)) && (count == 1))
	atomic_dec_32(&zp->z_sync_cnt);

	if (!zfs_has_ctldir(zp) && zp->z_zfsvfs->z_vscan &&
	ZTOV(zp)->v_type == VREG &&
	!(zp->z_pflags & ZFS_AV_QUARANTINED) && zp->z_size > 0)
	VERIFY(fs_vscan(vp, cr, 1) == 0);

	ZFS_EXIT(zfsvfs);
	return (0);
	}

	/*
	* Lseek support for finding holes (cmd == _FIO_SEEK_HOLE) and
	* data (cmd == _FIO_SEEK_DATA). "off" is an in/out parameter.
	*/
	static int
	zfs_holey(vnode_t vp, ulong_t cmd, offset_t off)
	{
	znode_t *zp = VTOZ(vp);
	uint64_t noff = (uint64_t)off; / new offset */
	uint64_t file_sz;
	int error;
	boolean_t hole;

	file_sz = zp->z_size;
	if (noff >= file_sz) {
	return (SET_ERROR(ENXIO));
	}

	if (cmd == _FIO_SEEK_HOLE)
	hole = B_TRUE;
	else
	hole = B_FALSE;

	error = dmu_offset_next(zp->z_zfsvfs->z_os, zp->z_id, hole, &noff);

	if (error == ESRCH)
	return (SET_ERROR(ENXIO));

	/* file was dirty, so fall back to using generic logic */
	if (error == EBUSY) {
	if (hole)
	*off = file_sz;

	return (0);
	}

	/*
	* We could find a hole that begins after the logical end-of-file,
	* because dmu_offset_next() only works on whole blocks. If the
	* EOF falls mid-block, then indicate that the "virtual hole"
	* at the end of the file begins at the logical EOF, rather than
	* at the end of the last block.
	*/
	if (noff > file_sz) {
	ASSERT(hole);
	noff = file_sz;
	}

	if (noff < *off)
	return (error);
	*off = noff;
	return (error);
	}

	/* ARGSUSED */
	static int
	zfs_ioctl(vnode_t vp, ulong_t com, intptr_t data, int flag, cred_t cred,
	int *rvalp)
	{
	offset_t off;
	int error;
	zfsvfs_t *zfsvfs;
	znode_t *zp;

	switch (com) {
	case _FIOFFS:
	{
	return (0);

	/*
	* The following two ioctls are used by bfu. Faking out,
	* necessary to avoid bfu errors.
	*/
	}
	case _FIOGDIO:
	case _FIOSDIO:
	{
	return (0);
	}

	case _FIO_SEEK_DATA:
	case _FIO_SEEK_HOLE:
	{
	off = (offset_t )data;
	zp = VTOZ(vp);
	zfsvfs = zp->z_zfsvfs;
	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	/* offset parameter is in/out */
	error = zfs_holey(vp, com, &off);
	ZFS_EXIT(zfsvfs);
	if (error)
	return (error);
	(offset_t )data = off;
	return (0);
	}
	}
	return (SET_ERROR(ENOTTY));
	}

	static vm_page_t
	page_busy(vnode_t *vp, int64_t start, int64_t off, int64_t nbytes)
	{
	vm_object_t obj;
	vm_page_t pp;
	int64_t end;

	/*
	* At present vm_page_clear_dirty extends the cleared range to DEV_BSIZE
	* aligned boundaries, if the range is not aligned. As a result a
	* DEV_BSIZE subrange with partially dirty data may get marked as clean.
	* It may happen that all DEV_BSIZE subranges are marked clean and thus
	* the whole page would be considered clean despite have some
	* dirty data.
	* For this reason we should shrink the range to DEV_BSIZE aligned
	* boundaries before calling vm_page_clear_dirty.
	*/
	end = rounddown2(off + nbytes, DEV_BSIZE);
	off = roundup2(off, DEV_BSIZE);
	nbytes = end - off;

	obj = vp->v_object;
	zfs_vmobject_assert_wlocked_12(obj);
	#if __FreeBSD_version < 1300050
	for (;;) {
	if ((pp = vm_page_lookup(obj, OFF_TO_IDX(start))) != NULL &&
	pp->valid) {
	if (vm_page_xbusied(pp)) {
	/*
	* Reference the page before unlocking and
	* sleeping so that the page daemon is less
	* likely to reclaim it.
	*/
	vm_page_reference(pp);
	vm_page_lock(pp);
	zfs_vmobject_wunlock(obj);
	vm_page_busy_sleep(pp, "zfsmwb", true);
	zfs_vmobject_wlock(obj);
	continue;
	}
	vm_page_sbusy(pp);
	} else if (pp != NULL) {
	ASSERT(!pp->valid);
	pp = NULL;
	}
	if (pp != NULL) {
	ASSERT3U(pp->valid, ==, VM_PAGE_BITS_ALL);
	vm_object_pip_add(obj, 1);
	pmap_remove_write(pp);
	if (nbytes != 0)
	vm_page_clear_dirty(pp, off, nbytes);
	}
	break;
	}
	#else
	vm_page_grab_valid_unlocked(&pp, obj, OFF_TO_IDX(start),
	VM_ALLOC_NOCREAT \| VM_ALLOC_SBUSY \| VM_ALLOC_NORMAL \|
	VM_ALLOC_IGN_SBUSY);
	if (pp != NULL) {
	ASSERT3U(pp->valid, ==, VM_PAGE_BITS_ALL);
	vm_object_pip_add(obj, 1);
	pmap_remove_write(pp);
	if (nbytes != 0)
	vm_page_clear_dirty(pp, off, nbytes);
	}
	#endif
	return (pp);
	}

	static void
	page_unbusy(vm_page_t pp)
	{

	vm_page_sunbusy(pp);
	#if __FreeBSD_version >= 1300041
	vm_object_pip_wakeup(pp->object);
	#else
	vm_object_pip_subtract(pp->object, 1);
	#endif
	}

	#if __FreeBSD_version > 1300051
	static vm_page_t
	page_hold(vnode_t *vp, int64_t start)
	{
	vm_object_t obj;
	vm_page_t m;

	obj = vp->v_object;
	vm_page_grab_valid_unlocked(&m, obj, OFF_TO_IDX(start),
	VM_ALLOC_NOCREAT \| VM_ALLOC_WIRED \| VM_ALLOC_IGN_SBUSY \|
	VM_ALLOC_NOBUSY);
	return (m);
	}
	#else
	static vm_page_t
	page_hold(vnode_t *vp, int64_t start)
	{
	vm_object_t obj;
	vm_page_t pp;

	obj = vp->v_object;
	zfs_vmobject_assert_wlocked(obj);

	for (;;) {
	if ((pp = vm_page_lookup(obj, OFF_TO_IDX(start))) != NULL &&
	pp->valid) {
	if (vm_page_xbusied(pp)) {
	/*
	* Reference the page before unlocking and
	* sleeping so that the page daemon is less
	* likely to reclaim it.
	*/
	vm_page_reference(pp);
	vm_page_lock(pp);
	zfs_vmobject_wunlock(obj);
	vm_page_busy_sleep(pp, "zfsmwb", true);
	zfs_vmobject_wlock(obj);
	continue;
	}

	ASSERT3U(pp->valid, ==, VM_PAGE_BITS_ALL);
	vm_page_wire_lock(pp);
	vm_page_hold(pp);
	vm_page_wire_unlock(pp);

	} else
	pp = NULL;
	break;
	}
	return (pp);
	}
	#endif

	static void
	page_unhold(vm_page_t pp)
	{

	vm_page_wire_lock(pp);
	#if __FreeBSD_version >= 1300035
	vm_page_unwire(pp, PQ_ACTIVE);
	#else
	vm_page_unhold(pp);
	#endif
	vm_page_wire_unlock(pp);
	}

	/*
	* When a file is memory mapped, we must keep the IO data synchronized
	* between the DMU cache and the memory mapped pages. What this means:
	*
	* On Write: If we find a memory mapped page, we write to both
	* the page and the dmu buffer.
	*/
	static void
	update_pages(vnode_t vp, int64_t start, int len, objset_t os, uint64_t oid,
	int segflg, dmu_tx_t *tx)
	{
	vm_object_t obj;
	struct sf_buf *sf;
	caddr_t va;
	int off;

	ASSERT(segflg != UIO_NOCOPY);
	ASSERT(vp->v_mount != NULL);
	obj = vp->v_object;
	ASSERT(obj != NULL);

	off = start & PAGEOFFSET;
	zfs_vmobject_wlock_12(obj);
	#if __FreeBSD_version >= 1300041
	vm_object_pip_add(obj, 1);
	#endif
	for (start &= PAGEMASK; len > 0; start += PAGESIZE) {
	vm_page_t pp;
	int nbytes = imin(PAGESIZE - off, len);

	if ((pp = page_busy(vp, start, off, nbytes)) != NULL) {
	zfs_vmobject_wunlock_12(obj);

	va = zfs_map_page(pp, &sf);
	(void) dmu_read(os, oid, start+off, nbytes,
	va+off, DMU_READ_PREFETCH);
	zfs_unmap_page(sf);

	zfs_vmobject_wlock_12(obj);
	page_unbusy(pp);
	}
	len -= nbytes;
	off = 0;
	}
	#if __FreeBSD_version >= 1300041
	vm_object_pip_wakeup(obj);
	#else
	vm_object_pip_wakeupn(obj, 0);
	#endif
	zfs_vmobject_wunlock_12(obj);
	}

	/*
	* Read with UIO_NOCOPY flag means that sendfile(2) requests
	* ZFS to populate a range of page cache pages with data.
	*
	* NOTE: this function could be optimized to pre-allocate
	* all pages in advance, drain exclusive busy on all of them,
	* map them into contiguous KVA region and populate them
	* in one single dmu_read() call.
	*/
	static int
	mappedread_sf(vnode_t vp, int nbytes, uio_t uio)
	{
	znode_t *zp = VTOZ(vp);
	objset_t *os = zp->z_zfsvfs->z_os;
	struct sf_buf *sf;
	vm_object_t obj;
	vm_page_t pp;
	int64_t start;
	caddr_t va;
	int len = nbytes;
	int error = 0;

	ASSERT(uio->uio_segflg == UIO_NOCOPY);
	ASSERT(vp->v_mount != NULL);
	obj = vp->v_object;
	ASSERT(obj != NULL);
	ASSERT((uio->uio_loffset & PAGEOFFSET) == 0);

	zfs_vmobject_wlock_12(obj);
	for (start = uio->uio_loffset; len > 0; start += PAGESIZE) {
	int bytes = MIN(PAGESIZE, len);

	pp = vm_page_grab_unlocked(obj, OFF_TO_IDX(start),
	VM_ALLOC_SBUSY \| VM_ALLOC_NORMAL \| VM_ALLOC_IGN_SBUSY);
	if (vm_page_none_valid(pp)) {
	zfs_vmobject_wunlock_12(obj);
	va = zfs_map_page(pp, &sf);
	error = dmu_read(os, zp->z_id, start, bytes, va,
	DMU_READ_PREFETCH);
	if (bytes != PAGESIZE && error == 0)
	bzero(va + bytes, PAGESIZE - bytes);
	zfs_unmap_page(sf);
	zfs_vmobject_wlock_12(obj);
	#if __FreeBSD_version >= 1300081
	if (error == 0) {
	vm_page_valid(pp);
	vm_page_activate(pp);
	vm_page_do_sunbusy(pp);
	} else {
	zfs_vmobject_wlock(obj);
	if (!vm_page_wired(pp) && pp->valid == 0 &&
	vm_page_busy_tryupgrade(pp))
	vm_page_free(pp);
	else
	vm_page_sunbusy(pp);
	zfs_vmobject_wunlock(obj);
	}
	#else
	vm_page_do_sunbusy(pp);
	vm_page_lock(pp);
	if (error) {
	if (pp->wire_count == 0 && pp->valid == 0 &&
	!vm_page_busied(pp))
	vm_page_free(pp);
	} else {
	pp->valid = VM_PAGE_BITS_ALL;
	vm_page_activate(pp);
	}
	vm_page_unlock(pp);
	#endif
	} else {
	ASSERT3U(pp->valid, ==, VM_PAGE_BITS_ALL);
	vm_page_do_sunbusy(pp);
	}
	if (error)
	break;
	uio->uio_resid -= bytes;
	uio->uio_offset += bytes;
	len -= bytes;
	}
	zfs_vmobject_wunlock_12(obj);
	return (error);
	}

	/*
	* When a file is memory mapped, we must keep the IO data synchronized
	* between the DMU cache and the memory mapped pages. What this means:
	*
	* On Read: We "read" preferentially from memory mapped pages,
	* else we default from the dmu buffer.
	*
	* NOTE: We will always "break up" the IO into PAGESIZE uiomoves when
	* the file is memory mapped.
	*/
	static int
	mappedread(vnode_t vp, int nbytes, uio_t uio)
	{
	znode_t *zp = VTOZ(vp);
	vm_object_t obj;
	int64_t start;
	int len = nbytes;
	int off;
	int error = 0;

	ASSERT(vp->v_mount != NULL);
	obj = vp->v_object;
	ASSERT(obj != NULL);

	start = uio->uio_loffset;
	off = start & PAGEOFFSET;
	zfs_vmobject_wlock_12(obj);
	for (start &= PAGEMASK; len > 0; start += PAGESIZE) {
	vm_page_t pp;
	uint64_t bytes = MIN(PAGESIZE - off, len);

	if ((pp = page_hold(vp, start))) {
	struct sf_buf *sf;
	caddr_t va;

	zfs_vmobject_wunlock_12(obj);
	va = zfs_map_page(pp, &sf);
	error = vn_io_fault_uiomove(va + off, bytes, uio);
	zfs_unmap_page(sf);
	zfs_vmobject_wlock_12(obj);
	page_unhold(pp);
	} else {
	zfs_vmobject_wunlock_12(obj);
	error = dmu_read_uio_dbuf(sa_get_db(zp->z_sa_hdl),
	uio, bytes);
	zfs_vmobject_wlock_12(obj);
	}
	len -= bytes;
	off = 0;
	if (error)
	break;
	}
	zfs_vmobject_wunlock_12(obj);
	return (error);
	}

	offset_t zfs_read_chunk_size = 1024 * 1024; /* Tunable */

	/*
	* Read bytes from specified file into supplied buffer.
	*
	* IN: vp - vnode of file to be read from.
	* uio - structure supplying read location, range info,
	* and return buffer.
	* ioflag - SYNC flags; used to provide FRSYNC semantics.
	* cr - credentials of caller.
	* ct - caller context
	*
	* OUT: uio - updated offset and range, buffer filled.
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Side Effects:
	* vp - atime updated if byte count > 0
	*/
	/* ARGSUSED */
	static int
	zfs_read(vnode_t vp, uio_t uio, int ioflag, cred_t *cr)
	{
	znode_t *zp = VTOZ(vp);
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;
	ssize_t n, nbytes, start_resid;
	int error = 0;
	int64_t nread;
	zfs_locked_range_t *lr;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	/* We don't copy out anything useful for directories. */
	if (vp->v_type == VDIR) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EISDIR));
	}

	if (zp->z_pflags & ZFS_AV_QUARANTINED) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EACCES));
	}

	/*
	* Validate file offset
	*/
	if (uio->uio_loffset < (offset_t)0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	/*
	* Fasttrack empty reads
	*/
	if (uio->uio_resid == 0) {
	ZFS_EXIT(zfsvfs);
	return (0);
	}

	/*
	* If we're in FRSYNC mode, sync out this znode before reading it.
	*/
	if (zfsvfs->z_log &&
	(ioflag & FRSYNC \|\| zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS))
	zil_commit(zfsvfs->z_log, zp->z_id);

	/*
	* Lock the range against changes.
	*/
	lr = zfs_rangelock_enter(&zp->z_rangelock, uio->uio_loffset,
	uio->uio_resid, RL_READER);

	/*
	* If we are reading past end-of-file we can skip
	* to the end; but we might still need to set atime.
	*/
	if (uio->uio_loffset >= zp->z_size) {
	error = 0;
	goto out;
	}

	ASSERT(uio->uio_loffset < zp->z_size);
	n = MIN(uio->uio_resid, zp->z_size - uio->uio_loffset);
	start_resid = n;

	while (n > 0) {
	nbytes = MIN(n, zfs_read_chunk_size -
	P2PHASE(uio->uio_loffset, zfs_read_chunk_size));

	if (uio->uio_segflg == UIO_NOCOPY)
	error = mappedread_sf(vp, nbytes, uio);
	else if (vn_has_cached_data(vp)) {
	error = mappedread(vp, nbytes, uio);
	} else {
	error = dmu_read_uio_dbuf(sa_get_db(zp->z_sa_hdl),
	uio, nbytes);
	}
	if (error) {
	/* convert checksum errors into IO errors */
	if (error == ECKSUM)
	error = SET_ERROR(EIO);
	break;
	}

	n -= nbytes;
	}

	nread = start_resid - n;
	dataset_kstats_update_read_kstats(&zfsvfs->z_kstat, nread);

	out:
	zfs_rangelock_exit(lr);

	ZFS_ACCESSTIME_STAMP(zfsvfs, zp);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Write the bytes to a file.
	*
	* IN: vp - vnode of file to be written to.
	* uio - structure supplying write location, range info,
	* and data buffer.
	* ioflag - FAPPEND, FSYNC, and/or FDSYNC. FAPPEND is
	* set if in append mode.
	* cr - credentials of caller.
	* ct - caller context (NFS/CIFS fem monitor only)
	*
	* OUT: uio - updated offset and range.
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* vp - ctime\|mtime updated if byte count > 0
	*/

	/* ARGSUSED */
	static int
	zfs_write(vnode_t vp, uio_t uio, int ioflag, cred_t *cr)
	{
	znode_t *zp = VTOZ(vp);
	rlim64_t limit = MAXOFFSET_T;
	ssize_t start_resid = uio->uio_resid;
	ssize_t tx_bytes;
	uint64_t end_size;
	dmu_buf_impl_t *db;
	dmu_tx_t *tx;
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;
	zilog_t *zilog;
	offset_t woff;
	ssize_t n, nbytes;
	zfs_locked_range_t *lr;
	int max_blksz = zfsvfs->z_max_blksz;
	int error = 0;
	arc_buf_t *abuf;
	iovec_t *aiov = NULL;
	xuio_t *xuio = NULL;
	int i_iov = 0;
	int iovcnt __unused = uio->uio_iovcnt;
	iovec_t *iovp = uio->uio_iov;
	int write_eof;
	int count = 0;
	sa_bulk_attr_t bulk[4];
	uint64_t mtime[2], ctime[2];
	uint64_t uid, gid, projid;
	int64_t nwritten;

	/*
	* Fasttrack empty write
	*/
	n = start_resid;
	if (n == 0)
	return (0);

	if (limit == RLIM64_INFINITY \|\| limit > MAXOFFSET_T)
	limit = MAXOFFSET_T;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_MTIME(zfsvfs), NULL, &mtime, 16);
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_CTIME(zfsvfs), NULL, &ctime, 16);
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_SIZE(zfsvfs), NULL,
	&zp->z_size, 8);
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_FLAGS(zfsvfs), NULL,
	&zp->z_pflags, 8);

	/*
	* Callers might not be able to detect properly that we are read-only,
	* so check it explicitly here.
	*/
	if (zfs_is_readonly(zfsvfs)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EROFS));
	}

	/*
	* If immutable or not appending then return EPERM.
	* Intentionally allow ZFS_READONLY through here.
	* See zfs_zaccess_common()
	*/
	if ((zp->z_pflags & ZFS_IMMUTABLE) \|\|
	((zp->z_pflags & ZFS_APPENDONLY) && !(ioflag & FAPPEND) &&
	(uio->uio_loffset < zp->z_size))) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EPERM));
	}

	zilog = zfsvfs->z_log;

	/*
	* Validate file offset
	*/
	woff = ioflag & FAPPEND ? zp->z_size : uio->uio_loffset;
	if (woff < 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	/*
	* If in append mode, set the io offset pointer to eof.
	*/
	if (ioflag & FAPPEND) {
	/*
	* Obtain an appending range lock to guarantee file append
	* semantics. We reset the write offset once we have the lock.
	*/
	lr = zfs_rangelock_enter(&zp->z_rangelock, 0, n, RL_APPEND);
	woff = lr->lr_offset;
	if (lr->lr_length == UINT64_MAX) {
	/*
	* We overlocked the file because this write will cause
	* the file block size to increase.
	* Note that zp_size cannot change with this lock held.
	*/
	woff = zp->z_size;
	}
	uio->uio_loffset = woff;
	} else {
	/*
	* Note that if the file block size will change as a result of
	* this write, then this range lock will lock the entire file
	* so that we can re-write the block safely.
	*/
	lr = zfs_rangelock_enter(&zp->z_rangelock, woff, n, RL_WRITER);
	}

	if (vn_rlimit_fsize(vp, uio, uio->uio_td)) {
	zfs_rangelock_exit(lr);
	ZFS_EXIT(zfsvfs);
	return (EFBIG);
	}

	if (woff >= limit) {
	zfs_rangelock_exit(lr);
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EFBIG));
	}

	if ((woff + n) > limit \|\| woff > (limit - n))
	n = limit - woff;

	/* Will this write extend the file length? */
	write_eof = (woff + n > zp->z_size);

	end_size = MAX(zp->z_size, woff + n);

	uid = zp->z_uid;
	gid = zp->z_gid;
	projid = zp->z_projid;

	/*
	* Write the file in reasonable size chunks. Each chunk is written
	* in a separate transaction; this keeps the intent log records small
	* and allows us to do more fine-grained space accounting.
	*/
	while (n > 0) {
	woff = uio->uio_loffset;

	if (zfs_id_overblockquota(zfsvfs, DMU_USERUSED_OBJECT, uid) \|\|
	zfs_id_overblockquota(zfsvfs, DMU_GROUPUSED_OBJECT, gid) \|\|
	(projid != ZFS_DEFAULT_PROJID &&
	zfs_id_overblockquota(zfsvfs, DMU_PROJECTUSED_OBJECT,
	projid))) {
	error = SET_ERROR(EDQUOT);
	break;
	}

	abuf = NULL;
	if (xuio) {
	ASSERT(i_iov < iovcnt);
	aiov = &iovp[i_iov];
	abuf = dmu_xuio_arcbuf(xuio, i_iov);
	dmu_xuio_clear(xuio, i_iov);
	DTRACE_PROBE3(zfs_cp_write, int, i_iov,
	iovec_t , aiov, arc_buf_t , abuf);
	ASSERT((aiov->iov_base == abuf->b_data) \|\|
	((char )aiov->iov_base - (char )abuf->b_data +
	aiov->iov_len == arc_buf_size(abuf)));
	i_iov++;
	} else if (n >= max_blksz &&
	woff >= zp->z_size &&
	P2PHASE(woff, max_blksz) == 0 &&
	zp->z_blksz == max_blksz) {
	/*
	* This write covers a full block. "Borrow" a buffer
	* from the dmu so that we can fill it before we enter
	* a transaction. This avoids the possibility of
	* holding up the transaction if the data copy hangs
	* up on a pagefault (e.g., from an NFS server mapping).
	*/
	size_t cbytes;

	abuf = dmu_request_arcbuf(sa_get_db(zp->z_sa_hdl),
	max_blksz);
	ASSERT(abuf != NULL);
	ASSERT(arc_buf_size(abuf) == max_blksz);
	if ((error = uiocopy(abuf->b_data, max_blksz,
	UIO_WRITE, uio, &cbytes))) {
	dmu_return_arcbuf(abuf);
	break;
	}
	ASSERT(cbytes == max_blksz);
	}

	/*
	* Start a transaction.
	*/
	tx = dmu_tx_create(zfsvfs->z_os);
	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_FALSE);
	db = (dmu_buf_impl_t *)sa_get_db(zp->z_sa_hdl);
	DB_DNODE_ENTER(db);
	dmu_tx_hold_write_by_dnode(tx, DB_DNODE(db), woff,
	MIN(n, max_blksz));
	DB_DNODE_EXIT(db);
	zfs_sa_upgrade_txholds(tx, zp);
	error = dmu_tx_assign(tx, TXG_WAIT);
	if (error) {
	dmu_tx_abort(tx);
	if (abuf != NULL)
	dmu_return_arcbuf(abuf);
	break;
	}

	/*
	* If zfs_range_lock() over-locked we grow the blocksize
	* and then reduce the lock range. This will only happen
	* on the first iteration since zfs_range_reduce() will
	* shrink down r_len to the appropriate size.
	*/
	if (lr->lr_length == UINT64_MAX) {
	uint64_t new_blksz;

	if (zp->z_blksz > max_blksz) {
	/*
	* File's blocksize is already larger than the
	* "recordsize" property. Only let it grow to
	* the next power of 2.
	*/
	ASSERT(!ISP2(zp->z_blksz));
	new_blksz = MIN(end_size,
	1 << highbit64(zp->z_blksz));
	} else {
	new_blksz = MIN(end_size, max_blksz);
	}
	zfs_grow_blocksize(zp, new_blksz, tx);
	zfs_rangelock_reduce(lr, woff, n);
	}

	/*
	* XXX - should we really limit each write to z_max_blksz?
	* Perhaps we should use SPA_MAXBLOCKSIZE chunks?
	*/
	nbytes = MIN(n, max_blksz - P2PHASE(woff, max_blksz));

	if (woff + nbytes > zp->z_size)
	vnode_pager_setsize(vp, woff + nbytes);

	if (abuf == NULL) {
	tx_bytes = uio->uio_resid;
	error = dmu_write_uio_dbuf(sa_get_db(zp->z_sa_hdl),
	uio, nbytes, tx);
	tx_bytes -= uio->uio_resid;
	} else {
	tx_bytes = nbytes;
	ASSERT(xuio == NULL \|\| tx_bytes == aiov->iov_len);
	/*
	* If this is not a full block write, but we are
	* extending the file past EOF and this data starts
	* block-aligned, use assign_arcbuf(). Otherwise,
	* write via dmu_write().
	*/
	if (tx_bytes < max_blksz && (!write_eof \|\|
	aiov->iov_base != abuf->b_data)) {
	ASSERT(xuio);
	dmu_write(zfsvfs->z_os, zp->z_id, woff,
	aiov->iov_len, aiov->iov_base, tx);
	dmu_return_arcbuf(abuf);
	xuio_stat_wbuf_copied();
	} else {
	ASSERT(xuio \|\| tx_bytes == max_blksz);
	dmu_assign_arcbuf(sa_get_db(zp->z_sa_hdl), woff,
	abuf, tx);
	}
	ASSERT(tx_bytes <= uio->uio_resid);
	uioskip(uio, tx_bytes);
	}
	if (tx_bytes && vn_has_cached_data(vp)) {
	update_pages(vp, woff, tx_bytes, zfsvfs->z_os,
	zp->z_id, uio->uio_segflg, tx);
	}

	/*
	* If we made no progress, we're done. If we made even
	* partial progress, update the znode and ZIL accordingly.
	*/
	if (tx_bytes == 0) {
	(void) sa_update(zp->z_sa_hdl, SA_ZPL_SIZE(zfsvfs),
	(void *)&zp->z_size, sizeof (uint64_t), tx);
	dmu_tx_commit(tx);
	ASSERT(error != 0);
	break;
	}

	/*
	* Clear Set-UID/Set-GID bits on successful write if not
	* privileged and at least one of the execute bits is set.
	*
	* It would be nice to to this after all writes have
	* been done, but that would still expose the ISUID/ISGID
	* to another app after the partial write is committed.
	*
	* Note: we don't call zfs_fuid_map_id() here because
	* user 0 is not an ephemeral uid.
	*/
	mutex_enter(&zp->z_acl_lock);
	if ((zp->z_mode & (S_IXUSR \| (S_IXUSR >> 3) \|
	(S_IXUSR >> 6))) != 0 &&
	(zp->z_mode & (S_ISUID \| S_ISGID)) != 0 &&
	secpolicy_vnode_setid_retain(vp, cr,
	(zp->z_mode & S_ISUID) != 0 && zp->z_uid == 0) != 0) {
	uint64_t newmode;
	zp->z_mode &= ~(S_ISUID \| S_ISGID);
	newmode = zp->z_mode;
	(void) sa_update(zp->z_sa_hdl, SA_ZPL_MODE(zfsvfs),
	(void *)&newmode, sizeof (uint64_t), tx);
	}
	mutex_exit(&zp->z_acl_lock);

	zfs_tstamp_update_setup(zp, CONTENT_MODIFIED, mtime, ctime);

	/*
	* Update the file size (zp_size) if it has changed;
	* account for possible concurrent updates.
	*/
	while ((end_size = zp->z_size) < uio->uio_loffset) {
	(void) atomic_cas_64(&zp->z_size, end_size,
	uio->uio_loffset);
	ASSERT(error == 0 \|\| error == EFAULT);
	}
	/*
	* If we are replaying and eof is non zero then force
	* the file size to the specified eof. Note, there's no
	* concurrency during replay.
	*/
	if (zfsvfs->z_replay && zfsvfs->z_replay_eof != 0)
	zp->z_size = zfsvfs->z_replay_eof;

	if (error == 0)
	error = sa_bulk_update(zp->z_sa_hdl, bulk, count, tx);
	else
	(void) sa_bulk_update(zp->z_sa_hdl, bulk, count, tx);

	zfs_log_write(zilog, tx, TX_WRITE, zp, woff, tx_bytes,
	ioflag, NULL, NULL);
	dmu_tx_commit(tx);

	if (error != 0)
	break;
	ASSERT(tx_bytes == nbytes);
	n -= nbytes;

	}

	zfs_rangelock_exit(lr);

	/*
	* If we're in replay mode, or we made no progress, return error.
	* Otherwise, it's at least a partial write, so it's successful.
	*/
	if (zfsvfs->z_replay \|\| uio->uio_resid == start_resid) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* EFAULT means that at least one page of the source buffer was not
	* available. VFS will re-try remaining I/O upon this error.
	*/
	if (error == EFAULT) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	if (ioflag & (FSYNC \| FDSYNC) \|\|
	zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, zp->z_id);

	nwritten = start_resid - uio->uio_resid;
	dataset_kstats_update_write_kstats(&zfsvfs->z_kstat, nwritten);

	ZFS_EXIT(zfsvfs);
	return (0);
	}

	int
	zfs_write_simple(znode_t zp, const void data, size_t len,
	loff_t pos, size_t *presid)
	{
	int error = 0;
	ssize_t resid;

	error = vn_rdwr(UIO_WRITE, ZTOV(zp), __DECONST(void *, data), len, pos,
	UIO_SYSSPACE, IO_SYNC, kcred, NOCRED, &resid, curthread);

	if (error) {
	return (SET_ERROR(error));
	} else if (presid == NULL) {
	if (resid != 0) {
	error = SET_ERROR(EIO);
	}
	} else {
	*presid = resid;
	}
	return (error);
	}

	static void
	zfs_get_done(zgd_t *zgd, int error)
	{
	znode_t *zp = zgd->zgd_private;
	objset_t *os = zp->z_zfsvfs->z_os;

	if (zgd->zgd_db)
	dmu_buf_rele(zgd->zgd_db, zgd);

	zfs_rangelock_exit(zgd->zgd_lr);

	/*
	* Release the vnode asynchronously as we currently have the
	* txg stopped from syncing.
	*/
	VN_RELE_ASYNC(ZTOV(zp), dsl_pool_zrele_taskq(dmu_objset_pool(os)));

	kmem_free(zgd, sizeof (zgd_t));
	}

	#ifdef ZFS_DEBUG
	static int zil_fault_io = 0;
	#endif

	/*
	* Get data to generate a TX_WRITE intent log record.
	*/
	int
	zfs_get_data(void arg, lr_write_t lr, char buf, struct lwb lwb, zio_t *zio)
	{
	zfsvfs_t *zfsvfs = arg;
	objset_t *os = zfsvfs->z_os;
	znode_t *zp;
	uint64_t object = lr->lr_foid;
	uint64_t offset = lr->lr_offset;
	uint64_t size = lr->lr_length;
	dmu_buf_t *db;
	zgd_t *zgd;
	int error = 0;

	ASSERT3P(lwb, !=, NULL);
	ASSERT3P(zio, !=, NULL);
	ASSERT3U(size, !=, 0);

	/*
	* Nothing to do if the file has been removed
	*/
	if (zfs_zget(zfsvfs, object, &zp) != 0)
	return (SET_ERROR(ENOENT));
	if (zp->z_unlinked) {
	/*
	* Release the vnode asynchronously as we currently have the
	* txg stopped from syncing.
	*/
	VN_RELE_ASYNC(ZTOV(zp),
	dsl_pool_zrele_taskq(dmu_objset_pool(os)));
	return (SET_ERROR(ENOENT));
	}

	zgd = (zgd_t *)kmem_zalloc(sizeof (zgd_t), KM_SLEEP);
	zgd->zgd_lwb = lwb;
	zgd->zgd_private = zp;

	/*
	* Write records come in two flavors: immediate and indirect.
	* For small writes it's cheaper to store the data with the
	* log record (immediate); for large writes it's cheaper to
	* sync the data and get a pointer to it (indirect) so that
	* we don't have to write the data twice.
	*/
	if (buf != NULL) { /* immediate write */
	zgd->zgd_lr = zfs_rangelock_enter(&zp->z_rangelock, offset,
	size, RL_READER);
	/* test for truncation needs to be done while range locked */
	if (offset >= zp->z_size) {
	error = SET_ERROR(ENOENT);
	} else {
	error = dmu_read(os, object, offset, size, buf,
	DMU_READ_NO_PREFETCH);
	}
	ASSERT(error == 0 \|\| error == ENOENT);
	} else { /* indirect write */
	/*
	* Have to lock the whole block to ensure when it's
	* written out and its checksum is being calculated
	* that no one can change the data. We need to re-check
	* blocksize after we get the lock in case it's changed!
	*/
	for (;;) {
	uint64_t blkoff;
	size = zp->z_blksz;
	blkoff = ISP2(size) ? P2PHASE(offset, size) : offset;
	offset -= blkoff;
	zgd->zgd_lr = zfs_rangelock_enter(&zp->z_rangelock,
	offset, size, RL_READER);
	if (zp->z_blksz == size)
	break;
	offset += blkoff;
	zfs_rangelock_exit(zgd->zgd_lr);
	}
	/* test for truncation needs to be done while range locked */
	if (lr->lr_offset >= zp->z_size)
	error = SET_ERROR(ENOENT);
	#ifdef ZFS_DEBUG
	if (zil_fault_io) {
	error = SET_ERROR(EIO);
	zil_fault_io = 0;
	}
	#endif
	if (error == 0)
	error = dmu_buf_hold(os, object, offset, zgd, &db,
	DMU_READ_NO_PREFETCH);

	if (error == 0) {
	blkptr_t *bp = &lr->lr_blkptr;

	zgd->zgd_db = db;
	zgd->zgd_bp = bp;

	ASSERT(db->db_offset == offset);
	ASSERT(db->db_size == size);

	error = dmu_sync(zio, lr->lr_common.lrc_txg,
	zfs_get_done, zgd);
	ASSERT(error \|\| lr->lr_length <= size);

	/*
	* On success, we need to wait for the write I/O
	* initiated by dmu_sync() to complete before we can
	* release this dbuf. We will finish everything up
	* in the zfs_get_done() callback.
	*/
	if (error == 0)
	return (0);

	if (error == EALREADY) {
	lr->lr_common.lrc_txtype = TX_WRITE2;
	/*
	* TX_WRITE2 relies on the data previously
	* written by the TX_WRITE that caused
	* EALREADY. We zero out the BP because
	* it is the old, currently-on-disk BP,
	* so there's no need to zio_flush() its
	* vdevs (flushing would needlesly hurt
	* performance, and doesn't work on
	* indirect vdevs).
	*/
	zgd->zgd_bp = NULL;
	BP_ZERO(bp);
	error = 0;
	}
	}
	}

	zfs_get_done(zgd, error);

	return (error);
	}

	/ARGSUSED/
	static int
	zfs_access(vnode_t vp, int mode, int flag, cred_t cr,
	caller_context_t *ct)
	{
	znode_t *zp = VTOZ(vp);
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;
	int error;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	if (flag & V_ACE_MASK)
	error = zfs_zaccess(zp, mode, flag, B_FALSE, cr);
	else
	error = zfs_zaccess_rwx(zp, mode, flag, cr);

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	static int
	zfs_dd_callback(struct mount mp, void arg, int lkflags, struct vnode **vpp)
	{
	int error;

	*vpp = arg;
	error = vn_lock(*vpp, lkflags);
	if (error != 0)
	vrele(*vpp);
	return (error);
	}

	static int
	zfs_lookup_lock(vnode_t dvp, vnode_t vp, const char *name, int lkflags)
	{
	znode_t *zdp = VTOZ(dvp);
	zfsvfs_t *zfsvfs __unused = zdp->z_zfsvfs;
	int error;
	int ltype;

	if (zfsvfs->z_replay == B_FALSE)
	ASSERT_VOP_LOCKED(dvp, __func__);
	#ifdef DIAGNOSTIC
	if ((zdp->z_pflags & ZFS_XATTR) == 0)
	VERIFY(!RRM_LOCK_HELD(&zfsvfs->z_teardown_lock));
	#endif

	if (name[0] == 0 \|\| (name[0] == '.' && name[1] == 0)) {
	ASSERT3P(dvp, ==, vp);
	vref(dvp);
	ltype = lkflags & LK_TYPE_MASK;
	if (ltype != VOP_ISLOCKED(dvp)) {
	if (ltype == LK_EXCLUSIVE)
	vn_lock(dvp, LK_UPGRADE \| LK_RETRY);
	else /* if (ltype == LK_SHARED) */
	vn_lock(dvp, LK_DOWNGRADE \| LK_RETRY);

	/*
	* Relock for the "." case could leave us with
	* reclaimed vnode.
	*/
	if (VN_IS_DOOMED(dvp)) {
	vrele(dvp);
	return (SET_ERROR(ENOENT));
	}
	}
	return (0);
	} else if (name[0] == '.' && name[1] == '.' && name[2] == 0) {
	/*
	* Note that in this case, dvp is the child vnode, and we
	* are looking up the parent vnode - exactly reverse from
	* normal operation. Unlocking dvp requires some rather
	* tricky unlock/relock dance to prevent mp from being freed;
	* use vn_vget_ino_gen() which takes care of all that.
	*
	* XXX Note that there is a time window when both vnodes are
	* unlocked. It is possible, although highly unlikely, that
	* during that window the parent-child relationship between
	* the vnodes may change, for example, get reversed.
	* In that case we would have a wrong lock order for the vnodes.
	* All other filesystems seem to ignore this problem, so we
	* do the same here.
	* A potential solution could be implemented as follows:
	* - using LK_NOWAIT when locking the second vnode and retrying
	* if necessary
	* - checking that the parent-child relationship still holds
	* after locking both vnodes and retrying if it doesn't
	*/
	error = vn_vget_ino_gen(dvp, zfs_dd_callback, vp, lkflags, &vp);
	return (error);
	} else {
	error = vn_lock(vp, lkflags);
	if (error != 0)
	vrele(vp);
	return (error);
	}
	}

	/*
	* Lookup an entry in a directory, or an extended attribute directory.
	* If it exists, return a held vnode reference for it.
	*
	* IN: dvp - vnode of directory to search.
	* nm - name of entry to lookup.
	* pnp - full pathname to lookup [UNUSED].
	* flags - LOOKUP_XATTR set if looking for an attribute.
	* rdir - root directory vnode [UNUSED].
	* cr - credentials of caller.
	* ct - caller context
	*
	* OUT: vpp - vnode of located entry, NULL if not found.
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* NA
	*/
	/* ARGSUSED */
	static int
	-zfs_lookup(vnode_t dvp, char nm, vnode_t *vpp, struct componentname cnp,
	- int nameiop, cred_t cr, kthread_t td, int flags, boolean_t cached)
	+zfs_lookup(vnode_t dvp, const char nm, vnode_t **vpp,
	+ struct componentname cnp, int nameiop, cred_t cr, kthread_t *td,
	+ int flags, boolean_t cached)
	{
	znode_t *zdp = VTOZ(dvp);
	znode_t *zp;
	zfsvfs_t *zfsvfs = zdp->z_zfsvfs;
	int error = 0;

	/*
	* Fast path lookup, however we must skip DNLC lookup
	* for case folding or normalizing lookups because the
	* DNLC code only stores the passed in name. This means
	* creating 'a' and removing 'A' on a case insensitive
	* file system would work, but DNLC still thinks 'a'
	* exists and won't let you create it again on the next
	* pass through fast path.
	*/
	if (!(flags & LOOKUP_XATTR)) {
	if (dvp->v_type != VDIR) {
	return (SET_ERROR(ENOTDIR));
	} else if (zdp->z_sa_hdl == NULL) {
	return (SET_ERROR(EIO));
	}
	}

	- DTRACE_PROBE2(zfs__fastpath__lookup__miss, vnode_t , dvp, char , nm);
	+ DTRACE_PROBE2(zfs__fastpath__lookup__miss, vnode_t *, dvp,
	+ const char *, nm);

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zdp);

	*vpp = NULL;

	if (flags & LOOKUP_XATTR) {
	/*
	* If the xattr property is off, refuse the lookup request.
	*/
	if (!(zfsvfs->z_flags & ZSB_XATTR)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EOPNOTSUPP));
	}

	/*
	* We don't allow recursive attributes..
	* Maybe someday we will.
	*/
	if (zdp->z_pflags & ZFS_XATTR) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	if ((error = zfs_get_xattrdir(VTOZ(dvp), &zp, cr, flags))) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	*vpp = ZTOV(zp);

	/*
	* Do we have permission to get into attribute directory?
	*/
	error = zfs_zaccess(zp, ACE_EXECUTE, 0, B_FALSE, cr);
	if (error) {
	vrele(ZTOV(zp));
	}

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Check accessibility of directory if we're not coming in via
	* VOP_CACHEDLOOKUP.
	*/
	if (!cached) {
	#ifdef NOEXECCHECK
	if ((cnp->cn_flags & NOEXECCHECK) != 0) {
	cnp->cn_flags &= ~NOEXECCHECK;
	} else
	#endif
	if ((error = zfs_zaccess(zdp, ACE_EXECUTE, 0, B_FALSE, cr))) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	}

	if (zfsvfs->z_utf8 && u8_validate(nm, strlen(nm),
	NULL, U8_VALIDATE_ENTIRE, &error) < 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EILSEQ));
	}


	/*
	* First handle the special cases.
	*/
	if ((cnp->cn_flags & ISDOTDOT) != 0) {
	/*
	* If we are a snapshot mounted under .zfs, return
	* the vp for the snapshot directory.
	*/
	if (zdp->z_id == zfsvfs->z_root && zfsvfs->z_parent != zfsvfs) {
	struct componentname cn;
	vnode_t *zfsctl_vp;
	int ltype;

	ZFS_EXIT(zfsvfs);
	ltype = VOP_ISLOCKED(dvp);
	VOP_UNLOCK1(dvp);
	error = zfsctl_root(zfsvfs->z_parent, LK_SHARED,
	&zfsctl_vp);
	if (error == 0) {
	cn.cn_nameptr = "snapshot";
	cn.cn_namelen = strlen(cn.cn_nameptr);
	cn.cn_nameiop = cnp->cn_nameiop;
	cn.cn_flags = cnp->cn_flags & ~ISDOTDOT;
	cn.cn_lkflags = cnp->cn_lkflags;
	error = VOP_LOOKUP(zfsctl_vp, vpp, &cn);
	vput(zfsctl_vp);
	}
	vn_lock(dvp, ltype \| LK_RETRY);
	return (error);
	}
	}
	if (zfs_has_ctldir(zdp) && strcmp(nm, ZFS_CTLDIR_NAME) == 0) {
	ZFS_EXIT(zfsvfs);
	if ((cnp->cn_flags & ISLASTCN) != 0 && nameiop != LOOKUP)
	return (SET_ERROR(ENOTSUP));
	error = zfsctl_root(zfsvfs, cnp->cn_lkflags, vpp);
	return (error);
	}

	/*
	* The loop is retry the lookup if the parent-child relationship
	* changes during the dot-dot locking complexities.
	*/
	for (;;) {
	uint64_t parent;

	error = zfs_dirlook(zdp, nm, &zp);
	if (error == 0)
	*vpp = ZTOV(zp);

	ZFS_EXIT(zfsvfs);
	if (error != 0)
	break;

	error = zfs_lookup_lock(dvp, *vpp, nm, cnp->cn_lkflags);
	if (error != 0) {
	/*
	* If we've got a locking error, then the vnode
	* got reclaimed because of a force unmount.
	* We never enter doomed vnodes into the name cache.
	*/
	*vpp = NULL;
	return (error);
	}

	if ((cnp->cn_flags & ISDOTDOT) == 0)
	break;

	ZFS_ENTER(zfsvfs);
	if (zdp->z_sa_hdl == NULL) {
	error = SET_ERROR(EIO);
	} else {
	error = sa_lookup(zdp->z_sa_hdl, SA_ZPL_PARENT(zfsvfs),
	&parent, sizeof (parent));
	}
	if (error != 0) {
	ZFS_EXIT(zfsvfs);
	vput(ZTOV(zp));
	break;
	}
	if (zp->z_id == parent) {
	ZFS_EXIT(zfsvfs);
	break;
	}
	vput(ZTOV(zp));
	}

	if (error != 0)
	*vpp = NULL;

	/* Translate errors and add SAVENAME when needed. */
	if (cnp->cn_flags & ISLASTCN) {
	switch (nameiop) {
	case CREATE:
	case RENAME:
	if (error == ENOENT) {
	error = EJUSTRETURN;
	cnp->cn_flags \|= SAVENAME;
	break;
	}
	/* FALLTHROUGH */
	case DELETE:
	if (error == 0)
	cnp->cn_flags \|= SAVENAME;
	break;
	}
	}

	/* Insert name into cache (as non-existent) if appropriate. */
	if (zfsvfs->z_use_namecache && !zfsvfs->z_replay &&
	error == ENOENT && (cnp->cn_flags & MAKEENTRY) != 0)
	cache_enter(dvp, NULL, cnp);

	/* Insert name into cache if appropriate. */
	if (zfsvfs->z_use_namecache && !zfsvfs->z_replay &&
	error == 0 && (cnp->cn_flags & MAKEENTRY)) {
	if (!(cnp->cn_flags & ISLASTCN) \|\|
	(nameiop != DELETE && nameiop != RENAME)) {
	cache_enter(dvp, *vpp, cnp);
	}
	}

	return (error);
	}

	/*
	* Attempt to create a new entry in a directory. If the entry
	* already exists, truncate the file if permissible, else return
	* an error. Return the vp of the created or trunc'd file.
	*
	* IN: dvp - vnode of directory to put new file entry in.
	* name - name of new file entry.
	* vap - attributes of new file.
	* excl - flag indicating exclusive or non-exclusive mode.
	* mode - mode to open file with.
	* cr - credentials of caller.
	* flag - large file flag [UNUSED].
	* ct - caller context
	* vsecp - ACL to be set
	*
	* OUT: vpp - vnode of created or trunc'd entry.
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* dvp - ctime\|mtime updated if new entry created
	* vp - ctime\|mtime always, atime if new
	*/

	/* ARGSUSED */
	int
	-zfs_create(znode_t dzp, char name, vattr_t *vap, int excl, int mode,
	+zfs_create(znode_t dzp, const char name, vattr_t *vap, int excl, int mode,
	znode_t *zpp, cred_t cr, int flag, vsecattr_t *vsecp)
	{
	znode_t *zp;
	zfsvfs_t *zfsvfs = dzp->z_zfsvfs;
	zilog_t *zilog;
	objset_t *os;
	dmu_tx_t *tx;
	int error;
	ksid_t *ksid;
	uid_t uid;
	gid_t gid = crgetgid(cr);
	uint64_t projid = ZFS_DEFAULT_PROJID;
	zfs_acl_ids_t acl_ids;
	boolean_t fuid_dirtied;
	uint64_t txtype;
	#ifdef DEBUG_VFS_LOCKS
	vnode_t *dvp = ZTOV(dzp);
	#endif

	/*
	* If we have an ephemeral id, ACL, or XVATTR then
	* make sure file system is at proper version
	*/

	ksid = crgetsid(cr, KSID_OWNER);
	if (ksid)
	uid = ksid_getid(ksid);
	else
	uid = crgetuid(cr);

	if (zfsvfs->z_use_fuids == B_FALSE &&
	(vsecp \|\| (vap->va_mask & AT_XVATTR) \|\|
	IS_EPHEMERAL(uid) \|\| IS_EPHEMERAL(gid)))
	return (SET_ERROR(EINVAL));

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(dzp);
	os = zfsvfs->z_os;
	zilog = zfsvfs->z_log;

	if (zfsvfs->z_utf8 && u8_validate(name, strlen(name),
	NULL, U8_VALIDATE_ENTIRE, &error) < 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EILSEQ));
	}

	if (vap->va_mask & AT_XVATTR) {
	if ((error = secpolicy_xvattr(ZTOV(dzp), (xvattr_t *)vap,
	crgetuid(cr), cr, vap->va_type)) != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	}

	*zpp = NULL;

	if ((vap->va_mode & S_ISVTX) && secpolicy_vnode_stky_modify(cr))
	vap->va_mode &= ~S_ISVTX;

	error = zfs_dirent_lookup(dzp, name, &zp, ZNEW);
	if (error) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	ASSERT3P(zp, ==, NULL);

	/*
	* Create a new file object and update the directory
	* to reference it.
	*/
	if ((error = zfs_zaccess(dzp, ACE_ADD_FILE, 0, B_FALSE, cr))) {
	goto out;
	}

	/*
	* We only support the creation of regular files in
	* extended attribute directories.
	*/

	if ((dzp->z_pflags & ZFS_XATTR) &&
	(vap->va_type != VREG)) {
	error = SET_ERROR(EINVAL);
	goto out;
	}

	if ((error = zfs_acl_ids_create(dzp, 0, vap,
	cr, vsecp, &acl_ids)) != 0)
	goto out;

	if (S_ISREG(vap->va_mode) \|\| S_ISDIR(vap->va_mode))
	projid = zfs_inherit_projid(dzp);
	if (zfs_acl_ids_overquota(zfsvfs, &acl_ids, projid)) {
	zfs_acl_ids_free(&acl_ids);
	error = SET_ERROR(EDQUOT);
	goto out;
	}

	getnewvnode_reserve_();

	tx = dmu_tx_create(os);

	dmu_tx_hold_sa_create(tx, acl_ids.z_aclp->z_acl_bytes +
	ZFS_SA_BASE_ATTR_SIZE);

	fuid_dirtied = zfsvfs->z_fuid_dirty;
	if (fuid_dirtied)
	zfs_fuid_txhold(zfsvfs, tx);
	dmu_tx_hold_zap(tx, dzp->z_id, TRUE, name);
	dmu_tx_hold_sa(tx, dzp->z_sa_hdl, B_FALSE);
	if (!zfsvfs->z_use_sa &&
	acl_ids.z_aclp->z_acl_bytes > ZFS_ACE_SPACE) {
	dmu_tx_hold_write(tx, DMU_NEW_OBJECT,
	0, acl_ids.z_aclp->z_acl_bytes);
	}
	error = dmu_tx_assign(tx, TXG_WAIT);
	if (error) {
	zfs_acl_ids_free(&acl_ids);
	dmu_tx_abort(tx);
	getnewvnode_drop_reserve();
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	zfs_mknode(dzp, vap, tx, cr, 0, &zp, &acl_ids);
	if (fuid_dirtied)
	zfs_fuid_sync(zfsvfs, tx);

	(void) zfs_link_create(dzp, name, zp, tx, ZNEW);
	txtype = zfs_log_create_txtype(Z_FILE, vsecp, vap);
	zfs_log_create(zilog, tx, txtype, dzp, zp, name,
	vsecp, acl_ids.z_fuidp, vap);
	zfs_acl_ids_free(&acl_ids);
	dmu_tx_commit(tx);

	getnewvnode_drop_reserve();

	out:
	VNCHECKREF(dvp);
	if (error == 0) {
	*zpp = zp;
	}

	if (zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, 0);

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Remove an entry from a directory.
	*
	* IN: dvp - vnode of directory to remove entry from.
	* name - name of entry to remove.
	* cr - credentials of caller.
	* ct - caller context
	* flags - case flags
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* dvp - ctime\|mtime
	* vp - ctime (if nlink > 0)
	*/

	/ARGSUSED/
	static int
	-zfs_remove_(vnode_t dvp, vnode_t vp, char name, cred_t cr)
	+zfs_remove_(vnode_t dvp, vnode_t vp, const char name, cred_t cr)
	{
	znode_t *dzp = VTOZ(dvp);
	znode_t *zp;
	znode_t *xzp;
	zfsvfs_t *zfsvfs = dzp->z_zfsvfs;
	zilog_t *zilog;
	uint64_t xattr_obj;
	uint64_t obj = 0;
	dmu_tx_t *tx;
	boolean_t unlinked;
	uint64_t txtype;
	int error;


	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(dzp);
	zp = VTOZ(vp);
	ZFS_VERIFY_ZP(zp);
	zilog = zfsvfs->z_log;

	xattr_obj = 0;
	xzp = NULL;

	if ((error = zfs_zaccess_delete(dzp, zp, cr))) {
	goto out;
	}

	/*
	* Need to use rmdir for removing directories.
	*/
	if (vp->v_type == VDIR) {
	error = SET_ERROR(EPERM);
	goto out;
	}

	vnevent_remove(vp, dvp, name, ct);

	obj = zp->z_id;

	/* are there any extended attributes? */
	error = sa_lookup(zp->z_sa_hdl, SA_ZPL_XATTR(zfsvfs),
	&xattr_obj, sizeof (xattr_obj));
	if (error == 0 && xattr_obj) {
	error = zfs_zget(zfsvfs, xattr_obj, &xzp);
	ASSERT0(error);
	}

	/*
	* We may delete the znode now, or we may put it in the unlinked set;
	* it depends on whether we're the last link, and on whether there are
	* other holds on the vnode. So we dmu_tx_hold() the right things to
	* allow for either case.
	*/
	tx = dmu_tx_create(zfsvfs->z_os);
	dmu_tx_hold_zap(tx, dzp->z_id, FALSE, name);
	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_FALSE);
	zfs_sa_upgrade_txholds(tx, zp);
	zfs_sa_upgrade_txholds(tx, dzp);

	if (xzp) {
	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_TRUE);
	dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE);
	}

	/* charge as an update -- would be nice not to charge at all */
	dmu_tx_hold_zap(tx, zfsvfs->z_unlinkedobj, FALSE, NULL);

	/*
	* Mark this transaction as typically resulting in a net free of space
	*/
	dmu_tx_mark_netfree(tx);

	error = dmu_tx_assign(tx, TXG_WAIT);
	if (error) {
	dmu_tx_abort(tx);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Remove the directory entry.
	*/
	error = zfs_link_destroy(dzp, name, zp, tx, ZEXISTS, &unlinked);

	if (error) {
	dmu_tx_commit(tx);
	goto out;
	}

	if (unlinked) {
	zfs_unlinked_add(zp, tx);
	vp->v_vflag \|= VV_NOSYNC;
	}
	/* XXX check changes to linux vnops */
	txtype = TX_REMOVE;
	zfs_log_remove(zilog, tx, txtype, dzp, name, obj, unlinked);

	dmu_tx_commit(tx);
	out:

	if (xzp)
	vrele(ZTOV(xzp));

	if (zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, 0);


	ZFS_EXIT(zfsvfs);
	return (error);
	}


	static int
	-zfs_lookup_internal(znode_t dzp, char name, vnode_t **vpp,
	+zfs_lookup_internal(znode_t dzp, const char name, vnode_t **vpp,
	struct componentname *cnp, int nameiop)
	{
	zfsvfs_t *zfsvfs = dzp->z_zfsvfs;
	int error;

	- cnp->cn_nameptr = name;
	+ cnp->cn_nameptr = __DECONST(char *, name);
	cnp->cn_namelen = strlen(name);
	cnp->cn_nameiop = nameiop;
	cnp->cn_flags = ISLASTCN \| SAVENAME;
	cnp->cn_lkflags = LK_EXCLUSIVE \| LK_RETRY;
	cnp->cn_cred = kcred;
	cnp->cn_thread = curthread;

	if (zfsvfs->z_use_namecache && !zfsvfs->z_replay) {
	struct vop_lookup_args a;

	a.a_gen.a_desc = &vop_lookup_desc;
	a.a_dvp = ZTOV(dzp);
	a.a_vpp = vpp;
	a.a_cnp = cnp;
	error = vfs_cache_lookup(&a);
	} else {
	error = zfs_lookup(ZTOV(dzp), name, vpp, cnp, nameiop, kcred,
	curthread, 0, B_FALSE);
	}
	#ifdef ZFS_DEBUG
	if (error) {
	printf("got error %d on name %s on op %d\n", error, name,
	nameiop);
	kdb_backtrace();
	}
	#endif
	return (error);
	}

	int
	-zfs_remove(znode_t dzp, char name, cred_t *cr, int flags)
	+zfs_remove(znode_t dzp, const char name, cred_t *cr, int flags)
	{
	vnode_t *vp;
	int error;
	struct componentname cn;

	if ((error = zfs_lookup_internal(dzp, name, &vp, &cn, DELETE)))
	return (error);

	error = zfs_remove_(ZTOV(dzp), vp, name, cr);
	vput(vp);
	return (error);
	}
	/*
	* Create a new directory and insert it into dvp using the name
	* provided. Return a pointer to the inserted directory.
	*
	* IN: dvp - vnode of directory to add subdir to.
	* dirname - name of new directory.
	* vap - attributes of new directory.
	* cr - credentials of caller.
	* ct - caller context
	* flags - case flags
	* vsecp - ACL to be set
	*
	* OUT: vpp - vnode of created directory.
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* dvp - ctime\|mtime updated
	* vp - ctime\|mtime\|atime updated
	*/
	/ARGSUSED/
	int
	-zfs_mkdir(znode_t dzp, char dirname, vattr_t vap, znode_t zpp, cred_t cr,
	- int flags, vsecattr_t *vsecp)
	+zfs_mkdir(znode_t dzp, const char dirname, vattr_t vap, znode_t *zpp,
	+ cred_t cr, int flags, vsecattr_t vsecp)
	{
	znode_t *zp;
	zfsvfs_t *zfsvfs = dzp->z_zfsvfs;
	zilog_t *zilog;
	uint64_t txtype;
	dmu_tx_t *tx;
	int error;
	ksid_t *ksid;
	uid_t uid;
	gid_t gid = crgetgid(cr);
	zfs_acl_ids_t acl_ids;
	boolean_t fuid_dirtied;

	ASSERT(vap->va_type == VDIR);

	/*
	* If we have an ephemeral id, ACL, or XVATTR then
	* make sure file system is at proper version
	*/

	ksid = crgetsid(cr, KSID_OWNER);
	if (ksid)
	uid = ksid_getid(ksid);
	else
	uid = crgetuid(cr);
	if (zfsvfs->z_use_fuids == B_FALSE &&
	((vap->va_mask & AT_XVATTR) \|\|
	IS_EPHEMERAL(uid) \|\| IS_EPHEMERAL(gid)))
	return (SET_ERROR(EINVAL));

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(dzp);
	zilog = zfsvfs->z_log;

	if (dzp->z_pflags & ZFS_XATTR) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	if (zfsvfs->z_utf8 && u8_validate(dirname,
	strlen(dirname), NULL, U8_VALIDATE_ENTIRE, &error) < 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EILSEQ));
	}

	if (vap->va_mask & AT_XVATTR) {
	if ((error = secpolicy_xvattr(ZTOV(dzp), (xvattr_t *)vap,
	crgetuid(cr), cr, vap->va_type)) != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	}

	if ((error = zfs_acl_ids_create(dzp, 0, vap, cr,
	NULL, &acl_ids)) != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* First make sure the new directory doesn't exist.
	*
	* Existence is checked first to make sure we don't return
	* EACCES instead of EEXIST which can cause some applications
	* to fail.
	*/
	*zpp = NULL;

	if ((error = zfs_dirent_lookup(dzp, dirname, &zp, ZNEW))) {
	zfs_acl_ids_free(&acl_ids);
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	ASSERT3P(zp, ==, NULL);

	if ((error = zfs_zaccess(dzp, ACE_ADD_SUBDIRECTORY, 0, B_FALSE, cr))) {
	zfs_acl_ids_free(&acl_ids);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	if (zfs_acl_ids_overquota(zfsvfs, &acl_ids, zfs_inherit_projid(dzp))) {
	zfs_acl_ids_free(&acl_ids);
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EDQUOT));
	}

	/*
	* Add a new entry to the directory.
	*/
	getnewvnode_reserve_();
	tx = dmu_tx_create(zfsvfs->z_os);
	dmu_tx_hold_zap(tx, dzp->z_id, TRUE, dirname);
	dmu_tx_hold_zap(tx, DMU_NEW_OBJECT, FALSE, NULL);
	fuid_dirtied = zfsvfs->z_fuid_dirty;
	if (fuid_dirtied)
	zfs_fuid_txhold(zfsvfs, tx);
	if (!zfsvfs->z_use_sa && acl_ids.z_aclp->z_acl_bytes > ZFS_ACE_SPACE) {
	dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0,
	acl_ids.z_aclp->z_acl_bytes);
	}

	dmu_tx_hold_sa_create(tx, acl_ids.z_aclp->z_acl_bytes +
	ZFS_SA_BASE_ATTR_SIZE);

	error = dmu_tx_assign(tx, TXG_WAIT);
	if (error) {
	zfs_acl_ids_free(&acl_ids);
	dmu_tx_abort(tx);
	getnewvnode_drop_reserve();
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Create new node.
	*/
	zfs_mknode(dzp, vap, tx, cr, 0, &zp, &acl_ids);

	if (fuid_dirtied)
	zfs_fuid_sync(zfsvfs, tx);

	/*
	* Now put new name in parent dir.
	*/
	(void) zfs_link_create(dzp, dirname, zp, tx, ZNEW);

	*zpp = zp;

	txtype = zfs_log_create_txtype(Z_DIR, NULL, vap);
	zfs_log_create(zilog, tx, txtype, dzp, zp, dirname, NULL,
	acl_ids.z_fuidp, vap);

	zfs_acl_ids_free(&acl_ids);

	dmu_tx_commit(tx);

	getnewvnode_drop_reserve();

	if (zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, 0);

	ZFS_EXIT(zfsvfs);
	return (0);
	}

	/*
	* Remove a directory subdir entry. If the current working
	* directory is the same as the subdir to be removed, the
	* remove will fail.
	*
	* IN: dvp - vnode of directory to remove from.
	* name - name of directory to be removed.
	* cwd - vnode of current working directory.
	* cr - credentials of caller.
	* ct - caller context
	* flags - case flags
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* dvp - ctime\|mtime updated
	*/
	/ARGSUSED/
	static int
	-zfs_rmdir_(vnode_t dvp, vnode_t vp, char name, cred_t cr)
	+zfs_rmdir_(vnode_t dvp, vnode_t vp, const char name, cred_t cr)
	{
	znode_t *dzp = VTOZ(dvp);
	znode_t *zp = VTOZ(vp);
	zfsvfs_t *zfsvfs = dzp->z_zfsvfs;
	zilog_t *zilog;
	dmu_tx_t *tx;
	int error;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(dzp);
	ZFS_VERIFY_ZP(zp);
	zilog = zfsvfs->z_log;


	if ((error = zfs_zaccess_delete(dzp, zp, cr))) {
	goto out;
	}

	if (vp->v_type != VDIR) {
	error = SET_ERROR(ENOTDIR);
	goto out;
	}

	vnevent_rmdir(vp, dvp, name, ct);

	tx = dmu_tx_create(zfsvfs->z_os);
	dmu_tx_hold_zap(tx, dzp->z_id, FALSE, name);
	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_FALSE);
	dmu_tx_hold_zap(tx, zfsvfs->z_unlinkedobj, FALSE, NULL);
	zfs_sa_upgrade_txholds(tx, zp);
	zfs_sa_upgrade_txholds(tx, dzp);
	dmu_tx_mark_netfree(tx);
	error = dmu_tx_assign(tx, TXG_WAIT);
	if (error) {
	dmu_tx_abort(tx);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	cache_purge(dvp);

	error = zfs_link_destroy(dzp, name, zp, tx, ZEXISTS, NULL);

	if (error == 0) {
	uint64_t txtype = TX_RMDIR;
	zfs_log_remove(zilog, tx, txtype, dzp, name,
	ZFS_NO_OBJECT, B_FALSE);
	}

	dmu_tx_commit(tx);

	cache_purge(vp);
	out:
	if (zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, 0);

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	int
	-zfs_rmdir(znode_t dzp, char name, znode_t cwd, cred_t cr, int flags)
	+zfs_rmdir(znode_t dzp, const char name, znode_t cwd, cred_t cr, int flags)
	{
	struct componentname cn;
	vnode_t *vp;
	int error;

	if ((error = zfs_lookup_internal(dzp, name, &vp, &cn, DELETE)))
	return (error);

	error = zfs_rmdir_(ZTOV(dzp), vp, name, cr);
	vput(vp);
	return (error);
	}

	/*
	* Read as many directory entries as will fit into the provided
	* buffer from the given directory cursor position (specified in
	* the uio structure).
	*
	* IN: vp - vnode of directory to read.
	* uio - structure supplying read location, range info,
	* and return buffer.
	* cr - credentials of caller.
	* ct - caller context
	* flags - case flags
	*
	* OUT: uio - updated offset and range, buffer filled.
	* eofp - set to true if end-of-file detected.
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* vp - atime updated
	*
	* Note that the low 4 bits of the cookie returned by zap is always zero.
	* This allows us to use the low range for "special" directory entries:
	* We use 0 for '.', and 1 for '..'. If this is the root of the filesystem,
	* we use the offset 2 for the '.zfs' directory.
	*/
	/* ARGSUSED */
	static int
	zfs_readdir(vnode_t vp, uio_t uio, cred_t cr, int eofp,
	int ncookies, ulong_t *cookies)
	{
	znode_t *zp = VTOZ(vp);
	iovec_t *iovp;
	edirent_t *eodp;
	dirent64_t *odp;
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;
	objset_t *os;
	caddr_t outbuf;
	size_t bufsize;
	zap_cursor_t zc;
	zap_attribute_t zap;
	uint_t bytes_wanted;
	uint64_t offset; /* must be unsigned; checks for < 1 */
	uint64_t parent;
	int local_eof;
	int outcount;
	int error;
	uint8_t prefetch;
	boolean_t check_sysattrs;
	uint8_t type;
	int ncooks;
	ulong_t *cooks = NULL;
	int flags = 0;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	if ((error = sa_lookup(zp->z_sa_hdl, SA_ZPL_PARENT(zfsvfs),
	&parent, sizeof (parent))) != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* If we are not given an eof variable,
	* use a local one.
	*/
	if (eofp == NULL)
	eofp = &local_eof;

	/*
	* Check for valid iov_len.
	*/
	if (uio->uio_iov->iov_len <= 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	/*
	* Quit if directory has been removed (posix)
	*/
	if ((*eofp = zp->z_unlinked) != 0) {
	ZFS_EXIT(zfsvfs);
	return (0);
	}

	error = 0;
	os = zfsvfs->z_os;
	offset = uio->uio_loffset;
	prefetch = zp->z_zn_prefetch;

	/*
	* Initialize the iterator cursor.
	*/
	if (offset <= 3) {
	/*
	* Start iteration from the beginning of the directory.
	*/
	zap_cursor_init(&zc, os, zp->z_id);
	} else {
	/*
	* The offset is a serialized cursor.
	*/
	zap_cursor_init_serialized(&zc, os, zp->z_id, offset);
	}

	/*
	* Get space to change directory entries into fs independent format.
	*/
	iovp = uio->uio_iov;
	bytes_wanted = iovp->iov_len;
	if (uio->uio_segflg != UIO_SYSSPACE \|\| uio->uio_iovcnt != 1) {
	bufsize = bytes_wanted;
	outbuf = kmem_alloc(bufsize, KM_SLEEP);
	odp = (struct dirent64 *)outbuf;
	} else {
	bufsize = bytes_wanted;
	outbuf = NULL;
	odp = (struct dirent64 *)iovp->iov_base;
	}
	eodp = (struct edirent *)odp;

	if (ncookies != NULL) {
	/*
	* Minimum entry size is dirent size and 1 byte for a file name.
	*/
	ncooks = uio->uio_resid / (sizeof (struct dirent) -
	sizeof (((struct dirent *)NULL)->d_name) + 1);
	cooks = malloc(ncooks * sizeof (ulong_t), M_TEMP, M_WAITOK);
	*cookies = cooks;
	*ncookies = ncooks;
	}
	/*
	* If this VFS supports the system attribute view interface; and
	* we're looking at an extended attribute directory; and we care
	* about normalization conflicts on this vfs; then we must check
	* for normalization conflicts with the sysattr name space.
	*/
	#ifdef TODO
	check_sysattrs = vfs_has_feature(vp->v_vfsp, VFSFT_SYSATTR_VIEWS) &&
	(vp->v_flag & V_XATTRDIR) && zfsvfs->z_norm &&
	(flags & V_RDDIR_ENTFLAGS);
	#else
	check_sysattrs = 0;
	#endif

	/*
	* Transform to file-system independent format
	*/
	outcount = 0;
	while (outcount < bytes_wanted) {
	ino64_t objnum;
	ushort_t reclen;
	off64_t *next = NULL;

	/*
	* Special case `.', `..', and `.zfs'.
	*/
	if (offset == 0) {
	(void) strcpy(zap.za_name, ".");
	zap.za_normalization_conflict = 0;
	objnum = zp->z_id;
	type = DT_DIR;
	} else if (offset == 1) {
	(void) strcpy(zap.za_name, "..");
	zap.za_normalization_conflict = 0;
	objnum = parent;
	type = DT_DIR;
	} else if (offset == 2 && zfs_show_ctldir(zp)) {
	(void) strcpy(zap.za_name, ZFS_CTLDIR_NAME);
	zap.za_normalization_conflict = 0;
	objnum = ZFSCTL_INO_ROOT;
	type = DT_DIR;
	} else {
	/*
	* Grab next entry.
	*/
	if ((error = zap_cursor_retrieve(&zc, &zap))) {
	if ((*eofp = (error == ENOENT)) != 0)
	break;
	else
	goto update;
	}

	if (zap.za_integer_length != 8 \|\|
	zap.za_num_integers != 1) {
	cmn_err(CE_WARN, "zap_readdir: bad directory "
	"entry, obj = %lld, offset = %lld\n",
	(u_longlong_t)zp->z_id,
	(u_longlong_t)offset);
	error = SET_ERROR(ENXIO);
	goto update;
	}

	objnum = ZFS_DIRENT_OBJ(zap.za_first_integer);
	/*
	* MacOS X can extract the object type here such as:
	* uint8_t type = ZFS_DIRENT_TYPE(zap.za_first_integer);
	*/
	type = ZFS_DIRENT_TYPE(zap.za_first_integer);

	if (check_sysattrs && !zap.za_normalization_conflict) {
	#ifdef TODO
	zap.za_normalization_conflict =
	xattr_sysattr_casechk(zap.za_name);
	#else
	panic("%s:%u: TODO", __func__, __LINE__);
	#endif
	}
	}

	if (flags & V_RDDIR_ACCFILTER) {
	/*
	* If we have no access at all, don't include
	* this entry in the returned information
	*/
	znode_t *ezp;
	if (zfs_zget(zp->z_zfsvfs, objnum, &ezp) != 0)
	goto skip_entry;
	if (!zfs_has_access(ezp, cr)) {
	vrele(ZTOV(ezp));
	goto skip_entry;
	}
	vrele(ZTOV(ezp));
	}

	if (flags & V_RDDIR_ENTFLAGS)
	reclen = EDIRENT_RECLEN(strlen(zap.za_name));
	else
	reclen = DIRENT64_RECLEN(strlen(zap.za_name));

	/*
	* Will this entry fit in the buffer?
	*/
	if (outcount + reclen > bufsize) {
	/*
	* Did we manage to fit anything in the buffer?
	*/
	if (!outcount) {
	error = SET_ERROR(EINVAL);
	goto update;
	}
	break;
	}
	if (flags & V_RDDIR_ENTFLAGS) {
	/*
	* Add extended flag entry:
	*/
	eodp->ed_ino = objnum;
	eodp->ed_reclen = reclen;
	/* NOTE: ed_off is the offset for the next entry */
	next = &(eodp->ed_off);
	eodp->ed_eflags = zap.za_normalization_conflict ?
	ED_CASE_CONFLICT : 0;
	(void) strncpy(eodp->ed_name, zap.za_name,
	EDIRENT_NAMELEN(reclen));
	eodp = (edirent_t *)((intptr_t)eodp + reclen);
	} else {
	/*
	* Add normal entry:
	*/
	odp->d_ino = objnum;
	odp->d_reclen = reclen;
	odp->d_namlen = strlen(zap.za_name);
	/* NOTE: d_off is the offset for the next entry. */
	next = &odp->d_off;
	strlcpy(odp->d_name, zap.za_name, odp->d_namlen + 1);
	odp->d_type = type;
	dirent_terminate(odp);
	odp = (dirent64_t *)((intptr_t)odp + reclen);
	}
	outcount += reclen;

	ASSERT(outcount <= bufsize);

	/* Prefetch znode */
	if (prefetch)
	dmu_prefetch(os, objnum, 0, 0, 0,
	ZIO_PRIORITY_SYNC_READ);

	skip_entry:
	/*
	* Move to the next entry, fill in the previous offset.
	*/
	if (offset > 2 \|\| (offset == 2 && !zfs_show_ctldir(zp))) {
	zap_cursor_advance(&zc);
	offset = zap_cursor_serialize(&zc);
	} else {
	offset += 1;
	}

	/* Fill the offset right after advancing the cursor. */
	if (next != NULL)
	*next = offset;
	if (cooks != NULL) {
	*cooks++ = offset;
	ncooks--;
	KASSERT(ncooks >= 0, ("ncookies=%d", ncooks));
	}
	}
	zp->z_zn_prefetch = B_FALSE; /* a lookup will re-enable pre-fetching */

	/* Subtract unused cookies */
	if (ncookies != NULL)
	*ncookies -= ncooks;

	if (uio->uio_segflg == UIO_SYSSPACE && uio->uio_iovcnt == 1) {
	iovp->iov_base += outcount;
	iovp->iov_len -= outcount;
	uio->uio_resid -= outcount;
	} else if ((error = uiomove(outbuf, (long)outcount, UIO_READ, uio))) {
	/*
	* Reset the pointer.
	*/
	offset = uio->uio_loffset;
	}

	update:
	zap_cursor_fini(&zc);
	if (uio->uio_segflg != UIO_SYSSPACE \|\| uio->uio_iovcnt != 1)
	kmem_free(outbuf, bufsize);

	if (error == ENOENT)
	error = 0;

	ZFS_ACCESSTIME_STAMP(zfsvfs, zp);

	uio->uio_loffset = offset;
	ZFS_EXIT(zfsvfs);
	if (error != 0 && cookies != NULL) {
	free(*cookies, M_TEMP);
	*cookies = NULL;
	*ncookies = 0;
	}
	return (error);
	}

	ulong_t zfs_fsync_sync_cnt = 4;

	static int
	zfs_fsync(vnode_t vp, int syncflag, cred_t cr, caller_context_t *ct)
	{
	znode_t *zp = VTOZ(vp);
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;

	(void) tsd_set(zfs_fsyncer_key, (void *)zfs_fsync_sync_cnt);

	if (zfsvfs->z_os->os_sync != ZFS_SYNC_DISABLED) {
	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);
	zil_commit(zfsvfs->z_log, zp->z_id);
	ZFS_EXIT(zfsvfs);
	}
	tsd_set(zfs_fsyncer_key, NULL);
	return (0);
	}


	/*
	* Get the requested file attributes and place them in the provided
	* vattr structure.
	*
	* IN: vp - vnode of file.
	* vap - va_mask identifies requested attributes.
	* If AT_XVATTR set, then optional attrs are requested
	* flags - ATTR_NOACLCHECK (CIFS server context)
	* cr - credentials of caller.
	*
	* OUT: vap - attribute values.
	*
	* RETURN: 0 (always succeeds).
	*/
	/* ARGSUSED */
	static int
	zfs_getattr(vnode_t vp, vattr_t vap, int flags, cred_t *cr)
	{
	znode_t *zp = VTOZ(vp);
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;
	int error = 0;
	uint32_t blksize;
	u_longlong_t nblocks;
	uint64_t mtime[2], ctime[2], crtime[2], rdev;
	xvattr_t xvap = (xvattr_t )vap; /* vap may be an xvattr_t * */
	xoptattr_t *xoap = NULL;
	boolean_t skipaclchk = (flags & ATTR_NOACLCHECK) ? B_TRUE : B_FALSE;
	sa_bulk_attr_t bulk[4];
	int count = 0;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	zfs_fuid_map_ids(zp, cr, &vap->va_uid, &vap->va_gid);

	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_MTIME(zfsvfs), NULL, &mtime, 16);
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_CTIME(zfsvfs), NULL, &ctime, 16);
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_CRTIME(zfsvfs), NULL, &crtime, 16);
	if (vp->v_type == VBLK \|\| vp->v_type == VCHR)
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_RDEV(zfsvfs), NULL,
	&rdev, 8);

	if ((error = sa_bulk_lookup(zp->z_sa_hdl, bulk, count)) != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* If ACL is trivial don't bother looking for ACE_READ_ATTRIBUTES.
	* Also, if we are the owner don't bother, since owner should
	* always be allowed to read basic attributes of file.
	*/
	if (!(zp->z_pflags & ZFS_ACL_TRIVIAL) &&
	(vap->va_uid != crgetuid(cr))) {
	if ((error = zfs_zaccess(zp, ACE_READ_ATTRIBUTES, 0,
	skipaclchk, cr))) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	}

	/*
	* Return all attributes. It's cheaper to provide the answer
	* than to determine whether we were asked the question.
	*/

	vap->va_type = IFTOVT(zp->z_mode);
	vap->va_mode = zp->z_mode & ~S_IFMT;
	vn_fsid(vp, vap);
	vap->va_nodeid = zp->z_id;
	vap->va_nlink = zp->z_links;
	if ((vp->v_flag & VROOT) && zfs_show_ctldir(zp) &&
	zp->z_links < ZFS_LINK_MAX)
	vap->va_nlink++;
	vap->va_size = zp->z_size;
	if (vp->v_type == VBLK \|\| vp->v_type == VCHR)
	vap->va_rdev = zfs_cmpldev(rdev);
	vap->va_seq = zp->z_seq;
	vap->va_flags = 0; /* FreeBSD: Reset chflags(2) flags. */
	vap->va_filerev = zp->z_seq;

	/*
	* Add in any requested optional attributes and the create time.
	* Also set the corresponding bits in the returned attribute bitmap.
	*/
	if ((xoap = xva_getxoptattr(xvap)) != NULL && zfsvfs->z_use_fuids) {
	if (XVA_ISSET_REQ(xvap, XAT_ARCHIVE)) {
	xoap->xoa_archive =
	((zp->z_pflags & ZFS_ARCHIVE) != 0);
	XVA_SET_RTN(xvap, XAT_ARCHIVE);
	}

	if (XVA_ISSET_REQ(xvap, XAT_READONLY)) {
	xoap->xoa_readonly =
	((zp->z_pflags & ZFS_READONLY) != 0);
	XVA_SET_RTN(xvap, XAT_READONLY);
	}

	if (XVA_ISSET_REQ(xvap, XAT_SYSTEM)) {
	xoap->xoa_system =
	((zp->z_pflags & ZFS_SYSTEM) != 0);
	XVA_SET_RTN(xvap, XAT_SYSTEM);
	}

	if (XVA_ISSET_REQ(xvap, XAT_HIDDEN)) {
	xoap->xoa_hidden =
	((zp->z_pflags & ZFS_HIDDEN) != 0);
	XVA_SET_RTN(xvap, XAT_HIDDEN);
	}

	if (XVA_ISSET_REQ(xvap, XAT_NOUNLINK)) {
	xoap->xoa_nounlink =
	((zp->z_pflags & ZFS_NOUNLINK) != 0);
	XVA_SET_RTN(xvap, XAT_NOUNLINK);
	}

	if (XVA_ISSET_REQ(xvap, XAT_IMMUTABLE)) {
	xoap->xoa_immutable =
	((zp->z_pflags & ZFS_IMMUTABLE) != 0);
	XVA_SET_RTN(xvap, XAT_IMMUTABLE);
	}

	if (XVA_ISSET_REQ(xvap, XAT_APPENDONLY)) {
	xoap->xoa_appendonly =
	((zp->z_pflags & ZFS_APPENDONLY) != 0);
	XVA_SET_RTN(xvap, XAT_APPENDONLY);
	}

	if (XVA_ISSET_REQ(xvap, XAT_NODUMP)) {
	xoap->xoa_nodump =
	((zp->z_pflags & ZFS_NODUMP) != 0);
	XVA_SET_RTN(xvap, XAT_NODUMP);
	}

	if (XVA_ISSET_REQ(xvap, XAT_OPAQUE)) {
	xoap->xoa_opaque =
	((zp->z_pflags & ZFS_OPAQUE) != 0);
	XVA_SET_RTN(xvap, XAT_OPAQUE);
	}

	if (XVA_ISSET_REQ(xvap, XAT_AV_QUARANTINED)) {
	xoap->xoa_av_quarantined =
	((zp->z_pflags & ZFS_AV_QUARANTINED) != 0);
	XVA_SET_RTN(xvap, XAT_AV_QUARANTINED);
	}

	if (XVA_ISSET_REQ(xvap, XAT_AV_MODIFIED)) {
	xoap->xoa_av_modified =
	((zp->z_pflags & ZFS_AV_MODIFIED) != 0);
	XVA_SET_RTN(xvap, XAT_AV_MODIFIED);
	}

	if (XVA_ISSET_REQ(xvap, XAT_AV_SCANSTAMP) &&
	vp->v_type == VREG) {
	zfs_sa_get_scanstamp(zp, xvap);
	}

	if (XVA_ISSET_REQ(xvap, XAT_REPARSE)) {
	xoap->xoa_reparse = ((zp->z_pflags & ZFS_REPARSE) != 0);
	XVA_SET_RTN(xvap, XAT_REPARSE);
	}
	if (XVA_ISSET_REQ(xvap, XAT_GEN)) {
	xoap->xoa_generation = zp->z_gen;
	XVA_SET_RTN(xvap, XAT_GEN);
	}

	if (XVA_ISSET_REQ(xvap, XAT_OFFLINE)) {
	xoap->xoa_offline =
	((zp->z_pflags & ZFS_OFFLINE) != 0);
	XVA_SET_RTN(xvap, XAT_OFFLINE);
	}

	if (XVA_ISSET_REQ(xvap, XAT_SPARSE)) {
	xoap->xoa_sparse =
	((zp->z_pflags & ZFS_SPARSE) != 0);
	XVA_SET_RTN(xvap, XAT_SPARSE);
	}

	if (XVA_ISSET_REQ(xvap, XAT_PROJINHERIT)) {
	xoap->xoa_projinherit =
	((zp->z_pflags & ZFS_PROJINHERIT) != 0);
	XVA_SET_RTN(xvap, XAT_PROJINHERIT);
	}

	if (XVA_ISSET_REQ(xvap, XAT_PROJID)) {
	xoap->xoa_projid = zp->z_projid;
	XVA_SET_RTN(xvap, XAT_PROJID);
	}
	}

	ZFS_TIME_DECODE(&vap->va_atime, zp->z_atime);
	ZFS_TIME_DECODE(&vap->va_mtime, mtime);
	ZFS_TIME_DECODE(&vap->va_ctime, ctime);
	ZFS_TIME_DECODE(&vap->va_birthtime, crtime);


	sa_object_size(zp->z_sa_hdl, &blksize, &nblocks);
	vap->va_blksize = blksize;
	vap->va_bytes = nblocks << 9; /* nblocks * 512 */

	if (zp->z_blksz == 0) {
	/*
	* Block size hasn't been set; suggest maximal I/O transfers.
	*/
	vap->va_blksize = zfsvfs->z_max_blksz;
	}

	ZFS_EXIT(zfsvfs);
	return (0);
	}

	/*
	* Set the file attributes to the values contained in the
	* vattr structure.
	*
	* IN: zp - znode of file to be modified.
	* vap - new attribute values.
	* If AT_XVATTR set, then optional attrs are being set
	* flags - ATTR_UTIME set if non-default time values provided.
	* - ATTR_NOACLCHECK (CIFS context only).
	* cr - credentials of caller.
	* ct - caller context
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* vp - ctime updated, mtime updated if size changed.
	*/
	/* ARGSUSED */
	int
	zfs_setattr(znode_t zp, vattr_t vap, int flags, cred_t *cr)
	{
	vnode_t *vp = ZTOV(zp);
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;
	objset_t *os = zfsvfs->z_os;
	zilog_t *zilog;
	dmu_tx_t *tx;
	vattr_t oldva;
	xvattr_t tmpxvattr;
	uint_t mask = vap->va_mask;
	uint_t saved_mask = 0;
	uint64_t saved_mode;
	int trim_mask = 0;
	uint64_t new_mode;
	uint64_t new_uid, new_gid;
	uint64_t xattr_obj;
	uint64_t mtime[2], ctime[2];
	uint64_t projid = ZFS_INVALID_PROJID;
	znode_t *attrzp;
	int need_policy = FALSE;
	int err, err2;
	zfs_fuid_info_t *fuidp = NULL;
	xvattr_t xvap = (xvattr_t )vap; /* vap may be an xvattr_t * */
	xoptattr_t *xoap;
	zfs_acl_t *aclp;
	boolean_t skipaclchk = (flags & ATTR_NOACLCHECK) ? B_TRUE : B_FALSE;
	boolean_t fuid_dirtied = B_FALSE;
	sa_bulk_attr_t bulk[7], xattr_bulk[7];
	int count = 0, xattr_count = 0;

	if (mask == 0)
	return (0);

	if (mask & AT_NOSET)
	return (SET_ERROR(EINVAL));

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	zilog = zfsvfs->z_log;

	/*
	* Make sure that if we have ephemeral uid/gid or xvattr specified
	* that file system is at proper version level
	*/

	if (zfsvfs->z_use_fuids == B_FALSE &&
	(((mask & AT_UID) && IS_EPHEMERAL(vap->va_uid)) \|\|
	((mask & AT_GID) && IS_EPHEMERAL(vap->va_gid)) \|\|
	(mask & AT_XVATTR))) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	if (mask & AT_SIZE && vp->v_type == VDIR) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EISDIR));
	}

	if (mask & AT_SIZE && vp->v_type != VREG && vp->v_type != VFIFO) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	/*
	* If this is an xvattr_t, then get a pointer to the structure of
	* optional attributes. If this is NULL, then we have a vattr_t.
	*/
	xoap = xva_getxoptattr(xvap);

	xva_init(&tmpxvattr);

	/*
	* Immutable files can only alter immutable bit and atime
	*/
	if ((zp->z_pflags & ZFS_IMMUTABLE) &&
	((mask & (AT_SIZE\|AT_UID\|AT_GID\|AT_MTIME\|AT_MODE)) \|\|
	((mask & AT_XVATTR) && XVA_ISSET_REQ(xvap, XAT_CREATETIME)))) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EPERM));
	}

	/*
	* Note: ZFS_READONLY is handled in zfs_zaccess_common.
	*/

	/*
	* Verify timestamps doesn't overflow 32 bits.
	* ZFS can handle large timestamps, but 32bit syscalls can't
	* handle times greater than 2039. This check should be removed
	* once large timestamps are fully supported.
	*/
	if (mask & (AT_ATIME \| AT_MTIME)) {
	if (((mask & AT_ATIME) && TIMESPEC_OVERFLOW(&vap->va_atime)) \|\|
	((mask & AT_MTIME) && TIMESPEC_OVERFLOW(&vap->va_mtime))) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EOVERFLOW));
	}
	}
	if (xoap != NULL && (mask & AT_XVATTR)) {
	if (XVA_ISSET_REQ(xvap, XAT_CREATETIME) &&
	TIMESPEC_OVERFLOW(&vap->va_birthtime)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EOVERFLOW));
	}

	if (XVA_ISSET_REQ(xvap, XAT_PROJID)) {
	if (!dmu_objset_projectquota_enabled(os) \|\|
	(!S_ISREG(zp->z_mode) && !S_ISDIR(zp->z_mode))) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EOPNOTSUPP));
	}

	projid = xoap->xoa_projid;
	if (unlikely(projid == ZFS_INVALID_PROJID)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	if (projid == zp->z_projid && zp->z_pflags & ZFS_PROJID)
	projid = ZFS_INVALID_PROJID;
	else
	need_policy = TRUE;
	}

	if (XVA_ISSET_REQ(xvap, XAT_PROJINHERIT) &&
	(xoap->xoa_projinherit !=
	((zp->z_pflags & ZFS_PROJINHERIT) != 0)) &&
	(!dmu_objset_projectquota_enabled(os) \|\|
	(!S_ISREG(zp->z_mode) && !S_ISDIR(zp->z_mode)))) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EOPNOTSUPP));
	}
	}

	attrzp = NULL;
	aclp = NULL;

	if (zfsvfs->z_vfs->vfs_flag & VFS_RDONLY) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EROFS));
	}

	/*
	* First validate permissions
	*/

	if (mask & AT_SIZE) {
	/*
	* XXX - Note, we are not providing any open
	* mode flags here (like FNDELAY), so we may
	* block if there are locks present... this
	* should be addressed in openat().
	*/
	/* XXX - would it be OK to generate a log record here? */
	err = zfs_freesp(zp, vap->va_size, 0, 0, FALSE);
	if (err) {
	ZFS_EXIT(zfsvfs);
	return (err);
	}
	}

	if (mask & (AT_ATIME\|AT_MTIME) \|\|
	((mask & AT_XVATTR) && (XVA_ISSET_REQ(xvap, XAT_HIDDEN) \|\|
	XVA_ISSET_REQ(xvap, XAT_READONLY) \|\|
	XVA_ISSET_REQ(xvap, XAT_ARCHIVE) \|\|
	XVA_ISSET_REQ(xvap, XAT_OFFLINE) \|\|
	XVA_ISSET_REQ(xvap, XAT_SPARSE) \|\|
	XVA_ISSET_REQ(xvap, XAT_CREATETIME) \|\|
	XVA_ISSET_REQ(xvap, XAT_SYSTEM)))) {
	need_policy = zfs_zaccess(zp, ACE_WRITE_ATTRIBUTES, 0,
	skipaclchk, cr);
	}

	if (mask & (AT_UID\|AT_GID)) {
	int idmask = (mask & (AT_UID\|AT_GID));
	int take_owner;
	int take_group;

	/*
	* NOTE: even if a new mode is being set,
	* we may clear S_ISUID/S_ISGID bits.
	*/

	if (!(mask & AT_MODE))
	vap->va_mode = zp->z_mode;

	/*
	* Take ownership or chgrp to group we are a member of
	*/

	take_owner = (mask & AT_UID) && (vap->va_uid == crgetuid(cr));
	take_group = (mask & AT_GID) &&
	zfs_groupmember(zfsvfs, vap->va_gid, cr);

	/*
	* If both AT_UID and AT_GID are set then take_owner and
	* take_group must both be set in order to allow taking
	* ownership.
	*
	* Otherwise, send the check through secpolicy_vnode_setattr()
	*
	*/

	if (((idmask == (AT_UID\|AT_GID)) && take_owner && take_group) \|\|
	((idmask == AT_UID) && take_owner) \|\|
	((idmask == AT_GID) && take_group)) {
	if (zfs_zaccess(zp, ACE_WRITE_OWNER, 0,
	skipaclchk, cr) == 0) {
	/*
	* Remove setuid/setgid for non-privileged users
	*/
	secpolicy_setid_clear(vap, vp, cr);
	trim_mask = (mask & (AT_UID\|AT_GID));
	} else {
	need_policy = TRUE;
	}
	} else {
	need_policy = TRUE;
	}
	}

	oldva.va_mode = zp->z_mode;
	zfs_fuid_map_ids(zp, cr, &oldva.va_uid, &oldva.va_gid);
	if (mask & AT_XVATTR) {
	/*
	* Update xvattr mask to include only those attributes
	* that are actually changing.
	*
	* the bits will be restored prior to actually setting
	* the attributes so the caller thinks they were set.
	*/
	if (XVA_ISSET_REQ(xvap, XAT_APPENDONLY)) {
	if (xoap->xoa_appendonly !=
	((zp->z_pflags & ZFS_APPENDONLY) != 0)) {
	need_policy = TRUE;
	} else {
	XVA_CLR_REQ(xvap, XAT_APPENDONLY);
	XVA_SET_REQ(&tmpxvattr, XAT_APPENDONLY);
	}
	}

	if (XVA_ISSET_REQ(xvap, XAT_PROJINHERIT)) {
	if (xoap->xoa_projinherit !=
	((zp->z_pflags & ZFS_PROJINHERIT) != 0)) {
	need_policy = TRUE;
	} else {
	XVA_CLR_REQ(xvap, XAT_PROJINHERIT);
	XVA_SET_REQ(&tmpxvattr, XAT_PROJINHERIT);
	}
	}

	if (XVA_ISSET_REQ(xvap, XAT_NOUNLINK)) {
	if (xoap->xoa_nounlink !=
	((zp->z_pflags & ZFS_NOUNLINK) != 0)) {
	need_policy = TRUE;
	} else {
	XVA_CLR_REQ(xvap, XAT_NOUNLINK);
	XVA_SET_REQ(&tmpxvattr, XAT_NOUNLINK);
	}
	}

	if (XVA_ISSET_REQ(xvap, XAT_IMMUTABLE)) {
	if (xoap->xoa_immutable !=
	((zp->z_pflags & ZFS_IMMUTABLE) != 0)) {
	need_policy = TRUE;
	} else {
	XVA_CLR_REQ(xvap, XAT_IMMUTABLE);
	XVA_SET_REQ(&tmpxvattr, XAT_IMMUTABLE);
	}
	}

	if (XVA_ISSET_REQ(xvap, XAT_NODUMP)) {
	if (xoap->xoa_nodump !=
	((zp->z_pflags & ZFS_NODUMP) != 0)) {
	need_policy = TRUE;
	} else {
	XVA_CLR_REQ(xvap, XAT_NODUMP);
	XVA_SET_REQ(&tmpxvattr, XAT_NODUMP);
	}
	}

	if (XVA_ISSET_REQ(xvap, XAT_AV_MODIFIED)) {
	if (xoap->xoa_av_modified !=
	((zp->z_pflags & ZFS_AV_MODIFIED) != 0)) {
	need_policy = TRUE;
	} else {
	XVA_CLR_REQ(xvap, XAT_AV_MODIFIED);
	XVA_SET_REQ(&tmpxvattr, XAT_AV_MODIFIED);
	}
	}

	if (XVA_ISSET_REQ(xvap, XAT_AV_QUARANTINED)) {
	if ((vp->v_type != VREG &&
	xoap->xoa_av_quarantined) \|\|
	xoap->xoa_av_quarantined !=
	((zp->z_pflags & ZFS_AV_QUARANTINED) != 0)) {
	need_policy = TRUE;
	} else {
	XVA_CLR_REQ(xvap, XAT_AV_QUARANTINED);
	XVA_SET_REQ(&tmpxvattr, XAT_AV_QUARANTINED);
	}
	}

	if (XVA_ISSET_REQ(xvap, XAT_REPARSE)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EPERM));
	}

	if (need_policy == FALSE &&
	(XVA_ISSET_REQ(xvap, XAT_AV_SCANSTAMP) \|\|
	XVA_ISSET_REQ(xvap, XAT_OPAQUE))) {
	need_policy = TRUE;
	}
	}

	if (mask & AT_MODE) {
	if (zfs_zaccess(zp, ACE_WRITE_ACL, 0, skipaclchk, cr) == 0) {
	err = secpolicy_setid_setsticky_clear(vp, vap,
	&oldva, cr);
	if (err) {
	ZFS_EXIT(zfsvfs);
	return (err);
	}
	trim_mask \|= AT_MODE;
	} else {
	need_policy = TRUE;
	}
	}

	if (need_policy) {
	/*
	* If trim_mask is set then take ownership
	* has been granted or write_acl is present and user
	* has the ability to modify mode. In that case remove
	* UID\|GID and or MODE from mask so that
	* secpolicy_vnode_setattr() doesn't revoke it.
	*/

	if (trim_mask) {
	saved_mask = vap->va_mask;
	vap->va_mask &= ~trim_mask;
	if (trim_mask & AT_MODE) {
	/*
	* Save the mode, as secpolicy_vnode_setattr()
	* will overwrite it with ova.va_mode.
	*/
	saved_mode = vap->va_mode;
	}
	}
	err = secpolicy_vnode_setattr(cr, vp, vap, &oldva, flags,
	(int ()(void , int, cred_t *))zfs_zaccess_unix, zp);
	if (err) {
	ZFS_EXIT(zfsvfs);
	return (err);
	}

	if (trim_mask) {
	vap->va_mask \|= saved_mask;
	if (trim_mask & AT_MODE) {
	/*
	* Recover the mode after
	* secpolicy_vnode_setattr().
	*/
	vap->va_mode = saved_mode;
	}
	}
	}

	/*
	* secpolicy_vnode_setattr, or take ownership may have
	* changed va_mask
	*/
	mask = vap->va_mask;

	if ((mask & (AT_UID \| AT_GID)) \|\| projid != ZFS_INVALID_PROJID) {
	err = sa_lookup(zp->z_sa_hdl, SA_ZPL_XATTR(zfsvfs),
	&xattr_obj, sizeof (xattr_obj));

	if (err == 0 && xattr_obj) {
	err = zfs_zget(zp->z_zfsvfs, xattr_obj, &attrzp);
	if (err == 0) {
	err = vn_lock(ZTOV(attrzp), LK_EXCLUSIVE);
	if (err != 0)
	vrele(ZTOV(attrzp));
	}
	if (err)
	goto out2;
	}
	if (mask & AT_UID) {
	new_uid = zfs_fuid_create(zfsvfs,
	(uint64_t)vap->va_uid, cr, ZFS_OWNER, &fuidp);
	if (new_uid != zp->z_uid &&
	zfs_id_overquota(zfsvfs, DMU_USERUSED_OBJECT,
	new_uid)) {
	if (attrzp)
	vput(ZTOV(attrzp));
	err = SET_ERROR(EDQUOT);
	goto out2;
	}
	}

	if (mask & AT_GID) {
	new_gid = zfs_fuid_create(zfsvfs, (uint64_t)vap->va_gid,
	cr, ZFS_GROUP, &fuidp);
	if (new_gid != zp->z_gid &&
	zfs_id_overquota(zfsvfs, DMU_GROUPUSED_OBJECT,
	new_gid)) {
	if (attrzp)
	vput(ZTOV(attrzp));
	err = SET_ERROR(EDQUOT);
	goto out2;
	}
	}

	if (projid != ZFS_INVALID_PROJID &&
	zfs_id_overquota(zfsvfs, DMU_PROJECTUSED_OBJECT, projid)) {
	if (attrzp)
	vput(ZTOV(attrzp));
	err = SET_ERROR(EDQUOT);
	goto out2;
	}
	}
	tx = dmu_tx_create(os);

	if (mask & AT_MODE) {
	uint64_t pmode = zp->z_mode;
	uint64_t acl_obj;
	new_mode = (pmode & S_IFMT) \| (vap->va_mode & ~S_IFMT);

	if (zp->z_zfsvfs->z_acl_mode == ZFS_ACL_RESTRICTED &&
	!(zp->z_pflags & ZFS_ACL_TRIVIAL)) {
	err = SET_ERROR(EPERM);
	goto out;
	}

	if ((err = zfs_acl_chmod_setattr(zp, &aclp, new_mode)))
	goto out;

	if (!zp->z_is_sa && ((acl_obj = zfs_external_acl(zp)) != 0)) {
	/*
	* Are we upgrading ACL from old V0 format
	* to V1 format?
	*/
	if (zfsvfs->z_version >= ZPL_VERSION_FUID &&
	zfs_znode_acl_version(zp) ==
	ZFS_ACL_VERSION_INITIAL) {
	dmu_tx_hold_free(tx, acl_obj, 0,
	DMU_OBJECT_END);
	dmu_tx_hold_write(tx, DMU_NEW_OBJECT,
	0, aclp->z_acl_bytes);
	} else {
	dmu_tx_hold_write(tx, acl_obj, 0,
	aclp->z_acl_bytes);
	}
	} else if (!zp->z_is_sa && aclp->z_acl_bytes > ZFS_ACE_SPACE) {
	dmu_tx_hold_write(tx, DMU_NEW_OBJECT,
	0, aclp->z_acl_bytes);
	}
	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_TRUE);
	} else {
	if (((mask & AT_XVATTR) &&
	XVA_ISSET_REQ(xvap, XAT_AV_SCANSTAMP)) \|\|
	(projid != ZFS_INVALID_PROJID &&
	!(zp->z_pflags & ZFS_PROJID)))
	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_TRUE);
	else
	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_FALSE);
	}

	if (attrzp) {
	dmu_tx_hold_sa(tx, attrzp->z_sa_hdl, B_FALSE);
	}

	fuid_dirtied = zfsvfs->z_fuid_dirty;
	if (fuid_dirtied)
	zfs_fuid_txhold(zfsvfs, tx);

	zfs_sa_upgrade_txholds(tx, zp);

	err = dmu_tx_assign(tx, TXG_WAIT);
	if (err)
	goto out;

	count = 0;
	/*
	* Set each attribute requested.
	* We group settings according to the locks they need to acquire.
	*
	* Note: you cannot set ctime directly, although it will be
	* updated as a side-effect of calling this function.
	*/

	if (projid != ZFS_INVALID_PROJID && !(zp->z_pflags & ZFS_PROJID)) {
	/*
	* For the existed object that is upgraded from old system,
	* its on-disk layout has no slot for the project ID attribute.
	* But quota accounting logic needs to access related slots by
	* offset directly. So we need to adjust old objects' layout
	* to make the project ID to some unified and fixed offset.
	*/
	if (attrzp)
	err = sa_add_projid(attrzp->z_sa_hdl, tx, projid);
	if (err == 0)
	err = sa_add_projid(zp->z_sa_hdl, tx, projid);

	if (unlikely(err == EEXIST))
	err = 0;
	else if (err != 0)
	goto out;
	else
	projid = ZFS_INVALID_PROJID;
	}

	if (mask & (AT_UID\|AT_GID\|AT_MODE))
	mutex_enter(&zp->z_acl_lock);

	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_FLAGS(zfsvfs), NULL,
	&zp->z_pflags, sizeof (zp->z_pflags));

	if (attrzp) {
	if (mask & (AT_UID\|AT_GID\|AT_MODE))
	mutex_enter(&attrzp->z_acl_lock);
	SA_ADD_BULK_ATTR(xattr_bulk, xattr_count,
	SA_ZPL_FLAGS(zfsvfs), NULL, &attrzp->z_pflags,
	sizeof (attrzp->z_pflags));
	if (projid != ZFS_INVALID_PROJID) {
	attrzp->z_projid = projid;
	SA_ADD_BULK_ATTR(xattr_bulk, xattr_count,
	SA_ZPL_PROJID(zfsvfs), NULL, &attrzp->z_projid,
	sizeof (attrzp->z_projid));
	}
	}

	if (mask & (AT_UID\|AT_GID)) {

	if (mask & AT_UID) {
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_UID(zfsvfs), NULL,
	&new_uid, sizeof (new_uid));
	zp->z_uid = new_uid;
	if (attrzp) {
	SA_ADD_BULK_ATTR(xattr_bulk, xattr_count,
	SA_ZPL_UID(zfsvfs), NULL, &new_uid,
	sizeof (new_uid));
	attrzp->z_uid = new_uid;
	}
	}

	if (mask & AT_GID) {
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_GID(zfsvfs),
	NULL, &new_gid, sizeof (new_gid));
	zp->z_gid = new_gid;
	if (attrzp) {
	SA_ADD_BULK_ATTR(xattr_bulk, xattr_count,
	SA_ZPL_GID(zfsvfs), NULL, &new_gid,
	sizeof (new_gid));
	attrzp->z_gid = new_gid;
	}
	}
	if (!(mask & AT_MODE)) {
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_MODE(zfsvfs),
	NULL, &new_mode, sizeof (new_mode));
	new_mode = zp->z_mode;
	}
	err = zfs_acl_chown_setattr(zp);
	ASSERT(err == 0);
	if (attrzp) {
	err = zfs_acl_chown_setattr(attrzp);
	ASSERT(err == 0);
	}
	}

	if (mask & AT_MODE) {
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_MODE(zfsvfs), NULL,
	&new_mode, sizeof (new_mode));
	zp->z_mode = new_mode;
	ASSERT3U((uintptr_t)aclp, !=, 0);
	err = zfs_aclset_common(zp, aclp, cr, tx);
	ASSERT0(err);
	if (zp->z_acl_cached)
	zfs_acl_free(zp->z_acl_cached);
	zp->z_acl_cached = aclp;
	aclp = NULL;
	}


	if (mask & AT_ATIME) {
	ZFS_TIME_ENCODE(&vap->va_atime, zp->z_atime);
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_ATIME(zfsvfs), NULL,
	&zp->z_atime, sizeof (zp->z_atime));
	}

	if (mask & AT_MTIME) {
	ZFS_TIME_ENCODE(&vap->va_mtime, mtime);
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_MTIME(zfsvfs), NULL,
	mtime, sizeof (mtime));
	}

	if (projid != ZFS_INVALID_PROJID) {
	zp->z_projid = projid;
	SA_ADD_BULK_ATTR(bulk, count,
	SA_ZPL_PROJID(zfsvfs), NULL, &zp->z_projid,
	sizeof (zp->z_projid));
	}

	/* XXX - shouldn't this be done before the ATIME/MTIME checks? */
	if (mask & AT_SIZE && !(mask & AT_MTIME)) {
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_MTIME(zfsvfs),
	NULL, mtime, sizeof (mtime));
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_CTIME(zfsvfs), NULL,
	&ctime, sizeof (ctime));
	zfs_tstamp_update_setup(zp, CONTENT_MODIFIED, mtime, ctime);
	} else if (mask != 0) {
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_CTIME(zfsvfs), NULL,
	&ctime, sizeof (ctime));
	zfs_tstamp_update_setup(zp, STATE_CHANGED, mtime, ctime);
	if (attrzp) {
	SA_ADD_BULK_ATTR(xattr_bulk, xattr_count,
	SA_ZPL_CTIME(zfsvfs), NULL,
	&ctime, sizeof (ctime));
	zfs_tstamp_update_setup(attrzp, STATE_CHANGED,
	mtime, ctime);
	}
	}

	/*
	* Do this after setting timestamps to prevent timestamp
	* update from toggling bit
	*/

	if (xoap && (mask & AT_XVATTR)) {

	if (XVA_ISSET_REQ(xvap, XAT_CREATETIME))
	xoap->xoa_createtime = vap->va_birthtime;
	/*
	* restore trimmed off masks
	* so that return masks can be set for caller.
	*/

	if (XVA_ISSET_REQ(&tmpxvattr, XAT_APPENDONLY)) {
	XVA_SET_REQ(xvap, XAT_APPENDONLY);
	}
	if (XVA_ISSET_REQ(&tmpxvattr, XAT_NOUNLINK)) {
	XVA_SET_REQ(xvap, XAT_NOUNLINK);
	}
	if (XVA_ISSET_REQ(&tmpxvattr, XAT_IMMUTABLE)) {
	XVA_SET_REQ(xvap, XAT_IMMUTABLE);
	}
	if (XVA_ISSET_REQ(&tmpxvattr, XAT_NODUMP)) {
	XVA_SET_REQ(xvap, XAT_NODUMP);
	}
	if (XVA_ISSET_REQ(&tmpxvattr, XAT_AV_MODIFIED)) {
	XVA_SET_REQ(xvap, XAT_AV_MODIFIED);
	}
	if (XVA_ISSET_REQ(&tmpxvattr, XAT_AV_QUARANTINED)) {
	XVA_SET_REQ(xvap, XAT_AV_QUARANTINED);
	}
	if (XVA_ISSET_REQ(&tmpxvattr, XAT_PROJINHERIT)) {
	XVA_SET_REQ(xvap, XAT_PROJINHERIT);
	}

	if (XVA_ISSET_REQ(xvap, XAT_AV_SCANSTAMP))
	ASSERT(vp->v_type == VREG);

	zfs_xvattr_set(zp, xvap, tx);
	}

	if (fuid_dirtied)
	zfs_fuid_sync(zfsvfs, tx);

	if (mask != 0)
	zfs_log_setattr(zilog, tx, TX_SETATTR, zp, vap, mask, fuidp);

	if (mask & (AT_UID\|AT_GID\|AT_MODE))
	mutex_exit(&zp->z_acl_lock);

	if (attrzp) {
	if (mask & (AT_UID\|AT_GID\|AT_MODE))
	mutex_exit(&attrzp->z_acl_lock);
	}
	out:
	if (err == 0 && attrzp) {
	err2 = sa_bulk_update(attrzp->z_sa_hdl, xattr_bulk,
	xattr_count, tx);
	ASSERT(err2 == 0);
	}

	if (attrzp)
	vput(ZTOV(attrzp));

	if (aclp)
	zfs_acl_free(aclp);

	if (fuidp) {
	zfs_fuid_info_free(fuidp);
	fuidp = NULL;
	}

	if (err) {
	dmu_tx_abort(tx);
	} else {
	err2 = sa_bulk_update(zp->z_sa_hdl, bulk, count, tx);
	dmu_tx_commit(tx);
	}

	out2:
	if (os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, 0);

	ZFS_EXIT(zfsvfs);
	return (err);
	}

	/*
	* We acquire all but fdvp locks using non-blocking acquisitions. If we
	* fail to acquire any lock in the path we will drop all held locks,
	* acquire the new lock in a blocking fashion, and then release it and
	* restart the rename. This acquire/release step ensures that we do not
	* spin on a lock waiting for release. On error release all vnode locks
	* and decrement references the way tmpfs_rename() would do.
	*/
	static int
	zfs_rename_relock(struct vnode sdvp, struct vnode *svpp,
	struct vnode tdvp, struct vnode *tvpp,
	const struct componentname scnp, const struct componentname tcnp)
	{
	zfsvfs_t *zfsvfs;
	struct vnode nvp, svp, *tvp;
	znode_t sdzp, tdzp, szp, tzp;
	const char *snm = scnp->cn_nameptr;
	const char *tnm = tcnp->cn_nameptr;
	int error;

	VOP_UNLOCK1(tdvp);
	if (tvpp != NULL && tvpp != tdvp)
	VOP_UNLOCK1(*tvpp);

	relock:
	error = vn_lock(sdvp, LK_EXCLUSIVE);
	if (error)
	goto out;
	sdzp = VTOZ(sdvp);

	error = vn_lock(tdvp, LK_EXCLUSIVE \| LK_NOWAIT);
	if (error != 0) {
	VOP_UNLOCK1(sdvp);
	if (error != EBUSY)
	goto out;
	error = vn_lock(tdvp, LK_EXCLUSIVE);
	if (error)
	goto out;
	VOP_UNLOCK1(tdvp);
	goto relock;
	}
	tdzp = VTOZ(tdvp);

	/*
	* Before using sdzp and tdzp we must ensure that they are live.
	* As a porting legacy from illumos we have two things to worry
	* about. One is typical for FreeBSD and it is that the vnode is
	* not reclaimed (doomed). The other is that the znode is live.
	* The current code can invalidate the znode without acquiring the
	* corresponding vnode lock if the object represented by the znode
	* and vnode is no longer valid after a rollback or receive operation.
	* z_teardown_lock hidden behind ZFS_ENTER and ZFS_EXIT is the lock
	* that protects the znodes from the invalidation.
	*/
	zfsvfs = sdzp->z_zfsvfs;
	ASSERT3P(zfsvfs, ==, tdzp->z_zfsvfs);
	ZFS_ENTER(zfsvfs);

	/*
	* We can not use ZFS_VERIFY_ZP() here because it could directly return
	* bypassing the cleanup code in the case of an error.
	*/
	if (tdzp->z_sa_hdl == NULL \|\| sdzp->z_sa_hdl == NULL) {
	ZFS_EXIT(zfsvfs);
	VOP_UNLOCK1(sdvp);
	VOP_UNLOCK1(tdvp);
	error = SET_ERROR(EIO);
	goto out;
	}

	/*
	* Re-resolve svp to be certain it still exists and fetch the
	* correct vnode.
	*/
	error = zfs_dirent_lookup(sdzp, snm, &szp, ZEXISTS);
	if (error != 0) {
	/* Source entry invalid or not there. */
	ZFS_EXIT(zfsvfs);
	VOP_UNLOCK1(sdvp);
	VOP_UNLOCK1(tdvp);
	if ((scnp->cn_flags & ISDOTDOT) != 0 \|\|
	(scnp->cn_namelen == 1 && scnp->cn_nameptr[0] == '.'))
	error = SET_ERROR(EINVAL);
	goto out;
	}
	svp = ZTOV(szp);

	/*
	* Re-resolve tvp, if it disappeared we just carry on.
	*/
	error = zfs_dirent_lookup(tdzp, tnm, &tzp, 0);
	if (error != 0) {
	ZFS_EXIT(zfsvfs);
	VOP_UNLOCK1(sdvp);
	VOP_UNLOCK1(tdvp);
	vrele(svp);
	if ((tcnp->cn_flags & ISDOTDOT) != 0)
	error = SET_ERROR(EINVAL);
	goto out;
	}
	if (tzp != NULL)
	tvp = ZTOV(tzp);
	else
	tvp = NULL;

	/*
	* At present the vnode locks must be acquired before z_teardown_lock,
	* although it would be more logical to use the opposite order.
	*/
	ZFS_EXIT(zfsvfs);

	/*
	* Now try acquire locks on svp and tvp.
	*/
	nvp = svp;
	error = vn_lock(nvp, LK_EXCLUSIVE \| LK_NOWAIT);
	if (error != 0) {
	VOP_UNLOCK1(sdvp);
	VOP_UNLOCK1(tdvp);
	if (tvp != NULL)
	vrele(tvp);
	if (error != EBUSY) {
	vrele(nvp);
	goto out;
	}
	error = vn_lock(nvp, LK_EXCLUSIVE);
	if (error != 0) {
	vrele(nvp);
	goto out;
	}
	VOP_UNLOCK1(nvp);
	/*
	* Concurrent rename race.
	* XXX ?
	*/
	if (nvp == tdvp) {
	vrele(nvp);
	error = SET_ERROR(EINVAL);
	goto out;
	}
	vrele(*svpp);
	*svpp = nvp;
	goto relock;
	}
	vrele(*svpp);
	*svpp = nvp;

	if (*tvpp != NULL)
	vrele(*tvpp);
	*tvpp = NULL;
	if (tvp != NULL) {
	nvp = tvp;
	error = vn_lock(nvp, LK_EXCLUSIVE \| LK_NOWAIT);
	if (error != 0) {
	VOP_UNLOCK1(sdvp);
	VOP_UNLOCK1(tdvp);
	VOP_UNLOCK1(*svpp);
	if (error != EBUSY) {
	vrele(nvp);
	goto out;
	}
	error = vn_lock(nvp, LK_EXCLUSIVE);
	if (error != 0) {
	vrele(nvp);
	goto out;
	}
	vput(nvp);
	goto relock;
	}
	*tvpp = nvp;
	}

	return (0);

	out:
	return (error);
	}

	/*
	* Note that we must use VRELE_ASYNC in this function as it walks
	* up the directory tree and vrele may need to acquire an exclusive
	* lock if a last reference to a vnode is dropped.
	*/
	static int
	zfs_rename_check(znode_t szp, znode_t sdzp, znode_t *tdzp)
	{
	zfsvfs_t *zfsvfs;
	znode_t zp, zp1;
	uint64_t parent;
	int error;

	zfsvfs = tdzp->z_zfsvfs;
	if (tdzp == szp)
	return (SET_ERROR(EINVAL));
	if (tdzp == sdzp)
	return (0);
	if (tdzp->z_id == zfsvfs->z_root)
	return (0);
	zp = tdzp;
	for (;;) {
	ASSERT(!zp->z_unlinked);
	if ((error = sa_lookup(zp->z_sa_hdl,
	SA_ZPL_PARENT(zfsvfs), &parent, sizeof (parent))) != 0)
	break;

	if (parent == szp->z_id) {
	error = SET_ERROR(EINVAL);
	break;
	}
	if (parent == zfsvfs->z_root)
	break;
	if (parent == sdzp->z_id)
	break;

	error = zfs_zget(zfsvfs, parent, &zp1);
	if (error != 0)
	break;

	if (zp != tdzp)
	VN_RELE_ASYNC(ZTOV(zp),
	dsl_pool_zrele_taskq(
	dmu_objset_pool(zfsvfs->z_os)));
	zp = zp1;
	}

	if (error == ENOTDIR)
	panic("checkpath: .. not a directory\n");
	if (zp != tdzp)
	VN_RELE_ASYNC(ZTOV(zp),
	dsl_pool_zrele_taskq(dmu_objset_pool(zfsvfs->z_os)));
	return (error);
	}

	+#if __FreeBSD_version < 1300110
	+static void
	+cache_rename(struct vnode fdvp, struct vnode fvp, struct vnode *tdvp,
	+ struct vnode tvp, struct componentname fcnp, struct componentname *tcnp)
	+{
	+
	+ cache_purge(fvp);
	+ if (tvp != NULL)
	+ cache_purge(tvp);
	+ cache_purge_negative(tdvp);
	+}
	+#endif
	+
	/*
	* Move an entry from the provided source directory to the target
	* directory. Change the entry name as indicated.
	*
	* IN: sdvp - Source directory containing the "old entry".
	* snm - Old entry name.
	* tdvp - Target directory to contain the "new entry".
	* tnm - New entry name.
	* cr - credentials of caller.
	* ct - caller context
	* flags - case flags
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* sdvp,tdvp - ctime\|mtime updated
	*/
	/ARGSUSED/
	static int
	zfs_rename_(vnode_t sdvp, vnode_t svpp, struct componentname scnp,
	vnode_t tdvp, vnode_t tvpp, struct componentname tcnp,
	cred_t *cr, int log)
	{
	zfsvfs_t *zfsvfs;
	znode_t sdzp, tdzp, szp, tzp;
	zilog_t *zilog = NULL;
	dmu_tx_t *tx;
	- char *snm = scnp->cn_nameptr;
	- char *tnm = tcnp->cn_nameptr;
	+ const char *snm = scnp->cn_nameptr;
	+ const char *tnm = tcnp->cn_nameptr;
	int error = 0;
	bool want_seqc_end __maybe_unused = false;

	/* Reject renames across filesystems. */
	if ((*svpp)->v_mount != tdvp->v_mount \|\|
	((tvpp) != NULL && (svpp)->v_mount != (*tvpp)->v_mount)) {
	error = SET_ERROR(EXDEV);
	goto out;
	}

	if (zfsctl_is_node(tdvp)) {
	error = SET_ERROR(EXDEV);
	goto out;
	}

	/*
	* Lock all four vnodes to ensure safety and semantics of renaming.
	*/
	error = zfs_rename_relock(sdvp, svpp, tdvp, tvpp, scnp, tcnp);
	if (error != 0) {
	/* no vnodes are locked in the case of error here */
	return (error);
	}

	tdzp = VTOZ(tdvp);
	sdzp = VTOZ(sdvp);
	zfsvfs = tdzp->z_zfsvfs;
	zilog = zfsvfs->z_log;

	/*
	* After we re-enter ZFS_ENTER() we will have to revalidate all
	* znodes involved.
	*/
	ZFS_ENTER(zfsvfs);

	if (zfsvfs->z_utf8 && u8_validate(tnm,
	strlen(tnm), NULL, U8_VALIDATE_ENTIRE, &error) < 0) {
	error = SET_ERROR(EILSEQ);
	goto unlockout;
	}

	/* If source and target are the same file, there is nothing to do. */
	if ((svpp) == (tvpp)) {
	error = 0;
	goto unlockout;
	}

	if (((svpp)->v_type == VDIR && (svpp)->v_mountedhere != NULL) \|\|
	((tvpp) != NULL && (tvpp)->v_type == VDIR &&
	(*tvpp)->v_mountedhere != NULL)) {
	error = SET_ERROR(EXDEV);
	goto unlockout;
	}

	/*
	* We can not use ZFS_VERIFY_ZP() here because it could directly return
	* bypassing the cleanup code in the case of an error.
	*/
	if (tdzp->z_sa_hdl == NULL \|\| sdzp->z_sa_hdl == NULL) {
	error = SET_ERROR(EIO);
	goto unlockout;
	}

	szp = VTOZ(*svpp);
	tzp = tvpp == NULL ? NULL : VTOZ(tvpp);
	if (szp->z_sa_hdl == NULL \|\| (tzp != NULL && tzp->z_sa_hdl == NULL)) {
	error = SET_ERROR(EIO);
	goto unlockout;
	}

	/*
	* This is to prevent the creation of links into attribute space
	* by renaming a linked file into/outof an attribute directory.
	* See the comment in zfs_link() for why this is considered bad.
	*/
	if ((tdzp->z_pflags & ZFS_XATTR) != (sdzp->z_pflags & ZFS_XATTR)) {
	error = SET_ERROR(EINVAL);
	goto unlockout;
	}

	/*
	* If we are using project inheritance, means if the directory has
	* ZFS_PROJINHERIT set, then its descendant directories will inherit
	* not only the project ID, but also the ZFS_PROJINHERIT flag. Under
	* such case, we only allow renames into our tree when the project
	* IDs are the same.
	*/
	if (tdzp->z_pflags & ZFS_PROJINHERIT &&
	tdzp->z_projid != szp->z_projid) {
	error = SET_ERROR(EXDEV);
	goto unlockout;
	}

	/*
	* Must have write access at the source to remove the old entry
	* and write access at the target to create the new entry.
	* Note that if target and source are the same, this can be
	* done in a single check.
	*/
	if ((error = zfs_zaccess_rename(sdzp, szp, tdzp, tzp, cr)))
	goto unlockout;

	if ((*svpp)->v_type == VDIR) {
	/*
	* Avoid ".", "..", and aliases of "." for obvious reasons.
	*/
	if ((scnp->cn_namelen == 1 && scnp->cn_nameptr[0] == '.') \|\|
	sdzp == szp \|\|
	(scnp->cn_flags \| tcnp->cn_flags) & ISDOTDOT) {
	error = EINVAL;
	goto unlockout;
	}

	/*
	* Check to make sure rename is valid.
	* Can't do a move like this: /usr/a/b to /usr/a/b/c/d
	*/
	if ((error = zfs_rename_check(szp, sdzp, tdzp)))
	goto unlockout;
	}

	/*
	* Does target exist?
	*/
	if (tzp) {
	/*
	* Source and target must be the same type.
	*/
	if ((*svpp)->v_type == VDIR) {
	if ((*tvpp)->v_type != VDIR) {
	error = SET_ERROR(ENOTDIR);
	goto unlockout;
	} else {
	cache_purge(tdvp);
	if (sdvp != tdvp)
	cache_purge(sdvp);
	}
	} else {
	if ((*tvpp)->v_type == VDIR) {
	error = SET_ERROR(EISDIR);
	goto unlockout;
	}
	}
	}

	vn_seqc_write_begin(*svpp);
	vn_seqc_write_begin(sdvp);
	if (*tvpp != NULL)
	vn_seqc_write_begin(*tvpp);
	if (tdvp != *tvpp)
	vn_seqc_write_begin(tdvp);
	#if __FreeBSD_version >= 1300102
	want_seqc_end = true;
	#endif
	vnevent_rename_src(*svpp, sdvp, scnp->cn_nameptr, ct);
	if (tzp)
	vnevent_rename_dest(*tvpp, tdvp, tnm, ct);

	/*
	* notify the target directory if it is not the same
	* as source directory.
	*/
	if (tdvp != sdvp) {
	vnevent_rename_dest_dir(tdvp, ct);
	}

	tx = dmu_tx_create(zfsvfs->z_os);
	dmu_tx_hold_sa(tx, szp->z_sa_hdl, B_FALSE);
	dmu_tx_hold_sa(tx, sdzp->z_sa_hdl, B_FALSE);
	dmu_tx_hold_zap(tx, sdzp->z_id, FALSE, snm);
	dmu_tx_hold_zap(tx, tdzp->z_id, TRUE, tnm);
	if (sdzp != tdzp) {
	dmu_tx_hold_sa(tx, tdzp->z_sa_hdl, B_FALSE);
	zfs_sa_upgrade_txholds(tx, tdzp);
	}
	if (tzp) {
	dmu_tx_hold_sa(tx, tzp->z_sa_hdl, B_FALSE);
	zfs_sa_upgrade_txholds(tx, tzp);
	}

	zfs_sa_upgrade_txholds(tx, szp);
	dmu_tx_hold_zap(tx, zfsvfs->z_unlinkedobj, FALSE, NULL);
	error = dmu_tx_assign(tx, TXG_WAIT);
	if (error) {
	dmu_tx_abort(tx);
	goto unlockout;
	}


	if (tzp) /* Attempt to remove the existing target */
	error = zfs_link_destroy(tdzp, tnm, tzp, tx, 0, NULL);

	if (error == 0) {
	error = zfs_link_create(tdzp, tnm, szp, tx, ZRENAMING);
	if (error == 0) {
	szp->z_pflags \|= ZFS_AV_MODIFIED;

	error = sa_update(szp->z_sa_hdl, SA_ZPL_FLAGS(zfsvfs),
	(void *)&szp->z_pflags, sizeof (uint64_t), tx);
	ASSERT0(error);

	error = zfs_link_destroy(sdzp, snm, szp, tx, ZRENAMING,
	NULL);
	if (error == 0) {
	zfs_log_rename(zilog, tx, TX_RENAME, sdzp,
	snm, tdzp, tnm, szp);

	/*
	* Update path information for the target vnode
	*/
	vn_renamepath(tdvp, *svpp, tnm, strlen(tnm));
	} else {
	/*
	* At this point, we have successfully created
	* the target name, but have failed to remove
	* the source name. Since the create was done
	* with the ZRENAMING flag, there are
	* complications; for one, the link count is
	* wrong. The easiest way to deal with this
	* is to remove the newly created target, and
	* return the original error. This must
	* succeed; fortunately, it is very unlikely to
	* fail, since we just created it.
	*/
	VERIFY3U(zfs_link_destroy(tdzp, tnm, szp, tx,
	ZRENAMING, NULL), ==, 0);
	}
	}
	if (error == 0) {
	- cache_purge(*svpp);
	- if (*tvpp != NULL)
	- cache_purge(*tvpp);
	- cache_purge_negative(tdvp);
	+ cache_rename(sdvp, svpp, tdvp, tvpp, scnp, tcnp);
	}
	}

	dmu_tx_commit(tx);

	unlockout: /* all 4 vnodes are locked, ZFS_ENTER called */
	ZFS_EXIT(zfsvfs);
	if (want_seqc_end) {
	vn_seqc_write_end(*svpp);
	vn_seqc_write_end(sdvp);
	if (*tvpp != NULL)
	vn_seqc_write_end(*tvpp);
	if (tdvp != *tvpp)
	vn_seqc_write_end(tdvp);
	want_seqc_end = false;
	}
	VOP_UNLOCK1(*svpp);
	VOP_UNLOCK1(sdvp);

	out: /* original two vnodes are locked */
	MPASS(!want_seqc_end);
	if (error == 0 && zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, 0);

	if (*tvpp != NULL)
	VOP_UNLOCK1(*tvpp);
	if (tdvp != *tvpp)
	VOP_UNLOCK1(tdvp);
	return (error);
	}

	int
	-zfs_rename(znode_t sdzp, char sname, znode_t tdzp, char tname,
	+zfs_rename(znode_t sdzp, const char sname, znode_t tdzp, const char tname,
	cred_t *cr, int flags)
	{
	struct componentname scn, tcn;
	vnode_t sdvp, tdvp;
	vnode_t svp, tvp;
	int error;
	svp = tvp = NULL;

	sdvp = ZTOV(sdzp);
	tdvp = ZTOV(tdzp);
	error = zfs_lookup_internal(sdzp, sname, &svp, &scn, DELETE);
	if (sdzp->z_zfsvfs->z_replay == B_FALSE)
	VOP_UNLOCK1(sdvp);
	if (error != 0)
	goto fail;
	VOP_UNLOCK1(svp);

	vn_lock(tdvp, LK_EXCLUSIVE \| LK_RETRY);
	error = zfs_lookup_internal(tdzp, tname, &tvp, &tcn, RENAME);
	if (error == EJUSTRETURN)
	tvp = NULL;
	else if (error != 0) {
	VOP_UNLOCK1(tdvp);
	goto fail;
	}

	error = zfs_rename_(sdvp, &svp, &scn, tdvp, &tvp, &tcn, cr, 0);
	fail:
	if (svp != NULL)
	vrele(svp);
	if (tvp != NULL)
	vrele(tvp);

	return (error);
	}

	/*
	* Insert the indicated symbolic reference entry into the directory.
	*
	* IN: dvp - Directory to contain new symbolic link.
	* link - Name for new symlink entry.
	* vap - Attributes of new entry.
	* cr - credentials of caller.
	* ct - caller context
	* flags - case flags
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* dvp - ctime\|mtime updated
	*/
	/ARGSUSED/
	int
	zfs_symlink(znode_t dzp, const char name, vattr_t *vap,
	const char link, znode_t zpp, cred_t cr, int flags)
	{
	znode_t *zp;
	dmu_tx_t *tx;
	zfsvfs_t *zfsvfs = dzp->z_zfsvfs;
	zilog_t *zilog;
	uint64_t len = strlen(link);
	int error;
	zfs_acl_ids_t acl_ids;
	boolean_t fuid_dirtied;
	uint64_t txtype = TX_SYMLINK;

	ASSERT(vap->va_type == VLNK);

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(dzp);
	zilog = zfsvfs->z_log;

	if (zfsvfs->z_utf8 && u8_validate(name, strlen(name),
	NULL, U8_VALIDATE_ENTIRE, &error) < 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EILSEQ));
	}

	if (len > MAXPATHLEN) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(ENAMETOOLONG));
	}

	if ((error = zfs_acl_ids_create(dzp, 0,
	vap, cr, NULL, &acl_ids)) != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Attempt to lock directory; fail if entry already exists.
	*/
	error = zfs_dirent_lookup(dzp, name, &zp, ZNEW);
	if (error) {
	zfs_acl_ids_free(&acl_ids);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	if ((error = zfs_zaccess(dzp, ACE_ADD_FILE, 0, B_FALSE, cr))) {
	zfs_acl_ids_free(&acl_ids);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	if (zfs_acl_ids_overquota(zfsvfs, &acl_ids,
	0 /* projid */)) {
	zfs_acl_ids_free(&acl_ids);
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EDQUOT));
	}

	getnewvnode_reserve_();
	tx = dmu_tx_create(zfsvfs->z_os);
	fuid_dirtied = zfsvfs->z_fuid_dirty;
	dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0, MAX(1, len));
	dmu_tx_hold_zap(tx, dzp->z_id, TRUE, name);
	dmu_tx_hold_sa_create(tx, acl_ids.z_aclp->z_acl_bytes +
	ZFS_SA_BASE_ATTR_SIZE + len);
	dmu_tx_hold_sa(tx, dzp->z_sa_hdl, B_FALSE);
	if (!zfsvfs->z_use_sa && acl_ids.z_aclp->z_acl_bytes > ZFS_ACE_SPACE) {
	dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0,
	acl_ids.z_aclp->z_acl_bytes);
	}
	if (fuid_dirtied)
	zfs_fuid_txhold(zfsvfs, tx);
	error = dmu_tx_assign(tx, TXG_WAIT);
	if (error) {
	zfs_acl_ids_free(&acl_ids);
	dmu_tx_abort(tx);
	getnewvnode_drop_reserve();
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Create a new object for the symlink.
	* for version 4 ZPL datsets the symlink will be an SA attribute
	*/
	zfs_mknode(dzp, vap, tx, cr, 0, &zp, &acl_ids);

	if (fuid_dirtied)
	zfs_fuid_sync(zfsvfs, tx);

	if (zp->z_is_sa)
	error = sa_update(zp->z_sa_hdl, SA_ZPL_SYMLINK(zfsvfs),
	__DECONST(void *, link), len, tx);
	else
	zfs_sa_symlink(zp, __DECONST(char *, link), len, tx);

	zp->z_size = len;
	(void) sa_update(zp->z_sa_hdl, SA_ZPL_SIZE(zfsvfs),
	&zp->z_size, sizeof (zp->z_size), tx);
	/*
	* Insert the new object into the directory.
	*/
	(void) zfs_link_create(dzp, name, zp, tx, ZNEW);

	- zfs_log_symlink(zilog, tx, txtype, dzp, zp,
	- __DECONST(char , name), __DECONST(char , link));
	+ zfs_log_symlink(zilog, tx, txtype, dzp, zp, name, link);
	*zpp = zp;

	zfs_acl_ids_free(&acl_ids);

	dmu_tx_commit(tx);

	getnewvnode_drop_reserve();

	if (zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, 0);

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Return, in the buffer contained in the provided uio structure,
	* the symbolic path referred to by vp.
	*
	* IN: vp - vnode of symbolic link.
	* uio - structure to contain the link path.
	* cr - credentials of caller.
	* ct - caller context
	*
	* OUT: uio - structure containing the link path.
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* vp - atime updated
	*/
	/* ARGSUSED */
	static int
	zfs_readlink(vnode_t vp, uio_t uio, cred_t cr, caller_context_t ct)
	{
	znode_t *zp = VTOZ(vp);
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;
	int error;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	if (zp->z_is_sa)
	error = sa_lookup_uio(zp->z_sa_hdl,
	SA_ZPL_SYMLINK(zfsvfs), uio);
	else
	error = zfs_sa_readlink(zp, uio);

	ZFS_ACCESSTIME_STAMP(zfsvfs, zp);

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Insert a new entry into directory tdvp referencing svp.
	*
	* IN: tdvp - Directory to contain new entry.
	* svp - vnode of new entry.
	* name - name of new entry.
	* cr - credentials of caller.
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* tdvp - ctime\|mtime updated
	* svp - ctime updated
	*/
	/* ARGSUSED */
	int
	-zfs_link(znode_t tdzp, znode_t szp, char name, cred_t cr,
	+zfs_link(znode_t tdzp, znode_t szp, const char name, cred_t cr,
	int flags)
	{
	znode_t *tzp;
	zfsvfs_t *zfsvfs = tdzp->z_zfsvfs;
	zilog_t *zilog;
	dmu_tx_t *tx;
	int error;
	uint64_t parent;
	uid_t owner;

	ASSERT(ZTOV(tdzp)->v_type == VDIR);

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(tdzp);
	zilog = zfsvfs->z_log;

	/*
	* POSIX dictates that we return EPERM here.
	* Better choices include ENOTSUP or EISDIR.
	*/
	if (ZTOV(szp)->v_type == VDIR) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EPERM));
	}

	ZFS_VERIFY_ZP(szp);

	/*
	* If we are using project inheritance, means if the directory has
	* ZFS_PROJINHERIT set, then its descendant directories will inherit
	* not only the project ID, but also the ZFS_PROJINHERIT flag. Under
	* such case, we only allow hard link creation in our tree when the
	* project IDs are the same.
	*/
	if (tdzp->z_pflags & ZFS_PROJINHERIT &&
	tdzp->z_projid != szp->z_projid) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EXDEV));
	}

	if (szp->z_pflags & (ZFS_APPENDONLY \|
	ZFS_IMMUTABLE \| ZFS_READONLY)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EPERM));
	}

	/* Prevent links to .zfs/shares files */

	if ((error = sa_lookup(szp->z_sa_hdl, SA_ZPL_PARENT(zfsvfs),
	&parent, sizeof (uint64_t))) != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	if (parent == zfsvfs->z_shares_dir) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EPERM));
	}

	if (zfsvfs->z_utf8 && u8_validate(name,
	strlen(name), NULL, U8_VALIDATE_ENTIRE, &error) < 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EILSEQ));
	}

	/*
	* We do not support links between attributes and non-attributes
	* because of the potential security risk of creating links
	* into "normal" file space in order to circumvent restrictions
	* imposed in attribute space.
	*/
	if ((szp->z_pflags & ZFS_XATTR) != (tdzp->z_pflags & ZFS_XATTR)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}


	owner = zfs_fuid_map_id(zfsvfs, szp->z_uid, cr, ZFS_OWNER);
	if (owner != crgetuid(cr) && secpolicy_basic_link(ZTOV(szp), cr) != 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EPERM));
	}

	if ((error = zfs_zaccess(tdzp, ACE_ADD_FILE, 0, B_FALSE, cr))) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Attempt to lock directory; fail if entry already exists.
	*/
	error = zfs_dirent_lookup(tdzp, name, &tzp, ZNEW);
	if (error) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	tx = dmu_tx_create(zfsvfs->z_os);
	dmu_tx_hold_sa(tx, szp->z_sa_hdl, B_FALSE);
	dmu_tx_hold_zap(tx, tdzp->z_id, TRUE, name);
	zfs_sa_upgrade_txholds(tx, szp);
	zfs_sa_upgrade_txholds(tx, tdzp);
	error = dmu_tx_assign(tx, TXG_WAIT);
	if (error) {
	dmu_tx_abort(tx);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	error = zfs_link_create(tdzp, name, szp, tx, 0);

	if (error == 0) {
	uint64_t txtype = TX_LINK;
	zfs_log_link(zilog, tx, txtype, tdzp, szp, name);
	}

	dmu_tx_commit(tx);

	if (error == 0) {
	vnevent_link(ZTOV(szp), ct);
	}

	if (zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, 0);

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Free or allocate space in a file. Currently, this function only
	* supports the `F_FREESP' command. However, this command is somewhat
	* misnamed, as its functionality includes the ability to allocate as
	* well as free space.
	*
	* IN: ip - inode of file to free data in.
	* cmd - action to take (only F_FREESP supported).
	* bfp - section of file to free/alloc.
	* flag - current file open mode flags.
	* offset - current file offset.
	* cr - credentials of caller.
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* ip - ctime\|mtime updated
	*/
	/* ARGSUSED */
	int
	zfs_space(znode_t zp, int cmd, flock64_t bfp, int flag,
	offset_t offset, cred_t *cr)
	{
	zfsvfs_t *zfsvfs = ZTOZSB(zp);
	uint64_t off, len;
	int error;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	if (cmd != F_FREESP) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	/*
	* Callers might not be able to detect properly that we are read-only,
	* so check it explicitly here.
	*/
	if (zfs_is_readonly(zfsvfs)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EROFS));
	}

	if (bfp->l_len < 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	/*
	* Permissions aren't checked on Solaris because on this OS
	* zfs_space() can only be called with an opened file handle.
	* On Linux we can get here through truncate_range() which
	* operates directly on inodes, so we need to check access rights.
	*/
	if ((error = zfs_zaccess(zp, ACE_WRITE_DATA, 0, B_FALSE, cr))) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	off = bfp->l_start;
	len = bfp->l_len; /* 0 means from off to end of file */

	error = zfs_freesp(zp, off, len, flag, TRUE);

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/ARGSUSED/
	static void
	zfs_inactive(vnode_t vp, cred_t cr, caller_context_t *ct)
	{
	znode_t *zp = VTOZ(vp);
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;
	int error;

	ZFS_RLOCK_TEARDOWN_INACTIVE(zfsvfs);
	if (zp->z_sa_hdl == NULL) {
	/*
	* The fs has been unmounted, or we did a
	* suspend/resume and this file no longer exists.
	*/
	ZFS_RUNLOCK_TEARDOWN_INACTIVE(zfsvfs);
	vrecycle(vp);
	return;
	}

	if (zp->z_unlinked) {
	/*
	* Fast path to recycle a vnode of a removed file.
	*/
	ZFS_RUNLOCK_TEARDOWN_INACTIVE(zfsvfs);
	vrecycle(vp);
	return;
	}

	if (zp->z_atime_dirty && zp->z_unlinked == 0) {
	dmu_tx_t *tx = dmu_tx_create(zfsvfs->z_os);

	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_FALSE);
	zfs_sa_upgrade_txholds(tx, zp);
	error = dmu_tx_assign(tx, TXG_WAIT);
	if (error) {
	dmu_tx_abort(tx);
	} else {
	(void) sa_update(zp->z_sa_hdl, SA_ZPL_ATIME(zfsvfs),
	(void *)&zp->z_atime, sizeof (zp->z_atime), tx);
	zp->z_atime_dirty = 0;
	dmu_tx_commit(tx);
	}
	}
	ZFS_RUNLOCK_TEARDOWN_INACTIVE(zfsvfs);
	}


	CTASSERT(sizeof (struct zfid_short) <= sizeof (struct fid));
	CTASSERT(sizeof (struct zfid_long) <= sizeof (struct fid));

	/ARGSUSED/
	static int
	zfs_fid(vnode_t vp, fid_t fidp, caller_context_t *ct)
	{
	znode_t *zp = VTOZ(vp);
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;
	uint32_t gen;
	uint64_t gen64;
	uint64_t object = zp->z_id;
	zfid_short_t *zfid;
	int size, i, error;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	if ((error = sa_lookup(zp->z_sa_hdl, SA_ZPL_GEN(zfsvfs),
	&gen64, sizeof (uint64_t))) != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	gen = (uint32_t)gen64;

	size = (zfsvfs->z_parent != zfsvfs) ? LONG_FID_LEN : SHORT_FID_LEN;
	fidp->fid_len = size;

	zfid = (zfid_short_t *)fidp;

	zfid->zf_len = size;

	for (i = 0; i < sizeof (zfid->zf_object); i++)
	zfid->zf_object[i] = (uint8_t)(object >> (8 * i));

	/* Must have a non-zero generation number to distinguish from .zfs */
	if (gen == 0)
	gen = 1;
	for (i = 0; i < sizeof (zfid->zf_gen); i++)
	zfid->zf_gen[i] = (uint8_t)(gen >> (8 * i));

	if (size == LONG_FID_LEN) {
	uint64_t objsetid = dmu_objset_id(zfsvfs->z_os);
	zfid_long_t *zlfid;

	zlfid = (zfid_long_t *)fidp;

	for (i = 0; i < sizeof (zlfid->zf_setid); i++)
	zlfid->zf_setid[i] = (uint8_t)(objsetid >> (8 * i));

	/* XXX - this should be the generation number for the objset */
	for (i = 0; i < sizeof (zlfid->zf_setgen); i++)
	zlfid->zf_setgen[i] = 0;
	}

	ZFS_EXIT(zfsvfs);
	return (0);
	}

	static int
	zfs_pathconf(vnode_t vp, int cmd, ulong_t valp, cred_t *cr,
	caller_context_t *ct)
	{
	+ znode_t *zp;
	+ zfsvfs_t *zfsvfs;

	switch (cmd) {
	case _PC_LINK_MAX:
	*valp = MIN(LONG_MAX, ZFS_LINK_MAX);
	return (0);

	case _PC_FILESIZEBITS:
	*valp = 64;
	return (0);
	case _PC_MIN_HOLE_SIZE:
	*valp = (int)SPA_MINBLOCKSIZE;
	return (0);
	case _PC_ACL_EXTENDED:
	+#if 0 /* POSIX ACLs are not implemented for ZFS on FreeBSD yet. */
	+ zp = VTOZ(vp);
	+ zfsvfs = zp->z_zfsvfs;
	+ ZFS_ENTER(zfsvfs);
	+ ZFS_VERIFY_ZP(zp);
	+ *valp = zfsvfs->z_acl_type == ZFSACLTYPE_POSIX ? 1 : 0;
	+ ZFS_EXIT(zfsvfs);
	+#else
	*valp = 0;
	+#endif
	return (0);

	case _PC_ACL_NFS4:
	- *valp = 1;
	+ zp = VTOZ(vp);
	+ zfsvfs = zp->z_zfsvfs;
	+ ZFS_ENTER(zfsvfs);
	+ ZFS_VERIFY_ZP(zp);
	+ *valp = zfsvfs->z_acl_type == ZFS_ACLTYPE_NFSV4 ? 1 : 0;
	+ ZFS_EXIT(zfsvfs);
	return (0);

	case _PC_ACL_PATH_MAX:
	*valp = ACL_MAX_ENTRIES;
	return (0);

	default:
	return (EOPNOTSUPP);
	}
	}

	/ARGSUSED/
	static int
	zfs_getsecattr(vnode_t vp, vsecattr_t vsecp, int flag, cred_t *cr,
	caller_context_t *ct)
	{
	znode_t *zp = VTOZ(vp);
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;
	int error;
	boolean_t skipaclchk = (flag & ATTR_NOACLCHECK) ? B_TRUE : B_FALSE;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);
	error = zfs_getacl(zp, vsecp, skipaclchk, cr);
	ZFS_EXIT(zfsvfs);

	return (error);
	}

	/ARGSUSED/
	int
	zfs_setsecattr(znode_t zp, vsecattr_t vsecp, int flag, cred_t *cr)
	{
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;
	int error;
	boolean_t skipaclchk = (flag & ATTR_NOACLCHECK) ? B_TRUE : B_FALSE;
	zilog_t *zilog = zfsvfs->z_log;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	error = zfs_setacl(zp, vsecp, skipaclchk, cr);

	if (zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, 0);

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	static int
	zfs_getpages(struct vnode vp, vm_page_t ma, int count, int *rbehind,
	int *rahead)
	{
	znode_t *zp = VTOZ(vp);
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;
	objset_t *os = zp->z_zfsvfs->z_os;
	zfs_locked_range_t *lr;
	vm_object_t object;
	off_t start, end, obj_size;
	uint_t blksz;
	int pgsin_b, pgsin_a;
	int error;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	start = IDX_TO_OFF(ma[0]->pindex);
	end = IDX_TO_OFF(ma[count - 1]->pindex + 1);

	/*
	* Lock a range covering all required and optional pages.
	* Note that we need to handle the case of the block size growing.
	*/
	for (;;) {
	blksz = zp->z_blksz;
	lr = zfs_rangelock_tryenter(&zp->z_rangelock,
	rounddown(start, blksz),
	roundup(end, blksz) - rounddown(start, blksz), RL_READER);
	if (lr == NULL) {
	if (rahead != NULL) {
	*rahead = 0;
	rahead = NULL;
	}
	if (rbehind != NULL) {
	*rbehind = 0;
	rbehind = NULL;
	}
	break;
	}
	if (blksz == zp->z_blksz)
	break;
	zfs_rangelock_exit(lr);
	}

	object = ma[0]->object;
	zfs_vmobject_wlock(object);
	obj_size = object->un_pager.vnp.vnp_size;
	zfs_vmobject_wunlock(object);
	if (IDX_TO_OFF(ma[count - 1]->pindex) >= obj_size) {
	if (lr != NULL)
	zfs_rangelock_exit(lr);
	ZFS_EXIT(zfsvfs);
	return (zfs_vm_pagerret_bad);
	}

	pgsin_b = 0;
	if (rbehind != NULL) {
	pgsin_b = OFF_TO_IDX(start - rounddown(start, blksz));
	pgsin_b = MIN(*rbehind, pgsin_b);
	}

	pgsin_a = 0;
	if (rahead != NULL) {
	pgsin_a = OFF_TO_IDX(roundup(end, blksz) - end);
	if (end + IDX_TO_OFF(pgsin_a) >= obj_size)
	pgsin_a = OFF_TO_IDX(round_page(obj_size) - end);
	pgsin_a = MIN(*rahead, pgsin_a);
	}

	/*
	* NB: we need to pass the exact byte size of the data that we expect
	* to read after accounting for the file size. This is required because
	* ZFS will panic if we request DMU to read beyond the end of the last
	* allocated block.
	*/
	error = dmu_read_pages(os, zp->z_id, ma, count, &pgsin_b, &pgsin_a,
	MIN(end, obj_size) - (end - PAGE_SIZE));

	if (lr != NULL)
	zfs_rangelock_exit(lr);
	ZFS_ACCESSTIME_STAMP(zfsvfs, zp);
	ZFS_EXIT(zfsvfs);

	if (error != 0)
	return (zfs_vm_pagerret_error);

	VM_CNT_INC(v_vnodein);
	VM_CNT_ADD(v_vnodepgsin, count + pgsin_b + pgsin_a);
	if (rbehind != NULL)
	*rbehind = pgsin_b;
	if (rahead != NULL)
	*rahead = pgsin_a;
	return (zfs_vm_pagerret_ok);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_getpages_args {
	struct vnode *a_vp;
	vm_page_t *a_m;
	int a_count;
	int *a_rbehind;
	int *a_rahead;
	};
	#endif

	static int
	zfs_freebsd_getpages(struct vop_getpages_args *ap)
	{

	return (zfs_getpages(ap->a_vp, ap->a_m, ap->a_count, ap->a_rbehind,
	ap->a_rahead));
	}

	static int
	zfs_putpages(struct vnode vp, vm_page_t ma, size_t len, int flags,
	int *rtvals)
	{
	znode_t *zp = VTOZ(vp);
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;
	zfs_locked_range_t *lr;
	dmu_tx_t *tx;
	struct sf_buf *sf;
	vm_object_t object;
	vm_page_t m;
	caddr_t va;
	size_t tocopy;
	size_t lo_len;
	vm_ooffset_t lo_off;
	vm_ooffset_t off;
	uint_t blksz;
	int ncount;
	int pcount;
	int err;
	int i;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	object = vp->v_object;
	pcount = btoc(len);
	ncount = pcount;

	KASSERT(ma[0]->object == object, ("mismatching object"));
	KASSERT(len > 0 && (len & PAGE_MASK) == 0, ("unexpected length"));

	for (i = 0; i < pcount; i++)
	rtvals[i] = zfs_vm_pagerret_error;

	off = IDX_TO_OFF(ma[0]->pindex);
	blksz = zp->z_blksz;
	lo_off = rounddown(off, blksz);
	lo_len = roundup(len + (off - lo_off), blksz);
	lr = zfs_rangelock_enter(&zp->z_rangelock, lo_off, lo_len, RL_WRITER);

	zfs_vmobject_wlock(object);
	if (len + off > object->un_pager.vnp.vnp_size) {
	if (object->un_pager.vnp.vnp_size > off) {
	int pgoff;

	len = object->un_pager.vnp.vnp_size - off;
	ncount = btoc(len);
	if ((pgoff = (int)len & PAGE_MASK) != 0) {
	/*
	* If the object is locked and the following
	* conditions hold, then the page's dirty
	* field cannot be concurrently changed by a
	* pmap operation.
	*/
	m = ma[ncount - 1];
	vm_page_assert_sbusied(m);
	KASSERT(!pmap_page_is_write_mapped(m),
	("zfs_putpages: page %p is not read-only",
	m));
	vm_page_clear_dirty(m, pgoff, PAGE_SIZE -
	pgoff);
	}
	} else {
	len = 0;
	ncount = 0;
	}
	if (ncount < pcount) {
	for (i = ncount; i < pcount; i++) {
	rtvals[i] = zfs_vm_pagerret_bad;
	}
	}
	}
	zfs_vmobject_wunlock(object);

	if (ncount == 0)
	goto out;

	if (zfs_id_overblockquota(zfsvfs, DMU_USERUSED_OBJECT, zp->z_uid) \|\|
	zfs_id_overblockquota(zfsvfs, DMU_GROUPUSED_OBJECT, zp->z_gid) \|\|
	(zp->z_projid != ZFS_DEFAULT_PROJID &&
	zfs_id_overblockquota(zfsvfs, DMU_PROJECTUSED_OBJECT,
	zp->z_projid))) {
	goto out;
	}

	tx = dmu_tx_create(zfsvfs->z_os);
	dmu_tx_hold_write(tx, zp->z_id, off, len);

	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_FALSE);
	zfs_sa_upgrade_txholds(tx, zp);
	err = dmu_tx_assign(tx, TXG_WAIT);
	if (err != 0) {
	dmu_tx_abort(tx);
	goto out;
	}

	if (zp->z_blksz < PAGE_SIZE) {
	for (i = 0; len > 0; off += tocopy, len -= tocopy, i++) {
	tocopy = len > PAGE_SIZE ? PAGE_SIZE : len;
	va = zfs_map_page(ma[i], &sf);
	dmu_write(zfsvfs->z_os, zp->z_id, off, tocopy, va, tx);
	zfs_unmap_page(sf);
	}
	} else {
	err = dmu_write_pages(zfsvfs->z_os, zp->z_id, off, len, ma, tx);
	}

	if (err == 0) {
	uint64_t mtime[2], ctime[2];
	sa_bulk_attr_t bulk[3];
	int count = 0;

	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_MTIME(zfsvfs), NULL,
	&mtime, 16);
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_CTIME(zfsvfs), NULL,
	&ctime, 16);
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_FLAGS(zfsvfs), NULL,
	&zp->z_pflags, 8);
	zfs_tstamp_update_setup(zp, CONTENT_MODIFIED, mtime, ctime);
	err = sa_bulk_update(zp->z_sa_hdl, bulk, count, tx);
	ASSERT0(err);
	/*
	* XXX we should be passing a callback to undirty
	* but that would make the locking messier
	*/
	zfs_log_write(zfsvfs->z_log, tx, TX_WRITE, zp, off,
	len, 0, NULL, NULL);

	zfs_vmobject_wlock(object);
	for (i = 0; i < ncount; i++) {
	rtvals[i] = zfs_vm_pagerret_ok;
	vm_page_undirty(ma[i]);
	}
	zfs_vmobject_wunlock(object);
	VM_CNT_INC(v_vnodeout);
	VM_CNT_ADD(v_vnodepgsout, ncount);
	}
	dmu_tx_commit(tx);

	out:
	zfs_rangelock_exit(lr);
	if ((flags & (zfs_vm_pagerput_sync \| zfs_vm_pagerput_inval)) != 0 \|\|
	zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zfsvfs->z_log, zp->z_id);
	ZFS_EXIT(zfsvfs);
	return (rtvals[0]);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_putpages_args {
	struct vnode *a_vp;
	vm_page_t *a_m;
	int a_count;
	int a_sync;
	int *a_rtvals;
	};
	#endif

	static int
	zfs_freebsd_putpages(struct vop_putpages_args *ap)
	{

	return (zfs_putpages(ap->a_vp, ap->a_m, ap->a_count, ap->a_sync,
	ap->a_rtvals));
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_bmap_args {
	struct vnode *a_vp;
	daddr_t a_bn;
	struct bufobj **a_bop;
	daddr_t *a_bnp;
	int *a_runp;
	int *a_runb;
	};
	#endif

	static int
	zfs_freebsd_bmap(struct vop_bmap_args *ap)
	{

	if (ap->a_bop != NULL)
	*ap->a_bop = &ap->a_vp->v_bufobj;
	if (ap->a_bnp != NULL)
	*ap->a_bnp = ap->a_bn;
	if (ap->a_runp != NULL)
	*ap->a_runp = 0;
	if (ap->a_runb != NULL)
	*ap->a_runb = 0;

	return (0);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_open_args {
	struct vnode *a_vp;
	int a_mode;
	struct ucred *a_cred;
	struct thread *a_td;
	};
	#endif

	static int
	zfs_freebsd_open(struct vop_open_args *ap)
	{
	vnode_t *vp = ap->a_vp;
	znode_t *zp = VTOZ(vp);
	int error;

	error = zfs_open(&vp, ap->a_mode, ap->a_cred);
	if (error == 0)
	vnode_create_vobject(vp, zp->z_size, ap->a_td);
	return (error);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_close_args {
	struct vnode *a_vp;
	int a_fflag;
	struct ucred *a_cred;
	struct thread *a_td;
	};
	#endif

	static int
	zfs_freebsd_close(struct vop_close_args *ap)
	{

	return (zfs_close(ap->a_vp, ap->a_fflag, 1, 0, ap->a_cred));
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_ioctl_args {
	struct vnode *a_vp;
	ulong_t a_command;
	caddr_t a_data;
	int a_fflag;
	struct ucred *cred;
	struct thread *td;
	};
	#endif

	static int
	zfs_freebsd_ioctl(struct vop_ioctl_args *ap)
	{

	return (zfs_ioctl(ap->a_vp, ap->a_command, (intptr_t)ap->a_data,
	ap->a_fflag, ap->a_cred, NULL));
	}

	static int
	ioflags(int ioflags)
	{
	int flags = 0;

	if (ioflags & IO_APPEND)
	flags \|= FAPPEND;
	if (ioflags & IO_NDELAY)
	flags \|= FNONBLOCK;
	if (ioflags & IO_SYNC)
	flags \|= (FSYNC \| FDSYNC \| FRSYNC);

	return (flags);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_read_args {
	struct vnode *a_vp;
	struct uio *a_uio;
	int a_ioflag;
	struct ucred *a_cred;
	};
	#endif

	static int
	zfs_freebsd_read(struct vop_read_args *ap)
	{

	return (zfs_read(ap->a_vp, ap->a_uio, ioflags(ap->a_ioflag),
	ap->a_cred));
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_write_args {
	struct vnode *a_vp;
	struct uio *a_uio;
	int a_ioflag;
	struct ucred *a_cred;
	};
	#endif

	static int
	zfs_freebsd_write(struct vop_write_args *ap)
	{

	return (zfs_write(ap->a_vp, ap->a_uio, ioflags(ap->a_ioflag),
	ap->a_cred));
	}

	#if __FreeBSD_version >= 1300102
	/*
	* VOP_FPLOOKUP_VEXEC routines are subject to special circumstances, see
	* the comment above cache_fplookup for details.
	*/
	static int
	zfs_freebsd_fplookup_vexec(struct vop_fplookup_vexec_args *v)
	{
	vnode_t *vp;
	znode_t *zp;
	uint64_t pflags;

	vp = v->a_vp;
	zp = VTOZ_SMR(vp);
	if (__predict_false(zp == NULL))
	return (EAGAIN);
	pflags = atomic_load_64(&zp->z_pflags);
	if (pflags & ZFS_AV_QUARANTINED)
	return (EAGAIN);
	if (pflags & ZFS_XATTR)
	return (EAGAIN);
	if ((pflags & ZFS_NO_EXECS_DENIED) == 0)
	return (EAGAIN);
	return (0);
	}
	#endif

	#ifndef _SYS_SYSPROTO_H_
	struct vop_access_args {
	struct vnode *a_vp;
	accmode_t a_accmode;
	struct ucred *a_cred;
	struct thread *a_td;
	};
	#endif

	static int
	zfs_freebsd_access(struct vop_access_args *ap)
	{
	vnode_t *vp = ap->a_vp;
	znode_t *zp = VTOZ(vp);
	accmode_t accmode;
	int error = 0;


	if (ap->a_accmode == VEXEC) {
	if (zfs_fastaccesschk_execute(zp, ap->a_cred) == 0)
	return (0);
	}

	/*
	* ZFS itself only knowns about VREAD, VWRITE, VEXEC and VAPPEND,
	*/
	accmode = ap->a_accmode & (VREAD\|VWRITE\|VEXEC\|VAPPEND);
	if (accmode != 0)
	error = zfs_access(ap->a_vp, accmode, 0, ap->a_cred, NULL);

	/*
	* VADMIN has to be handled by vaccess().
	*/
	if (error == 0) {
	accmode = ap->a_accmode & ~(VREAD\|VWRITE\|VEXEC\|VAPPEND);
	if (accmode != 0) {
	#if __FreeBSD_version >= 1300105
	error = vaccess(vp->v_type, zp->z_mode, zp->z_uid,
	zp->z_gid, accmode, ap->a_cred);
	#else
	error = vaccess(vp->v_type, zp->z_mode, zp->z_uid,
	zp->z_gid, accmode, ap->a_cred, NULL);
	#endif
	}
	}

	/*
	* For VEXEC, ensure that at least one execute bit is set for
	* non-directories.
	*/
	if (error == 0 && (ap->a_accmode & VEXEC) != 0 && vp->v_type != VDIR &&
	(zp->z_mode & (S_IXUSR \| S_IXGRP \| S_IXOTH)) == 0) {
	error = EACCES;
	}

	return (error);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_lookup_args {
	struct vnode *a_dvp;
	struct vnode **a_vpp;
	struct componentname *a_cnp;
	};
	#endif

	static int
	zfs_freebsd_lookup(struct vop_lookup_args *ap, boolean_t cached)
	{
	struct componentname *cnp = ap->a_cnp;
	char nm[NAME_MAX + 1];

	ASSERT(cnp->cn_namelen < sizeof (nm));
	strlcpy(nm, cnp->cn_nameptr, MIN(cnp->cn_namelen + 1, sizeof (nm)));

	return (zfs_lookup(ap->a_dvp, nm, ap->a_vpp, cnp, cnp->cn_nameiop,
	cnp->cn_cred, cnp->cn_thread, 0, cached));
	}

	static int
	zfs_freebsd_cachedlookup(struct vop_cachedlookup_args *ap)
	{

	return (zfs_freebsd_lookup((struct vop_lookup_args *)ap, B_TRUE));
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_lookup_args {
	struct vnode *a_dvp;
	struct vnode **a_vpp;
	struct componentname *a_cnp;
	};
	#endif

	static int
	zfs_cache_lookup(struct vop_lookup_args *ap)
	{
	zfsvfs_t *zfsvfs;

	zfsvfs = ap->a_dvp->v_mount->mnt_data;
	if (zfsvfs->z_use_namecache)
	return (vfs_cache_lookup(ap));
	else
	return (zfs_freebsd_lookup(ap, B_FALSE));
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_create_args {
	struct vnode *a_dvp;
	struct vnode **a_vpp;
	struct componentname *a_cnp;
	struct vattr *a_vap;
	};
	#endif

	static int
	zfs_freebsd_create(struct vop_create_args *ap)
	{
	zfsvfs_t *zfsvfs;
	struct componentname *cnp = ap->a_cnp;
	vattr_t *vap = ap->a_vap;
	znode_t *zp = NULL;
	int rc, mode;

	ASSERT(cnp->cn_flags & SAVENAME);

	vattr_init_mask(vap);
	mode = vap->va_mode & ALLPERMS;
	zfsvfs = ap->a_dvp->v_mount->mnt_data;
	*ap->a_vpp = NULL;

	rc = zfs_create(VTOZ(ap->a_dvp), cnp->cn_nameptr, vap, !EXCL, mode,
	&zp, cnp->cn_cred, 0 /* flag /, NULL / vsecattr */);
	if (rc == 0)
	*ap->a_vpp = ZTOV(zp);
	if (zfsvfs->z_use_namecache &&
	rc == 0 && (cnp->cn_flags & MAKEENTRY) != 0)
	cache_enter(ap->a_dvp, *ap->a_vpp, cnp);

	return (rc);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_remove_args {
	struct vnode *a_dvp;
	struct vnode *a_vp;
	struct componentname *a_cnp;
	};
	#endif

	static int
	zfs_freebsd_remove(struct vop_remove_args *ap)
	{

	ASSERT(ap->a_cnp->cn_flags & SAVENAME);

	return (zfs_remove_(ap->a_dvp, ap->a_vp, ap->a_cnp->cn_nameptr,
	ap->a_cnp->cn_cred));
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_mkdir_args {
	struct vnode *a_dvp;
	struct vnode **a_vpp;
	struct componentname *a_cnp;
	struct vattr *a_vap;
	};
	#endif

	static int
	zfs_freebsd_mkdir(struct vop_mkdir_args *ap)
	{
	vattr_t *vap = ap->a_vap;
	znode_t *zp = NULL;
	int rc;

	ASSERT(ap->a_cnp->cn_flags & SAVENAME);

	vattr_init_mask(vap);
	*ap->a_vpp = NULL;

	rc = zfs_mkdir(VTOZ(ap->a_dvp), ap->a_cnp->cn_nameptr, vap, &zp,
	ap->a_cnp->cn_cred, 0, NULL);

	if (rc == 0)
	*ap->a_vpp = ZTOV(zp);
	return (rc);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_rmdir_args {
	struct vnode *a_dvp;
	struct vnode *a_vp;
	struct componentname *a_cnp;
	};
	#endif

	static int
	zfs_freebsd_rmdir(struct vop_rmdir_args *ap)
	{
	struct componentname *cnp = ap->a_cnp;

	ASSERT(cnp->cn_flags & SAVENAME);

	return (zfs_rmdir_(ap->a_dvp, ap->a_vp, cnp->cn_nameptr, cnp->cn_cred));
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_readdir_args {
	struct vnode *a_vp;
	struct uio *a_uio;
	struct ucred *a_cred;
	int *a_eofflag;
	int *a_ncookies;
	ulong_t **a_cookies;
	};
	#endif

	static int
	zfs_freebsd_readdir(struct vop_readdir_args *ap)
	{

	return (zfs_readdir(ap->a_vp, ap->a_uio, ap->a_cred, ap->a_eofflag,
	ap->a_ncookies, ap->a_cookies));
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_fsync_args {
	struct vnode *a_vp;
	int a_waitfor;
	struct thread *a_td;
	};
	#endif

	static int
	zfs_freebsd_fsync(struct vop_fsync_args *ap)
	{

	vop_stdfsync(ap);
	return (zfs_fsync(ap->a_vp, 0, ap->a_td->td_ucred, NULL));
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_getattr_args {
	struct vnode *a_vp;
	struct vattr *a_vap;
	struct ucred *a_cred;
	};
	#endif

	static int
	zfs_freebsd_getattr(struct vop_getattr_args *ap)
	{
	vattr_t *vap = ap->a_vap;
	xvattr_t xvap;
	ulong_t fflags = 0;
	int error;

	xva_init(&xvap);
	xvap.xva_vattr = *vap;
	xvap.xva_vattr.va_mask \|= AT_XVATTR;

	/* Convert chflags into ZFS-type flags. */
	/* XXX: what about SF_SETTABLE?. */
	XVA_SET_REQ(&xvap, XAT_IMMUTABLE);
	XVA_SET_REQ(&xvap, XAT_APPENDONLY);
	XVA_SET_REQ(&xvap, XAT_NOUNLINK);
	XVA_SET_REQ(&xvap, XAT_NODUMP);
	XVA_SET_REQ(&xvap, XAT_READONLY);
	XVA_SET_REQ(&xvap, XAT_ARCHIVE);
	XVA_SET_REQ(&xvap, XAT_SYSTEM);
	XVA_SET_REQ(&xvap, XAT_HIDDEN);
	XVA_SET_REQ(&xvap, XAT_REPARSE);
	XVA_SET_REQ(&xvap, XAT_OFFLINE);
	XVA_SET_REQ(&xvap, XAT_SPARSE);

	error = zfs_getattr(ap->a_vp, (vattr_t *)&xvap, 0, ap->a_cred);
	if (error != 0)
	return (error);

	/* Convert ZFS xattr into chflags. */
	#define FLAG_CHECK(fflag, xflag, xfield) do { \
	if (XVA_ISSET_RTN(&xvap, (xflag)) && (xfield) != 0) \
	fflags \|= (fflag); \
	} while (0)
	FLAG_CHECK(SF_IMMUTABLE, XAT_IMMUTABLE,
	xvap.xva_xoptattrs.xoa_immutable);
	FLAG_CHECK(SF_APPEND, XAT_APPENDONLY,
	xvap.xva_xoptattrs.xoa_appendonly);
	FLAG_CHECK(SF_NOUNLINK, XAT_NOUNLINK,
	xvap.xva_xoptattrs.xoa_nounlink);
	FLAG_CHECK(UF_ARCHIVE, XAT_ARCHIVE,
	xvap.xva_xoptattrs.xoa_archive);
	FLAG_CHECK(UF_NODUMP, XAT_NODUMP,
	xvap.xva_xoptattrs.xoa_nodump);
	FLAG_CHECK(UF_READONLY, XAT_READONLY,
	xvap.xva_xoptattrs.xoa_readonly);
	FLAG_CHECK(UF_SYSTEM, XAT_SYSTEM,
	xvap.xva_xoptattrs.xoa_system);
	FLAG_CHECK(UF_HIDDEN, XAT_HIDDEN,
	xvap.xva_xoptattrs.xoa_hidden);
	FLAG_CHECK(UF_REPARSE, XAT_REPARSE,
	xvap.xva_xoptattrs.xoa_reparse);
	FLAG_CHECK(UF_OFFLINE, XAT_OFFLINE,
	xvap.xva_xoptattrs.xoa_offline);
	FLAG_CHECK(UF_SPARSE, XAT_SPARSE,
	xvap.xva_xoptattrs.xoa_sparse);

	#undef FLAG_CHECK
	*vap = xvap.xva_vattr;
	vap->va_flags = fflags;
	return (0);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_setattr_args {
	struct vnode *a_vp;
	struct vattr *a_vap;
	struct ucred *a_cred;
	};
	#endif

	static int
	zfs_freebsd_setattr(struct vop_setattr_args *ap)
	{
	vnode_t *vp = ap->a_vp;
	vattr_t *vap = ap->a_vap;
	cred_t *cred = ap->a_cred;
	xvattr_t xvap;
	ulong_t fflags;
	uint64_t zflags;

	vattr_init_mask(vap);
	vap->va_mask &= ~AT_NOSET;

	xva_init(&xvap);
	xvap.xva_vattr = *vap;

	zflags = VTOZ(vp)->z_pflags;

	if (vap->va_flags != VNOVAL) {
	zfsvfs_t *zfsvfs = VTOZ(vp)->z_zfsvfs;
	int error;

	if (zfsvfs->z_use_fuids == B_FALSE)
	return (EOPNOTSUPP);

	fflags = vap->va_flags;
	/*
	* XXX KDM
	* We need to figure out whether it makes sense to allow
	* UF_REPARSE through, since we don't really have other
	* facilities to handle reparse points and zfs_setattr()
	* doesn't currently allow setting that attribute anyway.
	*/
	if ((fflags & ~(SF_IMMUTABLE\|SF_APPEND\|SF_NOUNLINK\|UF_ARCHIVE\|
	UF_NODUMP\|UF_SYSTEM\|UF_HIDDEN\|UF_READONLY\|UF_REPARSE\|
	UF_OFFLINE\|UF_SPARSE)) != 0)
	return (EOPNOTSUPP);
	/*
	* Unprivileged processes are not permitted to unset system
	* flags, or modify flags if any system flags are set.
	* Privileged non-jail processes may not modify system flags
	* if securelevel > 0 and any existing system flags are set.
	* Privileged jail processes behave like privileged non-jail
	* processes if the PR_ALLOW_CHFLAGS permission bit is set;
	* otherwise, they behave like unprivileged processes.
	*/
	if (secpolicy_fs_owner(vp->v_mount, cred) == 0 \|\|
	spl_priv_check_cred(cred, PRIV_VFS_SYSFLAGS) == 0) {
	if (zflags &
	(ZFS_IMMUTABLE \| ZFS_APPENDONLY \| ZFS_NOUNLINK)) {
	error = securelevel_gt(cred, 0);
	if (error != 0)
	return (error);
	}
	} else {
	/*
	* Callers may only modify the file flags on
	* objects they have VADMIN rights for.
	*/
	if ((error = VOP_ACCESS(vp, VADMIN, cred,
	curthread)) != 0)
	return (error);
	if (zflags &
	(ZFS_IMMUTABLE \| ZFS_APPENDONLY \|
	ZFS_NOUNLINK)) {
	return (EPERM);
	}
	if (fflags &
	(SF_IMMUTABLE \| SF_APPEND \| SF_NOUNLINK)) {
	return (EPERM);
	}
	}

	#define FLAG_CHANGE(fflag, zflag, xflag, xfield) do { \
	if (((fflags & (fflag)) && !(zflags & (zflag))) \|\| \
	((zflags & (zflag)) && !(fflags & (fflag)))) { \
	XVA_SET_REQ(&xvap, (xflag)); \
	(xfield) = ((fflags & (fflag)) != 0); \
	} \
	} while (0)
	/* Convert chflags into ZFS-type flags. */
	/* XXX: what about SF_SETTABLE?. */
	FLAG_CHANGE(SF_IMMUTABLE, ZFS_IMMUTABLE, XAT_IMMUTABLE,
	xvap.xva_xoptattrs.xoa_immutable);
	FLAG_CHANGE(SF_APPEND, ZFS_APPENDONLY, XAT_APPENDONLY,
	xvap.xva_xoptattrs.xoa_appendonly);
	FLAG_CHANGE(SF_NOUNLINK, ZFS_NOUNLINK, XAT_NOUNLINK,
	xvap.xva_xoptattrs.xoa_nounlink);
	FLAG_CHANGE(UF_ARCHIVE, ZFS_ARCHIVE, XAT_ARCHIVE,
	xvap.xva_xoptattrs.xoa_archive);
	FLAG_CHANGE(UF_NODUMP, ZFS_NODUMP, XAT_NODUMP,
	xvap.xva_xoptattrs.xoa_nodump);
	FLAG_CHANGE(UF_READONLY, ZFS_READONLY, XAT_READONLY,
	xvap.xva_xoptattrs.xoa_readonly);
	FLAG_CHANGE(UF_SYSTEM, ZFS_SYSTEM, XAT_SYSTEM,
	xvap.xva_xoptattrs.xoa_system);
	FLAG_CHANGE(UF_HIDDEN, ZFS_HIDDEN, XAT_HIDDEN,
	xvap.xva_xoptattrs.xoa_hidden);
	FLAG_CHANGE(UF_REPARSE, ZFS_REPARSE, XAT_REPARSE,
	xvap.xva_xoptattrs.xoa_reparse);
	FLAG_CHANGE(UF_OFFLINE, ZFS_OFFLINE, XAT_OFFLINE,
	xvap.xva_xoptattrs.xoa_offline);
	FLAG_CHANGE(UF_SPARSE, ZFS_SPARSE, XAT_SPARSE,
	xvap.xva_xoptattrs.xoa_sparse);
	#undef FLAG_CHANGE
	}
	if (vap->va_birthtime.tv_sec != VNOVAL) {
	xvap.xva_vattr.va_mask \|= AT_XVATTR;
	XVA_SET_REQ(&xvap, XAT_CREATETIME);
	}
	return (zfs_setattr(VTOZ(vp), (vattr_t *)&xvap, 0, cred));
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_rename_args {
	struct vnode *a_fdvp;
	struct vnode *a_fvp;
	struct componentname *a_fcnp;
	struct vnode *a_tdvp;
	struct vnode *a_tvp;
	struct componentname *a_tcnp;
	};
	#endif

	static int
	zfs_freebsd_rename(struct vop_rename_args *ap)
	{
	vnode_t *fdvp = ap->a_fdvp;
	vnode_t *fvp = ap->a_fvp;
	vnode_t *tdvp = ap->a_tdvp;
	vnode_t *tvp = ap->a_tvp;
	int error;

	ASSERT(ap->a_fcnp->cn_flags & (SAVENAME\|SAVESTART));
	ASSERT(ap->a_tcnp->cn_flags & (SAVENAME\|SAVESTART));

	error = zfs_rename_(fdvp, &fvp, ap->a_fcnp, tdvp, &tvp,
	ap->a_tcnp, ap->a_fcnp->cn_cred, 1);

	vrele(fdvp);
	vrele(fvp);
	vrele(tdvp);
	if (tvp != NULL)
	vrele(tvp);

	return (error);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_symlink_args {
	struct vnode *a_dvp;
	struct vnode **a_vpp;
	struct componentname *a_cnp;
	struct vattr *a_vap;
	char *a_target;
	};
	#endif

	static int
	zfs_freebsd_symlink(struct vop_symlink_args *ap)
	{
	struct componentname *cnp = ap->a_cnp;
	vattr_t *vap = ap->a_vap;
	znode_t *zp = NULL;
	int rc;

	ASSERT(cnp->cn_flags & SAVENAME);

	vap->va_type = VLNK; /* FreeBSD: Syscall only sets va_mode. */
	vattr_init_mask(vap);
	*ap->a_vpp = NULL;

	rc = zfs_symlink(VTOZ(ap->a_dvp), cnp->cn_nameptr, vap,
	ap->a_target, &zp, cnp->cn_cred, 0 /* flags */);
	if (rc == 0)
	*ap->a_vpp = ZTOV(zp);
	return (rc);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_readlink_args {
	struct vnode *a_vp;
	struct uio *a_uio;
	struct ucred *a_cred;
	};
	#endif

	static int
	zfs_freebsd_readlink(struct vop_readlink_args *ap)
	{

	return (zfs_readlink(ap->a_vp, ap->a_uio, ap->a_cred, NULL));
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_link_args {
	struct vnode *a_tdvp;
	struct vnode *a_vp;
	struct componentname *a_cnp;
	};
	#endif

	static int
	zfs_freebsd_link(struct vop_link_args *ap)
	{
	struct componentname *cnp = ap->a_cnp;
	vnode_t *vp = ap->a_vp;
	vnode_t *tdvp = ap->a_tdvp;

	if (tdvp->v_mount != vp->v_mount)
	return (EXDEV);

	ASSERT(cnp->cn_flags & SAVENAME);

	return (zfs_link(VTOZ(tdvp), VTOZ(vp),
	cnp->cn_nameptr, cnp->cn_cred, 0));
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_inactive_args {
	struct vnode *a_vp;
	struct thread *a_td;
	};
	#endif

	static int
	zfs_freebsd_inactive(struct vop_inactive_args *ap)
	{
	vnode_t *vp = ap->a_vp;

	zfs_inactive(vp, ap->a_td->td_ucred, NULL);
	return (0);
	}

	#if __FreeBSD_version >= 1300042
	#ifndef _SYS_SYSPROTO_H_
	struct vop_need_inactive_args {
	struct vnode *a_vp;
	struct thread *a_td;
	};
	#endif

	static int
	zfs_freebsd_need_inactive(struct vop_need_inactive_args *ap)
	{
	vnode_t *vp = ap->a_vp;
	znode_t *zp = VTOZ(vp);
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;
	int need;

	if (vn_need_pageq_flush(vp))
	return (1);

	if (!ZFS_TRYRLOCK_TEARDOWN_INACTIVE(zfsvfs))
	return (1);
	need = (zp->z_sa_hdl == NULL \|\| zp->z_unlinked \|\| zp->z_atime_dirty);
	ZFS_RUNLOCK_TEARDOWN_INACTIVE(zfsvfs);

	return (need);
	}
	#endif

	#ifndef _SYS_SYSPROTO_H_
	struct vop_reclaim_args {
	struct vnode *a_vp;
	struct thread *a_td;
	};
	#endif

	static int
	zfs_freebsd_reclaim(struct vop_reclaim_args *ap)
	{
	vnode_t *vp = ap->a_vp;
	znode_t *zp = VTOZ(vp);
	zfsvfs_t *zfsvfs = zp->z_zfsvfs;

	ASSERT(zp != NULL);

	#if __FreeBSD_version < 1300042
	/* Destroy the vm object and flush associated pages. */
	vnode_destroy_vobject(vp);
	#endif
	/*
	* z_teardown_inactive_lock protects from a race with
	* zfs_znode_dmu_fini in zfsvfs_teardown during
	* force unmount.
	*/
	ZFS_RLOCK_TEARDOWN_INACTIVE(zfsvfs);
	if (zp->z_sa_hdl == NULL)
	zfs_znode_free(zp);
	else
	zfs_zinactive(zp);
	ZFS_RUNLOCK_TEARDOWN_INACTIVE(zfsvfs);

	vp->v_data = NULL;
	return (0);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_fid_args {
	struct vnode *a_vp;
	struct fid *a_fid;
	};
	#endif

	static int
	zfs_freebsd_fid(struct vop_fid_args *ap)
	{

	return (zfs_fid(ap->a_vp, (void *)ap->a_fid, NULL));
	}


	#ifndef _SYS_SYSPROTO_H_
	struct vop_pathconf_args {
	struct vnode *a_vp;
	int a_name;
	register_t *a_retval;
	} *ap;
	#endif

	static int
	zfs_freebsd_pathconf(struct vop_pathconf_args *ap)
	{
	ulong_t val;
	int error;

	error = zfs_pathconf(ap->a_vp, ap->a_name, &val,
	curthread->td_ucred, NULL);
	if (error == 0) {
	*ap->a_retval = val;
	return (error);
	}
	if (error != EOPNOTSUPP)
	return (error);

	switch (ap->a_name) {
	case _PC_NAME_MAX:
	*ap->a_retval = NAME_MAX;
	return (0);
	case _PC_PIPE_BUF:
	if (ap->a_vp->v_type == VDIR \|\| ap->a_vp->v_type == VFIFO) {
	*ap->a_retval = PIPE_BUF;
	return (0);
	}
	return (EINVAL);
	default:
	return (vop_stdpathconf(ap));
	}
	}

	/*
	* FreeBSD's extended attributes namespace defines file name prefix for ZFS'
	* extended attribute name:
	*
	* NAMESPACE PREFIX
	* system freebsd:system:
	* user (none, can be used to access ZFS fsattr(5) attributes
	* created on Solaris)
	*/
	static int
	zfs_create_attrname(int attrnamespace, const char name, char attrname,
	size_t size)
	{
	const char namespace, prefix, *suffix;

	/* We don't allow '/' character in attribute name. */
	if (strchr(name, '/') != NULL)
	return (EINVAL);
	/* We don't allow attribute names that start with "freebsd:" string. */
	if (strncmp(name, "freebsd:", 8) == 0)
	return (EINVAL);

	bzero(attrname, size);

	switch (attrnamespace) {
	case EXTATTR_NAMESPACE_USER:
	#if 0
	prefix = "freebsd:";
	namespace = EXTATTR_NAMESPACE_USER_STRING;
	suffix = ":";
	#else
	/*
	* This is the default namespace by which we can access all
	* attributes created on Solaris.
	*/
	prefix = namespace = suffix = "";
	#endif
	break;
	case EXTATTR_NAMESPACE_SYSTEM:
	prefix = "freebsd:";
	namespace = EXTATTR_NAMESPACE_SYSTEM_STRING;
	suffix = ":";
	break;
	case EXTATTR_NAMESPACE_EMPTY:
	default:
	return (EINVAL);
	}
	if (snprintf(attrname, size, "%s%s%s%s", prefix, namespace, suffix,
	name) >= size) {
	return (ENAMETOOLONG);
	}
	return (0);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_getextattr {
	IN struct vnode *a_vp;
	IN int a_attrnamespace;
	IN const char *a_name;
	INOUT struct uio *a_uio;
	OUT size_t *a_size;
	IN struct ucred *a_cred;
	IN struct thread *a_td;
	};
	#endif

	/*
	* Vnode operating to retrieve a named extended attribute.
	*/
	static int
	zfs_getextattr(struct vop_getextattr_args *ap)
	{
	zfsvfs_t *zfsvfs = VTOZ(ap->a_vp)->z_zfsvfs;
	struct thread *td = ap->a_td;
	struct nameidata nd;
	char attrname[255];
	struct vattr va;
	vnode_t xvp = NULL, vp;
	int error, flags;

	/*
	* If the xattr property is off, refuse the request.
	*/
	if (!(zfsvfs->z_flags & ZSB_XATTR)) {
	return (SET_ERROR(EOPNOTSUPP));
	}

	error = extattr_check_cred(ap->a_vp, ap->a_attrnamespace,
	ap->a_cred, ap->a_td, VREAD);
	if (error != 0)
	return (error);

	error = zfs_create_attrname(ap->a_attrnamespace, ap->a_name, attrname,
	sizeof (attrname));
	if (error != 0)
	return (error);

	ZFS_ENTER(zfsvfs);

	error = zfs_lookup(ap->a_vp, NULL, &xvp, NULL, 0, ap->a_cred, td,
	LOOKUP_XATTR, B_FALSE);
	if (error != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	flags = FREAD;
	NDINIT_ATVP(&nd, LOOKUP, NOFOLLOW, UIO_SYSSPACE, attrname,
	xvp, td);
	error = vn_open_cred(&nd, &flags, 0, VN_OPEN_INVFS, ap->a_cred, NULL);
	vp = nd.ni_vp;
	NDFREE(&nd, NDF_ONLY_PNBUF);
	if (error != 0) {
	ZFS_EXIT(zfsvfs);
	if (error == ENOENT)
	error = ENOATTR;
	return (error);
	}

	if (ap->a_size != NULL) {
	error = VOP_GETATTR(vp, &va, ap->a_cred);
	if (error == 0)
	*ap->a_size = (size_t)va.va_size;
	} else if (ap->a_uio != NULL)
	error = VOP_READ(vp, ap->a_uio, IO_UNIT, ap->a_cred);

	VOP_UNLOCK1(vp);
	vn_close(vp, flags, ap->a_cred, td);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_deleteextattr {
	IN struct vnode *a_vp;
	IN int a_attrnamespace;
	IN const char *a_name;
	IN struct ucred *a_cred;
	IN struct thread *a_td;
	};
	#endif

	/*
	* Vnode operation to remove a named attribute.
	*/
	static int
	zfs_deleteextattr(struct vop_deleteextattr_args *ap)
	{
	zfsvfs_t *zfsvfs = VTOZ(ap->a_vp)->z_zfsvfs;
	struct thread *td = ap->a_td;
	struct nameidata nd;
	char attrname[255];
	vnode_t xvp = NULL, vp;
	int error;

	/*
	* If the xattr property is off, refuse the request.
	*/
	if (!(zfsvfs->z_flags & ZSB_XATTR)) {
	return (SET_ERROR(EOPNOTSUPP));
	}

	error = extattr_check_cred(ap->a_vp, ap->a_attrnamespace,
	ap->a_cred, ap->a_td, VWRITE);
	if (error != 0)
	return (error);

	error = zfs_create_attrname(ap->a_attrnamespace, ap->a_name, attrname,
	sizeof (attrname));
	if (error != 0)
	return (error);

	ZFS_ENTER(zfsvfs);

	error = zfs_lookup(ap->a_vp, NULL, &xvp, NULL, 0, ap->a_cred, td,
	LOOKUP_XATTR, B_FALSE);
	if (error != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	NDINIT_ATVP(&nd, DELETE, NOFOLLOW \| LOCKPARENT \| LOCKLEAF,
	UIO_SYSSPACE, attrname, xvp, td);
	error = namei(&nd);
	vp = nd.ni_vp;
	if (error != 0) {
	ZFS_EXIT(zfsvfs);
	NDFREE(&nd, NDF_ONLY_PNBUF);
	if (error == ENOENT)
	error = ENOATTR;
	return (error);
	}

	error = VOP_REMOVE(nd.ni_dvp, vp, &nd.ni_cnd);
	NDFREE(&nd, NDF_ONLY_PNBUF);

	vput(nd.ni_dvp);
	if (vp == nd.ni_dvp)
	vrele(vp);
	else
	vput(vp);
	ZFS_EXIT(zfsvfs);

	return (error);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_setextattr {
	IN struct vnode *a_vp;
	IN int a_attrnamespace;
	IN const char *a_name;
	INOUT struct uio *a_uio;
	IN struct ucred *a_cred;
	IN struct thread *a_td;
	};
	#endif

	/*
	* Vnode operation to set a named attribute.
	*/
	static int
	zfs_setextattr(struct vop_setextattr_args *ap)
	{
	zfsvfs_t *zfsvfs = VTOZ(ap->a_vp)->z_zfsvfs;
	struct thread *td = ap->a_td;
	struct nameidata nd;
	char attrname[255];
	struct vattr va;
	vnode_t xvp = NULL, vp;
	int error, flags;

	/*
	* If the xattr property is off, refuse the request.
	*/
	if (!(zfsvfs->z_flags & ZSB_XATTR)) {
	return (SET_ERROR(EOPNOTSUPP));
	}

	error = extattr_check_cred(ap->a_vp, ap->a_attrnamespace,
	ap->a_cred, ap->a_td, VWRITE);
	if (error != 0)
	return (error);
	error = zfs_create_attrname(ap->a_attrnamespace, ap->a_name, attrname,
	sizeof (attrname));
	if (error != 0)
	return (error);

	ZFS_ENTER(zfsvfs);

	error = zfs_lookup(ap->a_vp, NULL, &xvp, NULL, 0, ap->a_cred, td,
	LOOKUP_XATTR \| CREATE_XATTR_DIR, B_FALSE);
	if (error != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	flags = FFLAGS(O_WRONLY \| O_CREAT);
	NDINIT_ATVP(&nd, LOOKUP, NOFOLLOW, UIO_SYSSPACE, attrname,
	xvp, td);
	error = vn_open_cred(&nd, &flags, 0600, VN_OPEN_INVFS, ap->a_cred,
	NULL);
	vp = nd.ni_vp;
	NDFREE(&nd, NDF_ONLY_PNBUF);
	if (error != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	VATTR_NULL(&va);
	va.va_size = 0;
	error = VOP_SETATTR(vp, &va, ap->a_cred);
	if (error == 0)
	VOP_WRITE(vp, ap->a_uio, IO_UNIT, ap->a_cred);

	VOP_UNLOCK1(vp);
	vn_close(vp, flags, ap->a_cred, td);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_listextattr {
	IN struct vnode *a_vp;
	IN int a_attrnamespace;
	INOUT struct uio *a_uio;
	OUT size_t *a_size;
	IN struct ucred *a_cred;
	IN struct thread *a_td;
	};
	#endif

	/*
	* Vnode operation to retrieve extended attributes on a vnode.
	*/
	static int
	zfs_listextattr(struct vop_listextattr_args *ap)
	{
	zfsvfs_t *zfsvfs = VTOZ(ap->a_vp)->z_zfsvfs;
	struct thread *td = ap->a_td;
	struct nameidata nd;
	char attrprefix[16];
	uint8_t dirbuf[sizeof (struct dirent)];
	struct dirent *dp;
	struct iovec aiov;
	struct uio auio, *uio = ap->a_uio;
	size_t *sizep = ap->a_size;
	size_t plen;
	vnode_t xvp = NULL, vp;
	int done, error, eof, pos;

	/*
	* If the xattr property is off, refuse the request.
	*/
	if (!(zfsvfs->z_flags & ZSB_XATTR)) {
	return (SET_ERROR(EOPNOTSUPP));
	}

	error = extattr_check_cred(ap->a_vp, ap->a_attrnamespace,
	ap->a_cred, ap->a_td, VREAD);
	if (error != 0)
	return (error);

	error = zfs_create_attrname(ap->a_attrnamespace, "", attrprefix,
	sizeof (attrprefix));
	if (error != 0)
	return (error);
	plen = strlen(attrprefix);

	ZFS_ENTER(zfsvfs);

	if (sizep != NULL)
	*sizep = 0;

	error = zfs_lookup(ap->a_vp, NULL, &xvp, NULL, 0, ap->a_cred, td,
	LOOKUP_XATTR, B_FALSE);
	if (error != 0) {
	ZFS_EXIT(zfsvfs);
	/*
	* ENOATTR means that the EA directory does not yet exist,
	* i.e. there are no extended attributes there.
	*/
	if (error == ENOATTR)
	error = 0;
	return (error);
	}

	NDINIT_ATVP(&nd, LOOKUP, NOFOLLOW \| LOCKLEAF \| LOCKSHARED,
	UIO_SYSSPACE, ".", xvp, td);
	error = namei(&nd);
	vp = nd.ni_vp;
	NDFREE(&nd, NDF_ONLY_PNBUF);
	if (error != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	auio.uio_iov = &aiov;
	auio.uio_iovcnt = 1;
	auio.uio_segflg = UIO_SYSSPACE;
	auio.uio_td = td;
	auio.uio_rw = UIO_READ;
	auio.uio_offset = 0;

	do {
	uint8_t nlen;

	aiov.iov_base = (void *)dirbuf;
	aiov.iov_len = sizeof (dirbuf);
	auio.uio_resid = sizeof (dirbuf);
	error = VOP_READDIR(vp, &auio, ap->a_cred, &eof, NULL, NULL);
	done = sizeof (dirbuf) - auio.uio_resid;
	if (error != 0)
	break;
	for (pos = 0; pos < done; ) {
	dp = (struct dirent *)(dirbuf + pos);
	pos += dp->d_reclen;
	/*
	* XXX: Temporarily we also accept DT_UNKNOWN, as this
	* is what we get when attribute was created on Solaris.
	*/
	if (dp->d_type != DT_REG && dp->d_type != DT_UNKNOWN)
	continue;
	if (plen == 0 &&
	strncmp(dp->d_name, "freebsd:", 8) == 0)
	continue;
	else if (strncmp(dp->d_name, attrprefix, plen) != 0)
	continue;
	nlen = dp->d_namlen - plen;
	if (sizep != NULL)
	*sizep += 1 + nlen;
	else if (uio != NULL) {
	/*
	* Format of extattr name entry is one byte for
	* length and the rest for name.
	*/
	error = uiomove(&nlen, 1, uio->uio_rw, uio);
	if (error == 0) {
	error = uiomove(dp->d_name + plen, nlen,
	uio->uio_rw, uio);
	}
	if (error != 0)
	break;
	}
	}
	} while (!eof && error == 0);

	vput(vp);
	ZFS_EXIT(zfsvfs);

	return (error);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_getacl_args {
	struct vnode *vp;
	acl_type_t type;
	struct acl *aclp;
	struct ucred *cred;
	struct thread *td;
	};
	#endif

	static int
	zfs_freebsd_getacl(struct vop_getacl_args *ap)
	{
	int error;
	vsecattr_t vsecattr;

	if (ap->a_type != ACL_TYPE_NFS4)
	return (EINVAL);

	vsecattr.vsa_mask = VSA_ACE \| VSA_ACECNT;
	if ((error = zfs_getsecattr(ap->a_vp, &vsecattr, 0, ap->a_cred, NULL)))
	return (error);

	error = acl_from_aces(ap->a_aclp, vsecattr.vsa_aclentp,
	vsecattr.vsa_aclcnt);
	if (vsecattr.vsa_aclentp != NULL)
	kmem_free(vsecattr.vsa_aclentp, vsecattr.vsa_aclentsz);

	return (error);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_setacl_args {
	struct vnode *vp;
	acl_type_t type;
	struct acl *aclp;
	struct ucred *cred;
	struct thread *td;
	};
	#endif

	static int
	zfs_freebsd_setacl(struct vop_setacl_args *ap)
	{
	int error;
	vsecattr_t vsecattr;
	int aclbsize; /* size of acl list in bytes */
	aclent_t *aaclp;

	if (ap->a_type != ACL_TYPE_NFS4)
	return (EINVAL);

	if (ap->a_aclp == NULL)
	return (EINVAL);

	if (ap->a_aclp->acl_cnt < 1 \|\| ap->a_aclp->acl_cnt > MAX_ACL_ENTRIES)
	return (EINVAL);

	/*
	* With NFSv4 ACLs, chmod(2) may need to add additional entries,
	* splitting every entry into two and appending "canonical six"
	* entries at the end. Don't allow for setting an ACL that would
	* cause chmod(2) to run out of ACL entries.
	*/
	if (ap->a_aclp->acl_cnt * 2 + 6 > ACL_MAX_ENTRIES)
	return (ENOSPC);

	error = acl_nfs4_check(ap->a_aclp, ap->a_vp->v_type == VDIR);
	if (error != 0)
	return (error);

	vsecattr.vsa_mask = VSA_ACE;
	aclbsize = ap->a_aclp->acl_cnt * sizeof (ace_t);
	vsecattr.vsa_aclentp = kmem_alloc(aclbsize, KM_SLEEP);
	aaclp = vsecattr.vsa_aclentp;
	vsecattr.vsa_aclentsz = aclbsize;

	aces_from_acl(vsecattr.vsa_aclentp, &vsecattr.vsa_aclcnt, ap->a_aclp);
	error = zfs_setsecattr(VTOZ(ap->a_vp), &vsecattr, 0, ap->a_cred);
	kmem_free(aaclp, aclbsize);

	return (error);
	}

	#ifndef _SYS_SYSPROTO_H_
	struct vop_aclcheck_args {
	struct vnode *vp;
	acl_type_t type;
	struct acl *aclp;
	struct ucred *cred;
	struct thread *td;
	};
	#endif

	static int
	zfs_freebsd_aclcheck(struct vop_aclcheck_args *ap)
	{

	return (EOPNOTSUPP);
	}

	static int
	zfs_vptocnp(struct vop_vptocnp_args *ap)
	{
	vnode_t *covered_vp;
	vnode_t *vp = ap->a_vp;
	zfsvfs_t *zfsvfs = vp->v_vfsp->vfs_data;
	znode_t *zp = VTOZ(vp);
	int ltype;
	int error;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	/*
	* If we are a snapshot mounted under .zfs, run the operation
	* on the covered vnode.
	*/
	if (zp->z_id != zfsvfs->z_root \|\| zfsvfs->z_parent == zfsvfs) {
	char name[MAXNAMLEN + 1];
	znode_t *dzp;
	size_t len;

	error = zfs_znode_parent_and_name(zp, &dzp, name);
	if (error == 0) {
	len = strlen(name);
	if (*ap->a_buflen < len)
	error = SET_ERROR(ENOMEM);
	}
	if (error == 0) {
	*ap->a_buflen -= len;
	bcopy(name, ap->a_buf + *ap->a_buflen, len);
	*ap->a_vpp = ZTOV(dzp);
	}
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	ZFS_EXIT(zfsvfs);

	covered_vp = vp->v_mount->mnt_vnodecovered;
	#if __FreeBSD_version >= 1300045
	enum vgetstate vs = vget_prep(covered_vp);
	#else
	vhold(covered_vp);
	#endif
	ltype = VOP_ISLOCKED(vp);
	VOP_UNLOCK1(vp);
	#if __FreeBSD_version >= 1300045
	error = vget_finish(covered_vp, LK_SHARED, vs);
	#else
	error = vget(covered_vp, LK_SHARED \| LK_VNHELD, curthread);
	#endif
	if (error == 0) {
	error = VOP_VPTOCNP(covered_vp, ap->a_vpp, ap->a_cred,
	ap->a_buf, ap->a_buflen);
	vput(covered_vp);
	}
	vn_lock(vp, ltype \| LK_RETRY);
	if (VN_IS_DOOMED(vp))
	error = SET_ERROR(ENOENT);
	return (error);
	}

	#ifdef DIAGNOSTIC
	#ifndef _SYS_SYSPROTO_H_
	struct vop_lock1_args {
	struct vnode *a_vp;
	int a_flags;
	char *file;
	int line;
	};
	#endif

	static int
	zfs_lock(struct vop_lock1_args *ap)
	{
	vnode_t *vp;
	znode_t *zp;
	int err;

	#if __FreeBSD_version >= 1300064
	err = vop_lock(ap);
	#else
	err = vop_stdlock(ap);
	#endif
	if (err == 0 && (ap->a_flags & LK_NOWAIT) == 0) {
	vp = ap->a_vp;
	zp = vp->v_data;
	if (vp->v_mount != NULL && !VN_IS_DOOMED(vp) &&
	zp != NULL && (zp->z_pflags & ZFS_XATTR) == 0)
	VERIFY(!RRM_LOCK_HELD(&zp->z_zfsvfs->z_teardown_lock));
	}
	return (err);
	}
	#endif

	struct vop_vector zfs_vnodeops;
	struct vop_vector zfs_fifoops;
	struct vop_vector zfs_shareops;

	struct vop_vector zfs_vnodeops = {
	.vop_default = &default_vnodeops,
	.vop_inactive = zfs_freebsd_inactive,
	#if __FreeBSD_version >= 1300042
	.vop_need_inactive = zfs_freebsd_need_inactive,
	#endif
	.vop_reclaim = zfs_freebsd_reclaim,
	#if __FreeBSD_version >= 1300102
	.vop_fplookup_vexec = zfs_freebsd_fplookup_vexec,
	#endif
	.vop_access = zfs_freebsd_access,
	.vop_allocate = VOP_EINVAL,
	.vop_lookup = zfs_cache_lookup,
	.vop_cachedlookup = zfs_freebsd_cachedlookup,
	.vop_getattr = zfs_freebsd_getattr,
	.vop_setattr = zfs_freebsd_setattr,
	.vop_create = zfs_freebsd_create,
	.vop_mknod = (vop_mknod_t *)zfs_freebsd_create,
	.vop_mkdir = zfs_freebsd_mkdir,
	.vop_readdir = zfs_freebsd_readdir,
	.vop_fsync = zfs_freebsd_fsync,
	.vop_open = zfs_freebsd_open,
	.vop_close = zfs_freebsd_close,
	.vop_rmdir = zfs_freebsd_rmdir,
	.vop_ioctl = zfs_freebsd_ioctl,
	.vop_link = zfs_freebsd_link,
	.vop_symlink = zfs_freebsd_symlink,
	.vop_readlink = zfs_freebsd_readlink,
	.vop_read = zfs_freebsd_read,
	.vop_write = zfs_freebsd_write,
	.vop_remove = zfs_freebsd_remove,
	.vop_rename = zfs_freebsd_rename,
	.vop_pathconf = zfs_freebsd_pathconf,
	.vop_bmap = zfs_freebsd_bmap,
	.vop_fid = zfs_freebsd_fid,
	.vop_getextattr = zfs_getextattr,
	.vop_deleteextattr = zfs_deleteextattr,
	.vop_setextattr = zfs_setextattr,
	.vop_listextattr = zfs_listextattr,
	.vop_getacl = zfs_freebsd_getacl,
	.vop_setacl = zfs_freebsd_setacl,
	.vop_aclcheck = zfs_freebsd_aclcheck,
	.vop_getpages = zfs_freebsd_getpages,
	.vop_putpages = zfs_freebsd_putpages,
	.vop_vptocnp = zfs_vptocnp,
	#if __FreeBSD_version >= 1300064
	#ifdef DIAGNOSTIC
	.vop_lock1 = zfs_lock,
	#else
	.vop_lock1 = vop_lock,
	#endif
	.vop_unlock = vop_unlock,
	.vop_islocked = vop_islocked,
	#else
	#ifdef DIAGNOSTIC
	.vop_lock1 = zfs_lock,
	#endif
	#endif
	};
	VFS_VOP_VECTOR_REGISTER(zfs_vnodeops);

	struct vop_vector zfs_fifoops = {
	.vop_default = &fifo_specops,
	.vop_fsync = zfs_freebsd_fsync,
	#if __FreeBSD_version >= 1300102
	.vop_fplookup_vexec = zfs_freebsd_fplookup_vexec,
	#endif
	.vop_access = zfs_freebsd_access,
	.vop_getattr = zfs_freebsd_getattr,
	.vop_inactive = zfs_freebsd_inactive,
	.vop_read = VOP_PANIC,
	.vop_reclaim = zfs_freebsd_reclaim,
	.vop_setattr = zfs_freebsd_setattr,
	.vop_write = VOP_PANIC,
	.vop_pathconf = zfs_freebsd_pathconf,
	.vop_fid = zfs_freebsd_fid,
	.vop_getacl = zfs_freebsd_getacl,
	.vop_setacl = zfs_freebsd_setacl,
	.vop_aclcheck = zfs_freebsd_aclcheck,
	};
	VFS_VOP_VECTOR_REGISTER(zfs_fifoops);

	/*
	* special share hidden files vnode operations template
	*/
	struct vop_vector zfs_shareops = {
	.vop_default = &default_vnodeops,
	+#if __FreeBSD_version >= 1300121
	+ .vop_fplookup_vexec = VOP_EAGAIN,
	+#endif
	.vop_access = zfs_freebsd_access,
	.vop_inactive = zfs_freebsd_inactive,
	.vop_reclaim = zfs_freebsd_reclaim,
	.vop_fid = zfs_freebsd_fid,
	.vop_pathconf = zfs_freebsd_pathconf,
	};
	VFS_VOP_VECTOR_REGISTER(zfs_shareops);
	Index: vendor-sys/openzfs/dist/module/os/linux/spl/README.md
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/spl/README.md (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/spl/README.md (revision 366774)
	@@ -1,16 +1,16 @@
	The Solaris Porting Layer, SPL, is a Linux kernel module which provides a
	-compatibility layer used by the [ZFS on Linux](https://zfsonlinux.org) project.
	+compatibility layer used by the [OpenZFS](https://github.com/openzfs/zfs) project.

	# Installation

	The latest version of the SPL is maintained as part of this repository.
	Only when building ZFS version 0.7.x or earlier must an external SPL release
	be used. These releases can be found at:

	* Version 0.7.x: https://github.com/zfsonlinux/spl/tree/spl-0.7-release
	* Version 0.6.5.x: https://github.com/zfsonlinux/spl/tree/spl-0.6.5-release

	# Release

	The SPL is released under a GPLv2 license.
	For more details see the NOTICE and THIRDPARTYLICENSE files; `UCRL-CODE-235197`
	Index: vendor-sys/openzfs/dist/module/os/linux/spl/spl-atomic.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/spl/spl-atomic.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/spl/spl-atomic.c (revision 366774)
	@@ -1,36 +1,35 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*
	* Solaris Porting Layer (SPL) Atomic Implementation.
	*/

	#include <sys/atomic.h>

	#ifdef ATOMIC_SPINLOCK
	/* Global atomic lock declarations */
	DEFINE_SPINLOCK(atomic32_lock);
	DEFINE_SPINLOCK(atomic64_lock);

	EXPORT_SYMBOL(atomic32_lock);
	EXPORT_SYMBOL(atomic64_lock);
	#endif /* ATOMIC_SPINLOCK */
	Index: vendor-sys/openzfs/dist/module/os/linux/spl/spl-condvar.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/spl/spl-condvar.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/spl/spl-condvar.c (revision 366774)
	@@ -1,510 +1,509 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*
	* Solaris Porting Layer (SPL) Credential Implementation.
	*/

	#include <sys/condvar.h>
	#include <sys/time.h>
	#include <sys/sysmacros.h>
	#include <linux/hrtimer.h>
	#include <linux/compiler_compat.h>
	#include <linux/mod_compat.h>

	#include <linux/sched.h>

	#ifdef HAVE_SCHED_SIGNAL_HEADER
	#include <linux/sched/signal.h>
	#endif

	#define MAX_HRTIMEOUT_SLACK_US 1000
	unsigned int spl_schedule_hrtimeout_slack_us = 0;

	static int
	param_set_hrtimeout_slack(const char buf, zfs_kernel_param_t kp)
	{
	unsigned long val;
	int error;

	error = kstrtoul(buf, 0, &val);
	if (error)
	return (error);

	if (val > MAX_HRTIMEOUT_SLACK_US)
	return (-EINVAL);

	error = param_set_uint(buf, kp);
	if (error < 0)
	return (error);

	return (0);
	}

	module_param_call(spl_schedule_hrtimeout_slack_us, param_set_hrtimeout_slack,
	param_get_uint, &spl_schedule_hrtimeout_slack_us, 0644);
	MODULE_PARM_DESC(spl_schedule_hrtimeout_slack_us,
	"schedule_hrtimeout_range() delta/slack value in us, default(0)");

	void
	__cv_init(kcondvar_t cvp, char name, kcv_type_t type, void *arg)
	{
	ASSERT(cvp);
	ASSERT(name == NULL);
	ASSERT(type == CV_DEFAULT);
	ASSERT(arg == NULL);

	cvp->cv_magic = CV_MAGIC;
	init_waitqueue_head(&cvp->cv_event);
	init_waitqueue_head(&cvp->cv_destroy);
	atomic_set(&cvp->cv_waiters, 0);
	atomic_set(&cvp->cv_refs, 1);
	cvp->cv_mutex = NULL;
	}
	EXPORT_SYMBOL(__cv_init);

	static int
	cv_destroy_wakeup(kcondvar_t *cvp)
	{
	if (!atomic_read(&cvp->cv_waiters) && !atomic_read(&cvp->cv_refs)) {
	ASSERT(cvp->cv_mutex == NULL);
	ASSERT(!waitqueue_active(&cvp->cv_event));
	return (1);
	}

	return (0);
	}

	void
	__cv_destroy(kcondvar_t *cvp)
	{
	ASSERT(cvp);
	ASSERT(cvp->cv_magic == CV_MAGIC);

	cvp->cv_magic = CV_DESTROY;
	atomic_dec(&cvp->cv_refs);

	/* Block until all waiters are woken and references dropped. */
	while (cv_destroy_wakeup(cvp) == 0)
	wait_event_timeout(cvp->cv_destroy, cv_destroy_wakeup(cvp), 1);

	ASSERT3P(cvp->cv_mutex, ==, NULL);
	ASSERT3S(atomic_read(&cvp->cv_refs), ==, 0);
	ASSERT3S(atomic_read(&cvp->cv_waiters), ==, 0);
	ASSERT3S(waitqueue_active(&cvp->cv_event), ==, 0);
	}
	EXPORT_SYMBOL(__cv_destroy);

	static void
	cv_wait_common(kcondvar_t cvp, kmutex_t mp, int state, int io)
	{
	DEFINE_WAIT(wait);
	kmutex_t *m;

	ASSERT(cvp);
	ASSERT(mp);
	ASSERT(cvp->cv_magic == CV_MAGIC);
	ASSERT(mutex_owned(mp));
	atomic_inc(&cvp->cv_refs);

	m = READ_ONCE(cvp->cv_mutex);
	if (!m)
	m = xchg(&cvp->cv_mutex, mp);
	/* Ensure the same mutex is used by all callers */
	ASSERT(m == NULL \|\| m == mp);

	prepare_to_wait_exclusive(&cvp->cv_event, &wait, state);
	atomic_inc(&cvp->cv_waiters);

	/*
	* Mutex should be dropped after prepare_to_wait() this
	* ensures we're linked in to the waiters list and avoids the
	* race where 'cvp->cv_waiters > 0' but the list is empty.
	*/
	mutex_exit(mp);
	if (io)
	io_schedule();
	else
	schedule();

	/* No more waiters a different mutex could be used */
	if (atomic_dec_and_test(&cvp->cv_waiters)) {
	/*
	* This is set without any lock, so it's racy. But this is
	* just for debug anyway, so make it best-effort
	*/
	cvp->cv_mutex = NULL;
	wake_up(&cvp->cv_destroy);
	}

	finish_wait(&cvp->cv_event, &wait);
	atomic_dec(&cvp->cv_refs);

	/*
	* Hold mutex after we release the cvp, otherwise we could dead lock
	* with a thread holding the mutex and call cv_destroy.
	*/
	mutex_enter(mp);
	}

	void
	__cv_wait(kcondvar_t cvp, kmutex_t mp)
	{
	cv_wait_common(cvp, mp, TASK_UNINTERRUPTIBLE, 0);
	}
	EXPORT_SYMBOL(__cv_wait);

	void
	__cv_wait_io(kcondvar_t cvp, kmutex_t mp)
	{
	cv_wait_common(cvp, mp, TASK_UNINTERRUPTIBLE, 1);
	}
	EXPORT_SYMBOL(__cv_wait_io);

	int
	__cv_wait_io_sig(kcondvar_t cvp, kmutex_t mp)
	{
	cv_wait_common(cvp, mp, TASK_INTERRUPTIBLE, 1);

	return (signal_pending(current) ? 0 : 1);
	}
	EXPORT_SYMBOL(__cv_wait_io_sig);

	int
	__cv_wait_sig(kcondvar_t cvp, kmutex_t mp)
	{
	cv_wait_common(cvp, mp, TASK_INTERRUPTIBLE, 0);

	return (signal_pending(current) ? 0 : 1);
	}
	EXPORT_SYMBOL(__cv_wait_sig);

	void
	__cv_wait_idle(kcondvar_t cvp, kmutex_t mp)
	{
	sigset_t blocked, saved;

	sigfillset(&blocked);
	(void) sigprocmask(SIG_BLOCK, &blocked, &saved);
	cv_wait_common(cvp, mp, TASK_INTERRUPTIBLE, 0);
	(void) sigprocmask(SIG_SETMASK, &saved, NULL);
	}
	EXPORT_SYMBOL(__cv_wait_idle);

	#if defined(HAVE_IO_SCHEDULE_TIMEOUT)
	#define spl_io_schedule_timeout(t) io_schedule_timeout(t)
	#else

	struct spl_task_timer {
	struct timer_list timer;
	struct task_struct *task;
	};

	static void
	__cv_wakeup(spl_timer_list_t t)
	{
	struct timer_list tmr = (struct timer_list )t;
	struct spl_task_timer *task_timer = from_timer(task_timer, tmr, timer);

	wake_up_process(task_timer->task);
	}

	static long
	spl_io_schedule_timeout(long time_left)
	{
	long expire_time = jiffies + time_left;
	struct spl_task_timer task_timer;
	struct timer_list *timer = &task_timer.timer;

	task_timer.task = current;

	timer_setup(timer, __cv_wakeup, 0);

	timer->expires = expire_time;
	add_timer(timer);

	io_schedule();

	del_timer_sync(timer);

	time_left = expire_time - jiffies;

	return (time_left < 0 ? 0 : time_left);
	}
	#endif

	/*
	* 'expire_time' argument is an absolute wall clock time in jiffies.
	* Return value is time left (expire_time - now) or -1 if timeout occurred.
	*/
	static clock_t
	__cv_timedwait_common(kcondvar_t cvp, kmutex_t mp, clock_t expire_time,
	int state, int io)
	{
	DEFINE_WAIT(wait);
	kmutex_t *m;
	clock_t time_left;

	ASSERT(cvp);
	ASSERT(mp);
	ASSERT(cvp->cv_magic == CV_MAGIC);
	ASSERT(mutex_owned(mp));

	/* XXX - Does not handle jiffie wrap properly */
	time_left = expire_time - jiffies;
	if (time_left <= 0)
	return (-1);

	atomic_inc(&cvp->cv_refs);
	m = READ_ONCE(cvp->cv_mutex);
	if (!m)
	m = xchg(&cvp->cv_mutex, mp);
	/* Ensure the same mutex is used by all callers */
	ASSERT(m == NULL \|\| m == mp);

	prepare_to_wait_exclusive(&cvp->cv_event, &wait, state);
	atomic_inc(&cvp->cv_waiters);

	/*
	* Mutex should be dropped after prepare_to_wait() this
	* ensures we're linked in to the waiters list and avoids the
	* race where 'cvp->cv_waiters > 0' but the list is empty.
	*/
	mutex_exit(mp);
	if (io)
	time_left = spl_io_schedule_timeout(time_left);
	else
	time_left = schedule_timeout(time_left);

	/* No more waiters a different mutex could be used */
	if (atomic_dec_and_test(&cvp->cv_waiters)) {
	/*
	* This is set without any lock, so it's racy. But this is
	* just for debug anyway, so make it best-effort
	*/
	cvp->cv_mutex = NULL;
	wake_up(&cvp->cv_destroy);
	}

	finish_wait(&cvp->cv_event, &wait);
	atomic_dec(&cvp->cv_refs);

	/*
	* Hold mutex after we release the cvp, otherwise we could dead lock
	* with a thread holding the mutex and call cv_destroy.
	*/
	mutex_enter(mp);
	return (time_left > 0 ? 1 : -1);
	}

	int
	__cv_timedwait(kcondvar_t cvp, kmutex_t mp, clock_t exp_time)
	{
	return (__cv_timedwait_common(cvp, mp, exp_time,
	TASK_UNINTERRUPTIBLE, 0));
	}
	EXPORT_SYMBOL(__cv_timedwait);

	int
	__cv_timedwait_io(kcondvar_t cvp, kmutex_t mp, clock_t exp_time)
	{
	return (__cv_timedwait_common(cvp, mp, exp_time,
	TASK_UNINTERRUPTIBLE, 1));
	}
	EXPORT_SYMBOL(__cv_timedwait_io);

	int
	__cv_timedwait_sig(kcondvar_t cvp, kmutex_t mp, clock_t exp_time)
	{
	int rc;

	rc = __cv_timedwait_common(cvp, mp, exp_time, TASK_INTERRUPTIBLE, 0);
	return (signal_pending(current) ? 0 : rc);
	}
	EXPORT_SYMBOL(__cv_timedwait_sig);

	int
	__cv_timedwait_idle(kcondvar_t cvp, kmutex_t mp, clock_t exp_time)
	{
	sigset_t blocked, saved;
	int rc;

	sigfillset(&blocked);
	(void) sigprocmask(SIG_BLOCK, &blocked, &saved);
	rc = __cv_timedwait_common(cvp, mp, exp_time,
	TASK_INTERRUPTIBLE, 0);
	(void) sigprocmask(SIG_SETMASK, &saved, NULL);

	return (rc);
	}
	EXPORT_SYMBOL(__cv_timedwait_idle);
	/*
	* 'expire_time' argument is an absolute clock time in nanoseconds.
	* Return value is time left (expire_time - now) or -1 if timeout occurred.
	*/
	static clock_t
	__cv_timedwait_hires(kcondvar_t cvp, kmutex_t mp, hrtime_t expire_time,
	hrtime_t res, int state)
	{
	DEFINE_WAIT(wait);
	kmutex_t *m;
	hrtime_t time_left;
	ktime_t ktime_left;
	u64 slack = 0;
	int rc;

	ASSERT(cvp);
	ASSERT(mp);
	ASSERT(cvp->cv_magic == CV_MAGIC);
	ASSERT(mutex_owned(mp));

	time_left = expire_time - gethrtime();
	if (time_left <= 0)
	return (-1);

	atomic_inc(&cvp->cv_refs);
	m = READ_ONCE(cvp->cv_mutex);
	if (!m)
	m = xchg(&cvp->cv_mutex, mp);
	/* Ensure the same mutex is used by all callers */
	ASSERT(m == NULL \|\| m == mp);

	prepare_to_wait_exclusive(&cvp->cv_event, &wait, state);
	atomic_inc(&cvp->cv_waiters);

	/*
	* Mutex should be dropped after prepare_to_wait() this
	* ensures we're linked in to the waiters list and avoids the
	* race where 'cvp->cv_waiters > 0' but the list is empty.
	*/
	mutex_exit(mp);

	ktime_left = ktime_set(0, time_left);
	slack = MIN(MAX(res, spl_schedule_hrtimeout_slack_us * NSEC_PER_USEC),
	MAX_HRTIMEOUT_SLACK_US * NSEC_PER_USEC);
	rc = schedule_hrtimeout_range(&ktime_left, slack, HRTIMER_MODE_REL);

	/* No more waiters a different mutex could be used */
	if (atomic_dec_and_test(&cvp->cv_waiters)) {
	/*
	* This is set without any lock, so it's racy. But this is
	* just for debug anyway, so make it best-effort
	*/
	cvp->cv_mutex = NULL;
	wake_up(&cvp->cv_destroy);
	}

	finish_wait(&cvp->cv_event, &wait);
	atomic_dec(&cvp->cv_refs);

	mutex_enter(mp);
	return (rc == -EINTR ? 1 : -1);
	}

	/*
	* Compatibility wrapper for the cv_timedwait_hires() Illumos interface.
	*/
	static int
	cv_timedwait_hires_common(kcondvar_t cvp, kmutex_t mp, hrtime_t tim,
	hrtime_t res, int flag, int state)
	{
	if (!(flag & CALLOUT_FLAG_ABSOLUTE))
	tim += gethrtime();

	return (__cv_timedwait_hires(cvp, mp, tim, res, state));
	}

	int
	cv_timedwait_hires(kcondvar_t cvp, kmutex_t mp, hrtime_t tim, hrtime_t res,
	int flag)
	{
	return (cv_timedwait_hires_common(cvp, mp, tim, res, flag,
	TASK_UNINTERRUPTIBLE));
	}
	EXPORT_SYMBOL(cv_timedwait_hires);

	int
	cv_timedwait_sig_hires(kcondvar_t cvp, kmutex_t mp, hrtime_t tim,
	hrtime_t res, int flag)
	{
	int rc;

	rc = cv_timedwait_hires_common(cvp, mp, tim, res, flag,
	TASK_INTERRUPTIBLE);
	return (signal_pending(current) ? 0 : rc);
	}
	EXPORT_SYMBOL(cv_timedwait_sig_hires);

	int
	cv_timedwait_idle_hires(kcondvar_t cvp, kmutex_t mp, hrtime_t tim,
	hrtime_t res, int flag)
	{
	sigset_t blocked, saved;
	int rc;

	sigfillset(&blocked);
	(void) sigprocmask(SIG_BLOCK, &blocked, &saved);
	rc = cv_timedwait_hires_common(cvp, mp, tim, res, flag,
	TASK_INTERRUPTIBLE);
	(void) sigprocmask(SIG_SETMASK, &saved, NULL);

	return (rc);
	}
	EXPORT_SYMBOL(cv_timedwait_idle_hires);

	void
	__cv_signal(kcondvar_t *cvp)
	{
	ASSERT(cvp);
	ASSERT(cvp->cv_magic == CV_MAGIC);
	atomic_inc(&cvp->cv_refs);

	/*
	* All waiters are added with WQ_FLAG_EXCLUSIVE so only one
	* waiter will be set runnable with each call to wake_up().
	* Additionally wake_up() holds a spin_lock associated with
	* the wait queue to ensure we don't race waking up processes.
	*/
	if (atomic_read(&cvp->cv_waiters) > 0)
	wake_up(&cvp->cv_event);

	atomic_dec(&cvp->cv_refs);
	}
	EXPORT_SYMBOL(__cv_signal);

	void
	__cv_broadcast(kcondvar_t *cvp)
	{
	ASSERT(cvp);
	ASSERT(cvp->cv_magic == CV_MAGIC);
	atomic_inc(&cvp->cv_refs);

	/*
	* Wake_up_all() will wake up all waiters even those which
	* have the WQ_FLAG_EXCLUSIVE flag set.
	*/
	if (atomic_read(&cvp->cv_waiters) > 0)
	wake_up_all(&cvp->cv_event);

	atomic_dec(&cvp->cv_refs);
	}
	EXPORT_SYMBOL(__cv_broadcast);
	Index: vendor-sys/openzfs/dist/module/os/linux/spl/spl-cred.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/spl/spl-cred.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/spl/spl-cred.c (revision 366774)
	@@ -1,196 +1,195 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*
	* Solaris Porting Layer (SPL) Credential Implementation.
	*/

	#include <sys/cred.h>

	static int
	cr_groups_search(const struct group_info *group_info, kgid_t grp)
	{
	unsigned int left, right, mid;
	int cmp;

	if (!group_info)
	return (0);

	left = 0;
	right = group_info->ngroups;
	while (left < right) {
	mid = (left + right) / 2;
	cmp = KGID_TO_SGID(grp) -
	KGID_TO_SGID(GROUP_AT(group_info, mid));

	if (cmp > 0)
	left = mid + 1;
	else if (cmp < 0)
	right = mid;
	else
	return (1);
	}
	return (0);
	}

	/* Hold a reference on the credential */
	void
	crhold(cred_t *cr)
	{
	(void) get_cred((const cred_t *)cr);
	}

	/* Free a reference on the credential */
	void
	crfree(cred_t *cr)
	{
	put_cred((const cred_t *)cr);
	}

	/* Return the number of supplemental groups */
	int
	crgetngroups(const cred_t *cr)
	{
	struct group_info *gi;
	int rc;

	gi = cr->group_info;
	rc = gi->ngroups;
	#ifndef HAVE_GROUP_INFO_GID
	/*
	* For Linux <= 4.8,
	* crgetgroups will only returns gi->blocks[0], which contains only
	* the first NGROUPS_PER_BLOCK groups.
	*/
	if (rc > NGROUPS_PER_BLOCK) {
	WARN_ON_ONCE(1);
	rc = NGROUPS_PER_BLOCK;
	}
	#endif
	return (rc);
	}

	/*
	* Return an array of supplemental gids. The returned address is safe
	* to use as long as the caller has taken a reference with crhold().
	*
	* Linux 4.9 API change, group_info changed from 2d array via ->blocks to 1d
	* array via ->gid.
	*/
	gid_t *
	crgetgroups(const cred_t *cr)
	{
	struct group_info *gi;
	gid_t *gids = NULL;

	gi = cr->group_info;
	#ifdef HAVE_GROUP_INFO_GID
	gids = KGIDP_TO_SGIDP(gi->gid);
	#else
	if (gi->nblocks > 0)
	gids = KGIDP_TO_SGIDP(gi->blocks[0]);
	#endif
	return (gids);
	}

	/* Check if the passed gid is available in supplied credential. */
	int
	groupmember(gid_t gid, const cred_t *cr)
	{
	struct group_info *gi;
	int rc;

	gi = cr->group_info;
	rc = cr_groups_search(gi, SGID_TO_KGID(gid));

	return (rc);
	}

	/* Return the effective user id */
	uid_t
	crgetuid(const cred_t *cr)
	{
	return (KUID_TO_SUID(cr->euid));
	}

	/* Return the real user id */
	uid_t
	crgetruid(const cred_t *cr)
	{
	return (KUID_TO_SUID(cr->uid));
	}

	/* Return the saved user id */
	uid_t
	crgetsuid(const cred_t *cr)
	{
	return (KUID_TO_SUID(cr->suid));
	}

	/* Return the filesystem user id */
	uid_t
	crgetfsuid(const cred_t *cr)
	{
	return (KUID_TO_SUID(cr->fsuid));
	}

	/* Return the effective group id */
	gid_t
	crgetgid(const cred_t *cr)
	{
	return (KGID_TO_SGID(cr->egid));
	}

	/* Return the real group id */
	gid_t
	crgetrgid(const cred_t *cr)
	{
	return (KGID_TO_SGID(cr->gid));
	}

	/* Return the saved group id */
	gid_t
	crgetsgid(const cred_t *cr)
	{
	return (KGID_TO_SGID(cr->sgid));
	}

	/* Return the filesystem group id */
	gid_t
	crgetfsgid(const cred_t *cr)
	{
	return (KGID_TO_SGID(cr->fsgid));
	}

	EXPORT_SYMBOL(crhold);
	EXPORT_SYMBOL(crfree);
	EXPORT_SYMBOL(crgetuid);
	EXPORT_SYMBOL(crgetruid);
	EXPORT_SYMBOL(crgetsuid);
	EXPORT_SYMBOL(crgetfsuid);
	EXPORT_SYMBOL(crgetgid);
	EXPORT_SYMBOL(crgetrgid);
	EXPORT_SYMBOL(crgetsgid);
	EXPORT_SYMBOL(crgetfsgid);
	EXPORT_SYMBOL(crgetngroups);
	EXPORT_SYMBOL(crgetgroups);
	EXPORT_SYMBOL(groupmember);
	Index: vendor-sys/openzfs/dist/module/os/linux/spl/spl-err.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/spl/spl-err.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/spl/spl-err.c (revision 366774)
	@@ -1,124 +1,123 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*
	* Solaris Porting Layer (SPL) Error Implementation.
	*/

	#include <sys/sysmacros.h>
	#include <sys/cmn_err.h>

	/*
	* It is often useful to actually have the panic crash the node so you
	* can then get notified of the event, get the crashdump for later
	* analysis and other such goodies.
	* But we would still default to the current default of not to do that.
	*/
	/* BEGIN CSTYLED */
	unsigned int spl_panic_halt;
	module_param(spl_panic_halt, uint, 0644);
	MODULE_PARM_DESC(spl_panic_halt, "Cause kernel panic on assertion failures");
	/* END CSTYLED */

	void
	spl_dumpstack(void)
	{
	printk("Showing stack for process %d\n", current->pid);
	dump_stack();
	}
	EXPORT_SYMBOL(spl_dumpstack);

	int
	spl_panic(const char file, const char func, int line, const char *fmt, ...)
	{
	const char *newfile;
	char msg[MAXMSGLEN];
	va_list ap;

	newfile = strrchr(file, '/');
	if (newfile != NULL)
	newfile = newfile + 1;
	else
	newfile = file;

	va_start(ap, fmt);
	(void) vsnprintf(msg, sizeof (msg), fmt, ap);
	va_end(ap);

	printk(KERN_EMERG "%s", msg);
	printk(KERN_EMERG "PANIC at %s:%d:%s()\n", newfile, line, func);
	if (spl_panic_halt)
	panic("%s", msg);

	spl_dumpstack();

	/* Halt the thread to facilitate further debugging */
	set_current_state(TASK_UNINTERRUPTIBLE);
	while (1)
	schedule();

	/* Unreachable */
	return (1);
	}
	EXPORT_SYMBOL(spl_panic);

	void
	vcmn_err(int ce, const char *fmt, va_list ap)
	{
	char msg[MAXMSGLEN];

	vsnprintf(msg, MAXMSGLEN, fmt, ap);

	switch (ce) {
	case CE_IGNORE:
	break;
	case CE_CONT:
	printk("%s", msg);
	break;
	case CE_NOTE:
	printk(KERN_NOTICE "NOTICE: %s\n", msg);
	break;
	case CE_WARN:
	printk(KERN_WARNING "WARNING: %s\n", msg);
	break;
	case CE_PANIC:
	printk(KERN_EMERG "PANIC: %s\n", msg);
	spl_dumpstack();

	/* Halt the thread to facilitate further debugging */
	set_current_state(TASK_UNINTERRUPTIBLE);
	while (1)
	schedule();
	}
	} /* vcmn_err() */
	EXPORT_SYMBOL(vcmn_err);

	void
	cmn_err(int ce, const char *fmt, ...)
	{
	va_list ap;

	va_start(ap, fmt);
	vcmn_err(ce, fmt, ap);
	va_end(ap);
	} /* cmn_err() */
	EXPORT_SYMBOL(cmn_err);
	Index: vendor-sys/openzfs/dist/module/os/linux/spl/spl-generic.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/spl/spl-generic.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/spl/spl-generic.c (revision 366774)
	@@ -1,844 +1,843 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*
	* Solaris Porting Layer (SPL) Generic Implementation.
	*/

	#include <sys/sysmacros.h>
	#include <sys/systeminfo.h>
	#include <sys/vmsystm.h>
	#include <sys/kmem.h>
	#include <sys/kmem_cache.h>
	#include <sys/vmem.h>
	#include <sys/mutex.h>
	#include <sys/rwlock.h>
	#include <sys/taskq.h>
	#include <sys/tsd.h>
	#include <sys/zmod.h>
	#include <sys/debug.h>
	#include <sys/proc.h>
	#include <sys/kstat.h>
	#include <sys/file.h>
	#include <sys/sunddi.h>
	#include <linux/ctype.h>
	#include <sys/disp.h>
	#include <sys/random.h>
	#include <sys/strings.h>
	#include <linux/kmod.h>
	#include "zfs_gitrev.h"
	#include <linux/mod_compat.h>
	#include <sys/cred.h>
	#include <sys/vnode.h>

	char spl_gitrev[64] = ZFS_META_GITREV;

	/* BEGIN CSTYLED */
	unsigned long spl_hostid = 0;
	EXPORT_SYMBOL(spl_hostid);
	/* BEGIN CSTYLED */
	module_param(spl_hostid, ulong, 0644);
	MODULE_PARM_DESC(spl_hostid, "The system hostid.");
	/* END CSTYLED */

	proc_t p0;
	EXPORT_SYMBOL(p0);

	/*
	* Xorshift Pseudo Random Number Generator based on work by Sebastiano Vigna
	*
	* "Further scramblings of Marsaglia's xorshift generators"
	* http://vigna.di.unimi.it/ftp/papers/xorshiftplus.pdf
	*
	* random_get_pseudo_bytes() is an API function on Illumos whose sole purpose
	* is to provide bytes containing random numbers. It is mapped to /dev/urandom
	* on Illumos, which uses a "FIPS 186-2 algorithm". No user of the SPL's
	* random_get_pseudo_bytes() needs bytes that are of cryptographic quality, so
	* we can implement it using a fast PRNG that we seed using Linux' actual
	* equivalent to random_get_pseudo_bytes(). We do this by providing each CPU
	* with an independent seed so that all calls to random_get_pseudo_bytes() are
	* free of atomic instructions.
	*
	* A consequence of using a fast PRNG is that using random_get_pseudo_bytes()
	* to generate words larger than 128 bits will paradoxically be limited to
	* `2^128 - 1` possibilities. This is because we have a sequence of `2^128 - 1`
	* 128-bit words and selecting the first will implicitly select the second. If
	* a caller finds this behavior undesirable, random_get_bytes() should be used
	* instead.
	*
	* XXX: Linux interrupt handlers that trigger within the critical section
	* formed by `s[1] = xp[1];` and `xp[0] = s[0];` and call this function will
	* see the same numbers. Nothing in the code currently calls this in an
	* interrupt handler, so this is considered to be okay. If that becomes a
	* problem, we could create a set of per-cpu variables for interrupt handlers
	* and use them when in_interrupt() from linux/preempt_mask.h evaluates to
	* true.
	*/
	void __percpu *spl_pseudo_entropy;

	/*
	* spl_rand_next()/spl_rand_jump() are copied from the following CC-0 licensed
	* file:
	*
	* http://xorshift.di.unimi.it/xorshift128plus.c
	*/

	static inline uint64_t
	spl_rand_next(uint64_t *s)
	{
	uint64_t s1 = s[0];
	const uint64_t s0 = s[1];
	s[0] = s0;
	s1 ^= s1 << 23; // a
	s[1] = s1 ^ s0 ^ (s1 >> 18) ^ (s0 >> 5); // b, c
	return (s[1] + s0);
	}

	static inline void
	spl_rand_jump(uint64_t *s)
	{
	static const uint64_t JUMP[] =
	{ 0x8a5cd789635d2dff, 0x121fd2155c472f96 };

	uint64_t s0 = 0;
	uint64_t s1 = 0;
	int i, b;
	for (i = 0; i < sizeof (JUMP) / sizeof (*JUMP); i++)
	for (b = 0; b < 64; b++) {
	if (JUMP[i] & 1ULL << b) {
	s0 ^= s[0];
	s1 ^= s[1];
	}
	(void) spl_rand_next(s);
	}

	s[0] = s0;
	s[1] = s1;
	}

	int
	random_get_pseudo_bytes(uint8_t *ptr, size_t len)
	{
	uint64_t *xp, s[2];

	ASSERT(ptr);

	xp = get_cpu_ptr(spl_pseudo_entropy);

	s[0] = xp[0];
	s[1] = xp[1];

	while (len) {
	union {
	uint64_t ui64;
	uint8_t byte[sizeof (uint64_t)];
	}entropy;
	int i = MIN(len, sizeof (uint64_t));

	len -= i;
	entropy.ui64 = spl_rand_next(s);

	while (i--)
	*ptr++ = entropy.byte[i];
	}

	xp[0] = s[0];
	xp[1] = s[1];

	put_cpu_ptr(spl_pseudo_entropy);

	return (0);
	}


	EXPORT_SYMBOL(random_get_pseudo_bytes);

	#if BITS_PER_LONG == 32

	/*
	* Support 64/64 => 64 division on a 32-bit platform. While the kernel
	* provides a div64_u64() function for this we do not use it because the
	* implementation is flawed. There are cases which return incorrect
	* results as late as linux-2.6.35. Until this is fixed upstream the
	* spl must provide its own implementation.
	*
	* This implementation is a slightly modified version of the algorithm
	* proposed by the book 'Hacker's Delight'. The original source can be
	* found here and is available for use without restriction.
	*
	* http://www.hackersdelight.org/HDcode/newCode/divDouble.c
	*/

	/*
	* Calculate number of leading of zeros for a 64-bit value.
	*/
	static int
	nlz64(uint64_t x)
	{
	register int n = 0;

	if (x == 0)
	return (64);

	if (x <= 0x00000000FFFFFFFFULL) { n = n + 32; x = x << 32; }
	if (x <= 0x0000FFFFFFFFFFFFULL) { n = n + 16; x = x << 16; }
	if (x <= 0x00FFFFFFFFFFFFFFULL) { n = n + 8; x = x << 8; }
	if (x <= 0x0FFFFFFFFFFFFFFFULL) { n = n + 4; x = x << 4; }
	if (x <= 0x3FFFFFFFFFFFFFFFULL) { n = n + 2; x = x << 2; }
	if (x <= 0x7FFFFFFFFFFFFFFFULL) { n = n + 1; }

	return (n);
	}

	/*
	* Newer kernels have a div_u64() function but we define our own
	* to simplify portability between kernel versions.
	*/
	static inline uint64_t
	__div_u64(uint64_t u, uint32_t v)
	{
	(void) do_div(u, v);
	return (u);
	}

	/*
	* Turn off missing prototypes warning for these functions. They are
	* replacements for libgcc-provided functions and will never be called
	* directly.
	*/
	#pragma GCC diagnostic push
	#pragma GCC diagnostic ignored "-Wmissing-prototypes"

	/*
	* Implementation of 64-bit unsigned division for 32-bit machines.
	*
	* First the procedure takes care of the case in which the divisor is a
	* 32-bit quantity. There are two subcases: (1) If the left half of the
	* dividend is less than the divisor, one execution of do_div() is all that
	* is required (overflow is not possible). (2) Otherwise it does two
	* divisions, using the grade school method.
	*/
	uint64_t
	__udivdi3(uint64_t u, uint64_t v)
	{
	uint64_t u0, u1, v1, q0, q1, k;
	int n;

	if (v >> 32 == 0) { // If v < 2**32:
	if (u >> 32 < v) { // If u/v cannot overflow,
	return (__div_u64(u, v)); // just do one division.
	} else { // If u/v would overflow:
	u1 = u >> 32; // Break u into two halves.
	u0 = u & 0xFFFFFFFF;
	q1 = __div_u64(u1, v); // First quotient digit.
	k = u1 - q1 * v; // First remainder, < v.
	u0 += (k << 32);
	q0 = __div_u64(u0, v); // Seconds quotient digit.
	return ((q1 << 32) + q0);
	}
	} else { // If v >= 2**32:
	n = nlz64(v); // 0 <= n <= 31.
	v1 = (v << n) >> 32; // Normalize divisor, MSB is 1.
	u1 = u >> 1; // To ensure no overflow.
	q1 = __div_u64(u1, v1); // Get quotient from
	q0 = (q1 << n) >> 31; // Undo normalization and
	// division of u by 2.
	if (q0 != 0) // Make q0 correct or
	q0 = q0 - 1; // too small by 1.
	if ((u - q0 * v) >= v)
	q0 = q0 + 1; // Now q0 is correct.

	return (q0);
	}
	}
	EXPORT_SYMBOL(__udivdi3);

	/* BEGIN CSTYLED */
	#ifndef abs64
	#define abs64(x) ({ uint64_t t = (x) >> 63; ((x) ^ t) - t; })
	#endif
	/* END CSTYLED */

	/*
	* Implementation of 64-bit signed division for 32-bit machines.
	*/
	int64_t
	__divdi3(int64_t u, int64_t v)
	{
	int64_t q, t;
	// cppcheck-suppress shiftTooManyBitsSigned
	q = __udivdi3(abs64(u), abs64(v));
	// cppcheck-suppress shiftTooManyBitsSigned
	t = (u ^ v) >> 63; // If u, v have different
	return ((q ^ t) - t); // signs, negate q.
	}
	EXPORT_SYMBOL(__divdi3);

	/*
	* Implementation of 64-bit unsigned modulo for 32-bit machines.
	*/
	uint64_t
	__umoddi3(uint64_t dividend, uint64_t divisor)
	{
	return (dividend - (divisor * __udivdi3(dividend, divisor)));
	}
	EXPORT_SYMBOL(__umoddi3);

	/* 64-bit signed modulo for 32-bit machines. */
	int64_t
	__moddi3(int64_t n, int64_t d)
	{
	int64_t q;
	boolean_t nn = B_FALSE;

	if (n < 0) {
	nn = B_TRUE;
	n = -n;
	}
	if (d < 0)
	d = -d;

	q = __umoddi3(n, d);

	return (nn ? -q : q);
	}
	EXPORT_SYMBOL(__moddi3);

	/*
	* Implementation of 64-bit unsigned division/modulo for 32-bit machines.
	*/
	uint64_t
	__udivmoddi4(uint64_t n, uint64_t d, uint64_t *r)
	{
	uint64_t q = __udivdi3(n, d);
	if (r)
	r = n - d q;
	return (q);
	}
	EXPORT_SYMBOL(__udivmoddi4);

	/*
	* Implementation of 64-bit signed division/modulo for 32-bit machines.
	*/
	int64_t
	__divmoddi4(int64_t n, int64_t d, int64_t *r)
	{
	int64_t q, rr;
	boolean_t nn = B_FALSE;
	boolean_t nd = B_FALSE;
	if (n < 0) {
	nn = B_TRUE;
	n = -n;
	}
	if (d < 0) {
	nd = B_TRUE;
	d = -d;
	}

	q = __udivmoddi4(n, d, (uint64_t *)&rr);

	if (nn != nd)
	q = -q;
	if (nn)
	rr = -rr;
	if (r)
	*r = rr;
	return (q);
	}
	EXPORT_SYMBOL(__divmoddi4);

	#if defined(__arm) \|\| defined(__arm__)
	/*
	* Implementation of 64-bit (un)signed division for 32-bit arm machines.
	*
	* Run-time ABI for the ARM Architecture (page 20). A pair of (unsigned)
	* long longs is returned in {{r0, r1}, {r2,r3}}, the quotient in {r0, r1},
	* and the remainder in {r2, r3}. The return type is specifically left
	* set to 'void' to ensure the compiler does not overwrite these registers
	* during the return. All results are in registers as per ABI
	*/
	void
	__aeabi_uldivmod(uint64_t u, uint64_t v)
	{
	uint64_t res;
	uint64_t mod;

	res = __udivdi3(u, v);
	mod = __umoddi3(u, v);
	{
	register uint32_t r0 asm("r0") = (res & 0xFFFFFFFF);
	register uint32_t r1 asm("r1") = (res >> 32);
	register uint32_t r2 asm("r2") = (mod & 0xFFFFFFFF);
	register uint32_t r3 asm("r3") = (mod >> 32);

	/* BEGIN CSTYLED */
	asm volatile(""
	: "+r"(r0), "+r"(r1), "+r"(r2),"+r"(r3) /* output */
	: "r"(r0), "r"(r1), "r"(r2), "r"(r3)); /* input */
	/* END CSTYLED */

	return; /* r0; */
	}
	}
	EXPORT_SYMBOL(__aeabi_uldivmod);

	void
	__aeabi_ldivmod(int64_t u, int64_t v)
	{
	int64_t res;
	uint64_t mod;

	res = __divdi3(u, v);
	mod = __umoddi3(u, v);
	{
	register uint32_t r0 asm("r0") = (res & 0xFFFFFFFF);
	register uint32_t r1 asm("r1") = (res >> 32);
	register uint32_t r2 asm("r2") = (mod & 0xFFFFFFFF);
	register uint32_t r3 asm("r3") = (mod >> 32);

	/* BEGIN CSTYLED */
	asm volatile(""
	: "+r"(r0), "+r"(r1), "+r"(r2),"+r"(r3) /* output */
	: "r"(r0), "r"(r1), "r"(r2), "r"(r3)); /* input */
	/* END CSTYLED */

	return; /* r0; */
	}
	}
	EXPORT_SYMBOL(__aeabi_ldivmod);
	#endif /* __arm \|\| __arm__ */

	#pragma GCC diagnostic pop

	#endif /* BITS_PER_LONG */

	/*
	* NOTE: The strtoxx behavior is solely based on my reading of the Solaris
	* ddi_strtol(9F) man page. I have not verified the behavior of these
	* functions against their Solaris counterparts. It is possible that I
	* may have misinterpreted the man page or the man page is incorrect.
	*/
	int ddi_strtoul(const char , char , int, unsigned long );
	int ddi_strtol(const char , char , int, long );
	int ddi_strtoull(const char , char , int, unsigned long long );
	int ddi_strtoll(const char , char , int, long long );

	#define define_ddi_strtoux(type, valtype) \
	int ddi_strtou##type(const char str, char *endptr, \
	int base, valtype *result) \
	{ \
	valtype last_value, value = 0; \
	char ptr = (char )str; \
	int flag = 1, digit; \
	\
	if (strlen(ptr) == 0) \
	return (EINVAL); \
	\
	/* Auto-detect base based on prefix */ \
	if (!base) { \
	if (str[0] == '0') { \
	if (tolower(str[1]) == 'x' && isxdigit(str[2])) { \
	base = 16; /* hex */ \
	ptr += 2; \
	} else if (str[1] >= '0' && str[1] < 8) { \
	base = 8; /* octal */ \
	ptr += 1; \
	} else { \
	return (EINVAL); \
	} \
	} else { \
	base = 10; /* decimal */ \
	} \
	} \
	\
	while (1) { \
	if (isdigit(*ptr)) \
	digit = *ptr - '0'; \
	else if (isalpha(*ptr)) \
	digit = tolower(*ptr) - 'a' + 10; \
	else \
	break; \
	\
	if (digit >= base) \
	break; \
	\
	last_value = value; \
	value = value * base + digit; \
	if (last_value > value) /* Overflow */ \
	return (ERANGE); \
	\
	flag = 1; \
	ptr++; \
	} \
	\
	if (flag) \
	*result = value; \
	\
	if (endptr) \
	endptr = (char )(flag ? ptr : str); \
	\
	return (0); \
	} \

	#define define_ddi_strtox(type, valtype) \
	int ddi_strto##type(const char str, char *endptr, \
	int base, valtype *result) \
	{ \
	int rc; \
	\
	if (*str == '-') { \
	rc = ddi_strtou##type(str + 1, endptr, base, result); \
	if (!rc) { \
	if (*endptr == str + 1) \
	endptr = (char )str; \
	else \
	result = -result; \
	} \
	} else { \
	rc = ddi_strtou##type(str, endptr, base, result); \
	} \
	\
	return (rc); \
	}

	define_ddi_strtoux(l, unsigned long)
	define_ddi_strtox(l, long)
	define_ddi_strtoux(ll, unsigned long long)
	define_ddi_strtox(ll, long long)

	EXPORT_SYMBOL(ddi_strtoul);
	EXPORT_SYMBOL(ddi_strtol);
	EXPORT_SYMBOL(ddi_strtoll);
	EXPORT_SYMBOL(ddi_strtoull);

	int
	ddi_copyin(const void from, void to, size_t len, int flags)
	{
	/* Fake ioctl() issued by kernel, 'from' is a kernel address */
	if (flags & FKIOCTL) {
	memcpy(to, from, len);
	return (0);
	}

	return (copyin(from, to, len));
	}
	EXPORT_SYMBOL(ddi_copyin);

	int
	ddi_copyout(const void from, void to, size_t len, int flags)
	{
	/* Fake ioctl() issued by kernel, 'from' is a kernel address */
	if (flags & FKIOCTL) {
	memcpy(to, from, len);
	return (0);
	}

	return (copyout(from, to, len));
	}
	EXPORT_SYMBOL(ddi_copyout);

	static ssize_t
	spl_kernel_read(struct file file, void buf, size_t count, loff_t *pos)
	{
	#if defined(HAVE_KERNEL_READ_PPOS)
	return (kernel_read(file, buf, count, pos));
	#else
	mm_segment_t saved_fs;
	ssize_t ret;

	saved_fs = get_fs();
	set_fs(KERNEL_DS);

	ret = vfs_read(file, (void __user *)buf, count, pos);

	set_fs(saved_fs);

	return (ret);
	#endif
	}

	static int
	spl_getattr(struct file filp, struct kstat stat)
	{
	int rc;

	ASSERT(filp);
	ASSERT(stat);

	#if defined(HAVE_4ARGS_VFS_GETATTR)
	rc = vfs_getattr(&filp->f_path, stat, STATX_BASIC_STATS,
	AT_STATX_SYNC_AS_STAT);
	#elif defined(HAVE_2ARGS_VFS_GETATTR)
	rc = vfs_getattr(&filp->f_path, stat);
	#else
	rc = vfs_getattr(filp->f_path.mnt, filp->f_dentry, stat);
	#endif
	if (rc)
	return (-rc);

	return (0);
	}

	/*
	* Read the unique system identifier from the /etc/hostid file.
	*
	* The behavior of /usr/bin/hostid on Linux systems with the
	* regular eglibc and coreutils is:
	*
	* 1. Generate the value if the /etc/hostid file does not exist
	* or if the /etc/hostid file is less than four bytes in size.
	*
	* 2. If the /etc/hostid file is at least 4 bytes, then return
	* the first four bytes [0..3] in native endian order.
	*
	* 3. Always ignore bytes [4..] if they exist in the file.
	*
	* Only the first four bytes are significant, even on systems that
	* have a 64-bit word size.
	*
	* See:
	*
	* eglibc: sysdeps/unix/sysv/linux/gethostid.c
	* coreutils: src/hostid.c
	*
	* Notes:
	*
	* The /etc/hostid file on Solaris is a text file that often reads:
	*
	* # DO NOT EDIT
	* "0123456789"
	*
	* Directly copying this file to Linux results in a constant
	* hostid of 4f442023 because the default comment constitutes
	* the first four bytes of the file.
	*
	*/

	char *spl_hostid_path = HW_HOSTID_PATH;
	module_param(spl_hostid_path, charp, 0444);
	MODULE_PARM_DESC(spl_hostid_path, "The system hostid file (/etc/hostid)");

	static int
	hostid_read(uint32_t *hostid)
	{
	uint64_t size;
	uint32_t value = 0;
	int error;
	loff_t off;
	struct file *filp;
	struct kstat stat;

	filp = filp_open(spl_hostid_path, 0, 0);

	if (IS_ERR(filp))
	return (ENOENT);

	error = spl_getattr(filp, &stat);
	if (error) {
	filp_close(filp, 0);
	return (error);
	}
	size = stat.size;
	if (size < sizeof (HW_HOSTID_MASK)) {
	filp_close(filp, 0);
	return (EINVAL);
	}

	off = 0;
	/*
	* Read directly into the variable like eglibc does.
	* Short reads are okay; native behavior is preserved.
	*/
	error = spl_kernel_read(filp, &value, sizeof (value), &off);
	if (error < 0) {
	filp_close(filp, 0);
	return (EIO);
	}

	/* Mask down to 32 bits like coreutils does. */
	*hostid = (value & HW_HOSTID_MASK);
	filp_close(filp, 0);

	return (0);
	}

	/*
	* Return the system hostid. Preferentially use the spl_hostid module option
	* when set, otherwise use the value in the /etc/hostid file.
	*/
	uint32_t
	zone_get_hostid(void *zone)
	{
	uint32_t hostid;

	ASSERT3P(zone, ==, NULL);

	if (spl_hostid != 0)
	return ((uint32_t)(spl_hostid & HW_HOSTID_MASK));

	if (hostid_read(&hostid) == 0)
	return (hostid);

	return (0);
	}
	EXPORT_SYMBOL(zone_get_hostid);

	static int
	spl_kvmem_init(void)
	{
	int rc = 0;

	rc = spl_kmem_init();
	if (rc)
	return (rc);

	rc = spl_vmem_init();
	if (rc) {
	spl_kmem_fini();
	return (rc);
	}

	return (rc);
	}

	/*
	* We initialize the random number generator with 128 bits of entropy from the
	* system random number generator. In the improbable case that we have a zero
	* seed, we fallback to the system jiffies, unless it is also zero, in which
	* situation we use a preprogrammed seed. We step forward by 2^64 iterations to
	* initialize each of the per-cpu seeds so that the sequences generated on each
	* CPU are guaranteed to never overlap in practice.
	*/
	static void __init
	spl_random_init(void)
	{
	uint64_t s[2];
	int i = 0;

	spl_pseudo_entropy = __alloc_percpu(2 * sizeof (uint64_t),
	sizeof (uint64_t));

	get_random_bytes(s, sizeof (s));

	if (s[0] == 0 && s[1] == 0) {
	if (jiffies != 0) {
	s[0] = jiffies;
	s[1] = ~0 - jiffies;
	} else {
	(void) memcpy(s, "improbable seed", sizeof (s));
	}
	printk("SPL: get_random_bytes() returned 0 "
	"when generating random seed. Setting initial seed to "
	"0x%016llx%016llx.\n", cpu_to_be64(s[0]),
	cpu_to_be64(s[1]));
	}

	for_each_possible_cpu(i) {
	uint64_t *wordp = per_cpu_ptr(spl_pseudo_entropy, i);

	spl_rand_jump(s);

	wordp[0] = s[0];
	wordp[1] = s[1];
	}
	}

	static void
	spl_random_fini(void)
	{
	free_percpu(spl_pseudo_entropy);
	}

	static void
	spl_kvmem_fini(void)
	{
	spl_vmem_fini();
	spl_kmem_fini();
	}

	static int __init
	spl_init(void)
	{
	int rc = 0;

	bzero(&p0, sizeof (proc_t));
	spl_random_init();

	if ((rc = spl_kvmem_init()))
	goto out1;

	if ((rc = spl_tsd_init()))
	goto out2;

	if ((rc = spl_taskq_init()))
	goto out3;

	if ((rc = spl_kmem_cache_init()))
	goto out4;

	if ((rc = spl_proc_init()))
	goto out5;

	if ((rc = spl_kstat_init()))
	goto out6;

	if ((rc = spl_zlib_init()))
	goto out7;

	return (rc);

	out7:
	spl_kstat_fini();
	out6:
	spl_proc_fini();
	out5:
	spl_kmem_cache_fini();
	out4:
	spl_taskq_fini();
	out3:
	spl_tsd_fini();
	out2:
	spl_kvmem_fini();
	out1:
	return (rc);
	}

	static void __exit
	spl_fini(void)
	{
	spl_zlib_fini();
	spl_kstat_fini();
	spl_proc_fini();
	spl_kmem_cache_fini();
	spl_taskq_fini();
	spl_tsd_fini();
	spl_kvmem_fini();
	spl_random_fini();
	}

	module_init(spl_init);
	module_exit(spl_fini);

	ZFS_MODULE_DESCRIPTION("Solaris Porting Layer");
	ZFS_MODULE_AUTHOR(ZFS_META_AUTHOR);
	ZFS_MODULE_LICENSE("GPL");
	ZFS_MODULE_VERSION(ZFS_META_VERSION "-" ZFS_META_RELEASE);
	Index: vendor-sys/openzfs/dist/module/os/linux/spl/spl-kmem-cache.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/spl/spl-kmem-cache.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/spl/spl-kmem-cache.c (revision 366774)
	@@ -1,1469 +1,1468 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#include <linux/percpu_compat.h>
	#include <sys/kmem.h>
	#include <sys/kmem_cache.h>
	#include <sys/taskq.h>
	#include <sys/timer.h>
	#include <sys/vmem.h>
	#include <sys/wait.h>
	#include <linux/slab.h>
	#include <linux/swap.h>
	#include <linux/prefetch.h>

	/*
	* Within the scope of spl-kmem.c file the kmem_cache_* definitions
	* are removed to allow access to the real Linux slab allocator.
	*/
	#undef kmem_cache_destroy
	#undef kmem_cache_create
	#undef kmem_cache_alloc
	#undef kmem_cache_free


	/*
	* Linux 3.16 replaced smp_mb__{before,after}_{atomic,clear}_{dec,inc,bit}()
	* with smp_mb__{before,after}_atomic() because they were redundant. This is
	* only used inside our SLAB allocator, so we implement an internal wrapper
	* here to give us smp_mb__{before,after}_atomic() on older kernels.
	*/
	#ifndef smp_mb__before_atomic
	#define smp_mb__before_atomic(x) smp_mb__before_clear_bit(x)
	#endif

	#ifndef smp_mb__after_atomic
	#define smp_mb__after_atomic(x) smp_mb__after_clear_bit(x)
	#endif

	/* BEGIN CSTYLED */

	/*
	* Cache magazines are an optimization designed to minimize the cost of
	* allocating memory. They do this by keeping a per-cpu cache of recently
	* freed objects, which can then be reallocated without taking a lock. This
	* can improve performance on highly contended caches. However, because
	* objects in magazines will prevent otherwise empty slabs from being
	* immediately released this may not be ideal for low memory machines.
	*
	* For this reason spl_kmem_cache_magazine_size can be used to set a maximum
	* magazine size. When this value is set to 0 the magazine size will be
	* automatically determined based on the object size. Otherwise magazines
	* will be limited to 2-256 objects per magazine (i.e per cpu). Magazines
	* may never be entirely disabled in this implementation.
	*/
	unsigned int spl_kmem_cache_magazine_size = 0;
	module_param(spl_kmem_cache_magazine_size, uint, 0444);
	MODULE_PARM_DESC(spl_kmem_cache_magazine_size,
	"Default magazine size (2-256), set automatically (0)");

	/*
	* The default behavior is to report the number of objects remaining in the
	* cache. This allows the Linux VM to repeatedly reclaim objects from the
	* cache when memory is low satisfy other memory allocations. Alternately,
	* setting this value to KMC_RECLAIM_ONCE limits how aggressively the cache
	* is reclaimed. This may increase the likelihood of out of memory events.
	*/
	unsigned int spl_kmem_cache_reclaim = 0 /* KMC_RECLAIM_ONCE */;
	module_param(spl_kmem_cache_reclaim, uint, 0644);
	MODULE_PARM_DESC(spl_kmem_cache_reclaim, "Single reclaim pass (0x1)");

	unsigned int spl_kmem_cache_obj_per_slab = SPL_KMEM_CACHE_OBJ_PER_SLAB;
	module_param(spl_kmem_cache_obj_per_slab, uint, 0644);
	MODULE_PARM_DESC(spl_kmem_cache_obj_per_slab, "Number of objects per slab");

	unsigned int spl_kmem_cache_max_size = SPL_KMEM_CACHE_MAX_SIZE;
	module_param(spl_kmem_cache_max_size, uint, 0644);
	MODULE_PARM_DESC(spl_kmem_cache_max_size, "Maximum size of slab in MB");

	/*
	* For small objects the Linux slab allocator should be used to make the most
	* efficient use of the memory. However, large objects are not supported by
	* the Linux slab and therefore the SPL implementation is preferred. A cutoff
	* of 16K was determined to be optimal for architectures using 4K pages.
	*/
	#if PAGE_SIZE == 4096
	unsigned int spl_kmem_cache_slab_limit = 16384;
	#else
	unsigned int spl_kmem_cache_slab_limit = 0;
	#endif
	module_param(spl_kmem_cache_slab_limit, uint, 0644);
	MODULE_PARM_DESC(spl_kmem_cache_slab_limit,
	"Objects less than N bytes use the Linux slab");

	/*
	* The number of threads available to allocate new slabs for caches. This
	* should not need to be tuned but it is available for performance analysis.
	*/
	unsigned int spl_kmem_cache_kmem_threads = 4;
	module_param(spl_kmem_cache_kmem_threads, uint, 0444);
	MODULE_PARM_DESC(spl_kmem_cache_kmem_threads,
	"Number of spl_kmem_cache threads");
	/* END CSTYLED */

	/*
	* Slab allocation interfaces
	*
	* While the Linux slab implementation was inspired by the Solaris
	* implementation I cannot use it to emulate the Solaris APIs. I
	* require two features which are not provided by the Linux slab.
	*
	* 1) Constructors AND destructors. Recent versions of the Linux
	* kernel have removed support for destructors. This is a deal
	* breaker for the SPL which contains particularly expensive
	* initializers for mutex's, condition variables, etc. We also
	* require a minimal level of cleanup for these data types unlike
	* many Linux data types which do need to be explicitly destroyed.
	*
	* 2) Virtual address space backed slab. Callers of the Solaris slab
	* expect it to work well for both small are very large allocations.
	* Because of memory fragmentation the Linux slab which is backed
	* by kmalloc'ed memory performs very badly when confronted with
	* large numbers of large allocations. Basing the slab on the
	* virtual address space removes the need for contiguous pages
	* and greatly improve performance for large allocations.
	*
	* For these reasons, the SPL has its own slab implementation with
	* the needed features. It is not as highly optimized as either the
	* Solaris or Linux slabs, but it should get me most of what is
	* needed until it can be optimized or obsoleted by another approach.
	*
	* One serious concern I do have about this method is the relatively
	* small virtual address space on 32bit arches. This will seriously
	* constrain the size of the slab caches and their performance.
	*/

	struct list_head spl_kmem_cache_list; /* List of caches */
	struct rw_semaphore spl_kmem_cache_sem; /* Cache list lock */
	taskq_t spl_kmem_cache_taskq; / Task queue for aging / reclaim */

	static void spl_cache_shrink(spl_kmem_cache_t skc, void obj);

	static void *
	kv_alloc(spl_kmem_cache_t *skc, int size, int flags)
	{
	gfp_t lflags = kmem_flags_convert(flags);
	void *ptr;

	ptr = spl_vmalloc(size, lflags \| __GFP_HIGHMEM);

	/* Resulting allocated memory will be page aligned */
	ASSERT(IS_P2ALIGNED(ptr, PAGE_SIZE));

	return (ptr);
	}

	static void
	kv_free(spl_kmem_cache_t skc, void ptr, int size)
	{
	ASSERT(IS_P2ALIGNED(ptr, PAGE_SIZE));

	/*
	* The Linux direct reclaim path uses this out of band value to
	* determine if forward progress is being made. Normally this is
	* incremented by kmem_freepages() which is part of the various
	* Linux slab implementations. However, since we are using none
	* of that infrastructure we are responsible for incrementing it.
	*/
	if (current->reclaim_state)
	current->reclaim_state->reclaimed_slab += size >> PAGE_SHIFT;

	vfree(ptr);
	}

	/*
	* Required space for each aligned sks.
	*/
	static inline uint32_t
	spl_sks_size(spl_kmem_cache_t *skc)
	{
	return (P2ROUNDUP_TYPED(sizeof (spl_kmem_slab_t),
	skc->skc_obj_align, uint32_t));
	}

	/*
	* Required space for each aligned object.
	*/
	static inline uint32_t
	spl_obj_size(spl_kmem_cache_t *skc)
	{
	uint32_t align = skc->skc_obj_align;

	return (P2ROUNDUP_TYPED(skc->skc_obj_size, align, uint32_t) +
	P2ROUNDUP_TYPED(sizeof (spl_kmem_obj_t), align, uint32_t));
	}

	uint64_t
	spl_kmem_cache_inuse(kmem_cache_t *cache)
	{
	return (cache->skc_obj_total);
	}
	EXPORT_SYMBOL(spl_kmem_cache_inuse);

	uint64_t
	spl_kmem_cache_entry_size(kmem_cache_t *cache)
	{
	return (cache->skc_obj_size);
	}
	EXPORT_SYMBOL(spl_kmem_cache_entry_size);

	/*
	* Lookup the spl_kmem_object_t for an object given that object.
	*/
	static inline spl_kmem_obj_t *
	spl_sko_from_obj(spl_kmem_cache_t skc, void obj)
	{
	return (obj + P2ROUNDUP_TYPED(skc->skc_obj_size,
	skc->skc_obj_align, uint32_t));
	}

	/*
	* It's important that we pack the spl_kmem_obj_t structure and the
	* actual objects in to one large address space to minimize the number
	* of calls to the allocator. It is far better to do a few large
	* allocations and then subdivide it ourselves. Now which allocator
	* we use requires balancing a few trade offs.
	*
	* For small objects we use kmem_alloc() because as long as you are
	* only requesting a small number of pages (ideally just one) its cheap.
	* However, when you start requesting multiple pages with kmem_alloc()
	* it gets increasingly expensive since it requires contiguous pages.
	* For this reason we shift to vmem_alloc() for slabs of large objects
	* which removes the need for contiguous pages. We do not use
	* vmem_alloc() in all cases because there is significant locking
	* overhead in __get_vm_area_node(). This function takes a single
	* global lock when acquiring an available virtual address range which
	* serializes all vmem_alloc()'s for all slab caches. Using slightly
	* different allocation functions for small and large objects should
	* give us the best of both worlds.
	*
	* +------------------------+
	* \| spl_kmem_slab_t --+-+ \|
	* \| skc_obj_size <-+ \| \|
	* \| spl_kmem_obj_t \| \|
	* \| skc_obj_size <---+ \|
	* \| spl_kmem_obj_t \| \|
	* \| ... v \|
	* +------------------------+
	*/
	static spl_kmem_slab_t *
	spl_slab_alloc(spl_kmem_cache_t *skc, int flags)
	{
	spl_kmem_slab_t *sks;
	void *base;
	uint32_t obj_size;

	base = kv_alloc(skc, skc->skc_slab_size, flags);
	if (base == NULL)
	return (NULL);

	sks = (spl_kmem_slab_t *)base;
	sks->sks_magic = SKS_MAGIC;
	sks->sks_objs = skc->skc_slab_objs;
	sks->sks_age = jiffies;
	sks->sks_cache = skc;
	INIT_LIST_HEAD(&sks->sks_list);
	INIT_LIST_HEAD(&sks->sks_free_list);
	sks->sks_ref = 0;
	obj_size = spl_obj_size(skc);

	for (int i = 0; i < sks->sks_objs; i++) {
	void obj = base + spl_sks_size(skc) + (i obj_size);

	ASSERT(IS_P2ALIGNED(obj, skc->skc_obj_align));
	spl_kmem_obj_t *sko = spl_sko_from_obj(skc, obj);
	sko->sko_addr = obj;
	sko->sko_magic = SKO_MAGIC;
	sko->sko_slab = sks;
	INIT_LIST_HEAD(&sko->sko_list);
	list_add_tail(&sko->sko_list, &sks->sks_free_list);
	}

	return (sks);
	}

	/*
	* Remove a slab from complete or partial list, it must be called with
	* the 'skc->skc_lock' held but the actual free must be performed
	* outside the lock to prevent deadlocking on vmem addresses.
	*/
	static void
	spl_slab_free(spl_kmem_slab_t *sks,
	struct list_head sks_list, struct list_head sko_list)
	{
	spl_kmem_cache_t *skc;

	ASSERT(sks->sks_magic == SKS_MAGIC);
	ASSERT(sks->sks_ref == 0);

	skc = sks->sks_cache;
	ASSERT(skc->skc_magic == SKC_MAGIC);

	/*
	* Update slab/objects counters in the cache, then remove the
	* slab from the skc->skc_partial_list. Finally add the slab
	* and all its objects in to the private work lists where the
	* destructors will be called and the memory freed to the system.
	*/
	skc->skc_obj_total -= sks->sks_objs;
	skc->skc_slab_total--;
	list_del(&sks->sks_list);
	list_add(&sks->sks_list, sks_list);
	list_splice_init(&sks->sks_free_list, sko_list);
	}

	/*
	* Reclaim empty slabs at the end of the partial list.
	*/
	static void
	spl_slab_reclaim(spl_kmem_cache_t *skc)
	{
	spl_kmem_slab_t sks = NULL, m = NULL;
	spl_kmem_obj_t sko = NULL, n = NULL;
	LIST_HEAD(sks_list);
	LIST_HEAD(sko_list);

	/*
	* Empty slabs and objects must be moved to a private list so they
	* can be safely freed outside the spin lock. All empty slabs are
	* at the end of skc->skc_partial_list, therefore once a non-empty
	* slab is found we can stop scanning.
	*/
	spin_lock(&skc->skc_lock);
	list_for_each_entry_safe_reverse(sks, m,
	&skc->skc_partial_list, sks_list) {

	if (sks->sks_ref > 0)
	break;

	spl_slab_free(sks, &sks_list, &sko_list);
	}
	spin_unlock(&skc->skc_lock);

	/*
	* The following two loops ensure all the object destructors are run,
	* and the slabs themselves are freed. This is all done outside the
	* skc->skc_lock since this allows the destructor to sleep, and
	* allows us to perform a conditional reschedule when a freeing a
	* large number of objects and slabs back to the system.
	*/

	list_for_each_entry_safe(sko, n, &sko_list, sko_list) {
	ASSERT(sko->sko_magic == SKO_MAGIC);
	}

	list_for_each_entry_safe(sks, m, &sks_list, sks_list) {
	ASSERT(sks->sks_magic == SKS_MAGIC);
	kv_free(skc, sks, skc->skc_slab_size);
	}
	}

	static spl_kmem_emergency_t *
	spl_emergency_search(struct rb_root root, void obj)
	{
	struct rb_node *node = root->rb_node;
	spl_kmem_emergency_t *ske;
	unsigned long address = (unsigned long)obj;

	while (node) {
	ske = container_of(node, spl_kmem_emergency_t, ske_node);

	if (address < ske->ske_obj)
	node = node->rb_left;
	else if (address > ske->ske_obj)
	node = node->rb_right;
	else
	return (ske);
	}

	return (NULL);
	}

	static int
	spl_emergency_insert(struct rb_root root, spl_kmem_emergency_t ske)
	{
	struct rb_node *new = &(root->rb_node), parent = NULL;
	spl_kmem_emergency_t *ske_tmp;
	unsigned long address = ske->ske_obj;

	while (*new) {
	ske_tmp = container_of(*new, spl_kmem_emergency_t, ske_node);

	parent = *new;
	if (address < ske_tmp->ske_obj)
	new = &((*new)->rb_left);
	else if (address > ske_tmp->ske_obj)
	new = &((*new)->rb_right);
	else
	return (0);
	}

	rb_link_node(&ske->ske_node, parent, new);
	rb_insert_color(&ske->ske_node, root);

	return (1);
	}

	/*
	* Allocate a single emergency object and track it in a red black tree.
	*/
	static int
	spl_emergency_alloc(spl_kmem_cache_t skc, int flags, void *obj)
	{
	gfp_t lflags = kmem_flags_convert(flags);
	spl_kmem_emergency_t *ske;
	int order = get_order(skc->skc_obj_size);
	int empty;

	/* Last chance use a partial slab if one now exists */
	spin_lock(&skc->skc_lock);
	empty = list_empty(&skc->skc_partial_list);
	spin_unlock(&skc->skc_lock);
	if (!empty)
	return (-EEXIST);

	ske = kmalloc(sizeof (*ske), lflags);
	if (ske == NULL)
	return (-ENOMEM);

	ske->ske_obj = __get_free_pages(lflags, order);
	if (ske->ske_obj == 0) {
	kfree(ske);
	return (-ENOMEM);
	}

	spin_lock(&skc->skc_lock);
	empty = spl_emergency_insert(&skc->skc_emergency_tree, ske);
	if (likely(empty)) {
	skc->skc_obj_total++;
	skc->skc_obj_emergency++;
	if (skc->skc_obj_emergency > skc->skc_obj_emergency_max)
	skc->skc_obj_emergency_max = skc->skc_obj_emergency;
	}
	spin_unlock(&skc->skc_lock);

	if (unlikely(!empty)) {
	free_pages(ske->ske_obj, order);
	kfree(ske);
	return (-EINVAL);
	}

	obj = (void )ske->ske_obj;

	return (0);
	}

	/*
	* Locate the passed object in the red black tree and free it.
	*/
	static int
	spl_emergency_free(spl_kmem_cache_t skc, void obj)
	{
	spl_kmem_emergency_t *ske;
	int order = get_order(skc->skc_obj_size);

	spin_lock(&skc->skc_lock);
	ske = spl_emergency_search(&skc->skc_emergency_tree, obj);
	if (ske) {
	rb_erase(&ske->ske_node, &skc->skc_emergency_tree);
	skc->skc_obj_emergency--;
	skc->skc_obj_total--;
	}
	spin_unlock(&skc->skc_lock);

	if (ske == NULL)
	return (-ENOENT);

	free_pages(ske->ske_obj, order);
	kfree(ske);

	return (0);
	}

	/*
	* Release objects from the per-cpu magazine back to their slab. The flush
	* argument contains the max number of entries to remove from the magazine.
	*/
	static void
	spl_cache_flush(spl_kmem_cache_t skc, spl_kmem_magazine_t skm, int flush)
	{
	spin_lock(&skc->skc_lock);

	ASSERT(skc->skc_magic == SKC_MAGIC);
	ASSERT(skm->skm_magic == SKM_MAGIC);

	int count = MIN(flush, skm->skm_avail);
	for (int i = 0; i < count; i++)
	spl_cache_shrink(skc, skm->skm_objs[i]);

	skm->skm_avail -= count;
	memmove(skm->skm_objs, &(skm->skm_objs[count]),
	sizeof (void ) skm->skm_avail);

	spin_unlock(&skc->skc_lock);
	}

	/*
	* Size a slab based on the size of each aligned object plus spl_kmem_obj_t.
	* When on-slab we want to target spl_kmem_cache_obj_per_slab. However,
	* for very small objects we may end up with more than this so as not
	* to waste space in the minimal allocation of a single page. Also for
	* very large objects we may use as few as spl_kmem_cache_obj_per_slab_min,
	* lower than this and we will fail.
	*/
	static int
	spl_slab_size(spl_kmem_cache_t skc, uint32_t objs, uint32_t *size)
	{
	uint32_t sks_size, obj_size, max_size, tgt_size, tgt_objs;

	sks_size = spl_sks_size(skc);
	obj_size = spl_obj_size(skc);
	max_size = (spl_kmem_cache_max_size * 1024 * 1024);
	tgt_size = (spl_kmem_cache_obj_per_slab * obj_size + sks_size);

	if (tgt_size <= max_size) {
	tgt_objs = (tgt_size - sks_size) / obj_size;
	} else {
	tgt_objs = (max_size - sks_size) / obj_size;
	tgt_size = (tgt_objs * obj_size) + sks_size;
	}

	if (tgt_objs == 0)
	return (-ENOSPC);

	*objs = tgt_objs;
	*size = tgt_size;

	return (0);
	}

	/*
	* Make a guess at reasonable per-cpu magazine size based on the size of
	* each object and the cost of caching N of them in each magazine. Long
	* term this should really adapt based on an observed usage heuristic.
	*/
	static int
	spl_magazine_size(spl_kmem_cache_t *skc)
	{
	uint32_t obj_size = spl_obj_size(skc);
	int size;

	if (spl_kmem_cache_magazine_size > 0)
	return (MAX(MIN(spl_kmem_cache_magazine_size, 256), 2));

	/* Per-magazine sizes below assume a 4Kib page size */
	if (obj_size > (PAGE_SIZE * 256))
	size = 4; /* Minimum 4Mib per-magazine */
	else if (obj_size > (PAGE_SIZE * 32))
	size = 16; /* Minimum 2Mib per-magazine */
	else if (obj_size > (PAGE_SIZE))
	size = 64; /* Minimum 256Kib per-magazine */
	else if (obj_size > (PAGE_SIZE / 4))
	size = 128; /* Minimum 128Kib per-magazine */
	else
	size = 256;

	return (size);
	}

	/*
	* Allocate a per-cpu magazine to associate with a specific core.
	*/
	static spl_kmem_magazine_t *
	spl_magazine_alloc(spl_kmem_cache_t *skc, int cpu)
	{
	spl_kmem_magazine_t *skm;
	int size = sizeof (spl_kmem_magazine_t) +
	sizeof (void ) skc->skc_mag_size;

	skm = kmalloc_node(size, GFP_KERNEL, cpu_to_node(cpu));
	if (skm) {
	skm->skm_magic = SKM_MAGIC;
	skm->skm_avail = 0;
	skm->skm_size = skc->skc_mag_size;
	skm->skm_refill = skc->skc_mag_refill;
	skm->skm_cache = skc;
	skm->skm_cpu = cpu;
	}

	return (skm);
	}

	/*
	* Free a per-cpu magazine associated with a specific core.
	*/
	static void
	spl_magazine_free(spl_kmem_magazine_t *skm)
	{
	ASSERT(skm->skm_magic == SKM_MAGIC);
	ASSERT(skm->skm_avail == 0);
	kfree(skm);
	}

	/*
	* Create all pre-cpu magazines of reasonable sizes.
	*/
	static int
	spl_magazine_create(spl_kmem_cache_t *skc)
	{
	int i = 0;

	ASSERT((skc->skc_flags & KMC_SLAB) == 0);

	skc->skc_mag = kzalloc(sizeof (spl_kmem_magazine_t )
	num_possible_cpus(), kmem_flags_convert(KM_SLEEP));
	skc->skc_mag_size = spl_magazine_size(skc);
	skc->skc_mag_refill = (skc->skc_mag_size + 1) / 2;

	for_each_possible_cpu(i) {
	skc->skc_mag[i] = spl_magazine_alloc(skc, i);
	if (!skc->skc_mag[i]) {
	for (i--; i >= 0; i--)
	spl_magazine_free(skc->skc_mag[i]);

	kfree(skc->skc_mag);
	return (-ENOMEM);
	}
	}

	return (0);
	}

	/*
	* Destroy all pre-cpu magazines.
	*/
	static void
	spl_magazine_destroy(spl_kmem_cache_t *skc)
	{
	spl_kmem_magazine_t *skm;
	int i = 0;

	ASSERT((skc->skc_flags & KMC_SLAB) == 0);

	for_each_possible_cpu(i) {
	skm = skc->skc_mag[i];
	spl_cache_flush(skc, skm, skm->skm_avail);
	spl_magazine_free(skm);
	}

	kfree(skc->skc_mag);
	}

	/*
	* Create a object cache based on the following arguments:
	* name cache name
	* size cache object size
	* align cache object alignment
	* ctor cache object constructor
	* dtor cache object destructor
	* reclaim cache object reclaim
	* priv cache private data for ctor/dtor/reclaim
	* vmp unused must be NULL
	* flags
	* KMC_KVMEM Force kvmem backed SPL cache
	* KMC_SLAB Force Linux slab backed cache
	* KMC_NODEBUG Disable debugging (unsupported)
	*/
	spl_kmem_cache_t *
	spl_kmem_cache_create(char *name, size_t size, size_t align,
	spl_kmem_ctor_t ctor, spl_kmem_dtor_t dtor, void *reclaim,
	void priv, void vmp, int flags)
	{
	gfp_t lflags = kmem_flags_convert(KM_SLEEP);
	spl_kmem_cache_t *skc;
	int rc;

	/*
	* Unsupported flags
	*/
	ASSERT(vmp == NULL);
	ASSERT(reclaim == NULL);

	might_sleep();

	skc = kzalloc(sizeof (*skc), lflags);
	if (skc == NULL)
	return (NULL);

	skc->skc_magic = SKC_MAGIC;
	skc->skc_name_size = strlen(name) + 1;
	skc->skc_name = (char *)kmalloc(skc->skc_name_size, lflags);
	if (skc->skc_name == NULL) {
	kfree(skc);
	return (NULL);
	}
	strncpy(skc->skc_name, name, skc->skc_name_size);

	skc->skc_ctor = ctor;
	skc->skc_dtor = dtor;
	skc->skc_private = priv;
	skc->skc_vmp = vmp;
	skc->skc_linux_cache = NULL;
	skc->skc_flags = flags;
	skc->skc_obj_size = size;
	skc->skc_obj_align = SPL_KMEM_CACHE_ALIGN;
	atomic_set(&skc->skc_ref, 0);

	INIT_LIST_HEAD(&skc->skc_list);
	INIT_LIST_HEAD(&skc->skc_complete_list);
	INIT_LIST_HEAD(&skc->skc_partial_list);
	skc->skc_emergency_tree = RB_ROOT;
	spin_lock_init(&skc->skc_lock);
	init_waitqueue_head(&skc->skc_waitq);
	skc->skc_slab_fail = 0;
	skc->skc_slab_create = 0;
	skc->skc_slab_destroy = 0;
	skc->skc_slab_total = 0;
	skc->skc_slab_alloc = 0;
	skc->skc_slab_max = 0;
	skc->skc_obj_total = 0;
	skc->skc_obj_alloc = 0;
	skc->skc_obj_max = 0;
	skc->skc_obj_deadlock = 0;
	skc->skc_obj_emergency = 0;
	skc->skc_obj_emergency_max = 0;

	rc = percpu_counter_init_common(&skc->skc_linux_alloc, 0,
	GFP_KERNEL);
	if (rc != 0) {
	kfree(skc);
	return (NULL);
	}

	/*
	* Verify the requested alignment restriction is sane.
	*/
	if (align) {
	VERIFY(ISP2(align));
	VERIFY3U(align, >=, SPL_KMEM_CACHE_ALIGN);
	VERIFY3U(align, <=, PAGE_SIZE);
	skc->skc_obj_align = align;
	}

	/*
	* When no specific type of slab is requested (kmem, vmem, or
	* linuxslab) then select a cache type based on the object size
	* and default tunables.
	*/
	if (!(skc->skc_flags & (KMC_SLAB \| KMC_KVMEM))) {
	if (spl_kmem_cache_slab_limit &&
	size <= (size_t)spl_kmem_cache_slab_limit) {
	/*
	* Objects smaller than spl_kmem_cache_slab_limit can
	* use the Linux slab for better space-efficiency.
	*/
	skc->skc_flags \|= KMC_SLAB;
	} else {
	/*
	* All other objects are considered large and are
	* placed on kvmem backed slabs.
	*/
	skc->skc_flags \|= KMC_KVMEM;
	}
	}

	/*
	* Given the type of slab allocate the required resources.
	*/
	if (skc->skc_flags & KMC_KVMEM) {
	rc = spl_slab_size(skc,
	&skc->skc_slab_objs, &skc->skc_slab_size);
	if (rc)
	goto out;

	rc = spl_magazine_create(skc);
	if (rc)
	goto out;
	} else {
	unsigned long slabflags = 0;

	if (size > (SPL_MAX_KMEM_ORDER_NR_PAGES * PAGE_SIZE)) {
	rc = EINVAL;
	goto out;
	}

	#if defined(SLAB_USERCOPY)
	/*
	* Required for PAX-enabled kernels if the slab is to be
	* used for copying between user and kernel space.
	*/
	slabflags \|= SLAB_USERCOPY;
	#endif

	#if defined(HAVE_KMEM_CACHE_CREATE_USERCOPY)
	/*
	* Newer grsec patchset uses kmem_cache_create_usercopy()
	* instead of SLAB_USERCOPY flag
	*/
	skc->skc_linux_cache = kmem_cache_create_usercopy(
	skc->skc_name, size, align, slabflags, 0, size, NULL);
	#else
	skc->skc_linux_cache = kmem_cache_create(
	skc->skc_name, size, align, slabflags, NULL);
	#endif
	if (skc->skc_linux_cache == NULL) {
	rc = ENOMEM;
	goto out;
	}
	}

	down_write(&spl_kmem_cache_sem);
	list_add_tail(&skc->skc_list, &spl_kmem_cache_list);
	up_write(&spl_kmem_cache_sem);

	return (skc);
	out:
	kfree(skc->skc_name);
	percpu_counter_destroy(&skc->skc_linux_alloc);
	kfree(skc);
	return (NULL);
	}
	EXPORT_SYMBOL(spl_kmem_cache_create);

	/*
	* Register a move callback for cache defragmentation.
	* XXX: Unimplemented but harmless to stub out for now.
	*/
	void
	spl_kmem_cache_set_move(spl_kmem_cache_t *skc,
	kmem_cbrc_t (move)(void , void , size_t, void *))
	{
	ASSERT(move != NULL);
	}
	EXPORT_SYMBOL(spl_kmem_cache_set_move);

	/*
	* Destroy a cache and all objects associated with the cache.
	*/
	void
	spl_kmem_cache_destroy(spl_kmem_cache_t *skc)
	{
	DECLARE_WAIT_QUEUE_HEAD(wq);
	taskqid_t id;

	ASSERT(skc->skc_magic == SKC_MAGIC);
	ASSERT(skc->skc_flags & (KMC_KVMEM \| KMC_SLAB));

	down_write(&spl_kmem_cache_sem);
	list_del_init(&skc->skc_list);
	up_write(&spl_kmem_cache_sem);

	/* Cancel any and wait for any pending delayed tasks */
	VERIFY(!test_and_set_bit(KMC_BIT_DESTROY, &skc->skc_flags));

	spin_lock(&skc->skc_lock);
	id = skc->skc_taskqid;
	spin_unlock(&skc->skc_lock);

	taskq_cancel_id(spl_kmem_cache_taskq, id);

	/*
	* Wait until all current callers complete, this is mainly
	* to catch the case where a low memory situation triggers a
	* cache reaping action which races with this destroy.
	*/
	wait_event(wq, atomic_read(&skc->skc_ref) == 0);

	if (skc->skc_flags & KMC_KVMEM) {
	spl_magazine_destroy(skc);
	spl_slab_reclaim(skc);
	} else {
	ASSERT(skc->skc_flags & KMC_SLAB);
	kmem_cache_destroy(skc->skc_linux_cache);
	}

	spin_lock(&skc->skc_lock);

	/*
	* Validate there are no objects in use and free all the
	* spl_kmem_slab_t, spl_kmem_obj_t, and object buffers.
	*/
	ASSERT3U(skc->skc_slab_alloc, ==, 0);
	ASSERT3U(skc->skc_obj_alloc, ==, 0);
	ASSERT3U(skc->skc_slab_total, ==, 0);
	ASSERT3U(skc->skc_obj_total, ==, 0);
	ASSERT3U(skc->skc_obj_emergency, ==, 0);
	ASSERT(list_empty(&skc->skc_complete_list));

	ASSERT3U(percpu_counter_sum(&skc->skc_linux_alloc), ==, 0);
	percpu_counter_destroy(&skc->skc_linux_alloc);

	spin_unlock(&skc->skc_lock);

	kfree(skc->skc_name);
	kfree(skc);
	}
	EXPORT_SYMBOL(spl_kmem_cache_destroy);

	/*
	* Allocate an object from a slab attached to the cache. This is used to
	* repopulate the per-cpu magazine caches in batches when they run low.
	*/
	static void *
	spl_cache_obj(spl_kmem_cache_t skc, spl_kmem_slab_t sks)
	{
	spl_kmem_obj_t *sko;

	ASSERT(skc->skc_magic == SKC_MAGIC);
	ASSERT(sks->sks_magic == SKS_MAGIC);

	sko = list_entry(sks->sks_free_list.next, spl_kmem_obj_t, sko_list);
	ASSERT(sko->sko_magic == SKO_MAGIC);
	ASSERT(sko->sko_addr != NULL);

	/* Remove from sks_free_list */
	list_del_init(&sko->sko_list);

	sks->sks_age = jiffies;
	sks->sks_ref++;
	skc->skc_obj_alloc++;

	/* Track max obj usage statistics */
	if (skc->skc_obj_alloc > skc->skc_obj_max)
	skc->skc_obj_max = skc->skc_obj_alloc;

	/* Track max slab usage statistics */
	if (sks->sks_ref == 1) {
	skc->skc_slab_alloc++;

	if (skc->skc_slab_alloc > skc->skc_slab_max)
	skc->skc_slab_max = skc->skc_slab_alloc;
	}

	return (sko->sko_addr);
	}

	/*
	* Generic slab allocation function to run by the global work queues.
	* It is responsible for allocating a new slab, linking it in to the list
	* of partial slabs, and then waking any waiters.
	*/
	static int
	__spl_cache_grow(spl_kmem_cache_t *skc, int flags)
	{
	spl_kmem_slab_t *sks;

	fstrans_cookie_t cookie = spl_fstrans_mark();
	sks = spl_slab_alloc(skc, flags);
	spl_fstrans_unmark(cookie);

	spin_lock(&skc->skc_lock);
	if (sks) {
	skc->skc_slab_total++;
	skc->skc_obj_total += sks->sks_objs;
	list_add_tail(&sks->sks_list, &skc->skc_partial_list);

	smp_mb__before_atomic();
	clear_bit(KMC_BIT_DEADLOCKED, &skc->skc_flags);
	smp_mb__after_atomic();
	}
	spin_unlock(&skc->skc_lock);

	return (sks == NULL ? -ENOMEM : 0);
	}

	static void
	spl_cache_grow_work(void *data)
	{
	spl_kmem_alloc_t ska = (spl_kmem_alloc_t )data;
	spl_kmem_cache_t *skc = ska->ska_cache;

	int error = __spl_cache_grow(skc, ska->ska_flags);

	atomic_dec(&skc->skc_ref);
	smp_mb__before_atomic();
	clear_bit(KMC_BIT_GROWING, &skc->skc_flags);
	smp_mb__after_atomic();
	if (error == 0)
	wake_up_all(&skc->skc_waitq);

	kfree(ska);
	}

	/*
	* Returns non-zero when a new slab should be available.
	*/
	static int
	spl_cache_grow_wait(spl_kmem_cache_t *skc)
	{
	return (!test_bit(KMC_BIT_GROWING, &skc->skc_flags));
	}

	/*
	* No available objects on any slabs, create a new slab. Note that this
	* functionality is disabled for KMC_SLAB caches which are backed by the
	* Linux slab.
	*/
	static int
	spl_cache_grow(spl_kmem_cache_t skc, int flags, void *obj)
	{
	int remaining, rc = 0;

	ASSERT0(flags & ~KM_PUBLIC_MASK);
	ASSERT(skc->skc_magic == SKC_MAGIC);
	ASSERT((skc->skc_flags & KMC_SLAB) == 0);
	might_sleep();
	*obj = NULL;

	/*
	* Before allocating a new slab wait for any reaping to complete and
	* then return so the local magazine can be rechecked for new objects.
	*/
	if (test_bit(KMC_BIT_REAPING, &skc->skc_flags)) {
	rc = spl_wait_on_bit(&skc->skc_flags, KMC_BIT_REAPING,
	TASK_UNINTERRUPTIBLE);
	return (rc ? rc : -EAGAIN);
	}

	/*
	* Note: It would be nice to reduce the overhead of context switch
	* and improve NUMA locality, by trying to allocate a new slab in the
	* current process context with KM_NOSLEEP flag.
	*
	* However, this can't be applied to vmem/kvmem due to a bug that
	* spl_vmalloc() doesn't honor gfp flags in page table allocation.
	*/

	/*
	* This is handled by dispatching a work request to the global work
	* queue. This allows us to asynchronously allocate a new slab while
	* retaining the ability to safely fall back to a smaller synchronous
	* allocations to ensure forward progress is always maintained.
	*/
	if (test_and_set_bit(KMC_BIT_GROWING, &skc->skc_flags) == 0) {
	spl_kmem_alloc_t *ska;

	ska = kmalloc(sizeof (*ska), kmem_flags_convert(flags));
	if (ska == NULL) {
	clear_bit_unlock(KMC_BIT_GROWING, &skc->skc_flags);
	smp_mb__after_atomic();
	wake_up_all(&skc->skc_waitq);
	return (-ENOMEM);
	}

	atomic_inc(&skc->skc_ref);
	ska->ska_cache = skc;
	ska->ska_flags = flags;
	taskq_init_ent(&ska->ska_tqe);
	taskq_dispatch_ent(spl_kmem_cache_taskq,
	spl_cache_grow_work, ska, 0, &ska->ska_tqe);
	}

	/*
	* The goal here is to only detect the rare case where a virtual slab
	* allocation has deadlocked. We must be careful to minimize the use
	* of emergency objects which are more expensive to track. Therefore,
	* we set a very long timeout for the asynchronous allocation and if
	* the timeout is reached the cache is flagged as deadlocked. From
	* this point only new emergency objects will be allocated until the
	* asynchronous allocation completes and clears the deadlocked flag.
	*/
	if (test_bit(KMC_BIT_DEADLOCKED, &skc->skc_flags)) {
	rc = spl_emergency_alloc(skc, flags, obj);
	} else {
	remaining = wait_event_timeout(skc->skc_waitq,
	spl_cache_grow_wait(skc), HZ / 10);

	if (!remaining) {
	spin_lock(&skc->skc_lock);
	if (test_bit(KMC_BIT_GROWING, &skc->skc_flags)) {
	set_bit(KMC_BIT_DEADLOCKED, &skc->skc_flags);
	skc->skc_obj_deadlock++;
	}
	spin_unlock(&skc->skc_lock);
	}

	rc = -ENOMEM;
	}

	return (rc);
	}

	/*
	* Refill a per-cpu magazine with objects from the slabs for this cache.
	* Ideally the magazine can be repopulated using existing objects which have
	* been released, however if we are unable to locate enough free objects new
	* slabs of objects will be created. On success NULL is returned, otherwise
	* the address of a single emergency object is returned for use by the caller.
	*/
	static void *
	spl_cache_refill(spl_kmem_cache_t skc, spl_kmem_magazine_t skm, int flags)
	{
	spl_kmem_slab_t *sks;
	int count = 0, rc, refill;
	void *obj = NULL;

	ASSERT(skc->skc_magic == SKC_MAGIC);
	ASSERT(skm->skm_magic == SKM_MAGIC);

	refill = MIN(skm->skm_refill, skm->skm_size - skm->skm_avail);
	spin_lock(&skc->skc_lock);

	while (refill > 0) {
	/* No slabs available we may need to grow the cache */
	if (list_empty(&skc->skc_partial_list)) {
	spin_unlock(&skc->skc_lock);

	local_irq_enable();
	rc = spl_cache_grow(skc, flags, &obj);
	local_irq_disable();

	/* Emergency object for immediate use by caller */
	if (rc == 0 && obj != NULL)
	return (obj);

	if (rc)
	goto out;

	/* Rescheduled to different CPU skm is not local */
	if (skm != skc->skc_mag[smp_processor_id()])
	goto out;

	/*
	* Potentially rescheduled to the same CPU but
	* allocations may have occurred from this CPU while
	* we were sleeping so recalculate max refill.
	*/
	refill = MIN(refill, skm->skm_size - skm->skm_avail);

	spin_lock(&skc->skc_lock);
	continue;
	}

	/* Grab the next available slab */
	sks = list_entry((&skc->skc_partial_list)->next,
	spl_kmem_slab_t, sks_list);
	ASSERT(sks->sks_magic == SKS_MAGIC);
	ASSERT(sks->sks_ref < sks->sks_objs);
	ASSERT(!list_empty(&sks->sks_free_list));

	/*
	* Consume as many objects as needed to refill the requested
	* cache. We must also be careful not to overfill it.
	*/
	while (sks->sks_ref < sks->sks_objs && refill-- > 0 &&
	++count) {
	ASSERT(skm->skm_avail < skm->skm_size);
	ASSERT(count < skm->skm_size);
	skm->skm_objs[skm->skm_avail++] =
	spl_cache_obj(skc, sks);
	}

	/* Move slab to skc_complete_list when full */
	if (sks->sks_ref == sks->sks_objs) {
	list_del(&sks->sks_list);
	list_add(&sks->sks_list, &skc->skc_complete_list);
	}
	}

	spin_unlock(&skc->skc_lock);
	out:
	return (NULL);
	}

	/*
	* Release an object back to the slab from which it came.
	*/
	static void
	spl_cache_shrink(spl_kmem_cache_t skc, void obj)
	{
	spl_kmem_slab_t *sks = NULL;
	spl_kmem_obj_t *sko = NULL;

	ASSERT(skc->skc_magic == SKC_MAGIC);

	sko = spl_sko_from_obj(skc, obj);
	ASSERT(sko->sko_magic == SKO_MAGIC);
	sks = sko->sko_slab;
	ASSERT(sks->sks_magic == SKS_MAGIC);
	ASSERT(sks->sks_cache == skc);
	list_add(&sko->sko_list, &sks->sks_free_list);

	sks->sks_age = jiffies;
	sks->sks_ref--;
	skc->skc_obj_alloc--;

	/*
	* Move slab to skc_partial_list when no longer full. Slabs
	* are added to the head to keep the partial list is quasi-full
	* sorted order. Fuller at the head, emptier at the tail.
	*/
	if (sks->sks_ref == (sks->sks_objs - 1)) {
	list_del(&sks->sks_list);
	list_add(&sks->sks_list, &skc->skc_partial_list);
	}

	/*
	* Move empty slabs to the end of the partial list so
	* they can be easily found and freed during reclamation.
	*/
	if (sks->sks_ref == 0) {
	list_del(&sks->sks_list);
	list_add_tail(&sks->sks_list, &skc->skc_partial_list);
	skc->skc_slab_alloc--;
	}
	}

	/*
	* Allocate an object from the per-cpu magazine, or if the magazine
	* is empty directly allocate from a slab and repopulate the magazine.
	*/
	void *
	spl_kmem_cache_alloc(spl_kmem_cache_t *skc, int flags)
	{
	spl_kmem_magazine_t *skm;
	void *obj = NULL;

	ASSERT0(flags & ~KM_PUBLIC_MASK);
	ASSERT(skc->skc_magic == SKC_MAGIC);
	ASSERT(!test_bit(KMC_BIT_DESTROY, &skc->skc_flags));

	/*
	* Allocate directly from a Linux slab. All optimizations are left
	* to the underlying cache we only need to guarantee that KM_SLEEP
	* callers will never fail.
	*/
	if (skc->skc_flags & KMC_SLAB) {
	struct kmem_cache *slc = skc->skc_linux_cache;
	do {
	obj = kmem_cache_alloc(slc, kmem_flags_convert(flags));
	} while ((obj == NULL) && !(flags & KM_NOSLEEP));

	if (obj != NULL) {
	/*
	* Even though we leave everything up to the
	* underlying cache we still keep track of
	* how many objects we've allocated in it for
	* better debuggability.
	*/
	percpu_counter_inc(&skc->skc_linux_alloc);
	}
	goto ret;
	}

	local_irq_disable();

	restart:
	/*
	* Safe to update per-cpu structure without lock, but
	* in the restart case we must be careful to reacquire
	* the local magazine since this may have changed
	* when we need to grow the cache.
	*/
	skm = skc->skc_mag[smp_processor_id()];
	ASSERT(skm->skm_magic == SKM_MAGIC);

	if (likely(skm->skm_avail)) {
	/* Object available in CPU cache, use it */
	obj = skm->skm_objs[--skm->skm_avail];
	} else {
	obj = spl_cache_refill(skc, skm, flags);
	if ((obj == NULL) && !(flags & KM_NOSLEEP))
	goto restart;

	local_irq_enable();
	goto ret;
	}

	local_irq_enable();
	ASSERT(obj);
	ASSERT(IS_P2ALIGNED(obj, skc->skc_obj_align));

	ret:
	/* Pre-emptively migrate object to CPU L1 cache */
	if (obj) {
	if (obj && skc->skc_ctor)
	skc->skc_ctor(obj, skc->skc_private, flags);
	else
	prefetchw(obj);
	}

	return (obj);
	}
	EXPORT_SYMBOL(spl_kmem_cache_alloc);

	/*
	* Free an object back to the local per-cpu magazine, there is no
	* guarantee that this is the same magazine the object was originally
	* allocated from. We may need to flush entire from the magazine
	* back to the slabs to make space.
	*/
	void
	spl_kmem_cache_free(spl_kmem_cache_t skc, void obj)
	{
	spl_kmem_magazine_t *skm;
	unsigned long flags;
	int do_reclaim = 0;
	int do_emergency = 0;

	ASSERT(skc->skc_magic == SKC_MAGIC);
	ASSERT(!test_bit(KMC_BIT_DESTROY, &skc->skc_flags));

	/*
	* Run the destructor
	*/
	if (skc->skc_dtor)
	skc->skc_dtor(obj, skc->skc_private);

	/*
	* Free the object from the Linux underlying Linux slab.
	*/
	if (skc->skc_flags & KMC_SLAB) {
	kmem_cache_free(skc->skc_linux_cache, obj);
	percpu_counter_dec(&skc->skc_linux_alloc);
	return;
	}

	/*
	* While a cache has outstanding emergency objects all freed objects
	* must be checked. However, since emergency objects will never use
	* a virtual address these objects can be safely excluded as an
	* optimization.
	*/
	if (!is_vmalloc_addr(obj)) {
	spin_lock(&skc->skc_lock);
	do_emergency = (skc->skc_obj_emergency > 0);
	spin_unlock(&skc->skc_lock);

	if (do_emergency && (spl_emergency_free(skc, obj) == 0))
	return;
	}

	local_irq_save(flags);

	/*
	* Safe to update per-cpu structure without lock, but
	* no remote memory allocation tracking is being performed
	* it is entirely possible to allocate an object from one
	* CPU cache and return it to another.
	*/
	skm = skc->skc_mag[smp_processor_id()];
	ASSERT(skm->skm_magic == SKM_MAGIC);

	/*
	* Per-CPU cache full, flush it to make space for this object,
	* this may result in an empty slab which can be reclaimed once
	* interrupts are re-enabled.
	*/
	if (unlikely(skm->skm_avail >= skm->skm_size)) {
	spl_cache_flush(skc, skm, skm->skm_refill);
	do_reclaim = 1;
	}

	/* Available space in cache, use it */
	skm->skm_objs[skm->skm_avail++] = obj;

	local_irq_restore(flags);

	if (do_reclaim)
	spl_slab_reclaim(skc);
	}
	EXPORT_SYMBOL(spl_kmem_cache_free);

	/*
	* Depending on how many and which objects are released it may simply
	* repopulate the local magazine which will then need to age-out. Objects
	* which cannot fit in the magazine will be released back to their slabs
	* which will also need to age out before being released. This is all just
	* best effort and we do not want to thrash creating and destroying slabs.
	*/
	void
	spl_kmem_cache_reap_now(spl_kmem_cache_t *skc)
	{
	ASSERT(skc->skc_magic == SKC_MAGIC);
	ASSERT(!test_bit(KMC_BIT_DESTROY, &skc->skc_flags));

	if (skc->skc_flags & KMC_SLAB)
	return;

	atomic_inc(&skc->skc_ref);

	/*
	* Prevent concurrent cache reaping when contended.
	*/
	if (test_and_set_bit(KMC_BIT_REAPING, &skc->skc_flags))
	goto out;

	/* Reclaim from the magazine and free all now empty slabs. */
	unsigned long irq_flags;
	local_irq_save(irq_flags);
	spl_kmem_magazine_t *skm = skc->skc_mag[smp_processor_id()];
	spl_cache_flush(skc, skm, skm->skm_avail);
	local_irq_restore(irq_flags);

	spl_slab_reclaim(skc);
	clear_bit_unlock(KMC_BIT_REAPING, &skc->skc_flags);
	smp_mb__after_atomic();
	wake_up_bit(&skc->skc_flags, KMC_BIT_REAPING);
	out:
	atomic_dec(&skc->skc_ref);
	}
	EXPORT_SYMBOL(spl_kmem_cache_reap_now);

	/*
	* This is stubbed out for code consistency with other platforms. There
	* is existing logic to prevent concurrent reaping so while this is ugly
	* it should do no harm.
	*/
	int
	spl_kmem_cache_reap_active()
	{
	return (0);
	}
	EXPORT_SYMBOL(spl_kmem_cache_reap_active);

	/*
	* Reap all free slabs from all registered caches.
	*/
	void
	spl_kmem_reap(void)
	{
	spl_kmem_cache_t *skc = NULL;

	down_read(&spl_kmem_cache_sem);
	list_for_each_entry(skc, &spl_kmem_cache_list, skc_list) {
	spl_kmem_cache_reap_now(skc);
	}
	up_read(&spl_kmem_cache_sem);
	}
	EXPORT_SYMBOL(spl_kmem_reap);

	int
	spl_kmem_cache_init(void)
	{
	init_rwsem(&spl_kmem_cache_sem);
	INIT_LIST_HEAD(&spl_kmem_cache_list);
	spl_kmem_cache_taskq = taskq_create("spl_kmem_cache",
	spl_kmem_cache_kmem_threads, maxclsyspri,
	spl_kmem_cache_kmem_threads * 8, INT_MAX,
	TASKQ_PREPOPULATE \| TASKQ_DYNAMIC);

	return (0);
	}

	void
	spl_kmem_cache_fini(void)
	{
	taskq_destroy(spl_kmem_cache_taskq);
	}
	Index: vendor-sys/openzfs/dist/module/os/linux/spl/spl-kmem.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/spl/spl-kmem.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/spl/spl-kmem.c (revision 366774)
	@@ -1,618 +1,617 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#include <sys/debug.h>
	#include <sys/sysmacros.h>
	#include <sys/kmem.h>
	#include <sys/vmem.h>

	/*
	* As a general rule kmem_alloc() allocations should be small, preferably
	* just a few pages since they must by physically contiguous. Therefore, a
	* rate limited warning will be printed to the console for any kmem_alloc()
	* which exceeds a reasonable threshold.
	*
	* The default warning threshold is set to sixteen pages but capped at 64K to
	* accommodate systems using large pages. This value was selected to be small
	* enough to ensure the largest allocations are quickly noticed and fixed.
	* But large enough to avoid logging any warnings when a allocation size is
	* larger than optimal but not a serious concern. Since this value is tunable,
	* developers are encouraged to set it lower when testing so any new largish
	* allocations are quickly caught. These warnings may be disabled by setting
	* the threshold to zero.
	*/
	/* BEGIN CSTYLED */
	unsigned int spl_kmem_alloc_warn = MIN(16 * PAGE_SIZE, 64 * 1024);
	module_param(spl_kmem_alloc_warn, uint, 0644);
	MODULE_PARM_DESC(spl_kmem_alloc_warn,
	"Warning threshold in bytes for a kmem_alloc()");
	EXPORT_SYMBOL(spl_kmem_alloc_warn);

	/*
	* Large kmem_alloc() allocations will fail if they exceed KMALLOC_MAX_SIZE.
	* Allocations which are marginally smaller than this limit may succeed but
	* should still be avoided due to the expense of locating a contiguous range
	* of free pages. Therefore, a maximum kmem size with reasonable safely
	* margin of 4x is set. Kmem_alloc() allocations larger than this maximum
	* will quickly fail. Vmem_alloc() allocations less than or equal to this
	* value will use kmalloc(), but shift to vmalloc() when exceeding this value.
	*/
	unsigned int spl_kmem_alloc_max = (KMALLOC_MAX_SIZE >> 2);
	module_param(spl_kmem_alloc_max, uint, 0644);
	MODULE_PARM_DESC(spl_kmem_alloc_max,
	"Maximum size in bytes for a kmem_alloc()");
	EXPORT_SYMBOL(spl_kmem_alloc_max);
	/* END CSTYLED */

	int
	kmem_debugging(void)
	{
	return (0);
	}
	EXPORT_SYMBOL(kmem_debugging);

	char *
	kmem_vasprintf(const char *fmt, va_list ap)
	{
	va_list aq;
	char *ptr;

	do {
	va_copy(aq, ap);
	ptr = kvasprintf(kmem_flags_convert(KM_SLEEP), fmt, aq);
	va_end(aq);
	} while (ptr == NULL);

	return (ptr);
	}
	EXPORT_SYMBOL(kmem_vasprintf);

	char *
	kmem_asprintf(const char *fmt, ...)
	{
	va_list ap;
	char *ptr;

	do {
	va_start(ap, fmt);
	ptr = kvasprintf(kmem_flags_convert(KM_SLEEP), fmt, ap);
	va_end(ap);
	} while (ptr == NULL);

	return (ptr);
	}
	EXPORT_SYMBOL(kmem_asprintf);

	static char *
	__strdup(const char *str, int flags)
	{
	char *ptr;
	int n;

	n = strlen(str);
	ptr = kmalloc(n + 1, kmem_flags_convert(flags));
	if (ptr)
	memcpy(ptr, str, n + 1);

	return (ptr);
	}

	char *
	kmem_strdup(const char *str)
	{
	return (__strdup(str, KM_SLEEP));
	}
	EXPORT_SYMBOL(kmem_strdup);

	void
	kmem_strfree(char *str)
	{
	kfree(str);
	}
	EXPORT_SYMBOL(kmem_strfree);

	void *
	spl_kvmalloc(size_t size, gfp_t lflags)
	{
	#ifdef HAVE_KVMALLOC
	/*
	* GFP_KERNEL allocations can safely use kvmalloc which may
	* improve performance by avoiding a) high latency caused by
	* vmalloc's on-access allocation, b) performance loss due to
	* MMU memory address mapping and c) vmalloc locking overhead.
	* This has the side-effect that the slab statistics will
	* incorrectly report this as a vmem allocation, but that is
	* purely cosmetic.
	*/
	if ((lflags & GFP_KERNEL) == GFP_KERNEL)
	return (kvmalloc(size, lflags));
	#endif

	gfp_t kmalloc_lflags = lflags;

	if (size > PAGE_SIZE) {
	/*
	* We need to set __GFP_NOWARN here since spl_kvmalloc is not
	* only called by spl_kmem_alloc_impl but can be called
	* directly with custom lflags, too. In that case
	* kmem_flags_convert does not get called, which would
	* implicitly set __GFP_NOWARN.
	*/
	kmalloc_lflags \|= __GFP_NOWARN;

	/*
	* N.B. __GFP_RETRY_MAYFAIL is supported only for large
	* e (>32kB) allocations.
	*
	* We have to override __GFP_RETRY_MAYFAIL by __GFP_NORETRY
	* for !costly requests because there is no other way to tell
	* the allocator that we want to fail rather than retry
	* endlessly.
	*/
	if (!(kmalloc_lflags & __GFP_RETRY_MAYFAIL) \|\|
	(size <= PAGE_SIZE << PAGE_ALLOC_COSTLY_ORDER)) {
	kmalloc_lflags \|= __GFP_NORETRY;
	}
	}

	/*
	* We first try kmalloc - even for big sizes - and fall back to
	* spl_vmalloc if that fails.
	*
	* For non-__GFP-RECLAIM allocations we always stick to
	* kmalloc_node, and fail when kmalloc is not successful (returns
	* NULL).
	* We cannot fall back to spl_vmalloc in this case because spl_vmalloc
	* internally uses GPF_KERNEL allocations.
	*/
	void *ptr = kmalloc_node(size, kmalloc_lflags, NUMA_NO_NODE);
	if (ptr \|\| size <= PAGE_SIZE \|\|
	(lflags & __GFP_RECLAIM) != __GFP_RECLAIM) {
	return (ptr);
	}

	return (spl_vmalloc(size, lflags \| __GFP_HIGHMEM));
	}

	/*
	* General purpose unified implementation of kmem_alloc(). It is an
	* amalgamation of Linux and Illumos allocator design. It should never be
	* exported to ensure that code using kmem_alloc()/kmem_zalloc() remains
	* relatively portable. Consumers may only access this function through
	* wrappers that enforce the common flags to ensure portability.
	*/
	inline void *
	spl_kmem_alloc_impl(size_t size, int flags, int node)
	{
	gfp_t lflags = kmem_flags_convert(flags);
	void *ptr;

	/*
	* Log abnormally large allocations and rate limit the console output.
	* Allocations larger than spl_kmem_alloc_warn should be performed
	* through the vmem_alloc()/vmem_zalloc() interfaces.
	*/
	if ((spl_kmem_alloc_warn > 0) && (size > spl_kmem_alloc_warn) &&
	!(flags & KM_VMEM)) {
	printk(KERN_WARNING
	"Large kmem_alloc(%lu, 0x%x), please file an issue at:\n"
	- "https://github.com/zfsonlinux/zfs/issues/new\n",
	+ "https://github.com/openzfs/zfs/issues/new\n",
	(unsigned long)size, flags);
	dump_stack();
	}

	/*
	* Use a loop because kmalloc_node() can fail when GFP_KERNEL is used
	* unlike kmem_alloc() with KM_SLEEP on Illumos.
	*/
	do {
	/*
	* Calling kmalloc_node() when the size >= spl_kmem_alloc_max
	* is unsafe. This must fail for all for kmem_alloc() and
	* kmem_zalloc() callers.
	*
	* For vmem_alloc() and vmem_zalloc() callers it is permissible
	* to use spl_vmalloc(). However, in general use of
	* spl_vmalloc() is strongly discouraged because a global lock
	* must be acquired. Contention on this lock can significantly
	* impact performance so frequently manipulating the virtual
	* address space is strongly discouraged.
	*/
	if (size > spl_kmem_alloc_max) {
	if (flags & KM_VMEM) {
	ptr = spl_vmalloc(size, lflags \| __GFP_HIGHMEM);
	} else {
	return (NULL);
	}
	} else {
	if (flags & KM_VMEM) {
	ptr = spl_kvmalloc(size, lflags);
	} else {
	ptr = kmalloc_node(size, lflags, node);
	}
	}

	if (likely(ptr) \|\| (flags & KM_NOSLEEP))
	return (ptr);

	/*
	* Try hard to satisfy the allocation. However, when progress
	* cannot be made, the allocation is allowed to fail.
	*/
	if ((lflags & GFP_KERNEL) == GFP_KERNEL)
	lflags \|= __GFP_RETRY_MAYFAIL;

	/*
	* Use cond_resched() instead of congestion_wait() to avoid
	* deadlocking systems where there are no block devices.
	*/
	cond_resched();
	} while (1);

	return (NULL);
	}

	inline void
	spl_kmem_free_impl(const void *buf, size_t size)
	{
	if (is_vmalloc_addr(buf))
	vfree(buf);
	else
	kfree(buf);
	}

	/*
	* Memory allocation and accounting for kmem_* * style allocations. When
	* DEBUG_KMEM is enabled the total memory allocated will be tracked and
	* any memory leaked will be reported during module unload.
	*
	* ./configure --enable-debug-kmem
	*/
	#ifdef DEBUG_KMEM

	/* Shim layer memory accounting */
	#ifdef HAVE_ATOMIC64_T
	atomic64_t kmem_alloc_used = ATOMIC64_INIT(0);
	unsigned long long kmem_alloc_max = 0;
	#else /* HAVE_ATOMIC64_T */
	atomic_t kmem_alloc_used = ATOMIC_INIT(0);
	unsigned long long kmem_alloc_max = 0;
	#endif /* HAVE_ATOMIC64_T */

	EXPORT_SYMBOL(kmem_alloc_used);
	EXPORT_SYMBOL(kmem_alloc_max);

	inline void *
	spl_kmem_alloc_debug(size_t size, int flags, int node)
	{
	void *ptr;

	ptr = spl_kmem_alloc_impl(size, flags, node);
	if (ptr) {
	kmem_alloc_used_add(size);
	if (unlikely(kmem_alloc_used_read() > kmem_alloc_max))
	kmem_alloc_max = kmem_alloc_used_read();
	}

	return (ptr);
	}

	inline void
	spl_kmem_free_debug(const void *ptr, size_t size)
	{
	kmem_alloc_used_sub(size);
	spl_kmem_free_impl(ptr, size);
	}

	/*
	* When DEBUG_KMEM_TRACKING is enabled not only will total bytes be tracked
	* but also the location of every alloc and free. When the SPL module is
	* unloaded a list of all leaked addresses and where they were allocated
	* will be dumped to the console. Enabling this feature has a significant
	* impact on performance but it makes finding memory leaks straight forward.
	*
	* Not surprisingly with debugging enabled the xmem_locks are very highly
	* contended particularly on xfree(). If we want to run with this detailed
	* debugging enabled for anything other than debugging we need to minimize
	* the contention by moving to a lock per xmem_table entry model.
	*
	* ./configure --enable-debug-kmem-tracking
	*/
	#ifdef DEBUG_KMEM_TRACKING

	#include <linux/hash.h>
	#include <linux/ctype.h>

	#define KMEM_HASH_BITS 10
	#define KMEM_TABLE_SIZE (1 << KMEM_HASH_BITS)

	typedef struct kmem_debug {
	struct hlist_node kd_hlist; /* Hash node linkage */
	struct list_head kd_list; /* List of all allocations */
	void kd_addr; / Allocation pointer */
	size_t kd_size; /* Allocation size */
	const char kd_func; / Allocation function */
	int kd_line; /* Allocation line */
	} kmem_debug_t;

	static spinlock_t kmem_lock;
	static struct hlist_head kmem_table[KMEM_TABLE_SIZE];
	static struct list_head kmem_list;

	static kmem_debug_t *
	kmem_del_init(spinlock_t lock, struct hlist_head table,
	int bits, const void *addr)
	{
	struct hlist_head *head;
	struct hlist_node *node = NULL;
	struct kmem_debug *p;
	unsigned long flags;

	spin_lock_irqsave(lock, flags);

	head = &table[hash_ptr((void *)addr, bits)];
	hlist_for_each(node, head) {
	p = list_entry(node, struct kmem_debug, kd_hlist);
	if (p->kd_addr == addr) {
	hlist_del_init(&p->kd_hlist);
	list_del_init(&p->kd_list);
	spin_unlock_irqrestore(lock, flags);
	return (p);
	}
	}

	spin_unlock_irqrestore(lock, flags);

	return (NULL);
	}

	inline void *
	spl_kmem_alloc_track(size_t size, int flags,
	const char *func, int line, int node)
	{
	void *ptr = NULL;
	kmem_debug_t *dptr;
	unsigned long irq_flags;

	dptr = kmalloc(sizeof (kmem_debug_t), kmem_flags_convert(flags));
	if (dptr == NULL)
	return (NULL);

	dptr->kd_func = __strdup(func, flags);
	if (dptr->kd_func == NULL) {
	kfree(dptr);
	return (NULL);
	}

	ptr = spl_kmem_alloc_debug(size, flags, node);
	if (ptr == NULL) {
	kfree(dptr->kd_func);
	kfree(dptr);
	return (NULL);
	}

	INIT_HLIST_NODE(&dptr->kd_hlist);
	INIT_LIST_HEAD(&dptr->kd_list);

	dptr->kd_addr = ptr;
	dptr->kd_size = size;
	dptr->kd_line = line;

	spin_lock_irqsave(&kmem_lock, irq_flags);
	hlist_add_head(&dptr->kd_hlist,
	&kmem_table[hash_ptr(ptr, KMEM_HASH_BITS)]);
	list_add_tail(&dptr->kd_list, &kmem_list);
	spin_unlock_irqrestore(&kmem_lock, irq_flags);

	return (ptr);
	}

	inline void
	spl_kmem_free_track(const void *ptr, size_t size)
	{
	kmem_debug_t *dptr;

	/* Ignore NULL pointer since we haven't tracked it at all */
	if (ptr == NULL)
	return;

	/* Must exist in hash due to kmem_alloc() */
	dptr = kmem_del_init(&kmem_lock, kmem_table, KMEM_HASH_BITS, ptr);
	ASSERT3P(dptr, !=, NULL);
	ASSERT3S(dptr->kd_size, ==, size);

	kfree(dptr->kd_func);
	kfree(dptr);

	spl_kmem_free_debug(ptr, size);
	}
	#endif /* DEBUG_KMEM_TRACKING */
	#endif /* DEBUG_KMEM */

	/*
	* Public kmem_alloc(), kmem_zalloc() and kmem_free() interfaces.
	*/
	void *
	spl_kmem_alloc(size_t size, int flags, const char *func, int line)
	{
	ASSERT0(flags & ~KM_PUBLIC_MASK);

	#if !defined(DEBUG_KMEM)
	return (spl_kmem_alloc_impl(size, flags, NUMA_NO_NODE));
	#elif !defined(DEBUG_KMEM_TRACKING)
	return (spl_kmem_alloc_debug(size, flags, NUMA_NO_NODE));
	#else
	return (spl_kmem_alloc_track(size, flags, func, line, NUMA_NO_NODE));
	#endif
	}
	EXPORT_SYMBOL(spl_kmem_alloc);

	void *
	spl_kmem_zalloc(size_t size, int flags, const char *func, int line)
	{
	ASSERT0(flags & ~KM_PUBLIC_MASK);

	flags \|= KM_ZERO;

	#if !defined(DEBUG_KMEM)
	return (spl_kmem_alloc_impl(size, flags, NUMA_NO_NODE));
	#elif !defined(DEBUG_KMEM_TRACKING)
	return (spl_kmem_alloc_debug(size, flags, NUMA_NO_NODE));
	#else
	return (spl_kmem_alloc_track(size, flags, func, line, NUMA_NO_NODE));
	#endif
	}
	EXPORT_SYMBOL(spl_kmem_zalloc);

	void
	spl_kmem_free(const void *buf, size_t size)
	{
	#if !defined(DEBUG_KMEM)
	return (spl_kmem_free_impl(buf, size));
	#elif !defined(DEBUG_KMEM_TRACKING)
	return (spl_kmem_free_debug(buf, size));
	#else
	return (spl_kmem_free_track(buf, size));
	#endif
	}
	EXPORT_SYMBOL(spl_kmem_free);

	#if defined(DEBUG_KMEM) && defined(DEBUG_KMEM_TRACKING)
	static char *
	spl_sprintf_addr(kmem_debug_t kd, char str, int len, int min)
	{
	int size = ((len - 1) < kd->kd_size) ? (len - 1) : kd->kd_size;
	int i, flag = 1;

	ASSERT(str != NULL && len >= 17);
	memset(str, 0, len);

	/*
	* Check for a fully printable string, and while we are at
	* it place the printable characters in the passed buffer.
	*/
	for (i = 0; i < size; i++) {
	str[i] = ((char *)(kd->kd_addr))[i];
	if (isprint(str[i])) {
	continue;
	} else {
	/*
	* Minimum number of printable characters found
	* to make it worthwhile to print this as ascii.
	*/
	if (i > min)
	break;

	flag = 0;
	break;
	}
	}

	if (!flag) {
	sprintf(str, "%02x%02x%02x%02x%02x%02x%02x%02x",
	((uint8_t )kd->kd_addr),
	((uint8_t )kd->kd_addr + 2),
	((uint8_t )kd->kd_addr + 4),
	((uint8_t )kd->kd_addr + 6),
	((uint8_t )kd->kd_addr + 8),
	((uint8_t )kd->kd_addr + 10),
	((uint8_t )kd->kd_addr + 12),
	((uint8_t )kd->kd_addr + 14));
	}

	return (str);
	}

	static int
	spl_kmem_init_tracking(struct list_head list, spinlock_t lock, int size)
	{
	int i;

	spin_lock_init(lock);
	INIT_LIST_HEAD(list);

	for (i = 0; i < size; i++)
	INIT_HLIST_HEAD(&kmem_table[i]);

	return (0);
	}

	static void
	spl_kmem_fini_tracking(struct list_head list, spinlock_t lock)
	{
	unsigned long flags;
	kmem_debug_t *kd = NULL;
	char str[17];

	spin_lock_irqsave(lock, flags);
	if (!list_empty(list))
	printk(KERN_WARNING "%-16s %-5s %-16s %s:%s\n", "address",
	"size", "data", "func", "line");

	list_for_each_entry(kd, list, kd_list) {
	printk(KERN_WARNING "%p %-5d %-16s %s:%d\n", kd->kd_addr,
	(int)kd->kd_size, spl_sprintf_addr(kd, str, 17, 8),
	kd->kd_func, kd->kd_line);
	}

	spin_unlock_irqrestore(lock, flags);
	}
	#endif /* DEBUG_KMEM && DEBUG_KMEM_TRACKING */

	int
	spl_kmem_init(void)
	{

	#ifdef DEBUG_KMEM
	kmem_alloc_used_set(0);



	#ifdef DEBUG_KMEM_TRACKING
	spl_kmem_init_tracking(&kmem_list, &kmem_lock, KMEM_TABLE_SIZE);
	#endif /* DEBUG_KMEM_TRACKING */
	#endif /* DEBUG_KMEM */

	return (0);
	}

	void
	spl_kmem_fini(void)
	{
	#ifdef DEBUG_KMEM
	/*
	* Display all unreclaimed memory addresses, including the
	* allocation size and the first few bytes of what's located
	* at that address to aid in debugging. Performance is not
	* a serious concern here since it is module unload time.
	*/
	if (kmem_alloc_used_read() != 0)
	printk(KERN_WARNING "kmem leaked %ld/%llu bytes\n",
	(unsigned long)kmem_alloc_used_read(), kmem_alloc_max);

	#ifdef DEBUG_KMEM_TRACKING
	spl_kmem_fini_tracking(&kmem_list, &kmem_lock);
	#endif /* DEBUG_KMEM_TRACKING */
	#endif /* DEBUG_KMEM */
	}
	Index: vendor-sys/openzfs/dist/module/os/linux/spl/spl-kstat.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/spl/spl-kstat.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/spl/spl-kstat.c (revision 366774)
	@@ -1,782 +1,781 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*
	* Solaris Porting Layer (SPL) Kstat Implementation.
	*
	* Links to Illumos.org for more information on kstat function:
	* [1] https://illumos.org/man/1M/kstat
	* [2] https://illumos.org/man/9f/kstat_create
	*/

	#include <linux/seq_file.h>
	#include <sys/kstat.h>
	#include <sys/vmem.h>
	#include <sys/cmn_err.h>
	#include <sys/sysmacros.h>

	static kmutex_t kstat_module_lock;
	static struct list_head kstat_module_list;
	static kid_t kstat_id;

	static int
	kstat_resize_raw(kstat_t *ksp)
	{
	if (ksp->ks_raw_bufsize == KSTAT_RAW_MAX)
	return (ENOMEM);

	vmem_free(ksp->ks_raw_buf, ksp->ks_raw_bufsize);
	ksp->ks_raw_bufsize = MIN(ksp->ks_raw_bufsize * 2, KSTAT_RAW_MAX);
	ksp->ks_raw_buf = vmem_alloc(ksp->ks_raw_bufsize, KM_SLEEP);

	return (0);
	}

	void
	kstat_waitq_enter(kstat_io_t *kiop)
	{
	hrtime_t new, delta;
	ulong_t wcnt;

	new = gethrtime();
	delta = new - kiop->wlastupdate;
	kiop->wlastupdate = new;
	wcnt = kiop->wcnt++;
	if (wcnt != 0) {
	kiop->wlentime += delta * wcnt;
	kiop->wtime += delta;
	}
	}
	EXPORT_SYMBOL(kstat_waitq_enter);

	void
	kstat_waitq_exit(kstat_io_t *kiop)
	{
	hrtime_t new, delta;
	ulong_t wcnt;

	new = gethrtime();
	delta = new - kiop->wlastupdate;
	kiop->wlastupdate = new;
	wcnt = kiop->wcnt--;
	ASSERT((int)wcnt > 0);
	kiop->wlentime += delta * wcnt;
	kiop->wtime += delta;
	}
	EXPORT_SYMBOL(kstat_waitq_exit);

	void
	kstat_runq_enter(kstat_io_t *kiop)
	{
	hrtime_t new, delta;
	ulong_t rcnt;

	new = gethrtime();
	delta = new - kiop->rlastupdate;
	kiop->rlastupdate = new;
	rcnt = kiop->rcnt++;
	if (rcnt != 0) {
	kiop->rlentime += delta * rcnt;
	kiop->rtime += delta;
	}
	}
	EXPORT_SYMBOL(kstat_runq_enter);

	void
	kstat_runq_exit(kstat_io_t *kiop)
	{
	hrtime_t new, delta;
	ulong_t rcnt;

	new = gethrtime();
	delta = new - kiop->rlastupdate;
	kiop->rlastupdate = new;
	rcnt = kiop->rcnt--;
	ASSERT((int)rcnt > 0);
	kiop->rlentime += delta * rcnt;
	kiop->rtime += delta;
	}
	EXPORT_SYMBOL(kstat_runq_exit);

	static int
	kstat_seq_show_headers(struct seq_file *f)
	{
	kstat_t ksp = (kstat_t )f->private;
	int rc = 0;

	ASSERT(ksp->ks_magic == KS_MAGIC);

	seq_printf(f, "%d %d 0x%02x %d %d %lld %lld\n",
	ksp->ks_kid, ksp->ks_type, ksp->ks_flags,
	ksp->ks_ndata, (int)ksp->ks_data_size,
	ksp->ks_crtime, ksp->ks_snaptime);

	switch (ksp->ks_type) {
	case KSTAT_TYPE_RAW:
	restart:
	if (ksp->ks_raw_ops.headers) {
	rc = ksp->ks_raw_ops.headers(
	ksp->ks_raw_buf, ksp->ks_raw_bufsize);
	if (rc == ENOMEM && !kstat_resize_raw(ksp))
	goto restart;
	if (!rc)
	seq_puts(f, ksp->ks_raw_buf);
	} else {
	seq_printf(f, "raw data\n");
	}
	break;
	case KSTAT_TYPE_NAMED:
	seq_printf(f, "%-31s %-4s %s\n",
	"name", "type", "data");
	break;
	case KSTAT_TYPE_INTR:
	seq_printf(f, "%-8s %-8s %-8s %-8s %-8s\n",
	"hard", "soft", "watchdog",
	"spurious", "multsvc");
	break;
	case KSTAT_TYPE_IO:
	seq_printf(f,
	"%-8s %-8s %-8s %-8s %-8s %-8s "
	"%-8s %-8s %-8s %-8s %-8s %-8s\n",
	"nread", "nwritten", "reads", "writes",
	"wtime", "wlentime", "wupdate",
	"rtime", "rlentime", "rupdate",
	"wcnt", "rcnt");
	break;
	case KSTAT_TYPE_TIMER:
	seq_printf(f,
	"%-31s %-8s "
	"%-8s %-8s %-8s %-8s %-8s\n",
	"name", "events", "elapsed",
	"min", "max", "start", "stop");
	break;
	default:
	PANIC("Undefined kstat type %d\n", ksp->ks_type);
	}

	return (-rc);
	}

	static int
	kstat_seq_show_raw(struct seq_file f, unsigned char p, int l)
	{
	int i, j;

	for (i = 0; ; i++) {
	seq_printf(f, "%03x:", i);

	for (j = 0; j < 16; j++) {
	if (i * 16 + j >= l) {
	seq_printf(f, "\n");
	goto out;
	}

	seq_printf(f, " %02x", (unsigned char)p[i * 16 + j]);
	}
	seq_printf(f, "\n");
	}
	out:
	return (0);
	}

	static int
	kstat_seq_show_named(struct seq_file f, kstat_named_t knp)
	{
	seq_printf(f, "%-31s %-4d ", knp->name, knp->data_type);

	switch (knp->data_type) {
	case KSTAT_DATA_CHAR:
	knp->value.c[15] = '\0'; /* NULL terminate */
	seq_printf(f, "%-16s", knp->value.c);
	break;
	/*
	* NOTE - We need to be more careful able what tokens are
	* used for each arch, for now this is correct for x86_64.
	*/
	case KSTAT_DATA_INT32:
	seq_printf(f, "%d", knp->value.i32);
	break;
	case KSTAT_DATA_UINT32:
	seq_printf(f, "%u", knp->value.ui32);
	break;
	case KSTAT_DATA_INT64:
	seq_printf(f, "%lld", (signed long long)knp->value.i64);
	break;
	case KSTAT_DATA_UINT64:
	seq_printf(f, "%llu",
	(unsigned long long)knp->value.ui64);
	break;
	case KSTAT_DATA_LONG:
	seq_printf(f, "%ld", knp->value.l);
	break;
	case KSTAT_DATA_ULONG:
	seq_printf(f, "%lu", knp->value.ul);
	break;
	case KSTAT_DATA_STRING:
	KSTAT_NAMED_STR_PTR(knp)
	[KSTAT_NAMED_STR_BUFLEN(knp)-1] = '\0';
	seq_printf(f, "%s", KSTAT_NAMED_STR_PTR(knp));
	break;
	default:
	PANIC("Undefined kstat data type %d\n", knp->data_type);
	}

	seq_printf(f, "\n");

	return (0);
	}

	static int
	kstat_seq_show_intr(struct seq_file f, kstat_intr_t kip)
	{
	seq_printf(f, "%-8u %-8u %-8u %-8u %-8u\n",
	kip->intrs[KSTAT_INTR_HARD],
	kip->intrs[KSTAT_INTR_SOFT],
	kip->intrs[KSTAT_INTR_WATCHDOG],
	kip->intrs[KSTAT_INTR_SPURIOUS],
	kip->intrs[KSTAT_INTR_MULTSVC]);

	return (0);
	}

	static int
	kstat_seq_show_io(struct seq_file f, kstat_io_t kip)
	{
	/* though wlentime & friends are signed, they will never be negative */
	seq_printf(f,
	"%-8llu %-8llu %-8u %-8u %-8llu %-8llu "
	"%-8llu %-8llu %-8llu %-8llu %-8u %-8u\n",
	kip->nread, kip->nwritten,
	kip->reads, kip->writes,
	kip->wtime, kip->wlentime, kip->wlastupdate,
	kip->rtime, kip->rlentime, kip->rlastupdate,
	kip->wcnt, kip->rcnt);

	return (0);
	}

	static int
	kstat_seq_show_timer(struct seq_file f, kstat_timer_t ktp)
	{
	seq_printf(f,
	"%-31s %-8llu %-8llu %-8llu %-8llu %-8llu %-8llu\n",
	ktp->name, ktp->num_events, ktp->elapsed_time,
	ktp->min_time, ktp->max_time,
	ktp->start_time, ktp->stop_time);

	return (0);
	}

	static int
	kstat_seq_show(struct seq_file f, void p)
	{
	kstat_t ksp = (kstat_t )f->private;
	int rc = 0;

	ASSERT(ksp->ks_magic == KS_MAGIC);

	switch (ksp->ks_type) {
	case KSTAT_TYPE_RAW:
	restart:
	if (ksp->ks_raw_ops.data) {
	rc = ksp->ks_raw_ops.data(
	ksp->ks_raw_buf, ksp->ks_raw_bufsize, p);
	if (rc == ENOMEM && !kstat_resize_raw(ksp))
	goto restart;
	if (!rc)
	seq_puts(f, ksp->ks_raw_buf);
	} else {
	ASSERT(ksp->ks_ndata == 1);
	rc = kstat_seq_show_raw(f, ksp->ks_data,
	ksp->ks_data_size);
	}
	break;
	case KSTAT_TYPE_NAMED:
	rc = kstat_seq_show_named(f, (kstat_named_t *)p);
	break;
	case KSTAT_TYPE_INTR:
	rc = kstat_seq_show_intr(f, (kstat_intr_t *)p);
	break;
	case KSTAT_TYPE_IO:
	rc = kstat_seq_show_io(f, (kstat_io_t *)p);
	break;
	case KSTAT_TYPE_TIMER:
	rc = kstat_seq_show_timer(f, (kstat_timer_t *)p);
	break;
	default:
	PANIC("Undefined kstat type %d\n", ksp->ks_type);
	}

	return (-rc);
	}

	static int
	kstat_default_update(kstat_t *ksp, int rw)
	{
	ASSERT(ksp != NULL);

	if (rw == KSTAT_WRITE)
	return (EACCES);

	return (0);
	}

	static void *
	kstat_seq_data_addr(kstat_t *ksp, loff_t n)
	{
	void *rc = NULL;

	switch (ksp->ks_type) {
	case KSTAT_TYPE_RAW:
	if (ksp->ks_raw_ops.addr)
	rc = ksp->ks_raw_ops.addr(ksp, n);
	else
	rc = ksp->ks_data;
	break;
	case KSTAT_TYPE_NAMED:
	rc = ksp->ks_data + n * sizeof (kstat_named_t);
	break;
	case KSTAT_TYPE_INTR:
	rc = ksp->ks_data + n * sizeof (kstat_intr_t);
	break;
	case KSTAT_TYPE_IO:
	rc = ksp->ks_data + n * sizeof (kstat_io_t);
	break;
	case KSTAT_TYPE_TIMER:
	rc = ksp->ks_data + n * sizeof (kstat_timer_t);
	break;
	default:
	PANIC("Undefined kstat type %d\n", ksp->ks_type);
	}

	return (rc);
	}

	static void *
	kstat_seq_start(struct seq_file f, loff_t pos)
	{
	loff_t n = *pos;
	kstat_t ksp = (kstat_t )f->private;
	ASSERT(ksp->ks_magic == KS_MAGIC);

	mutex_enter(ksp->ks_lock);

	if (ksp->ks_type == KSTAT_TYPE_RAW) {
	ksp->ks_raw_bufsize = PAGE_SIZE;
	ksp->ks_raw_buf = vmem_alloc(ksp->ks_raw_bufsize, KM_SLEEP);
	}

	/* Dynamically update kstat, on error existing kstats are used */
	(void) ksp->ks_update(ksp, KSTAT_READ);

	ksp->ks_snaptime = gethrtime();

	if (!(ksp->ks_flags & KSTAT_FLAG_NO_HEADERS) && !n &&
	kstat_seq_show_headers(f))
	return (NULL);

	if (n >= ksp->ks_ndata)
	return (NULL);

	return (kstat_seq_data_addr(ksp, n));
	}

	static void *
	kstat_seq_next(struct seq_file f, void p, loff_t *pos)
	{
	kstat_t ksp = (kstat_t )f->private;
	ASSERT(ksp->ks_magic == KS_MAGIC);

	++*pos;
	if (*pos >= ksp->ks_ndata)
	return (NULL);

	return (kstat_seq_data_addr(ksp, *pos));
	}

	static void
	kstat_seq_stop(struct seq_file f, void v)
	{
	kstat_t ksp = (kstat_t )f->private;
	ASSERT(ksp->ks_magic == KS_MAGIC);

	if (ksp->ks_type == KSTAT_TYPE_RAW)
	vmem_free(ksp->ks_raw_buf, ksp->ks_raw_bufsize);

	mutex_exit(ksp->ks_lock);
	}

	static struct seq_operations kstat_seq_ops = {
	.show = kstat_seq_show,
	.start = kstat_seq_start,
	.next = kstat_seq_next,
	.stop = kstat_seq_stop,
	};

	static kstat_module_t *
	kstat_find_module(char *name)
	{
	kstat_module_t *module = NULL;

	list_for_each_entry(module, &kstat_module_list, ksm_module_list) {
	if (strncmp(name, module->ksm_name, KSTAT_STRLEN) == 0)
	return (module);
	}

	return (NULL);
	}

	static kstat_module_t *
	kstat_create_module(char *name)
	{
	kstat_module_t *module;
	struct proc_dir_entry *pde;

	pde = proc_mkdir(name, proc_spl_kstat);
	if (pde == NULL)
	return (NULL);

	module = kmem_alloc(sizeof (kstat_module_t), KM_SLEEP);
	module->ksm_proc = pde;
	strlcpy(module->ksm_name, name, KSTAT_STRLEN+1);
	INIT_LIST_HEAD(&module->ksm_kstat_list);
	list_add_tail(&module->ksm_module_list, &kstat_module_list);

	return (module);

	}

	static void
	kstat_delete_module(kstat_module_t *module)
	{
	ASSERT(list_empty(&module->ksm_kstat_list));
	remove_proc_entry(module->ksm_name, proc_spl_kstat);
	list_del(&module->ksm_module_list);
	kmem_free(module, sizeof (kstat_module_t));
	}

	static int
	proc_kstat_open(struct inode inode, struct file filp)
	{
	struct seq_file *f;
	int rc;

	rc = seq_open(filp, &kstat_seq_ops);
	if (rc)
	return (rc);

	f = filp->private_data;
	f->private = PDE_DATA(inode);

	return (rc);
	}

	static ssize_t
	proc_kstat_write(struct file filp, const char __user buf, size_t len,
	loff_t *ppos)
	{
	struct seq_file *f = filp->private_data;
	kstat_t *ksp = f->private;
	int rc;

	ASSERT(ksp->ks_magic == KS_MAGIC);

	mutex_enter(ksp->ks_lock);
	rc = ksp->ks_update(ksp, KSTAT_WRITE);
	mutex_exit(ksp->ks_lock);

	if (rc)
	return (-rc);

	*ppos += len;
	return (len);
	}

	static const kstat_proc_op_t proc_kstat_operations = {
	#ifdef HAVE_PROC_OPS_STRUCT
	.proc_open = proc_kstat_open,
	.proc_write = proc_kstat_write,
	.proc_read = seq_read,
	.proc_lseek = seq_lseek,
	.proc_release = seq_release,
	#else
	.open = proc_kstat_open,
	.write = proc_kstat_write,
	.read = seq_read,
	.llseek = seq_lseek,
	.release = seq_release,
	#endif
	};

	void
	__kstat_set_raw_ops(kstat_t *ksp,
	int (headers)(char buf, size_t size),
	int (data)(char buf, size_t size, void *data),
	void (addr)(kstat_t *ksp, loff_t index))
	{
	ksp->ks_raw_ops.headers = headers;
	ksp->ks_raw_ops.data = data;
	ksp->ks_raw_ops.addr = addr;
	}
	EXPORT_SYMBOL(__kstat_set_raw_ops);

	void
	kstat_proc_entry_init(kstat_proc_entry_t kpep, const char module,
	const char *name)
	{
	kpep->kpe_owner = NULL;
	kpep->kpe_proc = NULL;
	INIT_LIST_HEAD(&kpep->kpe_list);
	strncpy(kpep->kpe_module, module, KSTAT_STRLEN);
	strncpy(kpep->kpe_name, name, KSTAT_STRLEN);
	}
	EXPORT_SYMBOL(kstat_proc_entry_init);

	kstat_t *
	__kstat_create(const char ks_module, int ks_instance, const char ks_name,
	const char *ks_class, uchar_t ks_type, uint_t ks_ndata,
	uchar_t ks_flags)
	{
	kstat_t *ksp;

	ASSERT(ks_module);
	ASSERT(ks_instance == 0);
	ASSERT(ks_name);

	if ((ks_type == KSTAT_TYPE_INTR) \|\| (ks_type == KSTAT_TYPE_IO))
	ASSERT(ks_ndata == 1);

	ksp = kmem_zalloc(sizeof (*ksp), KM_SLEEP);
	if (ksp == NULL)
	return (ksp);

	mutex_enter(&kstat_module_lock);
	ksp->ks_kid = kstat_id;
	kstat_id++;
	mutex_exit(&kstat_module_lock);

	ksp->ks_magic = KS_MAGIC;
	mutex_init(&ksp->ks_private_lock, NULL, MUTEX_DEFAULT, NULL);
	ksp->ks_lock = &ksp->ks_private_lock;

	ksp->ks_crtime = gethrtime();
	ksp->ks_snaptime = ksp->ks_crtime;
	ksp->ks_instance = ks_instance;
	strncpy(ksp->ks_class, ks_class, KSTAT_STRLEN);
	ksp->ks_type = ks_type;
	ksp->ks_flags = ks_flags;
	ksp->ks_update = kstat_default_update;
	ksp->ks_private = NULL;
	ksp->ks_raw_ops.headers = NULL;
	ksp->ks_raw_ops.data = NULL;
	ksp->ks_raw_ops.addr = NULL;
	ksp->ks_raw_buf = NULL;
	ksp->ks_raw_bufsize = 0;
	kstat_proc_entry_init(&ksp->ks_proc, ks_module, ks_name);

	switch (ksp->ks_type) {
	case KSTAT_TYPE_RAW:
	ksp->ks_ndata = 1;
	ksp->ks_data_size = ks_ndata;
	break;
	case KSTAT_TYPE_NAMED:
	ksp->ks_ndata = ks_ndata;
	ksp->ks_data_size = ks_ndata * sizeof (kstat_named_t);
	break;
	case KSTAT_TYPE_INTR:
	ksp->ks_ndata = ks_ndata;
	ksp->ks_data_size = ks_ndata * sizeof (kstat_intr_t);
	break;
	case KSTAT_TYPE_IO:
	ksp->ks_ndata = ks_ndata;
	ksp->ks_data_size = ks_ndata * sizeof (kstat_io_t);
	break;
	case KSTAT_TYPE_TIMER:
	ksp->ks_ndata = ks_ndata;
	ksp->ks_data_size = ks_ndata * sizeof (kstat_timer_t);
	break;
	default:
	PANIC("Undefined kstat type %d\n", ksp->ks_type);
	}

	if (ksp->ks_flags & KSTAT_FLAG_VIRTUAL) {
	ksp->ks_data = NULL;
	} else {
	ksp->ks_data = kmem_zalloc(ksp->ks_data_size, KM_SLEEP);
	if (ksp->ks_data == NULL) {
	kmem_free(ksp, sizeof (*ksp));
	ksp = NULL;
	}
	}

	return (ksp);
	}
	EXPORT_SYMBOL(__kstat_create);

	static int
	kstat_detect_collision(kstat_proc_entry_t *kpep)
	{
	kstat_module_t *module;
	kstat_proc_entry_t *tmp = NULL;
	char *parent;
	char *cp;

	parent = kmem_asprintf("%s", kpep->kpe_module);

	if ((cp = strrchr(parent, '/')) == NULL) {
	kmem_strfree(parent);
	return (0);
	}

	cp[0] = '\0';
	if ((module = kstat_find_module(parent)) != NULL) {
	list_for_each_entry(tmp, &module->ksm_kstat_list, kpe_list) {
	if (strncmp(tmp->kpe_name, cp+1, KSTAT_STRLEN) == 0) {
	kmem_strfree(parent);
	return (EEXIST);
	}
	}
	}

	kmem_strfree(parent);
	return (0);
	}

	/*
	* Add a file to the proc filesystem under the kstat namespace (i.e.
	* /proc/spl/kstat/). The file need not necessarily be implemented as a
	* kstat.
	*/
	void
	kstat_proc_entry_install(kstat_proc_entry_t *kpep, mode_t mode,
	const kstat_proc_op_t proc_ops, void data)
	{
	kstat_module_t *module;
	kstat_proc_entry_t *tmp = NULL;

	ASSERT(kpep);

	mutex_enter(&kstat_module_lock);

	module = kstat_find_module(kpep->kpe_module);
	if (module == NULL) {
	if (kstat_detect_collision(kpep) != 0) {
	cmn_err(CE_WARN, "kstat_create('%s', '%s'): namespace" \
	" collision", kpep->kpe_module, kpep->kpe_name);
	goto out;
	}
	module = kstat_create_module(kpep->kpe_module);
	if (module == NULL)
	goto out;
	}

	/*
	* Only one entry by this name per-module, on failure the module
	* shouldn't be deleted because we know it has at least one entry.
	*/
	list_for_each_entry(tmp, &module->ksm_kstat_list, kpe_list) {
	if (strncmp(tmp->kpe_name, kpep->kpe_name, KSTAT_STRLEN) == 0)
	goto out;
	}

	list_add_tail(&kpep->kpe_list, &module->ksm_kstat_list);

	kpep->kpe_owner = module;
	kpep->kpe_proc = proc_create_data(kpep->kpe_name, mode,
	module->ksm_proc, proc_ops, data);
	if (kpep->kpe_proc == NULL) {
	list_del_init(&kpep->kpe_list);
	if (list_empty(&module->ksm_kstat_list))
	kstat_delete_module(module);
	}
	out:
	mutex_exit(&kstat_module_lock);

	}
	EXPORT_SYMBOL(kstat_proc_entry_install);

	void
	__kstat_install(kstat_t *ksp)
	{
	ASSERT(ksp);
	mode_t mode;
	/* Specify permission modes for different kstats */
	if (strncmp(ksp->ks_proc.kpe_name, "dbufs", KSTAT_STRLEN) == 0) {
	mode = 0600;
	} else {
	mode = 0644;
	}
	kstat_proc_entry_install(
	&ksp->ks_proc, mode, &proc_kstat_operations, ksp);
	}
	EXPORT_SYMBOL(__kstat_install);

	void
	kstat_proc_entry_delete(kstat_proc_entry_t *kpep)
	{
	kstat_module_t *module = kpep->kpe_owner;
	if (kpep->kpe_proc)
	remove_proc_entry(kpep->kpe_name, module->ksm_proc);

	mutex_enter(&kstat_module_lock);
	list_del_init(&kpep->kpe_list);

	/*
	* Remove top level module directory if it wasn't empty before, but now
	* is.
	*/
	if (kpep->kpe_proc && list_empty(&module->ksm_kstat_list))
	kstat_delete_module(module);
	mutex_exit(&kstat_module_lock);

	}
	EXPORT_SYMBOL(kstat_proc_entry_delete);

	void
	__kstat_delete(kstat_t *ksp)
	{
	kstat_proc_entry_delete(&ksp->ks_proc);

	if (!(ksp->ks_flags & KSTAT_FLAG_VIRTUAL))
	kmem_free(ksp->ks_data, ksp->ks_data_size);

	ksp->ks_lock = NULL;
	mutex_destroy(&ksp->ks_private_lock);
	kmem_free(ksp, sizeof (*ksp));
	}
	EXPORT_SYMBOL(__kstat_delete);

	int
	spl_kstat_init(void)
	{
	mutex_init(&kstat_module_lock, NULL, MUTEX_DEFAULT, NULL);
	INIT_LIST_HEAD(&kstat_module_list);
	kstat_id = 0;
	return (0);
	}

	void
	spl_kstat_fini(void)
	{
	ASSERT(list_empty(&kstat_module_list));
	mutex_destroy(&kstat_module_lock);
	}
	Index: vendor-sys/openzfs/dist/module/os/linux/spl/spl-proc.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/spl/spl-proc.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/spl/spl-proc.c (revision 366774)
	@@ -1,791 +1,790 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*
	* Solaris Porting Layer (SPL) Proc Implementation.
	*/

	#include <sys/systeminfo.h>
	#include <sys/kstat.h>
	#include <sys/kmem.h>
	#include <sys/kmem_cache.h>
	#include <sys/vmem.h>
	#include <sys/taskq.h>
	#include <sys/proc.h>
	#include <linux/ctype.h>
	#include <linux/kmod.h>
	#include <linux/seq_file.h>
	#include <linux/uaccess.h>
	#include <linux/version.h>

	#if defined(CONSTIFY_PLUGIN) && LINUX_VERSION_CODE >= KERNEL_VERSION(3, 8, 0)
	typedef struct ctl_table __no_const spl_ctl_table;
	#else
	typedef struct ctl_table spl_ctl_table;
	#endif

	static unsigned long table_min = 0;
	static unsigned long table_max = ~0;

	static struct ctl_table_header *spl_header = NULL;
	static struct proc_dir_entry *proc_spl = NULL;
	static struct proc_dir_entry *proc_spl_kmem = NULL;
	static struct proc_dir_entry *proc_spl_kmem_slab = NULL;
	static struct proc_dir_entry *proc_spl_taskq_all = NULL;
	static struct proc_dir_entry *proc_spl_taskq = NULL;
	struct proc_dir_entry *proc_spl_kstat = NULL;

	static int
	proc_copyin_string(char kbuffer, int kbuffer_size, const char ubuffer,
	int ubuffer_size)
	{
	int size;

	if (ubuffer_size > kbuffer_size)
	return (-EOVERFLOW);

	if (copy_from_user((void )kbuffer, (void )ubuffer, ubuffer_size))
	return (-EFAULT);

	/* strip trailing whitespace */
	size = strnlen(kbuffer, ubuffer_size);
	while (size-- >= 0)
	if (!isspace(kbuffer[size]))
	break;

	/* empty string */
	if (size < 0)
	return (-EINVAL);

	/* no space to terminate */
	if (size == kbuffer_size)
	return (-EOVERFLOW);

	kbuffer[size + 1] = 0;
	return (0);
	}

	static int
	proc_copyout_string(char ubuffer, int ubuffer_size, const char kbuffer,
	char *append)
	{
	/*
	* NB if 'append' != NULL, it's a single character to append to the
	* copied out string - usually "\n", for /proc entries and
	* (i.e. a terminating zero byte) for sysctl entries
	*/
	int size = MIN(strlen(kbuffer), ubuffer_size);

	if (copy_to_user(ubuffer, kbuffer, size))
	return (-EFAULT);

	if (append != NULL && size < ubuffer_size) {
	if (copy_to_user(ubuffer + size, append, 1))
	return (-EFAULT);

	size++;
	}

	return (size);
	}

	#ifdef DEBUG_KMEM
	static int
	proc_domemused(struct ctl_table *table, int write,
	void __user buffer, size_t lenp, loff_t *ppos)
	{
	int rc = 0;
	unsigned long min = 0, max = ~0, val;
	spl_ctl_table dummy = *table;

	dummy.data = &val;
	dummy.proc_handler = &proc_dointvec;
	dummy.extra1 = &min;
	dummy.extra2 = &max;

	if (write) {
	ppos += lenp;
	} else {
	#ifdef HAVE_ATOMIC64_T
	val = atomic64_read((atomic64_t *)table->data);
	#else
	val = atomic_read((atomic_t *)table->data);
	#endif /* HAVE_ATOMIC64_T */
	rc = proc_doulongvec_minmax(&dummy, write, buffer, lenp, ppos);
	}

	return (rc);
	}
	#endif /* DEBUG_KMEM */

	static int
	proc_doslab(struct ctl_table *table, int write,
	void __user buffer, size_t lenp, loff_t *ppos)
	{
	int rc = 0;
	unsigned long min = 0, max = ~0, val = 0, mask;
	spl_ctl_table dummy = *table;
	spl_kmem_cache_t *skc = NULL;

	dummy.data = &val;
	dummy.proc_handler = &proc_dointvec;
	dummy.extra1 = &min;
	dummy.extra2 = &max;

	if (write) {
	ppos += lenp;
	} else {
	down_read(&spl_kmem_cache_sem);
	mask = (unsigned long)table->data;

	list_for_each_entry(skc, &spl_kmem_cache_list, skc_list) {

	/* Only use slabs of the correct kmem/vmem type */
	if (!(skc->skc_flags & mask))
	continue;

	/* Sum the specified field for selected slabs */
	switch (mask & (KMC_TOTAL \| KMC_ALLOC \| KMC_MAX)) {
	case KMC_TOTAL:
	val += skc->skc_slab_size * skc->skc_slab_total;
	break;
	case KMC_ALLOC:
	val += skc->skc_obj_size * skc->skc_obj_alloc;
	break;
	case KMC_MAX:
	val += skc->skc_obj_size * skc->skc_obj_max;
	break;
	}
	}

	up_read(&spl_kmem_cache_sem);
	rc = proc_doulongvec_minmax(&dummy, write, buffer, lenp, ppos);
	}

	return (rc);
	}

	static int
	proc_dohostid(struct ctl_table *table, int write,
	void __user buffer, size_t lenp, loff_t *ppos)
	{
	int len, rc = 0;
	char *end, str[32];

	if (write) {
	/*
	* We can't use proc_doulongvec_minmax() in the write
	* case here because hostid while a hex value has no
	* leading 0x which confuses the helper function.
	*/
	rc = proc_copyin_string(str, sizeof (str), buffer, *lenp);
	if (rc < 0)
	return (rc);

	spl_hostid = simple_strtoul(str, &end, 16);
	if (str == end)
	return (-EINVAL);

	} else {
	len = snprintf(str, sizeof (str), "%lx",
	(unsigned long) zone_get_hostid(NULL));
	if (*ppos >= len)
	rc = 0;
	else
	rc = proc_copyout_string(buffer,
	lenp, str + ppos, "\n");

	if (rc >= 0) {
	*lenp = rc;
	*ppos += rc;
	}
	}

	return (rc);
	}

	static void
	taskq_seq_show_headers(struct seq_file *f)
	{
	seq_printf(f, "%-25s %5s %5s %5s %5s %5s %5s %12s %5s %10s\n",
	"taskq", "act", "nthr", "spwn", "maxt", "pri",
	"mina", "maxa", "cura", "flags");
	}

	/* indices into the lheads array below */
	#define LHEAD_PEND 0
	#define LHEAD_PRIO 1
	#define LHEAD_DELAY 2
	#define LHEAD_WAIT 3
	#define LHEAD_ACTIVE 4
	#define LHEAD_SIZE 5

	/* BEGIN CSTYLED */
	static unsigned int spl_max_show_tasks = 512;
	module_param(spl_max_show_tasks, uint, 0644);
	MODULE_PARM_DESC(spl_max_show_tasks, "Max number of tasks shown in taskq proc");
	/* END CSTYLED */

	static int
	taskq_seq_show_impl(struct seq_file f, void p, boolean_t allflag)
	{
	taskq_t *tq = p;
	taskq_thread_t *tqt = NULL;
	spl_wait_queue_entry_t *wq;
	struct task_struct *tsk;
	taskq_ent_t *tqe;
	char name[100];
	struct list_head lheads[LHEAD_SIZE], lh;
	static char *list_names[LHEAD_SIZE] =
	{"pend", "prio", "delay", "wait", "active" };
	int i, j, have_lheads = 0;
	unsigned long wflags, flags;

	spin_lock_irqsave_nested(&tq->tq_lock, flags, tq->tq_lock_class);
	spin_lock_irqsave(&tq->tq_wait_waitq.lock, wflags);

	/* get the various lists and check whether they're empty */
	lheads[LHEAD_PEND] = &tq->tq_pend_list;
	lheads[LHEAD_PRIO] = &tq->tq_prio_list;
	lheads[LHEAD_DELAY] = &tq->tq_delay_list;
	#ifdef HAVE_WAIT_QUEUE_HEAD_ENTRY
	lheads[LHEAD_WAIT] = &tq->tq_wait_waitq.head;
	#else
	lheads[LHEAD_WAIT] = &tq->tq_wait_waitq.task_list;
	#endif
	lheads[LHEAD_ACTIVE] = &tq->tq_active_list;

	for (i = 0; i < LHEAD_SIZE; ++i) {
	if (list_empty(lheads[i]))
	lheads[i] = NULL;
	else
	++have_lheads;
	}

	/* early return in non-"all" mode if lists are all empty */
	if (!allflag && !have_lheads) {
	spin_unlock_irqrestore(&tq->tq_wait_waitq.lock, wflags);
	spin_unlock_irqrestore(&tq->tq_lock, flags);
	return (0);
	}

	/* unlock the waitq quickly */
	if (!lheads[LHEAD_WAIT])
	spin_unlock_irqrestore(&tq->tq_wait_waitq.lock, wflags);

	/* show the base taskq contents */
	snprintf(name, sizeof (name), "%s/%d", tq->tq_name, tq->tq_instance);
	seq_printf(f, "%-25s ", name);
	seq_printf(f, "%5d %5d %5d %5d %5d %5d %12d %5d %10x\n",
	tq->tq_nactive, tq->tq_nthreads, tq->tq_nspawn,
	tq->tq_maxthreads, tq->tq_pri, tq->tq_minalloc, tq->tq_maxalloc,
	tq->tq_nalloc, tq->tq_flags);

	/* show the active list */
	if (lheads[LHEAD_ACTIVE]) {
	j = 0;
	list_for_each_entry(tqt, &tq->tq_active_list, tqt_active_list) {
	if (j == 0)
	seq_printf(f, "\t%s:",
	list_names[LHEAD_ACTIVE]);
	else if (j == 2) {
	seq_printf(f, "\n\t ");
	j = 0;
	}
	seq_printf(f, " [%d]%pf(%ps)",
	tqt->tqt_thread->pid,
	tqt->tqt_task->tqent_func,
	tqt->tqt_task->tqent_arg);
	++j;
	}
	seq_printf(f, "\n");
	}

	for (i = LHEAD_PEND; i <= LHEAD_WAIT; ++i)
	if (lheads[i]) {
	j = 0;
	list_for_each(lh, lheads[i]) {
	if (spl_max_show_tasks != 0 &&
	j >= spl_max_show_tasks) {
	seq_printf(f, "\n\t(truncated)");
	break;
	}
	/* show the wait waitq list */
	if (i == LHEAD_WAIT) {
	#ifdef HAVE_WAIT_QUEUE_HEAD_ENTRY
	wq = list_entry(lh,
	spl_wait_queue_entry_t, entry);
	#else
	wq = list_entry(lh,
	spl_wait_queue_entry_t, task_list);
	#endif
	if (j == 0)
	seq_printf(f, "\t%s:",
	list_names[i]);
	else if (j % 8 == 0)
	seq_printf(f, "\n\t ");

	tsk = wq->private;
	seq_printf(f, " %d", tsk->pid);
	/* pend, prio and delay lists */
	} else {
	tqe = list_entry(lh, taskq_ent_t,
	tqent_list);
	if (j == 0)
	seq_printf(f, "\t%s:",
	list_names[i]);
	else if (j % 2 == 0)
	seq_printf(f, "\n\t ");

	seq_printf(f, " %pf(%ps)",
	tqe->tqent_func,
	tqe->tqent_arg);
	}
	++j;
	}
	seq_printf(f, "\n");
	}
	if (lheads[LHEAD_WAIT])
	spin_unlock_irqrestore(&tq->tq_wait_waitq.lock, wflags);
	spin_unlock_irqrestore(&tq->tq_lock, flags);

	return (0);
	}

	static int
	taskq_all_seq_show(struct seq_file f, void p)
	{
	return (taskq_seq_show_impl(f, p, B_TRUE));
	}

	static int
	taskq_seq_show(struct seq_file f, void p)
	{
	return (taskq_seq_show_impl(f, p, B_FALSE));
	}

	static void *
	taskq_seq_start(struct seq_file f, loff_t pos)
	{
	struct list_head *p;
	loff_t n = *pos;

	down_read(&tq_list_sem);
	if (!n)
	taskq_seq_show_headers(f);

	p = tq_list.next;
	while (n--) {
	p = p->next;
	if (p == &tq_list)
	return (NULL);
	}

	return (list_entry(p, taskq_t, tq_taskqs));
	}

	static void *
	taskq_seq_next(struct seq_file f, void p, loff_t *pos)
	{
	taskq_t *tq = p;

	++*pos;
	return ((tq->tq_taskqs.next == &tq_list) ?
	NULL : list_entry(tq->tq_taskqs.next, taskq_t, tq_taskqs));
	}

	static void
	slab_seq_show_headers(struct seq_file *f)
	{
	seq_printf(f,
	"--------------------- cache ----------"
	"--------------------------------------------- "
	"----- slab ------ "
	"---- object ----- "
	"--- emergency ---\n");
	seq_printf(f,
	"name "
	" flags size alloc slabsize objsize "
	"total alloc max "
	"total alloc max "
	"dlock alloc max\n");
	}

	static int
	slab_seq_show(struct seq_file f, void p)
	{
	spl_kmem_cache_t *skc = p;

	ASSERT(skc->skc_magic == SKC_MAGIC);

	if (skc->skc_flags & KMC_SLAB) {
	/*
	* This cache is backed by a generic Linux kmem cache which
	* has its own accounting. For these caches we only track
	* the number of active allocated objects that exist within
	* the underlying Linux slabs. For the overall statistics of
	* the underlying Linux cache please refer to /proc/slabinfo.
	*/
	spin_lock(&skc->skc_lock);
	uint64_t objs_allocated =
	percpu_counter_sum(&skc->skc_linux_alloc);
	seq_printf(f, "%-36s ", skc->skc_name);
	seq_printf(f, "0x%05lx %9s %9lu %8s %8u "
	"%5s %5s %5s %5s %5lu %5s %5s %5s %5s\n",
	(long unsigned)skc->skc_flags,
	"-",
	(long unsigned)(skc->skc_obj_size * objs_allocated),
	"-",
	(unsigned)skc->skc_obj_size,
	"-", "-", "-", "-",
	(long unsigned)objs_allocated,
	"-", "-", "-", "-");
	spin_unlock(&skc->skc_lock);
	return (0);
	}

	spin_lock(&skc->skc_lock);
	seq_printf(f, "%-36s ", skc->skc_name);
	seq_printf(f, "0x%05lx %9lu %9lu %8u %8u "
	"%5lu %5lu %5lu %5lu %5lu %5lu %5lu %5lu %5lu\n",
	(long unsigned)skc->skc_flags,
	(long unsigned)(skc->skc_slab_size * skc->skc_slab_total),
	(long unsigned)(skc->skc_obj_size * skc->skc_obj_alloc),
	(unsigned)skc->skc_slab_size,
	(unsigned)skc->skc_obj_size,
	(long unsigned)skc->skc_slab_total,
	(long unsigned)skc->skc_slab_alloc,
	(long unsigned)skc->skc_slab_max,
	(long unsigned)skc->skc_obj_total,
	(long unsigned)skc->skc_obj_alloc,
	(long unsigned)skc->skc_obj_max,
	(long unsigned)skc->skc_obj_deadlock,
	(long unsigned)skc->skc_obj_emergency,
	(long unsigned)skc->skc_obj_emergency_max);
	spin_unlock(&skc->skc_lock);
	return (0);
	}

	static void *
	slab_seq_start(struct seq_file f, loff_t pos)
	{
	struct list_head *p;
	loff_t n = *pos;

	down_read(&spl_kmem_cache_sem);
	if (!n)
	slab_seq_show_headers(f);

	p = spl_kmem_cache_list.next;
	while (n--) {
	p = p->next;
	if (p == &spl_kmem_cache_list)
	return (NULL);
	}

	return (list_entry(p, spl_kmem_cache_t, skc_list));
	}

	static void *
	slab_seq_next(struct seq_file f, void p, loff_t *pos)
	{
	spl_kmem_cache_t *skc = p;

	++*pos;
	return ((skc->skc_list.next == &spl_kmem_cache_list) ?
	NULL : list_entry(skc->skc_list.next, spl_kmem_cache_t, skc_list));
	}

	static void
	slab_seq_stop(struct seq_file f, void v)
	{
	up_read(&spl_kmem_cache_sem);
	}

	static struct seq_operations slab_seq_ops = {
	.show = slab_seq_show,
	.start = slab_seq_start,
	.next = slab_seq_next,
	.stop = slab_seq_stop,
	};

	static int
	proc_slab_open(struct inode inode, struct file filp)
	{
	return (seq_open(filp, &slab_seq_ops));
	}

	static const kstat_proc_op_t proc_slab_operations = {
	#ifdef HAVE_PROC_OPS_STRUCT
	.proc_open = proc_slab_open,
	.proc_read = seq_read,
	.proc_lseek = seq_lseek,
	.proc_release = seq_release,
	#else
	.open = proc_slab_open,
	.read = seq_read,
	.llseek = seq_lseek,
	.release = seq_release,
	#endif
	};

	static void
	taskq_seq_stop(struct seq_file f, void v)
	{
	up_read(&tq_list_sem);
	}

	static struct seq_operations taskq_all_seq_ops = {
	.show = taskq_all_seq_show,
	.start = taskq_seq_start,
	.next = taskq_seq_next,
	.stop = taskq_seq_stop,
	};

	static struct seq_operations taskq_seq_ops = {
	.show = taskq_seq_show,
	.start = taskq_seq_start,
	.next = taskq_seq_next,
	.stop = taskq_seq_stop,
	};

	static int
	proc_taskq_all_open(struct inode inode, struct file filp)
	{
	return (seq_open(filp, &taskq_all_seq_ops));
	}

	static int
	proc_taskq_open(struct inode inode, struct file filp)
	{
	return (seq_open(filp, &taskq_seq_ops));
	}

	static const kstat_proc_op_t proc_taskq_all_operations = {
	#ifdef HAVE_PROC_OPS_STRUCT
	.proc_open = proc_taskq_all_open,
	.proc_read = seq_read,
	.proc_lseek = seq_lseek,
	.proc_release = seq_release,
	#else
	.open = proc_taskq_all_open,
	.read = seq_read,
	.llseek = seq_lseek,
	.release = seq_release,
	#endif
	};

	static const kstat_proc_op_t proc_taskq_operations = {
	#ifdef HAVE_PROC_OPS_STRUCT
	.proc_open = proc_taskq_open,
	.proc_read = seq_read,
	.proc_lseek = seq_lseek,
	.proc_release = seq_release,
	#else
	.open = proc_taskq_open,
	.read = seq_read,
	.llseek = seq_lseek,
	.release = seq_release,
	#endif
	};

	static struct ctl_table spl_kmem_table[] = {
	#ifdef DEBUG_KMEM
	{
	.procname = "kmem_used",
	.data = &kmem_alloc_used,
	#ifdef HAVE_ATOMIC64_T
	.maxlen = sizeof (atomic64_t),
	#else
	.maxlen = sizeof (atomic_t),
	#endif /* HAVE_ATOMIC64_T */
	.mode = 0444,
	.proc_handler = &proc_domemused,
	},
	{
	.procname = "kmem_max",
	.data = &kmem_alloc_max,
	.maxlen = sizeof (unsigned long),
	.extra1 = &table_min,
	.extra2 = &table_max,
	.mode = 0444,
	.proc_handler = &proc_doulongvec_minmax,
	},
	#endif /* DEBUG_KMEM */
	{
	.procname = "slab_kvmem_total",
	.data = (void *)(KMC_KVMEM \| KMC_TOTAL),
	.maxlen = sizeof (unsigned long),
	.extra1 = &table_min,
	.extra2 = &table_max,
	.mode = 0444,
	.proc_handler = &proc_doslab,
	},
	{
	.procname = "slab_kvmem_alloc",
	.data = (void *)(KMC_KVMEM \| KMC_ALLOC),
	.maxlen = sizeof (unsigned long),
	.extra1 = &table_min,
	.extra2 = &table_max,
	.mode = 0444,
	.proc_handler = &proc_doslab,
	},
	{
	.procname = "slab_kvmem_max",
	.data = (void *)(KMC_KVMEM \| KMC_MAX),
	.maxlen = sizeof (unsigned long),
	.extra1 = &table_min,
	.extra2 = &table_max,
	.mode = 0444,
	.proc_handler = &proc_doslab,
	},
	{},
	};

	static struct ctl_table spl_kstat_table[] = {
	{},
	};

	static struct ctl_table spl_table[] = {
	/*
	* NB No .strategy entries have been provided since
	* sysctl(8) prefers to go via /proc for portability.
	*/
	{
	.procname = "gitrev",
	.data = spl_gitrev,
	.maxlen = sizeof (spl_gitrev),
	.mode = 0444,
	.proc_handler = &proc_dostring,
	},
	{
	.procname = "hostid",
	.data = &spl_hostid,
	.maxlen = sizeof (unsigned long),
	.mode = 0644,
	.proc_handler = &proc_dohostid,
	},
	{
	.procname = "kmem",
	.mode = 0555,
	.child = spl_kmem_table,
	},
	{
	.procname = "kstat",
	.mode = 0555,
	.child = spl_kstat_table,
	},
	{},
	};

	static struct ctl_table spl_dir[] = {
	{
	.procname = "spl",
	.mode = 0555,
	.child = spl_table,
	},
	{}
	};

	static struct ctl_table spl_root[] = {
	{
	.procname = "kernel",
	.mode = 0555,
	.child = spl_dir,
	},
	{}
	};

	int
	spl_proc_init(void)
	{
	int rc = 0;

	spl_header = register_sysctl_table(spl_root);
	if (spl_header == NULL)
	return (-EUNATCH);

	proc_spl = proc_mkdir("spl", NULL);
	if (proc_spl == NULL) {
	rc = -EUNATCH;
	goto out;
	}

	proc_spl_taskq_all = proc_create_data("taskq-all", 0444, proc_spl,
	&proc_taskq_all_operations, NULL);
	if (proc_spl_taskq_all == NULL) {
	rc = -EUNATCH;
	goto out;
	}

	proc_spl_taskq = proc_create_data("taskq", 0444, proc_spl,
	&proc_taskq_operations, NULL);
	if (proc_spl_taskq == NULL) {
	rc = -EUNATCH;
	goto out;
	}

	proc_spl_kmem = proc_mkdir("kmem", proc_spl);
	if (proc_spl_kmem == NULL) {
	rc = -EUNATCH;
	goto out;
	}

	proc_spl_kmem_slab = proc_create_data("slab", 0444, proc_spl_kmem,
	&proc_slab_operations, NULL);
	if (proc_spl_kmem_slab == NULL) {
	rc = -EUNATCH;
	goto out;
	}

	proc_spl_kstat = proc_mkdir("kstat", proc_spl);
	if (proc_spl_kstat == NULL) {
	rc = -EUNATCH;
	goto out;
	}
	out:
	if (rc) {
	remove_proc_entry("kstat", proc_spl);
	remove_proc_entry("slab", proc_spl_kmem);
	remove_proc_entry("kmem", proc_spl);
	remove_proc_entry("taskq-all", proc_spl);
	remove_proc_entry("taskq", proc_spl);
	remove_proc_entry("spl", NULL);
	unregister_sysctl_table(spl_header);
	}

	return (rc);
	}

	void
	spl_proc_fini(void)
	{
	remove_proc_entry("kstat", proc_spl);
	remove_proc_entry("slab", proc_spl_kmem);
	remove_proc_entry("kmem", proc_spl);
	remove_proc_entry("taskq-all", proc_spl);
	remove_proc_entry("taskq", proc_spl);
	remove_proc_entry("spl", NULL);

	ASSERT(spl_header != NULL);
	unregister_sysctl_table(spl_header);
	}
	Index: vendor-sys/openzfs/dist/module/os/linux/spl/spl-taskq.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/spl/spl-taskq.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/spl/spl-taskq.c (revision 366774)
	@@ -1,1308 +1,1307 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*
	* Solaris Porting Layer (SPL) Task Queue Implementation.
	*/

	#include <sys/timer.h>
	#include <sys/taskq.h>
	#include <sys/kmem.h>
	#include <sys/tsd.h>
	#include <sys/trace_spl.h>

	int spl_taskq_thread_bind = 0;
	module_param(spl_taskq_thread_bind, int, 0644);
	MODULE_PARM_DESC(spl_taskq_thread_bind, "Bind taskq thread to CPU by default");


	int spl_taskq_thread_dynamic = 1;
	module_param(spl_taskq_thread_dynamic, int, 0644);
	MODULE_PARM_DESC(spl_taskq_thread_dynamic, "Allow dynamic taskq threads");

	int spl_taskq_thread_priority = 1;
	module_param(spl_taskq_thread_priority, int, 0644);
	MODULE_PARM_DESC(spl_taskq_thread_priority,
	"Allow non-default priority for taskq threads");

	int spl_taskq_thread_sequential = 4;
	module_param(spl_taskq_thread_sequential, int, 0644);
	MODULE_PARM_DESC(spl_taskq_thread_sequential,
	"Create new taskq threads after N sequential tasks");

	/* Global system-wide dynamic task queue available for all consumers */
	taskq_t *system_taskq;
	EXPORT_SYMBOL(system_taskq);
	/* Global dynamic task queue for long delay */
	taskq_t *system_delay_taskq;
	EXPORT_SYMBOL(system_delay_taskq);

	/* Private dedicated taskq for creating new taskq threads on demand. */
	static taskq_t *dynamic_taskq;
	static taskq_thread_t taskq_thread_create(taskq_t );

	/* List of all taskqs */
	LIST_HEAD(tq_list);
	struct rw_semaphore tq_list_sem;
	static uint_t taskq_tsd;

	static int
	task_km_flags(uint_t flags)
	{
	if (flags & TQ_NOSLEEP)
	return (KM_NOSLEEP);

	if (flags & TQ_PUSHPAGE)
	return (KM_PUSHPAGE);

	return (KM_SLEEP);
	}

	/*
	* taskq_find_by_name - Find the largest instance number of a named taskq.
	*/
	static int
	taskq_find_by_name(const char *name)
	{
	struct list_head *tql = NULL;
	taskq_t *tq;

	list_for_each_prev(tql, &tq_list) {
	tq = list_entry(tql, taskq_t, tq_taskqs);
	if (strcmp(name, tq->tq_name) == 0)
	return (tq->tq_instance);
	}
	return (-1);
	}

	/*
	* NOTE: Must be called with tq->tq_lock held, returns a list_t which
	* is not attached to the free, work, or pending taskq lists.
	*/
	static taskq_ent_t *
	task_alloc(taskq_t tq, uint_t flags, unsigned long irqflags)
	{
	taskq_ent_t *t;
	int count = 0;

	ASSERT(tq);
	retry:
	/* Acquire taskq_ent_t's from free list if available */
	if (!list_empty(&tq->tq_free_list) && !(flags & TQ_NEW)) {
	t = list_entry(tq->tq_free_list.next, taskq_ent_t, tqent_list);

	ASSERT(!(t->tqent_flags & TQENT_FLAG_PREALLOC));
	ASSERT(!(t->tqent_flags & TQENT_FLAG_CANCEL));
	ASSERT(!timer_pending(&t->tqent_timer));

	list_del_init(&t->tqent_list);
	return (t);
	}

	/* Free list is empty and memory allocations are prohibited */
	if (flags & TQ_NOALLOC)
	return (NULL);

	/* Hit maximum taskq_ent_t pool size */
	if (tq->tq_nalloc >= tq->tq_maxalloc) {
	if (flags & TQ_NOSLEEP)
	return (NULL);

	/*
	* Sleep periodically polling the free list for an available
	* taskq_ent_t. Dispatching with TQ_SLEEP should always succeed
	* but we cannot block forever waiting for an taskq_ent_t to
	* show up in the free list, otherwise a deadlock can happen.
	*
	* Therefore, we need to allocate a new task even if the number
	* of allocated tasks is above tq->tq_maxalloc, but we still
	* end up delaying the task allocation by one second, thereby
	* throttling the task dispatch rate.
	*/
	spin_unlock_irqrestore(&tq->tq_lock, *irqflags);
	schedule_timeout(HZ / 100);
	spin_lock_irqsave_nested(&tq->tq_lock, *irqflags,
	tq->tq_lock_class);
	if (count < 100) {
	count++;
	goto retry;
	}
	}

	spin_unlock_irqrestore(&tq->tq_lock, *irqflags);
	t = kmem_alloc(sizeof (taskq_ent_t), task_km_flags(flags));
	spin_lock_irqsave_nested(&tq->tq_lock, *irqflags, tq->tq_lock_class);

	if (t) {
	taskq_init_ent(t);
	tq->tq_nalloc++;
	}

	return (t);
	}

	/*
	* NOTE: Must be called with tq->tq_lock held, expects the taskq_ent_t
	* to already be removed from the free, work, or pending taskq lists.
	*/
	static void
	task_free(taskq_t tq, taskq_ent_t t)
	{
	ASSERT(tq);
	ASSERT(t);
	ASSERT(list_empty(&t->tqent_list));
	ASSERT(!timer_pending(&t->tqent_timer));

	kmem_free(t, sizeof (taskq_ent_t));
	tq->tq_nalloc--;
	}

	/*
	* NOTE: Must be called with tq->tq_lock held, either destroys the
	* taskq_ent_t if too many exist or moves it to the free list for later use.
	*/
	static void
	task_done(taskq_t tq, taskq_ent_t t)
	{
	ASSERT(tq);
	ASSERT(t);

	/* Wake tasks blocked in taskq_wait_id() */
	wake_up_all(&t->tqent_waitq);

	list_del_init(&t->tqent_list);

	if (tq->tq_nalloc <= tq->tq_minalloc) {
	t->tqent_id = TASKQID_INVALID;
	t->tqent_func = NULL;
	t->tqent_arg = NULL;
	t->tqent_flags = 0;

	list_add_tail(&t->tqent_list, &tq->tq_free_list);
	} else {
	task_free(tq, t);
	}
	}

	/*
	* When a delayed task timer expires remove it from the delay list and
	* add it to the priority list in order for immediate processing.
	*/
	static void
	task_expire_impl(taskq_ent_t *t)
	{
	taskq_ent_t *w;
	taskq_t *tq = t->tqent_taskq;
	struct list_head *l = NULL;
	unsigned long flags;

	spin_lock_irqsave_nested(&tq->tq_lock, flags, tq->tq_lock_class);

	if (t->tqent_flags & TQENT_FLAG_CANCEL) {
	ASSERT(list_empty(&t->tqent_list));
	spin_unlock_irqrestore(&tq->tq_lock, flags);
	return;
	}

	t->tqent_birth = jiffies;
	DTRACE_PROBE1(taskq_ent__birth, taskq_ent_t *, t);

	/*
	* The priority list must be maintained in strict task id order
	* from lowest to highest for lowest_id to be easily calculable.
	*/
	list_del(&t->tqent_list);
	list_for_each_prev(l, &tq->tq_prio_list) {
	w = list_entry(l, taskq_ent_t, tqent_list);
	if (w->tqent_id < t->tqent_id) {
	list_add(&t->tqent_list, l);
	break;
	}
	}
	if (l == &tq->tq_prio_list)
	list_add(&t->tqent_list, &tq->tq_prio_list);

	spin_unlock_irqrestore(&tq->tq_lock, flags);

	wake_up(&tq->tq_work_waitq);
	}

	static void
	task_expire(spl_timer_list_t tl)
	{
	struct timer_list tmr = (struct timer_list )tl;
	taskq_ent_t *t = from_timer(t, tmr, tqent_timer);
	task_expire_impl(t);
	}

	/*
	* Returns the lowest incomplete taskqid_t. The taskqid_t may
	* be queued on the pending list, on the priority list, on the
	* delay list, or on the work list currently being handled, but
	* it is not 100% complete yet.
	*/
	static taskqid_t
	taskq_lowest_id(taskq_t *tq)
	{
	taskqid_t lowest_id = tq->tq_next_id;
	taskq_ent_t *t;
	taskq_thread_t *tqt;

	ASSERT(tq);

	if (!list_empty(&tq->tq_pend_list)) {
	t = list_entry(tq->tq_pend_list.next, taskq_ent_t, tqent_list);
	lowest_id = MIN(lowest_id, t->tqent_id);
	}

	if (!list_empty(&tq->tq_prio_list)) {
	t = list_entry(tq->tq_prio_list.next, taskq_ent_t, tqent_list);
	lowest_id = MIN(lowest_id, t->tqent_id);
	}

	if (!list_empty(&tq->tq_delay_list)) {
	t = list_entry(tq->tq_delay_list.next, taskq_ent_t, tqent_list);
	lowest_id = MIN(lowest_id, t->tqent_id);
	}

	if (!list_empty(&tq->tq_active_list)) {
	tqt = list_entry(tq->tq_active_list.next, taskq_thread_t,
	tqt_active_list);
	ASSERT(tqt->tqt_id != TASKQID_INVALID);
	lowest_id = MIN(lowest_id, tqt->tqt_id);
	}

	return (lowest_id);
	}

	/*
	* Insert a task into a list keeping the list sorted by increasing taskqid.
	*/
	static void
	taskq_insert_in_order(taskq_t tq, taskq_thread_t tqt)
	{
	taskq_thread_t *w;
	struct list_head *l = NULL;

	ASSERT(tq);
	ASSERT(tqt);

	list_for_each_prev(l, &tq->tq_active_list) {
	w = list_entry(l, taskq_thread_t, tqt_active_list);
	if (w->tqt_id < tqt->tqt_id) {
	list_add(&tqt->tqt_active_list, l);
	break;
	}
	}
	if (l == &tq->tq_active_list)
	list_add(&tqt->tqt_active_list, &tq->tq_active_list);
	}

	/*
	* Find and return a task from the given list if it exists. The list
	* must be in lowest to highest task id order.
	*/
	static taskq_ent_t *
	taskq_find_list(taskq_t tq, struct list_head lh, taskqid_t id)
	{
	struct list_head *l = NULL;
	taskq_ent_t *t;

	list_for_each(l, lh) {
	t = list_entry(l, taskq_ent_t, tqent_list);

	if (t->tqent_id == id)
	return (t);

	if (t->tqent_id > id)
	break;
	}

	return (NULL);
	}

	/*
	* Find an already dispatched task given the task id regardless of what
	* state it is in. If a task is still pending it will be returned.
	* If a task is executing, then -EBUSY will be returned instead.
	* If the task has already been run then NULL is returned.
	*/
	static taskq_ent_t *
	taskq_find(taskq_t *tq, taskqid_t id)
	{
	taskq_thread_t *tqt;
	struct list_head *l = NULL;
	taskq_ent_t *t;

	t = taskq_find_list(tq, &tq->tq_delay_list, id);
	if (t)
	return (t);

	t = taskq_find_list(tq, &tq->tq_prio_list, id);
	if (t)
	return (t);

	t = taskq_find_list(tq, &tq->tq_pend_list, id);
	if (t)
	return (t);

	list_for_each(l, &tq->tq_active_list) {
	tqt = list_entry(l, taskq_thread_t, tqt_active_list);
	if (tqt->tqt_id == id) {
	/*
	* Instead of returning tqt_task, we just return a non
	* NULL value to prevent misuse, since tqt_task only
	* has two valid fields.
	*/
	return (ERR_PTR(-EBUSY));
	}
	}

	return (NULL);
	}

	/*
	* Theory for the taskq_wait_id(), taskq_wait_outstanding(), and
	* taskq_wait() functions below.
	*
	* Taskq waiting is accomplished by tracking the lowest outstanding task
	* id and the next available task id. As tasks are dispatched they are
	* added to the tail of the pending, priority, or delay lists. As worker
	* threads become available the tasks are removed from the heads of these
	* lists and linked to the worker threads. This ensures the lists are
	* kept sorted by lowest to highest task id.
	*
	* Therefore the lowest outstanding task id can be quickly determined by
	* checking the head item from all of these lists. This value is stored
	* with the taskq as the lowest id. It only needs to be recalculated when
	* either the task with the current lowest id completes or is canceled.
	*
	* By blocking until the lowest task id exceeds the passed task id the
	* taskq_wait_outstanding() function can be easily implemented. Similarly,
	* by blocking until the lowest task id matches the next task id taskq_wait()
	* can be implemented.
	*
	* Callers should be aware that when there are multiple worked threads it
	* is possible for larger task ids to complete before smaller ones. Also
	* when the taskq contains delay tasks with small task ids callers may
	* block for a considerable length of time waiting for them to expire and
	* execute.
	*/
	static int
	taskq_wait_id_check(taskq_t *tq, taskqid_t id)
	{
	int rc;
	unsigned long flags;

	spin_lock_irqsave_nested(&tq->tq_lock, flags, tq->tq_lock_class);
	rc = (taskq_find(tq, id) == NULL);
	spin_unlock_irqrestore(&tq->tq_lock, flags);

	return (rc);
	}

	/*
	* The taskq_wait_id() function blocks until the passed task id completes.
	* This does not guarantee that all lower task ids have completed.
	*/
	void
	taskq_wait_id(taskq_t *tq, taskqid_t id)
	{
	wait_event(tq->tq_wait_waitq, taskq_wait_id_check(tq, id));
	}
	EXPORT_SYMBOL(taskq_wait_id);

	static int
	taskq_wait_outstanding_check(taskq_t *tq, taskqid_t id)
	{
	int rc;
	unsigned long flags;

	spin_lock_irqsave_nested(&tq->tq_lock, flags, tq->tq_lock_class);
	rc = (id < tq->tq_lowest_id);
	spin_unlock_irqrestore(&tq->tq_lock, flags);

	return (rc);
	}

	/*
	* The taskq_wait_outstanding() function will block until all tasks with a
	* lower taskqid than the passed 'id' have been completed. Note that all
	* task id's are assigned monotonically at dispatch time. Zero may be
	* passed for the id to indicate all tasks dispatch up to this point,
	* but not after, should be waited for.
	*/
	void
	taskq_wait_outstanding(taskq_t *tq, taskqid_t id)
	{
	id = id ? id : tq->tq_next_id - 1;
	wait_event(tq->tq_wait_waitq, taskq_wait_outstanding_check(tq, id));
	}
	EXPORT_SYMBOL(taskq_wait_outstanding);

	static int
	taskq_wait_check(taskq_t *tq)
	{
	int rc;
	unsigned long flags;

	spin_lock_irqsave_nested(&tq->tq_lock, flags, tq->tq_lock_class);
	rc = (tq->tq_lowest_id == tq->tq_next_id);
	spin_unlock_irqrestore(&tq->tq_lock, flags);

	return (rc);
	}

	/*
	* The taskq_wait() function will block until the taskq is empty.
	* This means that if a taskq re-dispatches work to itself taskq_wait()
	* callers will block indefinitely.
	*/
	void
	taskq_wait(taskq_t *tq)
	{
	wait_event(tq->tq_wait_waitq, taskq_wait_check(tq));
	}
	EXPORT_SYMBOL(taskq_wait);

	int
	taskq_member(taskq_t tq, kthread_t t)
	{
	return (tq == (taskq_t *)tsd_get_by_thread(taskq_tsd, t));
	}
	EXPORT_SYMBOL(taskq_member);

	taskq_t *
	taskq_of_curthread(void)
	{
	return (tsd_get(taskq_tsd));
	}
	EXPORT_SYMBOL(taskq_of_curthread);

	/*
	* Cancel an already dispatched task given the task id. Still pending tasks
	* will be immediately canceled, and if the task is active the function will
	* block until it completes. Preallocated tasks which are canceled must be
	* freed by the caller.
	*/
	int
	taskq_cancel_id(taskq_t *tq, taskqid_t id)
	{
	taskq_ent_t *t;
	int rc = ENOENT;
	unsigned long flags;

	ASSERT(tq);

	spin_lock_irqsave_nested(&tq->tq_lock, flags, tq->tq_lock_class);
	t = taskq_find(tq, id);
	if (t && t != ERR_PTR(-EBUSY)) {
	list_del_init(&t->tqent_list);
	t->tqent_flags \|= TQENT_FLAG_CANCEL;

	/*
	* When canceling the lowest outstanding task id we
	* must recalculate the new lowest outstanding id.
	*/
	if (tq->tq_lowest_id == t->tqent_id) {
	tq->tq_lowest_id = taskq_lowest_id(tq);
	ASSERT3S(tq->tq_lowest_id, >, t->tqent_id);
	}

	/*
	* The task_expire() function takes the tq->tq_lock so drop
	* drop the lock before synchronously cancelling the timer.
	*/
	if (timer_pending(&t->tqent_timer)) {
	spin_unlock_irqrestore(&tq->tq_lock, flags);
	del_timer_sync(&t->tqent_timer);
	spin_lock_irqsave_nested(&tq->tq_lock, flags,
	tq->tq_lock_class);
	}

	if (!(t->tqent_flags & TQENT_FLAG_PREALLOC))
	task_done(tq, t);

	rc = 0;
	}
	spin_unlock_irqrestore(&tq->tq_lock, flags);

	if (t == ERR_PTR(-EBUSY)) {
	taskq_wait_id(tq, id);
	rc = EBUSY;
	}

	return (rc);
	}
	EXPORT_SYMBOL(taskq_cancel_id);

	static int taskq_thread_spawn(taskq_t *tq);

	taskqid_t
	taskq_dispatch(taskq_t tq, task_func_t func, void arg, uint_t flags)
	{
	taskq_ent_t *t;
	taskqid_t rc = TASKQID_INVALID;
	unsigned long irqflags;

	ASSERT(tq);
	ASSERT(func);

	spin_lock_irqsave_nested(&tq->tq_lock, irqflags, tq->tq_lock_class);

	/* Taskq being destroyed and all tasks drained */
	if (!(tq->tq_flags & TASKQ_ACTIVE))
	goto out;

	/* Do not queue the task unless there is idle thread for it */
	ASSERT(tq->tq_nactive <= tq->tq_nthreads);
	if ((flags & TQ_NOQUEUE) && (tq->tq_nactive == tq->tq_nthreads)) {
	/* Dynamic taskq may be able to spawn another thread */
	if (!(tq->tq_flags & TASKQ_DYNAMIC) \|\|
	taskq_thread_spawn(tq) == 0)
	goto out;
	}

	if ((t = task_alloc(tq, flags, &irqflags)) == NULL)
	goto out;

	spin_lock(&t->tqent_lock);

	/* Queue to the front of the list to enforce TQ_NOQUEUE semantics */
	if (flags & TQ_NOQUEUE)
	list_add(&t->tqent_list, &tq->tq_prio_list);
	/* Queue to the priority list instead of the pending list */
	else if (flags & TQ_FRONT)
	list_add_tail(&t->tqent_list, &tq->tq_prio_list);
	else
	list_add_tail(&t->tqent_list, &tq->tq_pend_list);

	t->tqent_id = rc = tq->tq_next_id;
	tq->tq_next_id++;
	t->tqent_func = func;
	t->tqent_arg = arg;
	t->tqent_taskq = tq;
	t->tqent_timer.function = NULL;
	t->tqent_timer.expires = 0;

	t->tqent_birth = jiffies;
	DTRACE_PROBE1(taskq_ent__birth, taskq_ent_t *, t);

	ASSERT(!(t->tqent_flags & TQENT_FLAG_PREALLOC));

	spin_unlock(&t->tqent_lock);

	wake_up(&tq->tq_work_waitq);
	out:
	/* Spawn additional taskq threads if required. */
	if (!(flags & TQ_NOQUEUE) && tq->tq_nactive == tq->tq_nthreads)
	(void) taskq_thread_spawn(tq);

	spin_unlock_irqrestore(&tq->tq_lock, irqflags);
	return (rc);
	}
	EXPORT_SYMBOL(taskq_dispatch);

	taskqid_t
	taskq_dispatch_delay(taskq_t tq, task_func_t func, void arg,
	uint_t flags, clock_t expire_time)
	{
	taskqid_t rc = TASKQID_INVALID;
	taskq_ent_t *t;
	unsigned long irqflags;

	ASSERT(tq);
	ASSERT(func);

	spin_lock_irqsave_nested(&tq->tq_lock, irqflags, tq->tq_lock_class);

	/* Taskq being destroyed and all tasks drained */
	if (!(tq->tq_flags & TASKQ_ACTIVE))
	goto out;

	if ((t = task_alloc(tq, flags, &irqflags)) == NULL)
	goto out;

	spin_lock(&t->tqent_lock);

	/* Queue to the delay list for subsequent execution */
	list_add_tail(&t->tqent_list, &tq->tq_delay_list);

	t->tqent_id = rc = tq->tq_next_id;
	tq->tq_next_id++;
	t->tqent_func = func;
	t->tqent_arg = arg;
	t->tqent_taskq = tq;
	t->tqent_timer.function = task_expire;
	t->tqent_timer.expires = (unsigned long)expire_time;
	add_timer(&t->tqent_timer);

	ASSERT(!(t->tqent_flags & TQENT_FLAG_PREALLOC));

	spin_unlock(&t->tqent_lock);
	out:
	/* Spawn additional taskq threads if required. */
	if (tq->tq_nactive == tq->tq_nthreads)
	(void) taskq_thread_spawn(tq);
	spin_unlock_irqrestore(&tq->tq_lock, irqflags);
	return (rc);
	}
	EXPORT_SYMBOL(taskq_dispatch_delay);

	void
	taskq_dispatch_ent(taskq_t tq, task_func_t func, void arg, uint_t flags,
	taskq_ent_t *t)
	{
	unsigned long irqflags;
	ASSERT(tq);
	ASSERT(func);

	spin_lock_irqsave_nested(&tq->tq_lock, irqflags,
	tq->tq_lock_class);

	/* Taskq being destroyed and all tasks drained */
	if (!(tq->tq_flags & TASKQ_ACTIVE)) {
	t->tqent_id = TASKQID_INVALID;
	goto out;
	}

	if ((flags & TQ_NOQUEUE) && (tq->tq_nactive == tq->tq_nthreads)) {
	/* Dynamic taskq may be able to spawn another thread */
	if (!(tq->tq_flags & TASKQ_DYNAMIC) \|\|
	taskq_thread_spawn(tq) == 0)
	goto out2;
	flags \|= TQ_FRONT;
	}

	spin_lock(&t->tqent_lock);

	/*
	* Make sure the entry is not on some other taskq; it is important to
	* ASSERT() under lock
	*/
	ASSERT(taskq_empty_ent(t));

	/*
	* Mark it as a prealloc'd task. This is important
	* to ensure that we don't free it later.
	*/
	t->tqent_flags \|= TQENT_FLAG_PREALLOC;

	/* Queue to the priority list instead of the pending list */
	if (flags & TQ_FRONT)
	list_add_tail(&t->tqent_list, &tq->tq_prio_list);
	else
	list_add_tail(&t->tqent_list, &tq->tq_pend_list);

	t->tqent_id = tq->tq_next_id;
	tq->tq_next_id++;
	t->tqent_func = func;
	t->tqent_arg = arg;
	t->tqent_taskq = tq;

	t->tqent_birth = jiffies;
	DTRACE_PROBE1(taskq_ent__birth, taskq_ent_t *, t);

	spin_unlock(&t->tqent_lock);

	wake_up(&tq->tq_work_waitq);
	out:
	/* Spawn additional taskq threads if required. */
	if (tq->tq_nactive == tq->tq_nthreads)
	(void) taskq_thread_spawn(tq);
	out2:
	spin_unlock_irqrestore(&tq->tq_lock, irqflags);
	}
	EXPORT_SYMBOL(taskq_dispatch_ent);

	int
	taskq_empty_ent(taskq_ent_t *t)
	{
	return (list_empty(&t->tqent_list));
	}
	EXPORT_SYMBOL(taskq_empty_ent);

	void
	taskq_init_ent(taskq_ent_t *t)
	{
	spin_lock_init(&t->tqent_lock);
	init_waitqueue_head(&t->tqent_waitq);
	timer_setup(&t->tqent_timer, NULL, 0);
	INIT_LIST_HEAD(&t->tqent_list);
	t->tqent_id = 0;
	t->tqent_func = NULL;
	t->tqent_arg = NULL;
	t->tqent_flags = 0;
	t->tqent_taskq = NULL;
	}
	EXPORT_SYMBOL(taskq_init_ent);

	/*
	* Return the next pending task, preference is given to tasks on the
	* priority list which were dispatched with TQ_FRONT.
	*/
	static taskq_ent_t *
	taskq_next_ent(taskq_t *tq)
	{
	struct list_head *list;

	if (!list_empty(&tq->tq_prio_list))
	list = &tq->tq_prio_list;
	else if (!list_empty(&tq->tq_pend_list))
	list = &tq->tq_pend_list;
	else
	return (NULL);

	return (list_entry(list->next, taskq_ent_t, tqent_list));
	}

	/*
	* Spawns a new thread for the specified taskq.
	*/
	static void
	taskq_thread_spawn_task(void *arg)
	{
	taskq_t tq = (taskq_t )arg;
	unsigned long flags;

	if (taskq_thread_create(tq) == NULL) {
	/* restore spawning count if failed */
	spin_lock_irqsave_nested(&tq->tq_lock, flags,
	tq->tq_lock_class);
	tq->tq_nspawn--;
	spin_unlock_irqrestore(&tq->tq_lock, flags);
	}
	}

	/*
	* Spawn addition threads for dynamic taskqs (TASKQ_DYNAMIC) the current
	* number of threads is insufficient to handle the pending tasks. These
	* new threads must be created by the dedicated dynamic_taskq to avoid
	* deadlocks between thread creation and memory reclaim. The system_taskq
	* which is also a dynamic taskq cannot be safely used for this.
	*/
	static int
	taskq_thread_spawn(taskq_t *tq)
	{
	int spawning = 0;

	if (!(tq->tq_flags & TASKQ_DYNAMIC))
	return (0);

	if ((tq->tq_nthreads + tq->tq_nspawn < tq->tq_maxthreads) &&
	(tq->tq_flags & TASKQ_ACTIVE)) {
	spawning = (++tq->tq_nspawn);
	taskq_dispatch(dynamic_taskq, taskq_thread_spawn_task,
	tq, TQ_NOSLEEP);
	}

	return (spawning);
	}

	/*
	* Threads in a dynamic taskq should only exit once it has been completely
	* drained and no other threads are actively servicing tasks. This prevents
	* threads from being created and destroyed more than is required.
	*
	* The first thread is the thread list is treated as the primary thread.
	* There is nothing special about the primary thread but in order to avoid
	* all the taskq pids from changing we opt to make it long running.
	*/
	static int
	taskq_thread_should_stop(taskq_t tq, taskq_thread_t tqt)
	{
	if (!(tq->tq_flags & TASKQ_DYNAMIC))
	return (0);

	if (list_first_entry(&(tq->tq_thread_list), taskq_thread_t,
	tqt_thread_list) == tqt)
	return (0);

	return
	((tq->tq_nspawn == 0) && /* No threads are being spawned */
	(tq->tq_nactive == 0) && /* No threads are handling tasks */
	(tq->tq_nthreads > 1) && /* More than 1 thread is running */
	(!taskq_next_ent(tq)) && /* There are no pending tasks */
	(spl_taskq_thread_dynamic)); /* Dynamic taskqs are allowed */
	}

	static int
	taskq_thread(void *args)
	{
	DECLARE_WAITQUEUE(wait, current);
	sigset_t blocked;
	taskq_thread_t *tqt = args;
	taskq_t *tq;
	taskq_ent_t *t;
	int seq_tasks = 0;
	unsigned long flags;
	taskq_ent_t dup_task = {};

	ASSERT(tqt);
	ASSERT(tqt->tqt_tq);
	tq = tqt->tqt_tq;
	current->flags \|= PF_NOFREEZE;

	(void) spl_fstrans_mark();

	sigfillset(&blocked);
	sigprocmask(SIG_BLOCK, &blocked, NULL);
	flush_signals(current);

	tsd_set(taskq_tsd, tq);
	spin_lock_irqsave_nested(&tq->tq_lock, flags, tq->tq_lock_class);
	/*
	* If we are dynamically spawned, decrease spawning count. Note that
	* we could be created during taskq_create, in which case we shouldn't
	* do the decrement. But it's fine because taskq_create will reset
	* tq_nspawn later.
	*/
	if (tq->tq_flags & TASKQ_DYNAMIC)
	tq->tq_nspawn--;

	/* Immediately exit if more threads than allowed were created. */
	if (tq->tq_nthreads >= tq->tq_maxthreads)
	goto error;

	tq->tq_nthreads++;
	list_add_tail(&tqt->tqt_thread_list, &tq->tq_thread_list);
	wake_up(&tq->tq_wait_waitq);
	set_current_state(TASK_INTERRUPTIBLE);

	while (!kthread_should_stop()) {

	if (list_empty(&tq->tq_pend_list) &&
	list_empty(&tq->tq_prio_list)) {

	if (taskq_thread_should_stop(tq, tqt)) {
	wake_up_all(&tq->tq_wait_waitq);
	break;
	}

	add_wait_queue_exclusive(&tq->tq_work_waitq, &wait);
	spin_unlock_irqrestore(&tq->tq_lock, flags);

	schedule();
	seq_tasks = 0;

	spin_lock_irqsave_nested(&tq->tq_lock, flags,
	tq->tq_lock_class);
	remove_wait_queue(&tq->tq_work_waitq, &wait);
	} else {
	__set_current_state(TASK_RUNNING);
	}

	if ((t = taskq_next_ent(tq)) != NULL) {
	list_del_init(&t->tqent_list);

	/*
	* A TQENT_FLAG_PREALLOC task may be reused or freed
	* during the task function call. Store tqent_id and
	* tqent_flags here.
	*
	* Also use an on stack taskq_ent_t for tqt_task
	* assignment in this case; we want to make sure
	* to duplicate all fields, so the values are
	* correct when it's accessed via DTRACE_PROBE*.
	*/
	tqt->tqt_id = t->tqent_id;
	tqt->tqt_flags = t->tqent_flags;

	if (t->tqent_flags & TQENT_FLAG_PREALLOC) {
	dup_task = *t;
	t = &dup_task;
	}
	tqt->tqt_task = t;

	taskq_insert_in_order(tq, tqt);
	tq->tq_nactive++;
	spin_unlock_irqrestore(&tq->tq_lock, flags);

	DTRACE_PROBE1(taskq_ent__start, taskq_ent_t *, t);

	/* Perform the requested task */
	t->tqent_func(t->tqent_arg);

	DTRACE_PROBE1(taskq_ent__finish, taskq_ent_t *, t);

	spin_lock_irqsave_nested(&tq->tq_lock, flags,
	tq->tq_lock_class);
	tq->tq_nactive--;
	list_del_init(&tqt->tqt_active_list);
	tqt->tqt_task = NULL;

	/* For prealloc'd tasks, we don't free anything. */
	if (!(tqt->tqt_flags & TQENT_FLAG_PREALLOC))
	task_done(tq, t);

	/*
	* When the current lowest outstanding taskqid is
	* done calculate the new lowest outstanding id
	*/
	if (tq->tq_lowest_id == tqt->tqt_id) {
	tq->tq_lowest_id = taskq_lowest_id(tq);
	ASSERT3S(tq->tq_lowest_id, >, tqt->tqt_id);
	}

	/* Spawn additional taskq threads if required. */
	if ((++seq_tasks) > spl_taskq_thread_sequential &&
	taskq_thread_spawn(tq))
	seq_tasks = 0;

	tqt->tqt_id = TASKQID_INVALID;
	tqt->tqt_flags = 0;
	wake_up_all(&tq->tq_wait_waitq);
	} else {
	if (taskq_thread_should_stop(tq, tqt))
	break;
	}

	set_current_state(TASK_INTERRUPTIBLE);

	}

	__set_current_state(TASK_RUNNING);
	tq->tq_nthreads--;
	list_del_init(&tqt->tqt_thread_list);
	error:
	kmem_free(tqt, sizeof (taskq_thread_t));
	spin_unlock_irqrestore(&tq->tq_lock, flags);

	tsd_set(taskq_tsd, NULL);

	return (0);
	}

	static taskq_thread_t *
	taskq_thread_create(taskq_t *tq)
	{
	static int last_used_cpu = 0;
	taskq_thread_t *tqt;

	tqt = kmem_alloc(sizeof (*tqt), KM_PUSHPAGE);
	INIT_LIST_HEAD(&tqt->tqt_thread_list);
	INIT_LIST_HEAD(&tqt->tqt_active_list);
	tqt->tqt_tq = tq;
	tqt->tqt_id = TASKQID_INVALID;

	tqt->tqt_thread = spl_kthread_create(taskq_thread, tqt,
	"%s", tq->tq_name);
	if (tqt->tqt_thread == NULL) {
	kmem_free(tqt, sizeof (taskq_thread_t));
	return (NULL);
	}

	if (spl_taskq_thread_bind) {
	last_used_cpu = (last_used_cpu + 1) % num_online_cpus();
	kthread_bind(tqt->tqt_thread, last_used_cpu);
	}

	if (spl_taskq_thread_priority)
	set_user_nice(tqt->tqt_thread, PRIO_TO_NICE(tq->tq_pri));

	wake_up_process(tqt->tqt_thread);

	return (tqt);
	}

	taskq_t *
	taskq_create(const char *name, int nthreads, pri_t pri,
	int minalloc, int maxalloc, uint_t flags)
	{
	taskq_t *tq;
	taskq_thread_t *tqt;
	int count = 0, rc = 0, i;
	unsigned long irqflags;

	ASSERT(name != NULL);
	ASSERT(minalloc >= 0);
	ASSERT(maxalloc <= INT_MAX);
	ASSERT(!(flags & (TASKQ_CPR_SAFE))); /* Unsupported */

	/* Scale the number of threads using nthreads as a percentage */
	if (flags & TASKQ_THREADS_CPU_PCT) {
	ASSERT(nthreads <= 100);
	ASSERT(nthreads >= 0);
	nthreads = MIN(nthreads, 100);
	nthreads = MAX(nthreads, 0);
	nthreads = MAX((num_online_cpus() * nthreads) / 100, 1);
	}

	tq = kmem_alloc(sizeof (*tq), KM_PUSHPAGE);
	if (tq == NULL)
	return (NULL);

	spin_lock_init(&tq->tq_lock);
	INIT_LIST_HEAD(&tq->tq_thread_list);
	INIT_LIST_HEAD(&tq->tq_active_list);
	tq->tq_name = kmem_strdup(name);
	tq->tq_nactive = 0;
	tq->tq_nthreads = 0;
	tq->tq_nspawn = 0;
	tq->tq_maxthreads = nthreads;
	tq->tq_pri = pri;
	tq->tq_minalloc = minalloc;
	tq->tq_maxalloc = maxalloc;
	tq->tq_nalloc = 0;
	tq->tq_flags = (flags \| TASKQ_ACTIVE);
	tq->tq_next_id = TASKQID_INITIAL;
	tq->tq_lowest_id = TASKQID_INITIAL;
	INIT_LIST_HEAD(&tq->tq_free_list);
	INIT_LIST_HEAD(&tq->tq_pend_list);
	INIT_LIST_HEAD(&tq->tq_prio_list);
	INIT_LIST_HEAD(&tq->tq_delay_list);
	init_waitqueue_head(&tq->tq_work_waitq);
	init_waitqueue_head(&tq->tq_wait_waitq);
	tq->tq_lock_class = TQ_LOCK_GENERAL;
	INIT_LIST_HEAD(&tq->tq_taskqs);

	if (flags & TASKQ_PREPOPULATE) {
	spin_lock_irqsave_nested(&tq->tq_lock, irqflags,
	tq->tq_lock_class);

	for (i = 0; i < minalloc; i++)
	task_done(tq, task_alloc(tq, TQ_PUSHPAGE \| TQ_NEW,
	&irqflags));

	spin_unlock_irqrestore(&tq->tq_lock, irqflags);
	}

	if ((flags & TASKQ_DYNAMIC) && spl_taskq_thread_dynamic)
	nthreads = 1;

	for (i = 0; i < nthreads; i++) {
	tqt = taskq_thread_create(tq);
	if (tqt == NULL)
	rc = 1;
	else
	count++;
	}

	/* Wait for all threads to be started before potential destroy */
	wait_event(tq->tq_wait_waitq, tq->tq_nthreads == count);
	/*
	* taskq_thread might have touched nspawn, but we don't want them to
	* because they're not dynamically spawned. So we reset it to 0
	*/
	tq->tq_nspawn = 0;

	if (rc) {
	taskq_destroy(tq);
	tq = NULL;
	} else {
	down_write(&tq_list_sem);
	tq->tq_instance = taskq_find_by_name(name) + 1;
	list_add_tail(&tq->tq_taskqs, &tq_list);
	up_write(&tq_list_sem);
	}

	return (tq);
	}
	EXPORT_SYMBOL(taskq_create);

	void
	taskq_destroy(taskq_t *tq)
	{
	struct task_struct *thread;
	taskq_thread_t *tqt;
	taskq_ent_t *t;
	unsigned long flags;

	ASSERT(tq);
	spin_lock_irqsave_nested(&tq->tq_lock, flags, tq->tq_lock_class);
	tq->tq_flags &= ~TASKQ_ACTIVE;
	spin_unlock_irqrestore(&tq->tq_lock, flags);

	/*
	* When TASKQ_ACTIVE is clear new tasks may not be added nor may
	* new worker threads be spawned for dynamic taskq.
	*/
	if (dynamic_taskq != NULL)
	taskq_wait_outstanding(dynamic_taskq, 0);

	taskq_wait(tq);

	/* remove taskq from global list used by the kstats */
	down_write(&tq_list_sem);
	list_del(&tq->tq_taskqs);
	up_write(&tq_list_sem);

	spin_lock_irqsave_nested(&tq->tq_lock, flags, tq->tq_lock_class);
	/* wait for spawning threads to insert themselves to the list */
	while (tq->tq_nspawn) {
	spin_unlock_irqrestore(&tq->tq_lock, flags);
	schedule_timeout_interruptible(1);
	spin_lock_irqsave_nested(&tq->tq_lock, flags,
	tq->tq_lock_class);
	}

	/*
	* Signal each thread to exit and block until it does. Each thread
	* is responsible for removing itself from the list and freeing its
	* taskq_thread_t. This allows for idle threads to opt to remove
	* themselves from the taskq. They can be recreated as needed.
	*/
	while (!list_empty(&tq->tq_thread_list)) {
	tqt = list_entry(tq->tq_thread_list.next,
	taskq_thread_t, tqt_thread_list);
	thread = tqt->tqt_thread;
	spin_unlock_irqrestore(&tq->tq_lock, flags);

	kthread_stop(thread);

	spin_lock_irqsave_nested(&tq->tq_lock, flags,
	tq->tq_lock_class);
	}

	while (!list_empty(&tq->tq_free_list)) {
	t = list_entry(tq->tq_free_list.next, taskq_ent_t, tqent_list);

	ASSERT(!(t->tqent_flags & TQENT_FLAG_PREALLOC));

	list_del_init(&t->tqent_list);
	task_free(tq, t);
	}

	ASSERT0(tq->tq_nthreads);
	ASSERT0(tq->tq_nalloc);
	ASSERT0(tq->tq_nspawn);
	ASSERT(list_empty(&tq->tq_thread_list));
	ASSERT(list_empty(&tq->tq_active_list));
	ASSERT(list_empty(&tq->tq_free_list));
	ASSERT(list_empty(&tq->tq_pend_list));
	ASSERT(list_empty(&tq->tq_prio_list));
	ASSERT(list_empty(&tq->tq_delay_list));

	spin_unlock_irqrestore(&tq->tq_lock, flags);

	kmem_strfree(tq->tq_name);
	kmem_free(tq, sizeof (taskq_t));
	}
	EXPORT_SYMBOL(taskq_destroy);


	static unsigned int spl_taskq_kick = 0;

	/*
	* 2.6.36 API Change
	* module_param_cb is introduced to take kernel_param_ops and
	* module_param_call is marked as obsolete. Also set and get operations
	* were changed to take a 'const struct kernel_param *'.
	*/
	static int
	#ifdef module_param_cb
	param_set_taskq_kick(const char val, const struct kernel_param kp)
	#else
	param_set_taskq_kick(const char val, struct kernel_param kp)
	#endif
	{
	int ret;
	taskq_t *tq = NULL;
	taskq_ent_t *t;
	unsigned long flags;

	ret = param_set_uint(val, kp);
	if (ret < 0 \|\| !spl_taskq_kick)
	return (ret);
	/* reset value */
	spl_taskq_kick = 0;

	down_read(&tq_list_sem);
	list_for_each_entry(tq, &tq_list, tq_taskqs) {
	spin_lock_irqsave_nested(&tq->tq_lock, flags,
	tq->tq_lock_class);
	/* Check if the first pending is older than 5 seconds */
	t = taskq_next_ent(tq);
	if (t && time_after(jiffies, t->tqent_birth + 5*HZ)) {
	(void) taskq_thread_spawn(tq);
	printk(KERN_INFO "spl: Kicked taskq %s/%d\n",
	tq->tq_name, tq->tq_instance);
	}
	spin_unlock_irqrestore(&tq->tq_lock, flags);
	}
	up_read(&tq_list_sem);
	return (ret);
	}

	#ifdef module_param_cb
	static const struct kernel_param_ops param_ops_taskq_kick = {
	.set = param_set_taskq_kick,
	.get = param_get_uint,
	};
	module_param_cb(spl_taskq_kick, &param_ops_taskq_kick, &spl_taskq_kick, 0644);
	#else
	module_param_call(spl_taskq_kick, param_set_taskq_kick, param_get_uint,
	&spl_taskq_kick, 0644);
	#endif
	MODULE_PARM_DESC(spl_taskq_kick,
	"Write nonzero to kick stuck taskqs to spawn more threads");

	int
	spl_taskq_init(void)
	{
	init_rwsem(&tq_list_sem);
	tsd_create(&taskq_tsd, NULL);

	system_taskq = taskq_create("spl_system_taskq", MAX(boot_ncpus, 64),
	maxclsyspri, boot_ncpus, INT_MAX, TASKQ_PREPOPULATE\|TASKQ_DYNAMIC);
	if (system_taskq == NULL)
	return (1);

	system_delay_taskq = taskq_create("spl_delay_taskq", MAX(boot_ncpus, 4),
	maxclsyspri, boot_ncpus, INT_MAX, TASKQ_PREPOPULATE\|TASKQ_DYNAMIC);
	if (system_delay_taskq == NULL) {
	taskq_destroy(system_taskq);
	return (1);
	}

	dynamic_taskq = taskq_create("spl_dynamic_taskq", 1,
	maxclsyspri, boot_ncpus, INT_MAX, TASKQ_PREPOPULATE);
	if (dynamic_taskq == NULL) {
	taskq_destroy(system_taskq);
	taskq_destroy(system_delay_taskq);
	return (1);
	}

	/*
	* This is used to annotate tq_lock, so
	* taskq_dispatch -> taskq_thread_spawn -> taskq_dispatch
	* does not trigger a lockdep warning re: possible recursive locking
	*/
	dynamic_taskq->tq_lock_class = TQ_LOCK_DYNAMIC;

	return (0);
	}

	void
	spl_taskq_fini(void)
	{
	taskq_destroy(dynamic_taskq);
	dynamic_taskq = NULL;

	taskq_destroy(system_delay_taskq);
	system_delay_taskq = NULL;

	taskq_destroy(system_taskq);
	system_taskq = NULL;

	tsd_destroy(&taskq_tsd);
	}
	Index: vendor-sys/openzfs/dist/module/os/linux/spl/spl-thread.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/spl/spl-thread.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/spl/spl-thread.c (revision 366774)
	@@ -1,161 +1,160 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*
	* Solaris Porting Layer (SPL) Thread Implementation.
	*/

	#include <sys/thread.h>
	#include <sys/kmem.h>
	#include <sys/tsd.h>

	/*
	* Thread interfaces
	*/
	typedef struct thread_priv_s {
	unsigned long tp_magic; /* Magic */
	int tp_name_size; /* Name size */
	char tp_name; / Name (without _thread suffix) */
	void (tp_func)(void ); /* Registered function */
	void tp_args; / Args to be passed to function */
	size_t tp_len; /* Len to be passed to function */
	int tp_state; /* State to start thread at */
	pri_t tp_pri; /* Priority to start threat at */
	} thread_priv_t;

	static int
	thread_generic_wrapper(void *arg)
	{
	thread_priv_t tp = (thread_priv_t )arg;
	void (func)(void );
	void *args;

	ASSERT(tp->tp_magic == TP_MAGIC);
	func = tp->tp_func;
	args = tp->tp_args;
	set_current_state(tp->tp_state);
	set_user_nice((kthread_t *)current, PRIO_TO_NICE(tp->tp_pri));
	kmem_free(tp->tp_name, tp->tp_name_size);
	kmem_free(tp, sizeof (thread_priv_t));

	if (func)
	func(args);

	return (0);
	}

	void
	__thread_exit(void)
	{
	tsd_exit();
	complete_and_exit(NULL, 0);
	/* Unreachable */
	}
	EXPORT_SYMBOL(__thread_exit);

	/*
	* thread_create() may block forever if it cannot create a thread or
	* allocate memory. This is preferable to returning a NULL which Solaris
	* style callers likely never check for... since it can't fail.
	*/
	kthread_t *
	__thread_create(caddr_t stk, size_t stksize, thread_func_t func,
	const char name, void args, size_t len, proc_t *pp, int state, pri_t pri)
	{
	thread_priv_t *tp;
	struct task_struct *tsk;
	char *p;

	/* Option pp is simply ignored */
	/* Variable stack size unsupported */
	ASSERT(stk == NULL);

	tp = kmem_alloc(sizeof (thread_priv_t), KM_PUSHPAGE);
	if (tp == NULL)
	return (NULL);

	tp->tp_magic = TP_MAGIC;
	tp->tp_name_size = strlen(name) + 1;

	tp->tp_name = kmem_alloc(tp->tp_name_size, KM_PUSHPAGE);
	if (tp->tp_name == NULL) {
	kmem_free(tp, sizeof (thread_priv_t));
	return (NULL);
	}

	strncpy(tp->tp_name, name, tp->tp_name_size);

	/*
	* Strip trailing "_thread" from passed name which will be the func
	* name since the exposed API has no parameter for passing a name.
	*/
	p = strstr(tp->tp_name, "_thread");
	if (p)
	p[0] = '\0';

	tp->tp_func = func;
	tp->tp_args = args;
	tp->tp_len = len;
	tp->tp_state = state;
	tp->tp_pri = pri;

	tsk = spl_kthread_create(thread_generic_wrapper, (void *)tp,
	"%s", tp->tp_name);
	if (IS_ERR(tsk))
	return (NULL);

	wake_up_process(tsk);
	return ((kthread_t *)tsk);
	}
	EXPORT_SYMBOL(__thread_create);

	/*
	* spl_kthread_create - Wrapper providing pre-3.13 semantics for
	* kthread_create() in which it is not killable and less likely
	* to return -ENOMEM.
	*/
	struct task_struct *
	spl_kthread_create(int (func)(void ), void *data, const char namefmt[], ...)
	{
	struct task_struct *tsk;
	va_list args;
	char name[TASK_COMM_LEN];

	va_start(args, namefmt);
	vsnprintf(name, sizeof (name), namefmt, args);
	va_end(args);
	do {
	tsk = kthread_create(func, data, "%s", name);
	if (IS_ERR(tsk)) {
	if (signal_pending(current)) {
	clear_thread_flag(TIF_SIGPENDING);
	continue;
	}
	if (PTR_ERR(tsk) == -ENOMEM)
	continue;
	return (NULL);
	} else {
	return (tsk);
	}
	} while (1);
	}
	EXPORT_SYMBOL(spl_kthread_create);
	Index: vendor-sys/openzfs/dist/module/os/linux/spl/spl-tsd.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/spl/spl-tsd.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/spl/spl-tsd.c (revision 366774)
	@@ -1,720 +1,719 @@
	/*
	* Copyright (C) 2010 Lawrence Livermore National Security, LLC.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*
	*
	* Solaris Porting Layer (SPL) Thread Specific Data Implementation.
	*
	* Thread specific data has implemented using a hash table, this avoids
	* the need to add a member to the task structure and allows maximum
	* portability between kernels. This implementation has been optimized
	* to keep the tsd_set() and tsd_get() times as small as possible.
	*
	* The majority of the entries in the hash table are for specific tsd
	* entries. These entries are hashed by the product of their key and
	* pid because by design the key and pid are guaranteed to be unique.
	* Their product also has the desirable properly that it will be uniformly
	* distributed over the hash bins providing neither the pid nor key is zero.
	* Under linux the zero pid is always the init process and thus won't be
	* used, and this implementation is careful to never to assign a zero key.
	* By default the hash table is sized to 512 bins which is expected to
	* be sufficient for light to moderate usage of thread specific data.
	*
	* The hash table contains two additional type of entries. They first
	* type is entry is called a 'key' entry and it is added to the hash during
	* tsd_create(). It is used to store the address of the destructor function
	* and it is used as an anchor point. All tsd entries which use the same
	* key will be linked to this entry. This is used during tsd_destroy() to
	* quickly call the destructor function for all tsd associated with the key.
	* The 'key' entry may be looked up with tsd_hash_search() by passing the
	* key you wish to lookup and DTOR_PID constant as the pid.
	*
	* The second type of entry is called a 'pid' entry and it is added to the
	* hash the first time a process set a key. The 'pid' entry is also used
	* as an anchor and all tsd for the process will be linked to it. This
	* list is using during tsd_exit() to ensure all registered destructors
	* are run for the process. The 'pid' entry may be looked up with
	* tsd_hash_search() by passing the PID_KEY constant as the key, and
	* the process pid. Note that tsd_exit() is called by thread_exit()
	* so if your using the Solaris thread API you should not need to call
	* tsd_exit() directly.
	*
	*/

	#include <sys/kmem.h>
	#include <sys/thread.h>
	#include <sys/tsd.h>
	#include <linux/hash.h>

	typedef struct tsd_hash_bin {
	spinlock_t hb_lock;
	struct hlist_head hb_head;
	} tsd_hash_bin_t;

	typedef struct tsd_hash_table {
	spinlock_t ht_lock;
	uint_t ht_bits;
	uint_t ht_key;
	tsd_hash_bin_t *ht_bins;
	} tsd_hash_table_t;

	typedef struct tsd_hash_entry {
	uint_t he_key;
	pid_t he_pid;
	dtor_func_t he_dtor;
	void *he_value;
	struct hlist_node he_list;
	struct list_head he_key_list;
	struct list_head he_pid_list;
	} tsd_hash_entry_t;

	static tsd_hash_table_t *tsd_hash_table = NULL;


	/*
	* tsd_hash_search - searches hash table for tsd_hash_entry
	* @table: hash table
	* @key: search key
	* @pid: search pid
	*/
	static tsd_hash_entry_t *
	tsd_hash_search(tsd_hash_table_t *table, uint_t key, pid_t pid)
	{
	struct hlist_node *node = NULL;
	tsd_hash_entry_t *entry;
	tsd_hash_bin_t *bin;
	ulong_t hash;

	hash = hash_long((ulong_t)key * (ulong_t)pid, table->ht_bits);
	bin = &table->ht_bins[hash];
	spin_lock(&bin->hb_lock);
	hlist_for_each(node, &bin->hb_head) {
	entry = list_entry(node, tsd_hash_entry_t, he_list);
	if ((entry->he_key == key) && (entry->he_pid == pid)) {
	spin_unlock(&bin->hb_lock);
	return (entry);
	}
	}

	spin_unlock(&bin->hb_lock);
	return (NULL);
	}

	/*
	* tsd_hash_dtor - call the destructor and free all entries on the list
	* @work: list of hash entries
	*
	* For a list of entries which have all already been removed from the
	* hash call their registered destructor then free the associated memory.
	*/
	static void
	tsd_hash_dtor(struct hlist_head *work)
	{
	tsd_hash_entry_t *entry;

	while (!hlist_empty(work)) {
	entry = hlist_entry(work->first, tsd_hash_entry_t, he_list);
	hlist_del(&entry->he_list);

	if (entry->he_dtor && entry->he_pid != DTOR_PID)
	entry->he_dtor(entry->he_value);

	kmem_free(entry, sizeof (tsd_hash_entry_t));
	}
	}

	/*
	* tsd_hash_add - adds an entry to hash table
	* @table: hash table
	* @key: search key
	* @pid: search pid
	*
	* The caller is responsible for ensuring the unique key/pid do not
	* already exist in the hash table. This possible because all entries
	* are thread specific thus a concurrent thread will never attempt to
	* add this key/pid. Because multiple bins must be checked to add
	* links to the dtor and pid entries the entire table is locked.
	*/
	static int
	tsd_hash_add(tsd_hash_table_t table, uint_t key, pid_t pid, void value)
	{
	tsd_hash_entry_t entry, dtor_entry, *pid_entry;
	tsd_hash_bin_t *bin;
	ulong_t hash;
	int rc = 0;

	ASSERT3P(tsd_hash_search(table, key, pid), ==, NULL);

	/* New entry allocate structure, set value, and add to hash */
	entry = kmem_alloc(sizeof (tsd_hash_entry_t), KM_PUSHPAGE);
	if (entry == NULL)
	return (ENOMEM);

	entry->he_key = key;
	entry->he_pid = pid;
	entry->he_value = value;
	INIT_HLIST_NODE(&entry->he_list);
	INIT_LIST_HEAD(&entry->he_key_list);
	INIT_LIST_HEAD(&entry->he_pid_list);

	spin_lock(&table->ht_lock);

	/* Destructor entry must exist for all valid keys */
	dtor_entry = tsd_hash_search(table, entry->he_key, DTOR_PID);
	ASSERT3P(dtor_entry, !=, NULL);
	entry->he_dtor = dtor_entry->he_dtor;

	/* Process entry must exist for all valid processes */
	pid_entry = tsd_hash_search(table, PID_KEY, entry->he_pid);
	ASSERT3P(pid_entry, !=, NULL);

	hash = hash_long((ulong_t)key * (ulong_t)pid, table->ht_bits);
	bin = &table->ht_bins[hash];
	spin_lock(&bin->hb_lock);

	/* Add to the hash, key, and pid lists */
	hlist_add_head(&entry->he_list, &bin->hb_head);
	list_add(&entry->he_key_list, &dtor_entry->he_key_list);
	list_add(&entry->he_pid_list, &pid_entry->he_pid_list);

	spin_unlock(&bin->hb_lock);
	spin_unlock(&table->ht_lock);

	return (rc);
	}

	/*
	* tsd_hash_add_key - adds a destructor entry to the hash table
	* @table: hash table
	* @keyp: search key
	* @dtor: key destructor
	*
	* For every unique key there is a single entry in the hash which is used
	* as anchor. All other thread specific entries for this key are linked
	* to this anchor via the 'he_key_list' list head. On return they keyp
	* will be set to the next available key for the hash table.
	*/
	static int
	tsd_hash_add_key(tsd_hash_table_t table, uint_t keyp, dtor_func_t dtor)
	{
	tsd_hash_entry_t tmp_entry, entry;
	tsd_hash_bin_t *bin;
	ulong_t hash;
	int keys_checked = 0;

	ASSERT3P(table, !=, NULL);

	/* Allocate entry to be used as a destructor for this key */
	entry = kmem_alloc(sizeof (tsd_hash_entry_t), KM_PUSHPAGE);
	if (entry == NULL)
	return (ENOMEM);

	/* Determine next available key value */
	spin_lock(&table->ht_lock);
	do {
	/* Limited to TSD_KEYS_MAX concurrent unique keys */
	if (table->ht_key++ > TSD_KEYS_MAX)
	table->ht_key = 1;

	/* Ensure failure when all TSD_KEYS_MAX keys are in use */
	if (keys_checked++ >= TSD_KEYS_MAX) {
	spin_unlock(&table->ht_lock);
	return (ENOENT);
	}

	tmp_entry = tsd_hash_search(table, table->ht_key, DTOR_PID);
	} while (tmp_entry);

	/* Add destructor entry in to hash table */
	entry->he_key = *keyp = table->ht_key;
	entry->he_pid = DTOR_PID;
	entry->he_dtor = dtor;
	entry->he_value = NULL;
	INIT_HLIST_NODE(&entry->he_list);
	INIT_LIST_HEAD(&entry->he_key_list);
	INIT_LIST_HEAD(&entry->he_pid_list);

	hash = hash_long((ulong_t)keyp (ulong_t)DTOR_PID, table->ht_bits);
	bin = &table->ht_bins[hash];
	spin_lock(&bin->hb_lock);

	hlist_add_head(&entry->he_list, &bin->hb_head);

	spin_unlock(&bin->hb_lock);
	spin_unlock(&table->ht_lock);

	return (0);
	}

	/*
	* tsd_hash_add_pid - adds a process entry to the hash table
	* @table: hash table
	* @pid: search pid
	*
	* For every process there is a single entry in the hash which is used
	* as anchor. All other thread specific entries for this process are
	* linked to this anchor via the 'he_pid_list' list head.
	*/
	static int
	tsd_hash_add_pid(tsd_hash_table_t *table, pid_t pid)
	{
	tsd_hash_entry_t *entry;
	tsd_hash_bin_t *bin;
	ulong_t hash;

	/* Allocate entry to be used as the process reference */
	entry = kmem_alloc(sizeof (tsd_hash_entry_t), KM_PUSHPAGE);
	if (entry == NULL)
	return (ENOMEM);

	spin_lock(&table->ht_lock);
	entry->he_key = PID_KEY;
	entry->he_pid = pid;
	entry->he_dtor = NULL;
	entry->he_value = NULL;
	INIT_HLIST_NODE(&entry->he_list);
	INIT_LIST_HEAD(&entry->he_key_list);
	INIT_LIST_HEAD(&entry->he_pid_list);

	hash = hash_long((ulong_t)PID_KEY * (ulong_t)pid, table->ht_bits);
	bin = &table->ht_bins[hash];
	spin_lock(&bin->hb_lock);

	hlist_add_head(&entry->he_list, &bin->hb_head);

	spin_unlock(&bin->hb_lock);
	spin_unlock(&table->ht_lock);

	return (0);
	}

	/*
	* tsd_hash_del - delete an entry from hash table, key, and pid lists
	* @table: hash table
	* @key: search key
	* @pid: search pid
	*/
	static void
	tsd_hash_del(tsd_hash_table_t table, tsd_hash_entry_t entry)
	{
	hlist_del(&entry->he_list);
	list_del_init(&entry->he_key_list);
	list_del_init(&entry->he_pid_list);
	}

	/*
	* tsd_hash_table_init - allocate a hash table
	* @bits: hash table size
	*
	* A hash table with 2^bits bins will be created, it may not be resized
	* after the fact and must be free'd with tsd_hash_table_fini().
	*/
	static tsd_hash_table_t *
	tsd_hash_table_init(uint_t bits)
	{
	tsd_hash_table_t *table;
	int hash, size = (1 << bits);

	table = kmem_zalloc(sizeof (tsd_hash_table_t), KM_SLEEP);
	if (table == NULL)
	return (NULL);

	table->ht_bins = kmem_zalloc(sizeof (tsd_hash_bin_t) * size, KM_SLEEP);
	if (table->ht_bins == NULL) {
	kmem_free(table, sizeof (tsd_hash_table_t));
	return (NULL);
	}

	for (hash = 0; hash < size; hash++) {
	spin_lock_init(&table->ht_bins[hash].hb_lock);
	INIT_HLIST_HEAD(&table->ht_bins[hash].hb_head);
	}

	spin_lock_init(&table->ht_lock);
	table->ht_bits = bits;
	table->ht_key = 1;

	return (table);
	}

	/*
	* tsd_hash_table_fini - free a hash table
	* @table: hash table
	*
	* Free a hash table allocated by tsd_hash_table_init(). If the hash
	* table is not empty this function will call the proper destructor for
	* all remaining entries before freeing the memory used by those entries.
	*/
	static void
	tsd_hash_table_fini(tsd_hash_table_t *table)
	{
	HLIST_HEAD(work);
	tsd_hash_bin_t *bin;
	tsd_hash_entry_t *entry;
	int size, i;

	ASSERT3P(table, !=, NULL);
	spin_lock(&table->ht_lock);
	for (i = 0, size = (1 << table->ht_bits); i < size; i++) {
	bin = &table->ht_bins[i];
	spin_lock(&bin->hb_lock);
	while (!hlist_empty(&bin->hb_head)) {
	entry = hlist_entry(bin->hb_head.first,
	tsd_hash_entry_t, he_list);
	tsd_hash_del(table, entry);
	hlist_add_head(&entry->he_list, &work);
	}
	spin_unlock(&bin->hb_lock);
	}
	spin_unlock(&table->ht_lock);

	tsd_hash_dtor(&work);
	kmem_free(table->ht_bins, sizeof (tsd_hash_bin_t)*(1<<table->ht_bits));
	kmem_free(table, sizeof (tsd_hash_table_t));
	}

	/*
	* tsd_remove_entry - remove a tsd entry for this thread
	* @entry: entry to remove
	*
	* Remove the thread specific data @entry for this thread.
	* If this is the last entry for this thread, also remove the PID entry.
	*/
	static void
	tsd_remove_entry(tsd_hash_entry_t *entry)
	{
	HLIST_HEAD(work);
	tsd_hash_table_t *table;
	tsd_hash_entry_t *pid_entry;
	tsd_hash_bin_t pid_entry_bin, entry_bin;
	ulong_t hash;

	table = tsd_hash_table;
	ASSERT3P(table, !=, NULL);
	ASSERT3P(entry, !=, NULL);

	spin_lock(&table->ht_lock);

	hash = hash_long((ulong_t)entry->he_key *
	(ulong_t)entry->he_pid, table->ht_bits);
	entry_bin = &table->ht_bins[hash];

	/* save the possible pid_entry */
	pid_entry = list_entry(entry->he_pid_list.next, tsd_hash_entry_t,
	he_pid_list);

	/* remove entry */
	spin_lock(&entry_bin->hb_lock);
	tsd_hash_del(table, entry);
	hlist_add_head(&entry->he_list, &work);
	spin_unlock(&entry_bin->hb_lock);

	/* if pid_entry is indeed pid_entry, then remove it if it's empty */
	if (pid_entry->he_key == PID_KEY &&
	list_empty(&pid_entry->he_pid_list)) {
	hash = hash_long((ulong_t)pid_entry->he_key *
	(ulong_t)pid_entry->he_pid, table->ht_bits);
	pid_entry_bin = &table->ht_bins[hash];

	spin_lock(&pid_entry_bin->hb_lock);
	tsd_hash_del(table, pid_entry);
	hlist_add_head(&pid_entry->he_list, &work);
	spin_unlock(&pid_entry_bin->hb_lock);
	}

	spin_unlock(&table->ht_lock);

	tsd_hash_dtor(&work);
	}

	/*
	* tsd_set - set thread specific data
	* @key: lookup key
	* @value: value to set
	*
	* Caller must prevent racing tsd_create() or tsd_destroy(), protected
	* from racing tsd_get() or tsd_set() because it is thread specific.
	* This function has been optimized to be fast for the update case.
	* When setting the tsd initially it will be slower due to additional
	* required locking and potential memory allocations.
	*/
	int
	tsd_set(uint_t key, void *value)
	{
	tsd_hash_table_t *table;
	tsd_hash_entry_t *entry;
	pid_t pid;
	int rc;
	/* mark remove if value is NULL */
	boolean_t remove = (value == NULL);

	table = tsd_hash_table;
	pid = curthread->pid;
	ASSERT3P(table, !=, NULL);

	if ((key == 0) \|\| (key > TSD_KEYS_MAX))
	return (EINVAL);

	/* Entry already exists in hash table update value */
	entry = tsd_hash_search(table, key, pid);
	if (entry) {
	entry->he_value = value;
	/* remove the entry */
	if (remove)
	tsd_remove_entry(entry);
	return (0);
	}

	/* don't create entry if value is NULL */
	if (remove)
	return (0);

	/* Add a process entry to the hash if not yet exists */
	entry = tsd_hash_search(table, PID_KEY, pid);
	if (entry == NULL) {
	rc = tsd_hash_add_pid(table, pid);
	if (rc)
	return (rc);
	}

	rc = tsd_hash_add(table, key, pid, value);
	return (rc);
	}
	EXPORT_SYMBOL(tsd_set);

	/*
	* tsd_get - get thread specific data
	* @key: lookup key
	*
	* Caller must prevent racing tsd_create() or tsd_destroy(). This
	* implementation is designed to be fast and scalable, it does not
	* lock the entire table only a single hash bin.
	*/
	void *
	tsd_get(uint_t key)
	{
	tsd_hash_entry_t *entry;

	ASSERT3P(tsd_hash_table, !=, NULL);

	if ((key == 0) \|\| (key > TSD_KEYS_MAX))
	return (NULL);

	entry = tsd_hash_search(tsd_hash_table, key, curthread->pid);
	if (entry == NULL)
	return (NULL);

	return (entry->he_value);
	}
	EXPORT_SYMBOL(tsd_get);

	/*
	* tsd_get_by_thread - get thread specific data for specified thread
	* @key: lookup key
	* @thread: thread to lookup
	*
	* Caller must prevent racing tsd_create() or tsd_destroy(). This
	* implementation is designed to be fast and scalable, it does not
	* lock the entire table only a single hash bin.
	*/
	void *
	tsd_get_by_thread(uint_t key, kthread_t *thread)
	{
	tsd_hash_entry_t *entry;

	ASSERT3P(tsd_hash_table, !=, NULL);

	if ((key == 0) \|\| (key > TSD_KEYS_MAX))
	return (NULL);

	entry = tsd_hash_search(tsd_hash_table, key, thread->pid);
	if (entry == NULL)
	return (NULL);

	return (entry->he_value);
	}
	EXPORT_SYMBOL(tsd_get_by_thread);

	/*
	* tsd_create - create thread specific data key
	* @keyp: lookup key address
	* @dtor: destructor called during tsd_destroy() or tsd_exit()
	*
	* Provided key must be set to 0 or it assumed to be already in use.
	* The dtor is allowed to be NULL in which case no additional cleanup
	* for the data is performed during tsd_destroy() or tsd_exit().
	*
	* Caller must prevent racing tsd_set() or tsd_get(), this function is
	* safe from racing tsd_create(), tsd_destroy(), and tsd_exit().
	*/
	void
	tsd_create(uint_t *keyp, dtor_func_t dtor)
	{
	ASSERT3P(keyp, !=, NULL);
	if (*keyp)
	return;

	(void) tsd_hash_add_key(tsd_hash_table, keyp, dtor);
	}
	EXPORT_SYMBOL(tsd_create);

	/*
	* tsd_destroy - destroy thread specific data
	* @keyp: lookup key address
	*
	* Destroys the thread specific data on all threads which use this key.
	*
	* Caller must prevent racing tsd_set() or tsd_get(), this function is
	* safe from racing tsd_create(), tsd_destroy(), and tsd_exit().
	*/
	void
	tsd_destroy(uint_t *keyp)
	{
	HLIST_HEAD(work);
	tsd_hash_table_t *table;
	tsd_hash_entry_t dtor_entry, entry;
	tsd_hash_bin_t dtor_entry_bin, entry_bin;
	ulong_t hash;

	table = tsd_hash_table;
	ASSERT3P(table, !=, NULL);

	spin_lock(&table->ht_lock);
	dtor_entry = tsd_hash_search(table, *keyp, DTOR_PID);
	if (dtor_entry == NULL) {
	spin_unlock(&table->ht_lock);
	return;
	}

	/*
	* All threads which use this key must be linked off of the
	* DTOR_PID entry. They are removed from the hash table and
	* linked in to a private working list to be destroyed.
	*/
	while (!list_empty(&dtor_entry->he_key_list)) {
	entry = list_entry(dtor_entry->he_key_list.next,
	tsd_hash_entry_t, he_key_list);
	ASSERT3U(dtor_entry->he_key, ==, entry->he_key);
	ASSERT3P(dtor_entry->he_dtor, ==, entry->he_dtor);

	hash = hash_long((ulong_t)entry->he_key *
	(ulong_t)entry->he_pid, table->ht_bits);
	entry_bin = &table->ht_bins[hash];

	spin_lock(&entry_bin->hb_lock);
	tsd_hash_del(table, entry);
	hlist_add_head(&entry->he_list, &work);
	spin_unlock(&entry_bin->hb_lock);
	}

	hash = hash_long((ulong_t)dtor_entry->he_key *
	(ulong_t)dtor_entry->he_pid, table->ht_bits);
	dtor_entry_bin = &table->ht_bins[hash];

	spin_lock(&dtor_entry_bin->hb_lock);
	tsd_hash_del(table, dtor_entry);
	hlist_add_head(&dtor_entry->he_list, &work);
	spin_unlock(&dtor_entry_bin->hb_lock);
	spin_unlock(&table->ht_lock);

	tsd_hash_dtor(&work);
	*keyp = 0;
	}
	EXPORT_SYMBOL(tsd_destroy);

	/*
	* tsd_exit - destroys all thread specific data for this thread
	*
	* Destroys all the thread specific data for this thread.
	*
	* Caller must prevent racing tsd_set() or tsd_get(), this function is
	* safe from racing tsd_create(), tsd_destroy(), and tsd_exit().
	*/
	void
	tsd_exit(void)
	{
	HLIST_HEAD(work);
	tsd_hash_table_t *table;
	tsd_hash_entry_t pid_entry, entry;
	tsd_hash_bin_t pid_entry_bin, entry_bin;
	ulong_t hash;

	table = tsd_hash_table;
	ASSERT3P(table, !=, NULL);

	spin_lock(&table->ht_lock);
	pid_entry = tsd_hash_search(table, PID_KEY, curthread->pid);
	if (pid_entry == NULL) {
	spin_unlock(&table->ht_lock);
	return;
	}

	/*
	* All keys associated with this pid must be linked off of the
	* PID_KEY entry. They are removed from the hash table and
	* linked in to a private working list to be destroyed.
	*/

	while (!list_empty(&pid_entry->he_pid_list)) {
	entry = list_entry(pid_entry->he_pid_list.next,
	tsd_hash_entry_t, he_pid_list);
	ASSERT3U(pid_entry->he_pid, ==, entry->he_pid);

	hash = hash_long((ulong_t)entry->he_key *
	(ulong_t)entry->he_pid, table->ht_bits);
	entry_bin = &table->ht_bins[hash];

	spin_lock(&entry_bin->hb_lock);
	tsd_hash_del(table, entry);
	hlist_add_head(&entry->he_list, &work);
	spin_unlock(&entry_bin->hb_lock);
	}

	hash = hash_long((ulong_t)pid_entry->he_key *
	(ulong_t)pid_entry->he_pid, table->ht_bits);
	pid_entry_bin = &table->ht_bins[hash];

	spin_lock(&pid_entry_bin->hb_lock);
	tsd_hash_del(table, pid_entry);
	hlist_add_head(&pid_entry->he_list, &work);
	spin_unlock(&pid_entry_bin->hb_lock);
	spin_unlock(&table->ht_lock);

	tsd_hash_dtor(&work);
	}
	EXPORT_SYMBOL(tsd_exit);

	int
	spl_tsd_init(void)
	{
	tsd_hash_table = tsd_hash_table_init(TSD_HASH_TABLE_BITS_DEFAULT);
	if (tsd_hash_table == NULL)
	return (1);

	return (0);
	}

	void
	spl_tsd_fini(void)
	{
	tsd_hash_table_fini(tsd_hash_table);
	tsd_hash_table = NULL;
	}
	Index: vendor-sys/openzfs/dist/module/os/linux/spl/spl-vmem.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/spl/spl-vmem.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/spl/spl-vmem.c (revision 366774)
	@@ -1,91 +1,90 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*/

	#include <linux/percpu_compat.h>
	#include <sys/debug.h>
	#include <sys/vmem.h>
	#include <sys/kmem_cache.h>
	#include <sys/shrinker.h>
	#include <linux/module.h>

	/*
	* Public vmem_alloc(), vmem_zalloc() and vmem_free() interfaces.
	*/
	void *
	spl_vmem_alloc(size_t size, int flags, const char *func, int line)
	{
	ASSERT0(flags & ~KM_PUBLIC_MASK);

	flags \|= KM_VMEM;

	#if !defined(DEBUG_KMEM)
	return (spl_kmem_alloc_impl(size, flags, NUMA_NO_NODE));
	#elif !defined(DEBUG_KMEM_TRACKING)
	return (spl_kmem_alloc_debug(size, flags, NUMA_NO_NODE));
	#else
	return (spl_kmem_alloc_track(size, flags, func, line, NUMA_NO_NODE));
	#endif
	}
	EXPORT_SYMBOL(spl_vmem_alloc);

	void *
	spl_vmem_zalloc(size_t size, int flags, const char *func, int line)
	{
	ASSERT0(flags & ~KM_PUBLIC_MASK);

	flags \|= (KM_VMEM \| KM_ZERO);

	#if !defined(DEBUG_KMEM)
	return (spl_kmem_alloc_impl(size, flags, NUMA_NO_NODE));
	#elif !defined(DEBUG_KMEM_TRACKING)
	return (spl_kmem_alloc_debug(size, flags, NUMA_NO_NODE));
	#else
	return (spl_kmem_alloc_track(size, flags, func, line, NUMA_NO_NODE));
	#endif
	}
	EXPORT_SYMBOL(spl_vmem_zalloc);

	void
	spl_vmem_free(const void *buf, size_t size)
	{
	#if !defined(DEBUG_KMEM)
	return (spl_kmem_free_impl(buf, size));
	#elif !defined(DEBUG_KMEM_TRACKING)
	return (spl_kmem_free_debug(buf, size));
	#else
	return (spl_kmem_free_track(buf, size));
	#endif
	}
	EXPORT_SYMBOL(spl_vmem_free);

	int
	spl_vmem_init(void)
	{
	return (0);
	}

	void
	spl_vmem_fini(void)
	{
	}
	Index: vendor-sys/openzfs/dist/module/os/linux/spl/spl-xdr.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/spl/spl-xdr.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/spl/spl-xdr.c (revision 366774)
	@@ -1,513 +1,512 @@
	/*
	* Copyright (c) 2008-2010 Sun Microsystems, Inc.
	* Written by Ricardo Correia <Ricardo.M.Correia@Sun.COM>
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*
	* Solaris Porting Layer (SPL) XDR Implementation.
	*/

	#include <linux/string.h>
	#include <sys/kmem.h>
	#include <sys/debug.h>
	#include <sys/types.h>
	#include <sys/sysmacros.h>
	#include <rpc/xdr.h>

	/*
	* SPL's XDR mem implementation.
	*
	* This is used by libnvpair to serialize/deserialize the name-value pair data
	* structures into byte arrays in a well-defined and portable manner.
	*
	* These data structures are used by the DMU/ZFS to flexibly manipulate various
	* information in memory and later serialize it/deserialize it to disk.
	* Examples of usages include the pool configuration, lists of pool and dataset
	* properties, etc.
	*
	* Reference documentation for the XDR representation and XDR operations can be
	* found in RFC 1832 and xdr(3), respectively.
	*
	* === Implementation shortcomings ===
	*
	* It is assumed that the following C types have the following sizes:
	*
	* char/unsigned char: 1 byte
	* short/unsigned short: 2 bytes
	* int/unsigned int: 4 bytes
	* longlong_t/u_longlong_t: 8 bytes
	*
	* The C standard allows these types to be larger (and in the case of ints,
	* shorter), so if that is the case on some compiler/architecture, the build
	* will fail (on purpose).
	*
	* If someone wants to fix the code to work properly on such environments, then:
	*
	* 1) Preconditions should be added to xdrmem_enc functions to make sure the
	* caller doesn't pass arguments which exceed the expected range.
	* 2) Functions which take signed integers should be changed to properly do
	* sign extension.
	* 3) For ints with less than 32 bits, well.. I suspect you'll have bigger
	* problems than this implementation.
	*
	* It is also assumed that:
	*
	* 1) Chars have 8 bits.
	* 2) We can always do 32-bit-aligned int memory accesses and byte-aligned
	* memcpy, memset and memcmp.
	* 3) Arrays passed to xdr_array() are packed and the compiler/architecture
	* supports element-sized-aligned memory accesses.
	* 4) Negative integers are natively stored in two's complement binary
	* representation.
	*
	* No checks are done for the 4 assumptions above, though.
	*
	* === Caller expectations ===
	*
	* Existing documentation does not describe the semantics of XDR operations very
	* well. Therefore, some assumptions about failure semantics will be made and
	* will be described below:
	*
	* 1) If any encoding operation fails (e.g., due to lack of buffer space), the
	* the stream should be considered valid only up to the encoding operation
	* previous to the one that first failed. However, the stream size as returned
	* by xdr_control() cannot be considered to be strictly correct (it may be
	* bigger).
	*
	* Putting it another way, if there is an encoding failure it's undefined
	* whether anything is added to the stream in that operation and therefore
	* neither xdr_control() nor future encoding operations on the same stream can
	* be relied upon to produce correct results.
	*
	* 2) If a decoding operation fails, it's undefined whether anything will be
	* decoded into passed buffers/pointers during that operation, or what the
	* values on those buffers will look like.
	*
	* Future decoding operations on the same stream will also have similar
	* undefined behavior.
	*
	* 3) When the first decoding operation fails it is OK to trust the results of
	* previous decoding operations on the same stream, as long as the caller
	* expects a failure to be possible (e.g. due to end-of-stream).
	*
	* However, this is highly discouraged because the caller should know the
	* stream size and should be coded to expect any decoding failure to be data
	* corruption due to hardware, accidental or even malicious causes, which should
	* be handled gracefully in all cases.
	*
	* In very rare situations where there are strong reasons to believe the data
	* can be trusted to be valid and non-tampered with, then the caller may assume
	* a decoding failure to be a bug (e.g. due to mismatched data types) and may
	* fail non-gracefully.
	*
	* 4) Non-zero padding bytes will cause the decoding operation to fail.
	*
	* 5) Zero bytes on string types will also cause the decoding operation to fail.
	*
	* 6) It is assumed that either the pointer to the stream buffer given by the
	* caller is 32-bit aligned or the architecture supports non-32-bit-aligned int
	* memory accesses.
	*
	* 7) The stream buffer and encoding/decoding buffers/ptrs should not overlap.
	*
	* 8) If a caller passes pointers to non-kernel memory (e.g., pointers to user
	* space or MMIO space), the computer may explode.
	*/

	static struct xdr_ops xdrmem_encode_ops;
	static struct xdr_ops xdrmem_decode_ops;

	void
	xdrmem_create(XDR *xdrs, const caddr_t addr, const uint_t size,
	const enum xdr_op op)
	{
	switch (op) {
	case XDR_ENCODE:
	xdrs->x_ops = &xdrmem_encode_ops;
	break;
	case XDR_DECODE:
	xdrs->x_ops = &xdrmem_decode_ops;
	break;
	default:
	xdrs->x_ops = NULL; /* Let the caller know we failed */
	return;
	}

	xdrs->x_op = op;
	xdrs->x_addr = addr;
	xdrs->x_addr_end = addr + size;

	if (xdrs->x_addr_end < xdrs->x_addr) {
	xdrs->x_ops = NULL;
	}
	}
	EXPORT_SYMBOL(xdrmem_create);

	static bool_t
	xdrmem_control(XDR xdrs, int req, void info)
	{
	struct xdr_bytesrec rec = (struct xdr_bytesrec )info;

	if (req != XDR_GET_BYTES_AVAIL)
	return (FALSE);

	rec->xc_is_last_record = TRUE; /* always TRUE in xdrmem streams */
	rec->xc_num_avail = xdrs->x_addr_end - xdrs->x_addr;

	return (TRUE);
	}

	static bool_t
	xdrmem_enc_bytes(XDR *xdrs, caddr_t cp, const uint_t cnt)
	{
	uint_t size = roundup(cnt, 4);
	uint_t pad;

	if (size < cnt)
	return (FALSE); /* Integer overflow */

	if (xdrs->x_addr > xdrs->x_addr_end)
	return (FALSE);

	if (xdrs->x_addr_end - xdrs->x_addr < size)
	return (FALSE);

	memcpy(xdrs->x_addr, cp, cnt);

	xdrs->x_addr += cnt;

	pad = size - cnt;
	if (pad > 0) {
	memset(xdrs->x_addr, 0, pad);
	xdrs->x_addr += pad;
	}

	return (TRUE);
	}

	static bool_t
	xdrmem_dec_bytes(XDR *xdrs, caddr_t cp, const uint_t cnt)
	{
	static uint32_t zero = 0;
	uint_t size = roundup(cnt, 4);
	uint_t pad;

	if (size < cnt)
	return (FALSE); /* Integer overflow */

	if (xdrs->x_addr > xdrs->x_addr_end)
	return (FALSE);

	if (xdrs->x_addr_end - xdrs->x_addr < size)
	return (FALSE);

	memcpy(cp, xdrs->x_addr, cnt);
	xdrs->x_addr += cnt;

	pad = size - cnt;
	if (pad > 0) {
	/* An inverted memchr() would be useful here... */
	if (memcmp(&zero, xdrs->x_addr, pad) != 0)
	return (FALSE);

	xdrs->x_addr += pad;
	}

	return (TRUE);
	}

	static bool_t
	xdrmem_enc_uint32(XDR *xdrs, uint32_t val)
	{
	if (xdrs->x_addr + sizeof (uint32_t) > xdrs->x_addr_end)
	return (FALSE);

	((uint32_t )xdrs->x_addr) = cpu_to_be32(val);

	xdrs->x_addr += sizeof (uint32_t);

	return (TRUE);
	}

	static bool_t
	xdrmem_dec_uint32(XDR xdrs, uint32_t val)
	{
	if (xdrs->x_addr + sizeof (uint32_t) > xdrs->x_addr_end)
	return (FALSE);

	val = be32_to_cpu(((uint32_t *)xdrs->x_addr));

	xdrs->x_addr += sizeof (uint32_t);

	return (TRUE);
	}

	static bool_t
	xdrmem_enc_char(XDR xdrs, char cp)
	{
	uint32_t val;

	BUILD_BUG_ON(sizeof (char) != 1);
	val = ((unsigned char ) cp);

	return (xdrmem_enc_uint32(xdrs, val));
	}

	static bool_t
	xdrmem_dec_char(XDR xdrs, char cp)
	{
	uint32_t val;

	BUILD_BUG_ON(sizeof (char) != 1);

	if (!xdrmem_dec_uint32(xdrs, &val))
	return (FALSE);

	/*
	* If any of the 3 other bytes are non-zero then val will be greater
	* than 0xff and we fail because according to the RFC, this block does
	* not have a char encoded in it.
	*/
	if (val > 0xff)
	return (FALSE);

	((unsigned char ) cp) = val;

	return (TRUE);
	}

	static bool_t
	xdrmem_enc_ushort(XDR xdrs, unsigned short usp)
	{
	BUILD_BUG_ON(sizeof (unsigned short) != 2);

	return (xdrmem_enc_uint32(xdrs, *usp));
	}

	static bool_t
	xdrmem_dec_ushort(XDR xdrs, unsigned short usp)
	{
	uint32_t val;

	BUILD_BUG_ON(sizeof (unsigned short) != 2);

	if (!xdrmem_dec_uint32(xdrs, &val))
	return (FALSE);

	/*
	* Short ints are not in the RFC, but we assume similar logic as in
	* xdrmem_dec_char().
	*/
	if (val > 0xffff)
	return (FALSE);

	*usp = val;

	return (TRUE);
	}

	static bool_t
	xdrmem_enc_uint(XDR xdrs, unsigned up)
	{
	BUILD_BUG_ON(sizeof (unsigned) != 4);

	return (xdrmem_enc_uint32(xdrs, *up));
	}

	static bool_t
	xdrmem_dec_uint(XDR xdrs, unsigned up)
	{
	BUILD_BUG_ON(sizeof (unsigned) != 4);

	return (xdrmem_dec_uint32(xdrs, (uint32_t *)up));
	}

	static bool_t
	xdrmem_enc_ulonglong(XDR xdrs, u_longlong_t ullp)
	{
	BUILD_BUG_ON(sizeof (u_longlong_t) != 8);

	if (!xdrmem_enc_uint32(xdrs, *ullp >> 32))
	return (FALSE);

	return (xdrmem_enc_uint32(xdrs, *ullp & 0xffffffff));
	}

	static bool_t
	xdrmem_dec_ulonglong(XDR xdrs, u_longlong_t ullp)
	{
	uint32_t low, high;

	BUILD_BUG_ON(sizeof (u_longlong_t) != 8);

	if (!xdrmem_dec_uint32(xdrs, &high))
	return (FALSE);
	if (!xdrmem_dec_uint32(xdrs, &low))
	return (FALSE);

	*ullp = ((u_longlong_t)high << 32) \| low;

	return (TRUE);
	}

	static bool_t
	xdr_enc_array(XDR xdrs, caddr_t arrp, uint_t *sizep, const uint_t maxsize,
	const uint_t elsize, const xdrproc_t elproc)
	{
	uint_t i;
	caddr_t addr = *arrp;

	if (sizep > maxsize \|\| sizep > UINT_MAX / elsize)
	return (FALSE);

	if (!xdrmem_enc_uint(xdrs, sizep))
	return (FALSE);

	for (i = 0; i < *sizep; i++) {
	if (!elproc(xdrs, addr))
	return (FALSE);
	addr += elsize;
	}

	return (TRUE);
	}

	static bool_t
	xdr_dec_array(XDR xdrs, caddr_t arrp, uint_t *sizep, const uint_t maxsize,
	const uint_t elsize, const xdrproc_t elproc)
	{
	uint_t i, size;
	bool_t alloc = FALSE;
	caddr_t addr;

	if (!xdrmem_dec_uint(xdrs, sizep))
	return (FALSE);

	size = *sizep;

	if (size > maxsize \|\| size > UINT_MAX / elsize)
	return (FALSE);

	/*
	* The Solaris man page says: "If *arrp is NULL when decoding,
	* xdr_array() allocates memory and *arrp points to it".
	*/
	if (*arrp == NULL) {
	BUILD_BUG_ON(sizeof (uint_t) > sizeof (size_t));

	arrp = kmem_alloc(size elsize, KM_NOSLEEP);
	if (*arrp == NULL)
	return (FALSE);

	alloc = TRUE;
	}

	addr = *arrp;

	for (i = 0; i < size; i++) {
	if (!elproc(xdrs, addr)) {
	if (alloc)
	kmem_free(arrp, size elsize);
	return (FALSE);
	}
	addr += elsize;
	}

	return (TRUE);
	}

	static bool_t
	xdr_enc_string(XDR xdrs, char *sp, const uint_t maxsize)
	{
	size_t slen = strlen(*sp);
	uint_t len;

	if (slen > maxsize)
	return (FALSE);

	len = slen;

	if (!xdrmem_enc_uint(xdrs, &len))
	return (FALSE);

	return (xdrmem_enc_bytes(xdrs, *sp, len));
	}

	static bool_t
	xdr_dec_string(XDR xdrs, char *sp, const uint_t maxsize)
	{
	uint_t size;
	bool_t alloc = FALSE;

	if (!xdrmem_dec_uint(xdrs, &size))
	return (FALSE);

	if (size > maxsize \|\| size > UINT_MAX - 1)
	return (FALSE);

	/*
	* Solaris man page: "If *sp is NULL when decoding, xdr_string()
	* allocates memory and *sp points to it".
	*/
	if (*sp == NULL) {
	BUILD_BUG_ON(sizeof (uint_t) > sizeof (size_t));

	*sp = kmem_alloc(size + 1, KM_NOSLEEP);
	if (*sp == NULL)
	return (FALSE);

	alloc = TRUE;
	}

	if (!xdrmem_dec_bytes(xdrs, *sp, size))
	goto fail;

	if (memchr(*sp, 0, size) != NULL)
	goto fail;

	(*sp)[size] = '\0';

	return (TRUE);

	fail:
	if (alloc)
	kmem_free(*sp, size + 1);

	return (FALSE);
	}

	static struct xdr_ops xdrmem_encode_ops = {
	.xdr_control = xdrmem_control,
	.xdr_char = xdrmem_enc_char,
	.xdr_u_short = xdrmem_enc_ushort,
	.xdr_u_int = xdrmem_enc_uint,
	.xdr_u_longlong_t = xdrmem_enc_ulonglong,
	.xdr_opaque = xdrmem_enc_bytes,
	.xdr_string = xdr_enc_string,
	.xdr_array = xdr_enc_array
	};

	static struct xdr_ops xdrmem_decode_ops = {
	.xdr_control = xdrmem_control,
	.xdr_char = xdrmem_dec_char,
	.xdr_u_short = xdrmem_dec_ushort,
	.xdr_u_int = xdrmem_dec_uint,
	.xdr_u_longlong_t = xdrmem_dec_ulonglong,
	.xdr_opaque = xdrmem_dec_bytes,
	.xdr_string = xdr_dec_string,
	.xdr_array = xdr_dec_array
	};
	Index: vendor-sys/openzfs/dist/module/os/linux/spl/spl-zlib.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/spl/spl-zlib.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/spl/spl-zlib.c (revision 366774)
	@@ -1,218 +1,217 @@
	/*
	* Copyright (C) 2007-2010 Lawrence Livermore National Security, LLC.
	* Copyright (C) 2007 The Regents of the University of California.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* Written by Brian Behlendorf <behlendorf1@llnl.gov>.
	* UCRL-CODE-235197
	*
	* This file is part of the SPL, Solaris Porting Layer.
	- * For details, see <http://zfsonlinux.org/>.
	*
	* The SPL is free software; you can redistribute it and/or modify it
	* under the terms of the GNU General Public License as published by the
	* Free Software Foundation; either version 2 of the License, or (at your
	* option) any later version.
	*
	* The SPL is distributed in the hope that it will be useful, but WITHOUT
	* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
	* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
	* for more details.
	*
	* You should have received a copy of the GNU General Public License along
	* with the SPL. If not, see <http://www.gnu.org/licenses/>.
	*
	*
	* z_compress_level/z_uncompress are nearly identical copies of the
	* compress2/uncompress functions provided by the official zlib package
	* available at http://zlib.net/. The only changes made we to slightly
	* adapt the functions called to match the linux kernel implementation
	* of zlib. The full zlib license follows:
	*
	* zlib.h -- interface of the 'zlib' general purpose compression library
	* version 1.2.5, April 19th, 2010
	*
	* Copyright (C) 1995-2010 Jean-loup Gailly and Mark Adler
	*
	* This software is provided 'as-is', without any express or implied
	* warranty. In no event will the authors be held liable for any damages
	* arising from the use of this software.
	*
	* Permission is granted to anyone to use this software for any purpose,
	* including commercial applications, and to alter it and redistribute it
	* freely, subject to the following restrictions:
	*
	* 1. The origin of this software must not be misrepresented; you must not
	* claim that you wrote the original software. If you use this software
	* in a product, an acknowledgment in the product documentation would be
	* appreciated but is not required.
	* 2. Altered source versions must be plainly marked as such, and must not be
	* misrepresented as being the original software.
	* 3. This notice may not be removed or altered from any source distribution.
	*
	* Jean-loup Gailly
	* Mark Adler
	*/


	#include <linux/percpu_compat.h>
	#include <sys/kmem.h>
	#include <sys/kmem_cache.h>
	#include <sys/zmod.h>

	static spl_kmem_cache_t *zlib_workspace_cache;

	/*
	* A kmem_cache is used for the zlib workspaces to avoid having to vmalloc
	* and vfree for every call. Using a kmem_cache also has the advantage
	* that improves the odds that the memory used will be local to this cpu.
	* To further improve things it might be wise to create a dedicated per-cpu
	* workspace for use. This would take some additional care because we then
	* must disable preemption around the critical section, and verify that
	* zlib_deflate* and zlib_inflate* never internally call schedule().
	*/
	static void *
	zlib_workspace_alloc(int flags)
	{
	return (kmem_cache_alloc(zlib_workspace_cache, flags & ~(__GFP_FS)));
	}

	static void
	zlib_workspace_free(void *workspace)
	{
	kmem_cache_free(zlib_workspace_cache, workspace);
	}

	/*
	* Compresses the source buffer into the destination buffer. The level
	* parameter has the same meaning as in deflateInit. sourceLen is the byte
	* length of the source buffer. Upon entry, destLen is the total size of the
	* destination buffer, which must be at least 0.1% larger than sourceLen plus
	* 12 bytes. Upon exit, destLen is the actual size of the compressed buffer.
	*
	* compress2 returns Z_OK if success, Z_MEM_ERROR if there was not enough
	* memory, Z_BUF_ERROR if there was not enough room in the output buffer,
	* Z_STREAM_ERROR if the level parameter is invalid.
	*/
	int
	z_compress_level(void dest, size_t destLen, const void *source,
	size_t sourceLen, int level)
	{
	z_stream stream;
	int err;

	stream.next_in = (Byte *)source;
	stream.avail_in = (uInt)sourceLen;
	stream.next_out = dest;
	stream.avail_out = (uInt)*destLen;

	if ((size_t)stream.avail_out != *destLen)
	return (Z_BUF_ERROR);

	stream.workspace = zlib_workspace_alloc(KM_SLEEP);
	if (!stream.workspace)
	return (Z_MEM_ERROR);

	err = zlib_deflateInit(&stream, level);
	if (err != Z_OK) {
	zlib_workspace_free(stream.workspace);
	return (err);
	}

	err = zlib_deflate(&stream, Z_FINISH);
	if (err != Z_STREAM_END) {
	zlib_deflateEnd(&stream);
	zlib_workspace_free(stream.workspace);
	return (err == Z_OK ? Z_BUF_ERROR : err);
	}
	*destLen = stream.total_out;

	err = zlib_deflateEnd(&stream);
	zlib_workspace_free(stream.workspace);

	return (err);
	}
	EXPORT_SYMBOL(z_compress_level);

	/*
	* Decompresses the source buffer into the destination buffer. sourceLen is
	* the byte length of the source buffer. Upon entry, destLen is the total
	* size of the destination buffer, which must be large enough to hold the
	* entire uncompressed data. (The size of the uncompressed data must have
	* been saved previously by the compressor and transmitted to the decompressor
	* by some mechanism outside the scope of this compression library.)
	* Upon exit, destLen is the actual size of the compressed buffer.
	* This function can be used to decompress a whole file at once if the
	* input file is mmap'ed.
	*
	* uncompress returns Z_OK if success, Z_MEM_ERROR if there was not
	* enough memory, Z_BUF_ERROR if there was not enough room in the output
	* buffer, or Z_DATA_ERROR if the input data was corrupted.
	*/
	int
	z_uncompress(void dest, size_t destLen, const void *source, size_t sourceLen)
	{
	z_stream stream;
	int err;

	stream.next_in = (Byte *)source;
	stream.avail_in = (uInt)sourceLen;
	stream.next_out = dest;
	stream.avail_out = (uInt)*destLen;

	if ((size_t)stream.avail_out != *destLen)
	return (Z_BUF_ERROR);

	stream.workspace = zlib_workspace_alloc(KM_SLEEP);
	if (!stream.workspace)
	return (Z_MEM_ERROR);

	err = zlib_inflateInit(&stream);
	if (err != Z_OK) {
	zlib_workspace_free(stream.workspace);
	return (err);
	}

	err = zlib_inflate(&stream, Z_FINISH);
	if (err != Z_STREAM_END) {
	zlib_inflateEnd(&stream);
	zlib_workspace_free(stream.workspace);

	if (err == Z_NEED_DICT \|\|
	(err == Z_BUF_ERROR && stream.avail_in == 0))
	return (Z_DATA_ERROR);

	return (err);
	}
	*destLen = stream.total_out;

	err = zlib_inflateEnd(&stream);
	zlib_workspace_free(stream.workspace);

	return (err);
	}
	EXPORT_SYMBOL(z_uncompress);

	int
	spl_zlib_init(void)
	{
	int size;

	size = MAX(zlib_deflate_workspacesize(MAX_WBITS, MAX_MEM_LEVEL),
	zlib_inflate_workspacesize());

	zlib_workspace_cache = kmem_cache_create(
	"spl_zlib_workspace_cache",
	size, 0, NULL, NULL, NULL, NULL, NULL,
	KMC_KVMEM);
	if (!zlib_workspace_cache)
	return (1);

	return (0);
	}

	void
	spl_zlib_fini(void)
	{
	kmem_cache_destroy(zlib_workspace_cache);
	zlib_workspace_cache = NULL;
	}
	Index: vendor-sys/openzfs/dist/module/os/linux/zfs/zfs_acl.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/zfs/zfs_acl.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/zfs/zfs_acl.c (revision 366774)
	@@ -1,2932 +1,2932 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2013 by Delphix. All rights reserved.
	* Copyright 2014 Nexenta Systems, Inc. All rights reserved.
	*/


	#include <sys/types.h>
	#include <sys/param.h>
	#include <sys/time.h>
	#include <sys/sysmacros.h>
	#include <sys/vfs.h>
	#include <sys/vnode.h>
	#include <sys/sid.h>
	#include <sys/file.h>
	#include <sys/stat.h>
	#include <sys/kmem.h>
	#include <sys/cmn_err.h>
	#include <sys/errno.h>
	#include <sys/fs/zfs.h>
	#include <sys/policy.h>
	#include <sys/zfs_znode.h>
	#include <sys/zfs_fuid.h>
	#include <sys/zfs_acl.h>
	#include <sys/zfs_dir.h>
	#include <sys/zfs_quota.h>
	#include <sys/zfs_vfsops.h>
	#include <sys/dmu.h>
	#include <sys/dnode.h>
	#include <sys/zap.h>
	#include <sys/sa.h>
	#include <sys/trace_acl.h>
	#include <sys/zpl.h>

	#define ALLOW ACE_ACCESS_ALLOWED_ACE_TYPE
	#define DENY ACE_ACCESS_DENIED_ACE_TYPE
	#define MAX_ACE_TYPE ACE_SYSTEM_ALARM_CALLBACK_OBJECT_ACE_TYPE
	#define MIN_ACE_TYPE ALLOW

	#define OWNING_GROUP (ACE_GROUP\|ACE_IDENTIFIER_GROUP)
	#define EVERYONE_ALLOW_MASK (ACE_READ_ACL\|ACE_READ_ATTRIBUTES \| \
	ACE_READ_NAMED_ATTRS\|ACE_SYNCHRONIZE)
	#define EVERYONE_DENY_MASK (ACE_WRITE_ACL\|ACE_WRITE_OWNER \| \
	ACE_WRITE_ATTRIBUTES\|ACE_WRITE_NAMED_ATTRS)
	#define OWNER_ALLOW_MASK (ACE_WRITE_ACL \| ACE_WRITE_OWNER \| \
	ACE_WRITE_ATTRIBUTES\|ACE_WRITE_NAMED_ATTRS)

	#define ZFS_CHECKED_MASKS (ACE_READ_ACL\|ACE_READ_ATTRIBUTES\|ACE_READ_DATA\| \
	ACE_READ_NAMED_ATTRS\|ACE_WRITE_DATA\|ACE_WRITE_ATTRIBUTES\| \
	ACE_WRITE_NAMED_ATTRS\|ACE_APPEND_DATA\|ACE_EXECUTE\|ACE_WRITE_OWNER\| \
	ACE_WRITE_ACL\|ACE_DELETE\|ACE_DELETE_CHILD\|ACE_SYNCHRONIZE)

	#define WRITE_MASK_DATA (ACE_WRITE_DATA\|ACE_APPEND_DATA\|ACE_WRITE_NAMED_ATTRS)
	#define WRITE_MASK_ATTRS (ACE_WRITE_ACL\|ACE_WRITE_OWNER\|ACE_WRITE_ATTRIBUTES\| \
	ACE_DELETE\|ACE_DELETE_CHILD)
	#define WRITE_MASK (WRITE_MASK_DATA\|WRITE_MASK_ATTRS)

	#define OGE_CLEAR (ACE_READ_DATA\|ACE_LIST_DIRECTORY\|ACE_WRITE_DATA\| \
	ACE_ADD_FILE\|ACE_APPEND_DATA\|ACE_ADD_SUBDIRECTORY\|ACE_EXECUTE)

	#define OKAY_MASK_BITS (ACE_READ_DATA\|ACE_LIST_DIRECTORY\|ACE_WRITE_DATA\| \
	ACE_ADD_FILE\|ACE_APPEND_DATA\|ACE_ADD_SUBDIRECTORY\|ACE_EXECUTE)

	#define ALL_INHERIT (ACE_FILE_INHERIT_ACE\|ACE_DIRECTORY_INHERIT_ACE \| \
	ACE_NO_PROPAGATE_INHERIT_ACE\|ACE_INHERIT_ONLY_ACE\|ACE_INHERITED_ACE)

	#define RESTRICTED_CLEAR (ACE_WRITE_ACL\|ACE_WRITE_OWNER)

	#define V4_ACL_WIDE_FLAGS (ZFS_ACL_AUTO_INHERIT\|ZFS_ACL_DEFAULTED\|\
	ZFS_ACL_PROTECTED)

	#define ZFS_ACL_WIDE_FLAGS (V4_ACL_WIDE_FLAGS\|ZFS_ACL_TRIVIAL\|ZFS_INHERIT_ACE\|\
	ZFS_ACL_OBJ_ACE)

	#define ALL_MODE_EXECS (S_IXUSR \| S_IXGRP \| S_IXOTH)

	#define IDMAP_WK_CREATOR_OWNER_UID 2147483648U

	static uint16_t
	zfs_ace_v0_get_type(void *acep)
	{
	return (((zfs_oldace_t *)acep)->z_type);
	}

	static uint16_t
	zfs_ace_v0_get_flags(void *acep)
	{
	return (((zfs_oldace_t *)acep)->z_flags);
	}

	static uint32_t
	zfs_ace_v0_get_mask(void *acep)
	{
	return (((zfs_oldace_t *)acep)->z_access_mask);
	}

	static uint64_t
	zfs_ace_v0_get_who(void *acep)
	{
	return (((zfs_oldace_t *)acep)->z_fuid);
	}

	static void
	zfs_ace_v0_set_type(void *acep, uint16_t type)
	{
	((zfs_oldace_t *)acep)->z_type = type;
	}

	static void
	zfs_ace_v0_set_flags(void *acep, uint16_t flags)
	{
	((zfs_oldace_t *)acep)->z_flags = flags;
	}

	static void
	zfs_ace_v0_set_mask(void *acep, uint32_t mask)
	{
	((zfs_oldace_t *)acep)->z_access_mask = mask;
	}

	static void
	zfs_ace_v0_set_who(void *acep, uint64_t who)
	{
	((zfs_oldace_t *)acep)->z_fuid = who;
	}

	/ARGSUSED/
	static size_t
	zfs_ace_v0_size(void *acep)
	{
	return (sizeof (zfs_oldace_t));
	}

	static size_t
	zfs_ace_v0_abstract_size(void)
	{
	return (sizeof (zfs_oldace_t));
	}

	static int
	zfs_ace_v0_mask_off(void)
	{
	return (offsetof(zfs_oldace_t, z_access_mask));
	}

	/ARGSUSED/
	static int
	zfs_ace_v0_data(void acep, void *datap)
	{
	*datap = NULL;
	return (0);
	}

	static acl_ops_t zfs_acl_v0_ops = {
	.ace_mask_get = zfs_ace_v0_get_mask,
	.ace_mask_set = zfs_ace_v0_set_mask,
	.ace_flags_get = zfs_ace_v0_get_flags,
	.ace_flags_set = zfs_ace_v0_set_flags,
	.ace_type_get = zfs_ace_v0_get_type,
	.ace_type_set = zfs_ace_v0_set_type,
	.ace_who_get = zfs_ace_v0_get_who,
	.ace_who_set = zfs_ace_v0_set_who,
	.ace_size = zfs_ace_v0_size,
	.ace_abstract_size = zfs_ace_v0_abstract_size,
	.ace_mask_off = zfs_ace_v0_mask_off,
	.ace_data = zfs_ace_v0_data
	};

	static uint16_t
	zfs_ace_fuid_get_type(void *acep)
	{
	return (((zfs_ace_hdr_t *)acep)->z_type);
	}

	static uint16_t
	zfs_ace_fuid_get_flags(void *acep)
	{
	return (((zfs_ace_hdr_t *)acep)->z_flags);
	}

	static uint32_t
	zfs_ace_fuid_get_mask(void *acep)
	{
	return (((zfs_ace_hdr_t *)acep)->z_access_mask);
	}

	static uint64_t
	zfs_ace_fuid_get_who(void *args)
	{
	uint16_t entry_type;
	zfs_ace_t *acep = args;

	entry_type = acep->z_hdr.z_flags & ACE_TYPE_FLAGS;

	if (entry_type == ACE_OWNER \|\| entry_type == OWNING_GROUP \|\|
	entry_type == ACE_EVERYONE)
	return (-1);
	return (((zfs_ace_t *)acep)->z_fuid);
	}

	static void
	zfs_ace_fuid_set_type(void *acep, uint16_t type)
	{
	((zfs_ace_hdr_t *)acep)->z_type = type;
	}

	static void
	zfs_ace_fuid_set_flags(void *acep, uint16_t flags)
	{
	((zfs_ace_hdr_t *)acep)->z_flags = flags;
	}

	static void
	zfs_ace_fuid_set_mask(void *acep, uint32_t mask)
	{
	((zfs_ace_hdr_t *)acep)->z_access_mask = mask;
	}

	static void
	zfs_ace_fuid_set_who(void *arg, uint64_t who)
	{
	zfs_ace_t *acep = arg;

	uint16_t entry_type = acep->z_hdr.z_flags & ACE_TYPE_FLAGS;

	if (entry_type == ACE_OWNER \|\| entry_type == OWNING_GROUP \|\|
	entry_type == ACE_EVERYONE)
	return;
	acep->z_fuid = who;
	}

	static size_t
	zfs_ace_fuid_size(void *acep)
	{
	zfs_ace_hdr_t *zacep = acep;
	uint16_t entry_type;

	switch (zacep->z_type) {
	case ACE_ACCESS_ALLOWED_OBJECT_ACE_TYPE:
	case ACE_ACCESS_DENIED_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_AUDIT_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_ALARM_OBJECT_ACE_TYPE:
	return (sizeof (zfs_object_ace_t));
	case ALLOW:
	case DENY:
	entry_type =
	(((zfs_ace_hdr_t *)acep)->z_flags & ACE_TYPE_FLAGS);
	if (entry_type == ACE_OWNER \|\|
	entry_type == OWNING_GROUP \|\|
	entry_type == ACE_EVERYONE)
	return (sizeof (zfs_ace_hdr_t));
	/FALLTHROUGH/
	default:
	return (sizeof (zfs_ace_t));
	}
	}

	static size_t
	zfs_ace_fuid_abstract_size(void)
	{
	return (sizeof (zfs_ace_hdr_t));
	}

	static int
	zfs_ace_fuid_mask_off(void)
	{
	return (offsetof(zfs_ace_hdr_t, z_access_mask));
	}

	static int
	zfs_ace_fuid_data(void acep, void *datap)
	{
	zfs_ace_t *zacep = acep;
	zfs_object_ace_t *zobjp;

	switch (zacep->z_hdr.z_type) {
	case ACE_ACCESS_ALLOWED_OBJECT_ACE_TYPE:
	case ACE_ACCESS_DENIED_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_AUDIT_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_ALARM_OBJECT_ACE_TYPE:
	zobjp = acep;
	*datap = (caddr_t)zobjp + sizeof (zfs_ace_t);
	return (sizeof (zfs_object_ace_t) - sizeof (zfs_ace_t));
	default:
	*datap = NULL;
	return (0);
	}
	}

	static acl_ops_t zfs_acl_fuid_ops = {
	.ace_mask_get = zfs_ace_fuid_get_mask,
	.ace_mask_set = zfs_ace_fuid_set_mask,
	.ace_flags_get = zfs_ace_fuid_get_flags,
	.ace_flags_set = zfs_ace_fuid_set_flags,
	.ace_type_get = zfs_ace_fuid_get_type,
	.ace_type_set = zfs_ace_fuid_set_type,
	.ace_who_get = zfs_ace_fuid_get_who,
	.ace_who_set = zfs_ace_fuid_set_who,
	.ace_size = zfs_ace_fuid_size,
	.ace_abstract_size = zfs_ace_fuid_abstract_size,
	.ace_mask_off = zfs_ace_fuid_mask_off,
	.ace_data = zfs_ace_fuid_data
	};

	/*
	* The following three functions are provided for compatibility with
	* older ZPL version in order to determine if the file use to have
	* an external ACL and what version of ACL previously existed on the
	* file. Would really be nice to not need this, sigh.
	*/
	uint64_t
	zfs_external_acl(znode_t *zp)
	{
	zfs_acl_phys_t acl_phys;
	int error;

	if (zp->z_is_sa)
	return (0);

	/*
	* Need to deal with a potential
	* race where zfs_sa_upgrade could cause
	* z_isa_sa to change.
	*
	* If the lookup fails then the state of z_is_sa should have
	* changed.
	*/

	if ((error = sa_lookup(zp->z_sa_hdl, SA_ZPL_ZNODE_ACL(ZTOZSB(zp)),
	&acl_phys, sizeof (acl_phys))) == 0)
	return (acl_phys.z_acl_extern_obj);
	else {
	/*
	* after upgrade the SA_ZPL_ZNODE_ACL should have been
	* removed
	*/
	VERIFY(zp->z_is_sa && error == ENOENT);
	return (0);
	}
	}

	/*
	* Determine size of ACL in bytes
	*
	* This is more complicated than it should be since we have to deal
	* with old external ACLs.
	*/
	static int
	zfs_acl_znode_info(znode_t zp, int aclsize, int *aclcount,
	zfs_acl_phys_t *aclphys)
	{
	zfsvfs_t *zfsvfs = ZTOZSB(zp);
	uint64_t acl_count;
	int size;
	int error;

	ASSERT(MUTEX_HELD(&zp->z_acl_lock));
	if (zp->z_is_sa) {
	if ((error = sa_size(zp->z_sa_hdl, SA_ZPL_DACL_ACES(zfsvfs),
	&size)) != 0)
	return (error);
	*aclsize = size;
	if ((error = sa_lookup(zp->z_sa_hdl, SA_ZPL_DACL_COUNT(zfsvfs),
	&acl_count, sizeof (acl_count))) != 0)
	return (error);
	*aclcount = acl_count;
	} else {
	if ((error = sa_lookup(zp->z_sa_hdl, SA_ZPL_ZNODE_ACL(zfsvfs),
	aclphys, sizeof (*aclphys))) != 0)
	return (error);

	if (aclphys->z_acl_version == ZFS_ACL_VERSION_INITIAL) {
	*aclsize = ZFS_ACL_SIZE(aclphys->z_acl_size);
	*aclcount = aclphys->z_acl_size;
	} else {
	*aclsize = aclphys->z_acl_size;
	*aclcount = aclphys->z_acl_count;
	}
	}
	return (0);
	}

	int
	zfs_znode_acl_version(znode_t *zp)
	{
	zfs_acl_phys_t acl_phys;

	if (zp->z_is_sa)
	return (ZFS_ACL_VERSION_FUID);
	else {
	int error;

	/*
	* Need to deal with a potential
	* race where zfs_sa_upgrade could cause
	* z_isa_sa to change.
	*
	* If the lookup fails then the state of z_is_sa should have
	* changed.
	*/
	if ((error = sa_lookup(zp->z_sa_hdl,
	SA_ZPL_ZNODE_ACL(ZTOZSB(zp)),
	&acl_phys, sizeof (acl_phys))) == 0)
	return (acl_phys.z_acl_version);
	else {
	/*
	* After upgrade SA_ZPL_ZNODE_ACL should have
	* been removed.
	*/
	VERIFY(zp->z_is_sa && error == ENOENT);
	return (ZFS_ACL_VERSION_FUID);
	}
	}
	}

	static int
	zfs_acl_version(int version)
	{
	if (version < ZPL_VERSION_FUID)
	return (ZFS_ACL_VERSION_INITIAL);
	else
	return (ZFS_ACL_VERSION_FUID);
	}

	static int
	zfs_acl_version_zp(znode_t *zp)
	{
	return (zfs_acl_version(ZTOZSB(zp)->z_version));
	}

	zfs_acl_t *
	zfs_acl_alloc(int vers)
	{
	zfs_acl_t *aclp;

	aclp = kmem_zalloc(sizeof (zfs_acl_t), KM_SLEEP);
	list_create(&aclp->z_acl, sizeof (zfs_acl_node_t),
	offsetof(zfs_acl_node_t, z_next));
	aclp->z_version = vers;
	if (vers == ZFS_ACL_VERSION_FUID)
	aclp->z_ops = &zfs_acl_fuid_ops;
	else
	aclp->z_ops = &zfs_acl_v0_ops;
	return (aclp);
	}

	zfs_acl_node_t *
	zfs_acl_node_alloc(size_t bytes)
	{
	zfs_acl_node_t *aclnode;

	aclnode = kmem_zalloc(sizeof (zfs_acl_node_t), KM_SLEEP);
	if (bytes) {
	aclnode->z_acldata = kmem_alloc(bytes, KM_SLEEP);
	aclnode->z_allocdata = aclnode->z_acldata;
	aclnode->z_allocsize = bytes;
	aclnode->z_size = bytes;
	}

	return (aclnode);
	}

	static void
	zfs_acl_node_free(zfs_acl_node_t *aclnode)
	{
	if (aclnode->z_allocsize)
	kmem_free(aclnode->z_allocdata, aclnode->z_allocsize);
	kmem_free(aclnode, sizeof (zfs_acl_node_t));
	}

	static void
	zfs_acl_release_nodes(zfs_acl_t *aclp)
	{
	zfs_acl_node_t *aclnode;

	while ((aclnode = list_head(&aclp->z_acl))) {
	list_remove(&aclp->z_acl, aclnode);
	zfs_acl_node_free(aclnode);
	}
	aclp->z_acl_count = 0;
	aclp->z_acl_bytes = 0;
	}

	void
	zfs_acl_free(zfs_acl_t *aclp)
	{
	zfs_acl_release_nodes(aclp);
	list_destroy(&aclp->z_acl);
	kmem_free(aclp, sizeof (zfs_acl_t));
	}

	static boolean_t
	zfs_acl_valid_ace_type(uint_t type, uint_t flags)
	{
	uint16_t entry_type;

	switch (type) {
	case ALLOW:
	case DENY:
	case ACE_SYSTEM_AUDIT_ACE_TYPE:
	case ACE_SYSTEM_ALARM_ACE_TYPE:
	entry_type = flags & ACE_TYPE_FLAGS;
	return (entry_type == ACE_OWNER \|\|
	entry_type == OWNING_GROUP \|\|
	entry_type == ACE_EVERYONE \|\| entry_type == 0 \|\|
	entry_type == ACE_IDENTIFIER_GROUP);
	default:
	if (type >= MIN_ACE_TYPE && type <= MAX_ACE_TYPE)
	return (B_TRUE);
	}
	return (B_FALSE);
	}

	static boolean_t
	zfs_ace_valid(umode_t obj_mode, zfs_acl_t *aclp, uint16_t type, uint16_t iflags)
	{
	/*
	* first check type of entry
	*/

	if (!zfs_acl_valid_ace_type(type, iflags))
	return (B_FALSE);

	switch (type) {
	case ACE_ACCESS_ALLOWED_OBJECT_ACE_TYPE:
	case ACE_ACCESS_DENIED_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_AUDIT_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_ALARM_OBJECT_ACE_TYPE:
	if (aclp->z_version < ZFS_ACL_VERSION_FUID)
	return (B_FALSE);
	aclp->z_hints \|= ZFS_ACL_OBJ_ACE;
	}

	/*
	* next check inheritance level flags
	*/

	if (S_ISDIR(obj_mode) &&
	(iflags & (ACE_FILE_INHERIT_ACE\|ACE_DIRECTORY_INHERIT_ACE)))
	aclp->z_hints \|= ZFS_INHERIT_ACE;

	if (iflags & (ACE_INHERIT_ONLY_ACE\|ACE_NO_PROPAGATE_INHERIT_ACE)) {
	if ((iflags & (ACE_FILE_INHERIT_ACE\|
	ACE_DIRECTORY_INHERIT_ACE)) == 0) {
	return (B_FALSE);
	}
	}

	return (B_TRUE);
	}

	static void *
	zfs_acl_next_ace(zfs_acl_t aclp, void start, uint64_t *who,
	uint32_t access_mask, uint16_t iflags, uint16_t *type)
	{
	zfs_acl_node_t *aclnode;

	ASSERT(aclp);

	if (start == NULL) {
	aclnode = list_head(&aclp->z_acl);
	if (aclnode == NULL)
	return (NULL);

	aclp->z_next_ace = aclnode->z_acldata;
	aclp->z_curr_node = aclnode;
	aclnode->z_ace_idx = 0;
	}

	aclnode = aclp->z_curr_node;

	if (aclnode == NULL)
	return (NULL);

	if (aclnode->z_ace_idx >= aclnode->z_ace_count) {
	aclnode = list_next(&aclp->z_acl, aclnode);
	if (aclnode == NULL)
	return (NULL);
	else {
	aclp->z_curr_node = aclnode;
	aclnode->z_ace_idx = 0;
	aclp->z_next_ace = aclnode->z_acldata;
	}
	}

	if (aclnode->z_ace_idx < aclnode->z_ace_count) {
	void *acep = aclp->z_next_ace;
	size_t ace_size;

	/*
	* Make sure we don't overstep our bounds
	*/
	ace_size = aclp->z_ops->ace_size(acep);

	if (((caddr_t)acep + ace_size) >
	((caddr_t)aclnode->z_acldata + aclnode->z_size)) {
	return (NULL);
	}

	*iflags = aclp->z_ops->ace_flags_get(acep);
	*type = aclp->z_ops->ace_type_get(acep);
	*access_mask = aclp->z_ops->ace_mask_get(acep);
	*who = aclp->z_ops->ace_who_get(acep);
	aclp->z_next_ace = (caddr_t)aclp->z_next_ace + ace_size;
	aclnode->z_ace_idx++;

	return ((void *)acep);
	}
	return (NULL);
	}

	/ARGSUSED/
	static uint64_t
	zfs_ace_walk(void *datap, uint64_t cookie, int aclcnt,
	uint16_t flags, uint16_t type, uint32_t *mask)
	{
	zfs_acl_t *aclp = datap;
	zfs_ace_hdr_t acep = (zfs_ace_hdr_t )(uintptr_t)cookie;
	uint64_t who;

	acep = zfs_acl_next_ace(aclp, acep, &who, mask,
	flags, type);
	return ((uint64_t)(uintptr_t)acep);
	}

	/*
	* Copy ACE to internal ZFS format.
	* While processing the ACL each ACE will be validated for correctness.
	* ACE FUIDs will be created later.
	*/
	static int
	zfs_copy_ace_2_fuid(zfsvfs_t zfsvfs, umode_t obj_mode, zfs_acl_t aclp,
	void datap, zfs_ace_t z_acl, uint64_t aclcnt, size_t *size,
	zfs_fuid_info_t *fuidp, cred_t cr)
	{
	int i;
	uint16_t entry_type;
	zfs_ace_t *aceptr = z_acl;
	ace_t *acep = datap;
	zfs_object_ace_t *zobjacep;
	ace_object_t *aceobjp;

	for (i = 0; i != aclcnt; i++) {
	aceptr->z_hdr.z_access_mask = acep->a_access_mask;
	aceptr->z_hdr.z_flags = acep->a_flags;
	aceptr->z_hdr.z_type = acep->a_type;
	entry_type = aceptr->z_hdr.z_flags & ACE_TYPE_FLAGS;
	if (entry_type != ACE_OWNER && entry_type != OWNING_GROUP &&
	entry_type != ACE_EVERYONE) {
	aceptr->z_fuid = zfs_fuid_create(zfsvfs, acep->a_who,
	cr, (entry_type == 0) ?
	ZFS_ACE_USER : ZFS_ACE_GROUP, fuidp);
	}

	/*
	* Make sure ACE is valid
	*/
	if (zfs_ace_valid(obj_mode, aclp, aceptr->z_hdr.z_type,
	aceptr->z_hdr.z_flags) != B_TRUE)
	return (SET_ERROR(EINVAL));

	switch (acep->a_type) {
	case ACE_ACCESS_ALLOWED_OBJECT_ACE_TYPE:
	case ACE_ACCESS_DENIED_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_AUDIT_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_ALARM_OBJECT_ACE_TYPE:
	zobjacep = (zfs_object_ace_t *)aceptr;
	aceobjp = (ace_object_t *)acep;

	bcopy(aceobjp->a_obj_type, zobjacep->z_object_type,
	sizeof (aceobjp->a_obj_type));
	bcopy(aceobjp->a_inherit_obj_type,
	zobjacep->z_inherit_type,
	sizeof (aceobjp->a_inherit_obj_type));
	acep = (ace_t *)((caddr_t)acep + sizeof (ace_object_t));
	break;
	default:
	acep = (ace_t *)((caddr_t)acep + sizeof (ace_t));
	}

	aceptr = (zfs_ace_t *)((caddr_t)aceptr +
	aclp->z_ops->ace_size(aceptr));
	}

	*size = (caddr_t)aceptr - (caddr_t)z_acl;

	return (0);
	}

	/*
	* Copy ZFS ACEs to fixed size ace_t layout
	*/
	static void
	zfs_copy_fuid_2_ace(zfsvfs_t zfsvfs, zfs_acl_t aclp, cred_t *cr,
	void *datap, int filter)
	{
	uint64_t who;
	uint32_t access_mask;
	uint16_t iflags, type;
	zfs_ace_hdr_t *zacep = NULL;
	ace_t *acep = datap;
	ace_object_t *objacep;
	zfs_object_ace_t *zobjacep;
	size_t ace_size;
	uint16_t entry_type;

	while ((zacep = zfs_acl_next_ace(aclp, zacep,
	&who, &access_mask, &iflags, &type))) {

	switch (type) {
	case ACE_ACCESS_ALLOWED_OBJECT_ACE_TYPE:
	case ACE_ACCESS_DENIED_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_AUDIT_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_ALARM_OBJECT_ACE_TYPE:
	if (filter) {
	continue;
	}
	zobjacep = (zfs_object_ace_t *)zacep;
	objacep = (ace_object_t *)acep;
	bcopy(zobjacep->z_object_type,
	objacep->a_obj_type,
	sizeof (zobjacep->z_object_type));
	bcopy(zobjacep->z_inherit_type,
	objacep->a_inherit_obj_type,
	sizeof (zobjacep->z_inherit_type));
	ace_size = sizeof (ace_object_t);
	break;
	default:
	ace_size = sizeof (ace_t);
	break;
	}

	entry_type = (iflags & ACE_TYPE_FLAGS);
	if ((entry_type != ACE_OWNER &&
	entry_type != OWNING_GROUP &&
	entry_type != ACE_EVERYONE)) {
	acep->a_who = zfs_fuid_map_id(zfsvfs, who,
	cr, (entry_type & ACE_IDENTIFIER_GROUP) ?
	ZFS_ACE_GROUP : ZFS_ACE_USER);
	} else {
	acep->a_who = (uid_t)(int64_t)who;
	}
	acep->a_access_mask = access_mask;
	acep->a_flags = iflags;
	acep->a_type = type;
	acep = (ace_t *)((caddr_t)acep + ace_size);
	}
	}

	static int
	zfs_copy_ace_2_oldace(umode_t obj_mode, zfs_acl_t aclp, ace_t acep,
	zfs_oldace_t z_acl, int aclcnt, size_t size)
	{
	int i;
	zfs_oldace_t *aceptr = z_acl;

	for (i = 0; i != aclcnt; i++, aceptr++) {
	aceptr->z_access_mask = acep[i].a_access_mask;
	aceptr->z_type = acep[i].a_type;
	aceptr->z_flags = acep[i].a_flags;
	aceptr->z_fuid = acep[i].a_who;
	/*
	* Make sure ACE is valid
	*/
	if (zfs_ace_valid(obj_mode, aclp, aceptr->z_type,
	aceptr->z_flags) != B_TRUE)
	return (SET_ERROR(EINVAL));
	}
	*size = (caddr_t)aceptr - (caddr_t)z_acl;
	return (0);
	}

	/*
	* convert old ACL format to new
	*/
	void
	zfs_acl_xform(znode_t zp, zfs_acl_t aclp, cred_t *cr)
	{
	zfs_oldace_t *oldaclp;
	int i;
	uint16_t type, iflags;
	uint32_t access_mask;
	uint64_t who;
	void *cookie = NULL;
	zfs_acl_node_t *newaclnode;

	ASSERT(aclp->z_version == ZFS_ACL_VERSION_INITIAL);
	/*
	* First create the ACE in a contiguous piece of memory
	* for zfs_copy_ace_2_fuid().
	*
	* We only convert an ACL once, so this won't happen
	* every time.
	*/
	oldaclp = kmem_alloc(sizeof (zfs_oldace_t) * aclp->z_acl_count,
	KM_SLEEP);
	i = 0;
	while ((cookie = zfs_acl_next_ace(aclp, cookie, &who,
	&access_mask, &iflags, &type))) {
	oldaclp[i].z_flags = iflags;
	oldaclp[i].z_type = type;
	oldaclp[i].z_fuid = who;
	oldaclp[i++].z_access_mask = access_mask;
	}

	newaclnode = zfs_acl_node_alloc(aclp->z_acl_count *
	sizeof (zfs_object_ace_t));
	aclp->z_ops = &zfs_acl_fuid_ops;
	VERIFY(zfs_copy_ace_2_fuid(ZTOZSB(zp), ZTOI(zp)->i_mode,
	aclp, oldaclp, newaclnode->z_acldata, aclp->z_acl_count,
	&newaclnode->z_size, NULL, cr) == 0);
	newaclnode->z_ace_count = aclp->z_acl_count;
	aclp->z_version = ZFS_ACL_VERSION;
	kmem_free(oldaclp, aclp->z_acl_count * sizeof (zfs_oldace_t));

	/*
	* Release all previous ACL nodes
	*/

	zfs_acl_release_nodes(aclp);

	list_insert_head(&aclp->z_acl, newaclnode);

	aclp->z_acl_bytes = newaclnode->z_size;
	aclp->z_acl_count = newaclnode->z_ace_count;

	}

	/*
	* Convert unix access mask to v4 access mask
	*/
	static uint32_t
	zfs_unix_to_v4(uint32_t access_mask)
	{
	uint32_t new_mask = 0;

	if (access_mask & S_IXOTH)
	new_mask \|= ACE_EXECUTE;
	if (access_mask & S_IWOTH)
	new_mask \|= ACE_WRITE_DATA;
	if (access_mask & S_IROTH)
	new_mask \|= ACE_READ_DATA;
	return (new_mask);
	}

	static void
	zfs_set_ace(zfs_acl_t aclp, void acep, uint32_t access_mask,
	uint16_t access_type, uint64_t fuid, uint16_t entry_type)
	{
	uint16_t type = entry_type & ACE_TYPE_FLAGS;

	aclp->z_ops->ace_mask_set(acep, access_mask);
	aclp->z_ops->ace_type_set(acep, access_type);
	aclp->z_ops->ace_flags_set(acep, entry_type);
	if ((type != ACE_OWNER && type != OWNING_GROUP &&
	type != ACE_EVERYONE))
	aclp->z_ops->ace_who_set(acep, fuid);
	}

	/*
	* Determine mode of file based on ACL.
	*/
	uint64_t
	zfs_mode_compute(uint64_t fmode, zfs_acl_t *aclp,
	uint64_t *pflags, uint64_t fuid, uint64_t fgid)
	{
	int entry_type;
	mode_t mode;
	mode_t seen = 0;
	zfs_ace_hdr_t *acep = NULL;
	uint64_t who;
	uint16_t iflags, type;
	uint32_t access_mask;
	boolean_t an_exec_denied = B_FALSE;

	mode = (fmode & (S_IFMT \| S_ISUID \| S_ISGID \| S_ISVTX));

	while ((acep = zfs_acl_next_ace(aclp, acep, &who,
	&access_mask, &iflags, &type))) {

	if (!zfs_acl_valid_ace_type(type, iflags))
	continue;

	entry_type = (iflags & ACE_TYPE_FLAGS);

	/*
	* Skip over any inherit_only ACEs
	*/
	if (iflags & ACE_INHERIT_ONLY_ACE)
	continue;

	if (entry_type == ACE_OWNER \|\| (entry_type == 0 &&
	who == fuid)) {
	if ((access_mask & ACE_READ_DATA) &&
	(!(seen & S_IRUSR))) {
	seen \|= S_IRUSR;
	if (type == ALLOW) {
	mode \|= S_IRUSR;
	}
	}
	if ((access_mask & ACE_WRITE_DATA) &&
	(!(seen & S_IWUSR))) {
	seen \|= S_IWUSR;
	if (type == ALLOW) {
	mode \|= S_IWUSR;
	}
	}
	if ((access_mask & ACE_EXECUTE) &&
	(!(seen & S_IXUSR))) {
	seen \|= S_IXUSR;
	if (type == ALLOW) {
	mode \|= S_IXUSR;
	}
	}
	} else if (entry_type == OWNING_GROUP \|\|
	(entry_type == ACE_IDENTIFIER_GROUP && who == fgid)) {
	if ((access_mask & ACE_READ_DATA) &&
	(!(seen & S_IRGRP))) {
	seen \|= S_IRGRP;
	if (type == ALLOW) {
	mode \|= S_IRGRP;
	}
	}
	if ((access_mask & ACE_WRITE_DATA) &&
	(!(seen & S_IWGRP))) {
	seen \|= S_IWGRP;
	if (type == ALLOW) {
	mode \|= S_IWGRP;
	}
	}
	if ((access_mask & ACE_EXECUTE) &&
	(!(seen & S_IXGRP))) {
	seen \|= S_IXGRP;
	if (type == ALLOW) {
	mode \|= S_IXGRP;
	}
	}
	} else if (entry_type == ACE_EVERYONE) {
	if ((access_mask & ACE_READ_DATA)) {
	if (!(seen & S_IRUSR)) {
	seen \|= S_IRUSR;
	if (type == ALLOW) {
	mode \|= S_IRUSR;
	}
	}
	if (!(seen & S_IRGRP)) {
	seen \|= S_IRGRP;
	if (type == ALLOW) {
	mode \|= S_IRGRP;
	}
	}
	if (!(seen & S_IROTH)) {
	seen \|= S_IROTH;
	if (type == ALLOW) {
	mode \|= S_IROTH;
	}
	}
	}
	if ((access_mask & ACE_WRITE_DATA)) {
	if (!(seen & S_IWUSR)) {
	seen \|= S_IWUSR;
	if (type == ALLOW) {
	mode \|= S_IWUSR;
	}
	}
	if (!(seen & S_IWGRP)) {
	seen \|= S_IWGRP;
	if (type == ALLOW) {
	mode \|= S_IWGRP;
	}
	}
	if (!(seen & S_IWOTH)) {
	seen \|= S_IWOTH;
	if (type == ALLOW) {
	mode \|= S_IWOTH;
	}
	}
	}
	if ((access_mask & ACE_EXECUTE)) {
	if (!(seen & S_IXUSR)) {
	seen \|= S_IXUSR;
	if (type == ALLOW) {
	mode \|= S_IXUSR;
	}
	}
	if (!(seen & S_IXGRP)) {
	seen \|= S_IXGRP;
	if (type == ALLOW) {
	mode \|= S_IXGRP;
	}
	}
	if (!(seen & S_IXOTH)) {
	seen \|= S_IXOTH;
	if (type == ALLOW) {
	mode \|= S_IXOTH;
	}
	}
	}
	} else {
	/*
	* Only care if this IDENTIFIER_GROUP or
	* USER ACE denies execute access to someone,
	* mode is not affected
	*/
	if ((access_mask & ACE_EXECUTE) && type == DENY)
	an_exec_denied = B_TRUE;
	}
	}

	/*
	* Failure to allow is effectively a deny, so execute permission
	* is denied if it was never mentioned or if we explicitly
	* weren't allowed it.
	*/
	if (!an_exec_denied &&
	((seen & ALL_MODE_EXECS) != ALL_MODE_EXECS \|\|
	(mode & ALL_MODE_EXECS) != ALL_MODE_EXECS))
	an_exec_denied = B_TRUE;

	if (an_exec_denied)
	*pflags &= ~ZFS_NO_EXECS_DENIED;
	else
	*pflags \|= ZFS_NO_EXECS_DENIED;

	return (mode);
	}

	/*
	* Read an external acl object. If the intent is to modify, always
	* create a new acl and leave any cached acl in place.
	*/
	int
	zfs_acl_node_read(struct znode zp, boolean_t have_lock, zfs_acl_t *aclpp,
	boolean_t will_modify)
	{
	zfs_acl_t *aclp;
	int aclsize = 0;
	int acl_count = 0;
	zfs_acl_node_t *aclnode;
	zfs_acl_phys_t znode_acl;
	int version;
	int error;
	boolean_t drop_lock = B_FALSE;

	ASSERT(MUTEX_HELD(&zp->z_acl_lock));

	if (zp->z_acl_cached && !will_modify) {
	*aclpp = zp->z_acl_cached;
	return (0);
	}

	/*
	* close race where znode could be upgrade while trying to
	* read the znode attributes.
	*
	* But this could only happen if the file isn't already an SA
	* znode
	*/
	if (!zp->z_is_sa && !have_lock) {
	mutex_enter(&zp->z_lock);
	drop_lock = B_TRUE;
	}
	version = zfs_znode_acl_version(zp);

	if ((error = zfs_acl_znode_info(zp, &aclsize,
	&acl_count, &znode_acl)) != 0) {
	goto done;
	}

	aclp = zfs_acl_alloc(version);

	aclp->z_acl_count = acl_count;
	aclp->z_acl_bytes = aclsize;

	aclnode = zfs_acl_node_alloc(aclsize);
	aclnode->z_ace_count = aclp->z_acl_count;
	aclnode->z_size = aclsize;

	if (!zp->z_is_sa) {
	if (znode_acl.z_acl_extern_obj) {
	error = dmu_read(ZTOZSB(zp)->z_os,
	znode_acl.z_acl_extern_obj, 0, aclnode->z_size,
	aclnode->z_acldata, DMU_READ_PREFETCH);
	} else {
	bcopy(znode_acl.z_ace_data, aclnode->z_acldata,
	aclnode->z_size);
	}
	} else {
	error = sa_lookup(zp->z_sa_hdl, SA_ZPL_DACL_ACES(ZTOZSB(zp)),
	aclnode->z_acldata, aclnode->z_size);
	}

	if (error != 0) {
	zfs_acl_free(aclp);
	zfs_acl_node_free(aclnode);
	/* convert checksum errors into IO errors */
	if (error == ECKSUM)
	error = SET_ERROR(EIO);
	goto done;
	}

	list_insert_head(&aclp->z_acl, aclnode);

	*aclpp = aclp;
	if (!will_modify)
	zp->z_acl_cached = aclp;
	done:
	if (drop_lock)
	mutex_exit(&zp->z_lock);
	return (error);
	}

	/ARGSUSED/
	void
	zfs_acl_data_locator(void *dataptr, uint32_t length, uint32_t buflen,
	boolean_t start, void *userdata)
	{
	zfs_acl_locator_cb_t cb = (zfs_acl_locator_cb_t )userdata;

	if (start) {
	cb->cb_acl_node = list_head(&cb->cb_aclp->z_acl);
	} else {
	cb->cb_acl_node = list_next(&cb->cb_aclp->z_acl,
	cb->cb_acl_node);
	}
	*dataptr = cb->cb_acl_node->z_acldata;
	*length = cb->cb_acl_node->z_size;
	}

	int
	zfs_acl_chown_setattr(znode_t *zp)
	{
	int error;
	zfs_acl_t *aclp;

	if (ZTOZSB(zp)->z_acl_type == ZFS_ACLTYPE_POSIX)
	return (0);

	ASSERT(MUTEX_HELD(&zp->z_lock));
	ASSERT(MUTEX_HELD(&zp->z_acl_lock));

	error = zfs_acl_node_read(zp, B_TRUE, &aclp, B_FALSE);
	if (error == 0 && aclp->z_acl_count > 0)
	zp->z_mode = ZTOI(zp)->i_mode =
	zfs_mode_compute(zp->z_mode, aclp,
	&zp->z_pflags, KUID_TO_SUID(ZTOI(zp)->i_uid),
	KGID_TO_SGID(ZTOI(zp)->i_gid));

	/*
	* Some ZFS implementations (ZEVO) create neither a ZNODE_ACL
	* nor a DACL_ACES SA in which case ENOENT is returned from
	* zfs_acl_node_read() when the SA can't be located.
	* Allow chown/chgrp to succeed in these cases rather than
	* returning an error that makes no sense in the context of
	* the caller.
	*/
	if (error == ENOENT)
	return (0);

	return (error);
	}

	typedef struct trivial_acl {
	uint32_t allow0; /* allow mask for bits only in owner */
	uint32_t deny1; /* deny mask for bits not in owner */
	uint32_t deny2; /* deny mask for bits not in group */
	uint32_t owner; /* allow mask matching mode */
	uint32_t group; /* allow mask matching mode */
	uint32_t everyone; /* allow mask matching mode */
	} trivial_acl_t;

	static void
	acl_trivial_access_masks(mode_t mode, boolean_t isdir, trivial_acl_t *masks)
	{
	uint32_t read_mask = ACE_READ_DATA;
	uint32_t write_mask = ACE_WRITE_DATA\|ACE_APPEND_DATA;
	uint32_t execute_mask = ACE_EXECUTE;

	if (isdir)
	write_mask \|= ACE_DELETE_CHILD;

	masks->deny1 = 0;

	if (!(mode & S_IRUSR) && (mode & (S_IRGRP\|S_IROTH)))
	masks->deny1 \|= read_mask;
	if (!(mode & S_IWUSR) && (mode & (S_IWGRP\|S_IWOTH)))
	masks->deny1 \|= write_mask;
	if (!(mode & S_IXUSR) && (mode & (S_IXGRP\|S_IXOTH)))
	masks->deny1 \|= execute_mask;

	masks->deny2 = 0;
	if (!(mode & S_IRGRP) && (mode & S_IROTH))
	masks->deny2 \|= read_mask;
	if (!(mode & S_IWGRP) && (mode & S_IWOTH))
	masks->deny2 \|= write_mask;
	if (!(mode & S_IXGRP) && (mode & S_IXOTH))
	masks->deny2 \|= execute_mask;

	masks->allow0 = 0;
	if ((mode & S_IRUSR) && (!(mode & S_IRGRP) && (mode & S_IROTH)))
	masks->allow0 \|= read_mask;
	if ((mode & S_IWUSR) && (!(mode & S_IWGRP) && (mode & S_IWOTH)))
	masks->allow0 \|= write_mask;
	if ((mode & S_IXUSR) && (!(mode & S_IXGRP) && (mode & S_IXOTH)))
	masks->allow0 \|= execute_mask;

	masks->owner = ACE_WRITE_ATTRIBUTES\|ACE_WRITE_OWNER\|ACE_WRITE_ACL\|
	ACE_WRITE_NAMED_ATTRS\|ACE_READ_ACL\|ACE_READ_ATTRIBUTES\|
	ACE_READ_NAMED_ATTRS\|ACE_SYNCHRONIZE;
	if (mode & S_IRUSR)
	masks->owner \|= read_mask;
	if (mode & S_IWUSR)
	masks->owner \|= write_mask;
	if (mode & S_IXUSR)
	masks->owner \|= execute_mask;

	masks->group = ACE_READ_ACL\|ACE_READ_ATTRIBUTES\|ACE_READ_NAMED_ATTRS\|
	ACE_SYNCHRONIZE;
	if (mode & S_IRGRP)
	masks->group \|= read_mask;
	if (mode & S_IWGRP)
	masks->group \|= write_mask;
	if (mode & S_IXGRP)
	masks->group \|= execute_mask;

	masks->everyone = ACE_READ_ACL\|ACE_READ_ATTRIBUTES\|ACE_READ_NAMED_ATTRS\|
	ACE_SYNCHRONIZE;
	if (mode & S_IROTH)
	masks->everyone \|= read_mask;
	if (mode & S_IWOTH)
	masks->everyone \|= write_mask;
	if (mode & S_IXOTH)
	masks->everyone \|= execute_mask;
	}

	/*
	* ace_trivial:
	* determine whether an ace_t acl is trivial
	*
	* Trivialness implies that the acl is composed of only
	* owner, group, everyone entries. ACL can't
	* have read_acl denied, and write_owner/write_acl/write_attributes
	* can only be owner@ entry.
	*/
	static int
	ace_trivial_common(void *acep, int aclcnt,
	uint64_t (walk)(void , uint64_t, int aclcnt,
	uint16_t , uint16_t , uint32_t *))
	{
	uint16_t flags;
	uint32_t mask;
	uint16_t type;
	uint64_t cookie = 0;

	while ((cookie = walk(acep, cookie, aclcnt, &flags, &type, &mask))) {
	switch (flags & ACE_TYPE_FLAGS) {
	case ACE_OWNER:
	case ACE_GROUP\|ACE_IDENTIFIER_GROUP:
	case ACE_EVERYONE:
	break;
	default:
	return (1);
	}

	if (flags & (ACE_FILE_INHERIT_ACE\|
	ACE_DIRECTORY_INHERIT_ACE\|ACE_NO_PROPAGATE_INHERIT_ACE\|
	ACE_INHERIT_ONLY_ACE))
	return (1);

	/*
	* Special check for some special bits
	*
	* Don't allow anybody to deny reading basic
	* attributes or a files ACL.
	*/
	if ((mask & (ACE_READ_ACL\|ACE_READ_ATTRIBUTES)) &&
	(type == ACE_ACCESS_DENIED_ACE_TYPE))
	return (1);

	/*
	* Delete permission is never set by default
	*/
	if (mask & ACE_DELETE)
	return (1);

	/*
	* Child delete permission should be accompanied by write
	*/
	if ((mask & ACE_DELETE_CHILD) && !(mask & ACE_WRITE_DATA))
	return (1);

	/*
	* only allow owner@ to have
	* write_acl/write_owner/write_attributes/write_xattr/
	*/
	if (type == ACE_ACCESS_ALLOWED_ACE_TYPE &&
	(!(flags & ACE_OWNER) && (mask &
	(ACE_WRITE_OWNER\|ACE_WRITE_ACL\| ACE_WRITE_ATTRIBUTES\|
	ACE_WRITE_NAMED_ATTRS))))
	return (1);

	}

	return (0);
	}

	/*
	* common code for setting ACLs.
	*
	* This function is called from zfs_mode_update, zfs_perm_init, and zfs_setacl.
	* zfs_setacl passes a non-NULL inherit pointer (ihp) to indicate that it's
	* already checked the acl and knows whether to inherit.
	*/
	int
	zfs_aclset_common(znode_t zp, zfs_acl_t aclp, cred_t cr, dmu_tx_t tx)
	{
	int error;
	zfsvfs_t *zfsvfs = ZTOZSB(zp);
	dmu_object_type_t otype;
	zfs_acl_locator_cb_t locate = { 0 };
	uint64_t mode;
	sa_bulk_attr_t bulk[5];
	uint64_t ctime[2];
	int count = 0;
	zfs_acl_phys_t acl_phys;

	mode = zp->z_mode;

	mode = zfs_mode_compute(mode, aclp, &zp->z_pflags,
	KUID_TO_SUID(ZTOI(zp)->i_uid), KGID_TO_SGID(ZTOI(zp)->i_gid));

	zp->z_mode = ZTOI(zp)->i_mode = mode;
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_MODE(zfsvfs), NULL,
	&mode, sizeof (mode));
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_FLAGS(zfsvfs), NULL,
	&zp->z_pflags, sizeof (zp->z_pflags));
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_CTIME(zfsvfs), NULL,
	&ctime, sizeof (ctime));

	if (zp->z_acl_cached) {
	zfs_acl_free(zp->z_acl_cached);
	zp->z_acl_cached = NULL;
	}

	/*
	* Upgrade needed?
	*/
	if (!zfsvfs->z_use_fuids) {
	otype = DMU_OT_OLDACL;
	} else {
	if ((aclp->z_version == ZFS_ACL_VERSION_INITIAL) &&
	(zfsvfs->z_version >= ZPL_VERSION_FUID))
	zfs_acl_xform(zp, aclp, cr);
	ASSERT(aclp->z_version >= ZFS_ACL_VERSION_FUID);
	otype = DMU_OT_ACL;
	}

	/*
	* Arrgh, we have to handle old on disk format
	* as well as newer (preferred) SA format.
	*/

	if (zp->z_is_sa) { /* the easy case, just update the ACL attribute */
	locate.cb_aclp = aclp;
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_DACL_ACES(zfsvfs),
	zfs_acl_data_locator, &locate, aclp->z_acl_bytes);
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_DACL_COUNT(zfsvfs),
	NULL, &aclp->z_acl_count, sizeof (uint64_t));
	} else { /* Painful legacy way */
	zfs_acl_node_t *aclnode;
	uint64_t off = 0;
	uint64_t aoid;

	if ((error = sa_lookup(zp->z_sa_hdl, SA_ZPL_ZNODE_ACL(zfsvfs),
	&acl_phys, sizeof (acl_phys))) != 0)
	return (error);

	aoid = acl_phys.z_acl_extern_obj;

	if (aclp->z_acl_bytes > ZFS_ACE_SPACE) {
	/*
	* If ACL was previously external and we are now
	* converting to new ACL format then release old
	* ACL object and create a new one.
	*/
	if (aoid &&
	aclp->z_version != acl_phys.z_acl_version) {
	error = dmu_object_free(zfsvfs->z_os, aoid, tx);
	if (error)
	return (error);
	aoid = 0;
	}
	if (aoid == 0) {
	aoid = dmu_object_alloc(zfsvfs->z_os,
	otype, aclp->z_acl_bytes,
	otype == DMU_OT_ACL ?
	DMU_OT_SYSACL : DMU_OT_NONE,
	otype == DMU_OT_ACL ?
	DN_OLD_MAX_BONUSLEN : 0, tx);
	} else {
	(void) dmu_object_set_blocksize(zfsvfs->z_os,
	aoid, aclp->z_acl_bytes, 0, tx);
	}
	acl_phys.z_acl_extern_obj = aoid;
	for (aclnode = list_head(&aclp->z_acl); aclnode;
	aclnode = list_next(&aclp->z_acl, aclnode)) {
	if (aclnode->z_ace_count == 0)
	continue;
	dmu_write(zfsvfs->z_os, aoid, off,
	aclnode->z_size, aclnode->z_acldata, tx);
	off += aclnode->z_size;
	}
	} else {
	void *start = acl_phys.z_ace_data;
	/*
	* Migrating back embedded?
	*/
	if (acl_phys.z_acl_extern_obj) {
	error = dmu_object_free(zfsvfs->z_os,
	acl_phys.z_acl_extern_obj, tx);
	if (error)
	return (error);
	acl_phys.z_acl_extern_obj = 0;
	}

	for (aclnode = list_head(&aclp->z_acl); aclnode;
	aclnode = list_next(&aclp->z_acl, aclnode)) {
	if (aclnode->z_ace_count == 0)
	continue;
	bcopy(aclnode->z_acldata, start,
	aclnode->z_size);
	start = (caddr_t)start + aclnode->z_size;
	}
	}
	/*
	* If Old version then swap count/bytes to match old
	* layout of znode_acl_phys_t.
	*/
	if (aclp->z_version == ZFS_ACL_VERSION_INITIAL) {
	acl_phys.z_acl_size = aclp->z_acl_count;
	acl_phys.z_acl_count = aclp->z_acl_bytes;
	} else {
	acl_phys.z_acl_size = aclp->z_acl_bytes;
	acl_phys.z_acl_count = aclp->z_acl_count;
	}
	acl_phys.z_acl_version = aclp->z_version;

	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_ZNODE_ACL(zfsvfs), NULL,
	&acl_phys, sizeof (acl_phys));
	}

	/*
	* Replace ACL wide bits, but first clear them.
	*/
	zp->z_pflags &= ~ZFS_ACL_WIDE_FLAGS;

	zp->z_pflags \|= aclp->z_hints;

	if (ace_trivial_common(aclp, 0, zfs_ace_walk) == 0)
	zp->z_pflags \|= ZFS_ACL_TRIVIAL;

	zfs_tstamp_update_setup(zp, STATE_CHANGED, NULL, ctime);
	return (sa_bulk_update(zp->z_sa_hdl, bulk, count, tx));
	}

	static void
	zfs_acl_chmod(boolean_t isdir, uint64_t mode, boolean_t split, boolean_t trim,
	zfs_acl_t *aclp)
	{
	void *acep = NULL;
	uint64_t who;
	int new_count, new_bytes;
	int ace_size;
	int entry_type;
	uint16_t iflags, type;
	uint32_t access_mask;
	zfs_acl_node_t *newnode;
	size_t abstract_size = aclp->z_ops->ace_abstract_size();
	void *zacep;
	trivial_acl_t masks;

	new_count = new_bytes = 0;

	acl_trivial_access_masks((mode_t)mode, isdir, &masks);

	newnode = zfs_acl_node_alloc((abstract_size * 6) + aclp->z_acl_bytes);

	zacep = newnode->z_acldata;
	if (masks.allow0) {
	zfs_set_ace(aclp, zacep, masks.allow0, ALLOW, -1, ACE_OWNER);
	zacep = (void *)((uintptr_t)zacep + abstract_size);
	new_count++;
	new_bytes += abstract_size;
	}
	if (masks.deny1) {
	zfs_set_ace(aclp, zacep, masks.deny1, DENY, -1, ACE_OWNER);
	zacep = (void *)((uintptr_t)zacep + abstract_size);
	new_count++;
	new_bytes += abstract_size;
	}
	if (masks.deny2) {
	zfs_set_ace(aclp, zacep, masks.deny2, DENY, -1, OWNING_GROUP);
	zacep = (void *)((uintptr_t)zacep + abstract_size);
	new_count++;
	new_bytes += abstract_size;
	}

	while ((acep = zfs_acl_next_ace(aclp, acep, &who, &access_mask,
	&iflags, &type))) {
	entry_type = (iflags & ACE_TYPE_FLAGS);
	/*
	* ACEs used to represent the file mode may be divided
	* into an equivalent pair of inherit-only and regular
	* ACEs, if they are inheritable.
	* Skip regular ACEs, which are replaced by the new mode.
	*/
	if (split && (entry_type == ACE_OWNER \|\|
	entry_type == OWNING_GROUP \|\|
	entry_type == ACE_EVERYONE)) {
	if (!isdir \|\| !(iflags &
	(ACE_FILE_INHERIT_ACE\|ACE_DIRECTORY_INHERIT_ACE)))
	continue;
	/*
	* We preserve owner@, group@, or @everyone
	* permissions, if they are inheritable, by
	* copying them to inherit_only ACEs. This
	* prevents inheritable permissions from being
	* altered along with the file mode.
	*/
	iflags \|= ACE_INHERIT_ONLY_ACE;
	}

	/*
	* If this ACL has any inheritable ACEs, mark that in
	* the hints (which are later masked into the pflags)
	* so create knows to do inheritance.
	*/
	if (isdir && (iflags &
	(ACE_FILE_INHERIT_ACE\|ACE_DIRECTORY_INHERIT_ACE)))
	aclp->z_hints \|= ZFS_INHERIT_ACE;

	if ((type != ALLOW && type != DENY) \|\|
	(iflags & ACE_INHERIT_ONLY_ACE)) {
	switch (type) {
	case ACE_ACCESS_ALLOWED_OBJECT_ACE_TYPE:
	case ACE_ACCESS_DENIED_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_AUDIT_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_ALARM_OBJECT_ACE_TYPE:
	aclp->z_hints \|= ZFS_ACL_OBJ_ACE;
	break;
	}
	} else {
	/*
	* Limit permissions to be no greater than
	* group permissions.
	* The "aclinherit" and "aclmode" properties
	* affect policy for create and chmod(2),
	* respectively.
	*/
	if ((type == ALLOW) && trim)
	access_mask &= masks.group;
	}
	zfs_set_ace(aclp, zacep, access_mask, type, who, iflags);
	ace_size = aclp->z_ops->ace_size(acep);
	zacep = (void *)((uintptr_t)zacep + ace_size);
	new_count++;
	new_bytes += ace_size;
	}
	zfs_set_ace(aclp, zacep, masks.owner, ALLOW, -1, ACE_OWNER);
	zacep = (void *)((uintptr_t)zacep + abstract_size);
	zfs_set_ace(aclp, zacep, masks.group, ALLOW, -1, OWNING_GROUP);
	zacep = (void *)((uintptr_t)zacep + abstract_size);
	zfs_set_ace(aclp, zacep, masks.everyone, ALLOW, -1, ACE_EVERYONE);

	new_count += 3;
	new_bytes += abstract_size * 3;
	zfs_acl_release_nodes(aclp);
	aclp->z_acl_count = new_count;
	aclp->z_acl_bytes = new_bytes;
	newnode->z_ace_count = new_count;
	newnode->z_size = new_bytes;
	list_insert_tail(&aclp->z_acl, newnode);
	}

	int
	zfs_acl_chmod_setattr(znode_t zp, zfs_acl_t *aclp, uint64_t mode)
	{
	int error = 0;

	mutex_enter(&zp->z_acl_lock);
	mutex_enter(&zp->z_lock);
	if (ZTOZSB(zp)->z_acl_mode == ZFS_ACL_DISCARD)
	*aclp = zfs_acl_alloc(zfs_acl_version_zp(zp));
	else
	error = zfs_acl_node_read(zp, B_TRUE, aclp, B_TRUE);

	if (error == 0) {
	(*aclp)->z_hints = zp->z_pflags & V4_ACL_WIDE_FLAGS;
	zfs_acl_chmod(S_ISDIR(ZTOI(zp)->i_mode), mode, B_TRUE,
	(ZTOZSB(zp)->z_acl_mode == ZFS_ACL_GROUPMASK), *aclp);
	}
	mutex_exit(&zp->z_lock);
	mutex_exit(&zp->z_acl_lock);

	return (error);
	}

	/*
	* Should ACE be inherited?
	*/
	static int
	zfs_ace_can_use(umode_t obj_mode, uint16_t acep_flags)
	{
	int iflags = (acep_flags & 0xf);

	if (S_ISDIR(obj_mode) && (iflags & ACE_DIRECTORY_INHERIT_ACE))
	return (1);
	else if (iflags & ACE_FILE_INHERIT_ACE)
	return (!(S_ISDIR(obj_mode) &&
	(iflags & ACE_NO_PROPAGATE_INHERIT_ACE)));
	return (0);
	}

	/*
	* inherit inheritable ACEs from parent
	*/
	static zfs_acl_t *
	zfs_acl_inherit(zfsvfs_t zfsvfs, umode_t va_mode, zfs_acl_t paclp,
	uint64_t mode, boolean_t *need_chmod)
	{
	void *pacep = NULL;
	void *acep;
	zfs_acl_node_t *aclnode;
	zfs_acl_t *aclp = NULL;
	uint64_t who;
	uint32_t access_mask;
	uint16_t iflags, newflags, type;
	size_t ace_size;
	void data1, data2;
	size_t data1sz, data2sz;
	uint_t aclinherit;
	boolean_t isdir = S_ISDIR(va_mode);
	boolean_t isreg = S_ISREG(va_mode);

	*need_chmod = B_TRUE;

	aclp = zfs_acl_alloc(paclp->z_version);
	aclinherit = zfsvfs->z_acl_inherit;
	if (aclinherit == ZFS_ACL_DISCARD \|\| S_ISLNK(va_mode))
	return (aclp);

	while ((pacep = zfs_acl_next_ace(paclp, pacep, &who,
	&access_mask, &iflags, &type))) {

	/*
	* don't inherit bogus ACEs
	*/
	if (!zfs_acl_valid_ace_type(type, iflags))
	continue;

	/*
	* Check if ACE is inheritable by this vnode
	*/
	if ((aclinherit == ZFS_ACL_NOALLOW && type == ALLOW) \|\|
	!zfs_ace_can_use(va_mode, iflags))
	continue;

	/*
	* If owner@, group@, or everyone@ inheritable
	* then zfs_acl_chmod() isn't needed.
	*/
	if ((aclinherit == ZFS_ACL_PASSTHROUGH \|\|
	aclinherit == ZFS_ACL_PASSTHROUGH_X) &&
	((iflags & (ACE_OWNER\|ACE_EVERYONE)) \|\|
	((iflags & OWNING_GROUP) == OWNING_GROUP)) &&
	(isreg \|\| (isdir && (iflags & ACE_DIRECTORY_INHERIT_ACE))))
	*need_chmod = B_FALSE;

	/*
	* Strip inherited execute permission from file if
	* not in mode
	*/
	if (aclinherit == ZFS_ACL_PASSTHROUGH_X && type == ALLOW &&
	!isdir && ((mode & (S_IXUSR\|S_IXGRP\|S_IXOTH)) == 0)) {
	access_mask &= ~ACE_EXECUTE;
	}

	/*
	* Strip write_acl and write_owner from permissions
	* when inheriting an ACE
	*/
	if (aclinherit == ZFS_ACL_RESTRICTED && type == ALLOW) {
	access_mask &= ~RESTRICTED_CLEAR;
	}

	ace_size = aclp->z_ops->ace_size(pacep);
	aclnode = zfs_acl_node_alloc(ace_size);
	list_insert_tail(&aclp->z_acl, aclnode);
	acep = aclnode->z_acldata;

	zfs_set_ace(aclp, acep, access_mask, type,
	who, iflags\|ACE_INHERITED_ACE);

	/*
	* Copy special opaque data if any
	*/
	if ((data1sz = paclp->z_ops->ace_data(pacep, &data1)) != 0) {
	VERIFY((data2sz = aclp->z_ops->ace_data(acep,
	&data2)) == data1sz);
	bcopy(data1, data2, data2sz);
	}

	aclp->z_acl_count++;
	aclnode->z_ace_count++;
	aclp->z_acl_bytes += aclnode->z_size;
	newflags = aclp->z_ops->ace_flags_get(acep);

	/*
	* If ACE is not to be inherited further, or if the vnode is
	* not a directory, remove all inheritance flags
	*/
	if (!isdir \|\| (iflags & ACE_NO_PROPAGATE_INHERIT_ACE)) {
	newflags &= ~ALL_INHERIT;
	aclp->z_ops->ace_flags_set(acep,
	newflags\|ACE_INHERITED_ACE);
	continue;
	}

	/*
	* This directory has an inheritable ACE
	*/
	aclp->z_hints \|= ZFS_INHERIT_ACE;

	/*
	* If only FILE_INHERIT is set then turn on
	* inherit_only
	*/
	if ((iflags & (ACE_FILE_INHERIT_ACE \|
	ACE_DIRECTORY_INHERIT_ACE)) == ACE_FILE_INHERIT_ACE) {
	newflags \|= ACE_INHERIT_ONLY_ACE;
	aclp->z_ops->ace_flags_set(acep,
	newflags\|ACE_INHERITED_ACE);
	} else {
	newflags &= ~ACE_INHERIT_ONLY_ACE;
	aclp->z_ops->ace_flags_set(acep,
	newflags\|ACE_INHERITED_ACE);
	}
	}
	if (zfsvfs->z_acl_mode == ZFS_ACL_RESTRICTED &&
	aclp->z_acl_count != 0) {
	*need_chmod = B_FALSE;
	}

	return (aclp);
	}

	/*
	* Create file system object initial permissions
	* including inheritable ACEs.
	* Also, create FUIDs for owner and group.
	*/
	int
	zfs_acl_ids_create(znode_t dzp, int flag, vattr_t vap, cred_t *cr,
	vsecattr_t vsecp, zfs_acl_ids_t acl_ids)
	{
	int error;
	zfsvfs_t *zfsvfs = ZTOZSB(dzp);
	zfs_acl_t *paclp;
	gid_t gid = vap->va_gid;
	boolean_t need_chmod = B_TRUE;
	boolean_t trim = B_FALSE;
	boolean_t inherited = B_FALSE;

	bzero(acl_ids, sizeof (zfs_acl_ids_t));
	acl_ids->z_mode = vap->va_mode;

	if (vsecp)
	if ((error = zfs_vsec_2_aclp(zfsvfs, vap->va_mode, vsecp,
	cr, &acl_ids->z_fuidp, &acl_ids->z_aclp)) != 0)
	return (error);

	acl_ids->z_fuid = vap->va_uid;
	acl_ids->z_fgid = vap->va_gid;
	#ifdef HAVE_KSID
	/*
	* Determine uid and gid.
	*/
	if ((flag & IS_ROOT_NODE) \|\| zfsvfs->z_replay \|\|
	((flag & IS_XATTR) && (S_ISDIR(vap->va_mode)))) {
	acl_ids->z_fuid = zfs_fuid_create(zfsvfs, (uint64_t)vap->va_uid,
	cr, ZFS_OWNER, &acl_ids->z_fuidp);
	acl_ids->z_fgid = zfs_fuid_create(zfsvfs, (uint64_t)vap->va_gid,
	cr, ZFS_GROUP, &acl_ids->z_fuidp);
	gid = vap->va_gid;
	} else {
	acl_ids->z_fuid = zfs_fuid_create_cred(zfsvfs, ZFS_OWNER,
	cr, &acl_ids->z_fuidp);
	acl_ids->z_fgid = 0;
	if (vap->va_mask & AT_GID) {
	acl_ids->z_fgid = zfs_fuid_create(zfsvfs,
	(uint64_t)vap->va_gid,
	cr, ZFS_GROUP, &acl_ids->z_fuidp);
	gid = vap->va_gid;
	if (acl_ids->z_fgid != KGID_TO_SGID(ZTOI(dzp)->i_gid) &&
	!groupmember(vap->va_gid, cr) &&
	secpolicy_vnode_create_gid(cr) != 0)
	acl_ids->z_fgid = 0;
	}
	if (acl_ids->z_fgid == 0) {
	if (dzp->z_mode & S_ISGID) {
	char *domain;
	uint32_t rid;

	acl_ids->z_fgid = KGID_TO_SGID(
	ZTOI(dzp)->i_gid);
	gid = zfs_fuid_map_id(zfsvfs, acl_ids->z_fgid,
	cr, ZFS_GROUP);

	if (zfsvfs->z_use_fuids &&
	IS_EPHEMERAL(acl_ids->z_fgid)) {
	domain = zfs_fuid_idx_domain(
	&zfsvfs->z_fuid_idx,
	FUID_INDEX(acl_ids->z_fgid));
	rid = FUID_RID(acl_ids->z_fgid);
	zfs_fuid_node_add(&acl_ids->z_fuidp,
	domain, rid,
	FUID_INDEX(acl_ids->z_fgid),
	acl_ids->z_fgid, ZFS_GROUP);
	}
	} else {
	acl_ids->z_fgid = zfs_fuid_create_cred(zfsvfs,
	ZFS_GROUP, cr, &acl_ids->z_fuidp);
	gid = crgetgid(cr);
	}
	}
	}
	#endif /* HAVE_KSID */

	/*
	* If we're creating a directory, and the parent directory has the
	* set-GID bit set, set in on the new directory.
	* Otherwise, if the user is neither privileged nor a member of the
	* file's new group, clear the file's set-GID bit.
	*/

	if (!(flag & IS_ROOT_NODE) && (dzp->z_mode & S_ISGID) &&
	(S_ISDIR(vap->va_mode))) {
	acl_ids->z_mode \|= S_ISGID;
	} else {
	if ((acl_ids->z_mode & S_ISGID) &&
	secpolicy_vnode_setids_setgids(cr, gid) != 0)
	acl_ids->z_mode &= ~S_ISGID;
	}

	if (acl_ids->z_aclp == NULL) {
	mutex_enter(&dzp->z_acl_lock);
	mutex_enter(&dzp->z_lock);
	if (!(flag & IS_ROOT_NODE) &&
	(dzp->z_pflags & ZFS_INHERIT_ACE) &&
	!(dzp->z_pflags & ZFS_XATTR)) {
	VERIFY(0 == zfs_acl_node_read(dzp, B_TRUE,
	&paclp, B_FALSE));
	acl_ids->z_aclp = zfs_acl_inherit(zfsvfs,
	vap->va_mode, paclp, acl_ids->z_mode, &need_chmod);
	inherited = B_TRUE;
	} else {
	acl_ids->z_aclp =
	zfs_acl_alloc(zfs_acl_version_zp(dzp));
	acl_ids->z_aclp->z_hints \|= ZFS_ACL_TRIVIAL;
	}
	mutex_exit(&dzp->z_lock);
	mutex_exit(&dzp->z_acl_lock);

	if (need_chmod) {
	if (S_ISDIR(vap->va_mode))
	acl_ids->z_aclp->z_hints \|=
	ZFS_ACL_AUTO_INHERIT;

	if (zfsvfs->z_acl_mode == ZFS_ACL_GROUPMASK &&
	zfsvfs->z_acl_inherit != ZFS_ACL_PASSTHROUGH &&
	zfsvfs->z_acl_inherit != ZFS_ACL_PASSTHROUGH_X)
	trim = B_TRUE;
	zfs_acl_chmod(vap->va_mode, acl_ids->z_mode, B_FALSE,
	trim, acl_ids->z_aclp);
	}
	}

	if (inherited \|\| vsecp) {
	acl_ids->z_mode = zfs_mode_compute(acl_ids->z_mode,
	acl_ids->z_aclp, &acl_ids->z_aclp->z_hints,
	acl_ids->z_fuid, acl_ids->z_fgid);
	if (ace_trivial_common(acl_ids->z_aclp, 0, zfs_ace_walk) == 0)
	acl_ids->z_aclp->z_hints \|= ZFS_ACL_TRIVIAL;
	}

	return (0);
	}

	/*
	* Free ACL and fuid_infop, but not the acl_ids structure
	*/
	void
	zfs_acl_ids_free(zfs_acl_ids_t *acl_ids)
	{
	if (acl_ids->z_aclp)
	zfs_acl_free(acl_ids->z_aclp);
	if (acl_ids->z_fuidp)
	zfs_fuid_info_free(acl_ids->z_fuidp);
	acl_ids->z_aclp = NULL;
	acl_ids->z_fuidp = NULL;
	}

	boolean_t
	zfs_acl_ids_overquota(zfsvfs_t zv, zfs_acl_ids_t acl_ids, uint64_t projid)
	{
	return (zfs_id_overquota(zv, DMU_USERUSED_OBJECT, acl_ids->z_fuid) \|\|
	zfs_id_overquota(zv, DMU_GROUPUSED_OBJECT, acl_ids->z_fgid) \|\|
	(projid != ZFS_DEFAULT_PROJID && projid != ZFS_INVALID_PROJID &&
	zfs_id_overquota(zv, DMU_PROJECTUSED_OBJECT, projid)));
	}

	/*
	* Retrieve a file's ACL
	*/
	int
	zfs_getacl(znode_t zp, vsecattr_t vsecp, boolean_t skipaclchk, cred_t *cr)
	{
	zfs_acl_t *aclp;
	ulong_t mask;
	int error;
	int count = 0;
	int largeace = 0;

	mask = vsecp->vsa_mask & (VSA_ACE \| VSA_ACECNT \|
	VSA_ACE_ACLFLAGS \| VSA_ACE_ALLTYPES);

	if (mask == 0)
	return (SET_ERROR(ENOSYS));

	if ((error = zfs_zaccess(zp, ACE_READ_ACL, 0, skipaclchk, cr)))
	return (error);

	mutex_enter(&zp->z_acl_lock);

	error = zfs_acl_node_read(zp, B_FALSE, &aclp, B_FALSE);
	if (error != 0) {
	mutex_exit(&zp->z_acl_lock);
	return (error);
	}

	/*
	* Scan ACL to determine number of ACEs
	*/
	if ((zp->z_pflags & ZFS_ACL_OBJ_ACE) && !(mask & VSA_ACE_ALLTYPES)) {
	void *zacep = NULL;
	uint64_t who;
	uint32_t access_mask;
	uint16_t type, iflags;

	while ((zacep = zfs_acl_next_ace(aclp, zacep,
	&who, &access_mask, &iflags, &type))) {
	switch (type) {
	case ACE_ACCESS_ALLOWED_OBJECT_ACE_TYPE:
	case ACE_ACCESS_DENIED_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_AUDIT_OBJECT_ACE_TYPE:
	case ACE_SYSTEM_ALARM_OBJECT_ACE_TYPE:
	largeace++;
	continue;
	default:
	count++;
	}
	}
	vsecp->vsa_aclcnt = count;
	} else
	count = (int)aclp->z_acl_count;

	if (mask & VSA_ACECNT) {
	vsecp->vsa_aclcnt = count;
	}

	if (mask & VSA_ACE) {
	size_t aclsz;

	aclsz = count * sizeof (ace_t) +
	sizeof (ace_object_t) * largeace;

	vsecp->vsa_aclentp = kmem_alloc(aclsz, KM_SLEEP);
	vsecp->vsa_aclentsz = aclsz;

	if (aclp->z_version == ZFS_ACL_VERSION_FUID)
	zfs_copy_fuid_2_ace(ZTOZSB(zp), aclp, cr,
	vsecp->vsa_aclentp, !(mask & VSA_ACE_ALLTYPES));
	else {
	zfs_acl_node_t *aclnode;
	void *start = vsecp->vsa_aclentp;

	for (aclnode = list_head(&aclp->z_acl); aclnode;
	aclnode = list_next(&aclp->z_acl, aclnode)) {
	bcopy(aclnode->z_acldata, start,
	aclnode->z_size);
	start = (caddr_t)start + aclnode->z_size;
	}
	ASSERT((caddr_t)start - (caddr_t)vsecp->vsa_aclentp ==
	aclp->z_acl_bytes);
	}
	}
	if (mask & VSA_ACE_ACLFLAGS) {
	vsecp->vsa_aclflags = 0;
	if (zp->z_pflags & ZFS_ACL_DEFAULTED)
	vsecp->vsa_aclflags \|= ACL_DEFAULTED;
	if (zp->z_pflags & ZFS_ACL_PROTECTED)
	vsecp->vsa_aclflags \|= ACL_PROTECTED;
	if (zp->z_pflags & ZFS_ACL_AUTO_INHERIT)
	vsecp->vsa_aclflags \|= ACL_AUTO_INHERIT;
	}

	mutex_exit(&zp->z_acl_lock);

	return (0);
	}

	int
	zfs_vsec_2_aclp(zfsvfs_t *zfsvfs, umode_t obj_mode,
	vsecattr_t vsecp, cred_t cr, zfs_fuid_info_t fuidp, zfs_acl_t zaclp)
	{
	zfs_acl_t *aclp;
	zfs_acl_node_t *aclnode;
	int aclcnt = vsecp->vsa_aclcnt;
	int error;

	if (vsecp->vsa_aclcnt > MAX_ACL_ENTRIES \|\| vsecp->vsa_aclcnt <= 0)
	return (SET_ERROR(EINVAL));

	aclp = zfs_acl_alloc(zfs_acl_version(zfsvfs->z_version));

	aclp->z_hints = 0;
	aclnode = zfs_acl_node_alloc(aclcnt * sizeof (zfs_object_ace_t));
	if (aclp->z_version == ZFS_ACL_VERSION_INITIAL) {
	if ((error = zfs_copy_ace_2_oldace(obj_mode, aclp,
	(ace_t *)vsecp->vsa_aclentp, aclnode->z_acldata,
	aclcnt, &aclnode->z_size)) != 0) {
	zfs_acl_free(aclp);
	zfs_acl_node_free(aclnode);
	return (error);
	}
	} else {
	if ((error = zfs_copy_ace_2_fuid(zfsvfs, obj_mode, aclp,
	vsecp->vsa_aclentp, aclnode->z_acldata, aclcnt,
	&aclnode->z_size, fuidp, cr)) != 0) {
	zfs_acl_free(aclp);
	zfs_acl_node_free(aclnode);
	return (error);
	}
	}
	aclp->z_acl_bytes = aclnode->z_size;
	aclnode->z_ace_count = aclcnt;
	aclp->z_acl_count = aclcnt;
	list_insert_head(&aclp->z_acl, aclnode);

	/*
	* If flags are being set then add them to z_hints
	*/
	if (vsecp->vsa_mask & VSA_ACE_ACLFLAGS) {
	if (vsecp->vsa_aclflags & ACL_PROTECTED)
	aclp->z_hints \|= ZFS_ACL_PROTECTED;
	if (vsecp->vsa_aclflags & ACL_DEFAULTED)
	aclp->z_hints \|= ZFS_ACL_DEFAULTED;
	if (vsecp->vsa_aclflags & ACL_AUTO_INHERIT)
	aclp->z_hints \|= ZFS_ACL_AUTO_INHERIT;
	}

	*zaclp = aclp;

	return (0);
	}

	/*
	* Set a file's ACL
	*/
	int
	zfs_setacl(znode_t zp, vsecattr_t vsecp, boolean_t skipaclchk, cred_t *cr)
	{
	zfsvfs_t *zfsvfs = ZTOZSB(zp);
	zilog_t *zilog = zfsvfs->z_log;
	ulong_t mask = vsecp->vsa_mask & (VSA_ACE \| VSA_ACECNT);
	dmu_tx_t *tx;
	int error;
	zfs_acl_t *aclp;
	zfs_fuid_info_t *fuidp = NULL;
	boolean_t fuid_dirtied;
	uint64_t acl_obj;

	if (mask == 0)
	return (SET_ERROR(ENOSYS));

	if (zp->z_pflags & ZFS_IMMUTABLE)
	return (SET_ERROR(EPERM));

	if ((error = zfs_zaccess(zp, ACE_WRITE_ACL, 0, skipaclchk, cr)))
	return (error);

	error = zfs_vsec_2_aclp(zfsvfs, ZTOI(zp)->i_mode, vsecp, cr, &fuidp,
	&aclp);
	if (error)
	return (error);

	/*
	* If ACL wide flags aren't being set then preserve any
	* existing flags.
	*/
	if (!(vsecp->vsa_mask & VSA_ACE_ACLFLAGS)) {
	aclp->z_hints \|=
	(zp->z_pflags & V4_ACL_WIDE_FLAGS);
	}
	top:
	mutex_enter(&zp->z_acl_lock);
	mutex_enter(&zp->z_lock);

	tx = dmu_tx_create(zfsvfs->z_os);

	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_TRUE);

	fuid_dirtied = zfsvfs->z_fuid_dirty;
	if (fuid_dirtied)
	zfs_fuid_txhold(zfsvfs, tx);

	/*
	* If old version and ACL won't fit in bonus and we aren't
	* upgrading then take out necessary DMU holds
	*/

	if ((acl_obj = zfs_external_acl(zp)) != 0) {
	if (zfsvfs->z_version >= ZPL_VERSION_FUID &&
	zfs_znode_acl_version(zp) <= ZFS_ACL_VERSION_INITIAL) {
	dmu_tx_hold_free(tx, acl_obj, 0,
	DMU_OBJECT_END);
	dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0,
	aclp->z_acl_bytes);
	} else {
	dmu_tx_hold_write(tx, acl_obj, 0, aclp->z_acl_bytes);
	}
	} else if (!zp->z_is_sa && aclp->z_acl_bytes > ZFS_ACE_SPACE) {
	dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0, aclp->z_acl_bytes);
	}

	zfs_sa_upgrade_txholds(tx, zp);
	error = dmu_tx_assign(tx, TXG_NOWAIT);
	if (error) {
	mutex_exit(&zp->z_acl_lock);
	mutex_exit(&zp->z_lock);

	if (error == ERESTART) {
	dmu_tx_wait(tx);
	dmu_tx_abort(tx);
	goto top;
	}
	dmu_tx_abort(tx);
	zfs_acl_free(aclp);
	return (error);
	}

	error = zfs_aclset_common(zp, aclp, cr, tx);
	ASSERT(error == 0);
	ASSERT(zp->z_acl_cached == NULL);
	zp->z_acl_cached = aclp;

	if (fuid_dirtied)
	zfs_fuid_sync(zfsvfs, tx);

	zfs_log_acl(zilog, tx, zp, vsecp, fuidp);

	if (fuidp)
	zfs_fuid_info_free(fuidp);
	dmu_tx_commit(tx);

	mutex_exit(&zp->z_lock);
	mutex_exit(&zp->z_acl_lock);

	return (error);
	}

	/*
	* Check accesses of interest (AoI) against attributes of the dataset
	* such as read-only. Returns zero if no AoI conflict with dataset
	* attributes, otherwise an appropriate errno is returned.
	*/
	static int
	zfs_zaccess_dataset_check(znode_t *zp, uint32_t v4_mode)
	{
	if ((v4_mode & WRITE_MASK) && (zfs_is_readonly(ZTOZSB(zp))) &&
	(!Z_ISDEV(ZTOI(zp)->i_mode) \|\|
	(Z_ISDEV(ZTOI(zp)->i_mode) && (v4_mode & WRITE_MASK_ATTRS)))) {
	return (SET_ERROR(EROFS));
	}

	/*
	* Only check for READONLY on non-directories.
	*/
	if ((v4_mode & WRITE_MASK_DATA) &&
	((!S_ISDIR(ZTOI(zp)->i_mode) &&
	(zp->z_pflags & (ZFS_READONLY \| ZFS_IMMUTABLE))) \|\|
	(S_ISDIR(ZTOI(zp)->i_mode) &&
	(zp->z_pflags & ZFS_IMMUTABLE)))) {
	return (SET_ERROR(EPERM));
	}

	if ((v4_mode & (ACE_DELETE \| ACE_DELETE_CHILD)) &&
	(zp->z_pflags & ZFS_NOUNLINK)) {
	return (SET_ERROR(EPERM));
	}

	if (((v4_mode & (ACE_READ_DATA\|ACE_EXECUTE)) &&
	(zp->z_pflags & ZFS_AV_QUARANTINED))) {
	return (SET_ERROR(EACCES));
	}

	return (0);
	}

	/*
	* The primary usage of this function is to loop through all of the
	* ACEs in the znode, determining what accesses of interest (AoI) to
	* the caller are allowed or denied. The AoI are expressed as bits in
	* the working_mode parameter. As each ACE is processed, bits covered
	* by that ACE are removed from the working_mode. This removal
	* facilitates two things. The first is that when the working mode is
	* empty (= 0), we know we've looked at all the AoI. The second is
	* that the ACE interpretation rules don't allow a later ACE to undo
	* something granted or denied by an earlier ACE. Removing the
	* discovered access or denial enforces this rule. At the end of
	* processing the ACEs, all AoI that were found to be denied are
	* placed into the working_mode, giving the caller a mask of denied
	* accesses. Returns:
	* 0 if all AoI granted
	* EACCES if the denied mask is non-zero
	* other error if abnormal failure (e.g., IO error)
	*
	* A secondary usage of the function is to determine if any of the
	* AoI are granted. If an ACE grants any access in
	* the working_mode, we immediately short circuit out of the function.
	* This mode is chosen by setting anyaccess to B_TRUE. The
	* working_mode is not a denied access mask upon exit if the function
	* is used in this manner.
	*/
	static int
	zfs_zaccess_aces_check(znode_t zp, uint32_t working_mode,
	boolean_t anyaccess, cred_t *cr)
	{
	zfsvfs_t *zfsvfs = ZTOZSB(zp);
	zfs_acl_t *aclp;
	int error;
	uid_t uid = crgetuid(cr);
	uint64_t who;
	uint16_t type, iflags;
	uint16_t entry_type;
	uint32_t access_mask;
	uint32_t deny_mask = 0;
	zfs_ace_hdr_t *acep = NULL;
	boolean_t checkit;
	uid_t gowner;
	uid_t fowner;

	zfs_fuid_map_ids(zp, cr, &fowner, &gowner);

	mutex_enter(&zp->z_acl_lock);

	error = zfs_acl_node_read(zp, B_FALSE, &aclp, B_FALSE);
	if (error != 0) {
	mutex_exit(&zp->z_acl_lock);
	return (error);
	}

	ASSERT(zp->z_acl_cached);

	while ((acep = zfs_acl_next_ace(aclp, acep, &who, &access_mask,
	&iflags, &type))) {
	uint32_t mask_matched;

	if (!zfs_acl_valid_ace_type(type, iflags))
	continue;

	if (S_ISDIR(ZTOI(zp)->i_mode) &&
	(iflags & ACE_INHERIT_ONLY_ACE))
	continue;

	/* Skip ACE if it does not affect any AoI */
	mask_matched = (access_mask & *working_mode);
	if (!mask_matched)
	continue;

	entry_type = (iflags & ACE_TYPE_FLAGS);

	checkit = B_FALSE;

	switch (entry_type) {
	case ACE_OWNER:
	if (uid == fowner)
	checkit = B_TRUE;
	break;
	case OWNING_GROUP:
	who = gowner;
	/FALLTHROUGH/
	case ACE_IDENTIFIER_GROUP:
	checkit = zfs_groupmember(zfsvfs, who, cr);
	break;
	case ACE_EVERYONE:
	checkit = B_TRUE;
	break;

	/* USER Entry */
	default:
	if (entry_type == 0) {
	uid_t newid;

	newid = zfs_fuid_map_id(zfsvfs, who, cr,
	ZFS_ACE_USER);
	if (newid != IDMAP_WK_CREATOR_OWNER_UID &&
	uid == newid)
	checkit = B_TRUE;
	break;
	} else {
	mutex_exit(&zp->z_acl_lock);
	return (SET_ERROR(EIO));
	}
	}

	if (checkit) {
	if (type == DENY) {
	DTRACE_PROBE3(zfs__ace__denies,
	znode_t *, zp,
	zfs_ace_hdr_t *, acep,
	uint32_t, mask_matched);
	deny_mask \|= mask_matched;
	} else {
	DTRACE_PROBE3(zfs__ace__allows,
	znode_t *, zp,
	zfs_ace_hdr_t *, acep,
	uint32_t, mask_matched);
	if (anyaccess) {
	mutex_exit(&zp->z_acl_lock);
	return (0);
	}
	}
	*working_mode &= ~mask_matched;
	}

	/* Are we done? */
	if (*working_mode == 0)
	break;
	}

	mutex_exit(&zp->z_acl_lock);

	/* Put the found 'denies' back on the working mode */
	if (deny_mask) {
	*working_mode \|= deny_mask;
	return (SET_ERROR(EACCES));
	} else if (*working_mode) {
	return (-1);
	}

	return (0);
	}

	/*
	* Return true if any access whatsoever granted, we don't actually
	* care what access is granted.
	*/
	boolean_t
	zfs_has_access(znode_t zp, cred_t cr)
	{
	uint32_t have = ACE_ALL_PERMS;

	if (zfs_zaccess_aces_check(zp, &have, B_TRUE, cr) != 0) {
	uid_t owner;

	owner = zfs_fuid_map_id(ZTOZSB(zp),
	KUID_TO_SUID(ZTOI(zp)->i_uid), cr, ZFS_OWNER);
	return (secpolicy_vnode_any_access(cr, ZTOI(zp), owner) == 0);
	}
	return (B_TRUE);
	}

	static int
	zfs_zaccess_common(znode_t zp, uint32_t v4_mode, uint32_t working_mode,
	boolean_t check_privs, boolean_t skipaclchk, cred_t cr)
	{
	zfsvfs_t *zfsvfs = ZTOZSB(zp);
	int err;

	*working_mode = v4_mode;
	*check_privs = B_TRUE;

	/*
	* Short circuit empty requests
	*/
	if (v4_mode == 0 \|\| zfsvfs->z_replay) {
	*working_mode = 0;
	return (0);
	}

	if ((err = zfs_zaccess_dataset_check(zp, v4_mode)) != 0) {
	*check_privs = B_FALSE;
	return (err);
	}

	/*
	* The caller requested that the ACL check be skipped. This
	* would only happen if the caller checked VOP_ACCESS() with a
	* 32 bit ACE mask and already had the appropriate permissions.
	*/
	if (skipaclchk) {
	*working_mode = 0;
	return (0);
	}

	return (zfs_zaccess_aces_check(zp, working_mode, B_FALSE, cr));
	}

	static int
	zfs_zaccess_append(znode_t zp, uint32_t working_mode, boolean_t *check_privs,
	cred_t *cr)
	{
	if (*working_mode != ACE_WRITE_DATA)
	return (SET_ERROR(EACCES));

	return (zfs_zaccess_common(zp, ACE_APPEND_DATA, working_mode,
	check_privs, B_FALSE, cr));
	}

	int
	zfs_fastaccesschk_execute(znode_t zdp, cred_t cr)
	{
	boolean_t owner = B_FALSE;
	boolean_t groupmbr = B_FALSE;
	boolean_t is_attr;
	uid_t uid = crgetuid(cr);
	int error;

	if (zdp->z_pflags & ZFS_AV_QUARANTINED)
	return (SET_ERROR(EACCES));

	is_attr = ((zdp->z_pflags & ZFS_XATTR) &&
	(S_ISDIR(ZTOI(zdp)->i_mode)));
	if (is_attr)
	goto slow;


	mutex_enter(&zdp->z_acl_lock);

	if (zdp->z_pflags & ZFS_NO_EXECS_DENIED) {
	mutex_exit(&zdp->z_acl_lock);
	return (0);
	}

	if (KUID_TO_SUID(ZTOI(zdp)->i_uid) != 0 \|\|
	KGID_TO_SGID(ZTOI(zdp)->i_gid) != 0) {
	mutex_exit(&zdp->z_acl_lock);
	goto slow;
	}

	if (uid == KUID_TO_SUID(ZTOI(zdp)->i_uid)) {
	owner = B_TRUE;
	if (zdp->z_mode & S_IXUSR) {
	mutex_exit(&zdp->z_acl_lock);
	return (0);
	} else {
	mutex_exit(&zdp->z_acl_lock);
	goto slow;
	}
	}
	if (groupmember(KGID_TO_SGID(ZTOI(zdp)->i_gid), cr)) {
	groupmbr = B_TRUE;
	if (zdp->z_mode & S_IXGRP) {
	mutex_exit(&zdp->z_acl_lock);
	return (0);
	} else {
	mutex_exit(&zdp->z_acl_lock);
	goto slow;
	}
	}
	if (!owner && !groupmbr) {
	if (zdp->z_mode & S_IXOTH) {
	mutex_exit(&zdp->z_acl_lock);
	return (0);
	}
	}

	mutex_exit(&zdp->z_acl_lock);

	slow:
	DTRACE_PROBE(zfs__fastpath__execute__access__miss);
	ZFS_ENTER(ZTOZSB(zdp));
	error = zfs_zaccess(zdp, ACE_EXECUTE, 0, B_FALSE, cr);
	ZFS_EXIT(ZTOZSB(zdp));
	return (error);
	}

	/*
	* Determine whether Access should be granted/denied.
	*
	* The least priv subsystem is always consulted as a basic privilege
	* can define any form of access.
	*/
	int
	zfs_zaccess(znode_t zp, int mode, int flags, boolean_t skipaclchk, cred_t cr)
	{
	uint32_t working_mode;
	int error;
	int is_attr;
	boolean_t check_privs;
	znode_t *xzp;
	znode_t *check_zp = zp;
	mode_t needed_bits;
	uid_t owner;

	is_attr = ((zp->z_pflags & ZFS_XATTR) && S_ISDIR(ZTOI(zp)->i_mode));

	/*
	* If attribute then validate against base file
	*/
	if (is_attr) {
	if ((error = zfs_zget(ZTOZSB(zp),
	zp->z_xattr_parent, &xzp)) != 0) {
	return (error);
	}

	check_zp = xzp;

	/*
	* fixup mode to map to xattr perms
	*/

	if (mode & (ACE_WRITE_DATA\|ACE_APPEND_DATA)) {
	mode &= ~(ACE_WRITE_DATA\|ACE_APPEND_DATA);
	mode \|= ACE_WRITE_NAMED_ATTRS;
	}

	if (mode & (ACE_READ_DATA\|ACE_EXECUTE)) {
	mode &= ~(ACE_READ_DATA\|ACE_EXECUTE);
	mode \|= ACE_READ_NAMED_ATTRS;
	}
	}

	owner = zfs_fuid_map_id(ZTOZSB(zp), KUID_TO_SUID(ZTOI(zp)->i_uid),
	cr, ZFS_OWNER);
	/*
	* Map the bits required to the standard inode flags
	* S_IRUSR\|S_IWUSR\|S_IXUSR in the needed_bits. Map the bits
	* mapped by working_mode (currently missing) in missing_bits.
	* Call secpolicy_vnode_access2() with (needed_bits & ~checkmode),
	* needed_bits.
	*/
	needed_bits = 0;

	working_mode = mode;
	if ((working_mode & (ACE_READ_ACL\|ACE_READ_ATTRIBUTES)) &&
	owner == crgetuid(cr))
	working_mode &= ~(ACE_READ_ACL\|ACE_READ_ATTRIBUTES);

	if (working_mode & (ACE_READ_DATA\|ACE_READ_NAMED_ATTRS\|
	ACE_READ_ACL\|ACE_READ_ATTRIBUTES\|ACE_SYNCHRONIZE))
	needed_bits \|= S_IRUSR;
	if (working_mode & (ACE_WRITE_DATA\|ACE_WRITE_NAMED_ATTRS\|
	ACE_APPEND_DATA\|ACE_WRITE_ATTRIBUTES\|ACE_SYNCHRONIZE))
	needed_bits \|= S_IWUSR;
	if (working_mode & ACE_EXECUTE)
	needed_bits \|= S_IXUSR;

	if ((error = zfs_zaccess_common(check_zp, mode, &working_mode,
	&check_privs, skipaclchk, cr)) == 0) {
	if (is_attr)
	zrele(xzp);
	return (secpolicy_vnode_access2(cr, ZTOI(zp), owner,
	needed_bits, needed_bits));
	}

	if (error && !check_privs) {
	if (is_attr)
	zrele(xzp);
	return (error);
	}

	if (error && (flags & V_APPEND)) {
	error = zfs_zaccess_append(zp, &working_mode, &check_privs, cr);
	}

	if (error && check_privs) {
	mode_t checkmode = 0;

	/*
	* First check for implicit owner permission on
	* read_acl/read_attributes
	*/

	error = 0;
	ASSERT(working_mode != 0);

	if ((working_mode & (ACE_READ_ACL\|ACE_READ_ATTRIBUTES) &&
	owner == crgetuid(cr)))
	working_mode &= ~(ACE_READ_ACL\|ACE_READ_ATTRIBUTES);

	if (working_mode & (ACE_READ_DATA\|ACE_READ_NAMED_ATTRS\|
	ACE_READ_ACL\|ACE_READ_ATTRIBUTES\|ACE_SYNCHRONIZE))
	checkmode \|= S_IRUSR;
	if (working_mode & (ACE_WRITE_DATA\|ACE_WRITE_NAMED_ATTRS\|
	ACE_APPEND_DATA\|ACE_WRITE_ATTRIBUTES\|ACE_SYNCHRONIZE))
	checkmode \|= S_IWUSR;
	if (working_mode & ACE_EXECUTE)
	checkmode \|= S_IXUSR;

	error = secpolicy_vnode_access2(cr, ZTOI(check_zp), owner,
	needed_bits & ~checkmode, needed_bits);

	if (error == 0 && (working_mode & ACE_WRITE_OWNER))
	error = secpolicy_vnode_chown(cr, owner);
	if (error == 0 && (working_mode & ACE_WRITE_ACL))
	error = secpolicy_vnode_setdac(cr, owner);

	if (error == 0 && (working_mode &
	(ACE_DELETE\|ACE_DELETE_CHILD)))
	error = secpolicy_vnode_remove(cr);

	if (error == 0 && (working_mode & ACE_SYNCHRONIZE)) {
	error = secpolicy_vnode_chown(cr, owner);
	}
	if (error == 0) {
	/*
	* See if any bits other than those already checked
	* for are still present. If so then return EACCES
	*/
	if (working_mode & ~(ZFS_CHECKED_MASKS)) {
	error = SET_ERROR(EACCES);
	}
	}
	} else if (error == 0) {
	error = secpolicy_vnode_access2(cr, ZTOI(zp), owner,
	needed_bits, needed_bits);
	}

	if (is_attr)
	zrele(xzp);

	return (error);
	}

	/*
	* Translate traditional unix S_IRUSR/S_IWUSR/S_IXUSR mode into
	- * native ACL format and call zfs_zaccess()
	+ * NFSv4-style ZFS ACL format and call zfs_zaccess()
	*/
	int
	zfs_zaccess_rwx(znode_t zp, mode_t mode, int flags, cred_t cr)
	{
	return (zfs_zaccess(zp, zfs_unix_to_v4(mode >> 6), flags, B_FALSE, cr));
	}

	/*
	* Access function for secpolicy_vnode_setattr
	*/
	int
	zfs_zaccess_unix(znode_t zp, mode_t mode, cred_t cr)
	{
	int v4_mode = zfs_unix_to_v4(mode >> 6);

	return (zfs_zaccess(zp, v4_mode, 0, B_FALSE, cr));
	}

	/* See zfs_zaccess_delete() */
	int zfs_write_implies_delete_child = 1;

	/*
	* Determine whether delete access should be granted.
	*
	* The following chart outlines how we handle delete permissions which is
	* how recent versions of windows (Windows 2008) handles it. The efficiency
	* comes from not having to check the parent ACL where the object itself grants
	* delete:
	*
	* -------------------------------------------------------
	* \| Parent Dir \| Target Object Permissions \|
	* \| permissions \| \|
	* -------------------------------------------------------
	* \| \| ACL Allows \| ACL Denies\| Delete \|
	* \| \| Delete \| Delete \| unspecified\|
	* -------------------------------------------------------
	* \| ACL Allows \| Permit \| Deny * \| Permit \|
	* \| DELETE_CHILD \| \| \| \|
	* -------------------------------------------------------
	* \| ACL Denies \| Permit \| Deny \| Deny \|
	* \| DELETE_CHILD \| \| \| \|
	* -------------------------------------------------------
	* \| ACL specifies \| \| \| \|
	* \| only allow \| Permit \| Deny * \| Permit \|
	* \| write and \| \| \| \|
	* \| execute \| \| \| \|
	* -------------------------------------------------------
	* \| ACL denies \| \| \| \|
	* \| write and \| Permit \| Deny \| Deny \|
	* \| execute \| \| \| \|
	* -------------------------------------------------------
	* ^
	* \|
	* Re. execute permission on the directory: if that's missing,
	* the vnode lookup of the target will fail before we get here.
	*
	* Re [*] in the table above: NFSv4 would normally Permit delete for
	* these two cells of the matrix.
	* See acl.h for notes on which ACE_... flags should be checked for which
	* operations. Specifically, the NFSv4 committee recommendation is in
	* conflict with the Windows interpretation of DENY ACEs, where DENY ACEs
	* should take precedence ahead of ALLOW ACEs.
	*
	* This implementation always consults the target object's ACL first.
	* If a DENY ACE is present on the target object that specifies ACE_DELETE,
	* delete access is denied. If an ALLOW ACE with ACE_DELETE is present on
	* the target object, access is allowed. If and only if no entries with
	* ACE_DELETE are present in the object's ACL, check the container's ACL
	* for entries with ACE_DELETE_CHILD.
	*
	* A summary of the logic implemented from the table above is as follows:
	*
	* First check for DENY ACEs that apply.
	* If either target or container has a deny, EACCES.
	*
	* Delete access can then be summarized as follows:
	* 1: The object to be deleted grants ACE_DELETE, or
	* 2: The containing directory grants ACE_DELETE_CHILD.
	* In a Windows system, that would be the end of the story.
	* In this system, (2) has some complications...
	* 2a: "sticky" bit on a directory adds restrictions, and
	* 2b: existing ACEs from previous versions of ZFS may
	* not carry ACE_DELETE_CHILD where they should, so we
	* also allow delete when ACE_WRITE_DATA is granted.
	*
	* Note: 2b is technically a work-around for a prior bug,
	* which hopefully can go away some day. For those who
	* no longer need the work around, and for testing, this
	* work-around is made conditional via the tunable:
	* zfs_write_implies_delete_child
	*/
	int
	zfs_zaccess_delete(znode_t dzp, znode_t zp, cred_t *cr)
	{
	uint32_t wanted_dirperms;
	uint32_t dzp_working_mode = 0;
	uint32_t zp_working_mode = 0;
	int dzp_error, zp_error;
	boolean_t dzpcheck_privs;
	boolean_t zpcheck_privs;

	if (zp->z_pflags & (ZFS_IMMUTABLE \| ZFS_NOUNLINK))
	return (SET_ERROR(EPERM));

	/*
	* Case 1:
	* If target object grants ACE_DELETE then we are done. This is
	* indicated by a return value of 0. For this case we don't worry
	* about the sticky bit because sticky only applies to the parent
	* directory and this is the child access result.
	*
	* If we encounter a DENY ACE here, we're also done (EACCES).
	* Note that if we hit a DENY ACE here (on the target) it should
	* take precedence over a DENY ACE on the container, so that when
	* we have more complete auditing support we will be able to
	* report an access failure against the specific target.
	* (This is part of why we're checking the target first.)
	*/
	zp_error = zfs_zaccess_common(zp, ACE_DELETE, &zp_working_mode,
	&zpcheck_privs, B_FALSE, cr);
	if (zp_error == EACCES) {
	/* We hit a DENY ACE. */
	if (!zpcheck_privs)
	return (SET_ERROR(zp_error));
	return (secpolicy_vnode_remove(cr));

	}
	if (zp_error == 0)
	return (0);

	/*
	* Case 2:
	* If the containing directory grants ACE_DELETE_CHILD,
	* or we're in backward compatibility mode and the
	* containing directory has ACE_WRITE_DATA, allow.
	* Case 2b is handled with wanted_dirperms.
	*/
	wanted_dirperms = ACE_DELETE_CHILD;
	if (zfs_write_implies_delete_child)
	wanted_dirperms \|= ACE_WRITE_DATA;
	dzp_error = zfs_zaccess_common(dzp, wanted_dirperms,
	&dzp_working_mode, &dzpcheck_privs, B_FALSE, cr);
	if (dzp_error == EACCES) {
	/* We hit a DENY ACE. */
	if (!dzpcheck_privs)
	return (SET_ERROR(dzp_error));
	return (secpolicy_vnode_remove(cr));
	}

	/*
	* Cases 2a, 2b (continued)
	*
	* Note: dzp_working_mode now contains any permissions
	* that were NOT granted. Therefore, if any of the
	* wanted_dirperms WERE granted, we will have:
	* dzp_working_mode != wanted_dirperms
	* We're really asking if ANY of those permissions
	* were granted, and if so, grant delete access.
	*/
	if (dzp_working_mode != wanted_dirperms)
	dzp_error = 0;

	/*
	* dzp_error is 0 if the container granted us permissions to "modify".
	* If we do not have permission via one or more ACEs, our current
	* privileges may still permit us to modify the container.
	*
	* dzpcheck_privs is false when i.e. the FS is read-only.
	* Otherwise, do privilege checks for the container.
	*/
	if (dzp_error != 0 && dzpcheck_privs) {
	uid_t owner;

	/*
	* The secpolicy call needs the requested access and
	* the current access mode of the container, but it
	* only knows about Unix-style modes (VEXEC, VWRITE),
	* so this must condense the fine-grained ACE bits into
	* Unix modes.
	*
	* The VEXEC flag is easy, because we know that has
	* always been checked before we get here (during the
	* lookup of the target vnode). The container has not
	* granted us permissions to "modify", so we do not set
	* the VWRITE flag in the current access mode.
	*/
	owner = zfs_fuid_map_id(ZTOZSB(dzp),
	KUID_TO_SUID(ZTOI(dzp)->i_uid), cr, ZFS_OWNER);
	dzp_error = secpolicy_vnode_access2(cr, ZTOI(dzp),
	owner, S_IXUSR, S_IWUSR\|S_IXUSR);
	}
	if (dzp_error != 0) {
	/*
	* Note: We may have dzp_error = -1 here (from
	* zfs_zacess_common). Don't return that.
	*/
	return (SET_ERROR(EACCES));
	}


	/*
	* At this point, we know that the directory permissions allow
	* us to modify, but we still need to check for the additional
	* restrictions that apply when the "sticky bit" is set.
	*
	* Yes, zfs_sticky_remove_access() also checks this bit, but
	* checking it here and skipping the call below is nice when
	* you're watching all of this with dtrace.
	*/
	if ((dzp->z_mode & S_ISVTX) == 0)
	return (0);

	/*
	* zfs_sticky_remove_access will succeed if:
	* 1. The sticky bit is absent.
	* 2. We pass the sticky bit restrictions.
	* 3. We have privileges that always allow file removal.
	*/
	return (zfs_sticky_remove_access(dzp, zp, cr));
	}

	int
	zfs_zaccess_rename(znode_t sdzp, znode_t szp, znode_t *tdzp,
	znode_t tzp, cred_t cr)
	{
	int add_perm;
	int error;

	if (szp->z_pflags & ZFS_AV_QUARANTINED)
	return (SET_ERROR(EACCES));

	add_perm = S_ISDIR(ZTOI(szp)->i_mode) ?
	ACE_ADD_SUBDIRECTORY : ACE_ADD_FILE;

	/*
	* Rename permissions are combination of delete permission +
	* add file/subdir permission.
	*/

	/*
	* first make sure we do the delete portion.
	*
	* If that succeeds then check for add_file/add_subdir permissions
	*/

	if ((error = zfs_zaccess_delete(sdzp, szp, cr)))
	return (error);

	/*
	* If we have a tzp, see if we can delete it?
	*/
	if (tzp) {
	if ((error = zfs_zaccess_delete(tdzp, tzp, cr)))
	return (error);
	}

	/*
	* Now check for add permissions
	*/
	error = zfs_zaccess(tdzp, add_perm, 0, B_FALSE, cr);

	return (error);
	}
	Index: vendor-sys/openzfs/dist/module/os/linux/zfs/zfs_ctldir.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/zfs/zfs_ctldir.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/zfs/zfs_ctldir.c (revision 366774)
	@@ -1,1260 +1,1261 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (C) 2011 Lawrence Livermore National Security, LLC.
	* Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
	* LLNL-CODE-403049.
	* Rewritten for Linux by:
	* Rohan Puri <rohan.puri15@gmail.com>
	* Brian Behlendorf <behlendorf1@llnl.gov>
	* Copyright (c) 2013 by Delphix. All rights reserved.
	* Copyright 2015, OmniTI Computer Consulting, Inc. All rights reserved.
	* Copyright (c) 2018 George Melikov. All Rights Reserved.
	* Copyright (c) 2019 Datto, Inc. All rights reserved.
	* Copyright (c) 2020 The MathWorks, Inc. All rights reserved.
	*/

	/*
	* ZFS control directory (a.k.a. ".zfs")
	*
	* This directory provides a common location for all ZFS meta-objects.
	* Currently, this is only the 'snapshot' and 'shares' directory, but this may
	* expand in the future. The elements are built dynamically, as the hierarchy
	* does not actually exist on disk.
	*
	* For 'snapshot', we don't want to have all snapshots always mounted, because
	* this would take up a huge amount of space in /etc/mnttab. We have three
	* types of objects:
	*
	* ctldir ------> snapshotdir -------> snapshot
	* \|
	* \|
	* V
	* mounted fs
	*
	* The 'snapshot' node contains just enough information to lookup '..' and act
	* as a mountpoint for the snapshot. Whenever we lookup a specific snapshot, we
	* perform an automount of the underlying filesystem and return the
	* corresponding inode.
	*
	* All mounts are handled automatically by an user mode helper which invokes
	* the mount procedure. Unmounts are handled by allowing the mount
	* point to expire so the kernel may automatically unmount it.
	*
	* The '.zfs', '.zfs/snapshot', and all directories created under
	* '.zfs/snapshot' (ie: '.zfs/snapshot/<snapname>') all share the same
	* zfsvfs_t as the head filesystem (what '.zfs' lives under).
	*
	* File systems mounted on top of the '.zfs/snapshot/<snapname>' paths
	* (ie: snapshots) are complete ZFS filesystems and have their own unique
	* zfsvfs_t. However, the fsid reported by these mounts will be the same
	* as that used by the parent zfsvfs_t to make NFS happy.
	*/

	#include <sys/types.h>
	#include <sys/param.h>
	#include <sys/time.h>
	#include <sys/sysmacros.h>
	#include <sys/pathname.h>
	#include <sys/vfs.h>
	#include <sys/zfs_ctldir.h>
	#include <sys/zfs_ioctl.h>
	#include <sys/zfs_vfsops.h>
	#include <sys/zfs_vnops.h>
	#include <sys/stat.h>
	#include <sys/dmu.h>
	#include <sys/dmu_objset.h>
	#include <sys/dsl_destroy.h>
	#include <sys/dsl_deleg.h>
	#include <sys/zpl.h>
	#include <sys/mntent.h>
	#include "zfs_namecheck.h"

	/*
	* Two AVL trees are maintained which contain all currently automounted
	* snapshots. Every automounted snapshots maps to a single zfs_snapentry_t
	* entry which MUST:
	*
	* - be attached to both trees, and
	* - be unique, no duplicate entries are allowed.
	*
	* The zfs_snapshots_by_name tree is indexed by the full dataset name
	* while the zfs_snapshots_by_objsetid tree is indexed by the unique
	* objsetid. This allows for fast lookups either by name or objsetid.
	*/
	static avl_tree_t zfs_snapshots_by_name;
	static avl_tree_t zfs_snapshots_by_objsetid;
	static krwlock_t zfs_snapshot_lock;

	/*
	* Control Directory Tunables (.zfs)
	*/
	int zfs_expire_snapshot = ZFSCTL_EXPIRE_SNAPSHOT;
	int zfs_admin_snapshot = 0;

	typedef struct {
	char se_name; / full snapshot name */
	char se_path; / full mount path */
	spa_t se_spa; / pool spa */
	uint64_t se_objsetid; /* snapshot objset id */
	struct dentry se_root_dentry; / snapshot root dentry */
	taskqid_t se_taskqid; /* scheduled unmount taskqid */
	avl_node_t se_node_name; /* zfs_snapshots_by_name link */
	avl_node_t se_node_objsetid; /* zfs_snapshots_by_objsetid link */
	zfs_refcount_t se_refcount; /* reference count */
	} zfs_snapentry_t;

	static void zfsctl_snapshot_unmount_delay_impl(zfs_snapentry_t *se, int delay);

	/*
	* Allocate a new zfs_snapentry_t being careful to make a copy of the
	* the snapshot name and provided mount point. No reference is taken.
	*/
	static zfs_snapentry_t *
	-zfsctl_snapshot_alloc(char full_name, char full_path, spa_t *spa,
	+zfsctl_snapshot_alloc(const char full_name, const char full_path, spa_t *spa,
	uint64_t objsetid, struct dentry *root_dentry)
	{
	zfs_snapentry_t *se;

	se = kmem_zalloc(sizeof (zfs_snapentry_t), KM_SLEEP);

	se->se_name = kmem_strdup(full_name);
	se->se_path = kmem_strdup(full_path);
	se->se_spa = spa;
	se->se_objsetid = objsetid;
	se->se_root_dentry = root_dentry;
	se->se_taskqid = TASKQID_INVALID;

	zfs_refcount_create(&se->se_refcount);

	return (se);
	}

	/*
	* Free a zfs_snapentry_t the caller must ensure there are no active
	* references.
	*/
	static void
	zfsctl_snapshot_free(zfs_snapentry_t *se)
	{
	zfs_refcount_destroy(&se->se_refcount);
	kmem_strfree(se->se_name);
	kmem_strfree(se->se_path);

	kmem_free(se, sizeof (zfs_snapentry_t));
	}

	/*
	* Hold a reference on the zfs_snapentry_t.
	*/
	static void
	zfsctl_snapshot_hold(zfs_snapentry_t *se)
	{
	zfs_refcount_add(&se->se_refcount, NULL);
	}

	/*
	* Release a reference on the zfs_snapentry_t. When the number of
	* references drops to zero the structure will be freed.
	*/
	static void
	zfsctl_snapshot_rele(zfs_snapentry_t *se)
	{
	if (zfs_refcount_remove(&se->se_refcount, NULL) == 0)
	zfsctl_snapshot_free(se);
	}

	/*
	* Add a zfs_snapentry_t to both the zfs_snapshots_by_name and
	* zfs_snapshots_by_objsetid trees. While the zfs_snapentry_t is part
	* of the trees a reference is held.
	*/
	static void
	zfsctl_snapshot_add(zfs_snapentry_t *se)
	{
	ASSERT(RW_WRITE_HELD(&zfs_snapshot_lock));
	zfsctl_snapshot_hold(se);
	avl_add(&zfs_snapshots_by_name, se);
	avl_add(&zfs_snapshots_by_objsetid, se);
	}

	/*
	* Remove a zfs_snapentry_t from both the zfs_snapshots_by_name and
	* zfs_snapshots_by_objsetid trees. Upon removal a reference is dropped,
	* this can result in the structure being freed if that was the last
	* remaining reference.
	*/
	static void
	zfsctl_snapshot_remove(zfs_snapentry_t *se)
	{
	ASSERT(RW_WRITE_HELD(&zfs_snapshot_lock));
	avl_remove(&zfs_snapshots_by_name, se);
	avl_remove(&zfs_snapshots_by_objsetid, se);
	zfsctl_snapshot_rele(se);
	}

	/*
	* Snapshot name comparison function for the zfs_snapshots_by_name.
	*/
	static int
	snapentry_compare_by_name(const void a, const void b)
	{
	const zfs_snapentry_t *se_a = a;
	const zfs_snapentry_t *se_b = b;
	int ret;

	ret = strcmp(se_a->se_name, se_b->se_name);

	if (ret < 0)
	return (-1);
	else if (ret > 0)
	return (1);
	else
	return (0);
	}

	/*
	* Snapshot name comparison function for the zfs_snapshots_by_objsetid.
	*/
	static int
	snapentry_compare_by_objsetid(const void a, const void b)
	{
	const zfs_snapentry_t *se_a = a;
	const zfs_snapentry_t *se_b = b;

	if (se_a->se_spa != se_b->se_spa)
	return ((ulong_t)se_a->se_spa < (ulong_t)se_b->se_spa ? -1 : 1);

	if (se_a->se_objsetid < se_b->se_objsetid)
	return (-1);
	else if (se_a->se_objsetid > se_b->se_objsetid)
	return (1);
	else
	return (0);
	}

	/*
	* Find a zfs_snapentry_t in zfs_snapshots_by_name. If the snapname
	* is found a pointer to the zfs_snapentry_t is returned and a reference
	* taken on the structure. The caller is responsible for dropping the
	* reference with zfsctl_snapshot_rele(). If the snapname is not found
	* NULL will be returned.
	*/
	static zfs_snapentry_t *
	-zfsctl_snapshot_find_by_name(char *snapname)
	+zfsctl_snapshot_find_by_name(const char *snapname)
	{
	zfs_snapentry_t *se, search;

	ASSERT(RW_LOCK_HELD(&zfs_snapshot_lock));

	- search.se_name = snapname;
	+ search.se_name = (char *)snapname;
	se = avl_find(&zfs_snapshots_by_name, &search, NULL);
	if (se)
	zfsctl_snapshot_hold(se);

	return (se);
	}

	/*
	* Find a zfs_snapentry_t in zfs_snapshots_by_objsetid given the objset id
	* rather than the snapname. In all other respects it behaves the same
	* as zfsctl_snapshot_find_by_name().
	*/
	static zfs_snapentry_t *
	zfsctl_snapshot_find_by_objsetid(spa_t *spa, uint64_t objsetid)
	{
	zfs_snapentry_t *se, search;

	ASSERT(RW_LOCK_HELD(&zfs_snapshot_lock));

	search.se_spa = spa;
	search.se_objsetid = objsetid;
	se = avl_find(&zfs_snapshots_by_objsetid, &search, NULL);
	if (se)
	zfsctl_snapshot_hold(se);

	return (se);
	}

	/*
	* Rename a zfs_snapentry_t in the zfs_snapshots_by_name. The structure is
	* removed, renamed, and added back to the new correct location in the tree.
	*/
	static int
	-zfsctl_snapshot_rename(char old_snapname, char new_snapname)
	+zfsctl_snapshot_rename(const char old_snapname, const char new_snapname)
	{
	zfs_snapentry_t *se;

	ASSERT(RW_WRITE_HELD(&zfs_snapshot_lock));

	se = zfsctl_snapshot_find_by_name(old_snapname);
	if (se == NULL)
	return (SET_ERROR(ENOENT));

	zfsctl_snapshot_remove(se);
	kmem_strfree(se->se_name);
	se->se_name = kmem_strdup(new_snapname);
	zfsctl_snapshot_add(se);
	zfsctl_snapshot_rele(se);

	return (0);
	}

	/*
	* Delayed task responsible for unmounting an expired automounted snapshot.
	*/
	static void
	snapentry_expire(void *data)
	{
	zfs_snapentry_t se = (zfs_snapentry_t )data;
	spa_t *spa = se->se_spa;
	uint64_t objsetid = se->se_objsetid;

	if (zfs_expire_snapshot <= 0) {
	zfsctl_snapshot_rele(se);
	return;
	}

	se->se_taskqid = TASKQID_INVALID;
	(void) zfsctl_snapshot_unmount(se->se_name, MNT_EXPIRE);
	zfsctl_snapshot_rele(se);

	/*
	* Reschedule the unmount if the zfs_snapentry_t wasn't removed.
	* This can occur when the snapshot is busy.
	*/
	rw_enter(&zfs_snapshot_lock, RW_READER);
	if ((se = zfsctl_snapshot_find_by_objsetid(spa, objsetid)) != NULL) {
	zfsctl_snapshot_unmount_delay_impl(se, zfs_expire_snapshot);
	zfsctl_snapshot_rele(se);
	}
	rw_exit(&zfs_snapshot_lock);
	}

	/*
	* Cancel an automatic unmount of a snapname. This callback is responsible
	* for dropping the reference on the zfs_snapentry_t which was taken when
	* during dispatch.
	*/
	static void
	zfsctl_snapshot_unmount_cancel(zfs_snapentry_t *se)
	{
	if (taskq_cancel_id(system_delay_taskq, se->se_taskqid) == 0) {
	se->se_taskqid = TASKQID_INVALID;
	zfsctl_snapshot_rele(se);
	}
	}

	/*
	* Dispatch the unmount task for delayed handling with a hold protecting it.
	*/
	static void
	zfsctl_snapshot_unmount_delay_impl(zfs_snapentry_t *se, int delay)
	{
	ASSERT3S(se->se_taskqid, ==, TASKQID_INVALID);

	if (delay <= 0)
	return;

	zfsctl_snapshot_hold(se);
	se->se_taskqid = taskq_dispatch_delay(system_delay_taskq,
	snapentry_expire, se, TQ_SLEEP, ddi_get_lbolt() + delay * HZ);
	}

	/*
	* Schedule an automatic unmount of objset id to occur in delay seconds from
	* now. Any previous delayed unmount will be cancelled in favor of the
	* updated deadline. A reference is taken by zfsctl_snapshot_find_by_name()
	* and held until the outstanding task is handled or cancelled.
	*/
	int
	zfsctl_snapshot_unmount_delay(spa_t *spa, uint64_t objsetid, int delay)
	{
	zfs_snapentry_t *se;
	int error = ENOENT;

	rw_enter(&zfs_snapshot_lock, RW_READER);
	if ((se = zfsctl_snapshot_find_by_objsetid(spa, objsetid)) != NULL) {
	zfsctl_snapshot_unmount_cancel(se);
	zfsctl_snapshot_unmount_delay_impl(se, delay);
	zfsctl_snapshot_rele(se);
	error = 0;
	}
	rw_exit(&zfs_snapshot_lock);

	return (error);
	}

	/*
	* Check if snapname is currently mounted. Returned non-zero when mounted
	* and zero when unmounted.
	*/
	static boolean_t
	-zfsctl_snapshot_ismounted(char *snapname)
	+zfsctl_snapshot_ismounted(const char *snapname)
	{
	zfs_snapentry_t *se;
	boolean_t ismounted = B_FALSE;

	rw_enter(&zfs_snapshot_lock, RW_READER);
	if ((se = zfsctl_snapshot_find_by_name(snapname)) != NULL) {
	zfsctl_snapshot_rele(se);
	ismounted = B_TRUE;
	}
	rw_exit(&zfs_snapshot_lock);

	return (ismounted);
	}

	/*
	* Check if the given inode is a part of the virtual .zfs directory.
	*/
	boolean_t
	zfsctl_is_node(struct inode *ip)
	{
	return (ITOZ(ip)->z_is_ctldir);
	}

	/*
	* Check if the given inode is a .zfs/snapshots/snapname directory.
	*/
	boolean_t
	zfsctl_is_snapdir(struct inode *ip)
	{
	return (zfsctl_is_node(ip) && (ip->i_ino <= ZFSCTL_INO_SNAPDIRS));
	}

	/*
	* Allocate a new inode with the passed id and ops.
	*/
	static struct inode *
	zfsctl_inode_alloc(zfsvfs_t *zfsvfs, uint64_t id,
	const struct file_operations fops, const struct inode_operations ops)
	{
	inode_timespec_t now;
	struct inode *ip;
	znode_t *zp;

	ip = new_inode(zfsvfs->z_sb);
	if (ip == NULL)
	return (NULL);

	now = current_time(ip);
	zp = ITOZ(ip);
	ASSERT3P(zp->z_dirlocks, ==, NULL);
	ASSERT3P(zp->z_acl_cached, ==, NULL);
	ASSERT3P(zp->z_xattr_cached, ==, NULL);
	zp->z_id = id;
	zp->z_unlinked = B_FALSE;
	zp->z_atime_dirty = B_FALSE;
	zp->z_zn_prefetch = B_FALSE;
	zp->z_moved = B_FALSE;
	zp->z_is_sa = B_FALSE;
	zp->z_is_mapped = B_FALSE;
	zp->z_is_ctldir = B_TRUE;
	zp->z_is_stale = B_FALSE;
	zp->z_sa_hdl = NULL;
	zp->z_blksz = 0;
	zp->z_seq = 0;
	zp->z_mapcnt = 0;
	zp->z_size = 0;
	zp->z_pflags = 0;
	zp->z_mode = 0;
	zp->z_sync_cnt = 0;
	ip->i_generation = 0;
	ip->i_ino = id;
	ip->i_mode = (S_IFDIR \| S_IRWXUGO);
	ip->i_uid = SUID_TO_KUID(0);
	ip->i_gid = SGID_TO_KGID(0);
	ip->i_blkbits = SPA_MINBLOCKSHIFT;
	ip->i_atime = now;
	ip->i_mtime = now;
	ip->i_ctime = now;
	ip->i_fop = fops;
	ip->i_op = ops;
	#if defined(IOP_XATTR)
	ip->i_opflags &= ~IOP_XATTR;
	#endif

	if (insert_inode_locked(ip)) {
	unlock_new_inode(ip);
	iput(ip);
	return (NULL);
	}

	mutex_enter(&zfsvfs->z_znodes_lock);
	list_insert_tail(&zfsvfs->z_all_znodes, zp);
	zfsvfs->z_nr_znodes++;
	membar_producer();
	mutex_exit(&zfsvfs->z_znodes_lock);

	unlock_new_inode(ip);

	return (ip);
	}

	/*
	* Lookup the inode with given id, it will be allocated if needed.
	*/
	static struct inode *
	zfsctl_inode_lookup(zfsvfs_t *zfsvfs, uint64_t id,
	const struct file_operations fops, const struct inode_operations ops)
	{
	struct inode *ip = NULL;

	while (ip == NULL) {
	ip = ilookup(zfsvfs->z_sb, (unsigned long)id);
	if (ip)
	break;

	/* May fail due to concurrent zfsctl_inode_alloc() */
	ip = zfsctl_inode_alloc(zfsvfs, id, fops, ops);
	}

	return (ip);
	}

	/*
	* Create the '.zfs' directory. This directory is cached as part of the VFS
	* structure. This results in a hold on the zfsvfs_t. The code in zfs_umount()
	* therefore checks against a vfs_count of 2 instead of 1. This reference
	* is removed when the ctldir is destroyed in the unmount. All other entities
	* under the '.zfs' directory are created dynamically as needed.
	*
	* Because the dynamically created '.zfs' directory entries assume the use
	* of 64-bit inode numbers this support must be disabled on 32-bit systems.
	*/
	int
	zfsctl_create(zfsvfs_t *zfsvfs)
	{
	ASSERT(zfsvfs->z_ctldir == NULL);

	zfsvfs->z_ctldir = zfsctl_inode_alloc(zfsvfs, ZFSCTL_INO_ROOT,
	&zpl_fops_root, &zpl_ops_root);
	if (zfsvfs->z_ctldir == NULL)
	return (SET_ERROR(ENOENT));

	return (0);
	}

	/*
	* Destroy the '.zfs' directory or remove a snapshot from zfs_snapshots_by_name.
	* Only called when the filesystem is unmounted.
	*/
	void
	zfsctl_destroy(zfsvfs_t *zfsvfs)
	{
	if (zfsvfs->z_issnap) {
	zfs_snapentry_t *se;
	spa_t *spa = zfsvfs->z_os->os_spa;
	uint64_t objsetid = dmu_objset_id(zfsvfs->z_os);

	rw_enter(&zfs_snapshot_lock, RW_WRITER);
	se = zfsctl_snapshot_find_by_objsetid(spa, objsetid);
	if (se != NULL)
	zfsctl_snapshot_remove(se);
	rw_exit(&zfs_snapshot_lock);
	if (se != NULL) {
	zfsctl_snapshot_unmount_cancel(se);
	zfsctl_snapshot_rele(se);
	}
	} else if (zfsvfs->z_ctldir) {
	iput(zfsvfs->z_ctldir);
	zfsvfs->z_ctldir = NULL;
	}
	}

	/*
	* Given a root znode, retrieve the associated .zfs directory.
	* Add a hold to the vnode and return it.
	*/
	struct inode *
	zfsctl_root(znode_t *zp)
	{
	ASSERT(zfs_has_ctldir(zp));
	igrab(ZTOZSB(zp)->z_ctldir);
	return (ZTOZSB(zp)->z_ctldir);
	}

	/*
	* Generate a long fid to indicate a snapdir. We encode whether snapdir is
	* already mounted in gen field. We do this because nfsd lookup will not
	* trigger automount. Next time the nfsd does fh_to_dentry, we will notice
	* this and do automount and return ESTALE to force nfsd revalidate and follow
	* mount.
	*/
	static int
	zfsctl_snapdir_fid(struct inode ip, fid_t fidp)
	{
	zfid_short_t zfid = (zfid_short_t )fidp;
	zfid_long_t zlfid = (zfid_long_t )fidp;
	uint32_t gen = 0;
	uint64_t object;
	uint64_t objsetid;
	int i;
	struct dentry *dentry;

	if (fidp->fid_len < LONG_FID_LEN) {
	fidp->fid_len = LONG_FID_LEN;
	return (SET_ERROR(ENOSPC));
	}

	object = ip->i_ino;
	objsetid = ZFSCTL_INO_SNAPDIRS - ip->i_ino;
	zfid->zf_len = LONG_FID_LEN;

	dentry = d_obtain_alias(igrab(ip));
	if (!IS_ERR(dentry)) {
	gen = !!d_mountpoint(dentry);
	dput(dentry);
	}

	for (i = 0; i < sizeof (zfid->zf_object); i++)
	zfid->zf_object[i] = (uint8_t)(object >> (8 * i));

	for (i = 0; i < sizeof (zfid->zf_gen); i++)
	zfid->zf_gen[i] = (uint8_t)(gen >> (8 * i));

	for (i = 0; i < sizeof (zlfid->zf_setid); i++)
	zlfid->zf_setid[i] = (uint8_t)(objsetid >> (8 * i));

	for (i = 0; i < sizeof (zlfid->zf_setgen); i++)
	zlfid->zf_setgen[i] = 0;

	return (0);
	}

	/*
	* Generate an appropriate fid for an entry in the .zfs directory.
	*/
	int
	zfsctl_fid(struct inode ip, fid_t fidp)
	{
	znode_t *zp = ITOZ(ip);
	zfsvfs_t *zfsvfs = ITOZSB(ip);
	uint64_t object = zp->z_id;
	zfid_short_t *zfid;
	int i;

	ZFS_ENTER(zfsvfs);

	if (zfsctl_is_snapdir(ip)) {
	ZFS_EXIT(zfsvfs);
	return (zfsctl_snapdir_fid(ip, fidp));
	}

	if (fidp->fid_len < SHORT_FID_LEN) {
	fidp->fid_len = SHORT_FID_LEN;
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(ENOSPC));
	}

	zfid = (zfid_short_t *)fidp;

	zfid->zf_len = SHORT_FID_LEN;

	for (i = 0; i < sizeof (zfid->zf_object); i++)
	zfid->zf_object[i] = (uint8_t)(object >> (8 * i));

	/* .zfs znodes always have a generation number of 0 */
	for (i = 0; i < sizeof (zfid->zf_gen); i++)
	zfid->zf_gen[i] = 0;

	ZFS_EXIT(zfsvfs);
	return (0);
	}

	/*
	* Construct a full dataset name in full_name: "pool/dataset@snap_name"
	*/
	static int
	zfsctl_snapshot_name(zfsvfs_t zfsvfs, const char snap_name, int len,
	char *full_name)
	{
	objset_t *os = zfsvfs->z_os;

	if (zfs_component_namecheck(snap_name, NULL, NULL) != 0)
	return (SET_ERROR(EILSEQ));

	dmu_objset_name(os, full_name);
	if ((strlen(full_name) + 1 + strlen(snap_name)) >= len)
	return (SET_ERROR(ENAMETOOLONG));

	(void) strcat(full_name, "@");
	(void) strcat(full_name, snap_name);

	return (0);
	}

	/*
	* Returns full path in full_path: "/pool/dataset/.zfs/snapshot/snap_name/"
	*/
	static int
	zfsctl_snapshot_path_objset(zfsvfs_t *zfsvfs, uint64_t objsetid,
	int path_len, char *full_path)
	{
	objset_t *os = zfsvfs->z_os;
	fstrans_cookie_t cookie;
	char *snapname;
	boolean_t case_conflict;
	uint64_t id, pos = 0;
	int error = 0;

	if (zfsvfs->z_vfs->vfs_mntpoint == NULL)
	return (SET_ERROR(ENOENT));

	cookie = spl_fstrans_mark();
	snapname = kmem_alloc(ZFS_MAX_DATASET_NAME_LEN, KM_SLEEP);

	while (error == 0) {
	dsl_pool_config_enter(dmu_objset_pool(os), FTAG);
	error = dmu_snapshot_list_next(zfsvfs->z_os,
	ZFS_MAX_DATASET_NAME_LEN, snapname, &id, &pos,
	&case_conflict);
	dsl_pool_config_exit(dmu_objset_pool(os), FTAG);
	if (error)
	goto out;

	if (id == objsetid)
	break;
	}

	snprintf(full_path, path_len, "%s/.zfs/snapshot/%s",
	zfsvfs->z_vfs->vfs_mntpoint, snapname);
	out:
	kmem_free(snapname, ZFS_MAX_DATASET_NAME_LEN);
	spl_fstrans_unmark(cookie);

	return (error);
	}

	/*
	* Special case the handling of "..".
	*/
	int
	-zfsctl_root_lookup(struct inode dip, char name, struct inode **ipp,
	+zfsctl_root_lookup(struct inode dip, const char name, struct inode **ipp,
	int flags, cred_t cr, int direntflags, pathname_t *realpnp)
	{
	zfsvfs_t *zfsvfs = ITOZSB(dip);
	int error = 0;

	ZFS_ENTER(zfsvfs);

	if (strcmp(name, "..") == 0) {
	*ipp = dip->i_sb->s_root->d_inode;
	} else if (strcmp(name, ZFS_SNAPDIR_NAME) == 0) {
	*ipp = zfsctl_inode_lookup(zfsvfs, ZFSCTL_INO_SNAPDIR,
	&zpl_fops_snapdir, &zpl_ops_snapdir);
	} else if (strcmp(name, ZFS_SHAREDIR_NAME) == 0) {
	*ipp = zfsctl_inode_lookup(zfsvfs, ZFSCTL_INO_SHARES,
	&zpl_fops_shares, &zpl_ops_shares);
	} else {
	*ipp = NULL;
	}

	if (*ipp == NULL)
	error = SET_ERROR(ENOENT);

	ZFS_EXIT(zfsvfs);

	return (error);
	}

	/*
	* Lookup entry point for the 'snapshot' directory. Try to open the
	* snapshot if it exist, creating the pseudo filesystem inode as necessary.
	*/
	int
	-zfsctl_snapdir_lookup(struct inode dip, char name, struct inode **ipp,
	+zfsctl_snapdir_lookup(struct inode dip, const char name, struct inode **ipp,
	int flags, cred_t cr, int direntflags, pathname_t *realpnp)
	{
	zfsvfs_t *zfsvfs = ITOZSB(dip);
	uint64_t id;
	int error;

	ZFS_ENTER(zfsvfs);

	error = dmu_snapshot_lookup(zfsvfs->z_os, name, &id);
	if (error) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	*ipp = zfsctl_inode_lookup(zfsvfs, ZFSCTL_INO_SNAPDIRS - id,
	&simple_dir_operations, &simple_dir_inode_operations);
	if (*ipp == NULL)
	error = SET_ERROR(ENOENT);

	ZFS_EXIT(zfsvfs);

	return (error);
	}

	/*
	* Renaming a directory under '.zfs/snapshot' will automatically trigger
	* a rename of the snapshot to the new given name. The rename is confined
	* to the '.zfs/snapshot' directory snapshots cannot be moved elsewhere.
	*/
	int
	-zfsctl_snapdir_rename(struct inode sdip, char snm,
	- struct inode tdip, char tnm, cred_t *cr, int flags)
	+zfsctl_snapdir_rename(struct inode sdip, const char snm,
	+ struct inode tdip, const char tnm, cred_t *cr, int flags)
	{
	zfsvfs_t *zfsvfs = ITOZSB(sdip);
	char to, from, real, fsname;
	int error;

	if (!zfs_admin_snapshot)
	return (SET_ERROR(EACCES));

	ZFS_ENTER(zfsvfs);

	to = kmem_alloc(ZFS_MAX_DATASET_NAME_LEN, KM_SLEEP);
	from = kmem_alloc(ZFS_MAX_DATASET_NAME_LEN, KM_SLEEP);
	real = kmem_alloc(ZFS_MAX_DATASET_NAME_LEN, KM_SLEEP);
	fsname = kmem_alloc(ZFS_MAX_DATASET_NAME_LEN, KM_SLEEP);

	if (zfsvfs->z_case == ZFS_CASE_INSENSITIVE) {
	error = dmu_snapshot_realname(zfsvfs->z_os, snm, real,
	ZFS_MAX_DATASET_NAME_LEN, NULL);
	if (error == 0) {
	snm = real;
	} else if (error != ENOTSUP) {
	goto out;
	}
	}

	dmu_objset_name(zfsvfs->z_os, fsname);

	error = zfsctl_snapshot_name(ITOZSB(sdip), snm,
	ZFS_MAX_DATASET_NAME_LEN, from);
	if (error == 0)
	error = zfsctl_snapshot_name(ITOZSB(tdip), tnm,
	ZFS_MAX_DATASET_NAME_LEN, to);
	if (error == 0)
	error = zfs_secpolicy_rename_perms(from, to, cr);
	if (error != 0)
	goto out;

	/*
	* Cannot move snapshots out of the snapdir.
	*/
	if (sdip != tdip) {
	error = SET_ERROR(EINVAL);
	goto out;
	}

	/*
	* No-op when names are identical.
	*/
	if (strcmp(snm, tnm) == 0) {
	error = 0;
	goto out;
	}

	rw_enter(&zfs_snapshot_lock, RW_WRITER);

	error = dsl_dataset_rename_snapshot(fsname, snm, tnm, B_FALSE);
	if (error == 0)
	(void) zfsctl_snapshot_rename(snm, tnm);

	rw_exit(&zfs_snapshot_lock);
	out:
	kmem_free(from, ZFS_MAX_DATASET_NAME_LEN);
	kmem_free(to, ZFS_MAX_DATASET_NAME_LEN);
	kmem_free(real, ZFS_MAX_DATASET_NAME_LEN);
	kmem_free(fsname, ZFS_MAX_DATASET_NAME_LEN);

	ZFS_EXIT(zfsvfs);

	return (error);
	}

	/*
	* Removing a directory under '.zfs/snapshot' will automatically trigger
	* the removal of the snapshot with the given name.
	*/
	int
	-zfsctl_snapdir_remove(struct inode dip, char name, cred_t *cr, int flags)
	+zfsctl_snapdir_remove(struct inode dip, const char name, cred_t *cr,
	+ int flags)
	{
	zfsvfs_t *zfsvfs = ITOZSB(dip);
	char snapname, real;
	int error;

	if (!zfs_admin_snapshot)
	return (SET_ERROR(EACCES));

	ZFS_ENTER(zfsvfs);

	snapname = kmem_alloc(ZFS_MAX_DATASET_NAME_LEN, KM_SLEEP);
	real = kmem_alloc(ZFS_MAX_DATASET_NAME_LEN, KM_SLEEP);

	if (zfsvfs->z_case == ZFS_CASE_INSENSITIVE) {
	error = dmu_snapshot_realname(zfsvfs->z_os, name, real,
	ZFS_MAX_DATASET_NAME_LEN, NULL);
	if (error == 0) {
	name = real;
	} else if (error != ENOTSUP) {
	goto out;
	}
	}

	error = zfsctl_snapshot_name(ITOZSB(dip), name,
	ZFS_MAX_DATASET_NAME_LEN, snapname);
	if (error == 0)
	error = zfs_secpolicy_destroy_perms(snapname, cr);
	if (error != 0)
	goto out;

	error = zfsctl_snapshot_unmount(snapname, MNT_FORCE);
	if ((error == 0) \|\| (error == ENOENT))
	error = dsl_destroy_snapshot(snapname, B_FALSE);
	out:
	kmem_free(snapname, ZFS_MAX_DATASET_NAME_LEN);
	kmem_free(real, ZFS_MAX_DATASET_NAME_LEN);

	ZFS_EXIT(zfsvfs);

	return (error);
	}

	/*
	* Creating a directory under '.zfs/snapshot' will automatically trigger
	* the creation of a new snapshot with the given name.
	*/
	int
	-zfsctl_snapdir_mkdir(struct inode dip, char dirname, vattr_t *vap,
	+zfsctl_snapdir_mkdir(struct inode dip, const char dirname, vattr_t *vap,
	struct inode *ipp, cred_t cr, int flags)
	{
	zfsvfs_t *zfsvfs = ITOZSB(dip);
	char *dsname;
	int error;

	if (!zfs_admin_snapshot)
	return (SET_ERROR(EACCES));

	dsname = kmem_alloc(ZFS_MAX_DATASET_NAME_LEN, KM_SLEEP);

	if (zfs_component_namecheck(dirname, NULL, NULL) != 0) {
	error = SET_ERROR(EILSEQ);
	goto out;
	}

	dmu_objset_name(zfsvfs->z_os, dsname);

	error = zfs_secpolicy_snapshot_perms(dsname, cr);
	if (error != 0)
	goto out;

	if (error == 0) {
	error = dmu_objset_snapshot_one(dsname, dirname);
	if (error != 0)
	goto out;

	error = zfsctl_snapdir_lookup(dip, dirname, ipp,
	0, cr, NULL, NULL);
	}
	out:
	kmem_free(dsname, ZFS_MAX_DATASET_NAME_LEN);

	return (error);
	}

	/*
	* Flush everything out of the kernel's export table and such.
	* This is needed as once the snapshot is used over NFS, its
	* entries in svc_export and svc_expkey caches hold reference
	* to the snapshot mount point. There is no known way of flushing
	* only the entries related to the snapshot.
	*/
	static void
	exportfs_flush(void)
	{
	char *argv[] = { "/usr/sbin/exportfs", "-f", NULL };
	char *envp[] = { NULL };

	(void) call_usermodehelper(argv[0], argv, envp, UMH_WAIT_PROC);
	}

	/*
	* Attempt to unmount a snapshot by making a call to user space.
	* There is no assurance that this can or will succeed, is just a
	* best effort. In the case where it does fail, perhaps because
	* it's in use, the unmount will fail harmlessly.
	*/
	int
	-zfsctl_snapshot_unmount(char *snapname, int flags)
	+zfsctl_snapshot_unmount(const char *snapname, int flags)
	{
	char *argv[] = { "/usr/bin/env", "umount", "-t", "zfs", "-n", NULL,
	NULL };
	char *envp[] = { NULL };
	zfs_snapentry_t *se;
	int error;

	rw_enter(&zfs_snapshot_lock, RW_READER);
	if ((se = zfsctl_snapshot_find_by_name(snapname)) == NULL) {
	rw_exit(&zfs_snapshot_lock);
	return (SET_ERROR(ENOENT));
	}
	rw_exit(&zfs_snapshot_lock);

	exportfs_flush();

	if (flags & MNT_FORCE)
	argv[4] = "-fn";
	argv[5] = se->se_path;
	dprintf("unmount; path=%s\n", se->se_path);
	error = call_usermodehelper(argv[0], argv, envp, UMH_WAIT_PROC);
	zfsctl_snapshot_rele(se);


	/*
	* The umount system utility will return 256 on error. We must
	* assume this error is because the file system is busy so it is
	* converted to the more sensible EBUSY.
	*/
	if (error)
	error = SET_ERROR(EBUSY);

	return (error);
	}

	int
	zfsctl_snapshot_mount(struct path *path, int flags)
	{
	struct dentry *dentry = path->dentry;
	struct inode *ip = dentry->d_inode;
	zfsvfs_t *zfsvfs;
	zfsvfs_t *snap_zfsvfs;
	zfs_snapentry_t *se;
	char full_name, full_path;
	char *argv[] = { "/usr/bin/env", "mount", "-t", "zfs", "-n", NULL, NULL,
	NULL };
	char *envp[] = { NULL };
	int error;
	struct path spath;

	if (ip == NULL)
	return (SET_ERROR(EISDIR));

	zfsvfs = ITOZSB(ip);
	ZFS_ENTER(zfsvfs);

	full_name = kmem_zalloc(ZFS_MAX_DATASET_NAME_LEN, KM_SLEEP);
	full_path = kmem_zalloc(MAXPATHLEN, KM_SLEEP);

	error = zfsctl_snapshot_name(zfsvfs, dname(dentry),
	ZFS_MAX_DATASET_NAME_LEN, full_name);
	if (error)
	goto error;

	/*
	* Construct a mount point path from sb of the ctldir inode and dirent
	* name, instead of from d_path(), so that chroot'd process doesn't fail
	* on mount.zfs(8).
	*/
	snprintf(full_path, MAXPATHLEN, "%s/.zfs/snapshot/%s",
	zfsvfs->z_vfs->vfs_mntpoint ? zfsvfs->z_vfs->vfs_mntpoint : "",
	dname(dentry));

	/*
	* Multiple concurrent automounts of a snapshot are never allowed.
	* The snapshot may be manually mounted as many times as desired.
	*/
	if (zfsctl_snapshot_ismounted(full_name)) {
	error = 0;
	goto error;
	}

	/*
	* Attempt to mount the snapshot from user space. Normally this
	* would be done using the vfs_kern_mount() function, however that
	* function is marked GPL-only and cannot be used. On error we
	* careful to log the real error to the console and return EISDIR
	* to safely abort the automount. This should be very rare.
	*
	* If the user mode helper happens to return EBUSY, a concurrent
	* mount is already in progress in which case the error is ignored.
	* Take note that if the program was executed successfully the return
	* value from call_usermodehelper() will be (exitcode << 8 + signal).
	*/
	dprintf("mount; name=%s path=%s\n", full_name, full_path);
	argv[5] = full_name;
	argv[6] = full_path;
	error = call_usermodehelper(argv[0], argv, envp, UMH_WAIT_PROC);
	if (error) {
	if (!(error & MOUNT_BUSY << 8)) {
	zfs_dbgmsg("Unable to automount %s error=%d",
	full_path, error);
	error = SET_ERROR(EISDIR);
	} else {
	/*
	* EBUSY, this could mean a concurrent mount, or the
	* snapshot has already been mounted at completely
	* different place. We return 0 so VFS will retry. For
	* the latter case the VFS will retry several times
	* and return ELOOP, which is probably not a very good
	* behavior.
	*/
	error = 0;
	}
	goto error;
	}

	/*
	* Follow down in to the mounted snapshot and set MNT_SHRINKABLE
	* to identify this as an automounted filesystem.
	*/
	spath = *path;
	path_get(&spath);
	if (follow_down_one(&spath)) {
	snap_zfsvfs = ITOZSB(spath.dentry->d_inode);
	snap_zfsvfs->z_parent = zfsvfs;
	dentry = spath.dentry;
	spath.mnt->mnt_flags \|= MNT_SHRINKABLE;

	rw_enter(&zfs_snapshot_lock, RW_WRITER);
	se = zfsctl_snapshot_alloc(full_name, full_path,
	snap_zfsvfs->z_os->os_spa, dmu_objset_id(snap_zfsvfs->z_os),
	dentry);
	zfsctl_snapshot_add(se);
	zfsctl_snapshot_unmount_delay_impl(se, zfs_expire_snapshot);
	rw_exit(&zfs_snapshot_lock);
	}
	path_put(&spath);
	error:
	kmem_free(full_name, ZFS_MAX_DATASET_NAME_LEN);
	kmem_free(full_path, MAXPATHLEN);

	ZFS_EXIT(zfsvfs);

	return (error);
	}

	/*
	* Get the snapdir inode from fid
	*/
	int
	zfsctl_snapdir_vget(struct super_block *sb, uint64_t objsetid, int gen,
	struct inode **ipp)
	{
	int error;
	struct path path;
	char *mnt;
	struct dentry *dentry;

	mnt = kmem_alloc(MAXPATHLEN, KM_SLEEP);

	error = zfsctl_snapshot_path_objset(sb->s_fs_info, objsetid,
	MAXPATHLEN, mnt);
	if (error)
	goto out;

	/* Trigger automount */
	error = -kern_path(mnt, LOOKUP_FOLLOW\|LOOKUP_DIRECTORY, &path);
	if (error)
	goto out;

	path_put(&path);
	/*
	* Get the snapdir inode. Note, we don't want to use the above
	* path because it contains the root of the snapshot rather
	* than the snapdir.
	*/
	*ipp = ilookup(sb, ZFSCTL_INO_SNAPDIRS - objsetid);
	if (*ipp == NULL) {
	error = SET_ERROR(ENOENT);
	goto out;
	}

	/* check gen, see zfsctl_snapdir_fid */
	dentry = d_obtain_alias(igrab(*ipp));
	if (gen != (!IS_ERR(dentry) && d_mountpoint(dentry))) {
	iput(*ipp);
	*ipp = NULL;
	error = SET_ERROR(ENOENT);
	}
	if (!IS_ERR(dentry))
	dput(dentry);
	out:
	kmem_free(mnt, MAXPATHLEN);
	return (error);
	}

	int
	zfsctl_shares_lookup(struct inode dip, char name, struct inode **ipp,
	int flags, cred_t cr, int direntflags, pathname_t *realpnp)
	{
	zfsvfs_t *zfsvfs = ITOZSB(dip);
	znode_t *zp;
	znode_t *dzp;
	int error;

	ZFS_ENTER(zfsvfs);

	if (zfsvfs->z_shares_dir == 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(ENOTSUP));
	}

	if ((error = zfs_zget(zfsvfs, zfsvfs->z_shares_dir, &dzp)) == 0) {
	error = zfs_lookup(dzp, name, &zp, 0, cr, NULL, NULL);
	zrele(dzp);
	}

	ZFS_EXIT(zfsvfs);

	return (error);
	}

	/*
	* Initialize the various pieces we'll need to create and manipulate .zfs
	* directories. Currently this is unused but available.
	*/
	void
	zfsctl_init(void)
	{
	avl_create(&zfs_snapshots_by_name, snapentry_compare_by_name,
	sizeof (zfs_snapentry_t), offsetof(zfs_snapentry_t,
	se_node_name));
	avl_create(&zfs_snapshots_by_objsetid, snapentry_compare_by_objsetid,
	sizeof (zfs_snapentry_t), offsetof(zfs_snapentry_t,
	se_node_objsetid));
	rw_init(&zfs_snapshot_lock, NULL, RW_DEFAULT, NULL);
	}

	/*
	* Cleanup the various pieces we needed for .zfs directories. In particular
	* ensure the expiry timer is canceled safely.
	*/
	void
	zfsctl_fini(void)
	{
	avl_destroy(&zfs_snapshots_by_name);
	avl_destroy(&zfs_snapshots_by_objsetid);
	rw_destroy(&zfs_snapshot_lock);
	}

	module_param(zfs_admin_snapshot, int, 0644);
	MODULE_PARM_DESC(zfs_admin_snapshot, "Enable mkdir/rmdir/mv in .zfs/snapshot");

	module_param(zfs_expire_snapshot, int, 0644);
	MODULE_PARM_DESC(zfs_expire_snapshot, "Seconds to expire .zfs/snapshot");
	Index: vendor-sys/openzfs/dist/module/os/linux/zfs/zfs_dir.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/zfs/zfs_dir.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/zfs/zfs_dir.c (revision 366774)
	@@ -1,1224 +1,1225 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/

	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2013, 2016 by Delphix. All rights reserved.
	* Copyright 2017 Nexenta Systems, Inc.
	*/

	#include <sys/types.h>
	#include <sys/param.h>
	#include <sys/time.h>
	#include <sys/sysmacros.h>
	#include <sys/vfs.h>
	#include <sys/vnode.h>
	#include <sys/file.h>
	#include <sys/kmem.h>
	#include <sys/uio.h>
	#include <sys/pathname.h>
	#include <sys/cmn_err.h>
	#include <sys/errno.h>
	#include <sys/stat.h>
	#include <sys/sunddi.h>
	#include <sys/random.h>
	#include <sys/policy.h>
	#include <sys/zfs_dir.h>
	#include <sys/zfs_acl.h>
	#include <sys/zfs_vnops.h>
	#include <sys/fs/zfs.h>
	#include <sys/zap.h>
	#include <sys/dmu.h>
	#include <sys/atomic.h>
	#include <sys/zfs_ctldir.h>
	#include <sys/zfs_fuid.h>
	#include <sys/sa.h>
	#include <sys/zfs_sa.h>
	#include <sys/dmu_objset.h>
	#include <sys/dsl_dir.h>

	/*
	* zfs_match_find() is used by zfs_dirent_lock() to perform zap lookups
	* of names after deciding which is the appropriate lookup interface.
	*/
	static int
	-zfs_match_find(zfsvfs_t zfsvfs, znode_t dzp, char *name, matchtype_t mt,
	- boolean_t update, int deflags, pathname_t rpnp, uint64_t *zoid)
	+zfs_match_find(zfsvfs_t zfsvfs, znode_t dzp, const char *name,
	+ matchtype_t mt, boolean_t update, int deflags, pathname_t rpnp,
	+ uint64_t *zoid)
	{
	boolean_t conflict = B_FALSE;
	int error;

	if (zfsvfs->z_norm) {
	size_t bufsz = 0;
	char *buf = NULL;

	if (rpnp) {
	buf = rpnp->pn_buf;
	bufsz = rpnp->pn_bufsize;
	}

	/*
	* In the non-mixed case we only expect there would ever
	* be one match, but we need to use the normalizing lookup.
	*/
	error = zap_lookup_norm(zfsvfs->z_os, dzp->z_id, name, 8, 1,
	zoid, mt, buf, bufsz, &conflict);
	} else {
	error = zap_lookup(zfsvfs->z_os, dzp->z_id, name, 8, 1, zoid);
	}

	/*
	* Allow multiple entries provided the first entry is
	* the object id. Non-zpl consumers may safely make
	* use of the additional space.
	*
	* XXX: This should be a feature flag for compatibility
	*/
	if (error == EOVERFLOW)
	error = 0;

	if (zfsvfs->z_norm && !error && deflags)
	*deflags = conflict ? ED_CASE_CONFLICT : 0;

	zoid = ZFS_DIRENT_OBJ(zoid);

	return (error);
	}

	/*
	* Lock a directory entry. A dirlock on <dzp, name> protects that name
	* in dzp's directory zap object. As long as you hold a dirlock, you can
	* assume two things: (1) dzp cannot be reaped, and (2) no other thread
	* can change the zap entry for (i.e. link or unlink) this name.
	*
	* Input arguments:
	* dzp - znode for directory
	* name - name of entry to lock
	* flag - ZNEW: if the entry already exists, fail with EEXIST.
	* ZEXISTS: if the entry does not exist, fail with ENOENT.
	* ZSHARED: allow concurrent access with other ZSHARED callers.
	* ZXATTR: we want dzp's xattr directory
	* ZCILOOK: On a mixed sensitivity file system,
	* this lookup should be case-insensitive.
	* ZCIEXACT: On a purely case-insensitive file system,
	* this lookup should be case-sensitive.
	* ZRENAMING: we are locking for renaming, force narrow locks
	* ZHAVELOCK: Don't grab the z_name_lock for this call. The
	* current thread already holds it.
	*
	* Output arguments:
	* zpp - pointer to the znode for the entry (NULL if there isn't one)
	* dlpp - pointer to the dirlock for this entry (NULL on error)
	* direntflags - (case-insensitive lookup only)
	* flags if multiple case-sensitive matches exist in directory
	* realpnp - (case-insensitive lookup only)
	* actual name matched within the directory
	*
	* Return value: 0 on success or errno on failure.
	*
	* NOTE: Always checks for, and rejects, '.' and '..'.
	* NOTE: For case-insensitive file systems we take wide locks (see below),
	* but return znode pointers to a single match.
	*/
	int
	-zfs_dirent_lock(zfs_dirlock_t *dlpp, znode_t dzp, char name, znode_t *zpp,
	- int flag, int direntflags, pathname_t realpnp)
	+zfs_dirent_lock(zfs_dirlock_t *dlpp, znode_t dzp, char *name,
	+ znode_t *zpp, int flag, int direntflags, pathname_t *realpnp)
	{
	zfsvfs_t *zfsvfs = ZTOZSB(dzp);
	zfs_dirlock_t *dl;
	boolean_t update;
	matchtype_t mt = 0;
	uint64_t zoid;
	int error = 0;
	int cmpflags;

	*zpp = NULL;
	*dlpp = NULL;

	/*
	* Verify that we are not trying to lock '.', '..', or '.zfs'
	*/
	if ((name[0] == '.' &&
	(name[1] == '\0' \|\| (name[1] == '.' && name[2] == '\0'))) \|\|
	(zfs_has_ctldir(dzp) && strcmp(name, ZFS_CTLDIR_NAME) == 0))
	return (SET_ERROR(EEXIST));

	/*
	* Case sensitivity and normalization preferences are set when
	* the file system is created. These are stored in the
	* zfsvfs->z_case and zfsvfs->z_norm fields. These choices
	* affect what vnodes can be cached in the DNLC, how we
	* perform zap lookups, and the "width" of our dirlocks.
	*
	* A normal dirlock locks a single name. Note that with
	* normalization a name can be composed multiple ways, but
	* when normalized, these names all compare equal. A wide
	* dirlock locks multiple names. We need these when the file
	* system is supporting mixed-mode access. It is sometimes
	* necessary to lock all case permutations of file name at
	* once so that simultaneous case-insensitive/case-sensitive
	* behaves as rationally as possible.
	*/

	/*
	* When matching we may need to normalize & change case according to
	* FS settings.
	*
	* Note that a normalized match is necessary for a case insensitive
	* filesystem when the lookup request is not exact because normalization
	* can fold case independent of normalizing code point sequences.
	*
	* See the table above zfs_dropname().
	*/
	if (zfsvfs->z_norm != 0) {
	mt = MT_NORMALIZE;

	/*
	* Determine if the match needs to honor the case specified in
	* lookup, and if so keep track of that so that during
	* normalization we don't fold case.
	*/
	if ((zfsvfs->z_case == ZFS_CASE_INSENSITIVE &&
	(flag & ZCIEXACT)) \|\|
	(zfsvfs->z_case == ZFS_CASE_MIXED && !(flag & ZCILOOK))) {
	mt \|= MT_MATCH_CASE;
	}
	}

	/*
	* Only look in or update the DNLC if we are looking for the
	* name on a file system that does not require normalization
	* or case folding. We can also look there if we happen to be
	* on a non-normalizing, mixed sensitivity file system IF we
	* are looking for the exact name.
	*
	* Maybe can add TO-UPPERed version of name to dnlc in ci-only
	* case for performance improvement?
	*/
	update = !zfsvfs->z_norm \|\|
	(zfsvfs->z_case == ZFS_CASE_MIXED &&
	!(zfsvfs->z_norm & ~U8_TEXTPREP_TOUPPER) && !(flag & ZCILOOK));

	/*
	* ZRENAMING indicates we are in a situation where we should
	* take narrow locks regardless of the file system's
	* preferences for normalizing and case folding. This will
	* prevent us deadlocking trying to grab the same wide lock
	* twice if the two names happen to be case-insensitive
	* matches.
	*/
	if (flag & ZRENAMING)
	cmpflags = 0;
	else
	cmpflags = zfsvfs->z_norm;

	/*
	* Wait until there are no locks on this name.
	*
	* Don't grab the lock if it is already held. However, cannot
	* have both ZSHARED and ZHAVELOCK together.
	*/
	ASSERT(!(flag & ZSHARED) \|\| !(flag & ZHAVELOCK));
	if (!(flag & ZHAVELOCK))
	rw_enter(&dzp->z_name_lock, RW_READER);

	mutex_enter(&dzp->z_lock);
	for (;;) {
	if (dzp->z_unlinked && !(flag & ZXATTR)) {
	mutex_exit(&dzp->z_lock);
	if (!(flag & ZHAVELOCK))
	rw_exit(&dzp->z_name_lock);
	return (SET_ERROR(ENOENT));
	}
	for (dl = dzp->z_dirlocks; dl != NULL; dl = dl->dl_next) {
	if ((u8_strcmp(name, dl->dl_name, 0, cmpflags,
	U8_UNICODE_LATEST, &error) == 0) \|\| error != 0)
	break;
	}
	if (error != 0) {
	mutex_exit(&dzp->z_lock);
	if (!(flag & ZHAVELOCK))
	rw_exit(&dzp->z_name_lock);
	return (SET_ERROR(ENOENT));
	}
	if (dl == NULL) {
	/*
	* Allocate a new dirlock and add it to the list.
	*/
	dl = kmem_alloc(sizeof (zfs_dirlock_t), KM_SLEEP);
	cv_init(&dl->dl_cv, NULL, CV_DEFAULT, NULL);
	dl->dl_name = name;
	dl->dl_sharecnt = 0;
	dl->dl_namelock = 0;
	dl->dl_namesize = 0;
	dl->dl_dzp = dzp;
	dl->dl_next = dzp->z_dirlocks;
	dzp->z_dirlocks = dl;
	break;
	}
	if ((flag & ZSHARED) && dl->dl_sharecnt != 0)
	break;
	cv_wait(&dl->dl_cv, &dzp->z_lock);
	}

	/*
	* If the z_name_lock was NOT held for this dirlock record it.
	*/
	if (flag & ZHAVELOCK)
	dl->dl_namelock = 1;

	if ((flag & ZSHARED) && ++dl->dl_sharecnt > 1 && dl->dl_namesize == 0) {
	/*
	* We're the second shared reference to dl. Make a copy of
	* dl_name in case the first thread goes away before we do.
	* Note that we initialize the new name before storing its
	* pointer into dl_name, because the first thread may load
	* dl->dl_name at any time. It'll either see the old value,
	* which belongs to it, or the new shared copy; either is OK.
	*/
	dl->dl_namesize = strlen(dl->dl_name) + 1;
	name = kmem_alloc(dl->dl_namesize, KM_SLEEP);
	bcopy(dl->dl_name, name, dl->dl_namesize);
	dl->dl_name = name;
	}

	mutex_exit(&dzp->z_lock);

	/*
	* We have a dirlock on the name. (Note that it is the dirlock,
	* not the dzp's z_lock, that protects the name in the zap object.)
	* See if there's an object by this name; if so, put a hold on it.
	*/
	if (flag & ZXATTR) {
	error = sa_lookup(dzp->z_sa_hdl, SA_ZPL_XATTR(zfsvfs), &zoid,
	sizeof (zoid));
	if (error == 0)
	error = (zoid == 0 ? SET_ERROR(ENOENT) : 0);
	} else {
	error = zfs_match_find(zfsvfs, dzp, name, mt,
	update, direntflags, realpnp, &zoid);
	}
	if (error) {
	if (error != ENOENT \|\| (flag & ZEXISTS)) {
	zfs_dirent_unlock(dl);
	return (error);
	}
	} else {
	if (flag & ZNEW) {
	zfs_dirent_unlock(dl);
	return (SET_ERROR(EEXIST));
	}
	error = zfs_zget(zfsvfs, zoid, zpp);
	if (error) {
	zfs_dirent_unlock(dl);
	return (error);
	}
	}

	*dlpp = dl;

	return (0);
	}

	/*
	* Unlock this directory entry and wake anyone who was waiting for it.
	*/
	void
	zfs_dirent_unlock(zfs_dirlock_t *dl)
	{
	znode_t *dzp = dl->dl_dzp;
	zfs_dirlock_t *prev_dl, cur_dl;

	mutex_enter(&dzp->z_lock);

	if (!dl->dl_namelock)
	rw_exit(&dzp->z_name_lock);

	if (dl->dl_sharecnt > 1) {
	dl->dl_sharecnt--;
	mutex_exit(&dzp->z_lock);
	return;
	}
	prev_dl = &dzp->z_dirlocks;
	while ((cur_dl = *prev_dl) != dl)
	prev_dl = &cur_dl->dl_next;
	*prev_dl = dl->dl_next;
	cv_broadcast(&dl->dl_cv);
	mutex_exit(&dzp->z_lock);

	if (dl->dl_namesize != 0)
	kmem_free(dl->dl_name, dl->dl_namesize);
	cv_destroy(&dl->dl_cv);
	kmem_free(dl, sizeof (*dl));
	}

	/*
	* Look up an entry in a directory.
	*
	* NOTE: '.' and '..' are handled as special cases because
	* no directory entries are actually stored for them. If this is
	* the root of a filesystem, then '.zfs' is also treated as a
	* special pseudo-directory.
	*/
	int
	zfs_dirlook(znode_t dzp, char name, znode_t **zpp, int flags,
	int deflg, pathname_t rpnp)
	{
	zfs_dirlock_t *dl;
	znode_t *zp;
	struct inode *ip;
	int error = 0;
	uint64_t parent;

	if (name[0] == 0 \|\| (name[0] == '.' && name[1] == 0)) {
	*zpp = dzp;
	zhold(*zpp);
	} else if (name[0] == '.' && name[1] == '.' && name[2] == 0) {
	zfsvfs_t *zfsvfs = ZTOZSB(dzp);

	/*
	* If we are a snapshot mounted under .zfs, return
	* the inode pointer for the snapshot directory.
	*/
	if ((error = sa_lookup(dzp->z_sa_hdl,
	SA_ZPL_PARENT(zfsvfs), &parent, sizeof (parent))) != 0)
	return (error);

	if (parent == dzp->z_id && zfsvfs->z_parent != zfsvfs) {
	error = zfsctl_root_lookup(zfsvfs->z_parent->z_ctldir,
	"snapshot", &ip, 0, kcred, NULL, NULL);
	*zpp = ITOZ(ip);
	return (error);
	}
	rw_enter(&dzp->z_parent_lock, RW_READER);
	error = zfs_zget(zfsvfs, parent, &zp);
	if (error == 0)
	*zpp = zp;
	rw_exit(&dzp->z_parent_lock);
	} else if (zfs_has_ctldir(dzp) && strcmp(name, ZFS_CTLDIR_NAME) == 0) {
	ip = zfsctl_root(dzp);
	*zpp = ITOZ(ip);
	} else {
	int zf;

	zf = ZEXISTS \| ZSHARED;
	if (flags & FIGNORECASE)
	zf \|= ZCILOOK;

	error = zfs_dirent_lock(&dl, dzp, name, &zp, zf, deflg, rpnp);
	if (error == 0) {
	*zpp = zp;
	zfs_dirent_unlock(dl);
	dzp->z_zn_prefetch = B_TRUE; /* enable prefetching */
	}
	rpnp = NULL;
	}

	if ((flags & FIGNORECASE) && rpnp && !error)
	(void) strlcpy(rpnp->pn_buf, name, rpnp->pn_bufsize);

	return (error);
	}

	/*
	* unlinked Set (formerly known as the "delete queue") Error Handling
	*
	* When dealing with the unlinked set, we dmu_tx_hold_zap(), but we
	* don't specify the name of the entry that we will be manipulating. We
	* also fib and say that we won't be adding any new entries to the
	* unlinked set, even though we might (this is to lower the minimum file
	* size that can be deleted in a full filesystem). So on the small
	* chance that the nlink list is using a fat zap (ie. has more than
	* 2000 entries), we may not pre-read a block that's needed.
	* Therefore it is remotely possible for some of the assertions
	* regarding the unlinked set below to fail due to i/o error. On a
	* nondebug system, this will result in the space being leaked.
	*/
	void
	zfs_unlinked_add(znode_t zp, dmu_tx_t tx)
	{
	zfsvfs_t *zfsvfs = ZTOZSB(zp);

	ASSERT(zp->z_unlinked);
	ASSERT(ZTOI(zp)->i_nlink == 0);

	VERIFY3U(0, ==,
	zap_add_int(zfsvfs->z_os, zfsvfs->z_unlinkedobj, zp->z_id, tx));

	dataset_kstats_update_nunlinks_kstat(&zfsvfs->z_kstat, 1);
	}

	/*
	* Clean up any znodes that had no links when we either crashed or
	* (force) umounted the file system.
	*/
	static void
	zfs_unlinked_drain_task(void *arg)
	{
	zfsvfs_t *zfsvfs = arg;
	zap_cursor_t zc;
	zap_attribute_t zap;
	dmu_object_info_t doi;
	znode_t *zp;
	int error;

	ASSERT3B(zfsvfs->z_draining, ==, B_TRUE);

	/*
	* Iterate over the contents of the unlinked set.
	*/
	for (zap_cursor_init(&zc, zfsvfs->z_os, zfsvfs->z_unlinkedobj);
	zap_cursor_retrieve(&zc, &zap) == 0 && !zfsvfs->z_drain_cancel;
	zap_cursor_advance(&zc)) {

	/*
	* See what kind of object we have in list
	*/

	error = dmu_object_info(zfsvfs->z_os,
	zap.za_first_integer, &doi);
	if (error != 0)
	continue;

	ASSERT((doi.doi_type == DMU_OT_PLAIN_FILE_CONTENTS) \|\|
	(doi.doi_type == DMU_OT_DIRECTORY_CONTENTS));
	/*
	* We need to re-mark these list entries for deletion,
	* so we pull them back into core and set zp->z_unlinked.
	*/
	error = zfs_zget(zfsvfs, zap.za_first_integer, &zp);

	/*
	* We may pick up znodes that are already marked for deletion.
	* This could happen during the purge of an extended attribute
	* directory. All we need to do is skip over them, since they
	* are already in the system marked z_unlinked.
	*/
	if (error != 0)
	continue;

	zp->z_unlinked = B_TRUE;

	/*
	* zrele() decrements the znode's ref count and may cause
	* it to be synchronously freed. We interrupt freeing
	* of this znode by checking the return value of
	* dmu_objset_zfs_unmounting() in dmu_free_long_range()
	* when an unmount is requested.
	*/
	zrele(zp);
	ASSERT3B(zfsvfs->z_unmounted, ==, B_FALSE);
	}
	zap_cursor_fini(&zc);

	zfsvfs->z_draining = B_FALSE;
	zfsvfs->z_drain_task = TASKQID_INVALID;
	}

	/*
	* Sets z_draining then tries to dispatch async unlinked drain.
	* If that fails executes synchronous unlinked drain.
	*/
	void
	zfs_unlinked_drain(zfsvfs_t *zfsvfs)
	{
	ASSERT3B(zfsvfs->z_unmounted, ==, B_FALSE);
	ASSERT3B(zfsvfs->z_draining, ==, B_FALSE);

	zfsvfs->z_draining = B_TRUE;
	zfsvfs->z_drain_cancel = B_FALSE;

	zfsvfs->z_drain_task = taskq_dispatch(
	dsl_pool_unlinked_drain_taskq(dmu_objset_pool(zfsvfs->z_os)),
	zfs_unlinked_drain_task, zfsvfs, TQ_SLEEP);
	if (zfsvfs->z_drain_task == TASKQID_INVALID) {
	zfs_dbgmsg("async zfs_unlinked_drain dispatch failed");
	zfs_unlinked_drain_task(zfsvfs);
	}
	}

	/*
	* Wait for the unlinked drain taskq task to stop. This will interrupt the
	* unlinked set processing if it is in progress.
	*/
	void
	zfs_unlinked_drain_stop_wait(zfsvfs_t *zfsvfs)
	{
	ASSERT3B(zfsvfs->z_unmounted, ==, B_FALSE);

	if (zfsvfs->z_draining) {
	zfsvfs->z_drain_cancel = B_TRUE;
	taskq_cancel_id(dsl_pool_unlinked_drain_taskq(
	dmu_objset_pool(zfsvfs->z_os)), zfsvfs->z_drain_task);
	zfsvfs->z_drain_task = TASKQID_INVALID;
	zfsvfs->z_draining = B_FALSE;
	}
	}

	/*
	* Delete the entire contents of a directory. Return a count
	* of the number of entries that could not be deleted. If we encounter
	* an error, return a count of at least one so that the directory stays
	* in the unlinked set.
	*
	* NOTE: this function assumes that the directory is inactive,
	* so there is no need to lock its entries before deletion.
	* Also, it assumes the directory contents is only regular
	* files.
	*/
	static int
	zfs_purgedir(znode_t *dzp)
	{
	zap_cursor_t zc;
	zap_attribute_t zap;
	znode_t *xzp;
	dmu_tx_t *tx;
	zfsvfs_t *zfsvfs = ZTOZSB(dzp);
	zfs_dirlock_t dl;
	int skipped = 0;
	int error;

	for (zap_cursor_init(&zc, zfsvfs->z_os, dzp->z_id);
	(error = zap_cursor_retrieve(&zc, &zap)) == 0;
	zap_cursor_advance(&zc)) {
	error = zfs_zget(zfsvfs,
	ZFS_DIRENT_OBJ(zap.za_first_integer), &xzp);
	if (error) {
	skipped += 1;
	continue;
	}

	ASSERT(S_ISREG(ZTOI(xzp)->i_mode) \|\|
	S_ISLNK(ZTOI(xzp)->i_mode));

	tx = dmu_tx_create(zfsvfs->z_os);
	dmu_tx_hold_sa(tx, dzp->z_sa_hdl, B_FALSE);
	dmu_tx_hold_zap(tx, dzp->z_id, FALSE, zap.za_name);
	dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE);
	dmu_tx_hold_zap(tx, zfsvfs->z_unlinkedobj, FALSE, NULL);
	/* Is this really needed ? */
	zfs_sa_upgrade_txholds(tx, xzp);
	dmu_tx_mark_netfree(tx);
	error = dmu_tx_assign(tx, TXG_WAIT);
	if (error) {
	dmu_tx_abort(tx);
	zfs_zrele_async(xzp);
	skipped += 1;
	continue;
	}
	bzero(&dl, sizeof (dl));
	dl.dl_dzp = dzp;
	dl.dl_name = zap.za_name;

	error = zfs_link_destroy(&dl, xzp, tx, 0, NULL);
	if (error)
	skipped += 1;
	dmu_tx_commit(tx);

	zfs_zrele_async(xzp);
	}
	zap_cursor_fini(&zc);
	if (error != ENOENT)
	skipped += 1;
	return (skipped);
	}

	void
	zfs_rmnode(znode_t *zp)
	{
	zfsvfs_t *zfsvfs = ZTOZSB(zp);
	objset_t *os = zfsvfs->z_os;
	znode_t *xzp = NULL;
	dmu_tx_t *tx;
	uint64_t acl_obj;
	uint64_t xattr_obj;
	uint64_t links;
	int error;

	ASSERT(ZTOI(zp)->i_nlink == 0);
	ASSERT(atomic_read(&ZTOI(zp)->i_count) == 0);

	/*
	* If this is an attribute directory, purge its contents.
	*/
	if (S_ISDIR(ZTOI(zp)->i_mode) && (zp->z_pflags & ZFS_XATTR)) {
	if (zfs_purgedir(zp) != 0) {
	/*
	* Not enough space to delete some xattrs.
	* Leave it in the unlinked set.
	*/
	zfs_znode_dmu_fini(zp);

	return;
	}
	}

	/*
	* Free up all the data in the file. We don't do this for directories
	* because we need truncate and remove to be in the same tx, like in
	* zfs_znode_delete(). Otherwise, if we crash here we'll end up with
	* an inconsistent truncated zap object in the delete queue. Note a
	* truncated file is harmless since it only contains user data.
	*/
	if (S_ISREG(ZTOI(zp)->i_mode)) {
	error = dmu_free_long_range(os, zp->z_id, 0, DMU_OBJECT_END);
	if (error) {
	/*
	* Not enough space or we were interrupted by unmount.
	* Leave the file in the unlinked set.
	*/
	zfs_znode_dmu_fini(zp);
	return;
	}
	}

	/*
	* If the file has extended attributes, we're going to unlink
	* the xattr dir.
	*/
	error = sa_lookup(zp->z_sa_hdl, SA_ZPL_XATTR(zfsvfs),
	&xattr_obj, sizeof (xattr_obj));
	if (error == 0 && xattr_obj) {
	error = zfs_zget(zfsvfs, xattr_obj, &xzp);
	ASSERT(error == 0);
	}

	acl_obj = zfs_external_acl(zp);

	/*
	* Set up the final transaction.
	*/
	tx = dmu_tx_create(os);
	dmu_tx_hold_free(tx, zp->z_id, 0, DMU_OBJECT_END);
	dmu_tx_hold_zap(tx, zfsvfs->z_unlinkedobj, FALSE, NULL);
	if (xzp) {
	dmu_tx_hold_zap(tx, zfsvfs->z_unlinkedobj, TRUE, NULL);
	dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE);
	}
	if (acl_obj)
	dmu_tx_hold_free(tx, acl_obj, 0, DMU_OBJECT_END);

	zfs_sa_upgrade_txholds(tx, zp);
	error = dmu_tx_assign(tx, TXG_WAIT);
	if (error) {
	/*
	* Not enough space to delete the file. Leave it in the
	* unlinked set, leaking it until the fs is remounted (at
	* which point we'll call zfs_unlinked_drain() to process it).
	*/
	dmu_tx_abort(tx);
	zfs_znode_dmu_fini(zp);
	goto out;
	}

	if (xzp) {
	ASSERT(error == 0);
	mutex_enter(&xzp->z_lock);
	xzp->z_unlinked = B_TRUE; /* mark xzp for deletion */
	clear_nlink(ZTOI(xzp)); /* no more links to it */
	links = 0;
	VERIFY(0 == sa_update(xzp->z_sa_hdl, SA_ZPL_LINKS(zfsvfs),
	&links, sizeof (links), tx));
	mutex_exit(&xzp->z_lock);
	zfs_unlinked_add(xzp, tx);
	}

	mutex_enter(&os->os_dsl_dataset->ds_dir->dd_activity_lock);

	/*
	* Remove this znode from the unlinked set. If a has rollback has
	* occurred while a file is open and unlinked. Then when the file
	* is closed post rollback it will not exist in the rolled back
	* version of the unlinked object.
	*/
	error = zap_remove_int(zfsvfs->z_os, zfsvfs->z_unlinkedobj,
	zp->z_id, tx);
	VERIFY(error == 0 \|\| error == ENOENT);

	uint64_t count;
	if (zap_count(os, zfsvfs->z_unlinkedobj, &count) == 0 && count == 0) {
	cv_broadcast(&os->os_dsl_dataset->ds_dir->dd_activity_cv);
	}

	mutex_exit(&os->os_dsl_dataset->ds_dir->dd_activity_lock);

	dataset_kstats_update_nunlinked_kstat(&zfsvfs->z_kstat, 1);

	zfs_znode_delete(zp, tx);

	dmu_tx_commit(tx);
	out:
	if (xzp)
	zfs_zrele_async(xzp);
	}

	static uint64_t
	zfs_dirent(znode_t *zp, uint64_t mode)
	{
	uint64_t de = zp->z_id;

	if (ZTOZSB(zp)->z_version >= ZPL_VERSION_DIRENT_TYPE)
	de \|= IFTODT(mode) << 60;
	return (de);
	}

	/*
	* Link zp into dl. Can fail in the following cases :
	* - if zp has been unlinked.
	* - if the number of entries with the same hash (aka. colliding entries)
	* exceed the capacity of a leaf-block of fatzap and splitting of the
	* leaf-block does not help.
	*/
	int
	zfs_link_create(zfs_dirlock_t dl, znode_t zp, dmu_tx_t *tx, int flag)
	{
	znode_t *dzp = dl->dl_dzp;
	zfsvfs_t *zfsvfs = ZTOZSB(zp);
	uint64_t value;
	int zp_is_dir = S_ISDIR(ZTOI(zp)->i_mode);
	sa_bulk_attr_t bulk[5];
	uint64_t mtime[2], ctime[2];
	uint64_t links;
	int count = 0;
	int error;

	mutex_enter(&zp->z_lock);

	if (!(flag & ZRENAMING)) {
	if (zp->z_unlinked) { /* no new links to unlinked zp */
	ASSERT(!(flag & (ZNEW \| ZEXISTS)));
	mutex_exit(&zp->z_lock);
	return (SET_ERROR(ENOENT));
	}
	if (!(flag & ZNEW)) {
	/*
	* ZNEW nodes come from zfs_mknode() where the link
	* count has already been initialised
	*/
	inc_nlink(ZTOI(zp));
	links = ZTOI(zp)->i_nlink;
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_LINKS(zfsvfs),
	NULL, &links, sizeof (links));
	}
	}

	value = zfs_dirent(zp, zp->z_mode);
	error = zap_add(ZTOZSB(zp)->z_os, dzp->z_id, dl->dl_name, 8, 1,
	&value, tx);

	/*
	* zap_add could fail to add the entry if it exceeds the capacity of the
	* leaf-block and zap_leaf_split() failed to help.
	* The caller of this routine is responsible for failing the transaction
	* which will rollback the SA updates done above.
	*/
	if (error != 0) {
	if (!(flag & ZRENAMING) && !(flag & ZNEW))
	drop_nlink(ZTOI(zp));
	mutex_exit(&zp->z_lock);
	return (error);
	}

	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_PARENT(zfsvfs), NULL,
	&dzp->z_id, sizeof (dzp->z_id));
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_FLAGS(zfsvfs), NULL,
	&zp->z_pflags, sizeof (zp->z_pflags));

	if (!(flag & ZNEW)) {
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_CTIME(zfsvfs), NULL,
	ctime, sizeof (ctime));
	zfs_tstamp_update_setup(zp, STATE_CHANGED, mtime,
	ctime);
	}
	error = sa_bulk_update(zp->z_sa_hdl, bulk, count, tx);
	ASSERT(error == 0);

	mutex_exit(&zp->z_lock);

	mutex_enter(&dzp->z_lock);
	dzp->z_size++;
	if (zp_is_dir)
	inc_nlink(ZTOI(dzp));
	links = ZTOI(dzp)->i_nlink;
	count = 0;
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_SIZE(zfsvfs), NULL,
	&dzp->z_size, sizeof (dzp->z_size));
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_LINKS(zfsvfs), NULL,
	&links, sizeof (links));
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_MTIME(zfsvfs), NULL,
	mtime, sizeof (mtime));
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_CTIME(zfsvfs), NULL,
	ctime, sizeof (ctime));
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_FLAGS(zfsvfs), NULL,
	&dzp->z_pflags, sizeof (dzp->z_pflags));
	zfs_tstamp_update_setup(dzp, CONTENT_MODIFIED, mtime, ctime);
	error = sa_bulk_update(dzp->z_sa_hdl, bulk, count, tx);
	ASSERT(error == 0);
	mutex_exit(&dzp->z_lock);

	return (0);
	}

	/*
	* The match type in the code for this function should conform to:
	*
	* ------------------------------------------------------------------------
	* fs type \| z_norm \| lookup type \| match type
	* ---------\|-------------\|-------------\|----------------------------------
	* CS !norm \| 0 \| 0 \| 0 (exact)
	* CS norm \| formX \| 0 \| MT_NORMALIZE
	* CI !norm \| upper \| !ZCIEXACT \| MT_NORMALIZE
	* CI !norm \| upper \| ZCIEXACT \| MT_NORMALIZE \| MT_MATCH_CASE
	* CI norm \| upper\|formX \| !ZCIEXACT \| MT_NORMALIZE
	* CI norm \| upper\|formX \| ZCIEXACT \| MT_NORMALIZE \| MT_MATCH_CASE
	* CM !norm \| upper \| !ZCILOOK \| MT_NORMALIZE \| MT_MATCH_CASE
	* CM !norm \| upper \| ZCILOOK \| MT_NORMALIZE
	* CM norm \| upper\|formX \| !ZCILOOK \| MT_NORMALIZE \| MT_MATCH_CASE
	* CM norm \| upper\|formX \| ZCILOOK \| MT_NORMALIZE
	*
	* Abbreviations:
	* CS = Case Sensitive, CI = Case Insensitive, CM = Case Mixed
	* upper = case folding set by fs type on creation (U8_TEXTPREP_TOUPPER)
	* formX = unicode normalization form set on fs creation
	*/
	static int
	zfs_dropname(zfs_dirlock_t dl, znode_t zp, znode_t dzp, dmu_tx_t tx,
	int flag)
	{
	int error;

	if (ZTOZSB(zp)->z_norm) {
	matchtype_t mt = MT_NORMALIZE;

	if ((ZTOZSB(zp)->z_case == ZFS_CASE_INSENSITIVE &&
	(flag & ZCIEXACT)) \|\|
	(ZTOZSB(zp)->z_case == ZFS_CASE_MIXED &&
	!(flag & ZCILOOK))) {
	mt \|= MT_MATCH_CASE;
	}

	error = zap_remove_norm(ZTOZSB(zp)->z_os, dzp->z_id,
	dl->dl_name, mt, tx);
	} else {
	error = zap_remove(ZTOZSB(zp)->z_os, dzp->z_id, dl->dl_name,
	tx);
	}

	return (error);
	}

	/*
	* Unlink zp from dl, and mark zp for deletion if this was the last link. Can
	* fail if zp is a mount point (EBUSY) or a non-empty directory (ENOTEMPTY).
	* If 'unlinkedp' is NULL, we put unlinked znodes on the unlinked list.
	* If it's non-NULL, we use it to indicate whether the znode needs deletion,
	* and it's the caller's job to do it.
	*/
	int
	zfs_link_destroy(zfs_dirlock_t dl, znode_t zp, dmu_tx_t *tx, int flag,
	boolean_t *unlinkedp)
	{
	znode_t *dzp = dl->dl_dzp;
	zfsvfs_t *zfsvfs = ZTOZSB(dzp);
	int zp_is_dir = S_ISDIR(ZTOI(zp)->i_mode);
	boolean_t unlinked = B_FALSE;
	sa_bulk_attr_t bulk[5];
	uint64_t mtime[2], ctime[2];
	uint64_t links;
	int count = 0;
	int error;

	if (!(flag & ZRENAMING)) {
	mutex_enter(&zp->z_lock);

	if (zp_is_dir && !zfs_dirempty(zp)) {
	mutex_exit(&zp->z_lock);
	return (SET_ERROR(ENOTEMPTY));
	}

	/*
	* If we get here, we are going to try to remove the object.
	* First try removing the name from the directory; if that
	* fails, return the error.
	*/
	error = zfs_dropname(dl, zp, dzp, tx, flag);
	if (error != 0) {
	mutex_exit(&zp->z_lock);
	return (error);
	}

	if (ZTOI(zp)->i_nlink <= zp_is_dir) {
	zfs_panic_recover("zfs: link count on %lu is %u, "
	"should be at least %u", zp->z_id,
	(int)ZTOI(zp)->i_nlink, zp_is_dir + 1);
	set_nlink(ZTOI(zp), zp_is_dir + 1);
	}
	drop_nlink(ZTOI(zp));
	if (ZTOI(zp)->i_nlink == zp_is_dir) {
	zp->z_unlinked = B_TRUE;
	clear_nlink(ZTOI(zp));
	unlinked = B_TRUE;
	} else {
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_CTIME(zfsvfs),
	NULL, &ctime, sizeof (ctime));
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_FLAGS(zfsvfs),
	NULL, &zp->z_pflags, sizeof (zp->z_pflags));
	zfs_tstamp_update_setup(zp, STATE_CHANGED, mtime,
	ctime);
	}
	links = ZTOI(zp)->i_nlink;
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_LINKS(zfsvfs),
	NULL, &links, sizeof (links));
	error = sa_bulk_update(zp->z_sa_hdl, bulk, count, tx);
	count = 0;
	ASSERT(error == 0);
	mutex_exit(&zp->z_lock);
	} else {
	error = zfs_dropname(dl, zp, dzp, tx, flag);
	if (error != 0)
	return (error);
	}

	mutex_enter(&dzp->z_lock);
	dzp->z_size--; /* one dirent removed */
	if (zp_is_dir)
	drop_nlink(ZTOI(dzp)); /* ".." link from zp */
	links = ZTOI(dzp)->i_nlink;
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_LINKS(zfsvfs),
	NULL, &links, sizeof (links));
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_SIZE(zfsvfs),
	NULL, &dzp->z_size, sizeof (dzp->z_size));
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_CTIME(zfsvfs),
	NULL, ctime, sizeof (ctime));
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_MTIME(zfsvfs),
	NULL, mtime, sizeof (mtime));
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_FLAGS(zfsvfs),
	NULL, &dzp->z_pflags, sizeof (dzp->z_pflags));
	zfs_tstamp_update_setup(dzp, CONTENT_MODIFIED, mtime, ctime);
	error = sa_bulk_update(dzp->z_sa_hdl, bulk, count, tx);
	ASSERT(error == 0);
	mutex_exit(&dzp->z_lock);

	if (unlinkedp != NULL)
	*unlinkedp = unlinked;
	else if (unlinked)
	zfs_unlinked_add(zp, tx);

	return (0);
	}

	/*
	* Indicate whether the directory is empty. Works with or without z_lock
	* held, but can only be consider a hint in the latter case. Returns true
	* if only "." and ".." remain and there's no work in progress.
	*
	* The internal ZAP size, rather than zp->z_size, needs to be checked since
	* some consumers (Lustre) do not strictly maintain an accurate SA_ZPL_SIZE.
	*/
	boolean_t
	zfs_dirempty(znode_t *dzp)
	{
	zfsvfs_t *zfsvfs = ZTOZSB(dzp);
	uint64_t count;
	int error;

	if (dzp->z_dirlocks != NULL)
	return (B_FALSE);

	error = zap_count(zfsvfs->z_os, dzp->z_id, &count);
	if (error != 0 \|\| count != 0)
	return (B_FALSE);

	return (B_TRUE);
	}

	int
	zfs_make_xattrdir(znode_t zp, vattr_t vap, znode_t *xzpp, cred_t cr)
	{
	zfsvfs_t *zfsvfs = ZTOZSB(zp);
	znode_t *xzp;
	dmu_tx_t *tx;
	int error;
	zfs_acl_ids_t acl_ids;
	boolean_t fuid_dirtied;
	#ifdef ZFS_DEBUG
	uint64_t parent;
	#endif

	*xzpp = NULL;

	if ((error = zfs_zaccess(zp, ACE_WRITE_NAMED_ATTRS, 0, B_FALSE, cr)))
	return (error);

	if ((error = zfs_acl_ids_create(zp, IS_XATTR, vap, cr, NULL,
	&acl_ids)) != 0)
	return (error);
	if (zfs_acl_ids_overquota(zfsvfs, &acl_ids, zp->z_projid)) {
	zfs_acl_ids_free(&acl_ids);
	return (SET_ERROR(EDQUOT));
	}

	tx = dmu_tx_create(zfsvfs->z_os);
	dmu_tx_hold_sa_create(tx, acl_ids.z_aclp->z_acl_bytes +
	ZFS_SA_BASE_ATTR_SIZE);
	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_TRUE);
	dmu_tx_hold_zap(tx, DMU_NEW_OBJECT, FALSE, NULL);
	fuid_dirtied = zfsvfs->z_fuid_dirty;
	if (fuid_dirtied)
	zfs_fuid_txhold(zfsvfs, tx);
	error = dmu_tx_assign(tx, TXG_WAIT);
	if (error) {
	zfs_acl_ids_free(&acl_ids);
	dmu_tx_abort(tx);
	return (error);
	}
	zfs_mknode(zp, vap, tx, cr, IS_XATTR, &xzp, &acl_ids);

	if (fuid_dirtied)
	zfs_fuid_sync(zfsvfs, tx);

	#ifdef ZFS_DEBUG
	error = sa_lookup(xzp->z_sa_hdl, SA_ZPL_PARENT(zfsvfs),
	&parent, sizeof (parent));
	ASSERT(error == 0 && parent == zp->z_id);
	#endif

	VERIFY(0 == sa_update(zp->z_sa_hdl, SA_ZPL_XATTR(zfsvfs), &xzp->z_id,
	sizeof (xzp->z_id), tx));

	if (!zp->z_unlinked)
	(void) zfs_log_create(zfsvfs->z_log, tx, TX_MKXATTR, zp,
	xzp, "", NULL, acl_ids.z_fuidp, vap);

	zfs_acl_ids_free(&acl_ids);
	dmu_tx_commit(tx);

	*xzpp = xzp;

	return (0);
	}

	/*
	* Return a znode for the extended attribute directory for zp.
	* If the directory does not already exist, it is created
	*
	* IN: zp - znode to obtain attribute directory from
	* cr - credentials of caller
	* flags - flags from the VOP_LOOKUP call
	*
	* OUT: xipp - pointer to extended attribute znode
	*
	* RETURN: 0 on success
	* error number on failure
	*/
	int
	zfs_get_xattrdir(znode_t zp, znode_t xzpp, cred_t cr, int flags)
	{
	zfsvfs_t *zfsvfs = ZTOZSB(zp);
	znode_t *xzp;
	zfs_dirlock_t *dl;
	vattr_t va;
	int error;
	top:
	error = zfs_dirent_lock(&dl, zp, "", &xzp, ZXATTR, NULL, NULL);
	if (error)
	return (error);

	if (xzp != NULL) {
	*xzpp = xzp;
	zfs_dirent_unlock(dl);
	return (0);
	}

	if (!(flags & CREATE_XATTR_DIR)) {
	zfs_dirent_unlock(dl);
	return (SET_ERROR(ENOENT));
	}

	if (zfs_is_readonly(zfsvfs)) {
	zfs_dirent_unlock(dl);
	return (SET_ERROR(EROFS));
	}

	/*
	* The ability to 'create' files in an attribute
	* directory comes from the write_xattr permission on the base file.
	*
	* The ability to 'search' an attribute directory requires
	* read_xattr permission on the base file.
	*
	* Once in a directory the ability to read/write attributes
	* is controlled by the permissions on the attribute file.
	*/
	va.va_mask = ATTR_MODE \| ATTR_UID \| ATTR_GID;
	va.va_mode = S_IFDIR \| S_ISVTX \| 0777;
	zfs_fuid_map_ids(zp, cr, &va.va_uid, &va.va_gid);

	va.va_dentry = NULL;
	error = zfs_make_xattrdir(zp, &va, xzpp, cr);
	zfs_dirent_unlock(dl);

	if (error == ERESTART) {
	/* NB: we already did dmu_tx_wait() if necessary */
	goto top;
	}

	return (error);
	}

	/*
	* Decide whether it is okay to remove within a sticky directory.
	*
	* In sticky directories, write access is not sufficient;
	* you can remove entries from a directory only if:
	*
	* you own the directory,
	* you own the entry,
	* you have write access to the entry,
	* or you are privileged (checked in secpolicy...).
	*
	* The function returns 0 if remove access is granted.
	*/
	int
	zfs_sticky_remove_access(znode_t zdp, znode_t zp, cred_t *cr)
	{
	uid_t uid;
	uid_t downer;
	uid_t fowner;
	zfsvfs_t *zfsvfs = ZTOZSB(zdp);

	if (zfsvfs->z_replay)
	return (0);

	if ((zdp->z_mode & S_ISVTX) == 0)
	return (0);

	downer = zfs_fuid_map_id(zfsvfs, KUID_TO_SUID(ZTOI(zdp)->i_uid),
	cr, ZFS_OWNER);
	fowner = zfs_fuid_map_id(zfsvfs, KUID_TO_SUID(ZTOI(zp)->i_uid),
	cr, ZFS_OWNER);

	if ((uid = crgetuid(cr)) == downer \|\| uid == fowner \|\|
	zfs_zaccess(zp, ACE_WRITE_DATA, 0, B_FALSE, cr) == 0)
	return (0);
	else
	return (secpolicy_vnode_remove(cr));
	}
	Index: vendor-sys/openzfs/dist/module/os/linux/zfs/zfs_vfsops.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/zfs/zfs_vfsops.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/zfs/zfs_vfsops.c (revision 366774)
	@@ -1,2175 +1,2176 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2012, 2018 by Delphix. All rights reserved.
	*/

	/* Portions Copyright 2010 Robert Milkowski */

	#include <sys/types.h>
	#include <sys/param.h>
	#include <sys/sysmacros.h>
	#include <sys/kmem.h>
	#include <sys/pathname.h>
	#include <sys/vnode.h>
	#include <sys/vfs.h>
	#include <sys/mntent.h>
	#include <sys/cmn_err.h>
	#include <sys/zfs_znode.h>
	#include <sys/zfs_vnops.h>
	#include <sys/zfs_dir.h>
	#include <sys/zil.h>
	#include <sys/fs/zfs.h>
	#include <sys/dmu.h>
	#include <sys/dsl_prop.h>
	#include <sys/dsl_dataset.h>
	#include <sys/dsl_deleg.h>
	#include <sys/spa.h>
	#include <sys/zap.h>
	#include <sys/sa.h>
	#include <sys/sa_impl.h>
	#include <sys/policy.h>
	#include <sys/atomic.h>
	#include <sys/zfs_ioctl.h>
	#include <sys/zfs_ctldir.h>
	#include <sys/zfs_fuid.h>
	#include <sys/zfs_quota.h>
	#include <sys/sunddi.h>
	#include <sys/dmu_objset.h>
	#include <sys/dsl_dir.h>
	#include <sys/spa_boot.h>
	#include <sys/objlist.h>
	#include <sys/zpl.h>
	#include <linux/vfs_compat.h>
	#include "zfs_comutil.h"

	enum {
	TOKEN_RO,
	TOKEN_RW,
	TOKEN_SETUID,
	TOKEN_NOSETUID,
	TOKEN_EXEC,
	TOKEN_NOEXEC,
	TOKEN_DEVICES,
	TOKEN_NODEVICES,
	TOKEN_DIRXATTR,
	TOKEN_SAXATTR,
	TOKEN_XATTR,
	TOKEN_NOXATTR,
	TOKEN_ATIME,
	TOKEN_NOATIME,
	TOKEN_RELATIME,
	TOKEN_NORELATIME,
	TOKEN_NBMAND,
	TOKEN_NONBMAND,
	TOKEN_MNTPOINT,
	TOKEN_LAST,
	};

	static const match_table_t zpl_tokens = {
	{ TOKEN_RO, MNTOPT_RO },
	{ TOKEN_RW, MNTOPT_RW },
	{ TOKEN_SETUID, MNTOPT_SETUID },
	{ TOKEN_NOSETUID, MNTOPT_NOSETUID },
	{ TOKEN_EXEC, MNTOPT_EXEC },
	{ TOKEN_NOEXEC, MNTOPT_NOEXEC },
	{ TOKEN_DEVICES, MNTOPT_DEVICES },
	{ TOKEN_NODEVICES, MNTOPT_NODEVICES },
	{ TOKEN_DIRXATTR, MNTOPT_DIRXATTR },
	{ TOKEN_SAXATTR, MNTOPT_SAXATTR },
	{ TOKEN_XATTR, MNTOPT_XATTR },
	{ TOKEN_NOXATTR, MNTOPT_NOXATTR },
	{ TOKEN_ATIME, MNTOPT_ATIME },
	{ TOKEN_NOATIME, MNTOPT_NOATIME },
	{ TOKEN_RELATIME, MNTOPT_RELATIME },
	{ TOKEN_NORELATIME, MNTOPT_NORELATIME },
	{ TOKEN_NBMAND, MNTOPT_NBMAND },
	{ TOKEN_NONBMAND, MNTOPT_NONBMAND },
	{ TOKEN_MNTPOINT, MNTOPT_MNTPOINT "=%s" },
	{ TOKEN_LAST, NULL },
	};

	static void
	zfsvfs_vfs_free(vfs_t *vfsp)
	{
	if (vfsp != NULL) {
	if (vfsp->vfs_mntpoint != NULL)
	kmem_strfree(vfsp->vfs_mntpoint);

	kmem_free(vfsp, sizeof (vfs_t));
	}
	}

	static int
	zfsvfs_parse_option(char option, int token, substring_t args, vfs_t *vfsp)
	{
	switch (token) {
	case TOKEN_RO:
	vfsp->vfs_readonly = B_TRUE;
	vfsp->vfs_do_readonly = B_TRUE;
	break;
	case TOKEN_RW:
	vfsp->vfs_readonly = B_FALSE;
	vfsp->vfs_do_readonly = B_TRUE;
	break;
	case TOKEN_SETUID:
	vfsp->vfs_setuid = B_TRUE;
	vfsp->vfs_do_setuid = B_TRUE;
	break;
	case TOKEN_NOSETUID:
	vfsp->vfs_setuid = B_FALSE;
	vfsp->vfs_do_setuid = B_TRUE;
	break;
	case TOKEN_EXEC:
	vfsp->vfs_exec = B_TRUE;
	vfsp->vfs_do_exec = B_TRUE;
	break;
	case TOKEN_NOEXEC:
	vfsp->vfs_exec = B_FALSE;
	vfsp->vfs_do_exec = B_TRUE;
	break;
	case TOKEN_DEVICES:
	vfsp->vfs_devices = B_TRUE;
	vfsp->vfs_do_devices = B_TRUE;
	break;
	case TOKEN_NODEVICES:
	vfsp->vfs_devices = B_FALSE;
	vfsp->vfs_do_devices = B_TRUE;
	break;
	case TOKEN_DIRXATTR:
	vfsp->vfs_xattr = ZFS_XATTR_DIR;
	vfsp->vfs_do_xattr = B_TRUE;
	break;
	case TOKEN_SAXATTR:
	vfsp->vfs_xattr = ZFS_XATTR_SA;
	vfsp->vfs_do_xattr = B_TRUE;
	break;
	case TOKEN_XATTR:
	vfsp->vfs_xattr = ZFS_XATTR_DIR;
	vfsp->vfs_do_xattr = B_TRUE;
	break;
	case TOKEN_NOXATTR:
	vfsp->vfs_xattr = ZFS_XATTR_OFF;
	vfsp->vfs_do_xattr = B_TRUE;
	break;
	case TOKEN_ATIME:
	vfsp->vfs_atime = B_TRUE;
	vfsp->vfs_do_atime = B_TRUE;
	break;
	case TOKEN_NOATIME:
	vfsp->vfs_atime = B_FALSE;
	vfsp->vfs_do_atime = B_TRUE;
	break;
	case TOKEN_RELATIME:
	vfsp->vfs_relatime = B_TRUE;
	vfsp->vfs_do_relatime = B_TRUE;
	break;
	case TOKEN_NORELATIME:
	vfsp->vfs_relatime = B_FALSE;
	vfsp->vfs_do_relatime = B_TRUE;
	break;
	case TOKEN_NBMAND:
	vfsp->vfs_nbmand = B_TRUE;
	vfsp->vfs_do_nbmand = B_TRUE;
	break;
	case TOKEN_NONBMAND:
	vfsp->vfs_nbmand = B_FALSE;
	vfsp->vfs_do_nbmand = B_TRUE;
	break;
	case TOKEN_MNTPOINT:
	vfsp->vfs_mntpoint = match_strdup(&args[0]);
	if (vfsp->vfs_mntpoint == NULL)
	return (SET_ERROR(ENOMEM));

	break;
	default:
	break;
	}

	return (0);
	}

	/*
	* Parse the raw mntopts and return a vfs_t describing the options.
	*/
	static int
	zfsvfs_parse_options(char mntopts, vfs_t *vfsp)
	{
	vfs_t *tmp_vfsp;
	int error;

	tmp_vfsp = kmem_zalloc(sizeof (vfs_t), KM_SLEEP);

	if (mntopts != NULL) {
	substring_t args[MAX_OPT_ARGS];
	char tmp_mntopts, p, *t;
	int token;

	tmp_mntopts = t = kmem_strdup(mntopts);
	if (tmp_mntopts == NULL)
	return (SET_ERROR(ENOMEM));

	while ((p = strsep(&t, ",")) != NULL) {
	if (!*p)
	continue;

	args[0].to = args[0].from = NULL;
	token = match_token(p, zpl_tokens, args);
	error = zfsvfs_parse_option(p, token, args, tmp_vfsp);
	if (error) {
	kmem_strfree(tmp_mntopts);
	zfsvfs_vfs_free(tmp_vfsp);
	return (error);
	}
	}

	kmem_strfree(tmp_mntopts);
	}

	*vfsp = tmp_vfsp;

	return (0);
	}

	boolean_t
	zfs_is_readonly(zfsvfs_t *zfsvfs)
	{
	return (!!(zfsvfs->z_sb->s_flags & SB_RDONLY));
	}

	/ARGSUSED/
	int
	zfs_sync(struct super_block sb, int wait, cred_t cr)
	{
	zfsvfs_t *zfsvfs = sb->s_fs_info;

	/*
	* Semantically, the only requirement is that the sync be initiated.
	* The DMU syncs out txgs frequently, so there's nothing to do.
	*/
	if (!wait)
	return (0);

	if (zfsvfs != NULL) {
	/*
	* Sync a specific filesystem.
	*/
	dsl_pool_t *dp;

	ZFS_ENTER(zfsvfs);
	dp = dmu_objset_pool(zfsvfs->z_os);

	/*
	* If the system is shutting down, then skip any
	* filesystems which may exist on a suspended pool.
	*/
	if (spa_suspended(dp->dp_spa)) {
	ZFS_EXIT(zfsvfs);
	return (0);
	}

	if (zfsvfs->z_log != NULL)
	zil_commit(zfsvfs->z_log, 0);

	ZFS_EXIT(zfsvfs);
	} else {
	/*
	* Sync all ZFS filesystems. This is what happens when you
	* run sync(1M). Unlike other filesystems, ZFS honors the
	* request by waiting for all pools to commit all dirty data.
	*/
	spa_sync_allpools();
	}

	return (0);
	}

	static void
	atime_changed_cb(void *arg, uint64_t newval)
	{
	zfsvfs_t *zfsvfs = arg;
	struct super_block *sb = zfsvfs->z_sb;

	if (sb == NULL)
	return;
	/*
	* Update SB_NOATIME bit in VFS super block. Since atime update is
	* determined by atime_needs_update(), atime_needs_update() needs to
	* return false if atime is turned off, and not unconditionally return
	* false if atime is turned on.
	*/
	if (newval)
	sb->s_flags &= ~SB_NOATIME;
	else
	sb->s_flags \|= SB_NOATIME;
	}

	static void
	relatime_changed_cb(void *arg, uint64_t newval)
	{
	((zfsvfs_t *)arg)->z_relatime = newval;
	}

	static void
	xattr_changed_cb(void *arg, uint64_t newval)
	{
	zfsvfs_t *zfsvfs = arg;

	if (newval == ZFS_XATTR_OFF) {
	zfsvfs->z_flags &= ~ZSB_XATTR;
	} else {
	zfsvfs->z_flags \|= ZSB_XATTR;

	if (newval == ZFS_XATTR_SA)
	zfsvfs->z_xattr_sa = B_TRUE;
	else
	zfsvfs->z_xattr_sa = B_FALSE;
	}
	}

	static void
	acltype_changed_cb(void *arg, uint64_t newval)
	{
	zfsvfs_t *zfsvfs = arg;

	switch (newval) {
	+ case ZFS_ACLTYPE_NFSV4:
	case ZFS_ACLTYPE_OFF:
	zfsvfs->z_acl_type = ZFS_ACLTYPE_OFF;
	zfsvfs->z_sb->s_flags &= ~SB_POSIXACL;
	break;
	case ZFS_ACLTYPE_POSIX:
	#ifdef CONFIG_FS_POSIX_ACL
	zfsvfs->z_acl_type = ZFS_ACLTYPE_POSIX;
	zfsvfs->z_sb->s_flags \|= SB_POSIXACL;
	#else
	zfsvfs->z_acl_type = ZFS_ACLTYPE_OFF;
	zfsvfs->z_sb->s_flags &= ~SB_POSIXACL;
	#endif /* CONFIG_FS_POSIX_ACL */
	break;
	default:
	break;
	}
	}

	static void
	blksz_changed_cb(void *arg, uint64_t newval)
	{
	zfsvfs_t *zfsvfs = arg;
	ASSERT3U(newval, <=, spa_maxblocksize(dmu_objset_spa(zfsvfs->z_os)));
	ASSERT3U(newval, >=, SPA_MINBLOCKSIZE);
	ASSERT(ISP2(newval));

	zfsvfs->z_max_blksz = newval;
	}

	static void
	readonly_changed_cb(void *arg, uint64_t newval)
	{
	zfsvfs_t *zfsvfs = arg;
	struct super_block *sb = zfsvfs->z_sb;

	if (sb == NULL)
	return;

	if (newval)
	sb->s_flags \|= SB_RDONLY;
	else
	sb->s_flags &= ~SB_RDONLY;
	}

	static void
	devices_changed_cb(void *arg, uint64_t newval)
	{
	}

	static void
	setuid_changed_cb(void *arg, uint64_t newval)
	{
	}

	static void
	exec_changed_cb(void *arg, uint64_t newval)
	{
	}

	static void
	nbmand_changed_cb(void *arg, uint64_t newval)
	{
	zfsvfs_t *zfsvfs = arg;
	struct super_block *sb = zfsvfs->z_sb;

	if (sb == NULL)
	return;

	if (newval == TRUE)
	sb->s_flags \|= SB_MANDLOCK;
	else
	sb->s_flags &= ~SB_MANDLOCK;
	}

	static void
	snapdir_changed_cb(void *arg, uint64_t newval)
	{
	((zfsvfs_t *)arg)->z_show_ctldir = newval;
	}

	static void
	vscan_changed_cb(void *arg, uint64_t newval)
	{
	((zfsvfs_t *)arg)->z_vscan = newval;
	}

	static void
	acl_mode_changed_cb(void *arg, uint64_t newval)
	{
	zfsvfs_t *zfsvfs = arg;

	zfsvfs->z_acl_mode = newval;
	}

	static void
	acl_inherit_changed_cb(void *arg, uint64_t newval)
	{
	((zfsvfs_t *)arg)->z_acl_inherit = newval;
	}

	static int
	zfs_register_callbacks(vfs_t *vfsp)
	{
	struct dsl_dataset *ds = NULL;
	objset_t *os = NULL;
	zfsvfs_t *zfsvfs = NULL;
	int error = 0;

	ASSERT(vfsp);
	zfsvfs = vfsp->vfs_data;
	ASSERT(zfsvfs);
	os = zfsvfs->z_os;

	/*
	* The act of registering our callbacks will destroy any mount
	* options we may have. In order to enable temporary overrides
	* of mount options, we stash away the current values and
	* restore them after we register the callbacks.
	*/
	if (zfs_is_readonly(zfsvfs) \|\| !spa_writeable(dmu_objset_spa(os))) {
	vfsp->vfs_do_readonly = B_TRUE;
	vfsp->vfs_readonly = B_TRUE;
	}

	/*
	* Register property callbacks.
	*
	* It would probably be fine to just check for i/o error from
	* the first prop_register(), but I guess I like to go
	* overboard...
	*/
	ds = dmu_objset_ds(os);
	dsl_pool_config_enter(dmu_objset_pool(os), FTAG);
	error = dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_ATIME), atime_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_RELATIME), relatime_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_XATTR), xattr_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_RECORDSIZE), blksz_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_READONLY), readonly_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_DEVICES), devices_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_SETUID), setuid_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_EXEC), exec_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_SNAPDIR), snapdir_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_ACLTYPE), acltype_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_ACLMODE), acl_mode_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_ACLINHERIT), acl_inherit_changed_cb,
	zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_VSCAN), vscan_changed_cb, zfsvfs);
	error = error ? error : dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_NBMAND), nbmand_changed_cb, zfsvfs);
	dsl_pool_config_exit(dmu_objset_pool(os), FTAG);
	if (error)
	goto unregister;

	/*
	* Invoke our callbacks to restore temporary mount options.
	*/
	if (vfsp->vfs_do_readonly)
	readonly_changed_cb(zfsvfs, vfsp->vfs_readonly);
	if (vfsp->vfs_do_setuid)
	setuid_changed_cb(zfsvfs, vfsp->vfs_setuid);
	if (vfsp->vfs_do_exec)
	exec_changed_cb(zfsvfs, vfsp->vfs_exec);
	if (vfsp->vfs_do_devices)
	devices_changed_cb(zfsvfs, vfsp->vfs_devices);
	if (vfsp->vfs_do_xattr)
	xattr_changed_cb(zfsvfs, vfsp->vfs_xattr);
	if (vfsp->vfs_do_atime)
	atime_changed_cb(zfsvfs, vfsp->vfs_atime);
	if (vfsp->vfs_do_relatime)
	relatime_changed_cb(zfsvfs, vfsp->vfs_relatime);
	if (vfsp->vfs_do_nbmand)
	nbmand_changed_cb(zfsvfs, vfsp->vfs_nbmand);

	return (0);

	unregister:
	dsl_prop_unregister_all(ds, zfsvfs);
	return (error);
	}

	/*
	* Takes a dataset, a property, a value and that value's setpoint as
	* found in the ZAP. Checks if the property has been changed in the vfs.
	* If so, val and setpoint will be overwritten with updated content.
	* Otherwise, they are left unchanged.
	*/
	int
	zfs_get_temporary_prop(dsl_dataset_t ds, zfs_prop_t zfs_prop, uint64_t val,
	char *setpoint)
	{
	int error;
	zfsvfs_t *zfvp;
	vfs_t *vfsp;
	objset_t *os;
	uint64_t tmp = *val;

	error = dmu_objset_from_ds(ds, &os);
	if (error != 0)
	return (error);

	if (dmu_objset_type(os) != DMU_OST_ZFS)
	return (EINVAL);

	mutex_enter(&os->os_user_ptr_lock);
	zfvp = dmu_objset_get_user(os);
	mutex_exit(&os->os_user_ptr_lock);
	if (zfvp == NULL)
	return (ESRCH);

	vfsp = zfvp->z_vfs;

	switch (zfs_prop) {
	case ZFS_PROP_ATIME:
	if (vfsp->vfs_do_atime)
	tmp = vfsp->vfs_atime;
	break;
	case ZFS_PROP_RELATIME:
	if (vfsp->vfs_do_relatime)
	tmp = vfsp->vfs_relatime;
	break;
	case ZFS_PROP_DEVICES:
	if (vfsp->vfs_do_devices)
	tmp = vfsp->vfs_devices;
	break;
	case ZFS_PROP_EXEC:
	if (vfsp->vfs_do_exec)
	tmp = vfsp->vfs_exec;
	break;
	case ZFS_PROP_SETUID:
	if (vfsp->vfs_do_setuid)
	tmp = vfsp->vfs_setuid;
	break;
	case ZFS_PROP_READONLY:
	if (vfsp->vfs_do_readonly)
	tmp = vfsp->vfs_readonly;
	break;
	case ZFS_PROP_XATTR:
	if (vfsp->vfs_do_xattr)
	tmp = vfsp->vfs_xattr;
	break;
	case ZFS_PROP_NBMAND:
	if (vfsp->vfs_do_nbmand)
	tmp = vfsp->vfs_nbmand;
	break;
	default:
	return (ENOENT);
	}

	if (tmp != *val) {
	(void) strcpy(setpoint, "temporary");
	*val = tmp;
	}
	return (0);
	}

	/*
	* Associate this zfsvfs with the given objset, which must be owned.
	* This will cache a bunch of on-disk state from the objset in the
	* zfsvfs.
	*/
	static int
	zfsvfs_init(zfsvfs_t zfsvfs, objset_t os)
	{
	int error;
	uint64_t val;

	zfsvfs->z_max_blksz = SPA_OLD_MAXBLOCKSIZE;
	zfsvfs->z_show_ctldir = ZFS_SNAPDIR_VISIBLE;
	zfsvfs->z_os = os;

	error = zfs_get_zplprop(os, ZFS_PROP_VERSION, &zfsvfs->z_version);
	if (error != 0)
	return (error);
	if (zfsvfs->z_version >
	zfs_zpl_version_map(spa_version(dmu_objset_spa(os)))) {
	(void) printk("Can't mount a version %lld file system "
	"on a version %lld pool\n. Pool must be upgraded to mount "
	"this file system.\n", (u_longlong_t)zfsvfs->z_version,
	(u_longlong_t)spa_version(dmu_objset_spa(os)));
	return (SET_ERROR(ENOTSUP));
	}
	error = zfs_get_zplprop(os, ZFS_PROP_NORMALIZE, &val);
	if (error != 0)
	return (error);
	zfsvfs->z_norm = (int)val;

	error = zfs_get_zplprop(os, ZFS_PROP_UTF8ONLY, &val);
	if (error != 0)
	return (error);
	zfsvfs->z_utf8 = (val != 0);

	error = zfs_get_zplprop(os, ZFS_PROP_CASE, &val);
	if (error != 0)
	return (error);
	zfsvfs->z_case = (uint_t)val;

	if ((error = zfs_get_zplprop(os, ZFS_PROP_ACLTYPE, &val)) != 0)
	return (error);
	zfsvfs->z_acl_type = (uint_t)val;

	/*
	* Fold case on file systems that are always or sometimes case
	* insensitive.
	*/
	if (zfsvfs->z_case == ZFS_CASE_INSENSITIVE \|\|
	zfsvfs->z_case == ZFS_CASE_MIXED)
	zfsvfs->z_norm \|= U8_TEXTPREP_TOUPPER;

	zfsvfs->z_use_fuids = USE_FUIDS(zfsvfs->z_version, zfsvfs->z_os);
	zfsvfs->z_use_sa = USE_SA(zfsvfs->z_version, zfsvfs->z_os);

	uint64_t sa_obj = 0;
	if (zfsvfs->z_use_sa) {
	/* should either have both of these objects or none */
	error = zap_lookup(os, MASTER_NODE_OBJ, ZFS_SA_ATTRS, 8, 1,
	&sa_obj);
	if (error != 0)
	return (error);

	error = zfs_get_zplprop(os, ZFS_PROP_XATTR, &val);
	if ((error == 0) && (val == ZFS_XATTR_SA))
	zfsvfs->z_xattr_sa = B_TRUE;
	}

	error = zap_lookup(os, MASTER_NODE_OBJ, ZFS_ROOT_OBJ, 8, 1,
	&zfsvfs->z_root);
	if (error != 0)
	return (error);
	ASSERT(zfsvfs->z_root != 0);

	error = zap_lookup(os, MASTER_NODE_OBJ, ZFS_UNLINKED_SET, 8, 1,
	&zfsvfs->z_unlinkedobj);
	if (error != 0)
	return (error);

	error = zap_lookup(os, MASTER_NODE_OBJ,
	zfs_userquota_prop_prefixes[ZFS_PROP_USERQUOTA],
	8, 1, &zfsvfs->z_userquota_obj);
	if (error == ENOENT)
	zfsvfs->z_userquota_obj = 0;
	else if (error != 0)
	return (error);

	error = zap_lookup(os, MASTER_NODE_OBJ,
	zfs_userquota_prop_prefixes[ZFS_PROP_GROUPQUOTA],
	8, 1, &zfsvfs->z_groupquota_obj);
	if (error == ENOENT)
	zfsvfs->z_groupquota_obj = 0;
	else if (error != 0)
	return (error);

	error = zap_lookup(os, MASTER_NODE_OBJ,
	zfs_userquota_prop_prefixes[ZFS_PROP_PROJECTQUOTA],
	8, 1, &zfsvfs->z_projectquota_obj);
	if (error == ENOENT)
	zfsvfs->z_projectquota_obj = 0;
	else if (error != 0)
	return (error);

	error = zap_lookup(os, MASTER_NODE_OBJ,
	zfs_userquota_prop_prefixes[ZFS_PROP_USEROBJQUOTA],
	8, 1, &zfsvfs->z_userobjquota_obj);
	if (error == ENOENT)
	zfsvfs->z_userobjquota_obj = 0;
	else if (error != 0)
	return (error);

	error = zap_lookup(os, MASTER_NODE_OBJ,
	zfs_userquota_prop_prefixes[ZFS_PROP_GROUPOBJQUOTA],
	8, 1, &zfsvfs->z_groupobjquota_obj);
	if (error == ENOENT)
	zfsvfs->z_groupobjquota_obj = 0;
	else if (error != 0)
	return (error);

	error = zap_lookup(os, MASTER_NODE_OBJ,
	zfs_userquota_prop_prefixes[ZFS_PROP_PROJECTOBJQUOTA],
	8, 1, &zfsvfs->z_projectobjquota_obj);
	if (error == ENOENT)
	zfsvfs->z_projectobjquota_obj = 0;
	else if (error != 0)
	return (error);

	error = zap_lookup(os, MASTER_NODE_OBJ, ZFS_FUID_TABLES, 8, 1,
	&zfsvfs->z_fuid_obj);
	if (error == ENOENT)
	zfsvfs->z_fuid_obj = 0;
	else if (error != 0)
	return (error);

	error = zap_lookup(os, MASTER_NODE_OBJ, ZFS_SHARES_DIR, 8, 1,
	&zfsvfs->z_shares_dir);
	if (error == ENOENT)
	zfsvfs->z_shares_dir = 0;
	else if (error != 0)
	return (error);

	error = sa_setup(os, sa_obj, zfs_attr_table, ZPL_END,
	&zfsvfs->z_attr_table);
	if (error != 0)
	return (error);

	if (zfsvfs->z_version >= ZPL_VERSION_SA)
	sa_register_update_callback(os, zfs_sa_upgrade);

	return (0);
	}

	int
	zfsvfs_create(const char osname, boolean_t readonly, zfsvfs_t *zfvp)
	{
	objset_t *os;
	zfsvfs_t *zfsvfs;
	int error;
	boolean_t ro = (readonly \|\| (strchr(osname, '@') != NULL));

	zfsvfs = kmem_zalloc(sizeof (zfsvfs_t), KM_SLEEP);

	error = dmu_objset_own(osname, DMU_OST_ZFS, ro, B_TRUE, zfsvfs, &os);
	if (error != 0) {
	kmem_free(zfsvfs, sizeof (zfsvfs_t));
	return (error);
	}

	error = zfsvfs_create_impl(zfvp, zfsvfs, os);
	if (error != 0) {
	dmu_objset_disown(os, B_TRUE, zfsvfs);
	}
	return (error);
	}


	/*
	* Note: zfsvfs is assumed to be malloc'd, and will be freed by this function
	* on a failure. Do not pass in a statically allocated zfsvfs.
	*/
	int
	zfsvfs_create_impl(zfsvfs_t *zfvp, zfsvfs_t zfsvfs, objset_t *os)
	{
	int error;

	zfsvfs->z_vfs = NULL;
	zfsvfs->z_sb = NULL;
	zfsvfs->z_parent = zfsvfs;

	mutex_init(&zfsvfs->z_znodes_lock, NULL, MUTEX_DEFAULT, NULL);
	mutex_init(&zfsvfs->z_lock, NULL, MUTEX_DEFAULT, NULL);
	list_create(&zfsvfs->z_all_znodes, sizeof (znode_t),
	offsetof(znode_t, z_link_node));
	rrm_init(&zfsvfs->z_teardown_lock, B_FALSE);
	rw_init(&zfsvfs->z_teardown_inactive_lock, NULL, RW_DEFAULT, NULL);
	rw_init(&zfsvfs->z_fuid_lock, NULL, RW_DEFAULT, NULL);

	int size = MIN(1 << (highbit64(zfs_object_mutex_size) - 1),
	ZFS_OBJ_MTX_MAX);
	zfsvfs->z_hold_size = size;
	zfsvfs->z_hold_trees = vmem_zalloc(sizeof (avl_tree_t) * size,
	KM_SLEEP);
	zfsvfs->z_hold_locks = vmem_zalloc(sizeof (kmutex_t) * size, KM_SLEEP);
	for (int i = 0; i != size; i++) {
	avl_create(&zfsvfs->z_hold_trees[i], zfs_znode_hold_compare,
	sizeof (znode_hold_t), offsetof(znode_hold_t, zh_node));
	mutex_init(&zfsvfs->z_hold_locks[i], NULL, MUTEX_DEFAULT, NULL);
	}

	error = zfsvfs_init(zfsvfs, os);
	if (error != 0) {
	*zfvp = NULL;
	zfsvfs_free(zfsvfs);
	return (error);
	}

	zfsvfs->z_drain_task = TASKQID_INVALID;
	zfsvfs->z_draining = B_FALSE;
	zfsvfs->z_drain_cancel = B_TRUE;

	*zfvp = zfsvfs;
	return (0);
	}

	static int
	zfsvfs_setup(zfsvfs_t *zfsvfs, boolean_t mounting)
	{
	int error;
	boolean_t readonly = zfs_is_readonly(zfsvfs);

	error = zfs_register_callbacks(zfsvfs->z_vfs);
	if (error)
	return (error);

	zfsvfs->z_log = zil_open(zfsvfs->z_os, zfs_get_data);

	/*
	* If we are not mounting (ie: online recv), then we don't
	* have to worry about replaying the log as we blocked all
	* operations out since we closed the ZIL.
	*/
	if (mounting) {
	ASSERT3P(zfsvfs->z_kstat.dk_kstats, ==, NULL);
	dataset_kstats_create(&zfsvfs->z_kstat, zfsvfs->z_os);

	/*
	* During replay we remove the read only flag to
	* allow replays to succeed.
	*/
	if (readonly != 0) {
	readonly_changed_cb(zfsvfs, B_FALSE);
	} else {
	zap_stats_t zs;
	if (zap_get_stats(zfsvfs->z_os, zfsvfs->z_unlinkedobj,
	&zs) == 0) {
	dataset_kstats_update_nunlinks_kstat(
	&zfsvfs->z_kstat, zs.zs_num_entries);
	dprintf_ds(zfsvfs->z_os->os_dsl_dataset,
	"num_entries in unlinked set: %llu",
	zs.zs_num_entries);
	}
	zfs_unlinked_drain(zfsvfs);
	dsl_dir_t *dd = zfsvfs->z_os->os_dsl_dataset->ds_dir;
	dd->dd_activity_cancelled = B_FALSE;
	}

	/*
	* Parse and replay the intent log.
	*
	* Because of ziltest, this must be done after
	* zfs_unlinked_drain(). (Further note: ziltest
	* doesn't use readonly mounts, where
	* zfs_unlinked_drain() isn't called.) This is because
	* ziltest causes spa_sync() to think it's committed,
	* but actually it is not, so the intent log contains
	* many txg's worth of changes.
	*
	* In particular, if object N is in the unlinked set in
	* the last txg to actually sync, then it could be
	* actually freed in a later txg and then reallocated
	* in a yet later txg. This would write a "create
	* object N" record to the intent log. Normally, this
	* would be fine because the spa_sync() would have
	* written out the fact that object N is free, before
	* we could write the "create object N" intent log
	* record.
	*
	* But when we are in ziltest mode, we advance the "open
	* txg" without actually spa_sync()-ing the changes to
	* disk. So we would see that object N is still
	* allocated and in the unlinked set, and there is an
	* intent log record saying to allocate it.
	*/
	if (spa_writeable(dmu_objset_spa(zfsvfs->z_os))) {
	if (zil_replay_disable) {
	zil_destroy(zfsvfs->z_log, B_FALSE);
	} else {
	zfsvfs->z_replay = B_TRUE;
	zil_replay(zfsvfs->z_os, zfsvfs,
	zfs_replay_vector);
	zfsvfs->z_replay = B_FALSE;
	}
	}

	/* restore readonly bit */
	if (readonly != 0)
	readonly_changed_cb(zfsvfs, B_TRUE);
	}

	/*
	* Set the objset user_ptr to track its zfsvfs.
	*/
	mutex_enter(&zfsvfs->z_os->os_user_ptr_lock);
	dmu_objset_set_user(zfsvfs->z_os, zfsvfs);
	mutex_exit(&zfsvfs->z_os->os_user_ptr_lock);

	return (0);
	}

	void
	zfsvfs_free(zfsvfs_t *zfsvfs)
	{
	int i, size = zfsvfs->z_hold_size;

	zfs_fuid_destroy(zfsvfs);

	mutex_destroy(&zfsvfs->z_znodes_lock);
	mutex_destroy(&zfsvfs->z_lock);
	list_destroy(&zfsvfs->z_all_znodes);
	rrm_destroy(&zfsvfs->z_teardown_lock);
	rw_destroy(&zfsvfs->z_teardown_inactive_lock);
	rw_destroy(&zfsvfs->z_fuid_lock);
	for (i = 0; i != size; i++) {
	avl_destroy(&zfsvfs->z_hold_trees[i]);
	mutex_destroy(&zfsvfs->z_hold_locks[i]);
	}
	vmem_free(zfsvfs->z_hold_trees, sizeof (avl_tree_t) * size);
	vmem_free(zfsvfs->z_hold_locks, sizeof (kmutex_t) * size);
	zfsvfs_vfs_free(zfsvfs->z_vfs);
	dataset_kstats_destroy(&zfsvfs->z_kstat);
	kmem_free(zfsvfs, sizeof (zfsvfs_t));
	}

	static void
	zfs_set_fuid_feature(zfsvfs_t *zfsvfs)
	{
	zfsvfs->z_use_fuids = USE_FUIDS(zfsvfs->z_version, zfsvfs->z_os);
	zfsvfs->z_use_sa = USE_SA(zfsvfs->z_version, zfsvfs->z_os);
	}

	static void
	zfs_unregister_callbacks(zfsvfs_t *zfsvfs)
	{
	objset_t *os = zfsvfs->z_os;

	if (!dmu_objset_is_snapshot(os))
	dsl_prop_unregister_all(dmu_objset_ds(os), zfsvfs);
	}

	#ifdef HAVE_MLSLABEL
	/*
	* Check that the hex label string is appropriate for the dataset being
	* mounted into the global_zone proper.
	*
	* Return an error if the hex label string is not default or
	* admin_low/admin_high. For admin_low labels, the corresponding
	* dataset must be readonly.
	*/
	int
	zfs_check_global_label(const char dsname, const char hexsl)
	{
	if (strcasecmp(hexsl, ZFS_MLSLABEL_DEFAULT) == 0)
	return (0);
	if (strcasecmp(hexsl, ADMIN_HIGH) == 0)
	return (0);
	if (strcasecmp(hexsl, ADMIN_LOW) == 0) {
	/* must be readonly */
	uint64_t rdonly;

	if (dsl_prop_get_integer(dsname,
	zfs_prop_to_name(ZFS_PROP_READONLY), &rdonly, NULL))
	return (SET_ERROR(EACCES));
	return (rdonly ? 0 : SET_ERROR(EACCES));
	}
	return (SET_ERROR(EACCES));
	}
	#endif /* HAVE_MLSLABEL */

	static int
	zfs_statfs_project(zfsvfs_t zfsvfs, znode_t zp, struct kstatfs *statp,
	uint32_t bshift)
	{
	char buf[20 + DMU_OBJACCT_PREFIX_LEN];
	uint64_t offset = DMU_OBJACCT_PREFIX_LEN;
	uint64_t quota;
	uint64_t used;
	int err;

	strlcpy(buf, DMU_OBJACCT_PREFIX, DMU_OBJACCT_PREFIX_LEN + 1);
	err = zfs_id_to_fuidstr(zfsvfs, NULL, zp->z_projid, buf + offset,
	sizeof (buf) - offset, B_FALSE);
	if (err)
	return (err);

	if (zfsvfs->z_projectquota_obj == 0)
	goto objs;

	err = zap_lookup(zfsvfs->z_os, zfsvfs->z_projectquota_obj,
	buf + offset, 8, 1, &quota);
	if (err == ENOENT)
	goto objs;
	else if (err)
	return (err);

	err = zap_lookup(zfsvfs->z_os, DMU_PROJECTUSED_OBJECT,
	buf + offset, 8, 1, &used);
	if (unlikely(err == ENOENT)) {
	uint32_t blksize;
	u_longlong_t nblocks;

	/*
	* Quota accounting is async, so it is possible race case.
	* There is at least one object with the given project ID.
	*/
	sa_object_size(zp->z_sa_hdl, &blksize, &nblocks);
	if (unlikely(zp->z_blksz == 0))
	blksize = zfsvfs->z_max_blksz;

	used = blksize * nblocks;
	} else if (err) {
	return (err);
	}

	statp->f_blocks = quota >> bshift;
	statp->f_bfree = (quota > used) ? ((quota - used) >> bshift) : 0;
	statp->f_bavail = statp->f_bfree;

	objs:
	if (zfsvfs->z_projectobjquota_obj == 0)
	return (0);

	err = zap_lookup(zfsvfs->z_os, zfsvfs->z_projectobjquota_obj,
	buf + offset, 8, 1, &quota);
	if (err == ENOENT)
	return (0);
	else if (err)
	return (err);

	err = zap_lookup(zfsvfs->z_os, DMU_PROJECTUSED_OBJECT,
	buf, 8, 1, &used);
	if (unlikely(err == ENOENT)) {
	/*
	* Quota accounting is async, so it is possible race case.
	* There is at least one object with the given project ID.
	*/
	used = 1;
	} else if (err) {
	return (err);
	}

	statp->f_files = quota;
	statp->f_ffree = (quota > used) ? (quota - used) : 0;

	return (0);
	}

	int
	zfs_statvfs(struct inode ip, struct kstatfs statp)
	{
	zfsvfs_t *zfsvfs = ITOZSB(ip);
	uint64_t refdbytes, availbytes, usedobjs, availobjs;
	int err = 0;

	ZFS_ENTER(zfsvfs);

	dmu_objset_space(zfsvfs->z_os,
	&refdbytes, &availbytes, &usedobjs, &availobjs);

	uint64_t fsid = dmu_objset_fsid_guid(zfsvfs->z_os);
	/*
	* The underlying storage pool actually uses multiple block
	* size. Under Solaris frsize (fragment size) is reported as
	* the smallest block size we support, and bsize (block size)
	* as the filesystem's maximum block size. Unfortunately,
	* under Linux the fragment size and block size are often used
	* interchangeably. Thus we are forced to report both of them
	* as the filesystem's maximum block size.
	*/
	statp->f_frsize = zfsvfs->z_max_blksz;
	statp->f_bsize = zfsvfs->z_max_blksz;
	uint32_t bshift = fls(statp->f_bsize) - 1;

	/*
	* The following report "total" blocks of various kinds in
	* the file system, but reported in terms of f_bsize - the
	* "preferred" size.
	*/

	/* Round up so we never have a filesystem using 0 blocks. */
	refdbytes = P2ROUNDUP(refdbytes, statp->f_bsize);
	statp->f_blocks = (refdbytes + availbytes) >> bshift;
	statp->f_bfree = availbytes >> bshift;
	statp->f_bavail = statp->f_bfree; /* no root reservation */

	/*
	* statvfs() should really be called statufs(), because it assumes
	* static metadata. ZFS doesn't preallocate files, so the best
	* we can do is report the max that could possibly fit in f_files,
	* and that minus the number actually used in f_ffree.
	* For f_ffree, report the smaller of the number of objects available
	* and the number of blocks (each object will take at least a block).
	*/
	statp->f_ffree = MIN(availobjs, availbytes >> DNODE_SHIFT);
	statp->f_files = statp->f_ffree + usedobjs;
	statp->f_fsid.val[0] = (uint32_t)fsid;
	statp->f_fsid.val[1] = (uint32_t)(fsid >> 32);
	statp->f_type = ZFS_SUPER_MAGIC;
	statp->f_namelen = MAXNAMELEN - 1;

	/*
	* We have all of 40 characters to stuff a string here.
	* Is there anything useful we could/should provide?
	*/
	bzero(statp->f_spare, sizeof (statp->f_spare));

	if (dmu_objset_projectquota_enabled(zfsvfs->z_os) &&
	dmu_objset_projectquota_present(zfsvfs->z_os)) {
	znode_t *zp = ITOZ(ip);

	if (zp->z_pflags & ZFS_PROJINHERIT && zp->z_projid &&
	zpl_is_valid_projid(zp->z_projid))
	err = zfs_statfs_project(zfsvfs, zp, statp, bshift);
	}

	ZFS_EXIT(zfsvfs);
	return (err);
	}

	static int
	zfs_root(zfsvfs_t zfsvfs, struct inode *ipp)
	{
	znode_t *rootzp;
	int error;

	ZFS_ENTER(zfsvfs);

	error = zfs_zget(zfsvfs, zfsvfs->z_root, &rootzp);
	if (error == 0)
	*ipp = ZTOI(rootzp);

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Linux kernels older than 3.1 do not support a per-filesystem shrinker.
	* To accommodate this we must improvise and manually walk the list of znodes
	* attempting to prune dentries in order to be able to drop the inodes.
	*
	* To avoid scanning the same znodes multiple times they are always rotated
	* to the end of the z_all_znodes list. New znodes are inserted at the
	* end of the list so we're always scanning the oldest znodes first.
	*/
	static int
	zfs_prune_aliases(zfsvfs_t *zfsvfs, unsigned long nr_to_scan)
	{
	znode_t *zp_array, zp;
	int max_array = MIN(nr_to_scan, PAGE_SIZE * 8 / sizeof (znode_t *));
	int objects = 0;
	int i = 0, j = 0;

	zp_array = kmem_zalloc(max_array * sizeof (znode_t *), KM_SLEEP);

	mutex_enter(&zfsvfs->z_znodes_lock);
	while ((zp = list_head(&zfsvfs->z_all_znodes)) != NULL) {

	if ((i++ > nr_to_scan) \|\| (j >= max_array))
	break;

	ASSERT(list_link_active(&zp->z_link_node));
	list_remove(&zfsvfs->z_all_znodes, zp);
	list_insert_tail(&zfsvfs->z_all_znodes, zp);

	/* Skip active znodes and .zfs entries */
	if (MUTEX_HELD(&zp->z_lock) \|\| zp->z_is_ctldir)
	continue;

	if (igrab(ZTOI(zp)) == NULL)
	continue;

	zp_array[j] = zp;
	j++;
	}
	mutex_exit(&zfsvfs->z_znodes_lock);

	for (i = 0; i < j; i++) {
	zp = zp_array[i];

	ASSERT3P(zp, !=, NULL);
	d_prune_aliases(ZTOI(zp));

	if (atomic_read(&ZTOI(zp)->i_count) == 1)
	objects++;

	zrele(zp);
	}

	kmem_free(zp_array, max_array * sizeof (znode_t *));

	return (objects);
	}

	/*
	* The ARC has requested that the filesystem drop entries from the dentry
	* and inode caches. This can occur when the ARC needs to free meta data
	* blocks but can't because they are all pinned by entries in these caches.
	*/
	int
	zfs_prune(struct super_block sb, unsigned long nr_to_scan, int objects)
	{
	zfsvfs_t *zfsvfs = sb->s_fs_info;
	int error = 0;
	struct shrinker *shrinker = &sb->s_shrink;
	struct shrink_control sc = {
	.nr_to_scan = nr_to_scan,
	.gfp_mask = GFP_KERNEL,
	};

	ZFS_ENTER(zfsvfs);

	#if defined(HAVE_SPLIT_SHRINKER_CALLBACK) && \
	defined(SHRINK_CONTROL_HAS_NID) && \
	defined(SHRINKER_NUMA_AWARE)
	if (sb->s_shrink.flags & SHRINKER_NUMA_AWARE) {
	*objects = 0;
	for_each_online_node(sc.nid) {
	objects += (shrinker->scan_objects)(shrinker, &sc);
	}
	} else {
	objects = (shrinker->scan_objects)(shrinker, &sc);
	}

	#elif defined(HAVE_SPLIT_SHRINKER_CALLBACK)
	objects = (shrinker->scan_objects)(shrinker, &sc);
	#elif defined(HAVE_SINGLE_SHRINKER_CALLBACK)
	objects = (shrinker->shrink)(shrinker, &sc);
	#elif defined(HAVE_D_PRUNE_ALIASES)
	#define D_PRUNE_ALIASES_IS_DEFAULT
	*objects = zfs_prune_aliases(zfsvfs, nr_to_scan);
	#else
	#error "No available dentry and inode cache pruning mechanism."
	#endif

	#if defined(HAVE_D_PRUNE_ALIASES) && !defined(D_PRUNE_ALIASES_IS_DEFAULT)
	#undef D_PRUNE_ALIASES_IS_DEFAULT
	/*
	* Fall back to zfs_prune_aliases if the kernel's per-superblock
	* shrinker couldn't free anything, possibly due to the inodes being
	* allocated in a different memcg.
	*/
	if (*objects == 0)
	*objects = zfs_prune_aliases(zfsvfs, nr_to_scan);
	#endif

	ZFS_EXIT(zfsvfs);

	dprintf_ds(zfsvfs->z_os->os_dsl_dataset,
	"pruning, nr_to_scan=%lu objects=%d error=%d\n",
	nr_to_scan, *objects, error);

	return (error);
	}

	/*
	* Teardown the zfsvfs_t.
	*
	* Note, if 'unmounting' is FALSE, we return with the 'z_teardown_lock'
	* and 'z_teardown_inactive_lock' held.
	*/
	static int
	zfsvfs_teardown(zfsvfs_t *zfsvfs, boolean_t unmounting)
	{
	znode_t *zp;

	zfs_unlinked_drain_stop_wait(zfsvfs);

	/*
	* If someone has not already unmounted this file system,
	* drain the zrele_taskq to ensure all active references to the
	* zfsvfs_t have been handled only then can it be safely destroyed.
	*/
	if (zfsvfs->z_os) {
	/*
	* If we're unmounting we have to wait for the list to
	* drain completely.
	*
	* If we're not unmounting there's no guarantee the list
	* will drain completely, but iputs run from the taskq
	* may add the parents of dir-based xattrs to the taskq
	* so we want to wait for these.
	*
	* We can safely read z_nr_znodes without locking because the
	* VFS has already blocked operations which add to the
	* z_all_znodes list and thus increment z_nr_znodes.
	*/
	int round = 0;
	while (zfsvfs->z_nr_znodes > 0) {
	taskq_wait_outstanding(dsl_pool_zrele_taskq(
	dmu_objset_pool(zfsvfs->z_os)), 0);
	if (++round > 1 && !unmounting)
	break;
	}
	}

	rrm_enter(&zfsvfs->z_teardown_lock, RW_WRITER, FTAG);

	if (!unmounting) {
	/*
	* We purge the parent filesystem's super block as the
	* parent filesystem and all of its snapshots have their
	* inode's super block set to the parent's filesystem's
	* super block. Note, 'z_parent' is self referential
	* for non-snapshots.
	*/
	shrink_dcache_sb(zfsvfs->z_parent->z_sb);
	}

	/*
	* Close the zil. NB: Can't close the zil while zfs_inactive
	* threads are blocked as zil_close can call zfs_inactive.
	*/
	if (zfsvfs->z_log) {
	zil_close(zfsvfs->z_log);
	zfsvfs->z_log = NULL;
	}

	rw_enter(&zfsvfs->z_teardown_inactive_lock, RW_WRITER);

	/*
	* If we are not unmounting (ie: online recv) and someone already
	* unmounted this file system while we were doing the switcheroo,
	* or a reopen of z_os failed then just bail out now.
	*/
	if (!unmounting && (zfsvfs->z_unmounted \|\| zfsvfs->z_os == NULL)) {
	rw_exit(&zfsvfs->z_teardown_inactive_lock);
	rrm_exit(&zfsvfs->z_teardown_lock, FTAG);
	return (SET_ERROR(EIO));
	}

	/*
	* At this point there are no VFS ops active, and any new VFS ops
	* will fail with EIO since we have z_teardown_lock for writer (only
	* relevant for forced unmount).
	*
	* Release all holds on dbufs. We also grab an extra reference to all
	* the remaining inodes so that the kernel does not attempt to free
	* any inodes of a suspended fs. This can cause deadlocks since the
	* zfs_resume_fs() process may involve starting threads, which might
	* attempt to free unreferenced inodes to free up memory for the new
	* thread.
	*/
	if (!unmounting) {
	mutex_enter(&zfsvfs->z_znodes_lock);
	for (zp = list_head(&zfsvfs->z_all_znodes); zp != NULL;
	zp = list_next(&zfsvfs->z_all_znodes, zp)) {
	if (zp->z_sa_hdl)
	zfs_znode_dmu_fini(zp);
	if (igrab(ZTOI(zp)) != NULL)
	zp->z_suspended = B_TRUE;

	}
	mutex_exit(&zfsvfs->z_znodes_lock);
	}

	/*
	* If we are unmounting, set the unmounted flag and let new VFS ops
	* unblock. zfs_inactive will have the unmounted behavior, and all
	* other VFS ops will fail with EIO.
	*/
	if (unmounting) {
	zfsvfs->z_unmounted = B_TRUE;
	rw_exit(&zfsvfs->z_teardown_inactive_lock);
	rrm_exit(&zfsvfs->z_teardown_lock, FTAG);
	}

	/*
	* z_os will be NULL if there was an error in attempting to reopen
	* zfsvfs, so just return as the properties had already been
	*
	* unregistered and cached data had been evicted before.
	*/
	if (zfsvfs->z_os == NULL)
	return (0);

	/*
	* Unregister properties.
	*/
	zfs_unregister_callbacks(zfsvfs);

	/*
	* Evict cached data. We must write out any dirty data before
	* disowning the dataset.
	*/
	objset_t *os = zfsvfs->z_os;
	boolean_t os_dirty = B_FALSE;
	for (int t = 0; t < TXG_SIZE; t++) {
	if (dmu_objset_is_dirty(os, t)) {
	os_dirty = B_TRUE;
	break;
	}
	}
	if (!zfs_is_readonly(zfsvfs) && os_dirty) {
	txg_wait_synced(dmu_objset_pool(zfsvfs->z_os), 0);
	}
	dmu_objset_evict_dbufs(zfsvfs->z_os);
	dsl_dir_t *dd = os->os_dsl_dataset->ds_dir;
	dsl_dir_cancel_waiters(dd);

	return (0);
	}

	#if defined(HAVE_SUPER_SETUP_BDI_NAME)
	atomic_long_t zfs_bdi_seq = ATOMIC_LONG_INIT(0);
	#endif

	int
	zfs_domount(struct super_block sb, zfs_mnt_t zm, int silent)
	{
	const char *osname = zm->mnt_osname;
	struct inode *root_inode;
	uint64_t recordsize;
	int error = 0;
	zfsvfs_t *zfsvfs = NULL;
	vfs_t *vfs = NULL;

	ASSERT(zm);
	ASSERT(osname);

	error = zfsvfs_parse_options(zm->mnt_data, &vfs);
	if (error)
	return (error);

	error = zfsvfs_create(osname, vfs->vfs_readonly, &zfsvfs);
	if (error) {
	zfsvfs_vfs_free(vfs);
	goto out;
	}

	if ((error = dsl_prop_get_integer(osname, "recordsize",
	&recordsize, NULL))) {
	zfsvfs_vfs_free(vfs);
	goto out;
	}

	vfs->vfs_data = zfsvfs;
	zfsvfs->z_vfs = vfs;
	zfsvfs->z_sb = sb;
	sb->s_fs_info = zfsvfs;
	sb->s_magic = ZFS_SUPER_MAGIC;
	sb->s_maxbytes = MAX_LFS_FILESIZE;
	sb->s_time_gran = 1;
	sb->s_blocksize = recordsize;
	sb->s_blocksize_bits = ilog2(recordsize);

	error = -zpl_bdi_setup(sb, "zfs");
	if (error)
	goto out;

	sb->s_bdi->ra_pages = 0;

	/* Set callback operations for the file system. */
	sb->s_op = &zpl_super_operations;
	sb->s_xattr = zpl_xattr_handlers;
	sb->s_export_op = &zpl_export_operations;
	sb->s_d_op = &zpl_dentry_operations;

	/* Set features for file system. */
	zfs_set_fuid_feature(zfsvfs);

	if (dmu_objset_is_snapshot(zfsvfs->z_os)) {
	uint64_t pval;

	atime_changed_cb(zfsvfs, B_FALSE);
	readonly_changed_cb(zfsvfs, B_TRUE);
	if ((error = dsl_prop_get_integer(osname,
	"xattr", &pval, NULL)))
	goto out;
	xattr_changed_cb(zfsvfs, pval);
	if ((error = dsl_prop_get_integer(osname,
	"acltype", &pval, NULL)))
	goto out;
	acltype_changed_cb(zfsvfs, pval);
	zfsvfs->z_issnap = B_TRUE;
	zfsvfs->z_os->os_sync = ZFS_SYNC_DISABLED;
	zfsvfs->z_snap_defer_time = jiffies;

	mutex_enter(&zfsvfs->z_os->os_user_ptr_lock);
	dmu_objset_set_user(zfsvfs->z_os, zfsvfs);
	mutex_exit(&zfsvfs->z_os->os_user_ptr_lock);
	} else {
	if ((error = zfsvfs_setup(zfsvfs, B_TRUE)))
	goto out;
	}

	/* Allocate a root inode for the filesystem. */
	error = zfs_root(zfsvfs, &root_inode);
	if (error) {
	(void) zfs_umount(sb);
	goto out;
	}

	/* Allocate a root dentry for the filesystem */
	sb->s_root = d_make_root(root_inode);
	if (sb->s_root == NULL) {
	(void) zfs_umount(sb);
	error = SET_ERROR(ENOMEM);
	goto out;
	}

	if (!zfsvfs->z_issnap)
	zfsctl_create(zfsvfs);

	zfsvfs->z_arc_prune = arc_add_prune_callback(zpl_prune_sb, sb);
	out:
	if (error) {
	if (zfsvfs != NULL) {
	dmu_objset_disown(zfsvfs->z_os, B_TRUE, zfsvfs);
	zfsvfs_free(zfsvfs);
	}
	/*
	* make sure we don't have dangling sb->s_fs_info which
	* zfs_preumount will use.
	*/
	sb->s_fs_info = NULL;
	}

	return (error);
	}

	/*
	* Called when an unmount is requested and certain sanity checks have
	* already passed. At this point no dentries or inodes have been reclaimed
	* from their respective caches. We drop the extra reference on the .zfs
	* control directory to allow everything to be reclaimed. All snapshots
	* must already have been unmounted to reach this point.
	*/
	void
	zfs_preumount(struct super_block *sb)
	{
	zfsvfs_t *zfsvfs = sb->s_fs_info;

	/* zfsvfs is NULL when zfs_domount fails during mount */
	if (zfsvfs) {
	zfs_unlinked_drain_stop_wait(zfsvfs);
	zfsctl_destroy(sb->s_fs_info);
	/*
	* Wait for zrele_async before entering evict_inodes in
	* generic_shutdown_super. The reason we must finish before
	* evict_inodes is when lazytime is on, or when zfs_purgedir
	* calls zfs_zget, zrele would bump i_count from 0 to 1. This
	* would race with the i_count check in evict_inodes. This means
	* it could destroy the inode while we are still using it.
	*
	* We wait for two passes. xattr directories in the first pass
	* may add xattr entries in zfs_purgedir, so in the second pass
	* we wait for them. We don't use taskq_wait here because it is
	* a pool wide taskq. Other mounted filesystems can constantly
	* do zrele_async and there's no guarantee when taskq will be
	* empty.
	*/
	taskq_wait_outstanding(dsl_pool_zrele_taskq(
	dmu_objset_pool(zfsvfs->z_os)), 0);
	taskq_wait_outstanding(dsl_pool_zrele_taskq(
	dmu_objset_pool(zfsvfs->z_os)), 0);
	}
	}

	/*
	* Called once all other unmount released tear down has occurred.
	* It is our responsibility to release any remaining infrastructure.
	*/
	/ARGSUSED/
	int
	zfs_umount(struct super_block *sb)
	{
	zfsvfs_t *zfsvfs = sb->s_fs_info;
	objset_t *os;

	if (zfsvfs->z_arc_prune != NULL)
	arc_remove_prune_callback(zfsvfs->z_arc_prune);
	VERIFY(zfsvfs_teardown(zfsvfs, B_TRUE) == 0);
	os = zfsvfs->z_os;
	zpl_bdi_destroy(sb);

	/*
	* z_os will be NULL if there was an error in
	* attempting to reopen zfsvfs.
	*/
	if (os != NULL) {
	/*
	* Unset the objset user_ptr.
	*/
	mutex_enter(&os->os_user_ptr_lock);
	dmu_objset_set_user(os, NULL);
	mutex_exit(&os->os_user_ptr_lock);

	/*
	* Finally release the objset
	*/
	dmu_objset_disown(os, B_TRUE, zfsvfs);
	}

	zfsvfs_free(zfsvfs);
	return (0);
	}

	int
	zfs_remount(struct super_block sb, int flags, zfs_mnt_t *zm)
	{
	zfsvfs_t *zfsvfs = sb->s_fs_info;
	vfs_t *vfsp;
	boolean_t issnap = dmu_objset_is_snapshot(zfsvfs->z_os);
	int error;

	if ((issnap \|\| !spa_writeable(dmu_objset_spa(zfsvfs->z_os))) &&
	!(*flags & SB_RDONLY)) {
	*flags \|= SB_RDONLY;
	return (EROFS);
	}

	error = zfsvfs_parse_options(zm->mnt_data, &vfsp);
	if (error)
	return (error);

	if (!zfs_is_readonly(zfsvfs) && (*flags & SB_RDONLY))
	txg_wait_synced(dmu_objset_pool(zfsvfs->z_os), 0);

	zfs_unregister_callbacks(zfsvfs);
	zfsvfs_vfs_free(zfsvfs->z_vfs);

	vfsp->vfs_data = zfsvfs;
	zfsvfs->z_vfs = vfsp;
	if (!issnap)
	(void) zfs_register_callbacks(vfsp);

	return (error);
	}

	int
	zfs_vget(struct super_block sb, struct inode ipp, fid_t fidp)
	{
	zfsvfs_t *zfsvfs = sb->s_fs_info;
	znode_t *zp;
	uint64_t object = 0;
	uint64_t fid_gen = 0;
	uint64_t gen_mask;
	uint64_t zp_gen;
	int i, err;

	*ipp = NULL;

	if (fidp->fid_len == SHORT_FID_LEN \|\| fidp->fid_len == LONG_FID_LEN) {
	zfid_short_t zfid = (zfid_short_t )fidp;

	for (i = 0; i < sizeof (zfid->zf_object); i++)
	object \|= ((uint64_t)zfid->zf_object[i]) << (8 * i);

	for (i = 0; i < sizeof (zfid->zf_gen); i++)
	fid_gen \|= ((uint64_t)zfid->zf_gen[i]) << (8 * i);
	} else {
	return (SET_ERROR(EINVAL));
	}

	/* LONG_FID_LEN means snapdirs */
	if (fidp->fid_len == LONG_FID_LEN) {
	zfid_long_t zlfid = (zfid_long_t )fidp;
	uint64_t objsetid = 0;
	uint64_t setgen = 0;

	for (i = 0; i < sizeof (zlfid->zf_setid); i++)
	objsetid \|= ((uint64_t)zlfid->zf_setid[i]) << (8 * i);

	for (i = 0; i < sizeof (zlfid->zf_setgen); i++)
	setgen \|= ((uint64_t)zlfid->zf_setgen[i]) << (8 * i);

	if (objsetid != ZFSCTL_INO_SNAPDIRS - object) {
	dprintf("snapdir fid: objsetid (%llu) != "
	"ZFSCTL_INO_SNAPDIRS (%llu) - object (%llu)\n",
	objsetid, ZFSCTL_INO_SNAPDIRS, object);

	return (SET_ERROR(EINVAL));
	}

	if (fid_gen > 1 \|\| setgen != 0) {
	dprintf("snapdir fid: fid_gen (%llu) and setgen "
	"(%llu)\n", fid_gen, setgen);
	return (SET_ERROR(EINVAL));
	}

	return (zfsctl_snapdir_vget(sb, objsetid, fid_gen, ipp));
	}

	ZFS_ENTER(zfsvfs);
	/* A zero fid_gen means we are in the .zfs control directories */
	if (fid_gen == 0 &&
	(object == ZFSCTL_INO_ROOT \|\| object == ZFSCTL_INO_SNAPDIR)) {
	*ipp = zfsvfs->z_ctldir;
	ASSERT(*ipp != NULL);
	if (object == ZFSCTL_INO_SNAPDIR) {
	VERIFY(zfsctl_root_lookup(*ipp, "snapshot", ipp,
	0, kcred, NULL, NULL) == 0);
	} else {
	igrab(*ipp);
	}
	ZFS_EXIT(zfsvfs);
	return (0);
	}

	gen_mask = -1ULL >> (64 - 8 * i);

	dprintf("getting %llu [%llu mask %llx]\n", object, fid_gen, gen_mask);
	if ((err = zfs_zget(zfsvfs, object, &zp))) {
	ZFS_EXIT(zfsvfs);
	return (err);
	}

	/* Don't export xattr stuff */
	if (zp->z_pflags & ZFS_XATTR) {
	zrele(zp);
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(ENOENT));
	}

	(void) sa_lookup(zp->z_sa_hdl, SA_ZPL_GEN(zfsvfs), &zp_gen,
	sizeof (uint64_t));
	zp_gen = zp_gen & gen_mask;
	if (zp_gen == 0)
	zp_gen = 1;
	if ((fid_gen == 0) && (zfsvfs->z_root == object))
	fid_gen = zp_gen;
	if (zp->z_unlinked \|\| zp_gen != fid_gen) {
	dprintf("znode gen (%llu) != fid gen (%llu)\n", zp_gen,
	fid_gen);
	zrele(zp);
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(ENOENT));
	}

	*ipp = ZTOI(zp);
	if (*ipp)
	zfs_inode_update(ITOZ(*ipp));

	ZFS_EXIT(zfsvfs);
	return (0);
	}

	/*
	* Block out VFS ops and close zfsvfs_t
	*
	* Note, if successful, then we return with the 'z_teardown_lock' and
	* 'z_teardown_inactive_lock' write held. We leave ownership of the underlying
	* dataset and objset intact so that they can be atomically handed off during
	* a subsequent rollback or recv operation and the resume thereafter.
	*/
	int
	zfs_suspend_fs(zfsvfs_t *zfsvfs)
	{
	int error;

	if ((error = zfsvfs_teardown(zfsvfs, B_FALSE)) != 0)
	return (error);

	return (0);
	}

	/*
	* Rebuild SA and release VOPs. Note that ownership of the underlying dataset
	* is an invariant across any of the operations that can be performed while the
	* filesystem was suspended. Whether it succeeded or failed, the preconditions
	* are the same: the relevant objset and associated dataset are owned by
	* zfsvfs, held, and long held on entry.
	*/
	int
	zfs_resume_fs(zfsvfs_t zfsvfs, dsl_dataset_t ds)
	{
	int err, err2;
	znode_t *zp;

	ASSERT(RRM_WRITE_HELD(&zfsvfs->z_teardown_lock));
	ASSERT(RW_WRITE_HELD(&zfsvfs->z_teardown_inactive_lock));

	/*
	* We already own this, so just update the objset_t, as the one we
	* had before may have been evicted.
	*/
	objset_t *os;
	VERIFY3P(ds->ds_owner, ==, zfsvfs);
	VERIFY(dsl_dataset_long_held(ds));
	dsl_pool_t *dp = spa_get_dsl(dsl_dataset_get_spa(ds));
	dsl_pool_config_enter(dp, FTAG);
	VERIFY0(dmu_objset_from_ds(ds, &os));
	dsl_pool_config_exit(dp, FTAG);

	err = zfsvfs_init(zfsvfs, os);
	if (err != 0)
	goto bail;

	ds->ds_dir->dd_activity_cancelled = B_FALSE;
	VERIFY(zfsvfs_setup(zfsvfs, B_FALSE) == 0);

	zfs_set_fuid_feature(zfsvfs);
	zfsvfs->z_rollback_time = jiffies;

	/*
	* Attempt to re-establish all the active inodes with their
	* dbufs. If a zfs_rezget() fails, then we unhash the inode
	* and mark it stale. This prevents a collision if a new
	* inode/object is created which must use the same inode
	* number. The stale inode will be be released when the
	* VFS prunes the dentry holding the remaining references
	* on the stale inode.
	*/
	mutex_enter(&zfsvfs->z_znodes_lock);
	for (zp = list_head(&zfsvfs->z_all_znodes); zp;
	zp = list_next(&zfsvfs->z_all_znodes, zp)) {
	err2 = zfs_rezget(zp);
	if (err2) {
	remove_inode_hash(ZTOI(zp));
	zp->z_is_stale = B_TRUE;
	}

	/* see comment in zfs_suspend_fs() */
	if (zp->z_suspended) {
	zfs_zrele_async(zp);
	zp->z_suspended = B_FALSE;
	}
	}
	mutex_exit(&zfsvfs->z_znodes_lock);

	if (!zfs_is_readonly(zfsvfs) && !zfsvfs->z_unmounted) {
	/*
	* zfs_suspend_fs() could have interrupted freeing
	* of dnodes. We need to restart this freeing so
	* that we don't "leak" the space.
	*/
	zfs_unlinked_drain(zfsvfs);
	}

	/*
	* Most of the time zfs_suspend_fs is used for changing the contents
	* of the underlying dataset. ZFS rollback and receive operations
	* might create files for which negative dentries are present in
	* the cache. Since walking the dcache would require a lot of GPL-only
	* code duplication, it's much easier on these rather rare occasions
	* just to flush the whole dcache for the given dataset/filesystem.
	*/
	shrink_dcache_sb(zfsvfs->z_sb);

	bail:
	if (err != 0)
	zfsvfs->z_unmounted = B_TRUE;

	/* release the VFS ops */
	rw_exit(&zfsvfs->z_teardown_inactive_lock);
	rrm_exit(&zfsvfs->z_teardown_lock, FTAG);

	if (err != 0) {
	/*
	* Since we couldn't setup the sa framework, try to force
	* unmount this file system.
	*/
	if (zfsvfs->z_os)
	(void) zfs_umount(zfsvfs->z_sb);
	}
	return (err);
	}

	/*
	* Release VOPs and unmount a suspended filesystem.
	*/
	int
	zfs_end_fs(zfsvfs_t zfsvfs, dsl_dataset_t ds)
	{
	ASSERT(RRM_WRITE_HELD(&zfsvfs->z_teardown_lock));
	ASSERT(RW_WRITE_HELD(&zfsvfs->z_teardown_inactive_lock));

	/*
	* We already own this, so just hold and rele it to update the
	* objset_t, as the one we had before may have been evicted.
	*/
	objset_t *os;
	VERIFY3P(ds->ds_owner, ==, zfsvfs);
	VERIFY(dsl_dataset_long_held(ds));
	dsl_pool_t *dp = spa_get_dsl(dsl_dataset_get_spa(ds));
	dsl_pool_config_enter(dp, FTAG);
	VERIFY0(dmu_objset_from_ds(ds, &os));
	dsl_pool_config_exit(dp, FTAG);
	zfsvfs->z_os = os;

	/* release the VOPs */
	rw_exit(&zfsvfs->z_teardown_inactive_lock);
	rrm_exit(&zfsvfs->z_teardown_lock, FTAG);

	/*
	* Try to force unmount this file system.
	*/
	(void) zfs_umount(zfsvfs->z_sb);
	zfsvfs->z_unmounted = B_TRUE;
	return (0);
	}

	/*
	* Automounted snapshots rely on periodic revalidation
	* to defer snapshots from being automatically unmounted.
	*/

	inline void
	zfs_exit_fs(zfsvfs_t *zfsvfs)
	{
	if (!zfsvfs->z_issnap)
	return;

	if (time_after(jiffies, zfsvfs->z_snap_defer_time +
	MAX(zfs_expire_snapshot * HZ / 2, HZ))) {
	zfsvfs->z_snap_defer_time = jiffies;
	zfsctl_snapshot_unmount_delay(zfsvfs->z_os->os_spa,
	dmu_objset_id(zfsvfs->z_os),
	zfs_expire_snapshot);
	}
	}

	int
	zfs_set_version(zfsvfs_t *zfsvfs, uint64_t newvers)
	{
	int error;
	objset_t *os = zfsvfs->z_os;
	dmu_tx_t *tx;

	if (newvers < ZPL_VERSION_INITIAL \|\| newvers > ZPL_VERSION)
	return (SET_ERROR(EINVAL));

	if (newvers < zfsvfs->z_version)
	return (SET_ERROR(EINVAL));

	if (zfs_spa_version_map(newvers) >
	spa_version(dmu_objset_spa(zfsvfs->z_os)))
	return (SET_ERROR(ENOTSUP));

	tx = dmu_tx_create(os);
	dmu_tx_hold_zap(tx, MASTER_NODE_OBJ, B_FALSE, ZPL_VERSION_STR);
	if (newvers >= ZPL_VERSION_SA && !zfsvfs->z_use_sa) {
	dmu_tx_hold_zap(tx, MASTER_NODE_OBJ, B_TRUE,
	ZFS_SA_ATTRS);
	dmu_tx_hold_zap(tx, DMU_NEW_OBJECT, FALSE, NULL);
	}
	error = dmu_tx_assign(tx, TXG_WAIT);
	if (error) {
	dmu_tx_abort(tx);
	return (error);
	}

	error = zap_update(os, MASTER_NODE_OBJ, ZPL_VERSION_STR,
	8, 1, &newvers, tx);

	if (error) {
	dmu_tx_commit(tx);
	return (error);
	}

	if (newvers >= ZPL_VERSION_SA && !zfsvfs->z_use_sa) {
	uint64_t sa_obj;

	ASSERT3U(spa_version(dmu_objset_spa(zfsvfs->z_os)), >=,
	SPA_VERSION_SA);
	sa_obj = zap_create(os, DMU_OT_SA_MASTER_NODE,
	DMU_OT_NONE, 0, tx);

	error = zap_add(os, MASTER_NODE_OBJ,
	ZFS_SA_ATTRS, 8, 1, &sa_obj, tx);
	ASSERT0(error);

	VERIFY(0 == sa_set_sa_object(os, sa_obj));
	sa_register_update_callback(os, zfs_sa_upgrade);
	}

	spa_history_log_internal_ds(dmu_objset_ds(os), "upgrade", tx,
	"from %llu to %llu", zfsvfs->z_version, newvers);

	dmu_tx_commit(tx);

	zfsvfs->z_version = newvers;
	os->os_version = newvers;

	zfs_set_fuid_feature(zfsvfs);

	return (0);
	}

	/*
	* Read a property stored within the master node.
	*/
	int
	zfs_get_zplprop(objset_t os, zfs_prop_t prop, uint64_t value)
	{
	uint64_t *cached_copy = NULL;

	/*
	* Figure out where in the objset_t the cached copy would live, if it
	* is available for the requested property.
	*/
	if (os != NULL) {
	switch (prop) {
	case ZFS_PROP_VERSION:
	cached_copy = &os->os_version;
	break;
	case ZFS_PROP_NORMALIZE:
	cached_copy = &os->os_normalization;
	break;
	case ZFS_PROP_UTF8ONLY:
	cached_copy = &os->os_utf8only;
	break;
	case ZFS_PROP_CASE:
	cached_copy = &os->os_casesensitivity;
	break;
	default:
	break;
	}
	}
	if (cached_copy != NULL && *cached_copy != OBJSET_PROP_UNINITIALIZED) {
	value = cached_copy;
	return (0);
	}

	/*
	* If the property wasn't cached, look up the file system's value for
	* the property. For the version property, we look up a slightly
	* different string.
	*/
	const char *pname;
	int error = ENOENT;
	if (prop == ZFS_PROP_VERSION)
	pname = ZPL_VERSION_STR;
	else
	pname = zfs_prop_to_name(prop);

	if (os != NULL) {
	ASSERT3U(os->os_phys->os_type, ==, DMU_OST_ZFS);
	error = zap_lookup(os, MASTER_NODE_OBJ, pname, 8, 1, value);
	}

	if (error == ENOENT) {
	/* No value set, use the default value */
	switch (prop) {
	case ZFS_PROP_VERSION:
	*value = ZPL_VERSION;
	break;
	case ZFS_PROP_NORMALIZE:
	case ZFS_PROP_UTF8ONLY:
	*value = 0;
	break;
	case ZFS_PROP_CASE:
	*value = ZFS_CASE_SENSITIVE;
	break;
	case ZFS_PROP_ACLTYPE:
	*value = ZFS_ACLTYPE_OFF;
	break;
	default:
	return (error);
	}
	error = 0;
	}

	/*
	* If one of the methods for getting the property value above worked,
	* copy it into the objset_t's cache.
	*/
	if (error == 0 && cached_copy != NULL) {
	cached_copy = value;
	}

	return (error);
	}

	/*
	* Return true if the corresponding vfs's unmounted flag is set.
	* Otherwise return false.
	* If this function returns true we know VFS unmount has been initiated.
	*/
	boolean_t
	zfs_get_vfs_flag_unmounted(objset_t *os)
	{
	zfsvfs_t *zfvp;
	boolean_t unmounted = B_FALSE;

	ASSERT(dmu_objset_type(os) == DMU_OST_ZFS);

	mutex_enter(&os->os_user_ptr_lock);
	zfvp = dmu_objset_get_user(os);
	if (zfvp != NULL && zfvp->z_unmounted)
	unmounted = B_TRUE;
	mutex_exit(&os->os_user_ptr_lock);

	return (unmounted);
	}

	/ARGSUSED/
	void
	zfsvfs_update_fromname(const char oldname, const char newname)
	{
	/*
	* We don't need to do anything here, the devname is always current by
	* virtue of zfsvfs->z_sb->s_op->show_devname.
	*/
	}

	void
	zfs_init(void)
	{
	zfsctl_init();
	zfs_znode_init();
	dmu_objset_register_type(DMU_OST_ZFS, zpl_get_file_info);
	register_filesystem(&zpl_fs_type);
	}

	void
	zfs_fini(void)
	{
	/*
	* we don't use outstanding because zpl_posix_acl_free might add more.
	*/
	taskq_wait(system_delay_taskq);
	taskq_wait(system_taskq);
	unregister_filesystem(&zpl_fs_type);
	zfs_znode_fini();
	zfsctl_fini();
	}

	#if defined(_KERNEL)
	EXPORT_SYMBOL(zfs_suspend_fs);
	EXPORT_SYMBOL(zfs_resume_fs);
	EXPORT_SYMBOL(zfs_set_version);
	EXPORT_SYMBOL(zfsvfs_create);
	EXPORT_SYMBOL(zfsvfs_free);
	EXPORT_SYMBOL(zfs_is_readonly);
	EXPORT_SYMBOL(zfs_domount);
	EXPORT_SYMBOL(zfs_preumount);
	EXPORT_SYMBOL(zfs_umount);
	EXPORT_SYMBOL(zfs_remount);
	EXPORT_SYMBOL(zfs_statvfs);
	EXPORT_SYMBOL(zfs_vget);
	EXPORT_SYMBOL(zfs_prune);
	#endif
	Index: vendor-sys/openzfs/dist/module/os/linux/zfs/zfs_vnops.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/os/linux/zfs/zfs_vnops.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/os/linux/zfs/zfs_vnops.c (revision 366774)
	@@ -1,5031 +1,5031 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/

	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2012, 2018 by Delphix. All rights reserved.
	* Copyright (c) 2015 by Chunwei Chen. All rights reserved.
	* Copyright 2017 Nexenta Systems, Inc.
	*/

	/* Portions Copyright 2007 Jeremy Teo */
	/* Portions Copyright 2010 Robert Milkowski */


	#include <sys/types.h>
	#include <sys/param.h>
	#include <sys/time.h>
	#include <sys/sysmacros.h>
	#include <sys/vfs.h>
	#include <sys/file.h>
	#include <sys/stat.h>
	#include <sys/kmem.h>
	#include <sys/taskq.h>
	#include <sys/uio.h>
	#include <sys/vmsystm.h>
	#include <sys/atomic.h>
	#include <sys/pathname.h>
	#include <sys/cmn_err.h>
	#include <sys/errno.h>
	#include <sys/zfs_dir.h>
	#include <sys/zfs_acl.h>
	#include <sys/zfs_ioctl.h>
	#include <sys/fs/zfs.h>
	#include <sys/dmu.h>
	#include <sys/dmu_objset.h>
	#include <sys/spa.h>
	#include <sys/txg.h>
	#include <sys/dbuf.h>
	#include <sys/zap.h>
	#include <sys/sa.h>
	#include <sys/policy.h>
	#include <sys/sunddi.h>
	#include <sys/sid.h>
	#include <sys/zfs_ctldir.h>
	#include <sys/zfs_fuid.h>
	#include <sys/zfs_quota.h>
	#include <sys/zfs_sa.h>
	#include <sys/zfs_vnops.h>
	#include <sys/zfs_rlock.h>
	#include <sys/cred.h>
	#include <sys/zpl.h>
	#include <sys/zil.h>
	#include <sys/sa_impl.h>

	/*
	* Programming rules.
	*
	* Each vnode op performs some logical unit of work. To do this, the ZPL must
	* properly lock its in-core state, create a DMU transaction, do the work,
	* record this work in the intent log (ZIL), commit the DMU transaction,
	* and wait for the intent log to commit if it is a synchronous operation.
	* Moreover, the vnode ops must work in both normal and log replay context.
	* The ordering of events is important to avoid deadlocks and references
	* to freed memory. The example below illustrates the following Big Rules:
	*
	* (1) A check must be made in each zfs thread for a mounted file system.
	* This is done avoiding races using ZFS_ENTER(zfsvfs).
	* A ZFS_EXIT(zfsvfs) is needed before all returns. Any znodes
	* must be checked with ZFS_VERIFY_ZP(zp). Both of these macros
	* can return EIO from the calling function.
	*
	* (2) zrele() should always be the last thing except for zil_commit()
	* (if necessary) and ZFS_EXIT(). This is for 3 reasons:
	* First, if it's the last reference, the vnode/znode
	* can be freed, so the zp may point to freed memory. Second, the last
	* reference will call zfs_zinactive(), which may induce a lot of work --
	* pushing cached pages (which acquires range locks) and syncing out
	* cached atime changes. Third, zfs_zinactive() may require a new tx,
	* which could deadlock the system if you were already holding one.
	* If you must call zrele() within a tx then use zfs_zrele_async().
	*
	* (3) All range locks must be grabbed before calling dmu_tx_assign(),
	* as they can span dmu_tx_assign() calls.
	*
	* (4) If ZPL locks are held, pass TXG_NOWAIT as the second argument to
	* dmu_tx_assign(). This is critical because we don't want to block
	* while holding locks.
	*
	* If no ZPL locks are held (aside from ZFS_ENTER()), use TXG_WAIT. This
	* reduces lock contention and CPU usage when we must wait (note that if
	* throughput is constrained by the storage, nearly every transaction
	* must wait).
	*
	* Note, in particular, that if a lock is sometimes acquired before
	* the tx assigns, and sometimes after (e.g. z_lock), then failing
	* to use a non-blocking assign can deadlock the system. The scenario:
	*
	* Thread A has grabbed a lock before calling dmu_tx_assign().
	* Thread B is in an already-assigned tx, and blocks for this lock.
	* Thread A calls dmu_tx_assign(TXG_WAIT) and blocks in txg_wait_open()
	* forever, because the previous txg can't quiesce until B's tx commits.
	*
	* If dmu_tx_assign() returns ERESTART and zfsvfs->z_assign is TXG_NOWAIT,
	* then drop all locks, call dmu_tx_wait(), and try again. On subsequent
	* calls to dmu_tx_assign(), pass TXG_NOTHROTTLE in addition to TXG_NOWAIT,
	* to indicate that this operation has already called dmu_tx_wait().
	* This will ensure that we don't retry forever, waiting a short bit
	* each time.
	*
	* (5) If the operation succeeded, generate the intent log entry for it
	* before dropping locks. This ensures that the ordering of events
	* in the intent log matches the order in which they actually occurred.
	* During ZIL replay the zfs_log_* functions will update the sequence
	* number to indicate the zil transaction has replayed.
	*
	* (6) At the end of each vnode op, the DMU tx must always commit,
	* regardless of whether there were any errors.
	*
	* (7) After dropping all locks, invoke zil_commit(zilog, foid)
	* to ensure that synchronous semantics are provided when necessary.
	*
	* In general, this is how things should be ordered in each vnode op:
	*
	* ZFS_ENTER(zfsvfs); // exit if unmounted
	* top:
	* zfs_dirent_lock(&dl, ...) // lock directory entry (may igrab())
	* rw_enter(...); // grab any other locks you need
	* tx = dmu_tx_create(...); // get DMU tx
	* dmu_tx_hold_*(); // hold each object you might modify
	* error = dmu_tx_assign(tx, (waited ? TXG_NOTHROTTLE : 0) \| TXG_NOWAIT);
	* if (error) {
	* rw_exit(...); // drop locks
	* zfs_dirent_unlock(dl); // unlock directory entry
	* zrele(...); // release held znodes
	* if (error == ERESTART) {
	* waited = B_TRUE;
	* dmu_tx_wait(tx);
	* dmu_tx_abort(tx);
	* goto top;
	* }
	* dmu_tx_abort(tx); // abort DMU tx
	* ZFS_EXIT(zfsvfs); // finished in zfs
	* return (error); // really out of space
	* }
	* error = do_real_work(); // do whatever this VOP does
	* if (error == 0)
	* zfs_log_*(...); // on success, make ZIL entry
	* dmu_tx_commit(tx); // commit DMU tx -- error or not
	* rw_exit(...); // drop locks
	* zfs_dirent_unlock(dl); // unlock directory entry
	* zrele(...); // release held znodes
	* zil_commit(zilog, foid); // synchronous when necessary
	* ZFS_EXIT(zfsvfs); // finished in zfs
	* return (error); // done, report error
	*/

	/*
	* Virus scanning is unsupported. It would be possible to add a hook
	* here to performance the required virus scan. This could be done
	* entirely in the kernel or potentially as an update to invoke a
	* scanning utility.
	*/
	static int
	zfs_vscan(struct inode ip, cred_t cr, int async)
	{
	return (0);
	}

	/* ARGSUSED */
	int
	zfs_open(struct inode ip, int mode, int flag, cred_t cr)
	{
	znode_t *zp = ITOZ(ip);
	zfsvfs_t *zfsvfs = ITOZSB(ip);

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	/* Honor ZFS_APPENDONLY file attribute */
	if ((mode & FMODE_WRITE) && (zp->z_pflags & ZFS_APPENDONLY) &&
	((flag & O_APPEND) == 0)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EPERM));
	}

	/* Virus scan eligible files on open */
	if (!zfs_has_ctldir(zp) && zfsvfs->z_vscan && S_ISREG(ip->i_mode) &&
	!(zp->z_pflags & ZFS_AV_QUARANTINED) && zp->z_size > 0) {
	if (zfs_vscan(ip, cr, 0) != 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EACCES));
	}
	}

	/* Keep a count of the synchronous opens in the znode */
	if (flag & O_SYNC)
	atomic_inc_32(&zp->z_sync_cnt);

	ZFS_EXIT(zfsvfs);
	return (0);
	}

	/* ARGSUSED */
	int
	zfs_close(struct inode ip, int flag, cred_t cr)
	{
	znode_t *zp = ITOZ(ip);
	zfsvfs_t *zfsvfs = ITOZSB(ip);

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	/* Decrement the synchronous opens in the znode */
	if (flag & O_SYNC)
	atomic_dec_32(&zp->z_sync_cnt);

	if (!zfs_has_ctldir(zp) && zfsvfs->z_vscan && S_ISREG(ip->i_mode) &&
	!(zp->z_pflags & ZFS_AV_QUARANTINED) && zp->z_size > 0)
	VERIFY(zfs_vscan(ip, cr, 1) == 0);

	ZFS_EXIT(zfsvfs);
	return (0);
	}

	#if defined(SEEK_HOLE) && defined(SEEK_DATA)
	/*
	* Lseek support for finding holes (cmd == SEEK_HOLE) and
	* data (cmd == SEEK_DATA). "off" is an in/out parameter.
	*/
	static int
	zfs_holey_common(struct inode ip, int cmd, loff_t off)
	{
	znode_t *zp = ITOZ(ip);
	uint64_t noff = (uint64_t)off; / new offset */
	uint64_t file_sz;
	int error;
	boolean_t hole;

	file_sz = zp->z_size;
	if (noff >= file_sz) {
	return (SET_ERROR(ENXIO));
	}

	if (cmd == SEEK_HOLE)
	hole = B_TRUE;
	else
	hole = B_FALSE;

	error = dmu_offset_next(ZTOZSB(zp)->z_os, zp->z_id, hole, &noff);

	if (error == ESRCH)
	return (SET_ERROR(ENXIO));

	/* file was dirty, so fall back to using generic logic */
	if (error == EBUSY) {
	if (hole)
	*off = file_sz;

	return (0);
	}

	/*
	* We could find a hole that begins after the logical end-of-file,
	* because dmu_offset_next() only works on whole blocks. If the
	* EOF falls mid-block, then indicate that the "virtual hole"
	* at the end of the file begins at the logical EOF, rather than
	* at the end of the last block.
	*/
	if (noff > file_sz) {
	ASSERT(hole);
	noff = file_sz;
	}

	if (noff < *off)
	return (error);
	*off = noff;
	return (error);
	}

	int
	zfs_holey(struct inode ip, int cmd, loff_t off)
	{
	znode_t *zp = ITOZ(ip);
	zfsvfs_t *zfsvfs = ITOZSB(ip);
	int error;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	error = zfs_holey_common(ip, cmd, off);

	ZFS_EXIT(zfsvfs);
	return (error);
	}
	#endif /* SEEK_HOLE && SEEK_DATA */

	#if defined(_KERNEL)
	/*
	* When a file is memory mapped, we must keep the IO data synchronized
	* between the DMU cache and the memory mapped pages. What this means:
	*
	* On Write: If we find a memory mapped page, we write to both
	* the page and the dmu buffer.
	*/
	static void
	update_pages(struct inode *ip, int64_t start, int len,
	objset_t *os, uint64_t oid)
	{
	struct address_space *mp = ip->i_mapping;
	struct page *pp;
	uint64_t nbytes;
	int64_t off;
	void *pb;

	off = start & (PAGE_SIZE-1);
	for (start &= PAGE_MASK; len > 0; start += PAGE_SIZE) {
	nbytes = MIN(PAGE_SIZE - off, len);

	pp = find_lock_page(mp, start >> PAGE_SHIFT);
	if (pp) {
	if (mapping_writably_mapped(mp))
	flush_dcache_page(pp);

	pb = kmap(pp);
	(void) dmu_read(os, oid, start+off, nbytes, pb+off,
	DMU_READ_PREFETCH);
	kunmap(pp);

	if (mapping_writably_mapped(mp))
	flush_dcache_page(pp);

	mark_page_accessed(pp);
	SetPageUptodate(pp);
	ClearPageError(pp);
	unlock_page(pp);
	put_page(pp);
	}

	len -= nbytes;
	off = 0;
	}
	}

	/*
	* When a file is memory mapped, we must keep the IO data synchronized
	* between the DMU cache and the memory mapped pages. What this means:
	*
	* On Read: We "read" preferentially from memory mapped pages,
	* else we default from the dmu buffer.
	*
	* NOTE: We will always "break up" the IO into PAGESIZE uiomoves when
	* the file is memory mapped.
	*/
	static int
	mappedread(struct inode ip, int nbytes, uio_t uio)
	{
	struct address_space *mp = ip->i_mapping;
	struct page *pp;
	znode_t *zp = ITOZ(ip);
	int64_t start, off;
	uint64_t bytes;
	int len = nbytes;
	int error = 0;
	void *pb;

	start = uio->uio_loffset;
	off = start & (PAGE_SIZE-1);
	for (start &= PAGE_MASK; len > 0; start += PAGE_SIZE) {
	bytes = MIN(PAGE_SIZE - off, len);

	pp = find_lock_page(mp, start >> PAGE_SHIFT);
	if (pp) {
	ASSERT(PageUptodate(pp));
	unlock_page(pp);

	pb = kmap(pp);
	error = uiomove(pb + off, bytes, UIO_READ, uio);
	kunmap(pp);

	if (mapping_writably_mapped(mp))
	flush_dcache_page(pp);

	mark_page_accessed(pp);
	put_page(pp);
	} else {
	error = dmu_read_uio_dbuf(sa_get_db(zp->z_sa_hdl),
	uio, bytes);
	}

	len -= bytes;
	off = 0;
	if (error)
	break;
	}
	return (error);
	}
	#endif /* _KERNEL */

	unsigned long zfs_read_chunk_size = 1024 * 1024; /* Tunable */
	unsigned long zfs_delete_blocks = DMU_MAX_DELETEBLKCNT;

	/*
	* Read bytes from specified file into supplied buffer.
	*
	* IN: ip - inode of file to be read from.
	* uio - structure supplying read location, range info,
	* and return buffer.
	* ioflag - O_SYNC flags; used to provide FRSYNC semantics.
	* O_DIRECT flag; used to bypass page cache.
	* cr - credentials of caller.
	*
	* OUT: uio - updated offset and range, buffer filled.
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Side Effects:
	* inode - atime updated if byte count > 0
	*/
	/* ARGSUSED */
	int
	zfs_read(struct inode ip, uio_t uio, int ioflag, cred_t *cr)
	{
	int error = 0;
	boolean_t frsync = B_FALSE;

	znode_t *zp = ITOZ(ip);
	zfsvfs_t *zfsvfs = ITOZSB(ip);
	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	if (zp->z_pflags & ZFS_AV_QUARANTINED) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EACCES));
	}

	/*
	* Validate file offset
	*/
	if (uio->uio_loffset < (offset_t)0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	/*
	* Fasttrack empty reads
	*/
	if (uio->uio_resid == 0) {
	ZFS_EXIT(zfsvfs);
	return (0);
	}

	#ifdef FRSYNC
	/*
	* If we're in FRSYNC mode, sync out this znode before reading it.
	* Only do this for non-snapshots.
	*
	* Some platforms do not support FRSYNC and instead map it
	* to O_SYNC, which results in unnecessary calls to zil_commit. We
	* only honor FRSYNC requests on platforms which support it.
	*/
	frsync = !!(ioflag & FRSYNC);
	#endif
	if (zfsvfs->z_log &&
	(frsync \|\| zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS))
	zil_commit(zfsvfs->z_log, zp->z_id);

	/*
	* Lock the range against changes.
	*/
	zfs_locked_range_t *lr = zfs_rangelock_enter(&zp->z_rangelock,
	uio->uio_loffset, uio->uio_resid, RL_READER);

	/*
	* If we are reading past end-of-file we can skip
	* to the end; but we might still need to set atime.
	*/
	if (uio->uio_loffset >= zp->z_size) {
	error = 0;
	goto out;
	}

	ASSERT(uio->uio_loffset < zp->z_size);
	ssize_t n = MIN(uio->uio_resid, zp->z_size - uio->uio_loffset);
	ssize_t start_resid = n;

	#ifdef HAVE_UIO_ZEROCOPY
	xuio_t *xuio = NULL;
	if ((uio->uio_extflg == UIO_XUIO) &&
	(((xuio_t *)uio)->xu_type == UIOTYPE_ZEROCOPY)) {
	int nblk;
	int blksz = zp->z_blksz;
	uint64_t offset = uio->uio_loffset;

	xuio = (xuio_t *)uio;
	if ((ISP2(blksz))) {
	nblk = (P2ROUNDUP(offset + n, blksz) - P2ALIGN(offset,
	blksz)) / blksz;
	} else {
	ASSERT(offset + n <= blksz);
	nblk = 1;
	}
	(void) dmu_xuio_init(xuio, nblk);

	if (vn_has_cached_data(ip)) {
	/*
	* For simplicity, we always allocate a full buffer
	* even if we only expect to read a portion of a block.
	*/
	while (--nblk >= 0) {
	(void) dmu_xuio_add(xuio,
	dmu_request_arcbuf(sa_get_db(zp->z_sa_hdl),
	blksz), 0, blksz);
	}
	}
	}
	#endif /* HAVE_UIO_ZEROCOPY */

	while (n > 0) {
	ssize_t nbytes = MIN(n, zfs_read_chunk_size -
	P2PHASE(uio->uio_loffset, zfs_read_chunk_size));

	if (zp->z_is_mapped && !(ioflag & O_DIRECT)) {
	error = mappedread(ip, nbytes, uio);
	} else {
	error = dmu_read_uio_dbuf(sa_get_db(zp->z_sa_hdl),
	uio, nbytes);
	}

	if (error) {
	/* convert checksum errors into IO errors */
	if (error == ECKSUM)
	error = SET_ERROR(EIO);
	break;
	}

	n -= nbytes;
	}

	int64_t nread = start_resid - n;
	dataset_kstats_update_read_kstats(&zfsvfs->z_kstat, nread);
	task_io_account_read(nread);
	out:
	zfs_rangelock_exit(lr);

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Write the bytes to a file.
	*
	* IN: ip - inode of file to be written to.
	* uio - structure supplying write location, range info,
	* and data buffer.
	* ioflag - O_APPEND flag set if in append mode.
	* O_DIRECT flag; used to bypass page cache.
	* cr - credentials of caller.
	*
	* OUT: uio - updated offset and range.
	*
	* RETURN: 0 if success
	* error code if failure
	*
	* Timestamps:
	* ip - ctime\|mtime updated if byte count > 0
	*/

	/* ARGSUSED */
	int
	zfs_write(struct inode ip, uio_t uio, int ioflag, cred_t *cr)
	{
	int error = 0;
	ssize_t start_resid = uio->uio_resid;

	/*
	* Fasttrack empty write
	*/
	ssize_t n = start_resid;
	if (n == 0)
	return (0);

	rlim64_t limit = uio->uio_limit;
	if (limit == RLIM64_INFINITY \|\| limit > MAXOFFSET_T)
	limit = MAXOFFSET_T;

	znode_t *zp = ITOZ(ip);
	zfsvfs_t *zfsvfs = ZTOZSB(zp);
	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	sa_bulk_attr_t bulk[4];
	int count = 0;
	uint64_t mtime[2], ctime[2];
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_MTIME(zfsvfs), NULL, &mtime, 16);
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_CTIME(zfsvfs), NULL, &ctime, 16);
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_SIZE(zfsvfs), NULL,
	&zp->z_size, 8);
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_FLAGS(zfsvfs), NULL,
	&zp->z_pflags, 8);

	/*
	* Callers might not be able to detect properly that we are read-only,
	* so check it explicitly here.
	*/
	if (zfs_is_readonly(zfsvfs)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EROFS));
	}

	/*
	* If immutable or not appending then return EPERM
	*/
	if ((zp->z_pflags & (ZFS_IMMUTABLE \| ZFS_READONLY)) \|\|
	((zp->z_pflags & ZFS_APPENDONLY) && !(ioflag & O_APPEND) &&
	(uio->uio_loffset < zp->z_size))) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EPERM));
	}

	/*
	* Validate file offset
	*/
	offset_t woff = ioflag & O_APPEND ? zp->z_size : uio->uio_loffset;
	if (woff < 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	int max_blksz = zfsvfs->z_max_blksz;
	xuio_t *xuio = NULL;

	/*
	* Pre-fault the pages to ensure slow (eg NFS) pages
	* don't hold up txg.
	* Skip this if uio contains loaned arc_buf.
	*/
	#ifdef HAVE_UIO_ZEROCOPY
	if ((uio->uio_extflg == UIO_XUIO) &&
	(((xuio_t *)uio)->xu_type == UIOTYPE_ZEROCOPY))
	xuio = (xuio_t *)uio;
	else
	#endif
	if (uio_prefaultpages(MIN(n, max_blksz), uio)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EFAULT));
	}

	/*
	* If in append mode, set the io offset pointer to eof.
	*/
	zfs_locked_range_t *lr;
	if (ioflag & O_APPEND) {
	/*
	* Obtain an appending range lock to guarantee file append
	* semantics. We reset the write offset once we have the lock.
	*/
	lr = zfs_rangelock_enter(&zp->z_rangelock, 0, n, RL_APPEND);
	woff = lr->lr_offset;
	if (lr->lr_length == UINT64_MAX) {
	/*
	* We overlocked the file because this write will cause
	* the file block size to increase.
	* Note that zp_size cannot change with this lock held.
	*/
	woff = zp->z_size;
	}
	uio->uio_loffset = woff;
	} else {
	/*
	* Note that if the file block size will change as a result of
	* this write, then this range lock will lock the entire file
	* so that we can re-write the block safely.
	*/
	lr = zfs_rangelock_enter(&zp->z_rangelock, woff, n, RL_WRITER);
	}

	if (woff >= limit) {
	zfs_rangelock_exit(lr);
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EFBIG));
	}

	if ((woff + n) > limit \|\| woff > (limit - n))
	n = limit - woff;

	/* Will this write extend the file length? */
	int write_eof = (woff + n > zp->z_size);

	uint64_t end_size = MAX(zp->z_size, woff + n);
	zilog_t *zilog = zfsvfs->z_log;
	#ifdef HAVE_UIO_ZEROCOPY
	int i_iov = 0;
	const iovec_t *iovp = uio->uio_iov;
	int iovcnt __maybe_unused = uio->uio_iovcnt;
	#endif


	/*
	* Write the file in reasonable size chunks. Each chunk is written
	* in a separate transaction; this keeps the intent log records small
	* and allows us to do more fine-grained space accounting.
	*/
	while (n > 0) {
	woff = uio->uio_loffset;

	if (zfs_id_overblockquota(zfsvfs, DMU_USERUSED_OBJECT,
	KUID_TO_SUID(ip->i_uid)) \|\|
	zfs_id_overblockquota(zfsvfs, DMU_GROUPUSED_OBJECT,
	KGID_TO_SGID(ip->i_gid)) \|\|
	(zp->z_projid != ZFS_DEFAULT_PROJID &&
	zfs_id_overblockquota(zfsvfs, DMU_PROJECTUSED_OBJECT,
	zp->z_projid))) {
	error = SET_ERROR(EDQUOT);
	break;
	}

	arc_buf_t *abuf = NULL;
	const iovec_t *aiov = NULL;
	if (xuio) {
	#ifdef HAVE_UIO_ZEROCOPY
	ASSERT(i_iov < iovcnt);
	ASSERT3U(uio->uio_segflg, !=, UIO_BVEC);
	aiov = &iovp[i_iov];
	abuf = dmu_xuio_arcbuf(xuio, i_iov);
	dmu_xuio_clear(xuio, i_iov);
	ASSERT((aiov->iov_base == abuf->b_data) \|\|
	((char )aiov->iov_base - (char )abuf->b_data +
	aiov->iov_len == arc_buf_size(abuf)));
	i_iov++;
	#endif
	} else if (n >= max_blksz && woff >= zp->z_size &&
	P2PHASE(woff, max_blksz) == 0 &&
	zp->z_blksz == max_blksz) {
	/*
	* This write covers a full block. "Borrow" a buffer
	* from the dmu so that we can fill it before we enter
	* a transaction. This avoids the possibility of
	* holding up the transaction if the data copy hangs
	* up on a pagefault (e.g., from an NFS server mapping).
	*/
	size_t cbytes;

	abuf = dmu_request_arcbuf(sa_get_db(zp->z_sa_hdl),
	max_blksz);
	ASSERT(abuf != NULL);
	ASSERT(arc_buf_size(abuf) == max_blksz);
	if ((error = uiocopy(abuf->b_data, max_blksz,
	UIO_WRITE, uio, &cbytes))) {
	dmu_return_arcbuf(abuf);
	break;
	}
	ASSERT(cbytes == max_blksz);
	}

	/*
	* Start a transaction.
	*/
	dmu_tx_t *tx = dmu_tx_create(zfsvfs->z_os);
	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_FALSE);
	dmu_buf_impl_t db = (dmu_buf_impl_t )sa_get_db(zp->z_sa_hdl);
	DB_DNODE_ENTER(db);
	dmu_tx_hold_write_by_dnode(tx, DB_DNODE(db), woff,
	MIN(n, max_blksz));
	DB_DNODE_EXIT(db);
	zfs_sa_upgrade_txholds(tx, zp);
	error = dmu_tx_assign(tx, TXG_WAIT);
	if (error) {
	dmu_tx_abort(tx);
	if (abuf != NULL)
	dmu_return_arcbuf(abuf);
	break;
	}

	/*
	* If rangelock_enter() over-locked we grow the blocksize
	* and then reduce the lock range. This will only happen
	* on the first iteration since rangelock_reduce() will
	* shrink down lr_length to the appropriate size.
	*/
	if (lr->lr_length == UINT64_MAX) {
	uint64_t new_blksz;

	if (zp->z_blksz > max_blksz) {
	/*
	* File's blocksize is already larger than the
	* "recordsize" property. Only let it grow to
	* the next power of 2.
	*/
	ASSERT(!ISP2(zp->z_blksz));
	new_blksz = MIN(end_size,
	1 << highbit64(zp->z_blksz));
	} else {
	new_blksz = MIN(end_size, max_blksz);
	}
	zfs_grow_blocksize(zp, new_blksz, tx);
	zfs_rangelock_reduce(lr, woff, n);
	}

	/*
	* XXX - should we really limit each write to z_max_blksz?
	* Perhaps we should use SPA_MAXBLOCKSIZE chunks?
	*/
	ssize_t nbytes = MIN(n, max_blksz - P2PHASE(woff, max_blksz));

	ssize_t tx_bytes;
	if (abuf == NULL) {
	tx_bytes = uio->uio_resid;
	uio->uio_fault_disable = B_TRUE;
	error = dmu_write_uio_dbuf(sa_get_db(zp->z_sa_hdl),
	uio, nbytes, tx);
	uio->uio_fault_disable = B_FALSE;
	if (error == EFAULT) {
	dmu_tx_commit(tx);
	/*
	* Account for partial writes before
	* continuing the loop.
	* Update needs to occur before the next
	* uio_prefaultpages, or prefaultpages may
	* error, and we may break the loop early.
	*/
	if (tx_bytes != uio->uio_resid)
	n -= tx_bytes - uio->uio_resid;
	if (uio_prefaultpages(MIN(n, max_blksz), uio)) {
	break;
	}
	continue;
	} else if (error != 0) {
	dmu_tx_commit(tx);
	break;
	}
	tx_bytes -= uio->uio_resid;
	} else {
	tx_bytes = nbytes;
	ASSERT(xuio == NULL \|\| tx_bytes == aiov->iov_len);
	/*
	* If this is not a full block write, but we are
	* extending the file past EOF and this data starts
	* block-aligned, use assign_arcbuf(). Otherwise,
	* write via dmu_write().
	*/
	if (tx_bytes < max_blksz && (!write_eof \|\|
	aiov->iov_base != abuf->b_data)) {
	ASSERT(xuio);
	dmu_write(zfsvfs->z_os, zp->z_id, woff,
	/* cppcheck-suppress nullPointer */
	aiov->iov_len, aiov->iov_base, tx);
	dmu_return_arcbuf(abuf);
	xuio_stat_wbuf_copied();
	} else {
	ASSERT(xuio \|\| tx_bytes == max_blksz);
	error = dmu_assign_arcbuf_by_dbuf(
	sa_get_db(zp->z_sa_hdl), woff, abuf, tx);
	if (error != 0) {
	dmu_return_arcbuf(abuf);
	dmu_tx_commit(tx);
	break;
	}
	}
	ASSERT(tx_bytes <= uio->uio_resid);
	uioskip(uio, tx_bytes);
	}
	if (tx_bytes && zp->z_is_mapped && !(ioflag & O_DIRECT)) {
	update_pages(ip, woff,
	tx_bytes, zfsvfs->z_os, zp->z_id);
	}

	/*
	* If we made no progress, we're done. If we made even
	* partial progress, update the znode and ZIL accordingly.
	*/
	if (tx_bytes == 0) {
	(void) sa_update(zp->z_sa_hdl, SA_ZPL_SIZE(zfsvfs),
	(void *)&zp->z_size, sizeof (uint64_t), tx);
	dmu_tx_commit(tx);
	ASSERT(error != 0);
	break;
	}

	/*
	* Clear Set-UID/Set-GID bits on successful write if not
	* privileged and at least one of the execute bits is set.
	*
	* It would be nice to do this after all writes have
	* been done, but that would still expose the ISUID/ISGID
	* to another app after the partial write is committed.
	*
	* Note: we don't call zfs_fuid_map_id() here because
	* user 0 is not an ephemeral uid.
	*/
	mutex_enter(&zp->z_acl_lock);
	uint32_t uid = KUID_TO_SUID(ip->i_uid);
	if ((zp->z_mode & (S_IXUSR \| (S_IXUSR >> 3) \|
	(S_IXUSR >> 6))) != 0 &&
	(zp->z_mode & (S_ISUID \| S_ISGID)) != 0 &&
	secpolicy_vnode_setid_retain(cr,
	((zp->z_mode & S_ISUID) != 0 && uid == 0)) != 0) {
	uint64_t newmode;
	zp->z_mode &= ~(S_ISUID \| S_ISGID);
	ip->i_mode = newmode = zp->z_mode;
	(void) sa_update(zp->z_sa_hdl, SA_ZPL_MODE(zfsvfs),
	(void *)&newmode, sizeof (uint64_t), tx);
	}
	mutex_exit(&zp->z_acl_lock);

	zfs_tstamp_update_setup(zp, CONTENT_MODIFIED, mtime, ctime);

	/*
	* Update the file size (zp_size) if it has changed;
	* account for possible concurrent updates.
	*/
	while ((end_size = zp->z_size) < uio->uio_loffset) {
	(void) atomic_cas_64(&zp->z_size, end_size,
	uio->uio_loffset);
	ASSERT(error == 0);
	}
	/*
	* If we are replaying and eof is non zero then force
	* the file size to the specified eof. Note, there's no
	* concurrency during replay.
	*/
	if (zfsvfs->z_replay && zfsvfs->z_replay_eof != 0)
	zp->z_size = zfsvfs->z_replay_eof;

	error = sa_bulk_update(zp->z_sa_hdl, bulk, count, tx);

	zfs_log_write(zilog, tx, TX_WRITE, zp, woff, tx_bytes, ioflag,
	NULL, NULL);
	dmu_tx_commit(tx);

	if (error != 0)
	break;
	ASSERT(tx_bytes == nbytes);
	n -= nbytes;

	if (!xuio && n > 0) {
	if (uio_prefaultpages(MIN(n, max_blksz), uio)) {
	error = EFAULT;
	break;
	}
	}
	}

	zfs_inode_update(zp);
	zfs_rangelock_exit(lr);

	/*
	* If we're in replay mode, or we made no progress, return error.
	* Otherwise, it's at least a partial write, so it's successful.
	*/
	if (zfsvfs->z_replay \|\| uio->uio_resid == start_resid) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	if (ioflag & (O_SYNC \| O_DSYNC) \|\|
	zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, zp->z_id);

	int64_t nwritten = start_resid - uio->uio_resid;
	dataset_kstats_update_write_kstats(&zfsvfs->z_kstat, nwritten);
	task_io_account_write(nwritten);

	ZFS_EXIT(zfsvfs);
	return (0);
	}

	/*
	* Write the bytes to a file.
	*
	* IN: zp - znode of file to be written to
	* data - bytes to write
	* len - number of bytes to write
	* pos - offset to start writing at
	*
	* OUT: resid - remaining bytes to write
	*
	* RETURN: 0 if success
	* positive error code if failure
	*
	* Timestamps:
	* zp - ctime\|mtime updated if byte count > 0
	*/
	int
	zfs_write_simple(znode_t zp, const void data, size_t len,
	loff_t pos, size_t *resid)
	{
	ssize_t written;
	int error = 0;

	written = zpl_write_common(ZTOI(zp), data, len, &pos,
	UIO_SYSSPACE, 0, kcred);
	if (written < 0) {
	error = -written;
	} else if (resid == NULL) {
	if (written < len)
	error = SET_ERROR(EIO); /* short write */
	} else {
	*resid = len - written;
	}
	return (error);
	}

	/*
	* Drop a reference on the passed inode asynchronously. This ensures
	* that the caller will never drop the last reference on an inode in
	* the current context. Doing so while holding open a tx could result
	* in a deadlock if iput_final() re-enters the filesystem code.
	*/
	void
	zfs_zrele_async(znode_t *zp)
	{
	struct inode *ip = ZTOI(zp);
	objset_t *os = ITOZSB(ip)->z_os;

	ASSERT(atomic_read(&ip->i_count) > 0);
	ASSERT(os != NULL);

	if (atomic_read(&ip->i_count) == 1)
	VERIFY(taskq_dispatch(dsl_pool_zrele_taskq(dmu_objset_pool(os)),
	(task_func_t *)iput, ip, TQ_SLEEP) != TASKQID_INVALID);
	else
	zrele(zp);
	}

	/* ARGSUSED */
	static void
	zfs_get_done(zgd_t *zgd, int error)
	{
	znode_t *zp = zgd->zgd_private;

	if (zgd->zgd_db)
	dmu_buf_rele(zgd->zgd_db, zgd);

	zfs_rangelock_exit(zgd->zgd_lr);

	/*
	* Release the vnode asynchronously as we currently have the
	* txg stopped from syncing.
	*/
	zfs_zrele_async(zp);

	kmem_free(zgd, sizeof (zgd_t));
	}

	#ifdef ZFS_DEBUG
	static int zil_fault_io = 0;
	#endif

	/*
	* Get data to generate a TX_WRITE intent log record.
	*/
	int
	zfs_get_data(void arg, lr_write_t lr, char buf, struct lwb lwb, zio_t *zio)
	{
	zfsvfs_t *zfsvfs = arg;
	objset_t *os = zfsvfs->z_os;
	znode_t *zp;
	uint64_t object = lr->lr_foid;
	uint64_t offset = lr->lr_offset;
	uint64_t size = lr->lr_length;
	dmu_buf_t *db;
	zgd_t *zgd;
	int error = 0;

	ASSERT3P(lwb, !=, NULL);
	ASSERT3P(zio, !=, NULL);
	ASSERT3U(size, !=, 0);

	/*
	* Nothing to do if the file has been removed
	*/
	if (zfs_zget(zfsvfs, object, &zp) != 0)
	return (SET_ERROR(ENOENT));
	if (zp->z_unlinked) {
	/*
	* Release the vnode asynchronously as we currently have the
	* txg stopped from syncing.
	*/
	zfs_zrele_async(zp);
	return (SET_ERROR(ENOENT));
	}

	zgd = kmem_zalloc(sizeof (zgd_t), KM_SLEEP);
	zgd->zgd_lwb = lwb;
	zgd->zgd_private = zp;

	/*
	* Write records come in two flavors: immediate and indirect.
	* For small writes it's cheaper to store the data with the
	* log record (immediate); for large writes it's cheaper to
	* sync the data and get a pointer to it (indirect) so that
	* we don't have to write the data twice.
	*/
	if (buf != NULL) { /* immediate write */
	zgd->zgd_lr = zfs_rangelock_enter(&zp->z_rangelock,
	offset, size, RL_READER);
	/* test for truncation needs to be done while range locked */
	if (offset >= zp->z_size) {
	error = SET_ERROR(ENOENT);
	} else {
	error = dmu_read(os, object, offset, size, buf,
	DMU_READ_NO_PREFETCH);
	}
	ASSERT(error == 0 \|\| error == ENOENT);
	} else { /* indirect write */
	/*
	* Have to lock the whole block to ensure when it's
	* written out and its checksum is being calculated
	* that no one can change the data. We need to re-check
	* blocksize after we get the lock in case it's changed!
	*/
	for (;;) {
	uint64_t blkoff;
	size = zp->z_blksz;
	blkoff = ISP2(size) ? P2PHASE(offset, size) : offset;
	offset -= blkoff;
	zgd->zgd_lr = zfs_rangelock_enter(&zp->z_rangelock,
	offset, size, RL_READER);
	if (zp->z_blksz == size)
	break;
	offset += blkoff;
	zfs_rangelock_exit(zgd->zgd_lr);
	}
	/* test for truncation needs to be done while range locked */
	if (lr->lr_offset >= zp->z_size)
	error = SET_ERROR(ENOENT);
	#ifdef ZFS_DEBUG
	if (zil_fault_io) {
	error = SET_ERROR(EIO);
	zil_fault_io = 0;
	}
	#endif
	if (error == 0)
	error = dmu_buf_hold(os, object, offset, zgd, &db,
	DMU_READ_NO_PREFETCH);

	if (error == 0) {
	blkptr_t *bp = &lr->lr_blkptr;

	zgd->zgd_db = db;
	zgd->zgd_bp = bp;

	ASSERT(db->db_offset == offset);
	ASSERT(db->db_size == size);

	error = dmu_sync(zio, lr->lr_common.lrc_txg,
	zfs_get_done, zgd);
	ASSERT(error \|\| lr->lr_length <= size);

	/*
	* On success, we need to wait for the write I/O
	* initiated by dmu_sync() to complete before we can
	* release this dbuf. We will finish everything up
	* in the zfs_get_done() callback.
	*/
	if (error == 0)
	return (0);

	if (error == EALREADY) {
	lr->lr_common.lrc_txtype = TX_WRITE2;
	/*
	* TX_WRITE2 relies on the data previously
	* written by the TX_WRITE that caused
	* EALREADY. We zero out the BP because
	* it is the old, currently-on-disk BP.
	*/
	zgd->zgd_bp = NULL;
	BP_ZERO(bp);
	error = 0;
	}
	}
	}

	zfs_get_done(zgd, error);

	return (error);
	}

	/ARGSUSED/
	int
	zfs_access(struct inode ip, int mode, int flag, cred_t cr)
	{
	znode_t *zp = ITOZ(ip);
	zfsvfs_t *zfsvfs = ITOZSB(ip);
	int error;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	if (flag & V_ACE_MASK)
	error = zfs_zaccess(zp, mode, flag, B_FALSE, cr);
	else
	error = zfs_zaccess_rwx(zp, mode, flag, cr);

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Lookup an entry in a directory, or an extended attribute directory.
	* If it exists, return a held inode reference for it.
	*
	* IN: zdp - znode of directory to search.
	* nm - name of entry to lookup.
	* flags - LOOKUP_XATTR set if looking for an attribute.
	* cr - credentials of caller.
	* direntflags - directory lookup flags
	* realpnp - returned pathname.
	*
	* OUT: zpp - znode of located entry, NULL if not found.
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* NA
	*/
	/* ARGSUSED */
	int
	-zfs_lookup(znode_t zdp, char nm, znode_t **zpp, int flags,
	- cred_t cr, int direntflags, pathname_t *realpnp)
	+zfs_lookup(znode_t zdp, char nm, znode_t *zpp, int flags, cred_t cr,
	+ int direntflags, pathname_t realpnp)
	{
	zfsvfs_t *zfsvfs = ZTOZSB(zdp);
	int error = 0;

	/*
	* Fast path lookup, however we must skip DNLC lookup
	* for case folding or normalizing lookups because the
	* DNLC code only stores the passed in name. This means
	* creating 'a' and removing 'A' on a case insensitive
	* file system would work, but DNLC still thinks 'a'
	* exists and won't let you create it again on the next
	* pass through fast path.
	*/
	if (!(flags & (LOOKUP_XATTR \| FIGNORECASE))) {

	if (!S_ISDIR(ZTOI(zdp)->i_mode)) {
	return (SET_ERROR(ENOTDIR));
	} else if (zdp->z_sa_hdl == NULL) {
	return (SET_ERROR(EIO));
	}

	if (nm[0] == 0 \|\| (nm[0] == '.' && nm[1] == '\0')) {
	error = zfs_fastaccesschk_execute(zdp, cr);
	if (!error) {
	*zpp = zdp;
	zhold(*zpp);
	return (0);
	}
	return (error);
	}
	}

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zdp);

	*zpp = NULL;

	if (flags & LOOKUP_XATTR) {
	/*
	* We don't allow recursive attributes..
	* Maybe someday we will.
	*/
	if (zdp->z_pflags & ZFS_XATTR) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	if ((error = zfs_get_xattrdir(zdp, zpp, cr, flags))) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Do we have permission to get into attribute directory?
	*/

	if ((error = zfs_zaccess(*zpp, ACE_EXECUTE, 0,
	B_FALSE, cr))) {
	zrele(*zpp);
	*zpp = NULL;
	}

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	if (!S_ISDIR(ZTOI(zdp)->i_mode)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(ENOTDIR));
	}

	/*
	* Check accessibility of directory.
	*/

	if ((error = zfs_zaccess(zdp, ACE_EXECUTE, 0, B_FALSE, cr))) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	if (zfsvfs->z_utf8 && u8_validate(nm, strlen(nm),
	NULL, U8_VALIDATE_ENTIRE, &error) < 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EILSEQ));
	}

	error = zfs_dirlook(zdp, nm, zpp, flags, direntflags, realpnp);
	if ((error == 0) && (*zpp))
	zfs_inode_update(*zpp);

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Attempt to create a new entry in a directory. If the entry
	* already exists, truncate the file if permissible, else return
	* an error. Return the ip of the created or trunc'd file.
	*
	* IN: dzp - znode of directory to put new file entry in.
	* name - name of new file entry.
	* vap - attributes of new file.
	* excl - flag indicating exclusive or non-exclusive mode.
	* mode - mode to open file with.
	* cr - credentials of caller.
	* flag - file flag.
	* vsecp - ACL to be set
	*
	* OUT: zpp - znode of created or trunc'd entry.
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* dzp - ctime\|mtime updated if new entry created
	* zp - ctime\|mtime always, atime if new
	*/

	/* ARGSUSED */
	int
	zfs_create(znode_t dzp, char name, vattr_t *vap, int excl,
	int mode, znode_t *zpp, cred_t cr, int flag, vsecattr_t *vsecp)
	{
	znode_t *zp;
	zfsvfs_t *zfsvfs = ZTOZSB(dzp);
	zilog_t *zilog;
	objset_t *os;
	zfs_dirlock_t *dl;
	dmu_tx_t *tx;
	int error;
	uid_t uid;
	gid_t gid;
	zfs_acl_ids_t acl_ids;
	boolean_t fuid_dirtied;
	boolean_t have_acl = B_FALSE;
	boolean_t waited = B_FALSE;

	/*
	* If we have an ephemeral id, ACL, or XVATTR then
	* make sure file system is at proper version
	*/

	gid = crgetgid(cr);
	uid = crgetuid(cr);

	if (zfsvfs->z_use_fuids == B_FALSE &&
	(vsecp \|\| IS_EPHEMERAL(uid) \|\| IS_EPHEMERAL(gid)))
	return (SET_ERROR(EINVAL));

	if (name == NULL)
	return (SET_ERROR(EINVAL));

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(dzp);
	os = zfsvfs->z_os;
	zilog = zfsvfs->z_log;

	if (zfsvfs->z_utf8 && u8_validate(name, strlen(name),
	NULL, U8_VALIDATE_ENTIRE, &error) < 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EILSEQ));
	}

	if (vap->va_mask & ATTR_XVATTR) {
	if ((error = secpolicy_xvattr((xvattr_t *)vap,
	crgetuid(cr), cr, vap->va_mode)) != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	}

	top:
	*zpp = NULL;
	if (*name == '\0') {
	/*
	* Null component name refers to the directory itself.
	*/
	zhold(dzp);
	zp = dzp;
	dl = NULL;
	error = 0;
	} else {
	/* possible igrab(zp) */
	int zflg = 0;

	if (flag & FIGNORECASE)
	zflg \|= ZCILOOK;

	error = zfs_dirent_lock(&dl, dzp, name, &zp, zflg,
	NULL, NULL);
	if (error) {
	if (have_acl)
	zfs_acl_ids_free(&acl_ids);
	if (strcmp(name, "..") == 0)
	error = SET_ERROR(EISDIR);
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	}

	if (zp == NULL) {
	uint64_t txtype;
	uint64_t projid = ZFS_DEFAULT_PROJID;

	/*
	* Create a new file object and update the directory
	* to reference it.
	*/
	if ((error = zfs_zaccess(dzp, ACE_ADD_FILE, 0, B_FALSE, cr))) {
	if (have_acl)
	zfs_acl_ids_free(&acl_ids);
	goto out;
	}

	/*
	* We only support the creation of regular files in
	* extended attribute directories.
	*/

	if ((dzp->z_pflags & ZFS_XATTR) && !S_ISREG(vap->va_mode)) {
	if (have_acl)
	zfs_acl_ids_free(&acl_ids);
	error = SET_ERROR(EINVAL);
	goto out;
	}

	if (!have_acl && (error = zfs_acl_ids_create(dzp, 0, vap,
	cr, vsecp, &acl_ids)) != 0)
	goto out;
	have_acl = B_TRUE;

	if (S_ISREG(vap->va_mode) \|\| S_ISDIR(vap->va_mode))
	projid = zfs_inherit_projid(dzp);
	if (zfs_acl_ids_overquota(zfsvfs, &acl_ids, projid)) {
	zfs_acl_ids_free(&acl_ids);
	error = SET_ERROR(EDQUOT);
	goto out;
	}

	tx = dmu_tx_create(os);

	dmu_tx_hold_sa_create(tx, acl_ids.z_aclp->z_acl_bytes +
	ZFS_SA_BASE_ATTR_SIZE);

	fuid_dirtied = zfsvfs->z_fuid_dirty;
	if (fuid_dirtied)
	zfs_fuid_txhold(zfsvfs, tx);
	dmu_tx_hold_zap(tx, dzp->z_id, TRUE, name);
	dmu_tx_hold_sa(tx, dzp->z_sa_hdl, B_FALSE);
	if (!zfsvfs->z_use_sa &&
	acl_ids.z_aclp->z_acl_bytes > ZFS_ACE_SPACE) {
	dmu_tx_hold_write(tx, DMU_NEW_OBJECT,
	0, acl_ids.z_aclp->z_acl_bytes);
	}

	error = dmu_tx_assign(tx,
	(waited ? TXG_NOTHROTTLE : 0) \| TXG_NOWAIT);
	if (error) {
	zfs_dirent_unlock(dl);
	if (error == ERESTART) {
	waited = B_TRUE;
	dmu_tx_wait(tx);
	dmu_tx_abort(tx);
	goto top;
	}
	zfs_acl_ids_free(&acl_ids);
	dmu_tx_abort(tx);
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	zfs_mknode(dzp, vap, tx, cr, 0, &zp, &acl_ids);

	error = zfs_link_create(dl, zp, tx, ZNEW);
	if (error != 0) {
	/*
	* Since, we failed to add the directory entry for it,
	* delete the newly created dnode.
	*/
	zfs_znode_delete(zp, tx);
	remove_inode_hash(ZTOI(zp));
	zfs_acl_ids_free(&acl_ids);
	dmu_tx_commit(tx);
	goto out;
	}

	if (fuid_dirtied)
	zfs_fuid_sync(zfsvfs, tx);

	txtype = zfs_log_create_txtype(Z_FILE, vsecp, vap);
	if (flag & FIGNORECASE)
	txtype \|= TX_CI;
	zfs_log_create(zilog, tx, txtype, dzp, zp, name,
	vsecp, acl_ids.z_fuidp, vap);
	zfs_acl_ids_free(&acl_ids);
	dmu_tx_commit(tx);
	} else {
	int aflags = (flag & O_APPEND) ? V_APPEND : 0;

	if (have_acl)
	zfs_acl_ids_free(&acl_ids);
	have_acl = B_FALSE;

	/*
	* A directory entry already exists for this name.
	*/
	/*
	* Can't truncate an existing file if in exclusive mode.
	*/
	if (excl) {
	error = SET_ERROR(EEXIST);
	goto out;
	}
	/*
	* Can't open a directory for writing.
	*/
	if (S_ISDIR(ZTOI(zp)->i_mode)) {
	error = SET_ERROR(EISDIR);
	goto out;
	}
	/*
	* Verify requested access to file.
	*/
	if (mode && (error = zfs_zaccess_rwx(zp, mode, aflags, cr))) {
	goto out;
	}

	mutex_enter(&dzp->z_lock);
	dzp->z_seq++;
	mutex_exit(&dzp->z_lock);

	/*
	* Truncate regular files if requested.
	*/
	if (S_ISREG(ZTOI(zp)->i_mode) &&
	(vap->va_mask & ATTR_SIZE) && (vap->va_size == 0)) {
	/* we can't hold any locks when calling zfs_freesp() */
	if (dl) {
	zfs_dirent_unlock(dl);
	dl = NULL;
	}
	error = zfs_freesp(zp, 0, 0, mode, TRUE);
	}
	}
	out:

	if (dl)
	zfs_dirent_unlock(dl);

	if (error) {
	if (zp)
	zrele(zp);
	} else {
	zfs_inode_update(dzp);
	zfs_inode_update(zp);
	*zpp = zp;
	}

	if (zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, 0);

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/* ARGSUSED */
	int
	zfs_tmpfile(struct inode dip, vattr_t vap, int excl,
	int mode, struct inode *ipp, cred_t cr, int flag, vsecattr_t *vsecp)
	{
	znode_t zp = NULL, dzp = ITOZ(dip);
	zfsvfs_t *zfsvfs = ITOZSB(dip);
	objset_t *os;
	dmu_tx_t *tx;
	int error;
	uid_t uid;
	gid_t gid;
	zfs_acl_ids_t acl_ids;
	uint64_t projid = ZFS_DEFAULT_PROJID;
	boolean_t fuid_dirtied;
	boolean_t have_acl = B_FALSE;
	boolean_t waited = B_FALSE;

	/*
	* If we have an ephemeral id, ACL, or XVATTR then
	* make sure file system is at proper version
	*/

	gid = crgetgid(cr);
	uid = crgetuid(cr);

	if (zfsvfs->z_use_fuids == B_FALSE &&
	(vsecp \|\| IS_EPHEMERAL(uid) \|\| IS_EPHEMERAL(gid)))
	return (SET_ERROR(EINVAL));

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(dzp);
	os = zfsvfs->z_os;

	if (vap->va_mask & ATTR_XVATTR) {
	if ((error = secpolicy_xvattr((xvattr_t *)vap,
	crgetuid(cr), cr, vap->va_mode)) != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	}

	top:
	*ipp = NULL;

	/*
	* Create a new file object and update the directory
	* to reference it.
	*/
	if ((error = zfs_zaccess(dzp, ACE_ADD_FILE, 0, B_FALSE, cr))) {
	if (have_acl)
	zfs_acl_ids_free(&acl_ids);
	goto out;
	}

	if (!have_acl && (error = zfs_acl_ids_create(dzp, 0, vap,
	cr, vsecp, &acl_ids)) != 0)
	goto out;
	have_acl = B_TRUE;

	if (S_ISREG(vap->va_mode) \|\| S_ISDIR(vap->va_mode))
	projid = zfs_inherit_projid(dzp);
	if (zfs_acl_ids_overquota(zfsvfs, &acl_ids, projid)) {
	zfs_acl_ids_free(&acl_ids);
	error = SET_ERROR(EDQUOT);
	goto out;
	}

	tx = dmu_tx_create(os);

	dmu_tx_hold_sa_create(tx, acl_ids.z_aclp->z_acl_bytes +
	ZFS_SA_BASE_ATTR_SIZE);
	dmu_tx_hold_zap(tx, zfsvfs->z_unlinkedobj, FALSE, NULL);

	fuid_dirtied = zfsvfs->z_fuid_dirty;
	if (fuid_dirtied)
	zfs_fuid_txhold(zfsvfs, tx);
	if (!zfsvfs->z_use_sa &&
	acl_ids.z_aclp->z_acl_bytes > ZFS_ACE_SPACE) {
	dmu_tx_hold_write(tx, DMU_NEW_OBJECT,
	0, acl_ids.z_aclp->z_acl_bytes);
	}
	error = dmu_tx_assign(tx, (waited ? TXG_NOTHROTTLE : 0) \| TXG_NOWAIT);
	if (error) {
	if (error == ERESTART) {
	waited = B_TRUE;
	dmu_tx_wait(tx);
	dmu_tx_abort(tx);
	goto top;
	}
	zfs_acl_ids_free(&acl_ids);
	dmu_tx_abort(tx);
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	zfs_mknode(dzp, vap, tx, cr, IS_TMPFILE, &zp, &acl_ids);

	if (fuid_dirtied)
	zfs_fuid_sync(zfsvfs, tx);

	/* Add to unlinked set */
	zp->z_unlinked = B_TRUE;
	zfs_unlinked_add(zp, tx);
	zfs_acl_ids_free(&acl_ids);
	dmu_tx_commit(tx);
	out:

	if (error) {
	if (zp)
	zrele(zp);
	} else {
	zfs_inode_update(dzp);
	zfs_inode_update(zp);
	*ipp = ZTOI(zp);
	}

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Remove an entry from a directory.
	*
	* IN: dzp - znode of directory to remove entry from.
	* name - name of entry to remove.
	* cr - credentials of caller.
	* flags - case flags.
	*
	* RETURN: 0 if success
	* error code if failure
	*
	* Timestamps:
	* dzp - ctime\|mtime
	* ip - ctime (if nlink > 0)
	*/

	uint64_t null_xattr = 0;

	/ARGSUSED/
	int
	zfs_remove(znode_t dzp, char name, cred_t *cr, int flags)
	{
	znode_t *zp;
	znode_t *xzp;
	zfsvfs_t *zfsvfs = ZTOZSB(dzp);
	zilog_t *zilog;
	uint64_t acl_obj, xattr_obj;
	uint64_t xattr_obj_unlinked = 0;
	uint64_t obj = 0;
	uint64_t links;
	zfs_dirlock_t *dl;
	dmu_tx_t *tx;
	boolean_t may_delete_now, delete_now = FALSE;
	boolean_t unlinked, toobig = FALSE;
	uint64_t txtype;
	pathname_t *realnmp = NULL;
	pathname_t realnm;
	int error;
	int zflg = ZEXISTS;
	boolean_t waited = B_FALSE;

	if (name == NULL)
	return (SET_ERROR(EINVAL));

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(dzp);
	zilog = zfsvfs->z_log;

	if (flags & FIGNORECASE) {
	zflg \|= ZCILOOK;
	pn_alloc(&realnm);
	realnmp = &realnm;
	}

	top:
	xattr_obj = 0;
	xzp = NULL;
	/*
	* Attempt to lock directory; fail if entry doesn't exist.
	*/
	if ((error = zfs_dirent_lock(&dl, dzp, name, &zp, zflg,
	NULL, realnmp))) {
	if (realnmp)
	pn_free(realnmp);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	if ((error = zfs_zaccess_delete(dzp, zp, cr))) {
	goto out;
	}

	/*
	* Need to use rmdir for removing directories.
	*/
	if (S_ISDIR(ZTOI(zp)->i_mode)) {
	error = SET_ERROR(EPERM);
	goto out;
	}

	mutex_enter(&zp->z_lock);
	may_delete_now = atomic_read(&ZTOI(zp)->i_count) == 1 &&
	!(zp->z_is_mapped);
	mutex_exit(&zp->z_lock);

	/*
	* We may delete the znode now, or we may put it in the unlinked set;
	* it depends on whether we're the last link, and on whether there are
	* other holds on the inode. So we dmu_tx_hold() the right things to
	* allow for either case.
	*/
	obj = zp->z_id;
	tx = dmu_tx_create(zfsvfs->z_os);
	dmu_tx_hold_zap(tx, dzp->z_id, FALSE, name);
	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_FALSE);
	zfs_sa_upgrade_txholds(tx, zp);
	zfs_sa_upgrade_txholds(tx, dzp);
	if (may_delete_now) {
	toobig = zp->z_size > zp->z_blksz * zfs_delete_blocks;
	/* if the file is too big, only hold_free a token amount */
	dmu_tx_hold_free(tx, zp->z_id, 0,
	(toobig ? DMU_MAX_ACCESS : DMU_OBJECT_END));
	}

	/* are there any extended attributes? */
	error = sa_lookup(zp->z_sa_hdl, SA_ZPL_XATTR(zfsvfs),
	&xattr_obj, sizeof (xattr_obj));
	if (error == 0 && xattr_obj) {
	error = zfs_zget(zfsvfs, xattr_obj, &xzp);
	ASSERT0(error);
	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_TRUE);
	dmu_tx_hold_sa(tx, xzp->z_sa_hdl, B_FALSE);
	}

	mutex_enter(&zp->z_lock);
	if ((acl_obj = zfs_external_acl(zp)) != 0 && may_delete_now)
	dmu_tx_hold_free(tx, acl_obj, 0, DMU_OBJECT_END);
	mutex_exit(&zp->z_lock);

	/* charge as an update -- would be nice not to charge at all */
	dmu_tx_hold_zap(tx, zfsvfs->z_unlinkedobj, FALSE, NULL);

	/*
	* Mark this transaction as typically resulting in a net free of space
	*/
	dmu_tx_mark_netfree(tx);

	error = dmu_tx_assign(tx, (waited ? TXG_NOTHROTTLE : 0) \| TXG_NOWAIT);
	if (error) {
	zfs_dirent_unlock(dl);
	if (error == ERESTART) {
	waited = B_TRUE;
	dmu_tx_wait(tx);
	dmu_tx_abort(tx);
	zrele(zp);
	if (xzp)
	zrele(xzp);
	goto top;
	}
	if (realnmp)
	pn_free(realnmp);
	dmu_tx_abort(tx);
	zrele(zp);
	if (xzp)
	zrele(xzp);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Remove the directory entry.
	*/
	error = zfs_link_destroy(dl, zp, tx, zflg, &unlinked);

	if (error) {
	dmu_tx_commit(tx);
	goto out;
	}

	if (unlinked) {
	/*
	* Hold z_lock so that we can make sure that the ACL obj
	* hasn't changed. Could have been deleted due to
	* zfs_sa_upgrade().
	*/
	mutex_enter(&zp->z_lock);
	(void) sa_lookup(zp->z_sa_hdl, SA_ZPL_XATTR(zfsvfs),
	&xattr_obj_unlinked, sizeof (xattr_obj_unlinked));
	delete_now = may_delete_now && !toobig &&
	atomic_read(&ZTOI(zp)->i_count) == 1 &&
	!(zp->z_is_mapped) && xattr_obj == xattr_obj_unlinked &&
	zfs_external_acl(zp) == acl_obj;
	}

	if (delete_now) {
	if (xattr_obj_unlinked) {
	ASSERT3U(ZTOI(xzp)->i_nlink, ==, 2);
	mutex_enter(&xzp->z_lock);
	xzp->z_unlinked = B_TRUE;
	clear_nlink(ZTOI(xzp));
	links = 0;
	error = sa_update(xzp->z_sa_hdl, SA_ZPL_LINKS(zfsvfs),
	&links, sizeof (links), tx);
	ASSERT3U(error, ==, 0);
	mutex_exit(&xzp->z_lock);
	zfs_unlinked_add(xzp, tx);

	if (zp->z_is_sa)
	error = sa_remove(zp->z_sa_hdl,
	SA_ZPL_XATTR(zfsvfs), tx);
	else
	error = sa_update(zp->z_sa_hdl,
	SA_ZPL_XATTR(zfsvfs), &null_xattr,
	sizeof (uint64_t), tx);
	ASSERT0(error);
	}
	/*
	* Add to the unlinked set because a new reference could be
	* taken concurrently resulting in a deferred destruction.
	*/
	zfs_unlinked_add(zp, tx);
	mutex_exit(&zp->z_lock);
	} else if (unlinked) {
	mutex_exit(&zp->z_lock);
	zfs_unlinked_add(zp, tx);
	}

	txtype = TX_REMOVE;
	if (flags & FIGNORECASE)
	txtype \|= TX_CI;
	zfs_log_remove(zilog, tx, txtype, dzp, name, obj, unlinked);

	dmu_tx_commit(tx);
	out:
	if (realnmp)
	pn_free(realnmp);

	zfs_dirent_unlock(dl);
	zfs_inode_update(dzp);
	zfs_inode_update(zp);

	if (delete_now)
	zrele(zp);
	else
	zfs_zrele_async(zp);

	if (xzp) {
	zfs_inode_update(xzp);
	zfs_zrele_async(xzp);
	}

	if (zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, 0);

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Create a new directory and insert it into dzp using the name
	* provided. Return a pointer to the inserted directory.
	*
	* IN: dzp - znode of directory to add subdir to.
	* dirname - name of new directory.
	* vap - attributes of new directory.
	* cr - credentials of caller.
	* flags - case flags.
	* vsecp - ACL to be set
	*
	* OUT: zpp - znode of created directory.
	*
	* RETURN: 0 if success
	* error code if failure
	*
	* Timestamps:
	* dzp - ctime\|mtime updated
	* zpp - ctime\|mtime\|atime updated
	*/
	/ARGSUSED/
	int
	zfs_mkdir(znode_t dzp, char dirname, vattr_t vap, znode_t *zpp,
	cred_t cr, int flags, vsecattr_t vsecp)
	{
	znode_t *zp;
	zfsvfs_t *zfsvfs = ZTOZSB(dzp);
	zilog_t *zilog;
	zfs_dirlock_t *dl;
	uint64_t txtype;
	dmu_tx_t *tx;
	int error;
	int zf = ZNEW;
	uid_t uid;
	gid_t gid = crgetgid(cr);
	zfs_acl_ids_t acl_ids;
	boolean_t fuid_dirtied;
	boolean_t waited = B_FALSE;

	ASSERT(S_ISDIR(vap->va_mode));

	/*
	* If we have an ephemeral id, ACL, or XVATTR then
	* make sure file system is at proper version
	*/

	uid = crgetuid(cr);
	if (zfsvfs->z_use_fuids == B_FALSE &&
	(vsecp \|\| IS_EPHEMERAL(uid) \|\| IS_EPHEMERAL(gid)))
	return (SET_ERROR(EINVAL));

	if (dirname == NULL)
	return (SET_ERROR(EINVAL));

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(dzp);
	zilog = zfsvfs->z_log;

	if (dzp->z_pflags & ZFS_XATTR) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	if (zfsvfs->z_utf8 && u8_validate(dirname,
	strlen(dirname), NULL, U8_VALIDATE_ENTIRE, &error) < 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EILSEQ));
	}
	if (flags & FIGNORECASE)
	zf \|= ZCILOOK;

	if (vap->va_mask & ATTR_XVATTR) {
	if ((error = secpolicy_xvattr((xvattr_t *)vap,
	crgetuid(cr), cr, vap->va_mode)) != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	}

	if ((error = zfs_acl_ids_create(dzp, 0, vap, cr,
	vsecp, &acl_ids)) != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	/*
	* First make sure the new directory doesn't exist.
	*
	* Existence is checked first to make sure we don't return
	* EACCES instead of EEXIST which can cause some applications
	* to fail.
	*/
	top:
	*zpp = NULL;

	if ((error = zfs_dirent_lock(&dl, dzp, dirname, &zp, zf,
	NULL, NULL))) {
	zfs_acl_ids_free(&acl_ids);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	if ((error = zfs_zaccess(dzp, ACE_ADD_SUBDIRECTORY, 0, B_FALSE, cr))) {
	zfs_acl_ids_free(&acl_ids);
	zfs_dirent_unlock(dl);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	if (zfs_acl_ids_overquota(zfsvfs, &acl_ids, zfs_inherit_projid(dzp))) {
	zfs_acl_ids_free(&acl_ids);
	zfs_dirent_unlock(dl);
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EDQUOT));
	}

	/*
	* Add a new entry to the directory.
	*/
	tx = dmu_tx_create(zfsvfs->z_os);
	dmu_tx_hold_zap(tx, dzp->z_id, TRUE, dirname);
	dmu_tx_hold_zap(tx, DMU_NEW_OBJECT, FALSE, NULL);
	fuid_dirtied = zfsvfs->z_fuid_dirty;
	if (fuid_dirtied)
	zfs_fuid_txhold(zfsvfs, tx);
	if (!zfsvfs->z_use_sa && acl_ids.z_aclp->z_acl_bytes > ZFS_ACE_SPACE) {
	dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0,
	acl_ids.z_aclp->z_acl_bytes);
	}

	dmu_tx_hold_sa_create(tx, acl_ids.z_aclp->z_acl_bytes +
	ZFS_SA_BASE_ATTR_SIZE);

	error = dmu_tx_assign(tx, (waited ? TXG_NOTHROTTLE : 0) \| TXG_NOWAIT);
	if (error) {
	zfs_dirent_unlock(dl);
	if (error == ERESTART) {
	waited = B_TRUE;
	dmu_tx_wait(tx);
	dmu_tx_abort(tx);
	goto top;
	}
	zfs_acl_ids_free(&acl_ids);
	dmu_tx_abort(tx);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Create new node.
	*/
	zfs_mknode(dzp, vap, tx, cr, 0, &zp, &acl_ids);

	/*
	* Now put new name in parent dir.
	*/
	error = zfs_link_create(dl, zp, tx, ZNEW);
	if (error != 0) {
	zfs_znode_delete(zp, tx);
	remove_inode_hash(ZTOI(zp));
	goto out;
	}

	if (fuid_dirtied)
	zfs_fuid_sync(zfsvfs, tx);

	*zpp = zp;

	txtype = zfs_log_create_txtype(Z_DIR, vsecp, vap);
	if (flags & FIGNORECASE)
	txtype \|= TX_CI;
	zfs_log_create(zilog, tx, txtype, dzp, zp, dirname, vsecp,
	acl_ids.z_fuidp, vap);

	out:
	zfs_acl_ids_free(&acl_ids);

	dmu_tx_commit(tx);

	zfs_dirent_unlock(dl);

	if (zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, 0);

	if (error != 0) {
	zrele(zp);
	} else {
	zfs_inode_update(dzp);
	zfs_inode_update(zp);
	}
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Remove a directory subdir entry. If the current working
	* directory is the same as the subdir to be removed, the
	* remove will fail.
	*
	* IN: dzp - znode of directory to remove from.
	* name - name of directory to be removed.
	* cwd - inode of current working directory.
	* cr - credentials of caller.
	* flags - case flags
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* dzp - ctime\|mtime updated
	*/
	/ARGSUSED/
	int
	zfs_rmdir(znode_t dzp, char name, znode_t cwd, cred_t cr,
	int flags)
	{
	znode_t *zp;
	zfsvfs_t *zfsvfs = ZTOZSB(dzp);
	zilog_t *zilog;
	zfs_dirlock_t *dl;
	dmu_tx_t *tx;
	int error;
	int zflg = ZEXISTS;
	boolean_t waited = B_FALSE;

	if (name == NULL)
	return (SET_ERROR(EINVAL));

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(dzp);
	zilog = zfsvfs->z_log;

	if (flags & FIGNORECASE)
	zflg \|= ZCILOOK;
	top:
	zp = NULL;

	/*
	* Attempt to lock directory; fail if entry doesn't exist.
	*/
	if ((error = zfs_dirent_lock(&dl, dzp, name, &zp, zflg,
	NULL, NULL))) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	if ((error = zfs_zaccess_delete(dzp, zp, cr))) {
	goto out;
	}

	if (!S_ISDIR(ZTOI(zp)->i_mode)) {
	error = SET_ERROR(ENOTDIR);
	goto out;
	}

	if (zp == cwd) {
	error = SET_ERROR(EINVAL);
	goto out;
	}

	/*
	* Grab a lock on the directory to make sure that no one is
	* trying to add (or lookup) entries while we are removing it.
	*/
	rw_enter(&zp->z_name_lock, RW_WRITER);

	/*
	* Grab a lock on the parent pointer to make sure we play well
	* with the treewalk and directory rename code.
	*/
	rw_enter(&zp->z_parent_lock, RW_WRITER);

	tx = dmu_tx_create(zfsvfs->z_os);
	dmu_tx_hold_zap(tx, dzp->z_id, FALSE, name);
	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_FALSE);
	dmu_tx_hold_zap(tx, zfsvfs->z_unlinkedobj, FALSE, NULL);
	zfs_sa_upgrade_txholds(tx, zp);
	zfs_sa_upgrade_txholds(tx, dzp);
	dmu_tx_mark_netfree(tx);
	error = dmu_tx_assign(tx, (waited ? TXG_NOTHROTTLE : 0) \| TXG_NOWAIT);
	if (error) {
	rw_exit(&zp->z_parent_lock);
	rw_exit(&zp->z_name_lock);
	zfs_dirent_unlock(dl);
	if (error == ERESTART) {
	waited = B_TRUE;
	dmu_tx_wait(tx);
	dmu_tx_abort(tx);
	zrele(zp);
	goto top;
	}
	dmu_tx_abort(tx);
	zrele(zp);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	error = zfs_link_destroy(dl, zp, tx, zflg, NULL);

	if (error == 0) {
	uint64_t txtype = TX_RMDIR;
	if (flags & FIGNORECASE)
	txtype \|= TX_CI;
	zfs_log_remove(zilog, tx, txtype, dzp, name, ZFS_NO_OBJECT,
	B_FALSE);
	}

	dmu_tx_commit(tx);

	rw_exit(&zp->z_parent_lock);
	rw_exit(&zp->z_name_lock);
	out:
	zfs_dirent_unlock(dl);

	zfs_inode_update(dzp);
	zfs_inode_update(zp);
	zrele(zp);

	if (zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, 0);

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Read directory entries from the given directory cursor position and emit
	* name and position for each entry.
	*
	* IN: ip - inode of directory to read.
	* ctx - directory entry context.
	* cr - credentials of caller.
	*
	* RETURN: 0 if success
	* error code if failure
	*
	* Timestamps:
	* ip - atime updated
	*
	* Note that the low 4 bits of the cookie returned by zap is always zero.
	* This allows us to use the low range for "special" directory entries:
	* We use 0 for '.', and 1 for '..'. If this is the root of the filesystem,
	* we use the offset 2 for the '.zfs' directory.
	*/
	/* ARGSUSED */
	int
	zfs_readdir(struct inode ip, zpl_dir_context_t ctx, cred_t *cr)
	{
	znode_t *zp = ITOZ(ip);
	zfsvfs_t *zfsvfs = ITOZSB(ip);
	objset_t *os;
	zap_cursor_t zc;
	zap_attribute_t zap;
	int error;
	uint8_t prefetch;
	uint8_t type;
	int done = 0;
	uint64_t parent;
	uint64_t offset; /* must be unsigned; checks for < 1 */

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	if ((error = sa_lookup(zp->z_sa_hdl, SA_ZPL_PARENT(zfsvfs),
	&parent, sizeof (parent))) != 0)
	goto out;

	/*
	* Quit if directory has been removed (posix)
	*/
	if (zp->z_unlinked)
	goto out;

	error = 0;
	os = zfsvfs->z_os;
	offset = ctx->pos;
	prefetch = zp->z_zn_prefetch;

	/*
	* Initialize the iterator cursor.
	*/
	if (offset <= 3) {
	/*
	* Start iteration from the beginning of the directory.
	*/
	zap_cursor_init(&zc, os, zp->z_id);
	} else {
	/*
	* The offset is a serialized cursor.
	*/
	zap_cursor_init_serialized(&zc, os, zp->z_id, offset);
	}

	/*
	* Transform to file-system independent format
	*/
	while (!done) {
	uint64_t objnum;
	/*
	* Special case `.', `..', and `.zfs'.
	*/
	if (offset == 0) {
	(void) strcpy(zap.za_name, ".");
	zap.za_normalization_conflict = 0;
	objnum = zp->z_id;
	type = DT_DIR;
	} else if (offset == 1) {
	(void) strcpy(zap.za_name, "..");
	zap.za_normalization_conflict = 0;
	objnum = parent;
	type = DT_DIR;
	} else if (offset == 2 && zfs_show_ctldir(zp)) {
	(void) strcpy(zap.za_name, ZFS_CTLDIR_NAME);
	zap.za_normalization_conflict = 0;
	objnum = ZFSCTL_INO_ROOT;
	type = DT_DIR;
	} else {
	/*
	* Grab next entry.
	*/
	if ((error = zap_cursor_retrieve(&zc, &zap))) {
	if (error == ENOENT)
	break;
	else
	goto update;
	}

	/*
	* Allow multiple entries provided the first entry is
	* the object id. Non-zpl consumers may safely make
	* use of the additional space.
	*
	* XXX: This should be a feature flag for compatibility
	*/
	if (zap.za_integer_length != 8 \|\|
	zap.za_num_integers == 0) {
	cmn_err(CE_WARN, "zap_readdir: bad directory "
	"entry, obj = %lld, offset = %lld, "
	"length = %d, num = %lld\n",
	(u_longlong_t)zp->z_id,
	(u_longlong_t)offset,
	zap.za_integer_length,
	(u_longlong_t)zap.za_num_integers);
	error = SET_ERROR(ENXIO);
	goto update;
	}

	objnum = ZFS_DIRENT_OBJ(zap.za_first_integer);
	type = ZFS_DIRENT_TYPE(zap.za_first_integer);
	}

	done = !zpl_dir_emit(ctx, zap.za_name, strlen(zap.za_name),
	objnum, type);
	if (done)
	break;

	/* Prefetch znode */
	if (prefetch) {
	dmu_prefetch(os, objnum, 0, 0, 0,
	ZIO_PRIORITY_SYNC_READ);
	}

	/*
	* Move to the next entry, fill in the previous offset.
	*/
	if (offset > 2 \|\| (offset == 2 && !zfs_show_ctldir(zp))) {
	zap_cursor_advance(&zc);
	offset = zap_cursor_serialize(&zc);
	} else {
	offset += 1;
	}
	ctx->pos = offset;
	}
	zp->z_zn_prefetch = B_FALSE; /* a lookup will re-enable pre-fetching */

	update:
	zap_cursor_fini(&zc);
	if (error == ENOENT)
	error = 0;
	out:
	ZFS_EXIT(zfsvfs);

	return (error);
	}

	ulong_t zfs_fsync_sync_cnt = 4;

	int
	zfs_fsync(znode_t zp, int syncflag, cred_t cr)
	{
	zfsvfs_t *zfsvfs = ZTOZSB(zp);

	(void) tsd_set(zfs_fsyncer_key, (void *)zfs_fsync_sync_cnt);

	if (zfsvfs->z_os->os_sync != ZFS_SYNC_DISABLED) {
	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);
	zil_commit(zfsvfs->z_log, zp->z_id);
	ZFS_EXIT(zfsvfs);
	}
	tsd_set(zfs_fsyncer_key, NULL);

	return (0);
	}

	/*
	* Get the basic file attributes and place them in the provided kstat
	* structure. The inode is assumed to be the authoritative source
	* for most of the attributes. However, the znode currently has the
	* authoritative atime, blksize, and block count.
	*
	* IN: ip - inode of file.
	*
	* OUT: sp - kstat values.
	*
	* RETURN: 0 (always succeeds)
	*/
	/* ARGSUSED */
	int
	zfs_getattr_fast(struct inode ip, struct kstat sp)
	{
	znode_t *zp = ITOZ(ip);
	zfsvfs_t *zfsvfs = ITOZSB(ip);
	uint32_t blksize;
	u_longlong_t nblocks;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	mutex_enter(&zp->z_lock);

	generic_fillattr(ip, sp);
	/*
	* +1 link count for root inode with visible '.zfs' directory.
	*/
	if ((zp->z_id == zfsvfs->z_root) && zfs_show_ctldir(zp))
	if (sp->nlink < ZFS_LINK_MAX)
	sp->nlink++;

	sa_object_size(zp->z_sa_hdl, &blksize, &nblocks);
	sp->blksize = blksize;
	sp->blocks = nblocks;

	if (unlikely(zp->z_blksz == 0)) {
	/*
	* Block size hasn't been set; suggest maximal I/O transfers.
	*/
	sp->blksize = zfsvfs->z_max_blksz;
	}

	mutex_exit(&zp->z_lock);

	/*
	* Required to prevent NFS client from detecting different inode
	* numbers of snapshot root dentry before and after snapshot mount.
	*/
	if (zfsvfs->z_issnap) {
	if (ip->i_sb->s_root->d_inode == ip)
	sp->ino = ZFSCTL_INO_SNAPDIRS -
	dmu_objset_id(zfsvfs->z_os);
	}

	ZFS_EXIT(zfsvfs);

	return (0);
	}

	/*
	* For the operation of changing file's user/group/project, we need to
	* handle not only the main object that is assigned to the file directly,
	* but also the ones that are used by the file via hidden xattr directory.
	*
	* Because the xattr directory may contains many EA entries, as to it may
	* be impossible to change all of them via the transaction of changing the
	* main object's user/group/project attributes. Then we have to change them
	* via other multiple independent transactions one by one. It may be not good
	* solution, but we have no better idea yet.
	*/
	static int
	zfs_setattr_dir(znode_t *dzp)
	{
	struct inode *dxip = ZTOI(dzp);
	struct inode *xip = NULL;
	zfsvfs_t *zfsvfs = ZTOZSB(dzp);
	objset_t *os = zfsvfs->z_os;
	zap_cursor_t zc;
	zap_attribute_t zap;
	zfs_dirlock_t *dl;
	- znode_t *zp;
	+ znode_t *zp = NULL;
	dmu_tx_t *tx = NULL;
	uint64_t uid, gid;
	sa_bulk_attr_t bulk[4];
	int count;
	int err;

	zap_cursor_init(&zc, os, dzp->z_id);
	while ((err = zap_cursor_retrieve(&zc, &zap)) == 0) {
	count = 0;
	if (zap.za_integer_length != 8 \|\| zap.za_num_integers != 1) {
	err = ENXIO;
	break;
	}

	err = zfs_dirent_lock(&dl, dzp, (char *)zap.za_name, &zp,
	ZEXISTS, NULL, NULL);
	if (err == ENOENT)
	goto next;
	if (err)
	break;

	xip = ZTOI(zp);
	if (KUID_TO_SUID(xip->i_uid) == KUID_TO_SUID(dxip->i_uid) &&
	KGID_TO_SGID(xip->i_gid) == KGID_TO_SGID(dxip->i_gid) &&
	zp->z_projid == dzp->z_projid)
	goto next;

	tx = dmu_tx_create(os);
	if (!(zp->z_pflags & ZFS_PROJID))
	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_TRUE);
	else
	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_FALSE);

	err = dmu_tx_assign(tx, TXG_WAIT);
	if (err)
	break;

	mutex_enter(&dzp->z_lock);

	if (KUID_TO_SUID(xip->i_uid) != KUID_TO_SUID(dxip->i_uid)) {
	xip->i_uid = dxip->i_uid;
	uid = zfs_uid_read(dxip);
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_UID(zfsvfs), NULL,
	&uid, sizeof (uid));
	}

	if (KGID_TO_SGID(xip->i_gid) != KGID_TO_SGID(dxip->i_gid)) {
	xip->i_gid = dxip->i_gid;
	gid = zfs_gid_read(dxip);
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_GID(zfsvfs), NULL,
	&gid, sizeof (gid));
	}

	if (zp->z_projid != dzp->z_projid) {
	if (!(zp->z_pflags & ZFS_PROJID)) {
	zp->z_pflags \|= ZFS_PROJID;
	SA_ADD_BULK_ATTR(bulk, count,
	SA_ZPL_FLAGS(zfsvfs), NULL, &zp->z_pflags,
	sizeof (zp->z_pflags));
	}

	zp->z_projid = dzp->z_projid;
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_PROJID(zfsvfs),
	NULL, &zp->z_projid, sizeof (zp->z_projid));
	}

	mutex_exit(&dzp->z_lock);

	if (likely(count > 0)) {
	err = sa_bulk_update(zp->z_sa_hdl, bulk, count, tx);
	dmu_tx_commit(tx);
	} else {
	dmu_tx_abort(tx);
	}
	tx = NULL;
	if (err != 0 && err != ENOENT)
	break;

	next:
	if (zp) {
	zrele(zp);
	zp = NULL;
	zfs_dirent_unlock(dl);
	}
	zap_cursor_advance(&zc);
	}

	if (tx)
	dmu_tx_abort(tx);
	if (zp) {
	zrele(zp);
	zfs_dirent_unlock(dl);
	}
	zap_cursor_fini(&zc);

	return (err == ENOENT ? 0 : err);
	}

	/*
	* Set the file attributes to the values contained in the
	* vattr structure.
	*
	* IN: zp - znode of file to be modified.
	* vap - new attribute values.
	* If ATTR_XVATTR set, then optional attrs are being set
	* flags - ATTR_UTIME set if non-default time values provided.
	* - ATTR_NOACLCHECK (CIFS context only).
	* cr - credentials of caller.
	*
	* RETURN: 0 if success
	* error code if failure
	*
	* Timestamps:
	* ip - ctime updated, mtime updated if size changed.
	*/
	/* ARGSUSED */
	int
	zfs_setattr(znode_t zp, vattr_t vap, int flags, cred_t *cr)
	{
	struct inode *ip;
	zfsvfs_t *zfsvfs = ZTOZSB(zp);
	objset_t *os = zfsvfs->z_os;
	zilog_t *zilog;
	dmu_tx_t *tx;
	vattr_t oldva;
	xvattr_t *tmpxvattr;
	uint_t mask = vap->va_mask;
	uint_t saved_mask = 0;
	int trim_mask = 0;
	uint64_t new_mode;
	uint64_t new_kuid = 0, new_kgid = 0, new_uid, new_gid;
	uint64_t xattr_obj;
	uint64_t mtime[2], ctime[2], atime[2];
	uint64_t projid = ZFS_INVALID_PROJID;
	znode_t *attrzp;
	int need_policy = FALSE;
	int err, err2 = 0;
	zfs_fuid_info_t *fuidp = NULL;
	xvattr_t xvap = (xvattr_t )vap; /* vap may be an xvattr_t * */
	xoptattr_t *xoap;
	zfs_acl_t *aclp;
	boolean_t skipaclchk = (flags & ATTR_NOACLCHECK) ? B_TRUE : B_FALSE;
	boolean_t fuid_dirtied = B_FALSE;
	boolean_t handle_eadir = B_FALSE;
	sa_bulk_attr_t bulk, xattr_bulk;
	int count = 0, xattr_count = 0, bulks = 8;

	if (mask == 0)
	return (0);

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);
	ip = ZTOI(zp);

	/*
	* If this is a xvattr_t, then get a pointer to the structure of
	* optional attributes. If this is NULL, then we have a vattr_t.
	*/
	xoap = xva_getxoptattr(xvap);
	if (xoap != NULL && (mask & ATTR_XVATTR)) {
	if (XVA_ISSET_REQ(xvap, XAT_PROJID)) {
	if (!dmu_objset_projectquota_enabled(os) \|\|
	(!S_ISREG(ip->i_mode) && !S_ISDIR(ip->i_mode))) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(ENOTSUP));
	}

	projid = xoap->xoa_projid;
	if (unlikely(projid == ZFS_INVALID_PROJID)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	if (projid == zp->z_projid && zp->z_pflags & ZFS_PROJID)
	projid = ZFS_INVALID_PROJID;
	else
	need_policy = TRUE;
	}

	if (XVA_ISSET_REQ(xvap, XAT_PROJINHERIT) &&
	(xoap->xoa_projinherit !=
	((zp->z_pflags & ZFS_PROJINHERIT) != 0)) &&
	(!dmu_objset_projectquota_enabled(os) \|\|
	(!S_ISREG(ip->i_mode) && !S_ISDIR(ip->i_mode)))) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(ENOTSUP));
	}
	}

	zilog = zfsvfs->z_log;

	/*
	* Make sure that if we have ephemeral uid/gid or xvattr specified
	* that file system is at proper version level
	*/

	if (zfsvfs->z_use_fuids == B_FALSE &&
	(((mask & ATTR_UID) && IS_EPHEMERAL(vap->va_uid)) \|\|
	((mask & ATTR_GID) && IS_EPHEMERAL(vap->va_gid)) \|\|
	(mask & ATTR_XVATTR))) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	if (mask & ATTR_SIZE && S_ISDIR(ip->i_mode)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EISDIR));
	}

	if (mask & ATTR_SIZE && !S_ISREG(ip->i_mode) && !S_ISFIFO(ip->i_mode)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	tmpxvattr = kmem_alloc(sizeof (xvattr_t), KM_SLEEP);
	xva_init(tmpxvattr);

	bulk = kmem_alloc(sizeof (sa_bulk_attr_t) * bulks, KM_SLEEP);
	xattr_bulk = kmem_alloc(sizeof (sa_bulk_attr_t) * bulks, KM_SLEEP);

	/*
	* Immutable files can only alter immutable bit and atime
	*/
	if ((zp->z_pflags & ZFS_IMMUTABLE) &&
	((mask & (ATTR_SIZE\|ATTR_UID\|ATTR_GID\|ATTR_MTIME\|ATTR_MODE)) \|\|
	((mask & ATTR_XVATTR) && XVA_ISSET_REQ(xvap, XAT_CREATETIME)))) {
	err = SET_ERROR(EPERM);
	goto out3;
	}

	if ((mask & ATTR_SIZE) && (zp->z_pflags & ZFS_READONLY)) {
	err = SET_ERROR(EPERM);
	goto out3;
	}

	/*
	* Verify timestamps doesn't overflow 32 bits.
	* ZFS can handle large timestamps, but 32bit syscalls can't
	* handle times greater than 2039. This check should be removed
	* once large timestamps are fully supported.
	*/
	if (mask & (ATTR_ATIME \| ATTR_MTIME)) {
	if (((mask & ATTR_ATIME) &&
	TIMESPEC_OVERFLOW(&vap->va_atime)) \|\|
	((mask & ATTR_MTIME) &&
	TIMESPEC_OVERFLOW(&vap->va_mtime))) {
	err = SET_ERROR(EOVERFLOW);
	goto out3;
	}
	}

	top:
	attrzp = NULL;
	aclp = NULL;

	/* Can this be moved to before the top label? */
	if (zfs_is_readonly(zfsvfs)) {
	err = SET_ERROR(EROFS);
	goto out3;
	}

	/*
	* First validate permissions
	*/

	if (mask & ATTR_SIZE) {
	err = zfs_zaccess(zp, ACE_WRITE_DATA, 0, skipaclchk, cr);
	if (err)
	goto out3;

	/*
	* XXX - Note, we are not providing any open
	* mode flags here (like FNDELAY), so we may
	* block if there are locks present... this
	* should be addressed in openat().
	*/
	/* XXX - would it be OK to generate a log record here? */
	err = zfs_freesp(zp, vap->va_size, 0, 0, FALSE);
	if (err)
	goto out3;
	}

	if (mask & (ATTR_ATIME\|ATTR_MTIME) \|\|
	((mask & ATTR_XVATTR) && (XVA_ISSET_REQ(xvap, XAT_HIDDEN) \|\|
	XVA_ISSET_REQ(xvap, XAT_READONLY) \|\|
	XVA_ISSET_REQ(xvap, XAT_ARCHIVE) \|\|
	XVA_ISSET_REQ(xvap, XAT_OFFLINE) \|\|
	XVA_ISSET_REQ(xvap, XAT_SPARSE) \|\|
	XVA_ISSET_REQ(xvap, XAT_CREATETIME) \|\|
	XVA_ISSET_REQ(xvap, XAT_SYSTEM)))) {
	need_policy = zfs_zaccess(zp, ACE_WRITE_ATTRIBUTES, 0,
	skipaclchk, cr);
	}

	if (mask & (ATTR_UID\|ATTR_GID)) {
	int idmask = (mask & (ATTR_UID\|ATTR_GID));
	int take_owner;
	int take_group;

	/*
	* NOTE: even if a new mode is being set,
	* we may clear S_ISUID/S_ISGID bits.
	*/

	if (!(mask & ATTR_MODE))
	vap->va_mode = zp->z_mode;

	/*
	* Take ownership or chgrp to group we are a member of
	*/

	take_owner = (mask & ATTR_UID) && (vap->va_uid == crgetuid(cr));
	take_group = (mask & ATTR_GID) &&
	zfs_groupmember(zfsvfs, vap->va_gid, cr);

	/*
	* If both ATTR_UID and ATTR_GID are set then take_owner and
	* take_group must both be set in order to allow taking
	* ownership.
	*
	* Otherwise, send the check through secpolicy_vnode_setattr()
	*
	*/

	if (((idmask == (ATTR_UID\|ATTR_GID)) &&
	take_owner && take_group) \|\|
	((idmask == ATTR_UID) && take_owner) \|\|
	((idmask == ATTR_GID) && take_group)) {
	if (zfs_zaccess(zp, ACE_WRITE_OWNER, 0,
	skipaclchk, cr) == 0) {
	/*
	* Remove setuid/setgid for non-privileged users
	*/
	(void) secpolicy_setid_clear(vap, cr);
	trim_mask = (mask & (ATTR_UID\|ATTR_GID));
	} else {
	need_policy = TRUE;
	}
	} else {
	need_policy = TRUE;
	}
	}

	mutex_enter(&zp->z_lock);
	oldva.va_mode = zp->z_mode;
	zfs_fuid_map_ids(zp, cr, &oldva.va_uid, &oldva.va_gid);
	if (mask & ATTR_XVATTR) {
	/*
	* Update xvattr mask to include only those attributes
	* that are actually changing.
	*
	* the bits will be restored prior to actually setting
	* the attributes so the caller thinks they were set.
	*/
	if (XVA_ISSET_REQ(xvap, XAT_APPENDONLY)) {
	if (xoap->xoa_appendonly !=
	((zp->z_pflags & ZFS_APPENDONLY) != 0)) {
	need_policy = TRUE;
	} else {
	XVA_CLR_REQ(xvap, XAT_APPENDONLY);
	XVA_SET_REQ(tmpxvattr, XAT_APPENDONLY);
	}
	}

	if (XVA_ISSET_REQ(xvap, XAT_PROJINHERIT)) {
	if (xoap->xoa_projinherit !=
	((zp->z_pflags & ZFS_PROJINHERIT) != 0)) {
	need_policy = TRUE;
	} else {
	XVA_CLR_REQ(xvap, XAT_PROJINHERIT);
	XVA_SET_REQ(tmpxvattr, XAT_PROJINHERIT);
	}
	}

	if (XVA_ISSET_REQ(xvap, XAT_NOUNLINK)) {
	if (xoap->xoa_nounlink !=
	((zp->z_pflags & ZFS_NOUNLINK) != 0)) {
	need_policy = TRUE;
	} else {
	XVA_CLR_REQ(xvap, XAT_NOUNLINK);
	XVA_SET_REQ(tmpxvattr, XAT_NOUNLINK);
	}
	}

	if (XVA_ISSET_REQ(xvap, XAT_IMMUTABLE)) {
	if (xoap->xoa_immutable !=
	((zp->z_pflags & ZFS_IMMUTABLE) != 0)) {
	need_policy = TRUE;
	} else {
	XVA_CLR_REQ(xvap, XAT_IMMUTABLE);
	XVA_SET_REQ(tmpxvattr, XAT_IMMUTABLE);
	}
	}

	if (XVA_ISSET_REQ(xvap, XAT_NODUMP)) {
	if (xoap->xoa_nodump !=
	((zp->z_pflags & ZFS_NODUMP) != 0)) {
	need_policy = TRUE;
	} else {
	XVA_CLR_REQ(xvap, XAT_NODUMP);
	XVA_SET_REQ(tmpxvattr, XAT_NODUMP);
	}
	}

	if (XVA_ISSET_REQ(xvap, XAT_AV_MODIFIED)) {
	if (xoap->xoa_av_modified !=
	((zp->z_pflags & ZFS_AV_MODIFIED) != 0)) {
	need_policy = TRUE;
	} else {
	XVA_CLR_REQ(xvap, XAT_AV_MODIFIED);
	XVA_SET_REQ(tmpxvattr, XAT_AV_MODIFIED);
	}
	}

	if (XVA_ISSET_REQ(xvap, XAT_AV_QUARANTINED)) {
	if ((!S_ISREG(ip->i_mode) &&
	xoap->xoa_av_quarantined) \|\|
	xoap->xoa_av_quarantined !=
	((zp->z_pflags & ZFS_AV_QUARANTINED) != 0)) {
	need_policy = TRUE;
	} else {
	XVA_CLR_REQ(xvap, XAT_AV_QUARANTINED);
	XVA_SET_REQ(tmpxvattr, XAT_AV_QUARANTINED);
	}
	}

	if (XVA_ISSET_REQ(xvap, XAT_REPARSE)) {
	mutex_exit(&zp->z_lock);
	err = SET_ERROR(EPERM);
	goto out3;
	}

	if (need_policy == FALSE &&
	(XVA_ISSET_REQ(xvap, XAT_AV_SCANSTAMP) \|\|
	XVA_ISSET_REQ(xvap, XAT_OPAQUE))) {
	need_policy = TRUE;
	}
	}

	mutex_exit(&zp->z_lock);

	if (mask & ATTR_MODE) {
	if (zfs_zaccess(zp, ACE_WRITE_ACL, 0, skipaclchk, cr) == 0) {
	err = secpolicy_setid_setsticky_clear(ip, vap,
	&oldva, cr);
	if (err)
	goto out3;

	trim_mask \|= ATTR_MODE;
	} else {
	need_policy = TRUE;
	}
	}

	if (need_policy) {
	/*
	* If trim_mask is set then take ownership
	* has been granted or write_acl is present and user
	* has the ability to modify mode. In that case remove
	* UID\|GID and or MODE from mask so that
	* secpolicy_vnode_setattr() doesn't revoke it.
	*/

	if (trim_mask) {
	saved_mask = vap->va_mask;
	vap->va_mask &= ~trim_mask;
	}
	err = secpolicy_vnode_setattr(cr, ip, vap, &oldva, flags,
	(int ()(void , int, cred_t *))zfs_zaccess_unix, zp);
	if (err)
	goto out3;

	if (trim_mask)
	vap->va_mask \|= saved_mask;
	}

	/*
	* secpolicy_vnode_setattr, or take ownership may have
	* changed va_mask
	*/
	mask = vap->va_mask;

	if ((mask & (ATTR_UID \| ATTR_GID)) \|\| projid != ZFS_INVALID_PROJID) {
	handle_eadir = B_TRUE;
	err = sa_lookup(zp->z_sa_hdl, SA_ZPL_XATTR(zfsvfs),
	&xattr_obj, sizeof (xattr_obj));

	if (err == 0 && xattr_obj) {
	err = zfs_zget(ZTOZSB(zp), xattr_obj, &attrzp);
	if (err)
	goto out2;
	}
	if (mask & ATTR_UID) {
	new_kuid = zfs_fuid_create(zfsvfs,
	(uint64_t)vap->va_uid, cr, ZFS_OWNER, &fuidp);
	if (new_kuid != KUID_TO_SUID(ZTOI(zp)->i_uid) &&
	zfs_id_overquota(zfsvfs, DMU_USERUSED_OBJECT,
	new_kuid)) {
	if (attrzp)
	zrele(attrzp);
	err = SET_ERROR(EDQUOT);
	goto out2;
	}
	}

	if (mask & ATTR_GID) {
	new_kgid = zfs_fuid_create(zfsvfs,
	(uint64_t)vap->va_gid, cr, ZFS_GROUP, &fuidp);
	if (new_kgid != KGID_TO_SGID(ZTOI(zp)->i_gid) &&
	zfs_id_overquota(zfsvfs, DMU_GROUPUSED_OBJECT,
	new_kgid)) {
	if (attrzp)
	zrele(attrzp);
	err = SET_ERROR(EDQUOT);
	goto out2;
	}
	}

	if (projid != ZFS_INVALID_PROJID &&
	zfs_id_overquota(zfsvfs, DMU_PROJECTUSED_OBJECT, projid)) {
	if (attrzp)
	zrele(attrzp);
	err = EDQUOT;
	goto out2;
	}
	}
	tx = dmu_tx_create(os);

	if (mask & ATTR_MODE) {
	uint64_t pmode = zp->z_mode;
	uint64_t acl_obj;
	new_mode = (pmode & S_IFMT) \| (vap->va_mode & ~S_IFMT);

	if (ZTOZSB(zp)->z_acl_mode == ZFS_ACL_RESTRICTED &&
	!(zp->z_pflags & ZFS_ACL_TRIVIAL)) {
	err = EPERM;
	goto out;
	}

	if ((err = zfs_acl_chmod_setattr(zp, &aclp, new_mode)))
	goto out;

	mutex_enter(&zp->z_lock);
	if (!zp->z_is_sa && ((acl_obj = zfs_external_acl(zp)) != 0)) {
	/*
	* Are we upgrading ACL from old V0 format
	* to V1 format?
	*/
	if (zfsvfs->z_version >= ZPL_VERSION_FUID &&
	zfs_znode_acl_version(zp) ==
	ZFS_ACL_VERSION_INITIAL) {
	dmu_tx_hold_free(tx, acl_obj, 0,
	DMU_OBJECT_END);
	dmu_tx_hold_write(tx, DMU_NEW_OBJECT,
	0, aclp->z_acl_bytes);
	} else {
	dmu_tx_hold_write(tx, acl_obj, 0,
	aclp->z_acl_bytes);
	}
	} else if (!zp->z_is_sa && aclp->z_acl_bytes > ZFS_ACE_SPACE) {
	dmu_tx_hold_write(tx, DMU_NEW_OBJECT,
	0, aclp->z_acl_bytes);
	}
	mutex_exit(&zp->z_lock);
	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_TRUE);
	} else {
	if (((mask & ATTR_XVATTR) &&
	XVA_ISSET_REQ(xvap, XAT_AV_SCANSTAMP)) \|\|
	(projid != ZFS_INVALID_PROJID &&
	!(zp->z_pflags & ZFS_PROJID)))
	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_TRUE);
	else
	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_FALSE);
	}

	if (attrzp) {
	dmu_tx_hold_sa(tx, attrzp->z_sa_hdl, B_FALSE);
	}

	fuid_dirtied = zfsvfs->z_fuid_dirty;
	if (fuid_dirtied)
	zfs_fuid_txhold(zfsvfs, tx);

	zfs_sa_upgrade_txholds(tx, zp);

	err = dmu_tx_assign(tx, TXG_WAIT);
	if (err)
	goto out;

	count = 0;
	/*
	* Set each attribute requested.
	* We group settings according to the locks they need to acquire.
	*
	* Note: you cannot set ctime directly, although it will be
	* updated as a side-effect of calling this function.
	*/

	if (projid != ZFS_INVALID_PROJID && !(zp->z_pflags & ZFS_PROJID)) {
	/*
	* For the existed object that is upgraded from old system,
	* its on-disk layout has no slot for the project ID attribute.
	* But quota accounting logic needs to access related slots by
	* offset directly. So we need to adjust old objects' layout
	* to make the project ID to some unified and fixed offset.
	*/
	if (attrzp)
	err = sa_add_projid(attrzp->z_sa_hdl, tx, projid);
	if (err == 0)
	err = sa_add_projid(zp->z_sa_hdl, tx, projid);

	if (unlikely(err == EEXIST))
	err = 0;
	else if (err != 0)
	goto out;
	else
	projid = ZFS_INVALID_PROJID;
	}

	if (mask & (ATTR_UID\|ATTR_GID\|ATTR_MODE))
	mutex_enter(&zp->z_acl_lock);
	mutex_enter(&zp->z_lock);

	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_FLAGS(zfsvfs), NULL,
	&zp->z_pflags, sizeof (zp->z_pflags));

	if (attrzp) {
	if (mask & (ATTR_UID\|ATTR_GID\|ATTR_MODE))
	mutex_enter(&attrzp->z_acl_lock);
	mutex_enter(&attrzp->z_lock);
	SA_ADD_BULK_ATTR(xattr_bulk, xattr_count,
	SA_ZPL_FLAGS(zfsvfs), NULL, &attrzp->z_pflags,
	sizeof (attrzp->z_pflags));
	if (projid != ZFS_INVALID_PROJID) {
	attrzp->z_projid = projid;
	SA_ADD_BULK_ATTR(xattr_bulk, xattr_count,
	SA_ZPL_PROJID(zfsvfs), NULL, &attrzp->z_projid,
	sizeof (attrzp->z_projid));
	}
	}

	if (mask & (ATTR_UID\|ATTR_GID)) {

	if (mask & ATTR_UID) {
	ZTOI(zp)->i_uid = SUID_TO_KUID(new_kuid);
	new_uid = zfs_uid_read(ZTOI(zp));
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_UID(zfsvfs), NULL,
	&new_uid, sizeof (new_uid));
	if (attrzp) {
	SA_ADD_BULK_ATTR(xattr_bulk, xattr_count,
	SA_ZPL_UID(zfsvfs), NULL, &new_uid,
	sizeof (new_uid));
	ZTOI(attrzp)->i_uid = SUID_TO_KUID(new_uid);
	}
	}

	if (mask & ATTR_GID) {
	ZTOI(zp)->i_gid = SGID_TO_KGID(new_kgid);
	new_gid = zfs_gid_read(ZTOI(zp));
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_GID(zfsvfs),
	NULL, &new_gid, sizeof (new_gid));
	if (attrzp) {
	SA_ADD_BULK_ATTR(xattr_bulk, xattr_count,
	SA_ZPL_GID(zfsvfs), NULL, &new_gid,
	sizeof (new_gid));
	ZTOI(attrzp)->i_gid = SGID_TO_KGID(new_kgid);
	}
	}
	if (!(mask & ATTR_MODE)) {
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_MODE(zfsvfs),
	NULL, &new_mode, sizeof (new_mode));
	new_mode = zp->z_mode;
	}
	err = zfs_acl_chown_setattr(zp);
	ASSERT(err == 0);
	if (attrzp) {
	err = zfs_acl_chown_setattr(attrzp);
	ASSERT(err == 0);
	}
	}

	if (mask & ATTR_MODE) {
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_MODE(zfsvfs), NULL,
	&new_mode, sizeof (new_mode));
	zp->z_mode = ZTOI(zp)->i_mode = new_mode;
	ASSERT3P(aclp, !=, NULL);
	err = zfs_aclset_common(zp, aclp, cr, tx);
	ASSERT0(err);
	if (zp->z_acl_cached)
	zfs_acl_free(zp->z_acl_cached);
	zp->z_acl_cached = aclp;
	aclp = NULL;
	}

	if ((mask & ATTR_ATIME) \|\| zp->z_atime_dirty) {
	zp->z_atime_dirty = B_FALSE;
	ZFS_TIME_ENCODE(&ip->i_atime, atime);
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_ATIME(zfsvfs), NULL,
	&atime, sizeof (atime));
	}

	if (mask & (ATTR_MTIME \| ATTR_SIZE)) {
	ZFS_TIME_ENCODE(&vap->va_mtime, mtime);
	ZTOI(zp)->i_mtime = zpl_inode_timestamp_truncate(
	vap->va_mtime, ZTOI(zp));

	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_MTIME(zfsvfs), NULL,
	mtime, sizeof (mtime));
	}

	if (mask & (ATTR_CTIME \| ATTR_SIZE)) {
	ZFS_TIME_ENCODE(&vap->va_ctime, ctime);
	ZTOI(zp)->i_ctime = zpl_inode_timestamp_truncate(vap->va_ctime,
	ZTOI(zp));
	SA_ADD_BULK_ATTR(bulk, count, SA_ZPL_CTIME(zfsvfs), NULL,
	ctime, sizeof (ctime));
	}

	if (projid != ZFS_INVALID_PROJID) {
	zp->z_projid = projid;
	SA_ADD_BULK_ATTR(bulk, count,
	SA_ZPL_PROJID(zfsvfs), NULL, &zp->z_projid,
	sizeof (zp->z_projid));
	}

	if (attrzp && mask) {
	SA_ADD_BULK_ATTR(xattr_bulk, xattr_count,
	SA_ZPL_CTIME(zfsvfs), NULL, &ctime,
	sizeof (ctime));
	}

	/*
	* Do this after setting timestamps to prevent timestamp
	* update from toggling bit
	*/

	if (xoap && (mask & ATTR_XVATTR)) {

	/*
	* restore trimmed off masks
	* so that return masks can be set for caller.
	*/

	if (XVA_ISSET_REQ(tmpxvattr, XAT_APPENDONLY)) {
	XVA_SET_REQ(xvap, XAT_APPENDONLY);
	}
	if (XVA_ISSET_REQ(tmpxvattr, XAT_NOUNLINK)) {
	XVA_SET_REQ(xvap, XAT_NOUNLINK);
	}
	if (XVA_ISSET_REQ(tmpxvattr, XAT_IMMUTABLE)) {
	XVA_SET_REQ(xvap, XAT_IMMUTABLE);
	}
	if (XVA_ISSET_REQ(tmpxvattr, XAT_NODUMP)) {
	XVA_SET_REQ(xvap, XAT_NODUMP);
	}
	if (XVA_ISSET_REQ(tmpxvattr, XAT_AV_MODIFIED)) {
	XVA_SET_REQ(xvap, XAT_AV_MODIFIED);
	}
	if (XVA_ISSET_REQ(tmpxvattr, XAT_AV_QUARANTINED)) {
	XVA_SET_REQ(xvap, XAT_AV_QUARANTINED);
	}
	if (XVA_ISSET_REQ(tmpxvattr, XAT_PROJINHERIT)) {
	XVA_SET_REQ(xvap, XAT_PROJINHERIT);
	}

	if (XVA_ISSET_REQ(xvap, XAT_AV_SCANSTAMP))
	ASSERT(S_ISREG(ip->i_mode));

	zfs_xvattr_set(zp, xvap, tx);
	}

	if (fuid_dirtied)
	zfs_fuid_sync(zfsvfs, tx);

	if (mask != 0)
	zfs_log_setattr(zilog, tx, TX_SETATTR, zp, vap, mask, fuidp);

	mutex_exit(&zp->z_lock);
	if (mask & (ATTR_UID\|ATTR_GID\|ATTR_MODE))
	mutex_exit(&zp->z_acl_lock);

	if (attrzp) {
	if (mask & (ATTR_UID\|ATTR_GID\|ATTR_MODE))
	mutex_exit(&attrzp->z_acl_lock);
	mutex_exit(&attrzp->z_lock);
	}
	out:
	if (err == 0 && xattr_count > 0) {
	err2 = sa_bulk_update(attrzp->z_sa_hdl, xattr_bulk,
	xattr_count, tx);
	ASSERT(err2 == 0);
	}

	if (aclp)
	zfs_acl_free(aclp);

	if (fuidp) {
	zfs_fuid_info_free(fuidp);
	fuidp = NULL;
	}

	if (err) {
	dmu_tx_abort(tx);
	if (attrzp)
	zrele(attrzp);
	if (err == ERESTART)
	goto top;
	} else {
	if (count > 0)
	err2 = sa_bulk_update(zp->z_sa_hdl, bulk, count, tx);
	dmu_tx_commit(tx);
	if (attrzp) {
	if (err2 == 0 && handle_eadir)
	err2 = zfs_setattr_dir(attrzp);
	zrele(attrzp);
	}
	zfs_inode_update(zp);
	}

	out2:
	if (os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, 0);

	out3:
	kmem_free(xattr_bulk, sizeof (sa_bulk_attr_t) * bulks);
	kmem_free(bulk, sizeof (sa_bulk_attr_t) * bulks);
	kmem_free(tmpxvattr, sizeof (xvattr_t));
	ZFS_EXIT(zfsvfs);
	return (err);
	}

	typedef struct zfs_zlock {
	krwlock_t zl_rwlock; / lock we acquired */
	znode_t zl_znode; / znode we held */
	struct zfs_zlock zl_next; / next in list */
	} zfs_zlock_t;

	/*
	* Drop locks and release vnodes that were held by zfs_rename_lock().
	*/
	static void
	zfs_rename_unlock(zfs_zlock_t **zlpp)
	{
	zfs_zlock_t *zl;

	while ((zl = *zlpp) != NULL) {
	if (zl->zl_znode != NULL)
	zfs_zrele_async(zl->zl_znode);
	rw_exit(zl->zl_rwlock);
	*zlpp = zl->zl_next;
	kmem_free(zl, sizeof (*zl));
	}
	}

	/*
	* Search back through the directory tree, using the ".." entries.
	* Lock each directory in the chain to prevent concurrent renames.
	* Fail any attempt to move a directory into one of its own descendants.
	* XXX - z_parent_lock can overlap with map or grow locks
	*/
	static int
	zfs_rename_lock(znode_t szp, znode_t tdzp, znode_t sdzp, zfs_zlock_t *zlpp)
	{
	zfs_zlock_t *zl;
	znode_t *zp = tdzp;
	uint64_t rootid = ZTOZSB(zp)->z_root;
	uint64_t oidp = zp->z_id;
	krwlock_t *rwlp = &szp->z_parent_lock;
	krw_t rw = RW_WRITER;

	/*
	* First pass write-locks szp and compares to zp->z_id.
	* Later passes read-lock zp and compare to zp->z_parent.
	*/
	do {
	if (!rw_tryenter(rwlp, rw)) {
	/*
	* Another thread is renaming in this path.
	* Note that if we are a WRITER, we don't have any
	* parent_locks held yet.
	*/
	if (rw == RW_READER && zp->z_id > szp->z_id) {
	/*
	* Drop our locks and restart
	*/
	zfs_rename_unlock(&zl);
	*zlpp = NULL;
	zp = tdzp;
	oidp = zp->z_id;
	rwlp = &szp->z_parent_lock;
	rw = RW_WRITER;
	continue;
	} else {
	/*
	* Wait for other thread to drop its locks
	*/
	rw_enter(rwlp, rw);
	}
	}

	zl = kmem_alloc(sizeof (*zl), KM_SLEEP);
	zl->zl_rwlock = rwlp;
	zl->zl_znode = NULL;
	zl->zl_next = *zlpp;
	*zlpp = zl;

	if (oidp == szp->z_id) /* We're a descendant of szp */
	return (SET_ERROR(EINVAL));

	if (oidp == rootid) /* We've hit the top */
	return (0);

	if (rw == RW_READER) { /* i.e. not the first pass */
	int error = zfs_zget(ZTOZSB(zp), oidp, &zp);
	if (error)
	return (error);
	zl->zl_znode = zp;
	}
	(void) sa_lookup(zp->z_sa_hdl, SA_ZPL_PARENT(ZTOZSB(zp)),
	&oidp, sizeof (oidp));
	rwlp = &zp->z_parent_lock;
	rw = RW_READER;

	} while (zp->z_id != sdzp->z_id);

	return (0);
	}

	/*
	* Move an entry from the provided source directory to the target
	* directory. Change the entry name as indicated.
	*
	* IN: sdzp - Source directory containing the "old entry".
	* snm - Old entry name.
	* tdzp - Target directory to contain the "new entry".
	* tnm - New entry name.
	* cr - credentials of caller.
	* flags - case flags
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* sdzp,tdzp - ctime\|mtime updated
	*/
	/ARGSUSED/
	int
	zfs_rename(znode_t sdzp, char snm, znode_t tdzp, char tnm,
	cred_t *cr, int flags)
	{
	znode_t szp, tzp;
	zfsvfs_t *zfsvfs = ZTOZSB(sdzp);
	zilog_t *zilog;
	zfs_dirlock_t sdl, tdl;
	dmu_tx_t *tx;
	zfs_zlock_t *zl;
	int cmp, serr, terr;
	int error = 0;
	int zflg = 0;
	boolean_t waited = B_FALSE;

	if (snm == NULL \|\| tnm == NULL)
	return (SET_ERROR(EINVAL));

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(sdzp);
	zilog = zfsvfs->z_log;

	ZFS_VERIFY_ZP(tdzp);

	/*
	* We check i_sb because snapshots and the ctldir must have different
	* super blocks.
	*/
	if (ZTOI(tdzp)->i_sb != ZTOI(sdzp)->i_sb \|\|
	zfsctl_is_node(ZTOI(tdzp))) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EXDEV));
	}

	if (zfsvfs->z_utf8 && u8_validate(tnm,
	strlen(tnm), NULL, U8_VALIDATE_ENTIRE, &error) < 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EILSEQ));
	}

	if (flags & FIGNORECASE)
	zflg \|= ZCILOOK;

	top:
	szp = NULL;
	tzp = NULL;
	zl = NULL;

	/*
	* This is to prevent the creation of links into attribute space
	* by renaming a linked file into/outof an attribute directory.
	* See the comment in zfs_link() for why this is considered bad.
	*/
	if ((tdzp->z_pflags & ZFS_XATTR) != (sdzp->z_pflags & ZFS_XATTR)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	/*
	* Lock source and target directory entries. To prevent deadlock,
	* a lock ordering must be defined. We lock the directory with
	* the smallest object id first, or if it's a tie, the one with
	* the lexically first name.
	*/
	if (sdzp->z_id < tdzp->z_id) {
	cmp = -1;
	} else if (sdzp->z_id > tdzp->z_id) {
	cmp = 1;
	} else {
	/*
	* First compare the two name arguments without
	* considering any case folding.
	*/
	int nofold = (zfsvfs->z_norm & ~U8_TEXTPREP_TOUPPER);

	cmp = u8_strcmp(snm, tnm, 0, nofold, U8_UNICODE_LATEST, &error);
	ASSERT(error == 0 \|\| !zfsvfs->z_utf8);
	if (cmp == 0) {
	/*
	* POSIX: "If the old argument and the new argument
	* both refer to links to the same existing file,
	* the rename() function shall return successfully
	* and perform no other action."
	*/
	ZFS_EXIT(zfsvfs);
	return (0);
	}
	/*
	* If the file system is case-folding, then we may
	* have some more checking to do. A case-folding file
	* system is either supporting mixed case sensitivity
	* access or is completely case-insensitive. Note
	* that the file system is always case preserving.
	*
	* In mixed sensitivity mode case sensitive behavior
	* is the default. FIGNORECASE must be used to
	* explicitly request case insensitive behavior.
	*
	* If the source and target names provided differ only
	* by case (e.g., a request to rename 'tim' to 'Tim'),
	* we will treat this as a special case in the
	* case-insensitive mode: as long as the source name
	* is an exact match, we will allow this to proceed as
	* a name-change request.
	*/
	if ((zfsvfs->z_case == ZFS_CASE_INSENSITIVE \|\|
	(zfsvfs->z_case == ZFS_CASE_MIXED &&
	flags & FIGNORECASE)) &&
	u8_strcmp(snm, tnm, 0, zfsvfs->z_norm, U8_UNICODE_LATEST,
	&error) == 0) {
	/*
	* case preserving rename request, require exact
	* name matches
	*/
	zflg \|= ZCIEXACT;
	zflg &= ~ZCILOOK;
	}
	}

	/*
	* If the source and destination directories are the same, we should
	* grab the z_name_lock of that directory only once.
	*/
	if (sdzp == tdzp) {
	zflg \|= ZHAVELOCK;
	rw_enter(&sdzp->z_name_lock, RW_READER);
	}

	if (cmp < 0) {
	serr = zfs_dirent_lock(&sdl, sdzp, snm, &szp,
	ZEXISTS \| zflg, NULL, NULL);
	terr = zfs_dirent_lock(&tdl,
	tdzp, tnm, &tzp, ZRENAMING \| zflg, NULL, NULL);
	} else {
	terr = zfs_dirent_lock(&tdl,
	tdzp, tnm, &tzp, zflg, NULL, NULL);
	serr = zfs_dirent_lock(&sdl,
	sdzp, snm, &szp, ZEXISTS \| ZRENAMING \| zflg,
	NULL, NULL);
	}

	if (serr) {
	/*
	* Source entry invalid or not there.
	*/
	if (!terr) {
	zfs_dirent_unlock(tdl);
	if (tzp)
	zrele(tzp);
	}

	if (sdzp == tdzp)
	rw_exit(&sdzp->z_name_lock);

	if (strcmp(snm, "..") == 0)
	serr = EINVAL;
	ZFS_EXIT(zfsvfs);
	return (serr);
	}
	if (terr) {
	zfs_dirent_unlock(sdl);
	zrele(szp);

	if (sdzp == tdzp)
	rw_exit(&sdzp->z_name_lock);

	if (strcmp(tnm, "..") == 0)
	terr = EINVAL;
	ZFS_EXIT(zfsvfs);
	return (terr);
	}

	/*
	* If we are using project inheritance, means if the directory has
	* ZFS_PROJINHERIT set, then its descendant directories will inherit
	* not only the project ID, but also the ZFS_PROJINHERIT flag. Under
	* such case, we only allow renames into our tree when the project
	* IDs are the same.
	*/
	if (tdzp->z_pflags & ZFS_PROJINHERIT &&
	tdzp->z_projid != szp->z_projid) {
	error = SET_ERROR(EXDEV);
	goto out;
	}

	/*
	* Must have write access at the source to remove the old entry
	* and write access at the target to create the new entry.
	* Note that if target and source are the same, this can be
	* done in a single check.
	*/

	if ((error = zfs_zaccess_rename(sdzp, szp, tdzp, tzp, cr)))
	goto out;

	if (S_ISDIR(ZTOI(szp)->i_mode)) {
	/*
	* Check to make sure rename is valid.
	* Can't do a move like this: /usr/a/b to /usr/a/b/c/d
	*/
	if ((error = zfs_rename_lock(szp, tdzp, sdzp, &zl)))
	goto out;
	}

	/*
	* Does target exist?
	*/
	if (tzp) {
	/*
	* Source and target must be the same type.
	*/
	if (S_ISDIR(ZTOI(szp)->i_mode)) {
	if (!S_ISDIR(ZTOI(tzp)->i_mode)) {
	error = SET_ERROR(ENOTDIR);
	goto out;
	}
	} else {
	if (S_ISDIR(ZTOI(tzp)->i_mode)) {
	error = SET_ERROR(EISDIR);
	goto out;
	}
	}
	/*
	* POSIX dictates that when the source and target
	* entries refer to the same file object, rename
	* must do nothing and exit without error.
	*/
	if (szp->z_id == tzp->z_id) {
	error = 0;
	goto out;
	}
	}

	tx = dmu_tx_create(zfsvfs->z_os);
	dmu_tx_hold_sa(tx, szp->z_sa_hdl, B_FALSE);
	dmu_tx_hold_sa(tx, sdzp->z_sa_hdl, B_FALSE);
	dmu_tx_hold_zap(tx, sdzp->z_id, FALSE, snm);
	dmu_tx_hold_zap(tx, tdzp->z_id, TRUE, tnm);
	if (sdzp != tdzp) {
	dmu_tx_hold_sa(tx, tdzp->z_sa_hdl, B_FALSE);
	zfs_sa_upgrade_txholds(tx, tdzp);
	}
	if (tzp) {
	dmu_tx_hold_sa(tx, tzp->z_sa_hdl, B_FALSE);
	zfs_sa_upgrade_txholds(tx, tzp);
	}

	zfs_sa_upgrade_txholds(tx, szp);
	dmu_tx_hold_zap(tx, zfsvfs->z_unlinkedobj, FALSE, NULL);
	error = dmu_tx_assign(tx, (waited ? TXG_NOTHROTTLE : 0) \| TXG_NOWAIT);
	if (error) {
	if (zl != NULL)
	zfs_rename_unlock(&zl);
	zfs_dirent_unlock(sdl);
	zfs_dirent_unlock(tdl);

	if (sdzp == tdzp)
	rw_exit(&sdzp->z_name_lock);

	if (error == ERESTART) {
	waited = B_TRUE;
	dmu_tx_wait(tx);
	dmu_tx_abort(tx);
	zrele(szp);
	if (tzp)
	zrele(tzp);
	goto top;
	}
	dmu_tx_abort(tx);
	zrele(szp);
	if (tzp)
	zrele(tzp);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	if (tzp) /* Attempt to remove the existing target */
	error = zfs_link_destroy(tdl, tzp, tx, zflg, NULL);

	if (error == 0) {
	error = zfs_link_create(tdl, szp, tx, ZRENAMING);
	if (error == 0) {
	szp->z_pflags \|= ZFS_AV_MODIFIED;
	if (tdzp->z_pflags & ZFS_PROJINHERIT)
	szp->z_pflags \|= ZFS_PROJINHERIT;

	error = sa_update(szp->z_sa_hdl, SA_ZPL_FLAGS(zfsvfs),
	(void *)&szp->z_pflags, sizeof (uint64_t), tx);
	ASSERT0(error);

	error = zfs_link_destroy(sdl, szp, tx, ZRENAMING, NULL);
	if (error == 0) {
	zfs_log_rename(zilog, tx, TX_RENAME \|
	(flags & FIGNORECASE ? TX_CI : 0), sdzp,
	sdl->dl_name, tdzp, tdl->dl_name, szp);
	} else {
	/*
	* At this point, we have successfully created
	* the target name, but have failed to remove
	* the source name. Since the create was done
	* with the ZRENAMING flag, there are
	* complications; for one, the link count is
	* wrong. The easiest way to deal with this
	* is to remove the newly created target, and
	* return the original error. This must
	* succeed; fortunately, it is very unlikely to
	* fail, since we just created it.
	*/
	VERIFY3U(zfs_link_destroy(tdl, szp, tx,
	ZRENAMING, NULL), ==, 0);
	}
	} else {
	/*
	* If we had removed the existing target, subsequent
	* call to zfs_link_create() to add back the same entry
	* but, the new dnode (szp) should not fail.
	*/
	ASSERT(tzp == NULL);
	}
	}

	dmu_tx_commit(tx);
	out:
	if (zl != NULL)
	zfs_rename_unlock(&zl);

	zfs_dirent_unlock(sdl);
	zfs_dirent_unlock(tdl);

	zfs_inode_update(sdzp);
	if (sdzp == tdzp)
	rw_exit(&sdzp->z_name_lock);

	if (sdzp != tdzp)
	zfs_inode_update(tdzp);

	zfs_inode_update(szp);
	zrele(szp);
	if (tzp) {
	zfs_inode_update(tzp);
	zrele(tzp);
	}

	if (zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, 0);

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Insert the indicated symbolic reference entry into the directory.
	*
	* IN: dzp - Directory to contain new symbolic link.
	* name - Name of directory entry in dip.
	* vap - Attributes of new entry.
	* link - Name for new symlink entry.
	* cr - credentials of caller.
	* flags - case flags
	*
	* OUT: zpp - Znode for new symbolic link.
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* dip - ctime\|mtime updated
	*/
	/ARGSUSED/
	int
	zfs_symlink(znode_t dzp, char name, vattr_t vap, char link,
	znode_t *zpp, cred_t cr, int flags)
	{
	znode_t *zp;
	zfs_dirlock_t *dl;
	dmu_tx_t *tx;
	zfsvfs_t *zfsvfs = ZTOZSB(dzp);
	zilog_t *zilog;
	uint64_t len = strlen(link);
	int error;
	int zflg = ZNEW;
	zfs_acl_ids_t acl_ids;
	boolean_t fuid_dirtied;
	uint64_t txtype = TX_SYMLINK;
	boolean_t waited = B_FALSE;

	ASSERT(S_ISLNK(vap->va_mode));

	if (name == NULL)
	return (SET_ERROR(EINVAL));

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(dzp);
	zilog = zfsvfs->z_log;

	if (zfsvfs->z_utf8 && u8_validate(name, strlen(name),
	NULL, U8_VALIDATE_ENTIRE, &error) < 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EILSEQ));
	}
	if (flags & FIGNORECASE)
	zflg \|= ZCILOOK;

	if (len > MAXPATHLEN) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(ENAMETOOLONG));
	}

	if ((error = zfs_acl_ids_create(dzp, 0,
	vap, cr, NULL, &acl_ids)) != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	top:
	*zpp = NULL;

	/*
	* Attempt to lock directory; fail if entry already exists.
	*/
	error = zfs_dirent_lock(&dl, dzp, name, &zp, zflg, NULL, NULL);
	if (error) {
	zfs_acl_ids_free(&acl_ids);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	if ((error = zfs_zaccess(dzp, ACE_ADD_FILE, 0, B_FALSE, cr))) {
	zfs_acl_ids_free(&acl_ids);
	zfs_dirent_unlock(dl);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	if (zfs_acl_ids_overquota(zfsvfs, &acl_ids, ZFS_DEFAULT_PROJID)) {
	zfs_acl_ids_free(&acl_ids);
	zfs_dirent_unlock(dl);
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EDQUOT));
	}
	tx = dmu_tx_create(zfsvfs->z_os);
	fuid_dirtied = zfsvfs->z_fuid_dirty;
	dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0, MAX(1, len));
	dmu_tx_hold_zap(tx, dzp->z_id, TRUE, name);
	dmu_tx_hold_sa_create(tx, acl_ids.z_aclp->z_acl_bytes +
	ZFS_SA_BASE_ATTR_SIZE + len);
	dmu_tx_hold_sa(tx, dzp->z_sa_hdl, B_FALSE);
	if (!zfsvfs->z_use_sa && acl_ids.z_aclp->z_acl_bytes > ZFS_ACE_SPACE) {
	dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0,
	acl_ids.z_aclp->z_acl_bytes);
	}
	if (fuid_dirtied)
	zfs_fuid_txhold(zfsvfs, tx);
	error = dmu_tx_assign(tx, (waited ? TXG_NOTHROTTLE : 0) \| TXG_NOWAIT);
	if (error) {
	zfs_dirent_unlock(dl);
	if (error == ERESTART) {
	waited = B_TRUE;
	dmu_tx_wait(tx);
	dmu_tx_abort(tx);
	goto top;
	}
	zfs_acl_ids_free(&acl_ids);
	dmu_tx_abort(tx);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Create a new object for the symlink.
	* for version 4 ZPL datsets the symlink will be an SA attribute
	*/
	zfs_mknode(dzp, vap, tx, cr, 0, &zp, &acl_ids);

	if (fuid_dirtied)
	zfs_fuid_sync(zfsvfs, tx);

	mutex_enter(&zp->z_lock);
	if (zp->z_is_sa)
	error = sa_update(zp->z_sa_hdl, SA_ZPL_SYMLINK(zfsvfs),
	link, len, tx);
	else
	zfs_sa_symlink(zp, link, len, tx);
	mutex_exit(&zp->z_lock);

	zp->z_size = len;
	(void) sa_update(zp->z_sa_hdl, SA_ZPL_SIZE(zfsvfs),
	&zp->z_size, sizeof (zp->z_size), tx);
	/*
	* Insert the new object into the directory.
	*/
	error = zfs_link_create(dl, zp, tx, ZNEW);
	if (error != 0) {
	zfs_znode_delete(zp, tx);
	remove_inode_hash(ZTOI(zp));
	} else {
	if (flags & FIGNORECASE)
	txtype \|= TX_CI;
	zfs_log_symlink(zilog, tx, txtype, dzp, zp, name, link);

	zfs_inode_update(dzp);
	zfs_inode_update(zp);
	}

	zfs_acl_ids_free(&acl_ids);

	dmu_tx_commit(tx);

	zfs_dirent_unlock(dl);

	if (error == 0) {
	*zpp = zp;

	if (zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, 0);
	} else {
	zrele(zp);
	}

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Return, in the buffer contained in the provided uio structure,
	* the symbolic path referred to by ip.
	*
	* IN: ip - inode of symbolic link
	* uio - structure to contain the link path.
	* cr - credentials of caller.
	*
	* RETURN: 0 if success
	* error code if failure
	*
	* Timestamps:
	* ip - atime updated
	*/
	/* ARGSUSED */
	int
	zfs_readlink(struct inode ip, uio_t uio, cred_t *cr)
	{
	znode_t *zp = ITOZ(ip);
	zfsvfs_t *zfsvfs = ITOZSB(ip);
	int error;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	mutex_enter(&zp->z_lock);
	if (zp->z_is_sa)
	error = sa_lookup_uio(zp->z_sa_hdl,
	SA_ZPL_SYMLINK(zfsvfs), uio);
	else
	error = zfs_sa_readlink(zp, uio);
	mutex_exit(&zp->z_lock);

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/*
	* Insert a new entry into directory tdzp referencing szp.
	*
	* IN: tdzp - Directory to contain new entry.
	* szp - znode of new entry.
	* name - name of new entry.
	* cr - credentials of caller.
	* flags - case flags.
	*
	* RETURN: 0 if success
	* error code if failure
	*
	* Timestamps:
	* tdzp - ctime\|mtime updated
	* szp - ctime updated
	*/
	/* ARGSUSED */
	int
	zfs_link(znode_t tdzp, znode_t szp, char name, cred_t cr,
	int flags)
	{
	struct inode *sip = ZTOI(szp);
	znode_t *tzp;
	zfsvfs_t *zfsvfs = ZTOZSB(tdzp);
	zilog_t *zilog;
	zfs_dirlock_t *dl;
	dmu_tx_t *tx;
	int error;
	int zf = ZNEW;
	uint64_t parent;
	uid_t owner;
	boolean_t waited = B_FALSE;
	boolean_t is_tmpfile = 0;
	uint64_t txg;
	#ifdef HAVE_TMPFILE
	is_tmpfile = (sip->i_nlink == 0 && (sip->i_state & I_LINKABLE));
	#endif
	ASSERT(S_ISDIR(ZTOI(tdzp)->i_mode));

	if (name == NULL)
	return (SET_ERROR(EINVAL));

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(tdzp);
	zilog = zfsvfs->z_log;

	/*
	* POSIX dictates that we return EPERM here.
	* Better choices include ENOTSUP or EISDIR.
	*/
	if (S_ISDIR(sip->i_mode)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EPERM));
	}

	ZFS_VERIFY_ZP(szp);

	/*
	* If we are using project inheritance, means if the directory has
	* ZFS_PROJINHERIT set, then its descendant directories will inherit
	* not only the project ID, but also the ZFS_PROJINHERIT flag. Under
	* such case, we only allow hard link creation in our tree when the
	* project IDs are the same.
	*/
	if (tdzp->z_pflags & ZFS_PROJINHERIT &&
	tdzp->z_projid != szp->z_projid) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EXDEV));
	}

	/*
	* We check i_sb because snapshots and the ctldir must have different
	* super blocks.
	*/
	if (sip->i_sb != ZTOI(tdzp)->i_sb \|\| zfsctl_is_node(sip)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EXDEV));
	}

	/* Prevent links to .zfs/shares files */

	if ((error = sa_lookup(szp->z_sa_hdl, SA_ZPL_PARENT(zfsvfs),
	&parent, sizeof (uint64_t))) != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	if (parent == zfsvfs->z_shares_dir) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EPERM));
	}

	if (zfsvfs->z_utf8 && u8_validate(name,
	strlen(name), NULL, U8_VALIDATE_ENTIRE, &error) < 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EILSEQ));
	}
	if (flags & FIGNORECASE)
	zf \|= ZCILOOK;

	/*
	* We do not support links between attributes and non-attributes
	* because of the potential security risk of creating links
	* into "normal" file space in order to circumvent restrictions
	* imposed in attribute space.
	*/
	if ((szp->z_pflags & ZFS_XATTR) != (tdzp->z_pflags & ZFS_XATTR)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	owner = zfs_fuid_map_id(zfsvfs, KUID_TO_SUID(sip->i_uid),
	cr, ZFS_OWNER);
	if (owner != crgetuid(cr) && secpolicy_basic_link(cr) != 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EPERM));
	}

	if ((error = zfs_zaccess(tdzp, ACE_ADD_FILE, 0, B_FALSE, cr))) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	top:
	/*
	* Attempt to lock directory; fail if entry already exists.
	*/
	error = zfs_dirent_lock(&dl, tdzp, name, &tzp, zf, NULL, NULL);
	if (error) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	tx = dmu_tx_create(zfsvfs->z_os);
	dmu_tx_hold_sa(tx, szp->z_sa_hdl, B_FALSE);
	dmu_tx_hold_zap(tx, tdzp->z_id, TRUE, name);
	if (is_tmpfile)
	dmu_tx_hold_zap(tx, zfsvfs->z_unlinkedobj, FALSE, NULL);

	zfs_sa_upgrade_txholds(tx, szp);
	zfs_sa_upgrade_txholds(tx, tdzp);
	error = dmu_tx_assign(tx, (waited ? TXG_NOTHROTTLE : 0) \| TXG_NOWAIT);
	if (error) {
	zfs_dirent_unlock(dl);
	if (error == ERESTART) {
	waited = B_TRUE;
	dmu_tx_wait(tx);
	dmu_tx_abort(tx);
	goto top;
	}
	dmu_tx_abort(tx);
	ZFS_EXIT(zfsvfs);
	return (error);
	}
	/* unmark z_unlinked so zfs_link_create will not reject */
	if (is_tmpfile)
	szp->z_unlinked = B_FALSE;
	error = zfs_link_create(dl, szp, tx, 0);

	if (error == 0) {
	uint64_t txtype = TX_LINK;
	/*
	* tmpfile is created to be in z_unlinkedobj, so remove it.
	* Also, we don't log in ZIL, because all previous file
	* operation on the tmpfile are ignored by ZIL. Instead we
	* always wait for txg to sync to make sure all previous
	* operation are sync safe.
	*/
	if (is_tmpfile) {
	VERIFY(zap_remove_int(zfsvfs->z_os,
	zfsvfs->z_unlinkedobj, szp->z_id, tx) == 0);
	} else {
	if (flags & FIGNORECASE)
	txtype \|= TX_CI;
	zfs_log_link(zilog, tx, txtype, tdzp, szp, name);
	}
	} else if (is_tmpfile) {
	/* restore z_unlinked since when linking failed */
	szp->z_unlinked = B_TRUE;
	}
	txg = dmu_tx_get_txg(tx);
	dmu_tx_commit(tx);

	zfs_dirent_unlock(dl);

	if (!is_tmpfile && zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, 0);

	if (is_tmpfile && zfsvfs->z_os->os_sync != ZFS_SYNC_DISABLED)
	txg_wait_synced(dmu_objset_pool(zfsvfs->z_os), txg);

	zfs_inode_update(tdzp);
	zfs_inode_update(szp);
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	static void
	zfs_putpage_commit_cb(void *arg)
	{
	struct page *pp = arg;

	ClearPageError(pp);
	end_page_writeback(pp);
	}

	/*
	* Push a page out to disk, once the page is on stable storage the
	* registered commit callback will be run as notification of completion.
	*
	* IN: ip - page mapped for inode.
	* pp - page to push (page is locked)
	* wbc - writeback control data
	*
	* RETURN: 0 if success
	* error code if failure
	*
	* Timestamps:
	* ip - ctime\|mtime updated
	*/
	/* ARGSUSED */
	int
	zfs_putpage(struct inode ip, struct page pp, struct writeback_control *wbc)
	{
	znode_t *zp = ITOZ(ip);
	zfsvfs_t *zfsvfs = ITOZSB(ip);
	loff_t offset;
	loff_t pgoff;
	unsigned int pglen;
	dmu_tx_t *tx;
	caddr_t va;
	int err = 0;
	uint64_t mtime[2], ctime[2];
	sa_bulk_attr_t bulk[3];
	int cnt = 0;
	struct address_space *mapping;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	ASSERT(PageLocked(pp));

	pgoff = page_offset(pp); /* Page byte-offset in file */
	offset = i_size_read(ip); /* File length in bytes */
	pglen = MIN(PAGE_SIZE, /* Page length in bytes */
	P2ROUNDUP(offset, PAGE_SIZE)-pgoff);

	/* Page is beyond end of file */
	if (pgoff >= offset) {
	unlock_page(pp);
	ZFS_EXIT(zfsvfs);
	return (0);
	}

	/* Truncate page length to end of file */
	if (pgoff + pglen > offset)
	pglen = offset - pgoff;

	#if 0
	/*
	* FIXME: Allow mmap writes past its quota. The correct fix
	* is to register a page_mkwrite() handler to count the page
	* against its quota when it is about to be dirtied.
	*/
	if (zfs_id_overblockquota(zfsvfs, DMU_USERUSED_OBJECT,
	KUID_TO_SUID(ip->i_uid)) \|\|
	zfs_id_overblockquota(zfsvfs, DMU_GROUPUSED_OBJECT,
	KGID_TO_SGID(ip->i_gid)) \|\|
	(zp->z_projid != ZFS_DEFAULT_PROJID &&
	zfs_id_overblockquota(zfsvfs, DMU_PROJECTUSED_OBJECT,
	zp->z_projid))) {
	err = EDQUOT;
	}
	#endif

	/*
	* The ordering here is critical and must adhere to the following
	* rules in order to avoid deadlocking in either zfs_read() or
	* zfs_free_range() due to a lock inversion.
	*
	* 1) The page must be unlocked prior to acquiring the range lock.
	* This is critical because zfs_read() calls find_lock_page()
	* which may block on the page lock while holding the range lock.
	*
	* 2) Before setting or clearing write back on a page the range lock
	* must be held in order to prevent a lock inversion with the
	* zfs_free_range() function.
	*
	* This presents a problem because upon entering this function the
	* page lock is already held. To safely acquire the range lock the
	* page lock must be dropped. This creates a window where another
	* process could truncate, invalidate, dirty, or write out the page.
	*
	* Therefore, after successfully reacquiring the range and page locks
	* the current page state is checked. In the common case everything
	* will be as is expected and it can be written out. However, if
	* the page state has changed it must be handled accordingly.
	*/
	mapping = pp->mapping;
	redirty_page_for_writepage(wbc, pp);
	unlock_page(pp);

	zfs_locked_range_t *lr = zfs_rangelock_enter(&zp->z_rangelock,
	pgoff, pglen, RL_WRITER);
	lock_page(pp);

	/* Page mapping changed or it was no longer dirty, we're done */
	if (unlikely((mapping != pp->mapping) \|\| !PageDirty(pp))) {
	unlock_page(pp);
	zfs_rangelock_exit(lr);
	ZFS_EXIT(zfsvfs);
	return (0);
	}

	/* Another process started write block if required */
	if (PageWriteback(pp)) {
	unlock_page(pp);
	zfs_rangelock_exit(lr);

	if (wbc->sync_mode != WB_SYNC_NONE) {
	if (PageWriteback(pp))
	wait_on_page_bit(pp, PG_writeback);
	}

	ZFS_EXIT(zfsvfs);
	return (0);
	}

	/* Clear the dirty flag the required locks are held */
	if (!clear_page_dirty_for_io(pp)) {
	unlock_page(pp);
	zfs_rangelock_exit(lr);
	ZFS_EXIT(zfsvfs);
	return (0);
	}

	/*
	* Counterpart for redirty_page_for_writepage() above. This page
	* was in fact not skipped and should not be counted as if it were.
	*/
	wbc->pages_skipped--;
	set_page_writeback(pp);
	unlock_page(pp);

	tx = dmu_tx_create(zfsvfs->z_os);
	dmu_tx_hold_write(tx, zp->z_id, pgoff, pglen);
	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_FALSE);
	zfs_sa_upgrade_txholds(tx, zp);

	err = dmu_tx_assign(tx, TXG_NOWAIT);
	if (err != 0) {
	if (err == ERESTART)
	dmu_tx_wait(tx);

	dmu_tx_abort(tx);
	__set_page_dirty_nobuffers(pp);
	ClearPageError(pp);
	end_page_writeback(pp);
	zfs_rangelock_exit(lr);
	ZFS_EXIT(zfsvfs);
	return (err);
	}

	va = kmap(pp);
	ASSERT3U(pglen, <=, PAGE_SIZE);
	dmu_write(zfsvfs->z_os, zp->z_id, pgoff, pglen, va, tx);
	kunmap(pp);

	SA_ADD_BULK_ATTR(bulk, cnt, SA_ZPL_MTIME(zfsvfs), NULL, &mtime, 16);
	SA_ADD_BULK_ATTR(bulk, cnt, SA_ZPL_CTIME(zfsvfs), NULL, &ctime, 16);
	SA_ADD_BULK_ATTR(bulk, cnt, SA_ZPL_FLAGS(zfsvfs), NULL,
	&zp->z_pflags, 8);

	/* Preserve the mtime and ctime provided by the inode */
	ZFS_TIME_ENCODE(&ip->i_mtime, mtime);
	ZFS_TIME_ENCODE(&ip->i_ctime, ctime);
	zp->z_atime_dirty = B_FALSE;
	zp->z_seq++;

	err = sa_bulk_update(zp->z_sa_hdl, bulk, cnt, tx);

	zfs_log_write(zfsvfs->z_log, tx, TX_WRITE, zp, pgoff, pglen, 0,
	zfs_putpage_commit_cb, pp);
	dmu_tx_commit(tx);

	zfs_rangelock_exit(lr);

	if (wbc->sync_mode != WB_SYNC_NONE) {
	/*
	* Note that this is rarely called under writepages(), because
	* writepages() normally handles the entire commit for
	* performance reasons.
	*/
	zil_commit(zfsvfs->z_log, zp->z_id);
	}

	ZFS_EXIT(zfsvfs);
	return (err);
	}

	/*
	* Update the system attributes when the inode has been dirtied. For the
	* moment we only update the mode, atime, mtime, and ctime.
	*/
	int
	zfs_dirty_inode(struct inode *ip, int flags)
	{
	znode_t *zp = ITOZ(ip);
	zfsvfs_t *zfsvfs = ITOZSB(ip);
	dmu_tx_t *tx;
	uint64_t mode, atime[2], mtime[2], ctime[2];
	sa_bulk_attr_t bulk[4];
	int error = 0;
	int cnt = 0;

	if (zfs_is_readonly(zfsvfs) \|\| dmu_objset_is_snapshot(zfsvfs->z_os))
	return (0);

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	#ifdef I_DIRTY_TIME
	/*
	* This is the lazytime semantic introduced in Linux 4.0
	* This flag will only be called from update_time when lazytime is set.
	* (Note, I_DIRTY_SYNC will also set if not lazytime)
	* Fortunately mtime and ctime are managed within ZFS itself, so we
	* only need to dirty atime.
	*/
	if (flags == I_DIRTY_TIME) {
	zp->z_atime_dirty = B_TRUE;
	goto out;
	}
	#endif

	tx = dmu_tx_create(zfsvfs->z_os);

	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_FALSE);
	zfs_sa_upgrade_txholds(tx, zp);

	error = dmu_tx_assign(tx, TXG_WAIT);
	if (error) {
	dmu_tx_abort(tx);
	goto out;
	}

	mutex_enter(&zp->z_lock);
	zp->z_atime_dirty = B_FALSE;

	SA_ADD_BULK_ATTR(bulk, cnt, SA_ZPL_MODE(zfsvfs), NULL, &mode, 8);
	SA_ADD_BULK_ATTR(bulk, cnt, SA_ZPL_ATIME(zfsvfs), NULL, &atime, 16);
	SA_ADD_BULK_ATTR(bulk, cnt, SA_ZPL_MTIME(zfsvfs), NULL, &mtime, 16);
	SA_ADD_BULK_ATTR(bulk, cnt, SA_ZPL_CTIME(zfsvfs), NULL, &ctime, 16);

	/* Preserve the mode, mtime and ctime provided by the inode */
	ZFS_TIME_ENCODE(&ip->i_atime, atime);
	ZFS_TIME_ENCODE(&ip->i_mtime, mtime);
	ZFS_TIME_ENCODE(&ip->i_ctime, ctime);
	mode = ip->i_mode;

	zp->z_mode = mode;

	error = sa_bulk_update(zp->z_sa_hdl, bulk, cnt, tx);
	mutex_exit(&zp->z_lock);

	dmu_tx_commit(tx);
	out:
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/ARGSUSED/
	void
	zfs_inactive(struct inode *ip)
	{
	znode_t *zp = ITOZ(ip);
	zfsvfs_t *zfsvfs = ITOZSB(ip);
	uint64_t atime[2];
	int error;
	int need_unlock = 0;

	/* Only read lock if we haven't already write locked, e.g. rollback */
	if (!RW_WRITE_HELD(&zfsvfs->z_teardown_inactive_lock)) {
	need_unlock = 1;
	rw_enter(&zfsvfs->z_teardown_inactive_lock, RW_READER);
	}
	if (zp->z_sa_hdl == NULL) {
	if (need_unlock)
	rw_exit(&zfsvfs->z_teardown_inactive_lock);
	return;
	}

	if (zp->z_atime_dirty && zp->z_unlinked == B_FALSE) {
	dmu_tx_t *tx = dmu_tx_create(zfsvfs->z_os);

	dmu_tx_hold_sa(tx, zp->z_sa_hdl, B_FALSE);
	zfs_sa_upgrade_txholds(tx, zp);
	error = dmu_tx_assign(tx, TXG_WAIT);
	if (error) {
	dmu_tx_abort(tx);
	} else {
	ZFS_TIME_ENCODE(&ip->i_atime, atime);
	mutex_enter(&zp->z_lock);
	(void) sa_update(zp->z_sa_hdl, SA_ZPL_ATIME(zfsvfs),
	(void *)&atime, sizeof (atime), tx);
	zp->z_atime_dirty = B_FALSE;
	mutex_exit(&zp->z_lock);
	dmu_tx_commit(tx);
	}
	}

	zfs_zinactive(zp);
	if (need_unlock)
	rw_exit(&zfsvfs->z_teardown_inactive_lock);
	}

	/*
	* Fill pages with data from the disk.
	*/
	static int
	zfs_fillpage(struct inode ip, struct page pl[], int nr_pages)
	{
	znode_t *zp = ITOZ(ip);
	zfsvfs_t *zfsvfs = ITOZSB(ip);
	objset_t *os;
	struct page *cur_pp;
	u_offset_t io_off, total;
	size_t io_len;
	loff_t i_size;
	unsigned page_idx;
	int err;

	os = zfsvfs->z_os;
	io_len = nr_pages << PAGE_SHIFT;
	i_size = i_size_read(ip);
	io_off = page_offset(pl[0]);

	if (io_off + io_len > i_size)
	io_len = i_size - io_off;

	/*
	* Iterate over list of pages and read each page individually.
	*/
	page_idx = 0;
	for (total = io_off + io_len; io_off < total; io_off += PAGESIZE) {
	caddr_t va;

	cur_pp = pl[page_idx++];
	va = kmap(cur_pp);
	err = dmu_read(os, zp->z_id, io_off, PAGESIZE, va,
	DMU_READ_PREFETCH);
	kunmap(cur_pp);
	if (err) {
	/* convert checksum errors into IO errors */
	if (err == ECKSUM)
	err = SET_ERROR(EIO);
	return (err);
	}
	}

	return (0);
	}

	/*
	* Uses zfs_fillpage to read data from the file and fill the pages.
	*
	* IN: ip - inode of file to get data from.
	* pl - list of pages to read
	* nr_pages - number of pages to read
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* vp - atime updated
	*/
	/* ARGSUSED */
	int
	zfs_getpage(struct inode ip, struct page pl[], int nr_pages)
	{
	znode_t *zp = ITOZ(ip);
	zfsvfs_t *zfsvfs = ITOZSB(ip);
	int err;

	if (pl == NULL)
	return (0);

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	err = zfs_fillpage(ip, pl, nr_pages);

	ZFS_EXIT(zfsvfs);
	return (err);
	}

	/*
	* Check ZFS specific permissions to memory map a section of a file.
	*
	* IN: ip - inode of the file to mmap
	* off - file offset
	* addrp - start address in memory region
	* len - length of memory region
	* vm_flags- address flags
	*
	* RETURN: 0 if success
	* error code if failure
	*/
	/ARGSUSED/
	int
	zfs_map(struct inode ip, offset_t off, caddr_t addrp, size_t len,
	unsigned long vm_flags)
	{
	znode_t *zp = ITOZ(ip);
	zfsvfs_t *zfsvfs = ITOZSB(ip);

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	if ((vm_flags & VM_WRITE) && (zp->z_pflags &
	(ZFS_IMMUTABLE \| ZFS_READONLY \| ZFS_APPENDONLY))) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EPERM));
	}

	if ((vm_flags & (VM_READ \| VM_EXEC)) &&
	(zp->z_pflags & ZFS_AV_QUARANTINED)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EACCES));
	}

	if (off < 0 \|\| len > MAXOFFSET_T - off) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(ENXIO));
	}

	ZFS_EXIT(zfsvfs);
	return (0);
	}

	/*
	* Free or allocate space in a file. Currently, this function only
	* supports the `F_FREESP' command. However, this command is somewhat
	* misnamed, as its functionality includes the ability to allocate as
	* well as free space.
	*
	* IN: zp - znode of file to free data in.
	* cmd - action to take (only F_FREESP supported).
	* bfp - section of file to free/alloc.
	* flag - current file open mode flags.
	* offset - current file offset.
	* cr - credentials of caller.
	*
	* RETURN: 0 on success, error code on failure.
	*
	* Timestamps:
	* zp - ctime\|mtime updated
	*/
	/* ARGSUSED */
	int
	zfs_space(znode_t zp, int cmd, flock64_t bfp, int flag,
	offset_t offset, cred_t *cr)
	{
	zfsvfs_t *zfsvfs = ZTOZSB(zp);
	uint64_t off, len;
	int error;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	if (cmd != F_FREESP) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	/*
	* Callers might not be able to detect properly that we are read-only,
	* so check it explicitly here.
	*/
	if (zfs_is_readonly(zfsvfs)) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EROFS));
	}

	if (bfp->l_len < 0) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	/*
	* Permissions aren't checked on Solaris because on this OS
	* zfs_space() can only be called with an opened file handle.
	* On Linux we can get here through truncate_range() which
	* operates directly on inodes, so we need to check access rights.
	*/
	if ((error = zfs_zaccess(zp, ACE_WRITE_DATA, 0, B_FALSE, cr))) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	off = bfp->l_start;
	len = bfp->l_len; /* 0 means from off to end of file */

	error = zfs_freesp(zp, off, len, flag, TRUE);

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	/ARGSUSED/
	int
	zfs_fid(struct inode ip, fid_t fidp)
	{
	znode_t *zp = ITOZ(ip);
	zfsvfs_t *zfsvfs = ITOZSB(ip);
	uint32_t gen;
	uint64_t gen64;
	uint64_t object = zp->z_id;
	zfid_short_t *zfid;
	int size, i, error;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	if ((error = sa_lookup(zp->z_sa_hdl, SA_ZPL_GEN(zfsvfs),
	&gen64, sizeof (uint64_t))) != 0) {
	ZFS_EXIT(zfsvfs);
	return (error);
	}

	gen = (uint32_t)gen64;

	size = SHORT_FID_LEN;

	zfid = (zfid_short_t *)fidp;

	zfid->zf_len = size;

	for (i = 0; i < sizeof (zfid->zf_object); i++)
	zfid->zf_object[i] = (uint8_t)(object >> (8 * i));

	/* Must have a non-zero generation number to distinguish from .zfs */
	if (gen == 0)
	gen = 1;
	for (i = 0; i < sizeof (zfid->zf_gen); i++)
	zfid->zf_gen[i] = (uint8_t)(gen >> (8 * i));

	ZFS_EXIT(zfsvfs);
	return (0);
	}

	/ARGSUSED/
	int
	zfs_getsecattr(struct inode ip, vsecattr_t vsecp, int flag, cred_t *cr)
	{
	znode_t *zp = ITOZ(ip);
	zfsvfs_t *zfsvfs = ITOZSB(ip);
	int error;
	boolean_t skipaclchk = (flag & ATTR_NOACLCHECK) ? B_TRUE : B_FALSE;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);
	error = zfs_getacl(zp, vsecp, skipaclchk, cr);
	ZFS_EXIT(zfsvfs);

	return (error);
	}

	/ARGSUSED/
	int
	zfs_setsecattr(znode_t zp, vsecattr_t vsecp, int flag, cred_t *cr)
	{
	zfsvfs_t *zfsvfs = ZTOZSB(zp);
	int error;
	boolean_t skipaclchk = (flag & ATTR_NOACLCHECK) ? B_TRUE : B_FALSE;
	zilog_t *zilog = zfsvfs->z_log;

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);

	error = zfs_setacl(zp, vsecp, skipaclchk, cr);

	if (zfsvfs->z_os->os_sync == ZFS_SYNC_ALWAYS)
	zil_commit(zilog, 0);

	ZFS_EXIT(zfsvfs);
	return (error);
	}

	#ifdef HAVE_UIO_ZEROCOPY
	/*
	* The smallest read we may consider to loan out an arcbuf.
	* This must be a power of 2.
	*/
	int zcr_blksz_min = (1 << 10); /* 1K */
	/*
	* If set to less than the file block size, allow loaning out of an
	* arcbuf for a partial block read. This must be a power of 2.
	*/
	int zcr_blksz_max = (1 << 17); /* 128K */

	/ARGSUSED/
	static int
	zfs_reqzcbuf(struct inode ip, enum uio_rw ioflag, xuio_t xuio, cred_t *cr)
	{
	znode_t *zp = ITOZ(ip);
	zfsvfs_t *zfsvfs = ITOZSB(ip);
	int max_blksz = zfsvfs->z_max_blksz;
	uio_t *uio = &xuio->xu_uio;
	ssize_t size = uio->uio_resid;
	offset_t offset = uio->uio_loffset;
	int blksz;
	int fullblk, i;
	arc_buf_t *abuf;
	ssize_t maxsize;
	int preamble, postamble;

	if (xuio->xu_type != UIOTYPE_ZEROCOPY)
	return (SET_ERROR(EINVAL));

	ZFS_ENTER(zfsvfs);
	ZFS_VERIFY_ZP(zp);
	switch (ioflag) {
	case UIO_WRITE:
	/*
	* Loan out an arc_buf for write if write size is bigger than
	* max_blksz, and the file's block size is also max_blksz.
	*/
	blksz = max_blksz;
	if (size < blksz \|\| zp->z_blksz != blksz) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}
	/*
	* Caller requests buffers for write before knowing where the
	* write offset might be (e.g. NFS TCP write).
	*/
	if (offset == -1) {
	preamble = 0;
	} else {
	preamble = P2PHASE(offset, blksz);
	if (preamble) {
	preamble = blksz - preamble;
	size -= preamble;
	}
	}

	postamble = P2PHASE(size, blksz);
	size -= postamble;

	fullblk = size / blksz;
	(void) dmu_xuio_init(xuio,
	(preamble != 0) + fullblk + (postamble != 0));

	/*
	* Have to fix iov base/len for partial buffers. They
	* currently represent full arc_buf's.
	*/
	if (preamble) {
	/* data begins in the middle of the arc_buf */
	abuf = dmu_request_arcbuf(sa_get_db(zp->z_sa_hdl),
	blksz);
	ASSERT(abuf);
	(void) dmu_xuio_add(xuio, abuf,
	blksz - preamble, preamble);
	}

	for (i = 0; i < fullblk; i++) {
	abuf = dmu_request_arcbuf(sa_get_db(zp->z_sa_hdl),
	blksz);
	ASSERT(abuf);
	(void) dmu_xuio_add(xuio, abuf, 0, blksz);
	}

	if (postamble) {
	/* data ends in the middle of the arc_buf */
	abuf = dmu_request_arcbuf(sa_get_db(zp->z_sa_hdl),
	blksz);
	ASSERT(abuf);
	(void) dmu_xuio_add(xuio, abuf, 0, postamble);
	}
	break;
	case UIO_READ:
	/*
	* Loan out an arc_buf for read if the read size is larger than
	* the current file block size. Block alignment is not
	* considered. Partial arc_buf will be loaned out for read.
	*/
	blksz = zp->z_blksz;
	if (blksz < zcr_blksz_min)
	blksz = zcr_blksz_min;
	if (blksz > zcr_blksz_max)
	blksz = zcr_blksz_max;
	/* avoid potential complexity of dealing with it */
	if (blksz > max_blksz) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	maxsize = zp->z_size - uio->uio_loffset;
	if (size > maxsize)
	size = maxsize;

	if (size < blksz) {
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}
	break;
	default:
	ZFS_EXIT(zfsvfs);
	return (SET_ERROR(EINVAL));
	}

	uio->uio_extflg = UIO_XUIO;
	XUIO_XUZC_RW(xuio) = ioflag;
	ZFS_EXIT(zfsvfs);
	return (0);
	}

	/ARGSUSED/
	static int
	zfs_retzcbuf(struct inode ip, xuio_t xuio, cred_t *cr)
	{
	int i;
	arc_buf_t *abuf;
	int ioflag = XUIO_XUZC_RW(xuio);

	ASSERT(xuio->xu_type == UIOTYPE_ZEROCOPY);

	i = dmu_xuio_cnt(xuio);
	while (i-- > 0) {
	abuf = dmu_xuio_arcbuf(xuio, i);
	/*
	* if abuf == NULL, it must be a write buffer
	* that has been returned in zfs_write().
	*/
	if (abuf)
	dmu_return_arcbuf(abuf);
	ASSERT(abuf \|\| ioflag == UIO_WRITE);
	}

	dmu_xuio_fini(xuio);
	return (0);
	}
	#endif /* HAVE_UIO_ZEROCOPY */

	#if defined(_KERNEL)
	EXPORT_SYMBOL(zfs_open);
	EXPORT_SYMBOL(zfs_close);
	EXPORT_SYMBOL(zfs_read);
	EXPORT_SYMBOL(zfs_write);
	EXPORT_SYMBOL(zfs_access);
	EXPORT_SYMBOL(zfs_lookup);
	EXPORT_SYMBOL(zfs_create);
	EXPORT_SYMBOL(zfs_tmpfile);
	EXPORT_SYMBOL(zfs_remove);
	EXPORT_SYMBOL(zfs_mkdir);
	EXPORT_SYMBOL(zfs_rmdir);
	EXPORT_SYMBOL(zfs_readdir);
	EXPORT_SYMBOL(zfs_fsync);
	EXPORT_SYMBOL(zfs_getattr_fast);
	EXPORT_SYMBOL(zfs_setattr);
	EXPORT_SYMBOL(zfs_rename);
	EXPORT_SYMBOL(zfs_symlink);
	EXPORT_SYMBOL(zfs_readlink);
	EXPORT_SYMBOL(zfs_link);
	EXPORT_SYMBOL(zfs_inactive);
	EXPORT_SYMBOL(zfs_space);
	EXPORT_SYMBOL(zfs_fid);
	EXPORT_SYMBOL(zfs_getsecattr);
	EXPORT_SYMBOL(zfs_setsecattr);
	EXPORT_SYMBOL(zfs_getpage);
	EXPORT_SYMBOL(zfs_putpage);
	EXPORT_SYMBOL(zfs_dirty_inode);
	EXPORT_SYMBOL(zfs_map);

	/* BEGIN CSTYLED */
	module_param(zfs_delete_blocks, ulong, 0644);
	MODULE_PARM_DESC(zfs_delete_blocks, "Delete files larger than N blocks async");
	module_param(zfs_read_chunk_size, ulong, 0644);
	MODULE_PARM_DESC(zfs_read_chunk_size, "Bytes to read per chunk");
	/* END CSTYLED */

	#endif
	Index: vendor-sys/openzfs/dist/module/unicode/u8_textprep.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/unicode/u8_textprep.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/unicode/u8_textprep.c (revision 366774)
	@@ -1,2151 +1,2151 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright 2008 Sun Microsystems, Inc. All rights reserved.
	* Use is subject to license terms.
	*/




	/*
	* UTF-8 text preparation functions (PSARC/2007/149, PSARC/2007/458).
	*
	* Man pages: u8_textprep_open(9F), u8_textprep_buf(9F), u8_textprep_close(9F),
	* u8_textprep_str(9F), u8_strcmp(9F), and u8_validate(9F). See also
	* the section 3C man pages.
	* Interface stability: Committed.
	*/

	#include <sys/types.h>
	#include <sys/strings.h>
	#include <sys/param.h>
	#include <sys/sysmacros.h>
	#include <sys/debug.h>
	#include <sys/kmem.h>
	#include <sys/sunddi.h>
	#include <sys/u8_textprep.h>
	#include <sys/byteorder.h>
	#include <sys/errno.h>
	#include <sys/u8_textprep_data.h>
	#include <sys/mod.h>

	/* The maximum possible number of bytes in a UTF-8 character. */
	#define U8_MB_CUR_MAX (4)

	/*
	* The maximum number of bytes needed for a UTF-8 character to cover
	* U+0000 - U+FFFF, i.e., the coding space of now deprecated UCS-2.
	*/
	#define U8_MAX_BYTES_UCS2 (3)

	/* The maximum possible number of bytes in a Stream-Safe Text. */
	#define U8_STREAM_SAFE_TEXT_MAX (128)

	/*
	* The maximum number of characters in a combining/conjoining sequence and
	* the actual upperbound limit of a combining/conjoining sequence.
	*/
	#define U8_MAX_CHARS_A_SEQ (32)
	#define U8_UPPER_LIMIT_IN_A_SEQ (31)

	/* The combining class value for Starter. */
	#define U8_COMBINING_CLASS_STARTER (0)

	/*
	* Some Hangul related macros at below.
	*
	* The first and the last of Hangul syllables, Hangul Jamo Leading consonants,
	* Vowels, and optional Trailing consonants in Unicode scalar values.
	*
	* Please be noted that the U8_HANGUL_JAMO_T_FIRST is 0x11A7 at below not
	* the actual U+11A8. This is due to that the trailing consonant is optional
	* and thus we are doing a pre-calculation of subtracting one.
	*
	* Each of 19 modern leading consonants has total 588 possible syllables since
	* Hangul has 21 modern vowels and 27 modern trailing consonants plus 1 for
	* no trailing consonant case, i.e., 21 x 28 = 588.
	*
	* We also have bunch of Hangul related macros at below. Please bear in mind
	* that the U8_HANGUL_JAMO_1ST_BYTE can be used to check whether it is
	* a Hangul Jamo or not but the value does not guarantee that it is a Hangul
	* Jamo; it just guarantee that it will be most likely.
	*/
	#define U8_HANGUL_SYL_FIRST (0xAC00U)
	#define U8_HANGUL_SYL_LAST (0xD7A3U)

	#define U8_HANGUL_JAMO_L_FIRST (0x1100U)
	#define U8_HANGUL_JAMO_L_LAST (0x1112U)
	#define U8_HANGUL_JAMO_V_FIRST (0x1161U)
	#define U8_HANGUL_JAMO_V_LAST (0x1175U)
	#define U8_HANGUL_JAMO_T_FIRST (0x11A7U)
	#define U8_HANGUL_JAMO_T_LAST (0x11C2U)

	#define U8_HANGUL_V_COUNT (21)
	#define U8_HANGUL_VT_COUNT (588)
	#define U8_HANGUL_T_COUNT (28)

	#define U8_HANGUL_JAMO_1ST_BYTE (0xE1U)

	#define U8_SAVE_HANGUL_AS_UTF8(s, i, j, k, b) \
	(s)[(i)] = (uchar_t)(0xE0U \| ((uint32_t)(b) & 0xF000U) >> 12); \
	(s)[(j)] = (uchar_t)(0x80U \| ((uint32_t)(b) & 0x0FC0U) >> 6); \
	(s)[(k)] = (uchar_t)(0x80U \| ((uint32_t)(b) & 0x003FU));

	#define U8_HANGUL_JAMO_L(u) \
	((u) >= U8_HANGUL_JAMO_L_FIRST && (u) <= U8_HANGUL_JAMO_L_LAST)

	#define U8_HANGUL_JAMO_V(u) \
	((u) >= U8_HANGUL_JAMO_V_FIRST && (u) <= U8_HANGUL_JAMO_V_LAST)

	#define U8_HANGUL_JAMO_T(u) \
	((u) > U8_HANGUL_JAMO_T_FIRST && (u) <= U8_HANGUL_JAMO_T_LAST)

	#define U8_HANGUL_JAMO(u) \
	((u) >= U8_HANGUL_JAMO_L_FIRST && (u) <= U8_HANGUL_JAMO_T_LAST)

	#define U8_HANGUL_SYLLABLE(u) \
	((u) >= U8_HANGUL_SYL_FIRST && (u) <= U8_HANGUL_SYL_LAST)

	#define U8_HANGUL_COMPOSABLE_L_V(s, u) \
	((s) == U8_STATE_HANGUL_L && U8_HANGUL_JAMO_V((u)))

	#define U8_HANGUL_COMPOSABLE_LV_T(s, u) \
	((s) == U8_STATE_HANGUL_LV && U8_HANGUL_JAMO_T((u)))

	/* The types of decomposition mappings. */
	#define U8_DECOMP_BOTH (0xF5U)
	#define U8_DECOMP_CANONICAL (0xF6U)

	/* The indicator for 16-bit table. */
	#define U8_16BIT_TABLE_INDICATOR (0x8000U)

	/* The following are some convenience macros. */
	#define U8_PUT_3BYTES_INTO_UTF32(u, b1, b2, b3) \
	(u) = ((((uint32_t)(b1) & 0x0F) << 12) \| \
	(((uint32_t)(b2) & 0x3F) << 6) \| \
	((uint32_t)(b3) & 0x3F));

	#define U8_SIMPLE_SWAP(a, b, t) \
	(t) = (a); \
	(a) = (b); \
	(b) = (t);

	#define U8_ASCII_TOUPPER(c) \
	(((c) >= 'a' && (c) <= 'z') ? (c) - 'a' + 'A' : (c))

	#define U8_ASCII_TOLOWER(c) \
	(((c) >= 'A' && (c) <= 'Z') ? (c) - 'A' + 'a' : (c))

	#define U8_ISASCII(c) (((uchar_t)(c)) < 0x80U)
	/*
	* The following macro assumes that the two characters that are to be
	* swapped are adjacent to each other and 'a' comes before 'b'.
	*
	* If the assumptions are not met, then, the macro will fail.
	*/
	#define U8_SWAP_COMB_MARKS(a, b) \
	for (k = 0; k < disp[(a)]; k++) \
	u8t[k] = u8s[start[(a)] + k]; \
	for (k = 0; k < disp[(b)]; k++) \
	u8s[start[(a)] + k] = u8s[start[(b)] + k]; \
	start[(b)] = start[(a)] + disp[(b)]; \
	for (k = 0; k < disp[(a)]; k++) \
	u8s[start[(b)] + k] = u8t[k]; \
	U8_SIMPLE_SWAP(comb_class[(a)], comb_class[(b)], tc); \
	U8_SIMPLE_SWAP(disp[(a)], disp[(b)], tc);

	/* The possible states during normalization. */
	typedef enum {
	U8_STATE_START = 0,
	U8_STATE_HANGUL_L = 1,
	U8_STATE_HANGUL_LV = 2,
	U8_STATE_HANGUL_LVT = 3,
	U8_STATE_HANGUL_V = 4,
	U8_STATE_HANGUL_T = 5,
	U8_STATE_COMBINING_MARK = 6
	} u8_normalization_states_t;

	/*
	* The three vectors at below are used to check bytes of a given UTF-8
	* character are valid and not containing any malformed byte values.
	*
	* We used to have a quite relaxed UTF-8 binary representation but then there
	* was some security related issues and so the Unicode Consortium defined
	* and announced the UTF-8 Corrigendum at Unicode 3.1 and then refined it
	* one more time at the Unicode 3.2. The following three tables are based on
	* that.
	*/

	#define U8_ILLEGAL_NEXT_BYTE_COMMON(c) ((c) < 0x80 \|\| (c) > 0xBF)

	#define I_ U8_ILLEGAL_CHAR
	#define O_ U8_OUT_OF_RANGE_CHAR

	const int8_t u8_number_of_bytes[0x100] = {
	1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
	1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
	1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
	1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
	1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
	1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
	1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
	1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,

	/* 80 81 82 83 84 85 86 87 88 89 8A 8B 8C 8D 8E 8F */
	I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_,

	/* 90 91 92 93 94 95 96 97 98 99 9A 9B 9C 9D 9E 9F */
	I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_,

	/* A0 A1 A2 A3 A4 A5 A6 A7 A8 A9 AA AB AC AD AE AF */
	I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_,

	/* B0 B1 B2 B3 B4 B5 B6 B7 B8 B9 BA BB BC BD BE BF */
	I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_, I_,

	/* C0 C1 C2 C3 C4 C5 C6 C7 C8 C9 CA CB CC CD CE CF */
	I_, I_, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,

	/* D0 D1 D2 D3 D4 D5 D6 D7 D8 D9 DA DB DC DD DE DF */
	2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,

	/* E0 E1 E2 E3 E4 E5 E6 E7 E8 E9 EA EB EC ED EE EF */
	3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,

	/* F0 F1 F2 F3 F4 F5 F6 F7 F8 F9 FA FB FC FD FE FF */
	4, 4, 4, 4, 4, O_, O_, O_, O_, O_, O_, O_, O_, O_, O_, O_,
	};

	#undef I_
	#undef O_

	const uint8_t u8_valid_min_2nd_byte[0x100] = {
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	/* C0 C1 C2 C3 C4 C5 C6 C7 */
	0, 0, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80,
	/* C8 C9 CA CB CC CD CE CF */
	0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80,
	/* D0 D1 D2 D3 D4 D5 D6 D7 */
	0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80,
	/* D8 D9 DA DB DC DD DE DF */
	0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80,
	/* E0 E1 E2 E3 E4 E5 E6 E7 */
	0xa0, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80,
	/* E8 E9 EA EB EC ED EE EF */
	0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80,
	/* F0 F1 F2 F3 F4 F5 F6 F7 */
	0x90, 0x80, 0x80, 0x80, 0x80, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	};

	const uint8_t u8_valid_max_2nd_byte[0x100] = {
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	/* C0 C1 C2 C3 C4 C5 C6 C7 */
	0, 0, 0xbf, 0xbf, 0xbf, 0xbf, 0xbf, 0xbf,
	/* C8 C9 CA CB CC CD CE CF */
	0xbf, 0xbf, 0xbf, 0xbf, 0xbf, 0xbf, 0xbf, 0xbf,
	/* D0 D1 D2 D3 D4 D5 D6 D7 */
	0xbf, 0xbf, 0xbf, 0xbf, 0xbf, 0xbf, 0xbf, 0xbf,
	/* D8 D9 DA DB DC DD DE DF */
	0xbf, 0xbf, 0xbf, 0xbf, 0xbf, 0xbf, 0xbf, 0xbf,
	/* E0 E1 E2 E3 E4 E5 E6 E7 */
	0xbf, 0xbf, 0xbf, 0xbf, 0xbf, 0xbf, 0xbf, 0xbf,
	/* E8 E9 EA EB EC ED EE EF */
	0xbf, 0xbf, 0xbf, 0xbf, 0xbf, 0x9f, 0xbf, 0xbf,
	/* F0 F1 F2 F3 F4 F5 F6 F7 */
	0xbf, 0xbf, 0xbf, 0xbf, 0x8f, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	};


	/*
	* The u8_validate() validates on the given UTF-8 character string and
	* calculate the byte length. It is quite similar to mblen(3C) except that
	* this will validate against the list of characters if required and
	* specific to UTF-8 and Unicode.
	*/
	int
	-u8_validate(char u8str, size_t n, char list, int flag, int errnum)
	+u8_validate(const char u8str, size_t n, char list, int flag, int errnum)
	{
	uchar_t *ib;
	uchar_t *ibtail;
	uchar_t **p;
	uchar_t *s1;
	uchar_t *s2;
	uchar_t f;
	int sz;
	size_t i;
	int ret_val;
	boolean_t second;
	boolean_t no_need_to_validate_entire;
	boolean_t check_additional;
	boolean_t validate_ucs2_range_only;

	if (! u8str)
	return (0);

	ib = (uchar_t *)u8str;
	ibtail = ib + n;

	ret_val = 0;

	no_need_to_validate_entire = ! (flag & U8_VALIDATE_ENTIRE);
	check_additional = flag & U8_VALIDATE_CHECK_ADDITIONAL;
	validate_ucs2_range_only = flag & U8_VALIDATE_UCS2_RANGE;

	while (ib < ibtail) {
	/*
	* The first byte of a UTF-8 character tells how many
	* bytes will follow for the character. If the first byte
	* is an illegal byte value or out of range value, we just
	* return -1 with an appropriate error number.
	*/
	sz = u8_number_of_bytes[*ib];
	if (sz == U8_ILLEGAL_CHAR) {
	*errnum = EILSEQ;
	return (-1);
	}

	if (sz == U8_OUT_OF_RANGE_CHAR \|\|
	(validate_ucs2_range_only && sz > U8_MAX_BYTES_UCS2)) {
	*errnum = ERANGE;
	return (-1);
	}

	/*
	* If we don't have enough bytes to check on, that's also
	* an error. As you can see, we give illegal byte sequence
	* checking higher priority then EINVAL cases.
	*/
	if ((ibtail - ib) < sz) {
	*errnum = EINVAL;
	return (-1);
	}

	if (sz == 1) {
	ib++;
	ret_val++;
	} else {
	/*
	* Check on the multi-byte UTF-8 character. For more
	* details on this, see comment added for the used
	* data structures at the beginning of the file.
	*/
	f = *ib++;
	ret_val++;
	second = B_TRUE;
	for (i = 1; i < sz; i++) {
	if (second) {
	if (*ib < u8_valid_min_2nd_byte[f] \|\|
	*ib > u8_valid_max_2nd_byte[f]) {
	*errnum = EILSEQ;
	return (-1);
	}
	second = B_FALSE;
	} else if (U8_ILLEGAL_NEXT_BYTE_COMMON(*ib)) {
	*errnum = EILSEQ;
	return (-1);
	}
	ib++;
	ret_val++;
	}
	}

	if (check_additional) {
	for (p = (uchar_t **)list, i = 0; p[i]; i++) {
	s1 = ib - sz;
	s2 = p[i];
	while (s1 < ib) {
	if (s1 != s2 \|\| *s2 == '\0')
	break;
	s1++;
	s2++;
	}

	if (s1 >= ib && *s2 == '\0') {
	*errnum = EBADF;
	return (-1);
	}
	}
	}

	if (no_need_to_validate_entire)
	break;
	}

	return (ret_val);
	}

	/*
	* The do_case_conv() looks at the mapping tables and returns found
	* bytes if any. If not found, the input bytes are returned. The function
	* always terminate the return bytes with a null character assuming that
	* there are plenty of room to do so.
	*
	* The case conversions are simple case conversions mapping a character to
	* another character as specified in the Unicode data. The byte size of
	* the mapped character could be different from that of the input character.
	*
	* The return value is the byte length of the returned character excluding
	* the terminating null byte.
	*/
	static size_t
	do_case_conv(int uv, uchar_t u8s, uchar_t s, int sz, boolean_t is_it_toupper)
	{
	size_t i;
	uint16_t b1 = 0;
	uint16_t b2 = 0;
	uint16_t b3 = 0;
	uint16_t b3_tbl;
	uint16_t b3_base;
	uint16_t b4 = 0;
	size_t start_id;
	size_t end_id;

	/*
	* At this point, the only possible values for sz are 2, 3, and 4.
	* The u8s should point to a vector that is well beyond the size of
	* 5 bytes.
	*/
	if (sz == 2) {
	b3 = u8s[0] = s[0];
	b4 = u8s[1] = s[1];
	} else if (sz == 3) {
	b2 = u8s[0] = s[0];
	b3 = u8s[1] = s[1];
	b4 = u8s[2] = s[2];
	} else if (sz == 4) {
	b1 = u8s[0] = s[0];
	b2 = u8s[1] = s[1];
	b3 = u8s[2] = s[2];
	b4 = u8s[3] = s[3];
	} else {
	/* This is not possible but just in case as a fallback. */
	if (is_it_toupper)
	u8s = U8_ASCII_TOUPPER(s);
	else
	u8s = U8_ASCII_TOLOWER(s);
	u8s[1] = '\0';

	return (1);
	}
	u8s[sz] = '\0';

	/*
	* Let's find out if we have a corresponding character.
	*/
	b1 = u8_common_b1_tbl[uv][b1];
	if (b1 == U8_TBL_ELEMENT_NOT_DEF)
	return ((size_t)sz);

	b2 = u8_case_common_b2_tbl[uv][b1][b2];
	if (b2 == U8_TBL_ELEMENT_NOT_DEF)
	return ((size_t)sz);

	if (is_it_toupper) {
	b3_tbl = u8_toupper_b3_tbl[uv][b2][b3].tbl_id;
	if (b3_tbl == U8_TBL_ELEMENT_NOT_DEF)
	return ((size_t)sz);

	start_id = u8_toupper_b4_tbl[uv][b3_tbl][b4];
	end_id = u8_toupper_b4_tbl[uv][b3_tbl][b4 + 1];

	/* Either there is no match or an error at the table. */
	if (start_id >= end_id \|\| (end_id - start_id) > U8_MB_CUR_MAX)
	return ((size_t)sz);

	b3_base = u8_toupper_b3_tbl[uv][b2][b3].base;

	for (i = 0; start_id < end_id; start_id++)
	u8s[i++] = u8_toupper_final_tbl[uv][b3_base + start_id];
	} else {
	b3_tbl = u8_tolower_b3_tbl[uv][b2][b3].tbl_id;
	if (b3_tbl == U8_TBL_ELEMENT_NOT_DEF)
	return ((size_t)sz);

	start_id = u8_tolower_b4_tbl[uv][b3_tbl][b4];
	end_id = u8_tolower_b4_tbl[uv][b3_tbl][b4 + 1];

	if (start_id >= end_id \|\| (end_id - start_id) > U8_MB_CUR_MAX)
	return ((size_t)sz);

	b3_base = u8_tolower_b3_tbl[uv][b2][b3].base;

	for (i = 0; start_id < end_id; start_id++)
	u8s[i++] = u8_tolower_final_tbl[uv][b3_base + start_id];
	}

	/*
	* If i is still zero, that means there is no corresponding character.
	*/
	if (i == 0)
	return ((size_t)sz);

	u8s[i] = '\0';

	return (i);
	}

	/*
	* The do_case_compare() function compares the two input strings, s1 and s2,
	* one character at a time doing case conversions if applicable and return
	* the comparison result as like strcmp().
	*
	* Since, in empirical sense, most of text data are 7-bit ASCII characters,
	* we treat the 7-bit ASCII characters as a special case trying to yield
	* faster processing time.
	*/
	static int
	do_case_compare(size_t uv, uchar_t s1, uchar_t s2, size_t n1,
	size_t n2, boolean_t is_it_toupper, int *errnum)
	{
	int f;
	int sz1;
	int sz2;
	size_t j;
	size_t i1;
	size_t i2;
	uchar_t u8s1[U8_MB_CUR_MAX + 1];
	uchar_t u8s2[U8_MB_CUR_MAX + 1];

	i1 = i2 = 0;
	while (i1 < n1 && i2 < n2) {
	/*
	* Find out what would be the byte length for this UTF-8
	* character at string s1 and also find out if this is
	* an illegal start byte or not and if so, issue a proper
	* error number and yet treat this byte as a character.
	*/
	sz1 = u8_number_of_bytes[*s1];
	if (sz1 < 0) {
	*errnum = EILSEQ;
	sz1 = 1;
	}

	/*
	* For 7-bit ASCII characters mainly, we do a quick case
	* conversion right at here.
	*
	* If we don't have enough bytes for this character, issue
	* an EINVAL error and use what are available.
	*
	* If we have enough bytes, find out if there is
	* a corresponding uppercase character and if so, copy over
	* the bytes for a comparison later. If there is no
	* corresponding uppercase character, then, use what we have
	* for the comparison.
	*/
	if (sz1 == 1) {
	if (is_it_toupper)
	u8s1[0] = U8_ASCII_TOUPPER(*s1);
	else
	u8s1[0] = U8_ASCII_TOLOWER(*s1);
	s1++;
	u8s1[1] = '\0';
	} else if ((i1 + sz1) > n1) {
	*errnum = EINVAL;
	for (j = 0; (i1 + j) < n1; )
	u8s1[j++] = *s1++;
	u8s1[j] = '\0';
	} else {
	(void) do_case_conv(uv, u8s1, s1, sz1, is_it_toupper);
	s1 += sz1;
	}

	/* Do the same for the string s2. */
	sz2 = u8_number_of_bytes[*s2];
	if (sz2 < 0) {
	*errnum = EILSEQ;
	sz2 = 1;
	}

	if (sz2 == 1) {
	if (is_it_toupper)
	u8s2[0] = U8_ASCII_TOUPPER(*s2);
	else
	u8s2[0] = U8_ASCII_TOLOWER(*s2);
	s2++;
	u8s2[1] = '\0';
	} else if ((i2 + sz2) > n2) {
	*errnum = EINVAL;
	for (j = 0; (i2 + j) < n2; )
	u8s2[j++] = *s2++;
	u8s2[j] = '\0';
	} else {
	(void) do_case_conv(uv, u8s2, s2, sz2, is_it_toupper);
	s2 += sz2;
	}

	/* Now compare the two characters. */
	if (sz1 == 1 && sz2 == 1) {
	if (u8s1 > u8s2)
	return (1);
	if (u8s1 < u8s2)
	return (-1);
	} else {
	f = strcmp((const char )u8s1, (const char )u8s2);
	if (f != 0)
	return (f);
	}

	/*
	* They were the same. Let's move on to the next
	* characters then.
	*/
	i1 += sz1;
	i2 += sz2;
	}

	/*
	* We compared until the end of either or both strings.
	*
	* If we reached to or went over the ends for the both, that means
	* they are the same.
	*
	* If we reached only one of the two ends, that means the other string
	* has something which then the fact can be used to determine
	* the return value.
	*/
	if (i1 >= n1) {
	if (i2 >= n2)
	return (0);
	return (-1);
	}
	return (1);
	}

	/*
	* The combining_class() function checks on the given bytes and find out
	* the corresponding Unicode combining class value. The return value 0 means
	* it is a Starter. Any illegal UTF-8 character will also be treated as
	* a Starter.
	*/
	static uchar_t
	combining_class(size_t uv, uchar_t *s, size_t sz)
	{
	uint16_t b1 = 0;
	uint16_t b2 = 0;
	uint16_t b3 = 0;
	uint16_t b4 = 0;

	if (sz == 1 \|\| sz > 4)
	return (0);

	if (sz == 2) {
	b3 = s[0];
	b4 = s[1];
	} else if (sz == 3) {
	b2 = s[0];
	b3 = s[1];
	b4 = s[2];
	} else if (sz == 4) {
	b1 = s[0];
	b2 = s[1];
	b3 = s[2];
	b4 = s[3];
	}

	b1 = u8_common_b1_tbl[uv][b1];
	if (b1 == U8_TBL_ELEMENT_NOT_DEF)
	return (0);

	b2 = u8_combining_class_b2_tbl[uv][b1][b2];
	if (b2 == U8_TBL_ELEMENT_NOT_DEF)
	return (0);

	b3 = u8_combining_class_b3_tbl[uv][b2][b3];
	if (b3 == U8_TBL_ELEMENT_NOT_DEF)
	return (0);

	return (u8_combining_class_b4_tbl[uv][b3][b4]);
	}

	/*
	* The do_decomp() function finds out a matching decomposition if any
	* and return. If there is no match, the input bytes are copied and returned.
	* The function also checks if there is a Hangul, decomposes it if necessary
	* and returns.
	*
	* To save time, a single byte 7-bit ASCII character should be handled by
	* the caller.
	*
	* The function returns the number of bytes returned sans always terminating
	* the null byte. It will also return a state that will tell if there was
	* a Hangul character decomposed which then will be used by the caller.
	*/
	static size_t
	do_decomp(size_t uv, uchar_t u8s, uchar_t s, int sz,
	boolean_t canonical_decomposition, u8_normalization_states_t *state)
	{
	uint16_t b1 = 0;
	uint16_t b2 = 0;
	uint16_t b3 = 0;
	uint16_t b3_tbl;
	uint16_t b3_base;
	uint16_t b4 = 0;
	size_t start_id;
	size_t end_id;
	size_t i;
	uint32_t u1;

	if (sz == 2) {
	b3 = u8s[0] = s[0];
	b4 = u8s[1] = s[1];
	u8s[2] = '\0';
	} else if (sz == 3) {
	/* Convert it to a Unicode scalar value. */
	U8_PUT_3BYTES_INTO_UTF32(u1, s[0], s[1], s[2]);

	/*
	* If this is a Hangul syllable, we decompose it into
	* a leading consonant, a vowel, and an optional trailing
	* consonant and then return.
	*/
	if (U8_HANGUL_SYLLABLE(u1)) {
	u1 -= U8_HANGUL_SYL_FIRST;

	b1 = U8_HANGUL_JAMO_L_FIRST + u1 / U8_HANGUL_VT_COUNT;
	b2 = U8_HANGUL_JAMO_V_FIRST + (u1 % U8_HANGUL_VT_COUNT)
	/ U8_HANGUL_T_COUNT;
	b3 = u1 % U8_HANGUL_T_COUNT;

	U8_SAVE_HANGUL_AS_UTF8(u8s, 0, 1, 2, b1);
	U8_SAVE_HANGUL_AS_UTF8(u8s, 3, 4, 5, b2);
	if (b3) {
	b3 += U8_HANGUL_JAMO_T_FIRST;
	U8_SAVE_HANGUL_AS_UTF8(u8s, 6, 7, 8, b3);

	u8s[9] = '\0';
	*state = U8_STATE_HANGUL_LVT;
	return (9);
	}

	u8s[6] = '\0';
	*state = U8_STATE_HANGUL_LV;
	return (6);
	}

	b2 = u8s[0] = s[0];
	b3 = u8s[1] = s[1];
	b4 = u8s[2] = s[2];
	u8s[3] = '\0';

	/*
	* If this is a Hangul Jamo, we know there is nothing
	* further that we can decompose.
	*/
	if (U8_HANGUL_JAMO_L(u1)) {
	*state = U8_STATE_HANGUL_L;
	return (3);
	}

	if (U8_HANGUL_JAMO_V(u1)) {
	if (*state == U8_STATE_HANGUL_L)
	*state = U8_STATE_HANGUL_LV;
	else
	*state = U8_STATE_HANGUL_V;
	return (3);
	}

	if (U8_HANGUL_JAMO_T(u1)) {
	if (*state == U8_STATE_HANGUL_LV)
	*state = U8_STATE_HANGUL_LVT;
	else
	*state = U8_STATE_HANGUL_T;
	return (3);
	}
	} else if (sz == 4) {
	b1 = u8s[0] = s[0];
	b2 = u8s[1] = s[1];
	b3 = u8s[2] = s[2];
	b4 = u8s[3] = s[3];
	u8s[4] = '\0';
	} else {
	/*
	* This is a fallback and should not happen if the function
	* was called properly.
	*/
	u8s[0] = s[0];
	u8s[1] = '\0';
	*state = U8_STATE_START;
	return (1);
	}

	/*
	* At this point, this routine does not know what it would get.
	* The caller should sort it out if the state isn't a Hangul one.
	*/
	*state = U8_STATE_START;

	/* Try to find matching decomposition mapping byte sequence. */
	b1 = u8_common_b1_tbl[uv][b1];
	if (b1 == U8_TBL_ELEMENT_NOT_DEF)
	return ((size_t)sz);

	b2 = u8_decomp_b2_tbl[uv][b1][b2];
	if (b2 == U8_TBL_ELEMENT_NOT_DEF)
	return ((size_t)sz);

	b3_tbl = u8_decomp_b3_tbl[uv][b2][b3].tbl_id;
	if (b3_tbl == U8_TBL_ELEMENT_NOT_DEF)
	return ((size_t)sz);

	/*
	* If b3_tbl is bigger than or equal to U8_16BIT_TABLE_INDICATOR
	* which is 0x8000, this means we couldn't fit the mappings into
	* the cardinality of a unsigned byte.
	*/
	if (b3_tbl >= U8_16BIT_TABLE_INDICATOR) {
	b3_tbl -= U8_16BIT_TABLE_INDICATOR;
	start_id = u8_decomp_b4_16bit_tbl[uv][b3_tbl][b4];
	end_id = u8_decomp_b4_16bit_tbl[uv][b3_tbl][b4 + 1];
	} else {
	start_id = u8_decomp_b4_tbl[uv][b3_tbl][b4];
	end_id = u8_decomp_b4_tbl[uv][b3_tbl][b4 + 1];
	}

	/* This also means there wasn't any matching decomposition. */
	if (start_id >= end_id)
	return ((size_t)sz);

	/*
	* The final table for decomposition mappings has three types of
	* byte sequences depending on whether a mapping is for compatibility
	* decomposition, canonical decomposition, or both like the following:
	*
	* (1) Compatibility decomposition mappings:
	*
	* +---+---+-...-+---+
	* \| B0\| B1\| ... \| Bm\|
	* +---+---+-...-+---+
	*
	* The first byte, B0, is always less then 0xF5 (U8_DECOMP_BOTH).
	*
	* (2) Canonical decomposition mappings:
	*
	* +---+---+---+-...-+---+
	* \| T \| b0\| b1\| ... \| bn\|
	* +---+---+---+-...-+---+
	*
	* where the first byte, T, is 0xF6 (U8_DECOMP_CANONICAL).
	*
	* (3) Both mappings:
	*
	* +---+---+---+---+-...-+---+---+---+-...-+---+
	* \| T \| D \| b0\| b1\| ... \| bn\| B0\| B1\| ... \| Bm\|
	* +---+---+---+---+-...-+---+---+---+-...-+---+
	*
	* where T is 0xF5 (U8_DECOMP_BOTH) and D is a displacement
	* byte, b0 to bn are canonical mapping bytes and B0 to Bm are
	* compatibility mapping bytes.
	*
	* Note that compatibility decomposition means doing recursive
	* decompositions using both compatibility decomposition mappings and
	* canonical decomposition mappings. On the other hand, canonical
	* decomposition means doing recursive decompositions using only
	* canonical decomposition mappings. Since the table we have has gone
	* through the recursions already, we do not need to do so during
	* runtime, i.e., the table has been completely flattened out
	* already.
	*/

	b3_base = u8_decomp_b3_tbl[uv][b2][b3].base;

	/* Get the type, T, of the byte sequence. */
	b1 = u8_decomp_final_tbl[uv][b3_base + start_id];

	/*
	* If necessary, adjust start_id, end_id, or both. Note that if
	* this is compatibility decomposition mapping, there is no
	* adjustment.
	*/
	if (canonical_decomposition) {
	/* Is the mapping only for compatibility decomposition? */
	if (b1 < U8_DECOMP_BOTH)
	return ((size_t)sz);

	start_id++;

	if (b1 == U8_DECOMP_BOTH) {
	end_id = start_id +
	u8_decomp_final_tbl[uv][b3_base + start_id];
	start_id++;
	}
	} else {
	/*
	* Unless this is a compatibility decomposition mapping,
	* we adjust the start_id.
	*/
	if (b1 == U8_DECOMP_BOTH) {
	start_id++;
	start_id += u8_decomp_final_tbl[uv][b3_base + start_id];
	} else if (b1 == U8_DECOMP_CANONICAL) {
	start_id++;
	}
	}

	for (i = 0; start_id < end_id; start_id++)
	u8s[i++] = u8_decomp_final_tbl[uv][b3_base + start_id];
	u8s[i] = '\0';

	return (i);
	}

	/*
	* The find_composition_start() function uses the character bytes given and
	* find out the matching composition mappings if any and return the address
	* to the composition mappings as explained in the do_composition().
	*/
	static uchar_t *
	find_composition_start(size_t uv, uchar_t *s, size_t sz)
	{
	uint16_t b1 = 0;
	uint16_t b2 = 0;
	uint16_t b3 = 0;
	uint16_t b3_tbl;
	uint16_t b3_base;
	uint16_t b4 = 0;
	size_t start_id;
	size_t end_id;

	if (sz == 1) {
	b4 = s[0];
	} else if (sz == 2) {
	b3 = s[0];
	b4 = s[1];
	} else if (sz == 3) {
	b2 = s[0];
	b3 = s[1];
	b4 = s[2];
	} else if (sz == 4) {
	b1 = s[0];
	b2 = s[1];
	b3 = s[2];
	b4 = s[3];
	} else {
	/*
	* This is a fallback and should not happen if the function
	* was called properly.
	*/
	return (NULL);
	}

	b1 = u8_composition_b1_tbl[uv][b1];
	if (b1 == U8_TBL_ELEMENT_NOT_DEF)
	return (NULL);

	b2 = u8_composition_b2_tbl[uv][b1][b2];
	if (b2 == U8_TBL_ELEMENT_NOT_DEF)
	return (NULL);

	b3_tbl = u8_composition_b3_tbl[uv][b2][b3].tbl_id;
	if (b3_tbl == U8_TBL_ELEMENT_NOT_DEF)
	return (NULL);

	if (b3_tbl >= U8_16BIT_TABLE_INDICATOR) {
	b3_tbl -= U8_16BIT_TABLE_INDICATOR;
	start_id = u8_composition_b4_16bit_tbl[uv][b3_tbl][b4];
	end_id = u8_composition_b4_16bit_tbl[uv][b3_tbl][b4 + 1];
	} else {
	start_id = u8_composition_b4_tbl[uv][b3_tbl][b4];
	end_id = u8_composition_b4_tbl[uv][b3_tbl][b4 + 1];
	}

	if (start_id >= end_id)
	return (NULL);

	b3_base = u8_composition_b3_tbl[uv][b2][b3].base;

	return ((uchar_t *)&(u8_composition_final_tbl[uv][b3_base + start_id]));
	}

	/*
	* The blocked() function checks on the combining class values of previous
	* characters in this sequence and return whether it is blocked or not.
	*/
	static boolean_t
	blocked(uchar_t *comb_class, size_t last)
	{
	uchar_t my_comb_class;
	size_t i;

	my_comb_class = comb_class[last];
	for (i = 1; i < last; i++)
	if (comb_class[i] >= my_comb_class \|\|
	comb_class[i] == U8_COMBINING_CLASS_STARTER)
	return (B_TRUE);

	return (B_FALSE);
	}

	/*
	* The do_composition() reads the character string pointed by 's' and
	* do necessary canonical composition and then copy over the result back to
	* the 's'.
	*
	* The input argument 's' cannot contain more than 32 characters.
	*/
	static size_t
	do_composition(size_t uv, uchar_t s, uchar_t comb_class, uchar_t *start,
	uchar_t disp, size_t last, uchar_t os, uchar_t oslast)
	{
	uchar_t t[U8_STREAM_SAFE_TEXT_MAX + 1];
	uchar_t tc[U8_MB_CUR_MAX] = { '\0' };
	uint8_t saved_marks[U8_MAX_CHARS_A_SEQ];
	size_t saved_marks_count;
	uchar_t *p;
	uchar_t *saved_p;
	uchar_t *q;
	size_t i;
	size_t saved_i;
	size_t j;
	size_t k;
	size_t l;
	size_t C;
	size_t saved_l;
	size_t size;
	uint32_t u1;
	uint32_t u2;
	boolean_t match_not_found = B_TRUE;

	/*
	* This should never happen unless the callers are doing some strange
	* and unexpected things.
	*
	* The "last" is the index pointing to the last character not last + 1.
	*/
	if (last >= U8_MAX_CHARS_A_SEQ)
	last = U8_UPPER_LIMIT_IN_A_SEQ;

	for (i = l = 0; i <= last; i++) {
	/*
	* The last or any non-Starters at the beginning, we don't
	* have any chance to do composition and so we just copy them
	* to the temporary buffer.
	*/
	if (i >= last \|\| comb_class[i] != U8_COMBINING_CLASS_STARTER) {
	SAVE_THE_CHAR:
	p = s + start[i];
	size = disp[i];
	for (k = 0; k < size; k++)
	t[l++] = *p++;
	continue;
	}

	/*
	* If this could be a start of Hangul Jamos, then, we try to
	* conjoin them.
	*/
	if (s[start[i]] == U8_HANGUL_JAMO_1ST_BYTE) {
	U8_PUT_3BYTES_INTO_UTF32(u1, s[start[i]],
	s[start[i] + 1], s[start[i] + 2]);
	U8_PUT_3BYTES_INTO_UTF32(u2, s[start[i] + 3],
	s[start[i] + 4], s[start[i] + 5]);

	if (U8_HANGUL_JAMO_L(u1) && U8_HANGUL_JAMO_V(u2)) {
	u1 -= U8_HANGUL_JAMO_L_FIRST;
	u2 -= U8_HANGUL_JAMO_V_FIRST;
	u1 = U8_HANGUL_SYL_FIRST +
	(u1 * U8_HANGUL_V_COUNT + u2) *
	U8_HANGUL_T_COUNT;

	i += 2;
	if (i <= last) {
	U8_PUT_3BYTES_INTO_UTF32(u2,
	s[start[i]], s[start[i] + 1],
	s[start[i] + 2]);

	if (U8_HANGUL_JAMO_T(u2)) {
	u1 += u2 -
	U8_HANGUL_JAMO_T_FIRST;
	i++;
	}
	}

	U8_SAVE_HANGUL_AS_UTF8(t + l, 0, 1, 2, u1);
	i--;
	l += 3;
	continue;
	}
	}

	/*
	* Let's then find out if this Starter has composition
	* mapping.
	*/
	p = find_composition_start(uv, s + start[i], disp[i]);
	if (p == NULL)
	goto SAVE_THE_CHAR;

	/*
	* We have a Starter with composition mapping and the next
	* character is a non-Starter. Let's try to find out if
	* we can do composition.
	*/

	saved_p = p;
	saved_i = i;
	saved_l = l;
	saved_marks_count = 0;

	TRY_THE_NEXT_MARK:
	q = s + start[++i];
	size = disp[i];

	/*
	* The next for() loop compares the non-Starter pointed by
	* 'q' with the possible (joinable) characters pointed by 'p'.
	*
	* The composition final table entry pointed by the 'p'
	* looks like the following:
	*
	* +---+---+---+-...-+---+---+---+---+-...-+---+---+
	* \| C \| b0\| b2\| ... \| bn\| F \| B0\| B1\| ... \| Bm\| F \|
	* +---+---+---+-...-+---+---+---+---+-...-+---+---+
	*
	* where C is the count byte indicating the number of
	* mapping pairs where each pair would be look like
	* (b0-bn F, B0-Bm F). The b0-bn are the bytes of the second
	* character of a canonical decomposition and the B0-Bm are
	* the bytes of a matching composite character. The F is
	* a filler byte after each character as the separator.
	*/

	match_not_found = B_TRUE;

	for (C = *p++; C > 0; C--) {
	for (k = 0; k < size; p++, k++)
	if (*p != q[k])
	break;

	/* Have we found it? */
	if (k >= size && *p == U8_TBL_ELEMENT_FILLER) {
	match_not_found = B_FALSE;

	l = saved_l;

	while (*++p != U8_TBL_ELEMENT_FILLER)
	t[l++] = *p;

	break;
	}

	/* We didn't find; skip to the next pair. */
	if (*p != U8_TBL_ELEMENT_FILLER)
	while (*++p != U8_TBL_ELEMENT_FILLER)
	;
	while (*++p != U8_TBL_ELEMENT_FILLER)
	;
	p++;
	}

	/*
	* If there was no match, we will need to save the combining
	* mark for later appending. After that, if the next one
	* is a non-Starter and not blocked, then, we try once
	* again to do composition with the next non-Starter.
	*
	* If there was no match and this was a Starter, then,
	* this is a new start.
	*
	* If there was a match and a composition done and we have
	* more to check on, then, we retrieve a new composition final
	* table entry for the composite and then try to do the
	* composition again.
	*/

	if (match_not_found) {
	if (comb_class[i] == U8_COMBINING_CLASS_STARTER) {
	i--;
	goto SAVE_THE_CHAR;
	}

	saved_marks[saved_marks_count++] = i;
	}

	if (saved_l == l) {
	while (i < last) {
	if (blocked(comb_class, i + 1))
	saved_marks[saved_marks_count++] = ++i;
	else
	break;
	}
	if (i < last) {
	p = saved_p;
	goto TRY_THE_NEXT_MARK;
	}
	} else if (i < last) {
	p = find_composition_start(uv, t + saved_l,
	l - saved_l);
	if (p != NULL) {
	saved_p = p;
	goto TRY_THE_NEXT_MARK;
	}
	}

	/*
	* There is no more composition possible.
	*
	* If there was no composition what so ever then we copy
	* over the original Starter and then append any non-Starters
	* remaining at the target string sequentially after that.
	*/

	if (saved_l == l) {
	p = s + start[saved_i];
	size = disp[saved_i];
	for (j = 0; j < size; j++)
	t[l++] = *p++;
	}

	for (k = 0; k < saved_marks_count; k++) {
	p = s + start[saved_marks[k]];
	size = disp[saved_marks[k]];
	for (j = 0; j < size; j++)
	t[l++] = *p++;
	}
	}

	/*
	* If the last character is a Starter and if we have a character
	* (possibly another Starter) that can be turned into a composite,
	* we do so and we do so until there is no more of composition
	* possible.
	*/
	if (comb_class[last] == U8_COMBINING_CLASS_STARTER) {
	p = *os;
	saved_l = l - disp[last];

	while (p < oslast) {
	size = u8_number_of_bytes[*p];
	if (size <= 1 \|\| (p + size) > oslast)
	break;

	saved_p = p;

	for (i = 0; i < size; i++)
	tc[i] = *p++;

	q = find_composition_start(uv, t + saved_l,
	l - saved_l);
	if (q == NULL) {
	p = saved_p;
	break;
	}

	match_not_found = B_TRUE;

	for (C = *q++; C > 0; C--) {
	for (k = 0; k < size; q++, k++)
	if (*q != tc[k])
	break;

	if (k >= size && *q == U8_TBL_ELEMENT_FILLER) {
	match_not_found = B_FALSE;

	l = saved_l;

	while (*++q != U8_TBL_ELEMENT_FILLER) {
	/*
	* This is practically
	* impossible but we don't
	* want to take any chances.
	*/
	if (l >=
	U8_STREAM_SAFE_TEXT_MAX) {
	p = saved_p;
	goto SAFE_RETURN;
	}
	t[l++] = *q;
	}

	break;
	}

	if (*q != U8_TBL_ELEMENT_FILLER)
	while (*++q != U8_TBL_ELEMENT_FILLER)
	;
	while (*++q != U8_TBL_ELEMENT_FILLER)
	;
	q++;
	}

	if (match_not_found) {
	p = saved_p;
	break;
	}
	}
	SAFE_RETURN:
	*os = p;
	}

	/*
	* Now we copy over the temporary string to the target string.
	* Since composition always reduces the number of characters or
	* the number of characters stay, we don't need to worry about
	* the buffer overflow here.
	*/
	for (i = 0; i < l; i++)
	s[i] = t[i];
	s[l] = '\0';

	return (l);
	}

	/*
	* The collect_a_seq() function checks on the given string s, collect
	* a sequence of characters at u8s, and return the sequence. While it collects
	* a sequence, it also applies case conversion, canonical or compatibility
	* decomposition, canonical decomposition, or some or all of them and
	* in that order.
	*
	* The collected sequence cannot be bigger than 32 characters since if
	* it is having more than 31 characters, the sequence will be terminated
	* with a U+034F COMBINING GRAPHEME JOINER (CGJ) character and turned into
	* a Stream-Safe Text. The collected sequence is always terminated with
	* a null byte and the return value is the byte length of the sequence
	* including 0. The return value does not include the terminating
	* null byte.
	*/
	static size_t
	collect_a_seq(size_t uv, uchar_t u8s, uchar_t source, uchar_t slast,
	boolean_t is_it_toupper, boolean_t is_it_tolower,
	boolean_t canonical_decomposition, boolean_t compatibility_decomposition,
	boolean_t canonical_composition,
	int errnum, u8_normalization_states_t state)
	{
	uchar_t *s;
	int sz;
	int saved_sz;
	size_t i;
	size_t j;
	size_t k;
	size_t l;
	uchar_t comb_class[U8_MAX_CHARS_A_SEQ];
	uchar_t disp[U8_MAX_CHARS_A_SEQ];
	uchar_t start[U8_MAX_CHARS_A_SEQ];
	uchar_t u8t[U8_MB_CUR_MAX] = { '\0' };
	uchar_t uts[U8_STREAM_SAFE_TEXT_MAX + 1];
	uchar_t tc;
	size_t last;
	size_t saved_last;
	uint32_t u1;

	/*
	* Save the source string pointer which we will return a changed
	* pointer if we do processing.
	*/
	s = *source;

	/*
	* The following is a fallback for just in case callers are not
	* checking the string boundaries before the calling.
	*/
	if (s >= slast) {
	u8s[0] = '\0';

	return (0);
	}

	/*
	* As the first thing, let's collect a character and do case
	* conversion if necessary.
	*/

	sz = u8_number_of_bytes[*s];

	if (sz < 0) {
	*errnum = EILSEQ;

	u8s[0] = *s++;
	u8s[1] = '\0';

	*source = s;

	return (1);
	}

	if (sz == 1) {
	if (is_it_toupper)
	u8s[0] = U8_ASCII_TOUPPER(*s);
	else if (is_it_tolower)
	u8s[0] = U8_ASCII_TOLOWER(*s);
	else
	u8s[0] = *s;
	s++;
	u8s[1] = '\0';
	} else if ((s + sz) > slast) {
	*errnum = EINVAL;

	for (i = 0; s < slast; )
	u8s[i++] = *s++;
	u8s[i] = '\0';

	*source = s;

	return (i);
	} else {
	if (is_it_toupper \|\| is_it_tolower) {
	i = do_case_conv(uv, u8s, s, sz, is_it_toupper);
	s += sz;
	sz = i;
	} else {
	for (i = 0; i < sz; )
	u8s[i++] = *s++;
	u8s[i] = '\0';
	}
	}

	/*
	* And then canonical/compatibility decomposition followed by
	* an optional canonical composition. Please be noted that
	* canonical composition is done only when a decomposition is
	* done.
	*/
	if (canonical_decomposition \|\| compatibility_decomposition) {
	if (sz == 1) {
	*state = U8_STATE_START;

	saved_sz = 1;

	comb_class[0] = 0;
	start[0] = 0;
	disp[0] = 1;

	last = 1;
	} else {
	saved_sz = do_decomp(uv, u8s, u8s, sz,
	canonical_decomposition, state);

	last = 0;

	for (i = 0; i < saved_sz; ) {
	sz = u8_number_of_bytes[u8s[i]];

	comb_class[last] = combining_class(uv,
	u8s + i, sz);
	start[last] = i;
	disp[last] = sz;

	last++;
	i += sz;
	}

	/*
	* Decomposition yields various Hangul related
	* states but not on combining marks. We need to
	* find out at here by checking on the last
	* character.
	*/
	if (*state == U8_STATE_START) {
	if (comb_class[last - 1])
	*state = U8_STATE_COMBINING_MARK;
	}
	}

	saved_last = last;

	while (s < slast) {
	sz = u8_number_of_bytes[*s];

	/*
	* If this is an illegal character, an incomplete
	* character, or an 7-bit ASCII Starter character,
	* then we have collected a sequence; break and let
	* the next call deal with the two cases.
	*
	* Note that this is okay only if you are using this
	* function with a fixed length string, not on
	* a buffer with multiple calls of one chunk at a time.
	*/
	if (sz <= 1) {
	break;
	} else if ((s + sz) > slast) {
	break;
	} else {
	/*
	* If the previous character was a Hangul Jamo
	* and this character is a Hangul Jamo that
	* can be conjoined, we collect the Jamo.
	*/
	if (*s == U8_HANGUL_JAMO_1ST_BYTE) {
	U8_PUT_3BYTES_INTO_UTF32(u1,
	s, (s + 1), *(s + 2));

	if (U8_HANGUL_COMPOSABLE_L_V(*state,
	u1)) {
	i = 0;
	*state = U8_STATE_HANGUL_LV;
	goto COLLECT_A_HANGUL;
	}

	if (U8_HANGUL_COMPOSABLE_LV_T(*state,
	u1)) {
	i = 0;
	*state = U8_STATE_HANGUL_LVT;
	goto COLLECT_A_HANGUL;
	}
	}

	/*
	* Regardless of whatever it was, if this is
	* a Starter, we don't collect the character
	* since that's a new start and we will deal
	* with it at the next time.
	*/
	i = combining_class(uv, s, sz);
	if (i == U8_COMBINING_CLASS_STARTER)
	break;

	/*
	* We know the current character is a combining
	* mark. If the previous character wasn't
	* a Starter (not Hangul) or a combining mark,
	* then, we don't collect this combining mark.
	*/
	if (*state != U8_STATE_START &&
	*state != U8_STATE_COMBINING_MARK)
	break;

	*state = U8_STATE_COMBINING_MARK;
	COLLECT_A_HANGUL:
	/*
	* If we collected a Starter and combining
	* marks up to 30, i.e., total 31 characters,
	* then, we terminate this degenerately long
	* combining sequence with a U+034F COMBINING
	* GRAPHEME JOINER (CGJ) which is 0xCD 0x8F in
	* UTF-8 and turn this into a Stream-Safe
	* Text. This will be extremely rare but
	* possible.
	*
	* The following will also guarantee that
	* we are not writing more than 32 characters
	* plus a NULL at u8s[].
	*/
	if (last >= U8_UPPER_LIMIT_IN_A_SEQ) {
	TURN_STREAM_SAFE:
	*state = U8_STATE_START;
	comb_class[last] = 0;
	start[last] = saved_sz;
	disp[last] = 2;
	last++;

	u8s[saved_sz++] = 0xCD;
	u8s[saved_sz++] = 0x8F;

	break;
	}

	/*
	* Some combining marks also do decompose into
	* another combining mark or marks.
	*/
	if (*state == U8_STATE_COMBINING_MARK) {
	k = last;
	l = sz;
	i = do_decomp(uv, uts, s, sz,
	canonical_decomposition, state);
	for (j = 0; j < i; ) {
	sz = u8_number_of_bytes[uts[j]];

	comb_class[last] =
	combining_class(uv,
	uts + j, sz);
	start[last] = saved_sz + j;
	disp[last] = sz;

	last++;
	if (last >=
	U8_UPPER_LIMIT_IN_A_SEQ) {
	last = k;
	goto TURN_STREAM_SAFE;
	}
	j += sz;
	}

	*state = U8_STATE_COMBINING_MARK;
	sz = i;
	s += l;

	for (i = 0; i < sz; i++)
	u8s[saved_sz++] = uts[i];
	} else {
	comb_class[last] = i;
	start[last] = saved_sz;
	disp[last] = sz;
	last++;

	for (i = 0; i < sz; i++)
	u8s[saved_sz++] = *s++;
	}

	/*
	* If this is U+0345 COMBINING GREEK
	* YPOGEGRAMMENI (0xCD 0x85 in UTF-8), a.k.a.,
	* iota subscript, and need to be converted to
	* uppercase letter, convert it to U+0399 GREEK
	* CAPITAL LETTER IOTA (0xCE 0x99 in UTF-8),
	* i.e., convert to capital adscript form as
	* specified in the Unicode standard.
	*
	* This is the only special case of (ambiguous)
	* case conversion at combining marks and
	* probably the standard will never have
	* anything similar like this in future.
	*/
	if (is_it_toupper && sz >= 2 &&
	u8s[saved_sz - 2] == 0xCD &&
	u8s[saved_sz - 1] == 0x85) {
	u8s[saved_sz - 2] = 0xCE;
	u8s[saved_sz - 1] = 0x99;
	}
	}
	}

	/*
	* Let's try to ensure a canonical ordering for the collected
	* combining marks. We do this only if we have collected
	* at least one more non-Starter. (The decomposition mapping
	* data tables have fully (and recursively) expanded and
	* canonically ordered decompositions.)
	*
	* The U8_SWAP_COMB_MARKS() convenience macro has some
	* assumptions and we are meeting the assumptions.
	*/
	last--;
	if (last >= saved_last) {
	for (i = 0; i < last; i++)
	for (j = last; j > i; j--)
	if (comb_class[j] &&
	comb_class[j - 1] > comb_class[j]) {
	U8_SWAP_COMB_MARKS(j - 1, j);
	}
	}

	*source = s;

	if (! canonical_composition) {
	u8s[saved_sz] = '\0';
	return (saved_sz);
	}

	/*
	* Now do the canonical composition. Note that we do this
	* only after a canonical or compatibility decomposition to
	* finish up NFC or NFKC.
	*/
	sz = do_composition(uv, u8s, comb_class, start, disp, last,
	&s, slast);
	}

	*source = s;

	return ((size_t)sz);
	}

	/*
	* The do_norm_compare() function does string comparison based on Unicode
	* simple case mappings and Unicode Normalization definitions.
	*
	* It does so by collecting a sequence of character at a time and comparing
	* the collected sequences from the strings.
	*
	* The meanings on the return values are the same as the usual strcmp().
	*/
	static int
	do_norm_compare(size_t uv, uchar_t s1, uchar_t s2, size_t n1, size_t n2,
	int flag, int *errnum)
	{
	int result;
	size_t sz1;
	size_t sz2;
	uchar_t u8s1[U8_STREAM_SAFE_TEXT_MAX + 1];
	uchar_t u8s2[U8_STREAM_SAFE_TEXT_MAX + 1];
	uchar_t *s1last;
	uchar_t *s2last;
	boolean_t is_it_toupper;
	boolean_t is_it_tolower;
	boolean_t canonical_decomposition;
	boolean_t compatibility_decomposition;
	boolean_t canonical_composition;
	u8_normalization_states_t state;

	s1last = s1 + n1;
	s2last = s2 + n2;

	is_it_toupper = flag & U8_TEXTPREP_TOUPPER;
	is_it_tolower = flag & U8_TEXTPREP_TOLOWER;
	canonical_decomposition = flag & U8_CANON_DECOMP;
	compatibility_decomposition = flag & U8_COMPAT_DECOMP;
	canonical_composition = flag & U8_CANON_COMP;

	while (s1 < s1last && s2 < s2last) {
	/*
	* If the current character is a 7-bit ASCII and the last
	* character, or, if the current character and the next
	* character are both some 7-bit ASCII characters then
	* we treat the current character as a sequence.
	*
	* In any other cases, we need to call collect_a_seq().
	*/

	if (U8_ISASCII(*s1) && ((s1 + 1) >= s1last \|\|
	((s1 + 1) < s1last && U8_ISASCII(*(s1 + 1))))) {
	if (is_it_toupper)
	u8s1[0] = U8_ASCII_TOUPPER(*s1);
	else if (is_it_tolower)
	u8s1[0] = U8_ASCII_TOLOWER(*s1);
	else
	u8s1[0] = *s1;
	u8s1[1] = '\0';
	sz1 = 1;
	s1++;
	} else {
	state = U8_STATE_START;
	sz1 = collect_a_seq(uv, u8s1, &s1, s1last,
	is_it_toupper, is_it_tolower,
	canonical_decomposition,
	compatibility_decomposition,
	canonical_composition, errnum, &state);
	}

	if (U8_ISASCII(*s2) && ((s2 + 1) >= s2last \|\|
	((s2 + 1) < s2last && U8_ISASCII(*(s2 + 1))))) {
	if (is_it_toupper)
	u8s2[0] = U8_ASCII_TOUPPER(*s2);
	else if (is_it_tolower)
	u8s2[0] = U8_ASCII_TOLOWER(*s2);
	else
	u8s2[0] = *s2;
	u8s2[1] = '\0';
	sz2 = 1;
	s2++;
	} else {
	state = U8_STATE_START;
	sz2 = collect_a_seq(uv, u8s2, &s2, s2last,
	is_it_toupper, is_it_tolower,
	canonical_decomposition,
	compatibility_decomposition,
	canonical_composition, errnum, &state);
	}

	/*
	* Now compare the two characters. If they are the same,
	* we move on to the next character sequences.
	*/
	if (sz1 == 1 && sz2 == 1) {
	if (u8s1 > u8s2)
	return (1);
	if (u8s1 < u8s2)
	return (-1);
	} else {
	result = strcmp((const char )u8s1, (const char )u8s2);
	if (result != 0)
	return (result);
	}
	}

	/*
	* We compared until the end of either or both strings.
	*
	* If we reached to or went over the ends for the both, that means
	* they are the same.
	*
	* If we reached only one end, that means the other string has
	* something which then can be used to determine the return value.
	*/
	if (s1 >= s1last) {
	if (s2 >= s2last)
	return (0);
	return (-1);
	}
	return (1);
	}

	/*
	* The u8_strcmp() function compares two UTF-8 strings quite similar to
	* the strcmp(). For the comparison, however, Unicode Normalization specific
	* equivalency and Unicode simple case conversion mappings based equivalency
	* can be requested and checked against.
	*/
	int
	u8_strcmp(const char s1, const char s2, size_t n, int flag, size_t uv,
	int *errnum)
	{
	int f;
	size_t n1;
	size_t n2;

	*errnum = 0;

	/*
	* Check on the requested Unicode version, case conversion, and
	* normalization flag values.
	*/

	if (uv > U8_UNICODE_LATEST) {
	*errnum = ERANGE;
	uv = U8_UNICODE_LATEST;
	}

	if (flag == 0) {
	flag = U8_STRCMP_CS;
	} else {
	f = flag & (U8_STRCMP_CS \| U8_STRCMP_CI_UPPER \|
	U8_STRCMP_CI_LOWER);
	if (f == 0) {
	flag \|= U8_STRCMP_CS;
	} else if (f != U8_STRCMP_CS && f != U8_STRCMP_CI_UPPER &&
	f != U8_STRCMP_CI_LOWER) {
	*errnum = EBADF;
	flag = U8_STRCMP_CS;
	}

	f = flag & (U8_CANON_DECOMP \| U8_COMPAT_DECOMP \| U8_CANON_COMP);
	if (f && f != U8_STRCMP_NFD && f != U8_STRCMP_NFC &&
	f != U8_STRCMP_NFKD && f != U8_STRCMP_NFKC) {
	*errnum = EBADF;
	flag = U8_STRCMP_CS;
	}
	}

	if (flag == U8_STRCMP_CS) {
	return (n == 0 ? strcmp(s1, s2) : strncmp(s1, s2, n));
	}

	n1 = strlen(s1);
	n2 = strlen(s2);
	if (n != 0) {
	if (n < n1)
	n1 = n;
	if (n < n2)
	n2 = n;
	}

	/*
	* Simple case conversion can be done much faster and so we do
	* them separately here.
	*/
	if (flag == U8_STRCMP_CI_UPPER) {
	return (do_case_compare(uv, (uchar_t )s1, (uchar_t )s2,
	n1, n2, B_TRUE, errnum));
	} else if (flag == U8_STRCMP_CI_LOWER) {
	return (do_case_compare(uv, (uchar_t )s1, (uchar_t )s2,
	n1, n2, B_FALSE, errnum));
	}

	return (do_norm_compare(uv, (uchar_t )s1, (uchar_t )s2, n1, n2,
	flag, errnum));
	}

	size_t
	u8_textprep_str(char inarray, size_t inlen, char outarray, size_t outlen,
	int flag, size_t unicode_version, int *errnum)
	{
	int f;
	int sz;
	uchar_t *ib;
	uchar_t *ibtail;
	uchar_t *ob;
	uchar_t *obtail;
	boolean_t do_not_ignore_null;
	boolean_t do_not_ignore_invalid;
	boolean_t is_it_toupper;
	boolean_t is_it_tolower;
	boolean_t canonical_decomposition;
	boolean_t compatibility_decomposition;
	boolean_t canonical_composition;
	size_t ret_val;
	size_t i;
	size_t j;
	uchar_t u8s[U8_STREAM_SAFE_TEXT_MAX + 1];
	u8_normalization_states_t state;

	if (unicode_version > U8_UNICODE_LATEST) {
	*errnum = ERANGE;
	return ((size_t)-1);
	}

	f = flag & (U8_TEXTPREP_TOUPPER \| U8_TEXTPREP_TOLOWER);
	if (f == (U8_TEXTPREP_TOUPPER \| U8_TEXTPREP_TOLOWER)) {
	*errnum = EBADF;
	return ((size_t)-1);
	}

	f = flag & (U8_CANON_DECOMP \| U8_COMPAT_DECOMP \| U8_CANON_COMP);
	if (f && f != U8_TEXTPREP_NFD && f != U8_TEXTPREP_NFC &&
	f != U8_TEXTPREP_NFKD && f != U8_TEXTPREP_NFKC) {
	*errnum = EBADF;
	return ((size_t)-1);
	}

	if (inarray == NULL \|\| *inlen == 0)
	return (0);

	if (outarray == NULL) {
	*errnum = E2BIG;
	return ((size_t)-1);
	}

	ib = (uchar_t *)inarray;
	ob = (uchar_t *)outarray;
	ibtail = ib + *inlen;
	obtail = ob + *outlen;

	do_not_ignore_null = !(flag & U8_TEXTPREP_IGNORE_NULL);
	do_not_ignore_invalid = !(flag & U8_TEXTPREP_IGNORE_INVALID);
	is_it_toupper = flag & U8_TEXTPREP_TOUPPER;
	is_it_tolower = flag & U8_TEXTPREP_TOLOWER;

	ret_val = 0;

	/*
	* If we don't have a normalization flag set, we do the simple case
	* conversion based text preparation separately below. Text
	* preparation involving Normalization will be done in the false task
	* block, again, separately since it will take much more time and
	* resource than doing simple case conversions.
	*/
	if (f == 0) {
	while (ib < ibtail) {
	if (*ib == '\0' && do_not_ignore_null)
	break;

	sz = u8_number_of_bytes[*ib];

	if (sz < 0) {
	if (do_not_ignore_invalid) {
	*errnum = EILSEQ;
	ret_val = (size_t)-1;
	break;
	}

	sz = 1;
	ret_val++;
	}

	if (sz == 1) {
	if (ob >= obtail) {
	*errnum = E2BIG;
	ret_val = (size_t)-1;
	break;
	}

	if (is_it_toupper)
	ob = U8_ASCII_TOUPPER(ib);
	else if (is_it_tolower)
	ob = U8_ASCII_TOLOWER(ib);
	else
	ob = ib;
	ib++;
	ob++;
	} else if ((ib + sz) > ibtail) {
	if (do_not_ignore_invalid) {
	*errnum = EINVAL;
	ret_val = (size_t)-1;
	break;
	}

	if ((obtail - ob) < (ibtail - ib)) {
	*errnum = E2BIG;
	ret_val = (size_t)-1;
	break;
	}

	/*
	* We treat the remaining incomplete character
	* bytes as a character.
	*/
	ret_val++;

	while (ib < ibtail)
	ob++ = ib++;
	} else {
	if (is_it_toupper \|\| is_it_tolower) {
	i = do_case_conv(unicode_version, u8s,
	ib, sz, is_it_toupper);

	if ((obtail - ob) < i) {
	*errnum = E2BIG;
	ret_val = (size_t)-1;
	break;
	}

	ib += sz;

	for (sz = 0; sz < i; sz++)
	*ob++ = u8s[sz];
	} else {
	if ((obtail - ob) < sz) {
	*errnum = E2BIG;
	ret_val = (size_t)-1;
	break;
	}

	for (i = 0; i < sz; i++)
	ob++ = ib++;
	}
	}
	}
	} else {
	canonical_decomposition = flag & U8_CANON_DECOMP;
	compatibility_decomposition = flag & U8_COMPAT_DECOMP;
	canonical_composition = flag & U8_CANON_COMP;

	while (ib < ibtail) {
	if (*ib == '\0' && do_not_ignore_null)
	break;

	/*
	* If the current character is a 7-bit ASCII
	* character and it is the last character, or,
	* if the current character is a 7-bit ASCII
	* character and the next character is also a 7-bit
	* ASCII character, then, we copy over this
	* character without going through collect_a_seq().
	*
	* In any other cases, we need to look further with
	* the collect_a_seq() function.
	*/
	if (U8_ISASCII(*ib) && ((ib + 1) >= ibtail \|\|
	((ib + 1) < ibtail && U8_ISASCII(*(ib + 1))))) {
	if (ob >= obtail) {
	*errnum = E2BIG;
	ret_val = (size_t)-1;
	break;
	}

	if (is_it_toupper)
	ob = U8_ASCII_TOUPPER(ib);
	else if (is_it_tolower)
	ob = U8_ASCII_TOLOWER(ib);
	else
	ob = ib;
	ib++;
	ob++;
	} else {
	*errnum = 0;
	state = U8_STATE_START;

	j = collect_a_seq(unicode_version, u8s,
	&ib, ibtail,
	is_it_toupper,
	is_it_tolower,
	canonical_decomposition,
	compatibility_decomposition,
	canonical_composition,
	errnum, &state);

	if (*errnum && do_not_ignore_invalid) {
	ret_val = (size_t)-1;
	break;
	}

	if ((obtail - ob) < j) {
	*errnum = E2BIG;
	ret_val = (size_t)-1;
	break;
	}

	for (i = 0; i < j; i++)
	*ob++ = u8s[i];
	}
	}
	}

	*inlen = ibtail - ib;
	*outlen = obtail - ob;

	return (ret_val);
	}

	#if defined(_KERNEL)
	static int __init
	unicode_init(void)
	{
	return (0);
	}

	static void __exit
	unicode_fini(void)
	{
	}

	module_init(unicode_init);
	module_exit(unicode_fini);
	#endif

	ZFS_MODULE_DESCRIPTION("Unicode implementation");
	ZFS_MODULE_AUTHOR(ZFS_META_AUTHOR);
	ZFS_MODULE_LICENSE(ZFS_META_LICENSE);
	ZFS_MODULE_VERSION(ZFS_META_VERSION "-" ZFS_META_RELEASE);

	EXPORT_SYMBOL(u8_validate);
	EXPORT_SYMBOL(u8_strcmp);
	EXPORT_SYMBOL(u8_textprep_str);
	Index: vendor-sys/openzfs/dist/module/zcommon/zfs_prop.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/zcommon/zfs_prop.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/zcommon/zfs_prop.c (revision 366774)
	@@ -1,1053 +1,1052 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2011, 2018 by Delphix. All rights reserved.
	* Copyright (c) 2013 by Saso Kiselkov. All rights reserved.
	* Copyright 2016, Joyent, Inc.
	* Copyright (c) 2019, Klara Inc.
	* Copyright (c) 2019, Allan Jude
	*/

	/* Portions Copyright 2010 Robert Milkowski */

	#include <sys/zio.h>
	#include <sys/spa.h>
	#include <sys/u8_textprep.h>
	#include <sys/zfs_acl.h>
	#include <sys/zfs_ioctl.h>
	#include <sys/zfs_znode.h>
	#include <sys/dsl_crypt.h>

	#include "zfs_prop.h"
	#include "zfs_deleg.h"
	#include "zfs_fletcher.h"

	#if !defined(_KERNEL)
	#include <stdlib.h>
	#include <string.h>
	#include <ctype.h>
	#endif

	static zprop_desc_t zfs_prop_table[ZFS_NUM_PROPS];

	/* Note this is indexed by zfs_userquota_prop_t, keep the order the same */
	const char *zfs_userquota_prop_prefixes[] = {
	"userused@",
	"userquota@",
	"groupused@",
	"groupquota@",
	"userobjused@",
	"userobjquota@",
	"groupobjused@",
	"groupobjquota@",
	"projectused@",
	"projectquota@",
	"projectobjused@",
	"projectobjquota@"
	};

	zprop_desc_t *
	zfs_prop_get_table(void)
	{
	return (zfs_prop_table);
	}

	void
	zfs_prop_init(void)
	{
	static zprop_index_t checksum_table[] = {
	{ "on", ZIO_CHECKSUM_ON },
	{ "off", ZIO_CHECKSUM_OFF },
	{ "fletcher2", ZIO_CHECKSUM_FLETCHER_2 },
	{ "fletcher4", ZIO_CHECKSUM_FLETCHER_4 },
	{ "sha256", ZIO_CHECKSUM_SHA256 },
	{ "noparity", ZIO_CHECKSUM_NOPARITY },
	{ "sha512", ZIO_CHECKSUM_SHA512 },
	{ "skein", ZIO_CHECKSUM_SKEIN },
	#if !defined(__FreeBSD__)

	{ "edonr", ZIO_CHECKSUM_EDONR },
	#endif
	{ NULL }
	};

	static zprop_index_t dedup_table[] = {
	{ "on", ZIO_CHECKSUM_ON },
	{ "off", ZIO_CHECKSUM_OFF },
	{ "verify", ZIO_CHECKSUM_ON \| ZIO_CHECKSUM_VERIFY },
	{ "sha256", ZIO_CHECKSUM_SHA256 },
	{ "sha256,verify",
	ZIO_CHECKSUM_SHA256 \| ZIO_CHECKSUM_VERIFY },
	{ "sha512", ZIO_CHECKSUM_SHA512 },
	{ "sha512,verify",
	ZIO_CHECKSUM_SHA512 \| ZIO_CHECKSUM_VERIFY },
	{ "skein", ZIO_CHECKSUM_SKEIN },
	{ "skein,verify",
	ZIO_CHECKSUM_SKEIN \| ZIO_CHECKSUM_VERIFY },
	#if !defined(__FreeBSD__)

	{ "edonr,verify",
	ZIO_CHECKSUM_EDONR \| ZIO_CHECKSUM_VERIFY },
	#endif
	{ NULL }
	};

	static zprop_index_t compress_table[] = {
	{ "on", ZIO_COMPRESS_ON },
	{ "off", ZIO_COMPRESS_OFF },
	{ "lzjb", ZIO_COMPRESS_LZJB },
	{ "gzip", ZIO_COMPRESS_GZIP_6 }, /* gzip default */
	{ "gzip-1", ZIO_COMPRESS_GZIP_1 },
	{ "gzip-2", ZIO_COMPRESS_GZIP_2 },
	{ "gzip-3", ZIO_COMPRESS_GZIP_3 },
	{ "gzip-4", ZIO_COMPRESS_GZIP_4 },
	{ "gzip-5", ZIO_COMPRESS_GZIP_5 },
	{ "gzip-6", ZIO_COMPRESS_GZIP_6 },
	{ "gzip-7", ZIO_COMPRESS_GZIP_7 },
	{ "gzip-8", ZIO_COMPRESS_GZIP_8 },
	{ "gzip-9", ZIO_COMPRESS_GZIP_9 },
	{ "zle", ZIO_COMPRESS_ZLE },
	{ "lz4", ZIO_COMPRESS_LZ4 },
	{ "zstd", ZIO_COMPRESS_ZSTD },
	{ "zstd-fast",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_DEFAULT) },

	/*
	* ZSTD 1-19 are synthetic. We store the compression level in a
	* separate hidden property to avoid wasting a large amount of
	* space in the ZIO_COMPRESS enum.
	*
	* The compression level is also stored within the header of the
	* compressed block since we may need it for later recompression
	* to avoid checksum errors (L2ARC).
	*
	* Note that the level here is defined as bit shifted mask on
	* top of the method.
	*/
	{ "zstd-1", ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_1) },
	{ "zstd-2", ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_2) },
	{ "zstd-3", ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_3) },
	{ "zstd-4", ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_4) },
	{ "zstd-5", ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_5) },
	{ "zstd-6", ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_6) },
	{ "zstd-7", ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_7) },
	{ "zstd-8", ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_8) },
	{ "zstd-9", ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_9) },
	{ "zstd-10", ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_10) },
	{ "zstd-11", ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_11) },
	{ "zstd-12", ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_12) },
	{ "zstd-13", ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_13) },
	{ "zstd-14", ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_14) },
	{ "zstd-15", ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_15) },
	{ "zstd-16", ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_16) },
	{ "zstd-17", ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_17) },
	{ "zstd-18", ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_18) },
	{ "zstd-19", ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_19) },

	/*
	* The ZSTD-Fast levels are also synthetic.
	*/
	{ "zstd-fast-1",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_1) },
	{ "zstd-fast-2",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_2) },
	{ "zstd-fast-3",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_3) },
	{ "zstd-fast-4",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_4) },
	{ "zstd-fast-5",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_5) },
	{ "zstd-fast-6",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_6) },
	{ "zstd-fast-7",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_7) },
	{ "zstd-fast-8",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_8) },
	{ "zstd-fast-9",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_9) },
	{ "zstd-fast-10",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_10) },
	{ "zstd-fast-20",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_20) },
	{ "zstd-fast-30",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_30) },
	{ "zstd-fast-40",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_40) },
	{ "zstd-fast-50",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_50) },
	{ "zstd-fast-60",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_60) },
	{ "zstd-fast-70",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_70) },
	{ "zstd-fast-80",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_80) },
	{ "zstd-fast-90",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_90) },
	{ "zstd-fast-100",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_100) },
	{ "zstd-fast-500",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_500) },
	{ "zstd-fast-1000",
	ZIO_COMPLEVEL_ZSTD(ZIO_ZSTD_LEVEL_FAST_1000) },
	{ NULL }
	};

	static zprop_index_t crypto_table[] = {
	{ "on", ZIO_CRYPT_ON },
	{ "off", ZIO_CRYPT_OFF },
	{ "aes-128-ccm", ZIO_CRYPT_AES_128_CCM },
	{ "aes-192-ccm", ZIO_CRYPT_AES_192_CCM },
	{ "aes-256-ccm", ZIO_CRYPT_AES_256_CCM },
	{ "aes-128-gcm", ZIO_CRYPT_AES_128_GCM },
	{ "aes-192-gcm", ZIO_CRYPT_AES_192_GCM },
	{ "aes-256-gcm", ZIO_CRYPT_AES_256_GCM },
	{ NULL }
	};

	static zprop_index_t keyformat_table[] = {
	{ "none", ZFS_KEYFORMAT_NONE },
	{ "raw", ZFS_KEYFORMAT_RAW },
	{ "hex", ZFS_KEYFORMAT_HEX },
	{ "passphrase", ZFS_KEYFORMAT_PASSPHRASE },
	{ NULL }
	};

	static zprop_index_t snapdir_table[] = {
	{ "hidden", ZFS_SNAPDIR_HIDDEN },
	{ "visible", ZFS_SNAPDIR_VISIBLE },
	{ NULL }
	};

	static zprop_index_t snapdev_table[] = {
	{ "hidden", ZFS_SNAPDEV_HIDDEN },
	{ "visible", ZFS_SNAPDEV_VISIBLE },
	{ NULL }
	};

	static zprop_index_t acl_mode_table[] = {
	{ "discard", ZFS_ACL_DISCARD },
	{ "groupmask", ZFS_ACL_GROUPMASK },
	{ "passthrough", ZFS_ACL_PASSTHROUGH },
	{ "restricted", ZFS_ACL_RESTRICTED },
	{ NULL }
	};

	static zprop_index_t acltype_table[] = {
	{ "off", ZFS_ACLTYPE_OFF },
	{ "posix", ZFS_ACLTYPE_POSIX },
	+ { "nfsv4", ZFS_ACLTYPE_NFSV4 },
	{ "disabled", ZFS_ACLTYPE_OFF }, /* bkwrd compatibility */
	{ "noacl", ZFS_ACLTYPE_OFF }, /* bkwrd compatibility */
	{ "posixacl", ZFS_ACLTYPE_POSIX }, /* bkwrd compatibility */
	{ NULL }
	};

	static zprop_index_t acl_inherit_table[] = {
	{ "discard", ZFS_ACL_DISCARD },
	{ "noallow", ZFS_ACL_NOALLOW },
	{ "restricted", ZFS_ACL_RESTRICTED },
	{ "passthrough", ZFS_ACL_PASSTHROUGH },
	{ "secure", ZFS_ACL_RESTRICTED }, /* bkwrd compatibility */
	{ "passthrough-x", ZFS_ACL_PASSTHROUGH_X },
	{ NULL }
	};

	static zprop_index_t case_table[] = {
	{ "sensitive", ZFS_CASE_SENSITIVE },
	{ "insensitive", ZFS_CASE_INSENSITIVE },
	{ "mixed", ZFS_CASE_MIXED },
	{ NULL }
	};

	static zprop_index_t copies_table[] = {
	{ "1", 1 },
	{ "2", 2 },
	{ "3", 3 },
	{ NULL }
	};

	/*
	* Use the unique flags we have to send to u8_strcmp() and/or
	* u8_textprep() to represent the various normalization property
	* values.
	*/
	static zprop_index_t normalize_table[] = {
	{ "none", 0 },
	{ "formD", U8_TEXTPREP_NFD },
	{ "formKC", U8_TEXTPREP_NFKC },
	{ "formC", U8_TEXTPREP_NFC },
	{ "formKD", U8_TEXTPREP_NFKD },
	{ NULL }
	};

	static zprop_index_t version_table[] = {
	{ "1", 1 },
	{ "2", 2 },
	{ "3", 3 },
	{ "4", 4 },
	{ "5", 5 },
	{ "current", ZPL_VERSION },
	{ NULL }
	};

	static zprop_index_t boolean_table[] = {
	{ "off", 0 },
	{ "on", 1 },
	{ NULL }
	};

	static zprop_index_t keystatus_table[] = {
	{ "none", ZFS_KEYSTATUS_NONE},
	{ "unavailable", ZFS_KEYSTATUS_UNAVAILABLE},
	{ "available", ZFS_KEYSTATUS_AVAILABLE},
	{ NULL }
	};

	static zprop_index_t logbias_table[] = {
	{ "latency", ZFS_LOGBIAS_LATENCY },
	{ "throughput", ZFS_LOGBIAS_THROUGHPUT },
	{ NULL }
	};

	static zprop_index_t canmount_table[] = {
	{ "off", ZFS_CANMOUNT_OFF },
	{ "on", ZFS_CANMOUNT_ON },
	{ "noauto", ZFS_CANMOUNT_NOAUTO },
	{ NULL }
	};

	static zprop_index_t cache_table[] = {
	{ "none", ZFS_CACHE_NONE },
	{ "metadata", ZFS_CACHE_METADATA },
	{ "all", ZFS_CACHE_ALL },
	{ NULL }
	};

	static zprop_index_t sync_table[] = {
	{ "standard", ZFS_SYNC_STANDARD },
	{ "always", ZFS_SYNC_ALWAYS },
	{ "disabled", ZFS_SYNC_DISABLED },
	{ NULL }
	};

	static zprop_index_t xattr_table[] = {
	{ "off", ZFS_XATTR_OFF },
	{ "on", ZFS_XATTR_DIR },
	{ "sa", ZFS_XATTR_SA },
	{ "dir", ZFS_XATTR_DIR },
	{ NULL }
	};

	static zprop_index_t dnsize_table[] = {
	{ "legacy", ZFS_DNSIZE_LEGACY },
	{ "auto", ZFS_DNSIZE_AUTO },
	{ "1k", ZFS_DNSIZE_1K },
	{ "2k", ZFS_DNSIZE_2K },
	{ "4k", ZFS_DNSIZE_4K },
	{ "8k", ZFS_DNSIZE_8K },
	{ "16k", ZFS_DNSIZE_16K },
	{ NULL }
	};

	static zprop_index_t redundant_metadata_table[] = {
	{ "all", ZFS_REDUNDANT_METADATA_ALL },
	{ "most", ZFS_REDUNDANT_METADATA_MOST },
	{ NULL }
	};

	static zprop_index_t volmode_table[] = {
	{ "default", ZFS_VOLMODE_DEFAULT },
	{ "full", ZFS_VOLMODE_GEOM },
	{ "geom", ZFS_VOLMODE_GEOM },
	{ "dev", ZFS_VOLMODE_DEV },
	{ "none", ZFS_VOLMODE_NONE },
	{ NULL }
	};

	/* inherit index properties */
	zprop_register_index(ZFS_PROP_REDUNDANT_METADATA, "redundant_metadata",
	ZFS_REDUNDANT_METADATA_ALL,
	PROP_INHERIT, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME,
	"all \| most", "REDUND_MD",
	redundant_metadata_table);
	zprop_register_index(ZFS_PROP_SYNC, "sync", ZFS_SYNC_STANDARD,
	PROP_INHERIT, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME,
	"standard \| always \| disabled", "SYNC",
	sync_table);
	zprop_register_index(ZFS_PROP_CHECKSUM, "checksum",
	ZIO_CHECKSUM_DEFAULT, PROP_INHERIT, ZFS_TYPE_FILESYSTEM \|
	ZFS_TYPE_VOLUME,
	#if !defined(__FreeBSD__)
	"on \| off \| fletcher2 \| fletcher4 \| sha256 \| sha512 \| skein"
	" \| edonr",
	#else
	"on \| off \| fletcher2 \| fletcher4 \| sha256 \| sha512 \| skein",
	#endif
	"CHECKSUM", checksum_table);
	zprop_register_index(ZFS_PROP_DEDUP, "dedup", ZIO_CHECKSUM_OFF,
	PROP_INHERIT, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME,
	"on \| off \| verify \| sha256[,verify] \| sha512[,verify] \| "
	#if !defined(__FreeBSD__)
	"skein[,verify] \| edonr,verify",
	#else
	"skein[,verify]",
	#endif
	"DEDUP", dedup_table);
	zprop_register_index(ZFS_PROP_COMPRESSION, "compression",
	ZIO_COMPRESS_DEFAULT, PROP_INHERIT,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME,
	"on \| off \| lzjb \| gzip \| gzip-[1-9] \| zle \| lz4 \| "
	"zstd \| zstd-[1-19] \| "
	"zstd-fast-[1-10,20,30,40,50,60,70,80,90,100,500,1000]",
	"COMPRESS", compress_table);
	zprop_register_index(ZFS_PROP_SNAPDIR, "snapdir", ZFS_SNAPDIR_HIDDEN,
	PROP_INHERIT, ZFS_TYPE_FILESYSTEM,
	"hidden \| visible", "SNAPDIR", snapdir_table);
	zprop_register_index(ZFS_PROP_SNAPDEV, "snapdev", ZFS_SNAPDEV_HIDDEN,
	PROP_INHERIT, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME,
	"hidden \| visible", "SNAPDEV", snapdev_table);
	zprop_register_index(ZFS_PROP_ACLMODE, "aclmode", ZFS_ACL_DISCARD,
	PROP_INHERIT, ZFS_TYPE_FILESYSTEM,
	"discard \| groupmask \| passthrough \| restricted", "ACLMODE",
	acl_mode_table);
	-#ifndef __FreeBSD__
	- zprop_register_index(ZFS_PROP_ACLTYPE, "acltype", ZFS_ACLTYPE_OFF,
	- PROP_INHERIT, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_SNAPSHOT,
	- "off \| posix", "ACLTYPE", acltype_table);
	+ zprop_register_index(ZFS_PROP_ACLTYPE, "acltype",
	+#ifdef __linux__
	+ /* Linux doesn't natively support ZFS's NFSv4-style ACLs. */
	+ ZFS_ACLTYPE_OFF,
	+#else
	+ ZFS_ACLTYPE_NFSV4,
	#endif
	+ PROP_INHERIT, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_SNAPSHOT,
	+ "off \| nfsv4 \| posix", "ACLTYPE", acltype_table);
	zprop_register_index(ZFS_PROP_ACLINHERIT, "aclinherit",
	ZFS_ACL_RESTRICTED, PROP_INHERIT, ZFS_TYPE_FILESYSTEM,
	"discard \| noallow \| restricted \| passthrough \| passthrough-x",
	"ACLINHERIT", acl_inherit_table);
	zprop_register_index(ZFS_PROP_COPIES, "copies", 1, PROP_INHERIT,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME,
	"1 \| 2 \| 3", "COPIES", copies_table);
	zprop_register_index(ZFS_PROP_PRIMARYCACHE, "primarycache",
	ZFS_CACHE_ALL, PROP_INHERIT,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_SNAPSHOT \| ZFS_TYPE_VOLUME,
	"all \| none \| metadata", "PRIMARYCACHE", cache_table);
	zprop_register_index(ZFS_PROP_SECONDARYCACHE, "secondarycache",
	ZFS_CACHE_ALL, PROP_INHERIT,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_SNAPSHOT \| ZFS_TYPE_VOLUME,
	"all \| none \| metadata", "SECONDARYCACHE", cache_table);
	zprop_register_index(ZFS_PROP_LOGBIAS, "logbias", ZFS_LOGBIAS_LATENCY,
	PROP_INHERIT, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME,
	"latency \| throughput", "LOGBIAS", logbias_table);
	zprop_register_index(ZFS_PROP_XATTR, "xattr", ZFS_XATTR_DIR,
	PROP_INHERIT, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_SNAPSHOT,
	"on \| off \| dir \| sa", "XATTR", xattr_table);
	zprop_register_index(ZFS_PROP_DNODESIZE, "dnodesize",
	ZFS_DNSIZE_LEGACY, PROP_INHERIT, ZFS_TYPE_FILESYSTEM,
	"legacy \| auto \| 1k \| 2k \| 4k \| 8k \| 16k", "DNSIZE", dnsize_table);
	zprop_register_index(ZFS_PROP_VOLMODE, "volmode",
	ZFS_VOLMODE_DEFAULT, PROP_INHERIT,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME,
	"default \| full \| geom \| dev \| none", "VOLMODE", volmode_table);

	/* inherit index (boolean) properties */
	zprop_register_index(ZFS_PROP_ATIME, "atime", 1, PROP_INHERIT,
	ZFS_TYPE_FILESYSTEM, "on \| off", "ATIME", boolean_table);
	zprop_register_index(ZFS_PROP_RELATIME, "relatime", 0, PROP_INHERIT,
	ZFS_TYPE_FILESYSTEM, "on \| off", "RELATIME", boolean_table);
	zprop_register_index(ZFS_PROP_DEVICES, "devices", 1, PROP_INHERIT,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_SNAPSHOT, "on \| off", "DEVICES",
	boolean_table);
	zprop_register_index(ZFS_PROP_EXEC, "exec", 1, PROP_INHERIT,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_SNAPSHOT, "on \| off", "EXEC",
	boolean_table);
	zprop_register_index(ZFS_PROP_SETUID, "setuid", 1, PROP_INHERIT,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_SNAPSHOT, "on \| off", "SETUID",
	boolean_table);
	zprop_register_index(ZFS_PROP_READONLY, "readonly", 0, PROP_INHERIT,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME, "on \| off", "RDONLY",
	boolean_table);
	#ifdef __FreeBSD__
	zprop_register_index(ZFS_PROP_ZONED, "jailed", 0, PROP_INHERIT,
	ZFS_TYPE_FILESYSTEM, "on \| off", "JAILED", boolean_table);
	#else
	zprop_register_index(ZFS_PROP_ZONED, "zoned", 0, PROP_INHERIT,
	ZFS_TYPE_FILESYSTEM, "on \| off", "ZONED", boolean_table);
	#endif
	zprop_register_index(ZFS_PROP_VSCAN, "vscan", 0, PROP_INHERIT,
	ZFS_TYPE_FILESYSTEM, "on \| off", "VSCAN", boolean_table);
	zprop_register_index(ZFS_PROP_NBMAND, "nbmand", 0, PROP_INHERIT,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_SNAPSHOT, "on \| off", "NBMAND",
	boolean_table);
	zprop_register_index(ZFS_PROP_OVERLAY, "overlay", 1, PROP_INHERIT,
	ZFS_TYPE_FILESYSTEM, "on \| off", "OVERLAY", boolean_table);

	/* default index properties */
	zprop_register_index(ZFS_PROP_VERSION, "version", 0, PROP_DEFAULT,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_SNAPSHOT,
	"1 \| 2 \| 3 \| 4 \| 5 \| current", "VERSION", version_table);
	zprop_register_index(ZFS_PROP_CANMOUNT, "canmount", ZFS_CANMOUNT_ON,
	PROP_DEFAULT, ZFS_TYPE_FILESYSTEM, "on \| off \| noauto",
	"CANMOUNT", canmount_table);

	/* readonly index properties */
	zprop_register_index(ZFS_PROP_MOUNTED, "mounted", 0, PROP_READONLY,
	ZFS_TYPE_FILESYSTEM, "yes \| no", "MOUNTED", boolean_table);
	zprop_register_index(ZFS_PROP_DEFER_DESTROY, "defer_destroy", 0,
	PROP_READONLY, ZFS_TYPE_SNAPSHOT, "yes \| no", "DEFER_DESTROY",
	boolean_table);
	zprop_register_index(ZFS_PROP_KEYSTATUS, "keystatus",
	ZFS_KEYSTATUS_NONE, PROP_READONLY, ZFS_TYPE_DATASET,
	"none \| unavailable \| available",
	"KEYSTATUS", keystatus_table);

	/* set once index properties */
	zprop_register_index(ZFS_PROP_NORMALIZE, "normalization", 0,
	PROP_ONETIME, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_SNAPSHOT,
	"none \| formC \| formD \| formKC \| formKD", "NORMALIZATION",
	normalize_table);
	zprop_register_index(ZFS_PROP_CASE, "casesensitivity",
	ZFS_CASE_SENSITIVE, PROP_ONETIME, ZFS_TYPE_FILESYSTEM \|
	ZFS_TYPE_SNAPSHOT,
	"sensitive \| insensitive \| mixed", "CASE", case_table);
	zprop_register_index(ZFS_PROP_KEYFORMAT, "keyformat",
	ZFS_KEYFORMAT_NONE, PROP_ONETIME_DEFAULT,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME,
	"none \| raw \| hex \| passphrase", "KEYFORMAT", keyformat_table);
	zprop_register_index(ZFS_PROP_ENCRYPTION, "encryption",
	ZIO_CRYPT_DEFAULT, PROP_ONETIME, ZFS_TYPE_DATASET,
	"on \| off \| aes-128-ccm \| aes-192-ccm \| aes-256-ccm \| "
	"aes-128-gcm \| aes-192-gcm \| aes-256-gcm", "ENCRYPTION",
	crypto_table);

	/* set once index (boolean) properties */
	zprop_register_index(ZFS_PROP_UTF8ONLY, "utf8only", 0, PROP_ONETIME,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_SNAPSHOT,
	"on \| off", "UTF8ONLY", boolean_table);

	/* string properties */
	zprop_register_string(ZFS_PROP_ORIGIN, "origin", NULL, PROP_READONLY,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME, "<snapshot>", "ORIGIN");
	zprop_register_string(ZFS_PROP_CLONES, "clones", NULL, PROP_READONLY,
	ZFS_TYPE_SNAPSHOT, "<dataset>[,...]", "CLONES");
	zprop_register_string(ZFS_PROP_MOUNTPOINT, "mountpoint", "/",
	PROP_INHERIT, ZFS_TYPE_FILESYSTEM, "<path> \| legacy \| none",
	"MOUNTPOINT");
	zprop_register_string(ZFS_PROP_SHARENFS, "sharenfs", "off",
	PROP_INHERIT, ZFS_TYPE_FILESYSTEM, "on \| off \| share(1M) options",
	"SHARENFS");
	zprop_register_string(ZFS_PROP_TYPE, "type", NULL, PROP_READONLY,
	ZFS_TYPE_DATASET \| ZFS_TYPE_BOOKMARK,
	"filesystem \| volume \| snapshot \| bookmark", "TYPE");
	zprop_register_string(ZFS_PROP_SHARESMB, "sharesmb", "off",
	PROP_INHERIT, ZFS_TYPE_FILESYSTEM,
	"on \| off \| sharemgr(1M) options", "SHARESMB");
	zprop_register_string(ZFS_PROP_MLSLABEL, "mlslabel",
	ZFS_MLSLABEL_DEFAULT, PROP_INHERIT, ZFS_TYPE_DATASET,
	"<sensitivity label>", "MLSLABEL");
	zprop_register_string(ZFS_PROP_SELINUX_CONTEXT, "context",
	"none", PROP_DEFAULT, ZFS_TYPE_DATASET, "<selinux context>",
	"CONTEXT");
	zprop_register_string(ZFS_PROP_SELINUX_FSCONTEXT, "fscontext",
	"none", PROP_DEFAULT, ZFS_TYPE_DATASET, "<selinux fscontext>",
	"FSCONTEXT");
	zprop_register_string(ZFS_PROP_SELINUX_DEFCONTEXT, "defcontext",
	"none", PROP_DEFAULT, ZFS_TYPE_DATASET, "<selinux defcontext>",
	"DEFCONTEXT");
	zprop_register_string(ZFS_PROP_SELINUX_ROOTCONTEXT, "rootcontext",
	"none", PROP_DEFAULT, ZFS_TYPE_DATASET, "<selinux rootcontext>",
	"ROOTCONTEXT");
	zprop_register_string(ZFS_PROP_RECEIVE_RESUME_TOKEN,
	"receive_resume_token",
	NULL, PROP_READONLY, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME,
	"<string token>", "RESUMETOK");
	zprop_register_string(ZFS_PROP_ENCRYPTION_ROOT, "encryptionroot", NULL,
	PROP_READONLY, ZFS_TYPE_DATASET, "<filesystem \| volume>",
	"ENCROOT");
	zprop_register_string(ZFS_PROP_KEYLOCATION, "keylocation",
	"none", PROP_DEFAULT, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME,
	"prompt \| <file URI>", "KEYLOCATION");
	zprop_register_string(ZFS_PROP_REDACT_SNAPS,
	"redact_snaps", NULL, PROP_READONLY,
	ZFS_TYPE_DATASET \| ZFS_TYPE_BOOKMARK, "<snapshot>[,...]",
	"RSNAPS");

	/* readonly number properties */
	zprop_register_number(ZFS_PROP_USED, "used", 0, PROP_READONLY,
	ZFS_TYPE_DATASET, "<size>", "USED");
	zprop_register_number(ZFS_PROP_AVAILABLE, "available", 0, PROP_READONLY,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME, "<size>", "AVAIL");
	zprop_register_number(ZFS_PROP_REFERENCED, "referenced", 0,
	PROP_READONLY, ZFS_TYPE_DATASET \| ZFS_TYPE_BOOKMARK, "<size>",
	"REFER");
	zprop_register_number(ZFS_PROP_COMPRESSRATIO, "compressratio", 0,
	PROP_READONLY, ZFS_TYPE_DATASET \| ZFS_TYPE_BOOKMARK,
	"<1.00x or higher if compressed>", "RATIO");
	zprop_register_number(ZFS_PROP_REFRATIO, "refcompressratio", 0,
	PROP_READONLY, ZFS_TYPE_DATASET,
	"<1.00x or higher if compressed>", "REFRATIO");
	zprop_register_number(ZFS_PROP_VOLBLOCKSIZE, "volblocksize",
	ZVOL_DEFAULT_BLOCKSIZE, PROP_ONETIME,
	ZFS_TYPE_VOLUME, "512 to 128k, power of 2", "VOLBLOCK");
	zprop_register_number(ZFS_PROP_USEDSNAP, "usedbysnapshots", 0,
	PROP_READONLY, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME, "<size>",
	"USEDSNAP");
	zprop_register_number(ZFS_PROP_USEDDS, "usedbydataset", 0,
	PROP_READONLY, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME, "<size>",
	"USEDDS");
	zprop_register_number(ZFS_PROP_USEDCHILD, "usedbychildren", 0,
	PROP_READONLY, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME, "<size>",
	"USEDCHILD");
	zprop_register_number(ZFS_PROP_USEDREFRESERV, "usedbyrefreservation", 0,
	PROP_READONLY,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME, "<size>", "USEDREFRESERV");
	zprop_register_number(ZFS_PROP_USERREFS, "userrefs", 0, PROP_READONLY,
	ZFS_TYPE_SNAPSHOT, "<count>", "USERREFS");
	zprop_register_number(ZFS_PROP_WRITTEN, "written", 0, PROP_READONLY,
	ZFS_TYPE_DATASET, "<size>", "WRITTEN");
	zprop_register_number(ZFS_PROP_LOGICALUSED, "logicalused", 0,
	PROP_READONLY, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME, "<size>",
	"LUSED");
	zprop_register_number(ZFS_PROP_LOGICALREFERENCED, "logicalreferenced",
	0, PROP_READONLY, ZFS_TYPE_DATASET \| ZFS_TYPE_BOOKMARK, "<size>",
	"LREFER");
	zprop_register_number(ZFS_PROP_FILESYSTEM_COUNT, "filesystem_count",
	UINT64_MAX, PROP_READONLY, ZFS_TYPE_FILESYSTEM,
	"<count>", "FSCOUNT");
	zprop_register_number(ZFS_PROP_SNAPSHOT_COUNT, "snapshot_count",
	UINT64_MAX, PROP_READONLY, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME,
	"<count>", "SSCOUNT");
	zprop_register_number(ZFS_PROP_GUID, "guid", 0, PROP_READONLY,
	ZFS_TYPE_DATASET \| ZFS_TYPE_BOOKMARK, "<uint64>", "GUID");
	zprop_register_number(ZFS_PROP_CREATETXG, "createtxg", 0, PROP_READONLY,
	ZFS_TYPE_DATASET \| ZFS_TYPE_BOOKMARK, "<uint64>", "CREATETXG");
	zprop_register_number(ZFS_PROP_PBKDF2_ITERS, "pbkdf2iters",
	0, PROP_ONETIME_DEFAULT, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME,
	"<iters>", "PBKDF2ITERS");
	zprop_register_number(ZFS_PROP_OBJSETID, "objsetid", 0,
	PROP_READONLY, ZFS_TYPE_DATASET, "<uint64>", "OBJSETID");

	/* default number properties */
	zprop_register_number(ZFS_PROP_QUOTA, "quota", 0, PROP_DEFAULT,
	ZFS_TYPE_FILESYSTEM, "<size> \| none", "QUOTA");
	zprop_register_number(ZFS_PROP_RESERVATION, "reservation", 0,
	PROP_DEFAULT, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME,
	"<size> \| none", "RESERV");
	zprop_register_number(ZFS_PROP_VOLSIZE, "volsize", 0, PROP_DEFAULT,
	ZFS_TYPE_SNAPSHOT \| ZFS_TYPE_VOLUME, "<size>", "VOLSIZE");
	zprop_register_number(ZFS_PROP_REFQUOTA, "refquota", 0, PROP_DEFAULT,
	ZFS_TYPE_FILESYSTEM, "<size> \| none", "REFQUOTA");
	zprop_register_number(ZFS_PROP_REFRESERVATION, "refreservation", 0,
	PROP_DEFAULT, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME,
	"<size> \| none", "REFRESERV");
	zprop_register_number(ZFS_PROP_FILESYSTEM_LIMIT, "filesystem_limit",
	UINT64_MAX, PROP_DEFAULT, ZFS_TYPE_FILESYSTEM,
	"<count> \| none", "FSLIMIT");
	zprop_register_number(ZFS_PROP_SNAPSHOT_LIMIT, "snapshot_limit",
	UINT64_MAX, PROP_DEFAULT, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME,
	"<count> \| none", "SSLIMIT");

	/* inherit number properties */
	zprop_register_number(ZFS_PROP_RECORDSIZE, "recordsize",
	SPA_OLD_MAXBLOCKSIZE, PROP_INHERIT,
	ZFS_TYPE_FILESYSTEM, "512 to 1M, power of 2", "RECSIZE");
	zprop_register_number(ZFS_PROP_SPECIAL_SMALL_BLOCKS,
	"special_small_blocks", 0, PROP_INHERIT, ZFS_TYPE_FILESYSTEM,
	"zero or 512 to 1M, power of 2", "SPECIAL_SMALL_BLOCKS");

	/* hidden properties */
	zprop_register_hidden(ZFS_PROP_NUMCLONES, "numclones", PROP_TYPE_NUMBER,
	PROP_READONLY, ZFS_TYPE_SNAPSHOT, "NUMCLONES");
	zprop_register_hidden(ZFS_PROP_NAME, "name", PROP_TYPE_STRING,
	PROP_READONLY, ZFS_TYPE_DATASET \| ZFS_TYPE_BOOKMARK, "NAME");
	zprop_register_hidden(ZFS_PROP_ISCSIOPTIONS, "iscsioptions",
	PROP_TYPE_STRING, PROP_INHERIT, ZFS_TYPE_VOLUME, "ISCSIOPTIONS");
	zprop_register_hidden(ZFS_PROP_STMF_SHAREINFO, "stmf_sbd_lu",
	PROP_TYPE_STRING, PROP_INHERIT, ZFS_TYPE_VOLUME,
	"STMF_SBD_LU");
	zprop_register_hidden(ZFS_PROP_USERACCOUNTING, "useraccounting",
	PROP_TYPE_NUMBER, PROP_READONLY, ZFS_TYPE_DATASET,
	"USERACCOUNTING");
	zprop_register_hidden(ZFS_PROP_UNIQUE, "unique", PROP_TYPE_NUMBER,
	PROP_READONLY, ZFS_TYPE_DATASET, "UNIQUE");
	zprop_register_hidden(ZFS_PROP_INCONSISTENT, "inconsistent",
	PROP_TYPE_NUMBER, PROP_READONLY, ZFS_TYPE_DATASET, "INCONSISTENT");
	zprop_register_hidden(ZFS_PROP_IVSET_GUID, "ivsetguid",
	PROP_TYPE_NUMBER, PROP_READONLY,
	ZFS_TYPE_DATASET \| ZFS_TYPE_BOOKMARK, "IVSETGUID");
	zprop_register_hidden(ZFS_PROP_PREV_SNAP, "prevsnap", PROP_TYPE_STRING,
	PROP_READONLY, ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME, "PREVSNAP");
	zprop_register_hidden(ZFS_PROP_PBKDF2_SALT, "pbkdf2salt",
	PROP_TYPE_NUMBER, PROP_ONETIME_DEFAULT,
	ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_VOLUME, "PBKDF2SALT");
	zprop_register_hidden(ZFS_PROP_KEY_GUID, "keyguid", PROP_TYPE_NUMBER,
	PROP_READONLY, ZFS_TYPE_DATASET, "KEYGUID");
	zprop_register_hidden(ZFS_PROP_REDACTED, "redacted", PROP_TYPE_NUMBER,
	PROP_READONLY, ZFS_TYPE_DATASET, "REDACTED");

	/*
	* Properties that are obsolete and not used. These are retained so
	* that we don't have to change the values of the zfs_prop_t enum, or
	* have NULL pointers in the zfs_prop_table[].
	*/
	-#ifdef __FreeBSD__
	- zprop_register_impl(ZFS_PROP_ACLTYPE, "acltype", PROP_TYPE_INDEX,
	- ZFS_ACLTYPE_OFF, NULL, PROP_INHERIT,
	- ZFS_TYPE_FILESYSTEM \| ZFS_TYPE_SNAPSHOT,
	- "off \| posix", "ACLTYPE", B_FALSE, B_FALSE, acltype_table);
	-#endif
	zprop_register_hidden(ZFS_PROP_REMAPTXG, "remaptxg", PROP_TYPE_NUMBER,
	PROP_READONLY, ZFS_TYPE_DATASET, "REMAPTXG");

	/* oddball properties */
	zprop_register_impl(ZFS_PROP_CREATION, "creation", PROP_TYPE_NUMBER, 0,
	NULL, PROP_READONLY, ZFS_TYPE_DATASET \| ZFS_TYPE_BOOKMARK,
	"<date>", "CREATION", B_FALSE, B_TRUE, NULL);
	}

	boolean_t
	zfs_prop_delegatable(zfs_prop_t prop)
	{
	zprop_desc_t *pd = &zfs_prop_table[prop];

	/* The mlslabel property is never delegatable. */
	if (prop == ZFS_PROP_MLSLABEL)
	return (B_FALSE);

	return (pd->pd_attr != PROP_READONLY);
	}

	/*
	* Given a zfs dataset property name, returns the corresponding property ID.
	*/
	zfs_prop_t
	zfs_name_to_prop(const char *propname)
	{
	return (zprop_name_to_prop(propname, ZFS_TYPE_DATASET));
	}

	/*
	* For user property names, we allow all lowercase alphanumeric characters, plus
	* a few useful punctuation characters.
	*/
	static int
	valid_char(char c)
	{
	return ((c >= 'a' && c <= 'z') \|\|
	(c >= '0' && c <= '9') \|\|
	c == '-' \|\| c == '_' \|\| c == '.' \|\| c == ':');
	}

	/*
	* Returns true if this is a valid user-defined property (one with a ':').
	*/
	boolean_t
	zfs_prop_user(const char *name)
	{
	int i;
	char c;
	boolean_t foundsep = B_FALSE;

	for (i = 0; i < strlen(name); i++) {
	c = name[i];
	if (!valid_char(c))
	return (B_FALSE);
	if (c == ':')
	foundsep = B_TRUE;
	}

	if (!foundsep)
	return (B_FALSE);

	return (B_TRUE);
	}

	/*
	* Returns true if this is a valid userspace-type property (one with a '@').
	* Note that after the @, any character is valid (eg, another @, for SID
	* user@domain).
	*/
	boolean_t
	zfs_prop_userquota(const char *name)
	{
	zfs_userquota_prop_t prop;

	for (prop = 0; prop < ZFS_NUM_USERQUOTA_PROPS; prop++) {
	if (strncmp(name, zfs_userquota_prop_prefixes[prop],
	strlen(zfs_userquota_prop_prefixes[prop])) == 0) {
	return (B_TRUE);
	}
	}

	return (B_FALSE);
	}

	/*
	* Returns true if this is a valid written@ property.
	* Note that after the @, any character is valid (eg, another @, for
	* written@pool/fs@origin).
	*/
	boolean_t
	zfs_prop_written(const char *name)
	{
	static const char *prop_prefix = "written@";
	static const char *book_prefix = "written#";
	return (strncmp(name, prop_prefix, strlen(prop_prefix)) == 0 \|\|
	strncmp(name, book_prefix, strlen(book_prefix)) == 0);
	}

	/*
	* Tables of index types, plus functions to convert between the user view
	* (strings) and internal representation (uint64_t).
	*/
	int
	zfs_prop_string_to_index(zfs_prop_t prop, const char string, uint64_t index)
	{
	return (zprop_string_to_index(prop, string, index, ZFS_TYPE_DATASET));
	}

	int
	zfs_prop_index_to_string(zfs_prop_t prop, uint64_t index, const char **string)
	{
	return (zprop_index_to_string(prop, index, string, ZFS_TYPE_DATASET));
	}

	uint64_t
	zfs_prop_random_value(zfs_prop_t prop, uint64_t seed)
	{
	return (zprop_random_value(prop, seed, ZFS_TYPE_DATASET));
	}

	/*
	* Returns TRUE if the property applies to any of the given dataset types.
	*/
	boolean_t
	zfs_prop_valid_for_type(int prop, zfs_type_t types, boolean_t headcheck)
	{
	return (zprop_valid_for_type(prop, types, headcheck));
	}

	zprop_type_t
	zfs_prop_get_type(zfs_prop_t prop)
	{
	return (zfs_prop_table[prop].pd_proptype);
	}

	/*
	* Returns TRUE if the property is readonly.
	*/
	boolean_t
	zfs_prop_readonly(zfs_prop_t prop)
	{
	return (zfs_prop_table[prop].pd_attr == PROP_READONLY \|\|
	zfs_prop_table[prop].pd_attr == PROP_ONETIME \|\|
	zfs_prop_table[prop].pd_attr == PROP_ONETIME_DEFAULT);
	}

	/*
	* Returns TRUE if the property is visible (not hidden).
	*/
	boolean_t
	zfs_prop_visible(zfs_prop_t prop)
	{
	return (zfs_prop_table[prop].pd_visible &&
	zfs_prop_table[prop].pd_zfs_mod_supported);
	}

	/*
	* Returns TRUE if the property is only allowed to be set once.
	*/
	boolean_t
	zfs_prop_setonce(zfs_prop_t prop)
	{
	return (zfs_prop_table[prop].pd_attr == PROP_ONETIME \|\|
	zfs_prop_table[prop].pd_attr == PROP_ONETIME_DEFAULT);
	}

	const char *
	zfs_prop_default_string(zfs_prop_t prop)
	{
	return (zfs_prop_table[prop].pd_strdefault);
	}

	uint64_t
	zfs_prop_default_numeric(zfs_prop_t prop)
	{
	return (zfs_prop_table[prop].pd_numdefault);
	}

	/*
	* Given a dataset property ID, returns the corresponding name.
	* Assuming the zfs dataset property ID is valid.
	*/
	const char *
	zfs_prop_to_name(zfs_prop_t prop)
	{
	return (zfs_prop_table[prop].pd_name);
	}

	/*
	* Returns TRUE if the property is inheritable.
	*/
	boolean_t
	zfs_prop_inheritable(zfs_prop_t prop)
	{
	return (zfs_prop_table[prop].pd_attr == PROP_INHERIT \|\|
	zfs_prop_table[prop].pd_attr == PROP_ONETIME);
	}

	/*
	* Returns TRUE if property is one of the encryption properties that requires
	* a loaded encryption key to modify.
	*/
	boolean_t
	zfs_prop_encryption_key_param(zfs_prop_t prop)
	{
	/*
	* keylocation does not count as an encryption property. It can be
	* changed at will without needing the master keys.
	*/
	return (prop == ZFS_PROP_PBKDF2_SALT \|\| prop == ZFS_PROP_PBKDF2_ITERS \|\|
	prop == ZFS_PROP_KEYFORMAT);
	}

	/*
	* Helper function used by both kernelspace and userspace to check the
	* keylocation property. If encrypted is set, the keylocation must be valid
	* for an encrypted dataset.
	*/
	boolean_t
	zfs_prop_valid_keylocation(const char *str, boolean_t encrypted)
	{
	if (strcmp("none", str) == 0)
	return (!encrypted);
	else if (strcmp("prompt", str) == 0)
	return (B_TRUE);
	else if (strlen(str) > 8 && strncmp("file:///", str, 8) == 0)
	return (B_TRUE);

	return (B_FALSE);
	}


	#ifndef _KERNEL
	#include <libzfs.h>

	/*
	* Returns a string describing the set of acceptable values for the given
	* zfs property, or NULL if it cannot be set.
	*/
	const char *
	zfs_prop_values(zfs_prop_t prop)
	{
	return (zfs_prop_table[prop].pd_values);
	}

	/*
	* Returns TRUE if this property is a string type. Note that index types
	* (compression, checksum) are treated as strings in userland, even though they
	* are stored numerically on disk.
	*/
	int
	zfs_prop_is_string(zfs_prop_t prop)
	{
	return (zfs_prop_table[prop].pd_proptype == PROP_TYPE_STRING \|\|
	zfs_prop_table[prop].pd_proptype == PROP_TYPE_INDEX);
	}

	/*
	* Returns the column header for the given property. Used only in
	* 'zfs list -o', but centralized here with the other property information.
	*/
	const char *
	zfs_prop_column_name(zfs_prop_t prop)
	{
	return (zfs_prop_table[prop].pd_colname);
	}

	/*
	* Returns whether the given property should be displayed right-justified for
	* 'zfs list'.
	*/
	boolean_t
	zfs_prop_align_right(zfs_prop_t prop)
	{
	return (zfs_prop_table[prop].pd_rightalign);
	}

	#endif

	#if defined(_KERNEL)

	#include <sys/simd.h>

	#if defined(HAVE_KERNEL_FPU_INTERNAL)
	union fpregs_state **zfs_kfpu_fpregs;
	EXPORT_SYMBOL(zfs_kfpu_fpregs);
	#endif /* HAVE_KERNEL_FPU_INTERNAL */

	static int __init
	zcommon_init(void)
	{
	int error = kfpu_init();
	if (error)
	return (error);

	fletcher_4_init();

	return (0);
	}

	static void __exit
	zcommon_fini(void)
	{
	fletcher_4_fini();
	kfpu_fini();
	}

	module_init(zcommon_init);
	module_exit(zcommon_fini);

	#endif

	ZFS_MODULE_DESCRIPTION("Generic ZFS support");
	ZFS_MODULE_AUTHOR(ZFS_META_AUTHOR);
	ZFS_MODULE_LICENSE(ZFS_META_LICENSE);
	ZFS_MODULE_VERSION(ZFS_META_VERSION "-" ZFS_META_RELEASE);

	/* zfs dataset property functions */
	EXPORT_SYMBOL(zfs_userquota_prop_prefixes);
	EXPORT_SYMBOL(zfs_prop_init);
	EXPORT_SYMBOL(zfs_prop_get_type);
	EXPORT_SYMBOL(zfs_prop_get_table);
	EXPORT_SYMBOL(zfs_prop_delegatable);
	EXPORT_SYMBOL(zfs_prop_visible);

	/* Dataset property functions shared between libzfs and kernel. */
	EXPORT_SYMBOL(zfs_prop_default_string);
	EXPORT_SYMBOL(zfs_prop_default_numeric);
	EXPORT_SYMBOL(zfs_prop_readonly);
	EXPORT_SYMBOL(zfs_prop_inheritable);
	EXPORT_SYMBOL(zfs_prop_encryption_key_param);
	EXPORT_SYMBOL(zfs_prop_valid_keylocation);
	EXPORT_SYMBOL(zfs_prop_setonce);
	EXPORT_SYMBOL(zfs_prop_to_name);
	EXPORT_SYMBOL(zfs_name_to_prop);
	EXPORT_SYMBOL(zfs_prop_user);
	EXPORT_SYMBOL(zfs_prop_userquota);
	EXPORT_SYMBOL(zfs_prop_index_to_string);
	EXPORT_SYMBOL(zfs_prop_string_to_index);
	EXPORT_SYMBOL(zfs_prop_valid_for_type);
	EXPORT_SYMBOL(zfs_prop_written);
	Index: vendor-sys/openzfs/dist/module/zfs/dmu_objset.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/zfs/dmu_objset.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/zfs/dmu_objset.c (revision 366774)
	@@ -1,2992 +1,2992 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/

	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2012, 2018 by Delphix. All rights reserved.
	* Copyright (c) 2013 by Saso Kiselkov. All rights reserved.
	* Copyright (c) 2013, Joyent, Inc. All rights reserved.
	* Copyright (c) 2014 Spectra Logic Corporation, All rights reserved.
	* Copyright (c) 2015, STRATO AG, Inc. All rights reserved.
	* Copyright (c) 2016 Actifio, Inc. All rights reserved.
	* Copyright 2017 Nexenta Systems, Inc.
	* Copyright (c) 2017 Open-E, Inc. All Rights Reserved.
	* Copyright (c) 2018, loli10K <ezomori.nozomu@gmail.com>. All rights reserved.
	* Copyright (c) 2019, Klara Inc.
	* Copyright (c) 2019, Allan Jude
	*/

	/* Portions Copyright 2010 Robert Milkowski */

	#include <sys/cred.h>
	#include <sys/zfs_context.h>
	#include <sys/dmu_objset.h>
	#include <sys/dsl_dir.h>
	#include <sys/dsl_dataset.h>
	#include <sys/dsl_prop.h>
	#include <sys/dsl_pool.h>
	#include <sys/dsl_synctask.h>
	#include <sys/dsl_deleg.h>
	#include <sys/dnode.h>
	#include <sys/dbuf.h>
	#include <sys/zvol.h>
	#include <sys/dmu_tx.h>
	#include <sys/zap.h>
	#include <sys/zil.h>
	#include <sys/dmu_impl.h>
	#include <sys/zfs_ioctl.h>
	#include <sys/sa.h>
	#include <sys/zfs_onexit.h>
	#include <sys/dsl_destroy.h>
	#include <sys/vdev.h>
	#include <sys/zfeature.h>
	#include <sys/policy.h>
	#include <sys/spa_impl.h>
	#include <sys/dmu_recv.h>
	#include <sys/zfs_project.h>
	#include "zfs_namecheck.h"

	/*
	* Needed to close a window in dnode_move() that allows the objset to be freed
	* before it can be safely accessed.
	*/
	krwlock_t os_lock;

	/*
	* Tunable to overwrite the maximum number of threads for the parallelization
	* of dmu_objset_find_dp, needed to speed up the import of pools with many
	* datasets.
	* Default is 4 times the number of leaf vdevs.
	*/
	int dmu_find_threads = 0;

	/*
	* Backfill lower metadnode objects after this many have been freed.
	* Backfilling negatively impacts object creation rates, so only do it
	* if there are enough holes to fill.
	*/
	int dmu_rescan_dnode_threshold = 1 << DN_MAX_INDBLKSHIFT;

	static char *upgrade_tag = "upgrade_tag";

	static void dmu_objset_find_dp_cb(void *arg);

	static void dmu_objset_upgrade(objset_t *os, dmu_objset_upgrade_cb_t cb);
	static void dmu_objset_upgrade_stop(objset_t *os);

	void
	dmu_objset_init(void)
	{
	rw_init(&os_lock, NULL, RW_DEFAULT, NULL);
	}

	void
	dmu_objset_fini(void)
	{
	rw_destroy(&os_lock);
	}

	spa_t *
	dmu_objset_spa(objset_t *os)
	{
	return (os->os_spa);
	}

	zilog_t *
	dmu_objset_zil(objset_t *os)
	{
	return (os->os_zil);
	}

	dsl_pool_t *
	dmu_objset_pool(objset_t *os)
	{
	dsl_dataset_t *ds;

	if ((ds = os->os_dsl_dataset) != NULL && ds->ds_dir)
	return (ds->ds_dir->dd_pool);
	else
	return (spa_get_dsl(os->os_spa));
	}

	dsl_dataset_t *
	dmu_objset_ds(objset_t *os)
	{
	return (os->os_dsl_dataset);
	}

	dmu_objset_type_t
	dmu_objset_type(objset_t *os)
	{
	return (os->os_phys->os_type);
	}

	void
	dmu_objset_name(objset_t os, char buf)
	{
	dsl_dataset_name(os->os_dsl_dataset, buf);
	}

	uint64_t
	dmu_objset_id(objset_t *os)
	{
	dsl_dataset_t *ds = os->os_dsl_dataset;

	return (ds ? ds->ds_object : 0);
	}

	uint64_t
	dmu_objset_dnodesize(objset_t *os)
	{
	return (os->os_dnodesize);
	}

	zfs_sync_type_t
	dmu_objset_syncprop(objset_t *os)
	{
	return (os->os_sync);
	}

	zfs_logbias_op_t
	dmu_objset_logbias(objset_t *os)
	{
	return (os->os_logbias);
	}

	static void
	checksum_changed_cb(void *arg, uint64_t newval)
	{
	objset_t *os = arg;

	/*
	* Inheritance should have been done by now.
	*/
	ASSERT(newval != ZIO_CHECKSUM_INHERIT);

	os->os_checksum = zio_checksum_select(newval, ZIO_CHECKSUM_ON_VALUE);
	}

	static void
	compression_changed_cb(void *arg, uint64_t newval)
	{
	objset_t *os = arg;

	/*
	* Inheritance and range checking should have been done by now.
	*/
	ASSERT(newval != ZIO_COMPRESS_INHERIT);

	os->os_compress = zio_compress_select(os->os_spa,
	ZIO_COMPRESS_ALGO(newval), ZIO_COMPRESS_ON);
	os->os_complevel = zio_complevel_select(os->os_spa, os->os_compress,
	ZIO_COMPRESS_LEVEL(newval), ZIO_COMPLEVEL_DEFAULT);
	}

	static void
	copies_changed_cb(void *arg, uint64_t newval)
	{
	objset_t *os = arg;

	/*
	* Inheritance and range checking should have been done by now.
	*/
	ASSERT(newval > 0);
	ASSERT(newval <= spa_max_replication(os->os_spa));

	os->os_copies = newval;
	}

	static void
	dedup_changed_cb(void *arg, uint64_t newval)
	{
	objset_t *os = arg;
	spa_t *spa = os->os_spa;
	enum zio_checksum checksum;

	/*
	* Inheritance should have been done by now.
	*/
	ASSERT(newval != ZIO_CHECKSUM_INHERIT);

	checksum = zio_checksum_dedup_select(spa, newval, ZIO_CHECKSUM_OFF);

	os->os_dedup_checksum = checksum & ZIO_CHECKSUM_MASK;
	os->os_dedup_verify = !!(checksum & ZIO_CHECKSUM_VERIFY);
	}

	static void
	primary_cache_changed_cb(void *arg, uint64_t newval)
	{
	objset_t *os = arg;

	/*
	* Inheritance and range checking should have been done by now.
	*/
	ASSERT(newval == ZFS_CACHE_ALL \|\| newval == ZFS_CACHE_NONE \|\|
	newval == ZFS_CACHE_METADATA);

	os->os_primary_cache = newval;
	}

	static void
	secondary_cache_changed_cb(void *arg, uint64_t newval)
	{
	objset_t *os = arg;

	/*
	* Inheritance and range checking should have been done by now.
	*/
	ASSERT(newval == ZFS_CACHE_ALL \|\| newval == ZFS_CACHE_NONE \|\|
	newval == ZFS_CACHE_METADATA);

	os->os_secondary_cache = newval;
	}

	static void
	sync_changed_cb(void *arg, uint64_t newval)
	{
	objset_t *os = arg;

	/*
	* Inheritance and range checking should have been done by now.
	*/
	ASSERT(newval == ZFS_SYNC_STANDARD \|\| newval == ZFS_SYNC_ALWAYS \|\|
	newval == ZFS_SYNC_DISABLED);

	os->os_sync = newval;
	if (os->os_zil)
	zil_set_sync(os->os_zil, newval);
	}

	static void
	redundant_metadata_changed_cb(void *arg, uint64_t newval)
	{
	objset_t *os = arg;

	/*
	* Inheritance and range checking should have been done by now.
	*/
	ASSERT(newval == ZFS_REDUNDANT_METADATA_ALL \|\|
	newval == ZFS_REDUNDANT_METADATA_MOST);

	os->os_redundant_metadata = newval;
	}

	static void
	dnodesize_changed_cb(void *arg, uint64_t newval)
	{
	objset_t *os = arg;

	switch (newval) {
	case ZFS_DNSIZE_LEGACY:
	os->os_dnodesize = DNODE_MIN_SIZE;
	break;
	case ZFS_DNSIZE_AUTO:
	/*
	* Choose a dnode size that will work well for most
	* workloads if the user specified "auto". Future code
	* improvements could dynamically select a dnode size
	* based on observed workload patterns.
	*/
	os->os_dnodesize = DNODE_MIN_SIZE * 2;
	break;
	case ZFS_DNSIZE_1K:
	case ZFS_DNSIZE_2K:
	case ZFS_DNSIZE_4K:
	case ZFS_DNSIZE_8K:
	case ZFS_DNSIZE_16K:
	os->os_dnodesize = newval;
	break;
	}
	}

	static void
	smallblk_changed_cb(void *arg, uint64_t newval)
	{
	objset_t *os = arg;

	/*
	* Inheritance and range checking should have been done by now.
	*/
	ASSERT(newval <= SPA_OLD_MAXBLOCKSIZE);
	ASSERT(ISP2(newval));

	os->os_zpl_special_smallblock = newval;
	}

	static void
	logbias_changed_cb(void *arg, uint64_t newval)
	{
	objset_t *os = arg;

	ASSERT(newval == ZFS_LOGBIAS_LATENCY \|\|
	newval == ZFS_LOGBIAS_THROUGHPUT);
	os->os_logbias = newval;
	if (os->os_zil)
	zil_set_logbias(os->os_zil, newval);
	}

	static void
	recordsize_changed_cb(void *arg, uint64_t newval)
	{
	objset_t *os = arg;

	os->os_recordsize = newval;
	}

	void
	dmu_objset_byteswap(void *buf, size_t size)
	{
	objset_phys_t *osp = buf;

	ASSERT(size == OBJSET_PHYS_SIZE_V1 \|\| size == OBJSET_PHYS_SIZE_V2 \|\|
	size == sizeof (objset_phys_t));
	dnode_byteswap(&osp->os_meta_dnode);
	byteswap_uint64_array(&osp->os_zil_header, sizeof (zil_header_t));
	osp->os_type = BSWAP_64(osp->os_type);
	osp->os_flags = BSWAP_64(osp->os_flags);
	if (size >= OBJSET_PHYS_SIZE_V2) {
	dnode_byteswap(&osp->os_userused_dnode);
	dnode_byteswap(&osp->os_groupused_dnode);
	if (size >= sizeof (objset_phys_t))
	dnode_byteswap(&osp->os_projectused_dnode);
	}
	}

	/*
	* The hash is a CRC-based hash of the objset_t pointer and the object number.
	*/
	static uint64_t
	dnode_hash(const objset_t *os, uint64_t obj)
	{
	uintptr_t osv = (uintptr_t)os;
	uint64_t crc = -1ULL;

	ASSERT(zfs_crc64_table[128] == ZFS_CRC64_POLY);
	/*
	* The low 6 bits of the pointer don't have much entropy, because
	* the objset_t is larger than 2^6 bytes long.
	*/
	crc = (crc >> 8) ^ zfs_crc64_table[(crc ^ (osv >> 6)) & 0xFF];
	crc = (crc >> 8) ^ zfs_crc64_table[(crc ^ (obj >> 0)) & 0xFF];
	crc = (crc >> 8) ^ zfs_crc64_table[(crc ^ (obj >> 8)) & 0xFF];
	crc = (crc >> 8) ^ zfs_crc64_table[(crc ^ (obj >> 16)) & 0xFF];

	crc ^= (osv>>14) ^ (obj>>24);

	return (crc);
	}

	static unsigned int
	dnode_multilist_index_func(multilist_t ml, void obj)
	{
	dnode_t *dn = obj;
	return (dnode_hash(dn->dn_objset, dn->dn_object) %
	multilist_get_num_sublists(ml));
	}

	/*
	* Instantiates the objset_t in-memory structure corresponding to the
	* objset_phys_t that's pointed to by the specified blkptr_t.
	*/
	int
	dmu_objset_open_impl(spa_t spa, dsl_dataset_t ds, blkptr_t *bp,
	objset_t **osp)
	{
	objset_t *os;
	int i, err;

	ASSERT(ds == NULL \|\| MUTEX_HELD(&ds->ds_opening_lock));
	ASSERT(!BP_IS_REDACTED(bp));

	/*
	* We need the pool config lock to get properties.
	*/
	ASSERT(ds == NULL \|\| dsl_pool_config_held(ds->ds_dir->dd_pool));

	/*
	* The $ORIGIN dataset (if it exists) doesn't have an associated
	* objset, so there's no reason to open it. The $ORIGIN dataset
	* will not exist on pools older than SPA_VERSION_ORIGIN.
	*/
	if (ds != NULL && spa_get_dsl(spa) != NULL &&
	spa_get_dsl(spa)->dp_origin_snap != NULL) {
	ASSERT3P(ds->ds_dir, !=,
	spa_get_dsl(spa)->dp_origin_snap->ds_dir);
	}

	os = kmem_zalloc(sizeof (objset_t), KM_SLEEP);
	os->os_dsl_dataset = ds;
	os->os_spa = spa;
	os->os_rootbp = bp;
	if (!BP_IS_HOLE(os->os_rootbp)) {
	arc_flags_t aflags = ARC_FLAG_WAIT;
	zbookmark_phys_t zb;
	int size;
	enum zio_flag zio_flags = ZIO_FLAG_CANFAIL;
	SET_BOOKMARK(&zb, ds ? ds->ds_object : DMU_META_OBJSET,
	ZB_ROOT_OBJECT, ZB_ROOT_LEVEL, ZB_ROOT_BLKID);

	if (DMU_OS_IS_L2CACHEABLE(os))
	aflags \|= ARC_FLAG_L2CACHE;

	if (ds != NULL && ds->ds_dir->dd_crypto_obj != 0) {
	ASSERT3U(BP_GET_COMPRESS(bp), ==, ZIO_COMPRESS_OFF);
	ASSERT(BP_IS_AUTHENTICATED(bp));
	zio_flags \|= ZIO_FLAG_RAW;
	}

	dprintf_bp(os->os_rootbp, "reading %s", "");
	err = arc_read(NULL, spa, os->os_rootbp,
	arc_getbuf_func, &os->os_phys_buf,
	ZIO_PRIORITY_SYNC_READ, zio_flags, &aflags, &zb);
	if (err != 0) {
	kmem_free(os, sizeof (objset_t));
	/* convert checksum errors into IO errors */
	if (err == ECKSUM)
	err = SET_ERROR(EIO);
	return (err);
	}

	if (spa_version(spa) < SPA_VERSION_USERSPACE)
	size = OBJSET_PHYS_SIZE_V1;
	else if (!spa_feature_is_enabled(spa,
	SPA_FEATURE_PROJECT_QUOTA))
	size = OBJSET_PHYS_SIZE_V2;
	else
	size = sizeof (objset_phys_t);

	/* Increase the blocksize if we are permitted. */
	if (arc_buf_size(os->os_phys_buf) < size) {
	arc_buf_t *buf = arc_alloc_buf(spa, &os->os_phys_buf,
	ARC_BUFC_METADATA, size);
	bzero(buf->b_data, size);
	bcopy(os->os_phys_buf->b_data, buf->b_data,
	arc_buf_size(os->os_phys_buf));
	arc_buf_destroy(os->os_phys_buf, &os->os_phys_buf);
	os->os_phys_buf = buf;
	}

	os->os_phys = os->os_phys_buf->b_data;
	os->os_flags = os->os_phys->os_flags;
	} else {
	int size = spa_version(spa) >= SPA_VERSION_USERSPACE ?
	sizeof (objset_phys_t) : OBJSET_PHYS_SIZE_V1;
	os->os_phys_buf = arc_alloc_buf(spa, &os->os_phys_buf,
	ARC_BUFC_METADATA, size);
	os->os_phys = os->os_phys_buf->b_data;
	bzero(os->os_phys, size);
	}
	/*
	* These properties will be filled in by the logic in zfs_get_zplprop()
	* when they are queried for the first time.
	*/
	os->os_version = OBJSET_PROP_UNINITIALIZED;
	os->os_normalization = OBJSET_PROP_UNINITIALIZED;
	os->os_utf8only = OBJSET_PROP_UNINITIALIZED;
	os->os_casesensitivity = OBJSET_PROP_UNINITIALIZED;

	/*
	* Note: the changed_cb will be called once before the register
	* func returns, thus changing the checksum/compression from the
	* default (fletcher2/off). Snapshots don't need to know about
	* checksum/compression/copies.
	*/
	if (ds != NULL) {
	os->os_encrypted = (ds->ds_dir->dd_crypto_obj != 0);

	err = dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_PRIMARYCACHE),
	primary_cache_changed_cb, os);
	if (err == 0) {
	err = dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_SECONDARYCACHE),
	secondary_cache_changed_cb, os);
	}
	if (!ds->ds_is_snapshot) {
	if (err == 0) {
	err = dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_CHECKSUM),
	checksum_changed_cb, os);
	}
	if (err == 0) {
	err = dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_COMPRESSION),
	compression_changed_cb, os);
	}
	if (err == 0) {
	err = dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_COPIES),
	copies_changed_cb, os);
	}
	if (err == 0) {
	err = dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_DEDUP),
	dedup_changed_cb, os);
	}
	if (err == 0) {
	err = dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_LOGBIAS),
	logbias_changed_cb, os);
	}
	if (err == 0) {
	err = dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_SYNC),
	sync_changed_cb, os);
	}
	if (err == 0) {
	err = dsl_prop_register(ds,
	zfs_prop_to_name(
	ZFS_PROP_REDUNDANT_METADATA),
	redundant_metadata_changed_cb, os);
	}
	if (err == 0) {
	err = dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_RECORDSIZE),
	recordsize_changed_cb, os);
	}
	if (err == 0) {
	err = dsl_prop_register(ds,
	zfs_prop_to_name(ZFS_PROP_DNODESIZE),
	dnodesize_changed_cb, os);
	}
	if (err == 0) {
	err = dsl_prop_register(ds,
	zfs_prop_to_name(
	ZFS_PROP_SPECIAL_SMALL_BLOCKS),
	smallblk_changed_cb, os);
	}
	}
	if (err != 0) {
	arc_buf_destroy(os->os_phys_buf, &os->os_phys_buf);
	kmem_free(os, sizeof (objset_t));
	return (err);
	}
	} else {
	/* It's the meta-objset. */
	os->os_checksum = ZIO_CHECKSUM_FLETCHER_4;
	os->os_compress = ZIO_COMPRESS_ON;
	os->os_complevel = ZIO_COMPLEVEL_DEFAULT;
	os->os_encrypted = B_FALSE;
	os->os_copies = spa_max_replication(spa);
	os->os_dedup_checksum = ZIO_CHECKSUM_OFF;
	os->os_dedup_verify = B_FALSE;
	os->os_logbias = ZFS_LOGBIAS_LATENCY;
	os->os_sync = ZFS_SYNC_STANDARD;
	os->os_primary_cache = ZFS_CACHE_ALL;
	os->os_secondary_cache = ZFS_CACHE_ALL;
	os->os_dnodesize = DNODE_MIN_SIZE;
	}

	if (ds == NULL \|\| !ds->ds_is_snapshot)
	os->os_zil_header = os->os_phys->os_zil_header;
	os->os_zil = zil_alloc(os, &os->os_zil_header);

	for (i = 0; i < TXG_SIZE; i++) {
	os->os_dirty_dnodes[i] = multilist_create(sizeof (dnode_t),
	offsetof(dnode_t, dn_dirty_link[i]),
	dnode_multilist_index_func);
	}
	list_create(&os->os_dnodes, sizeof (dnode_t),
	offsetof(dnode_t, dn_link));
	list_create(&os->os_downgraded_dbufs, sizeof (dmu_buf_impl_t),
	offsetof(dmu_buf_impl_t, db_link));

	list_link_init(&os->os_evicting_node);

	mutex_init(&os->os_lock, NULL, MUTEX_DEFAULT, NULL);
	mutex_init(&os->os_userused_lock, NULL, MUTEX_DEFAULT, NULL);
	mutex_init(&os->os_obj_lock, NULL, MUTEX_DEFAULT, NULL);
	mutex_init(&os->os_user_ptr_lock, NULL, MUTEX_DEFAULT, NULL);
	os->os_obj_next_percpu_len = boot_ncpus;
	os->os_obj_next_percpu = kmem_zalloc(os->os_obj_next_percpu_len *
	sizeof (os->os_obj_next_percpu[0]), KM_SLEEP);

	dnode_special_open(os, &os->os_phys->os_meta_dnode,
	DMU_META_DNODE_OBJECT, &os->os_meta_dnode);
	if (OBJSET_BUF_HAS_USERUSED(os->os_phys_buf)) {
	dnode_special_open(os, &os->os_phys->os_userused_dnode,
	DMU_USERUSED_OBJECT, &os->os_userused_dnode);
	dnode_special_open(os, &os->os_phys->os_groupused_dnode,
	DMU_GROUPUSED_OBJECT, &os->os_groupused_dnode);
	if (OBJSET_BUF_HAS_PROJECTUSED(os->os_phys_buf))
	dnode_special_open(os,
	&os->os_phys->os_projectused_dnode,
	DMU_PROJECTUSED_OBJECT, &os->os_projectused_dnode);
	}

	mutex_init(&os->os_upgrade_lock, NULL, MUTEX_DEFAULT, NULL);

	*osp = os;
	return (0);
	}

	int
	dmu_objset_from_ds(dsl_dataset_t ds, objset_t *osp)
	{
	int err = 0;

	/*
	* We need the pool_config lock to manipulate the dsl_dataset_t.
	* Even if the dataset is long-held, we need the pool_config lock
	* to open the objset, as it needs to get properties.
	*/
	ASSERT(dsl_pool_config_held(ds->ds_dir->dd_pool));

	mutex_enter(&ds->ds_opening_lock);
	if (ds->ds_objset == NULL) {
	objset_t *os;
	rrw_enter(&ds->ds_bp_rwlock, RW_READER, FTAG);
	err = dmu_objset_open_impl(dsl_dataset_get_spa(ds),
	ds, dsl_dataset_get_blkptr(ds), &os);
	rrw_exit(&ds->ds_bp_rwlock, FTAG);

	if (err == 0) {
	mutex_enter(&ds->ds_lock);
	ASSERT(ds->ds_objset == NULL);
	ds->ds_objset = os;
	mutex_exit(&ds->ds_lock);
	}
	}
	*osp = ds->ds_objset;
	mutex_exit(&ds->ds_opening_lock);
	return (err);
	}

	/*
	* Holds the pool while the objset is held. Therefore only one objset
	* can be held at a time.
	*/
	int
	dmu_objset_hold_flags(const char name, boolean_t decrypt, void tag,
	objset_t **osp)
	{
	dsl_pool_t *dp;
	dsl_dataset_t *ds;
	int err;
	ds_hold_flags_t flags = (decrypt) ? DS_HOLD_FLAG_DECRYPT : 0;

	err = dsl_pool_hold(name, tag, &dp);
	if (err != 0)
	return (err);
	err = dsl_dataset_hold_flags(dp, name, flags, tag, &ds);
	if (err != 0) {
	dsl_pool_rele(dp, tag);
	return (err);
	}

	err = dmu_objset_from_ds(ds, osp);
	if (err != 0) {
	dsl_dataset_rele(ds, tag);
	dsl_pool_rele(dp, tag);
	}

	return (err);
	}

	int
	dmu_objset_hold(const char name, void tag, objset_t **osp)
	{
	return (dmu_objset_hold_flags(name, B_FALSE, tag, osp));
	}

	static int
	dmu_objset_own_impl(dsl_dataset_t *ds, dmu_objset_type_t type,
	boolean_t readonly, boolean_t decrypt, void tag, objset_t *osp)
	{
	int err;

	err = dmu_objset_from_ds(ds, osp);
	if (err != 0) {
	return (err);
	} else if (type != DMU_OST_ANY && type != (*osp)->os_phys->os_type) {
	return (SET_ERROR(EINVAL));
	} else if (!readonly && dsl_dataset_is_snapshot(ds)) {
	return (SET_ERROR(EROFS));
	} else if (!readonly && decrypt &&
	dsl_dir_incompatible_encryption_version(ds->ds_dir)) {
	return (SET_ERROR(EROFS));
	}

	/* if we are decrypting, we can now check MACs in os->os_phys_buf */
	if (decrypt && arc_is_unauthenticated((*osp)->os_phys_buf)) {
	zbookmark_phys_t zb;

	SET_BOOKMARK(&zb, ds->ds_object, ZB_ROOT_OBJECT,
	ZB_ROOT_LEVEL, ZB_ROOT_BLKID);
	err = arc_untransform((osp)->os_phys_buf, (osp)->os_spa,
	&zb, B_FALSE);
	if (err != 0)
	return (err);

	ASSERT0(arc_is_unauthenticated((*osp)->os_phys_buf));
	}

	return (0);
	}

	/*
	* dsl_pool must not be held when this is called.
	* Upon successful return, there will be a longhold on the dataset,
	* and the dsl_pool will not be held.
	*/
	int
	dmu_objset_own(const char *name, dmu_objset_type_t type,
	boolean_t readonly, boolean_t decrypt, void tag, objset_t *osp)
	{
	dsl_pool_t *dp;
	dsl_dataset_t *ds;
	int err;
	ds_hold_flags_t flags = (decrypt) ? DS_HOLD_FLAG_DECRYPT : 0;

	err = dsl_pool_hold(name, FTAG, &dp);
	if (err != 0)
	return (err);
	err = dsl_dataset_own(dp, name, flags, tag, &ds);
	if (err != 0) {
	dsl_pool_rele(dp, FTAG);
	return (err);
	}
	err = dmu_objset_own_impl(ds, type, readonly, decrypt, tag, osp);
	if (err != 0) {
	dsl_dataset_disown(ds, flags, tag);
	dsl_pool_rele(dp, FTAG);
	return (err);
	}

	/*
	* User accounting requires the dataset to be decrypted and rw.
	* We also don't begin user accounting during claiming to help
	* speed up pool import times and to keep this txg reserved
	* completely for recovery work.
	*/
	if ((dmu_objset_userobjspace_upgradable(*osp) \|\|
	dmu_objset_projectquota_upgradable(*osp)) &&
	!readonly && !dp->dp_spa->spa_claiming &&
	(ds->ds_dir->dd_crypto_obj == 0 \|\| decrypt))
	dmu_objset_id_quota_upgrade(*osp);

	dsl_pool_rele(dp, FTAG);
	return (0);
	}

	int
	dmu_objset_own_obj(dsl_pool_t *dp, uint64_t obj, dmu_objset_type_t type,
	boolean_t readonly, boolean_t decrypt, void tag, objset_t *osp)
	{
	dsl_dataset_t *ds;
	int err;
	ds_hold_flags_t flags = (decrypt) ? DS_HOLD_FLAG_DECRYPT : 0;

	err = dsl_dataset_own_obj(dp, obj, flags, tag, &ds);
	if (err != 0)
	return (err);

	err = dmu_objset_own_impl(ds, type, readonly, decrypt, tag, osp);
	if (err != 0) {
	dsl_dataset_disown(ds, flags, tag);
	return (err);
	}

	return (0);
	}

	void
	dmu_objset_rele_flags(objset_t os, boolean_t decrypt, void tag)
	{
	ds_hold_flags_t flags = (decrypt) ? DS_HOLD_FLAG_DECRYPT : 0;

	dsl_pool_t *dp = dmu_objset_pool(os);
	dsl_dataset_rele_flags(os->os_dsl_dataset, flags, tag);
	dsl_pool_rele(dp, tag);
	}

	void
	dmu_objset_rele(objset_t os, void tag)
	{
	dmu_objset_rele_flags(os, B_FALSE, tag);
	}

	/*
	* When we are called, os MUST refer to an objset associated with a dataset
	* that is owned by 'tag'; that is, is held and long held by 'tag' and ds_owner
	* == tag. We will then release and reacquire ownership of the dataset while
	* holding the pool config_rwlock to avoid intervening namespace or ownership
	* changes may occur.
	*
	* This exists solely to accommodate zfs_ioc_userspace_upgrade()'s desire to
	* release the hold on its dataset and acquire a new one on the dataset of the
	* same name so that it can be partially torn down and reconstructed.
	*/
	void
	dmu_objset_refresh_ownership(dsl_dataset_t ds, dsl_dataset_t *newds,
	boolean_t decrypt, void *tag)
	{
	dsl_pool_t *dp;
	char name[ZFS_MAX_DATASET_NAME_LEN];

	VERIFY3P(ds, !=, NULL);
	VERIFY3P(ds->ds_owner, ==, tag);
	VERIFY(dsl_dataset_long_held(ds));

	dsl_dataset_name(ds, name);
	dp = ds->ds_dir->dd_pool;
	dsl_pool_config_enter(dp, FTAG);
	dsl_dataset_disown(ds, decrypt, tag);
	VERIFY0(dsl_dataset_own(dp, name,
	(decrypt) ? DS_HOLD_FLAG_DECRYPT : 0, tag, newds));
	dsl_pool_config_exit(dp, FTAG);
	}

	void
	dmu_objset_disown(objset_t os, boolean_t decrypt, void tag)
	{
	/*
	* Stop upgrading thread
	*/
	dmu_objset_upgrade_stop(os);
	dsl_dataset_disown(os->os_dsl_dataset,
	(decrypt) ? DS_HOLD_FLAG_DECRYPT : 0, tag);
	}

	void
	dmu_objset_evict_dbufs(objset_t *os)
	{
	dnode_t *dn_marker;
	dnode_t *dn;

	dn_marker = kmem_alloc(sizeof (dnode_t), KM_SLEEP);

	mutex_enter(&os->os_lock);
	dn = list_head(&os->os_dnodes);
	while (dn != NULL) {
	/*
	* Skip dnodes without holds. We have to do this dance
	* because dnode_add_ref() only works if there is already a
	* hold. If the dnode has no holds, then it has no dbufs.
	*/
	if (dnode_add_ref(dn, FTAG)) {
	list_insert_after(&os->os_dnodes, dn, dn_marker);
	mutex_exit(&os->os_lock);

	dnode_evict_dbufs(dn);
	dnode_rele(dn, FTAG);

	mutex_enter(&os->os_lock);
	dn = list_next(&os->os_dnodes, dn_marker);
	list_remove(&os->os_dnodes, dn_marker);
	} else {
	dn = list_next(&os->os_dnodes, dn);
	}
	}
	mutex_exit(&os->os_lock);

	kmem_free(dn_marker, sizeof (dnode_t));

	if (DMU_USERUSED_DNODE(os) != NULL) {
	if (DMU_PROJECTUSED_DNODE(os) != NULL)
	dnode_evict_dbufs(DMU_PROJECTUSED_DNODE(os));
	dnode_evict_dbufs(DMU_GROUPUSED_DNODE(os));
	dnode_evict_dbufs(DMU_USERUSED_DNODE(os));
	}
	dnode_evict_dbufs(DMU_META_DNODE(os));
	}

	/*
	* Objset eviction processing is split into into two pieces.
	* The first marks the objset as evicting, evicts any dbufs that
	* have a refcount of zero, and then queues up the objset for the
	* second phase of eviction. Once os->os_dnodes has been cleared by
	* dnode_buf_pageout()->dnode_destroy(), the second phase is executed.
	* The second phase closes the special dnodes, dequeues the objset from
	* the list of those undergoing eviction, and finally frees the objset.
	*
	* NOTE: Due to asynchronous eviction processing (invocation of
	* dnode_buf_pageout()), it is possible for the meta dnode for the
	* objset to have no holds even though os->os_dnodes is not empty.
	*/
	void
	dmu_objset_evict(objset_t *os)
	{
	dsl_dataset_t *ds = os->os_dsl_dataset;

	for (int t = 0; t < TXG_SIZE; t++)
	ASSERT(!dmu_objset_is_dirty(os, t));

	if (ds)
	dsl_prop_unregister_all(ds, os);

	if (os->os_sa)
	sa_tear_down(os);

	dmu_objset_evict_dbufs(os);

	mutex_enter(&os->os_lock);
	spa_evicting_os_register(os->os_spa, os);
	if (list_is_empty(&os->os_dnodes)) {
	mutex_exit(&os->os_lock);
	dmu_objset_evict_done(os);
	} else {
	mutex_exit(&os->os_lock);
	}


	}

	void
	dmu_objset_evict_done(objset_t *os)
	{
	ASSERT3P(list_head(&os->os_dnodes), ==, NULL);

	dnode_special_close(&os->os_meta_dnode);
	if (DMU_USERUSED_DNODE(os)) {
	if (DMU_PROJECTUSED_DNODE(os))
	dnode_special_close(&os->os_projectused_dnode);
	dnode_special_close(&os->os_userused_dnode);
	dnode_special_close(&os->os_groupused_dnode);
	}
	zil_free(os->os_zil);

	arc_buf_destroy(os->os_phys_buf, &os->os_phys_buf);

	/*
	* This is a barrier to prevent the objset from going away in
	* dnode_move() until we can safely ensure that the objset is still in
	* use. We consider the objset valid before the barrier and invalid
	* after the barrier.
	*/
	rw_enter(&os_lock, RW_READER);
	rw_exit(&os_lock);

	kmem_free(os->os_obj_next_percpu,
	os->os_obj_next_percpu_len * sizeof (os->os_obj_next_percpu[0]));

	mutex_destroy(&os->os_lock);
	mutex_destroy(&os->os_userused_lock);
	mutex_destroy(&os->os_obj_lock);
	mutex_destroy(&os->os_user_ptr_lock);
	mutex_destroy(&os->os_upgrade_lock);
	for (int i = 0; i < TXG_SIZE; i++) {
	multilist_destroy(os->os_dirty_dnodes[i]);
	}
	spa_evicting_os_deregister(os->os_spa, os);
	kmem_free(os, sizeof (objset_t));
	}

	inode_timespec_t
	dmu_objset_snap_cmtime(objset_t *os)
	{
	return (dsl_dir_snap_cmtime(os->os_dsl_dataset->ds_dir));
	}

	objset_t *
	dmu_objset_create_impl_dnstats(spa_t spa, dsl_dataset_t ds, blkptr_t *bp,
	dmu_objset_type_t type, int levels, int blksz, int ibs, dmu_tx_t *tx)
	{
	objset_t *os;
	dnode_t *mdn;

	ASSERT(dmu_tx_is_syncing(tx));

	if (blksz == 0)
	blksz = DNODE_BLOCK_SIZE;
	if (ibs == 0)
	ibs = DN_MAX_INDBLKSHIFT;

	if (ds != NULL)
	VERIFY0(dmu_objset_from_ds(ds, &os));
	else
	VERIFY0(dmu_objset_open_impl(spa, NULL, bp, &os));

	mdn = DMU_META_DNODE(os);

	dnode_allocate(mdn, DMU_OT_DNODE, blksz, ibs, DMU_OT_NONE, 0,
	DNODE_MIN_SLOTS, tx);

	/*
	* We don't want to have to increase the meta-dnode's nlevels
	* later, because then we could do it in quiescing context while
	* we are also accessing it in open context.
	*
	* This precaution is not necessary for the MOS (ds == NULL),
	* because the MOS is only updated in syncing context.
	* This is most fortunate: the MOS is the only objset that
	* needs to be synced multiple times as spa_sync() iterates
	* to convergence, so minimizing its dn_nlevels matters.
	*/
	if (ds != NULL) {
	if (levels == 0) {
	levels = 1;

	/*
	* Determine the number of levels necessary for the
	* meta-dnode to contain DN_MAX_OBJECT dnodes. Note
	* that in order to ensure that we do not overflow
	* 64 bits, there has to be a nlevels that gives us a
	* number of blocks > DN_MAX_OBJECT but < 2^64.
	* Therefore, (mdn->dn_indblkshift - SPA_BLKPTRSHIFT)
	* (10) must be less than (64 - log2(DN_MAX_OBJECT))
	* (16).
	*/
	while ((uint64_t)mdn->dn_nblkptr <<
	(mdn->dn_datablkshift - DNODE_SHIFT + (levels - 1) *
	(mdn->dn_indblkshift - SPA_BLKPTRSHIFT)) <
	DN_MAX_OBJECT)
	levels++;
	}

	mdn->dn_next_nlevels[tx->tx_txg & TXG_MASK] =
	mdn->dn_nlevels = levels;
	}

	ASSERT(type != DMU_OST_NONE);
	ASSERT(type != DMU_OST_ANY);
	ASSERT(type < DMU_OST_NUMTYPES);
	os->os_phys->os_type = type;

	/*
	* Enable user accounting if it is enabled and this is not an
	* encrypted receive.
	*/
	if (dmu_objset_userused_enabled(os) &&
	(!os->os_encrypted \|\| !dmu_objset_is_receiving(os))) {
	os->os_phys->os_flags \|= OBJSET_FLAG_USERACCOUNTING_COMPLETE;
	if (dmu_objset_userobjused_enabled(os)) {
	ds->ds_feature_activation[
	SPA_FEATURE_USEROBJ_ACCOUNTING] = (void *)B_TRUE;
	os->os_phys->os_flags \|=
	OBJSET_FLAG_USEROBJACCOUNTING_COMPLETE;
	}
	if (dmu_objset_projectquota_enabled(os)) {
	ds->ds_feature_activation[
	SPA_FEATURE_PROJECT_QUOTA] = (void *)B_TRUE;
	os->os_phys->os_flags \|=
	OBJSET_FLAG_PROJECTQUOTA_COMPLETE;
	}
	os->os_flags = os->os_phys->os_flags;
	}

	dsl_dataset_dirty(ds, tx);

	return (os);
	}

	/* called from dsl for meta-objset */
	objset_t *
	dmu_objset_create_impl(spa_t spa, dsl_dataset_t ds, blkptr_t *bp,
	dmu_objset_type_t type, dmu_tx_t *tx)
	{
	return (dmu_objset_create_impl_dnstats(spa, ds, bp, type, 0, 0, 0, tx));
	}

	typedef struct dmu_objset_create_arg {
	const char *doca_name;
	cred_t *doca_cred;
	proc_t *doca_proc;
	void (doca_userfunc)(objset_t os, void *arg,
	cred_t cr, dmu_tx_t tx);
	void *doca_userarg;
	dmu_objset_type_t doca_type;
	uint64_t doca_flags;
	dsl_crypto_params_t *doca_dcp;
	} dmu_objset_create_arg_t;

	/ARGSUSED/
	static int
	dmu_objset_create_check(void arg, dmu_tx_t tx)
	{
	dmu_objset_create_arg_t *doca = arg;
	dsl_pool_t *dp = dmu_tx_pool(tx);
	dsl_dir_t *pdd;
	dsl_dataset_t *parentds;
	objset_t *parentos;
	const char *tail;
	int error;

	if (strchr(doca->doca_name, '@') != NULL)
	return (SET_ERROR(EINVAL));

	if (strlen(doca->doca_name) >= ZFS_MAX_DATASET_NAME_LEN)
	return (SET_ERROR(ENAMETOOLONG));

	if (dataset_nestcheck(doca->doca_name) != 0)
	return (SET_ERROR(ENAMETOOLONG));

	error = dsl_dir_hold(dp, doca->doca_name, FTAG, &pdd, &tail);
	if (error != 0)
	return (error);
	if (tail == NULL) {
	dsl_dir_rele(pdd, FTAG);
	return (SET_ERROR(EEXIST));
	}

	error = dmu_objset_create_crypt_check(pdd, doca->doca_dcp, NULL);
	if (error != 0) {
	dsl_dir_rele(pdd, FTAG);
	return (error);
	}

	error = dsl_fs_ss_limit_check(pdd, 1, ZFS_PROP_FILESYSTEM_LIMIT, NULL,
	doca->doca_cred, doca->doca_proc);
	if (error != 0) {
	dsl_dir_rele(pdd, FTAG);
	return (error);
	}

	/* can't create below anything but filesystems (eg. no ZVOLs) */
	error = dsl_dataset_hold_obj(pdd->dd_pool,
	dsl_dir_phys(pdd)->dd_head_dataset_obj, FTAG, &parentds);
	if (error != 0) {
	dsl_dir_rele(pdd, FTAG);
	return (error);
	}
	error = dmu_objset_from_ds(parentds, &parentos);
	if (error != 0) {
	dsl_dataset_rele(parentds, FTAG);
	dsl_dir_rele(pdd, FTAG);
	return (error);
	}
	if (dmu_objset_type(parentos) != DMU_OST_ZFS) {
	dsl_dataset_rele(parentds, FTAG);
	dsl_dir_rele(pdd, FTAG);
	return (SET_ERROR(ZFS_ERR_WRONG_PARENT));
	}
	dsl_dataset_rele(parentds, FTAG);
	dsl_dir_rele(pdd, FTAG);

	return (error);
	}

	static void
	dmu_objset_create_sync(void arg, dmu_tx_t tx)
	{
	dmu_objset_create_arg_t *doca = arg;
	dsl_pool_t *dp = dmu_tx_pool(tx);
	spa_t *spa = dp->dp_spa;
	dsl_dir_t *pdd;
	const char *tail;
	dsl_dataset_t *ds;
	uint64_t obj;
	blkptr_t *bp;
	objset_t *os;
	zio_t *rzio;

	VERIFY0(dsl_dir_hold(dp, doca->doca_name, FTAG, &pdd, &tail));

	obj = dsl_dataset_create_sync(pdd, tail, NULL, doca->doca_flags,
	doca->doca_cred, doca->doca_dcp, tx);

	VERIFY0(dsl_dataset_hold_obj_flags(pdd->dd_pool, obj,
	DS_HOLD_FLAG_DECRYPT, FTAG, &ds));
	rrw_enter(&ds->ds_bp_rwlock, RW_READER, FTAG);
	bp = dsl_dataset_get_blkptr(ds);
	os = dmu_objset_create_impl(spa, ds, bp, doca->doca_type, tx);
	rrw_exit(&ds->ds_bp_rwlock, FTAG);

	if (doca->doca_userfunc != NULL) {
	doca->doca_userfunc(os, doca->doca_userarg,
	doca->doca_cred, tx);
	}

	/*
	* The doca_userfunc() may write out some data that needs to be
	* encrypted if the dataset is encrypted (specifically the root
	* directory). This data must be written out before the encryption
	* key mapping is removed by dsl_dataset_rele_flags(). Force the
	* I/O to occur immediately by invoking the relevant sections of
	* dsl_pool_sync().
	*/
	if (os->os_encrypted) {
	dsl_dataset_t *tmpds = NULL;
	boolean_t need_sync_done = B_FALSE;

	mutex_enter(&ds->ds_lock);
	ds->ds_owner = FTAG;
	mutex_exit(&ds->ds_lock);

	rzio = zio_root(spa, NULL, NULL, ZIO_FLAG_MUSTSUCCEED);
	tmpds = txg_list_remove_this(&dp->dp_dirty_datasets, ds,
	tx->tx_txg);
	if (tmpds != NULL) {
	dsl_dataset_sync(ds, rzio, tx);
	need_sync_done = B_TRUE;
	}
	VERIFY0(zio_wait(rzio));

	dmu_objset_do_userquota_updates(os, tx);
	taskq_wait(dp->dp_sync_taskq);
	if (txg_list_member(&dp->dp_dirty_datasets, ds, tx->tx_txg)) {
	ASSERT3P(ds->ds_key_mapping, !=, NULL);
	key_mapping_rele(spa, ds->ds_key_mapping, ds);
	}

	rzio = zio_root(spa, NULL, NULL, ZIO_FLAG_MUSTSUCCEED);
	tmpds = txg_list_remove_this(&dp->dp_dirty_datasets, ds,
	tx->tx_txg);
	if (tmpds != NULL) {
	dmu_buf_rele(ds->ds_dbuf, ds);
	dsl_dataset_sync(ds, rzio, tx);
	}
	VERIFY0(zio_wait(rzio));

	if (need_sync_done) {
	ASSERT3P(ds->ds_key_mapping, !=, NULL);
	key_mapping_rele(spa, ds->ds_key_mapping, ds);
	dsl_dataset_sync_done(ds, tx);
	}

	mutex_enter(&ds->ds_lock);
	ds->ds_owner = NULL;
	mutex_exit(&ds->ds_lock);
	}

	spa_history_log_internal_ds(ds, "create", tx, " ");

	dsl_dataset_rele_flags(ds, DS_HOLD_FLAG_DECRYPT, FTAG);
	dsl_dir_rele(pdd, FTAG);
	}

	int
	dmu_objset_create(const char *name, dmu_objset_type_t type, uint64_t flags,
	dsl_crypto_params_t dcp, dmu_objset_create_sync_func_t func, void arg)
	{
	dmu_objset_create_arg_t doca;
	dsl_crypto_params_t tmp_dcp = { 0 };

	doca.doca_name = name;
	doca.doca_cred = CRED();
	doca.doca_proc = curproc;
	doca.doca_flags = flags;
	doca.doca_userfunc = func;
	doca.doca_userarg = arg;
	doca.doca_type = type;

	/*
	* Some callers (mostly for testing) do not provide a dcp on their
	* own but various code inside the sync task will require it to be
	* allocated. Rather than adding NULL checks throughout this code
	* or adding dummy dcp's to all of the callers we simply create a
	* dummy one here and use that. This zero dcp will have the same
	* effect as asking for inheritance of all encryption params.
	*/
	doca.doca_dcp = (dcp != NULL) ? dcp : &tmp_dcp;

	int rv = dsl_sync_task(name,
	dmu_objset_create_check, dmu_objset_create_sync, &doca,
	6, ZFS_SPACE_CHECK_NORMAL);

	if (rv == 0)
	zvol_create_minor(name);
	return (rv);
	}

	typedef struct dmu_objset_clone_arg {
	const char *doca_clone;
	const char *doca_origin;
	cred_t *doca_cred;
	proc_t *doca_proc;
	} dmu_objset_clone_arg_t;

	/ARGSUSED/
	static int
	dmu_objset_clone_check(void arg, dmu_tx_t tx)
	{
	dmu_objset_clone_arg_t *doca = arg;
	dsl_dir_t *pdd;
	const char *tail;
	int error;
	dsl_dataset_t *origin;
	dsl_pool_t *dp = dmu_tx_pool(tx);

	if (strchr(doca->doca_clone, '@') != NULL)
	return (SET_ERROR(EINVAL));

	if (strlen(doca->doca_clone) >= ZFS_MAX_DATASET_NAME_LEN)
	return (SET_ERROR(ENAMETOOLONG));

	error = dsl_dir_hold(dp, doca->doca_clone, FTAG, &pdd, &tail);
	if (error != 0)
	return (error);
	if (tail == NULL) {
	dsl_dir_rele(pdd, FTAG);
	return (SET_ERROR(EEXIST));
	}

	error = dsl_fs_ss_limit_check(pdd, 1, ZFS_PROP_FILESYSTEM_LIMIT, NULL,
	doca->doca_cred, doca->doca_proc);
	if (error != 0) {
	dsl_dir_rele(pdd, FTAG);
	return (SET_ERROR(EDQUOT));
	}

	error = dsl_dataset_hold(dp, doca->doca_origin, FTAG, &origin);
	if (error != 0) {
	dsl_dir_rele(pdd, FTAG);
	return (error);
	}

	/* You can only clone snapshots, not the head datasets. */
	if (!origin->ds_is_snapshot) {
	dsl_dataset_rele(origin, FTAG);
	dsl_dir_rele(pdd, FTAG);
	return (SET_ERROR(EINVAL));
	}

	dsl_dataset_rele(origin, FTAG);
	dsl_dir_rele(pdd, FTAG);

	return (0);
	}

	static void
	dmu_objset_clone_sync(void arg, dmu_tx_t tx)
	{
	dmu_objset_clone_arg_t *doca = arg;
	dsl_pool_t *dp = dmu_tx_pool(tx);
	dsl_dir_t *pdd;
	const char *tail;
	dsl_dataset_t origin, ds;
	uint64_t obj;
	char namebuf[ZFS_MAX_DATASET_NAME_LEN];

	VERIFY0(dsl_dir_hold(dp, doca->doca_clone, FTAG, &pdd, &tail));
	VERIFY0(dsl_dataset_hold(dp, doca->doca_origin, FTAG, &origin));

	obj = dsl_dataset_create_sync(pdd, tail, origin, 0,
	doca->doca_cred, NULL, tx);

	VERIFY0(dsl_dataset_hold_obj(pdd->dd_pool, obj, FTAG, &ds));
	dsl_dataset_name(origin, namebuf);
	spa_history_log_internal_ds(ds, "clone", tx,
	"origin=%s (%llu)", namebuf, (u_longlong_t)origin->ds_object);
	dsl_dataset_rele(ds, FTAG);
	dsl_dataset_rele(origin, FTAG);
	dsl_dir_rele(pdd, FTAG);
	}

	int
	dmu_objset_clone(const char clone, const char origin)
	{
	dmu_objset_clone_arg_t doca;

	doca.doca_clone = clone;
	doca.doca_origin = origin;
	doca.doca_cred = CRED();
	doca.doca_proc = curproc;

	int rv = dsl_sync_task(clone,
	dmu_objset_clone_check, dmu_objset_clone_sync, &doca,
	6, ZFS_SPACE_CHECK_NORMAL);

	if (rv == 0)
	zvol_create_minor(clone);

	return (rv);
	}

	int
	dmu_objset_snapshot_one(const char fsname, const char snapname)
	{
	int err;
	char *longsnap = kmem_asprintf("%s@%s", fsname, snapname);
	nvlist_t *snaps = fnvlist_alloc();

	fnvlist_add_boolean(snaps, longsnap);
	kmem_strfree(longsnap);
	err = dsl_dataset_snapshot(snaps, NULL, NULL);
	fnvlist_free(snaps);
	return (err);
	}

	static void
	dmu_objset_upgrade_task_cb(void *data)
	{
	objset_t *os = data;

	mutex_enter(&os->os_upgrade_lock);
	os->os_upgrade_status = EINTR;
	if (!os->os_upgrade_exit) {
	mutex_exit(&os->os_upgrade_lock);

	os->os_upgrade_status = os->os_upgrade_cb(os);
	mutex_enter(&os->os_upgrade_lock);
	}
	os->os_upgrade_exit = B_TRUE;
	os->os_upgrade_id = 0;
	mutex_exit(&os->os_upgrade_lock);
	dsl_dataset_long_rele(dmu_objset_ds(os), upgrade_tag);
	}

	static void
	dmu_objset_upgrade(objset_t *os, dmu_objset_upgrade_cb_t cb)
	{
	if (os->os_upgrade_id != 0)
	return;

	ASSERT(dsl_pool_config_held(dmu_objset_pool(os)));
	dsl_dataset_long_hold(dmu_objset_ds(os), upgrade_tag);

	mutex_enter(&os->os_upgrade_lock);
	if (os->os_upgrade_id == 0 && os->os_upgrade_status == 0) {
	os->os_upgrade_exit = B_FALSE;
	os->os_upgrade_cb = cb;
	os->os_upgrade_id = taskq_dispatch(
	os->os_spa->spa_upgrade_taskq,
	dmu_objset_upgrade_task_cb, os, TQ_SLEEP);
	if (os->os_upgrade_id == TASKQID_INVALID) {
	dsl_dataset_long_rele(dmu_objset_ds(os), upgrade_tag);
	os->os_upgrade_status = ENOMEM;
	}
	}
	mutex_exit(&os->os_upgrade_lock);
	}

	static void
	dmu_objset_upgrade_stop(objset_t *os)
	{
	mutex_enter(&os->os_upgrade_lock);
	os->os_upgrade_exit = B_TRUE;
	if (os->os_upgrade_id != 0) {
	taskqid_t id = os->os_upgrade_id;

	os->os_upgrade_id = 0;
	mutex_exit(&os->os_upgrade_lock);

	if ((taskq_cancel_id(os->os_spa->spa_upgrade_taskq, id)) == 0) {
	dsl_dataset_long_rele(dmu_objset_ds(os), upgrade_tag);
	}
	txg_wait_synced(os->os_spa->spa_dsl_pool, 0);
	} else {
	mutex_exit(&os->os_upgrade_lock);
	}
	}

	static void
	dmu_objset_sync_dnodes(multilist_sublist_t list, dmu_tx_t tx)
	{
	dnode_t *dn;

	while ((dn = multilist_sublist_head(list)) != NULL) {
	ASSERT(dn->dn_object != DMU_META_DNODE_OBJECT);
	ASSERT(dn->dn_dbuf->db_data_pending);
	/*
	* Initialize dn_zio outside dnode_sync() because the
	* meta-dnode needs to set it outside dnode_sync().
	*/
	dn->dn_zio = dn->dn_dbuf->db_data_pending->dr_zio;
	ASSERT(dn->dn_zio);

	ASSERT3U(dn->dn_nlevels, <=, DN_MAX_LEVELS);
	multilist_sublist_remove(list, dn);

	/*
	* If we are not doing useraccounting (os_synced_dnodes == NULL)
	* we are done with this dnode for this txg. Unset dn_dirty_txg
	* if later txgs aren't dirtying it so that future holders do
	* not get a stale value. Otherwise, we will do this in
	* userquota_updates_task() when processing has completely
	* finished for this txg.
	*/
	multilist_t *newlist = dn->dn_objset->os_synced_dnodes;
	if (newlist != NULL) {
	(void) dnode_add_ref(dn, newlist);
	multilist_insert(newlist, dn);
	} else {
	mutex_enter(&dn->dn_mtx);
	if (dn->dn_dirty_txg == tx->tx_txg)
	dn->dn_dirty_txg = 0;
	mutex_exit(&dn->dn_mtx);
	}

	dnode_sync(dn, tx);
	}
	}

	/* ARGSUSED */
	static void
	dmu_objset_write_ready(zio_t zio, arc_buf_t abuf, void *arg)
	{
	blkptr_t *bp = zio->io_bp;
	objset_t *os = arg;
	dnode_phys_t *dnp = &os->os_phys->os_meta_dnode;
	uint64_t fill = 0;

	ASSERT(!BP_IS_EMBEDDED(bp));
	ASSERT3U(BP_GET_TYPE(bp), ==, DMU_OT_OBJSET);
	ASSERT0(BP_GET_LEVEL(bp));

	/*
	* Update rootbp fill count: it should be the number of objects
	* allocated in the object set (not counting the "special"
	* objects that are stored in the objset_phys_t -- the meta
	* dnode and user/group/project accounting objects).
	*/
	for (int i = 0; i < dnp->dn_nblkptr; i++)
	fill += BP_GET_FILL(&dnp->dn_blkptr[i]);

	BP_SET_FILL(bp, fill);

	if (os->os_dsl_dataset != NULL)
	rrw_enter(&os->os_dsl_dataset->ds_bp_rwlock, RW_WRITER, FTAG);
	os->os_rootbp = bp;
	if (os->os_dsl_dataset != NULL)
	rrw_exit(&os->os_dsl_dataset->ds_bp_rwlock, FTAG);
	}

	/* ARGSUSED */
	static void
	dmu_objset_write_done(zio_t zio, arc_buf_t abuf, void *arg)
	{
	blkptr_t *bp = zio->io_bp;
	blkptr_t *bp_orig = &zio->io_bp_orig;
	objset_t *os = arg;

	if (zio->io_flags & ZIO_FLAG_IO_REWRITE) {
	ASSERT(BP_EQUAL(bp, bp_orig));
	} else {
	dsl_dataset_t *ds = os->os_dsl_dataset;
	dmu_tx_t *tx = os->os_synctx;

	(void) dsl_dataset_block_kill(ds, bp_orig, tx, B_TRUE);
	dsl_dataset_block_born(ds, bp, tx);
	}
	kmem_free(bp, sizeof (*bp));
	}

	typedef struct sync_dnodes_arg {
	multilist_t *sda_list;
	int sda_sublist_idx;
	multilist_t *sda_newlist;
	dmu_tx_t *sda_tx;
	} sync_dnodes_arg_t;

	static void
	sync_dnodes_task(void *arg)
	{
	sync_dnodes_arg_t *sda = arg;

	multilist_sublist_t *ms =
	multilist_sublist_lock(sda->sda_list, sda->sda_sublist_idx);

	dmu_objset_sync_dnodes(ms, sda->sda_tx);

	multilist_sublist_unlock(ms);

	kmem_free(sda, sizeof (*sda));
	}


	/* called from dsl */
	void
	dmu_objset_sync(objset_t os, zio_t pio, dmu_tx_t *tx)
	{
	int txgoff;
	zbookmark_phys_t zb;
	zio_prop_t zp;
	zio_t *zio;
	list_t *list;
	dbuf_dirty_record_t *dr;
	int num_sublists;
	multilist_t *ml;
	blkptr_t blkptr_copy = kmem_alloc(sizeof (os->os_rootbp), KM_SLEEP);
	blkptr_copy = os->os_rootbp;

	dprintf_ds(os->os_dsl_dataset, "txg=%llu\n", tx->tx_txg);

	ASSERT(dmu_tx_is_syncing(tx));
	/* XXX the write_done callback should really give us the tx... */
	os->os_synctx = tx;

	if (os->os_dsl_dataset == NULL) {
	/*
	* This is the MOS. If we have upgraded,
	* spa_max_replication() could change, so reset
	* os_copies here.
	*/
	os->os_copies = spa_max_replication(os->os_spa);
	}

	/*
	* Create the root block IO
	*/
	SET_BOOKMARK(&zb, os->os_dsl_dataset ?
	os->os_dsl_dataset->ds_object : DMU_META_OBJSET,
	ZB_ROOT_OBJECT, ZB_ROOT_LEVEL, ZB_ROOT_BLKID);
	arc_release(os->os_phys_buf, &os->os_phys_buf);

	dmu_write_policy(os, NULL, 0, 0, &zp);

	/*
	* If we are either claiming the ZIL or doing a raw receive, write
	* out the os_phys_buf raw. Neither of these actions will effect the
	* MAC at this point.
	*/
	if (os->os_raw_receive \|\|
	os->os_next_write_raw[tx->tx_txg & TXG_MASK]) {
	ASSERT(os->os_encrypted);
	arc_convert_to_raw(os->os_phys_buf,
	os->os_dsl_dataset->ds_object, ZFS_HOST_BYTEORDER,
	DMU_OT_OBJSET, NULL, NULL, NULL);
	}

	zio = arc_write(pio, os->os_spa, tx->tx_txg,
	blkptr_copy, os->os_phys_buf, DMU_OS_IS_L2CACHEABLE(os),
	&zp, dmu_objset_write_ready, NULL, NULL, dmu_objset_write_done,
	os, ZIO_PRIORITY_ASYNC_WRITE, ZIO_FLAG_MUSTSUCCEED, &zb);

	/*
	* Sync special dnodes - the parent IO for the sync is the root block
	*/
	DMU_META_DNODE(os)->dn_zio = zio;
	dnode_sync(DMU_META_DNODE(os), tx);

	os->os_phys->os_flags = os->os_flags;

	if (DMU_USERUSED_DNODE(os) &&
	DMU_USERUSED_DNODE(os)->dn_type != DMU_OT_NONE) {
	DMU_USERUSED_DNODE(os)->dn_zio = zio;
	dnode_sync(DMU_USERUSED_DNODE(os), tx);
	DMU_GROUPUSED_DNODE(os)->dn_zio = zio;
	dnode_sync(DMU_GROUPUSED_DNODE(os), tx);
	}

	if (DMU_PROJECTUSED_DNODE(os) &&
	DMU_PROJECTUSED_DNODE(os)->dn_type != DMU_OT_NONE) {
	DMU_PROJECTUSED_DNODE(os)->dn_zio = zio;
	dnode_sync(DMU_PROJECTUSED_DNODE(os), tx);
	}

	txgoff = tx->tx_txg & TXG_MASK;

	if (dmu_objset_userused_enabled(os) &&
	(!os->os_encrypted \|\| !dmu_objset_is_receiving(os))) {
	/*
	* We must create the list here because it uses the
	* dn_dirty_link[] of this txg. But it may already
	* exist because we call dsl_dataset_sync() twice per txg.
	*/
	if (os->os_synced_dnodes == NULL) {
	os->os_synced_dnodes =
	multilist_create(sizeof (dnode_t),
	offsetof(dnode_t, dn_dirty_link[txgoff]),
	dnode_multilist_index_func);
	} else {
	ASSERT3U(os->os_synced_dnodes->ml_offset, ==,
	offsetof(dnode_t, dn_dirty_link[txgoff]));
	}
	}

	ml = os->os_dirty_dnodes[txgoff];
	num_sublists = multilist_get_num_sublists(ml);
	for (int i = 0; i < num_sublists; i++) {
	if (multilist_sublist_is_empty_idx(ml, i))
	continue;
	sync_dnodes_arg_t sda = kmem_alloc(sizeof (sda), KM_SLEEP);
	sda->sda_list = ml;
	sda->sda_sublist_idx = i;
	sda->sda_tx = tx;
	(void) taskq_dispatch(dmu_objset_pool(os)->dp_sync_taskq,
	sync_dnodes_task, sda, 0);
	/* callback frees sda */
	}
	taskq_wait(dmu_objset_pool(os)->dp_sync_taskq);

	list = &DMU_META_DNODE(os)->dn_dirty_records[txgoff];
	while ((dr = list_head(list)) != NULL) {
	ASSERT0(dr->dr_dbuf->db_level);
	list_remove(list, dr);
	zio_nowait(dr->dr_zio);
	}

	/* Enable dnode backfill if enough objects have been freed. */
	if (os->os_freed_dnodes >= dmu_rescan_dnode_threshold) {
	os->os_rescan_dnodes = B_TRUE;
	os->os_freed_dnodes = 0;
	}

	/*
	* Free intent log blocks up to this tx.
	*/
	zil_sync(os->os_zil, tx);
	os->os_phys->os_zil_header = os->os_zil_header;
	zio_nowait(zio);
	}

	boolean_t
	dmu_objset_is_dirty(objset_t *os, uint64_t txg)
	{
	return (!multilist_is_empty(os->os_dirty_dnodes[txg & TXG_MASK]));
	}

	static file_info_cb_t *file_cbs[DMU_OST_NUMTYPES];

	void
	dmu_objset_register_type(dmu_objset_type_t ost, file_info_cb_t *cb)
	{
	file_cbs[ost] = cb;
	}

	int
	dmu_get_file_info(objset_t os, dmu_object_type_t bonustype, const void data,
	zfs_file_info_t *zfi)
	{
	file_info_cb_t *cb = file_cbs[os->os_phys->os_type];
	if (cb == NULL)
	return (EINVAL);
	return (cb(bonustype, data, zfi));
	}

	boolean_t
	dmu_objset_userused_enabled(objset_t *os)
	{
	return (spa_version(os->os_spa) >= SPA_VERSION_USERSPACE &&
	file_cbs[os->os_phys->os_type] != NULL &&
	DMU_USERUSED_DNODE(os) != NULL);
	}

	boolean_t
	dmu_objset_userobjused_enabled(objset_t *os)
	{
	return (dmu_objset_userused_enabled(os) &&
	spa_feature_is_enabled(os->os_spa, SPA_FEATURE_USEROBJ_ACCOUNTING));
	}

	boolean_t
	dmu_objset_projectquota_enabled(objset_t *os)
	{
	return (file_cbs[os->os_phys->os_type] != NULL &&
	DMU_PROJECTUSED_DNODE(os) != NULL &&
	spa_feature_is_enabled(os->os_spa, SPA_FEATURE_PROJECT_QUOTA));
	}

	typedef struct userquota_node {
	/* must be in the first filed, see userquota_update_cache() */
	char uqn_id[20 + DMU_OBJACCT_PREFIX_LEN];
	int64_t uqn_delta;
	avl_node_t uqn_node;
	} userquota_node_t;

	typedef struct userquota_cache {
	avl_tree_t uqc_user_deltas;
	avl_tree_t uqc_group_deltas;
	avl_tree_t uqc_project_deltas;
	} userquota_cache_t;

	static int
	userquota_compare(const void l, const void r)
	{
	const userquota_node_t *luqn = l;
	const userquota_node_t *ruqn = r;
	int rv;

	/*
	* NB: can only access uqn_id because userquota_update_cache() doesn't
	* pass in an entire userquota_node_t.
	*/
	rv = strcmp(luqn->uqn_id, ruqn->uqn_id);

	return (TREE_ISIGN(rv));
	}

	static void
	do_userquota_cacheflush(objset_t os, userquota_cache_t cache, dmu_tx_t *tx)
	{
	void *cookie;
	userquota_node_t *uqn;

	ASSERT(dmu_tx_is_syncing(tx));

	cookie = NULL;
	while ((uqn = avl_destroy_nodes(&cache->uqc_user_deltas,
	&cookie)) != NULL) {
	/*
	* os_userused_lock protects against concurrent calls to
	* zap_increment_int(). It's needed because zap_increment_int()
	* is not thread-safe (i.e. not atomic).
	*/
	mutex_enter(&os->os_userused_lock);
	VERIFY0(zap_increment(os, DMU_USERUSED_OBJECT,
	uqn->uqn_id, uqn->uqn_delta, tx));
	mutex_exit(&os->os_userused_lock);
	kmem_free(uqn, sizeof (*uqn));
	}
	avl_destroy(&cache->uqc_user_deltas);

	cookie = NULL;
	while ((uqn = avl_destroy_nodes(&cache->uqc_group_deltas,
	&cookie)) != NULL) {
	mutex_enter(&os->os_userused_lock);
	VERIFY0(zap_increment(os, DMU_GROUPUSED_OBJECT,
	uqn->uqn_id, uqn->uqn_delta, tx));
	mutex_exit(&os->os_userused_lock);
	kmem_free(uqn, sizeof (*uqn));
	}
	avl_destroy(&cache->uqc_group_deltas);

	if (dmu_objset_projectquota_enabled(os)) {
	cookie = NULL;
	while ((uqn = avl_destroy_nodes(&cache->uqc_project_deltas,
	&cookie)) != NULL) {
	mutex_enter(&os->os_userused_lock);
	VERIFY0(zap_increment(os, DMU_PROJECTUSED_OBJECT,
	uqn->uqn_id, uqn->uqn_delta, tx));
	mutex_exit(&os->os_userused_lock);
	kmem_free(uqn, sizeof (*uqn));
	}
	avl_destroy(&cache->uqc_project_deltas);
	}
	}

	static void
	userquota_update_cache(avl_tree_t avl, const char id, int64_t delta)
	{
	userquota_node_t *uqn;
	avl_index_t idx;

	ASSERT(strlen(id) < sizeof (uqn->uqn_id));
	/*
	* Use id directly for searching because uqn_id is the first field of
	* userquota_node_t and fields after uqn_id won't be accessed in
	* avl_find().
	*/
	uqn = avl_find(avl, (const void *)id, &idx);
	if (uqn == NULL) {
	uqn = kmem_zalloc(sizeof (*uqn), KM_SLEEP);
	strlcpy(uqn->uqn_id, id, sizeof (uqn->uqn_id));
	avl_insert(avl, uqn, idx);
	}
	uqn->uqn_delta += delta;
	}

	static void
	do_userquota_update(objset_t os, userquota_cache_t cache, uint64_t used,
	uint64_t flags, uint64_t user, uint64_t group, uint64_t project,
	boolean_t subtract)
	{
	if (flags & DNODE_FLAG_USERUSED_ACCOUNTED) {
	int64_t delta = DNODE_MIN_SIZE + used;
	char name[20];

	if (subtract)
	delta = -delta;

	(void) snprintf(name, sizeof (name), "%llx", (longlong_t)user);
	userquota_update_cache(&cache->uqc_user_deltas, name, delta);

	(void) snprintf(name, sizeof (name), "%llx", (longlong_t)group);
	userquota_update_cache(&cache->uqc_group_deltas, name, delta);

	if (dmu_objset_projectquota_enabled(os)) {
	(void) snprintf(name, sizeof (name), "%llx",
	(longlong_t)project);
	userquota_update_cache(&cache->uqc_project_deltas,
	name, delta);
	}
	}
	}

	static void
	do_userobjquota_update(objset_t os, userquota_cache_t cache, uint64_t flags,
	uint64_t user, uint64_t group, uint64_t project, boolean_t subtract)
	{
	if (flags & DNODE_FLAG_USEROBJUSED_ACCOUNTED) {
	char name[20 + DMU_OBJACCT_PREFIX_LEN];
	int delta = subtract ? -1 : 1;

	(void) snprintf(name, sizeof (name), DMU_OBJACCT_PREFIX "%llx",
	(longlong_t)user);
	userquota_update_cache(&cache->uqc_user_deltas, name, delta);

	(void) snprintf(name, sizeof (name), DMU_OBJACCT_PREFIX "%llx",
	(longlong_t)group);
	userquota_update_cache(&cache->uqc_group_deltas, name, delta);

	if (dmu_objset_projectquota_enabled(os)) {
	(void) snprintf(name, sizeof (name),
	DMU_OBJACCT_PREFIX "%llx", (longlong_t)project);
	userquota_update_cache(&cache->uqc_project_deltas,
	name, delta);
	}
	}
	}

	typedef struct userquota_updates_arg {
	objset_t *uua_os;
	int uua_sublist_idx;
	dmu_tx_t *uua_tx;
	} userquota_updates_arg_t;

	static void
	userquota_updates_task(void *arg)
	{
	userquota_updates_arg_t *uua = arg;
	objset_t *os = uua->uua_os;
	dmu_tx_t *tx = uua->uua_tx;
	dnode_t *dn;
	userquota_cache_t cache = { { 0 } };

	multilist_sublist_t *list =
	multilist_sublist_lock(os->os_synced_dnodes, uua->uua_sublist_idx);

	ASSERT(multilist_sublist_head(list) == NULL \|\|
	dmu_objset_userused_enabled(os));
	avl_create(&cache.uqc_user_deltas, userquota_compare,
	sizeof (userquota_node_t), offsetof(userquota_node_t, uqn_node));
	avl_create(&cache.uqc_group_deltas, userquota_compare,
	sizeof (userquota_node_t), offsetof(userquota_node_t, uqn_node));
	if (dmu_objset_projectquota_enabled(os))
	avl_create(&cache.uqc_project_deltas, userquota_compare,
	sizeof (userquota_node_t), offsetof(userquota_node_t,
	uqn_node));

	while ((dn = multilist_sublist_head(list)) != NULL) {
	int flags;
	ASSERT(!DMU_OBJECT_IS_SPECIAL(dn->dn_object));
	ASSERT(dn->dn_phys->dn_type == DMU_OT_NONE \|\|
	dn->dn_phys->dn_flags &
	DNODE_FLAG_USERUSED_ACCOUNTED);

	flags = dn->dn_id_flags;
	ASSERT(flags);
	if (flags & DN_ID_OLD_EXIST) {
	do_userquota_update(os, &cache, dn->dn_oldused,
	dn->dn_oldflags, dn->dn_olduid, dn->dn_oldgid,
	dn->dn_oldprojid, B_TRUE);
	do_userobjquota_update(os, &cache, dn->dn_oldflags,
	dn->dn_olduid, dn->dn_oldgid,
	dn->dn_oldprojid, B_TRUE);
	}
	if (flags & DN_ID_NEW_EXIST) {
	do_userquota_update(os, &cache,
	DN_USED_BYTES(dn->dn_phys), dn->dn_phys->dn_flags,
	dn->dn_newuid, dn->dn_newgid,
	dn->dn_newprojid, B_FALSE);
	do_userobjquota_update(os, &cache,
	dn->dn_phys->dn_flags, dn->dn_newuid, dn->dn_newgid,
	dn->dn_newprojid, B_FALSE);
	}

	mutex_enter(&dn->dn_mtx);
	dn->dn_oldused = 0;
	dn->dn_oldflags = 0;
	if (dn->dn_id_flags & DN_ID_NEW_EXIST) {
	dn->dn_olduid = dn->dn_newuid;
	dn->dn_oldgid = dn->dn_newgid;
	dn->dn_oldprojid = dn->dn_newprojid;
	dn->dn_id_flags \|= DN_ID_OLD_EXIST;
	if (dn->dn_bonuslen == 0)
	dn->dn_id_flags \|= DN_ID_CHKED_SPILL;
	else
	dn->dn_id_flags \|= DN_ID_CHKED_BONUS;
	}
	dn->dn_id_flags &= ~(DN_ID_NEW_EXIST);
	if (dn->dn_dirty_txg == spa_syncing_txg(os->os_spa))
	dn->dn_dirty_txg = 0;
	mutex_exit(&dn->dn_mtx);

	multilist_sublist_remove(list, dn);
	dnode_rele(dn, os->os_synced_dnodes);
	}
	do_userquota_cacheflush(os, &cache, tx);
	multilist_sublist_unlock(list);
	kmem_free(uua, sizeof (*uua));
	}

	void
	dmu_objset_do_userquota_updates(objset_t os, dmu_tx_t tx)
	{
	int num_sublists;

	if (!dmu_objset_userused_enabled(os))
	return;

	/*
	* If this is a raw receive just return and handle accounting
	* later when we have the keys loaded. We also don't do user
	* accounting during claiming since the datasets are not owned
	* for the duration of claiming and this txg should only be
	* used for recovery.
	*/
	if (os->os_encrypted && dmu_objset_is_receiving(os))
	return;

	if (tx->tx_txg <= os->os_spa->spa_claim_max_txg)
	return;

	/* Allocate the user/group/project used objects if necessary. */
	if (DMU_USERUSED_DNODE(os)->dn_type == DMU_OT_NONE) {
	VERIFY0(zap_create_claim(os,
	DMU_USERUSED_OBJECT,
	DMU_OT_USERGROUP_USED, DMU_OT_NONE, 0, tx));
	VERIFY0(zap_create_claim(os,
	DMU_GROUPUSED_OBJECT,
	DMU_OT_USERGROUP_USED, DMU_OT_NONE, 0, tx));
	}

	if (dmu_objset_projectquota_enabled(os) &&
	DMU_PROJECTUSED_DNODE(os)->dn_type == DMU_OT_NONE) {
	VERIFY0(zap_create_claim(os, DMU_PROJECTUSED_OBJECT,
	DMU_OT_USERGROUP_USED, DMU_OT_NONE, 0, tx));
	}

	num_sublists = multilist_get_num_sublists(os->os_synced_dnodes);
	for (int i = 0; i < num_sublists; i++) {
	if (multilist_sublist_is_empty_idx(os->os_synced_dnodes, i))
	continue;
	userquota_updates_arg_t *uua =
	kmem_alloc(sizeof (*uua), KM_SLEEP);
	uua->uua_os = os;
	uua->uua_sublist_idx = i;
	uua->uua_tx = tx;
	/* note: caller does taskq_wait() */
	(void) taskq_dispatch(dmu_objset_pool(os)->dp_sync_taskq,
	userquota_updates_task, uua, 0);
	/* callback frees uua */
	}
	}

	/*
	* Returns a pointer to data to find uid/gid from
	*
	* If a dirty record for transaction group that is syncing can't
	* be found then NULL is returned. In the NULL case it is assumed
	* the uid/gid aren't changing.
	*/
	static void *
	dmu_objset_userquota_find_data(dmu_buf_impl_t db, dmu_tx_t tx)
	{
	dbuf_dirty_record_t *dr;
	void *data;

	if (db->db_dirtycnt == 0)
	return (db->db.db_data); /* Nothing is changing */

	dr = dbuf_find_dirty_eq(db, tx->tx_txg);

	if (dr == NULL) {
	data = NULL;
	} else {
	dnode_t *dn;

	DB_DNODE_ENTER(dr->dr_dbuf);
	dn = DB_DNODE(dr->dr_dbuf);

	if (dn->dn_bonuslen == 0 &&
	dr->dr_dbuf->db_blkid == DMU_SPILL_BLKID)
	data = dr->dt.dl.dr_data->b_data;
	else
	data = dr->dt.dl.dr_data;

	DB_DNODE_EXIT(dr->dr_dbuf);
	}

	return (data);
	}

	void
	dmu_objset_userquota_get_ids(dnode_t dn, boolean_t before, dmu_tx_t tx)
	{
	objset_t *os = dn->dn_objset;
	void *data = NULL;
	dmu_buf_impl_t *db = NULL;
	int flags = dn->dn_id_flags;
	int error;
	boolean_t have_spill = B_FALSE;

	if (!dmu_objset_userused_enabled(dn->dn_objset))
	return;

	/*
	* Raw receives introduce a problem with user accounting. Raw
	* receives cannot update the user accounting info because the
	* user ids and the sizes are encrypted. To guarantee that we
	* never end up with bad user accounting, we simply disable it
	* during raw receives. We also disable this for normal receives
	* so that an incremental raw receive may be done on top of an
	* existing non-raw receive.
	*/
	if (os->os_encrypted && dmu_objset_is_receiving(os))
	return;

	if (before && (flags & (DN_ID_CHKED_BONUS\|DN_ID_OLD_EXIST\|
	DN_ID_CHKED_SPILL)))
	return;

	if (before && dn->dn_bonuslen != 0)
	data = DN_BONUS(dn->dn_phys);
	else if (!before && dn->dn_bonuslen != 0) {
	if (dn->dn_bonus) {
	db = dn->dn_bonus;
	mutex_enter(&db->db_mtx);
	data = dmu_objset_userquota_find_data(db, tx);
	} else {
	data = DN_BONUS(dn->dn_phys);
	}
	} else if (dn->dn_bonuslen == 0 && dn->dn_bonustype == DMU_OT_SA) {
	int rf = 0;

	if (RW_WRITE_HELD(&dn->dn_struct_rwlock))
	rf \|= DB_RF_HAVESTRUCT;
	error = dmu_spill_hold_by_dnode(dn,
	rf \| DB_RF_MUST_SUCCEED,
	FTAG, (dmu_buf_t **)&db);
	ASSERT(error == 0);
	mutex_enter(&db->db_mtx);
	data = (before) ? db->db.db_data :
	dmu_objset_userquota_find_data(db, tx);
	have_spill = B_TRUE;
	} else {
	mutex_enter(&dn->dn_mtx);
	dn->dn_id_flags \|= DN_ID_CHKED_BONUS;
	mutex_exit(&dn->dn_mtx);
	return;
	}

	/*
	* Must always call the callback in case the object
	* type has changed and that type isn't an object type to track
	*/
	zfs_file_info_t zfi;
	error = file_cbs[os->os_phys->os_type](dn->dn_bonustype, data, &zfi);

	if (before) {
	ASSERT(data);
	dn->dn_olduid = zfi.zfi_user;
	dn->dn_oldgid = zfi.zfi_group;
	dn->dn_oldprojid = zfi.zfi_project;
	} else if (data) {
	dn->dn_newuid = zfi.zfi_user;
	dn->dn_newgid = zfi.zfi_group;
	dn->dn_newprojid = zfi.zfi_project;
	}

	/*
	* Preserve existing uid/gid when the callback can't determine
	* what the new uid/gid are and the callback returned EEXIST.
	* The EEXIST error tells us to just use the existing uid/gid.
	* If we don't know what the old values are then just assign
	* them to 0, since that is a new file being created.
	*/
	if (!before && data == NULL && error == EEXIST) {
	if (flags & DN_ID_OLD_EXIST) {
	dn->dn_newuid = dn->dn_olduid;
	dn->dn_newgid = dn->dn_oldgid;
	dn->dn_newprojid = dn->dn_oldprojid;
	} else {
	dn->dn_newuid = 0;
	dn->dn_newgid = 0;
	dn->dn_newprojid = ZFS_DEFAULT_PROJID;
	}
	error = 0;
	}

	if (db)
	mutex_exit(&db->db_mtx);

	mutex_enter(&dn->dn_mtx);
	if (error == 0 && before)
	dn->dn_id_flags \|= DN_ID_OLD_EXIST;
	if (error == 0 && !before)
	dn->dn_id_flags \|= DN_ID_NEW_EXIST;

	if (have_spill) {
	dn->dn_id_flags \|= DN_ID_CHKED_SPILL;
	} else {
	dn->dn_id_flags \|= DN_ID_CHKED_BONUS;
	}
	mutex_exit(&dn->dn_mtx);
	if (have_spill)
	dmu_buf_rele((dmu_buf_t *)db, FTAG);
	}

	boolean_t
	dmu_objset_userspace_present(objset_t *os)
	{
	return (os->os_phys->os_flags &
	OBJSET_FLAG_USERACCOUNTING_COMPLETE);
	}

	boolean_t
	dmu_objset_userobjspace_present(objset_t *os)
	{
	return (os->os_phys->os_flags &
	OBJSET_FLAG_USEROBJACCOUNTING_COMPLETE);
	}

	boolean_t
	dmu_objset_projectquota_present(objset_t *os)
	{
	return (os->os_phys->os_flags &
	OBJSET_FLAG_PROJECTQUOTA_COMPLETE);
	}

	static int
	dmu_objset_space_upgrade(objset_t *os)
	{
	uint64_t obj;
	int err = 0;

	/*
	* We simply need to mark every object dirty, so that it will be
	* synced out and now accounted. If this is called
	* concurrently, or if we already did some work before crashing,
	* that's fine, since we track each object's accounted state
	* independently.
	*/

	for (obj = 0; err == 0; err = dmu_object_next(os, &obj, FALSE, 0)) {
	dmu_tx_t *tx;
	dmu_buf_t *db;
	int objerr;

	mutex_enter(&os->os_upgrade_lock);
	if (os->os_upgrade_exit)
	err = SET_ERROR(EINTR);
	mutex_exit(&os->os_upgrade_lock);
	if (err != 0)
	return (err);

	if (issig(JUSTLOOKING) && issig(FORREAL))
	return (SET_ERROR(EINTR));

	objerr = dmu_bonus_hold(os, obj, FTAG, &db);
	if (objerr != 0)
	continue;
	tx = dmu_tx_create(os);
	dmu_tx_hold_bonus(tx, obj);
	objerr = dmu_tx_assign(tx, TXG_WAIT);
	if (objerr != 0) {
	dmu_buf_rele(db, FTAG);
	dmu_tx_abort(tx);
	continue;
	}
	dmu_buf_will_dirty(db, tx);
	dmu_buf_rele(db, FTAG);
	dmu_tx_commit(tx);
	}
	return (0);
	}

	int
	dmu_objset_userspace_upgrade(objset_t *os)
	{
	int err = 0;

	if (dmu_objset_userspace_present(os))
	return (0);
	if (dmu_objset_is_snapshot(os))
	return (SET_ERROR(EINVAL));
	if (!dmu_objset_userused_enabled(os))
	return (SET_ERROR(ENOTSUP));

	err = dmu_objset_space_upgrade(os);
	if (err)
	return (err);

	os->os_flags \|= OBJSET_FLAG_USERACCOUNTING_COMPLETE;
	txg_wait_synced(dmu_objset_pool(os), 0);
	return (0);
	}

	static int
	dmu_objset_id_quota_upgrade_cb(objset_t *os)
	{
	int err = 0;

	if (dmu_objset_userobjspace_present(os) &&
	dmu_objset_projectquota_present(os))
	return (0);
	if (dmu_objset_is_snapshot(os))
	return (SET_ERROR(EINVAL));
	if (!dmu_objset_userobjused_enabled(os))
	return (SET_ERROR(ENOTSUP));
	if (!dmu_objset_projectquota_enabled(os) &&
	dmu_objset_userobjspace_present(os))
	return (SET_ERROR(ENOTSUP));

	dmu_objset_ds(os)->ds_feature_activation[
	SPA_FEATURE_USEROBJ_ACCOUNTING] = (void *)B_TRUE;
	if (dmu_objset_projectquota_enabled(os))
	dmu_objset_ds(os)->ds_feature_activation[
	SPA_FEATURE_PROJECT_QUOTA] = (void *)B_TRUE;

	err = dmu_objset_space_upgrade(os);
	if (err)
	return (err);

	os->os_flags \|= OBJSET_FLAG_USEROBJACCOUNTING_COMPLETE;
	if (dmu_objset_projectquota_enabled(os))
	os->os_flags \|= OBJSET_FLAG_PROJECTQUOTA_COMPLETE;

	txg_wait_synced(dmu_objset_pool(os), 0);
	return (0);
	}

	void
	dmu_objset_id_quota_upgrade(objset_t *os)
	{
	dmu_objset_upgrade(os, dmu_objset_id_quota_upgrade_cb);
	}

	boolean_t
	dmu_objset_userobjspace_upgradable(objset_t *os)
	{
	return (dmu_objset_type(os) == DMU_OST_ZFS &&
	!dmu_objset_is_snapshot(os) &&
	dmu_objset_userobjused_enabled(os) &&
	!dmu_objset_userobjspace_present(os) &&
	spa_writeable(dmu_objset_spa(os)));
	}

	boolean_t
	dmu_objset_projectquota_upgradable(objset_t *os)
	{
	return (dmu_objset_type(os) == DMU_OST_ZFS &&
	!dmu_objset_is_snapshot(os) &&
	dmu_objset_projectquota_enabled(os) &&
	!dmu_objset_projectquota_present(os) &&
	spa_writeable(dmu_objset_spa(os)));
	}

	void
	dmu_objset_space(objset_t os, uint64_t refdbytesp, uint64_t *availbytesp,
	uint64_t usedobjsp, uint64_t availobjsp)
	{
	dsl_dataset_space(os->os_dsl_dataset, refdbytesp, availbytesp,
	usedobjsp, availobjsp);
	}

	uint64_t
	dmu_objset_fsid_guid(objset_t *os)
	{
	return (dsl_dataset_fsid_guid(os->os_dsl_dataset));
	}

	void
	dmu_objset_fast_stat(objset_t os, dmu_objset_stats_t stat)
	{
	stat->dds_type = os->os_phys->os_type;
	if (os->os_dsl_dataset)
	dsl_dataset_fast_stat(os->os_dsl_dataset, stat);
	}

	void
	dmu_objset_stats(objset_t os, nvlist_t nv)
	{
	ASSERT(os->os_dsl_dataset \|\|
	os->os_phys->os_type == DMU_OST_META);

	if (os->os_dsl_dataset != NULL)
	dsl_dataset_stats(os->os_dsl_dataset, nv);

	dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_TYPE,
	os->os_phys->os_type);
	dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_USERACCOUNTING,
	dmu_objset_userspace_present(os));
	}

	int
	dmu_objset_is_snapshot(objset_t *os)
	{
	if (os->os_dsl_dataset != NULL)
	return (os->os_dsl_dataset->ds_is_snapshot);
	else
	return (B_FALSE);
	}

	int
	-dmu_snapshot_realname(objset_t os, char name, char *real, int maxlen,
	+dmu_snapshot_realname(objset_t os, const char name, char *real, int maxlen,
	boolean_t *conflict)
	{
	dsl_dataset_t *ds = os->os_dsl_dataset;
	uint64_t ignored;

	if (dsl_dataset_phys(ds)->ds_snapnames_zapobj == 0)
	return (SET_ERROR(ENOENT));

	return (zap_lookup_norm(ds->ds_dir->dd_pool->dp_meta_objset,
	dsl_dataset_phys(ds)->ds_snapnames_zapobj, name, 8, 1, &ignored,
	MT_NORMALIZE, real, maxlen, conflict));
	}

	int
	dmu_snapshot_list_next(objset_t os, int namelen, char name,
	uint64_t idp, uint64_t offp, boolean_t *case_conflict)
	{
	dsl_dataset_t *ds = os->os_dsl_dataset;
	zap_cursor_t cursor;
	zap_attribute_t attr;

	ASSERT(dsl_pool_config_held(dmu_objset_pool(os)));

	if (dsl_dataset_phys(ds)->ds_snapnames_zapobj == 0)
	return (SET_ERROR(ENOENT));

	zap_cursor_init_serialized(&cursor,
	ds->ds_dir->dd_pool->dp_meta_objset,
	dsl_dataset_phys(ds)->ds_snapnames_zapobj, *offp);

	if (zap_cursor_retrieve(&cursor, &attr) != 0) {
	zap_cursor_fini(&cursor);
	return (SET_ERROR(ENOENT));
	}

	if (strlen(attr.za_name) + 1 > namelen) {
	zap_cursor_fini(&cursor);
	return (SET_ERROR(ENAMETOOLONG));
	}

	(void) strlcpy(name, attr.za_name, namelen);
	if (idp)
	*idp = attr.za_first_integer;
	if (case_conflict)
	*case_conflict = attr.za_normalization_conflict;
	zap_cursor_advance(&cursor);
	*offp = zap_cursor_serialize(&cursor);
	zap_cursor_fini(&cursor);

	return (0);
	}

	int
	dmu_snapshot_lookup(objset_t os, const char name, uint64_t *value)
	{
	return (dsl_dataset_snap_lookup(os->os_dsl_dataset, name, value));
	}

	int
	dmu_dir_list_next(objset_t os, int namelen, char name,
	uint64_t idp, uint64_t offp)
	{
	dsl_dir_t *dd = os->os_dsl_dataset->ds_dir;
	zap_cursor_t cursor;
	zap_attribute_t attr;

	/* there is no next dir on a snapshot! */
	if (os->os_dsl_dataset->ds_object !=
	dsl_dir_phys(dd)->dd_head_dataset_obj)
	return (SET_ERROR(ENOENT));

	zap_cursor_init_serialized(&cursor,
	dd->dd_pool->dp_meta_objset,
	dsl_dir_phys(dd)->dd_child_dir_zapobj, *offp);

	if (zap_cursor_retrieve(&cursor, &attr) != 0) {
	zap_cursor_fini(&cursor);
	return (SET_ERROR(ENOENT));
	}

	if (strlen(attr.za_name) + 1 > namelen) {
	zap_cursor_fini(&cursor);
	return (SET_ERROR(ENAMETOOLONG));
	}

	(void) strlcpy(name, attr.za_name, namelen);
	if (idp)
	*idp = attr.za_first_integer;
	zap_cursor_advance(&cursor);
	*offp = zap_cursor_serialize(&cursor);
	zap_cursor_fini(&cursor);

	return (0);
	}

	typedef struct dmu_objset_find_ctx {
	taskq_t *dc_tq;
	dsl_pool_t *dc_dp;
	uint64_t dc_ddobj;
	char dc_ddname; / last component of ddobj's name */
	int (dc_func)(dsl_pool_t , dsl_dataset_t , void );
	void *dc_arg;
	int dc_flags;
	kmutex_t *dc_error_lock;
	int *dc_error;
	} dmu_objset_find_ctx_t;

	static void
	dmu_objset_find_dp_impl(dmu_objset_find_ctx_t *dcp)
	{
	dsl_pool_t *dp = dcp->dc_dp;
	dsl_dir_t *dd;
	dsl_dataset_t *ds;
	zap_cursor_t zc;
	zap_attribute_t *attr;
	uint64_t thisobj;
	int err = 0;

	/* don't process if there already was an error */
	if (*dcp->dc_error != 0)
	goto out;

	/*
	* Note: passing the name (dc_ddname) here is optional, but it
	* improves performance because we don't need to call
	* zap_value_search() to determine the name.
	*/
	err = dsl_dir_hold_obj(dp, dcp->dc_ddobj, dcp->dc_ddname, FTAG, &dd);
	if (err != 0)
	goto out;

	/* Don't visit hidden ($MOS & $ORIGIN) objsets. */
	if (dd->dd_myname[0] == '$') {
	dsl_dir_rele(dd, FTAG);
	goto out;
	}

	thisobj = dsl_dir_phys(dd)->dd_head_dataset_obj;
	attr = kmem_alloc(sizeof (zap_attribute_t), KM_SLEEP);

	/*
	* Iterate over all children.
	*/
	if (dcp->dc_flags & DS_FIND_CHILDREN) {
	for (zap_cursor_init(&zc, dp->dp_meta_objset,
	dsl_dir_phys(dd)->dd_child_dir_zapobj);
	zap_cursor_retrieve(&zc, attr) == 0;
	(void) zap_cursor_advance(&zc)) {
	ASSERT3U(attr->za_integer_length, ==,
	sizeof (uint64_t));
	ASSERT3U(attr->za_num_integers, ==, 1);

	dmu_objset_find_ctx_t *child_dcp =
	kmem_alloc(sizeof (*child_dcp), KM_SLEEP);
	child_dcp = dcp;
	child_dcp->dc_ddobj = attr->za_first_integer;
	child_dcp->dc_ddname = spa_strdup(attr->za_name);
	if (dcp->dc_tq != NULL)
	(void) taskq_dispatch(dcp->dc_tq,
	dmu_objset_find_dp_cb, child_dcp, TQ_SLEEP);
	else
	dmu_objset_find_dp_impl(child_dcp);
	}
	zap_cursor_fini(&zc);
	}

	/*
	* Iterate over all snapshots.
	*/
	if (dcp->dc_flags & DS_FIND_SNAPSHOTS) {
	dsl_dataset_t *ds;
	err = dsl_dataset_hold_obj(dp, thisobj, FTAG, &ds);

	if (err == 0) {
	uint64_t snapobj;

	snapobj = dsl_dataset_phys(ds)->ds_snapnames_zapobj;
	dsl_dataset_rele(ds, FTAG);

	for (zap_cursor_init(&zc, dp->dp_meta_objset, snapobj);
	zap_cursor_retrieve(&zc, attr) == 0;
	(void) zap_cursor_advance(&zc)) {
	ASSERT3U(attr->za_integer_length, ==,
	sizeof (uint64_t));
	ASSERT3U(attr->za_num_integers, ==, 1);

	err = dsl_dataset_hold_obj(dp,
	attr->za_first_integer, FTAG, &ds);
	if (err != 0)
	break;
	err = dcp->dc_func(dp, ds, dcp->dc_arg);
	dsl_dataset_rele(ds, FTAG);
	if (err != 0)
	break;
	}
	zap_cursor_fini(&zc);
	}
	}

	kmem_free(attr, sizeof (zap_attribute_t));

	if (err != 0) {
	dsl_dir_rele(dd, FTAG);
	goto out;
	}

	/*
	* Apply to self.
	*/
	err = dsl_dataset_hold_obj(dp, thisobj, FTAG, &ds);

	/*
	* Note: we hold the dir while calling dsl_dataset_hold_obj() so
	* that the dir will remain cached, and we won't have to re-instantiate
	* it (which could be expensive due to finding its name via
	* zap_value_search()).
	*/
	dsl_dir_rele(dd, FTAG);
	if (err != 0)
	goto out;
	err = dcp->dc_func(dp, ds, dcp->dc_arg);
	dsl_dataset_rele(ds, FTAG);

	out:
	if (err != 0) {
	mutex_enter(dcp->dc_error_lock);
	/* only keep first error */
	if (*dcp->dc_error == 0)
	*dcp->dc_error = err;
	mutex_exit(dcp->dc_error_lock);
	}

	if (dcp->dc_ddname != NULL)
	spa_strfree(dcp->dc_ddname);
	kmem_free(dcp, sizeof (*dcp));
	}

	static void
	dmu_objset_find_dp_cb(void *arg)
	{
	dmu_objset_find_ctx_t *dcp = arg;
	dsl_pool_t *dp = dcp->dc_dp;

	/*
	* We need to get a pool_config_lock here, as there are several
	* assert(pool_config_held) down the stack. Getting a lock via
	* dsl_pool_config_enter is risky, as it might be stalled by a
	* pending writer. This would deadlock, as the write lock can
	* only be granted when our parent thread gives up the lock.
	* The _prio interface gives us priority over a pending writer.
	*/
	dsl_pool_config_enter_prio(dp, FTAG);

	dmu_objset_find_dp_impl(dcp);

	dsl_pool_config_exit(dp, FTAG);
	}

	/*
	* Find objsets under and including ddobj, call func(ds) on each.
	* The order for the enumeration is completely undefined.
	* func is called with dsl_pool_config held.
	*/
	int
	dmu_objset_find_dp(dsl_pool_t *dp, uint64_t ddobj,
	int func(dsl_pool_t , dsl_dataset_t , void ), void arg, int flags)
	{
	int error = 0;
	taskq_t *tq = NULL;
	int ntasks;
	dmu_objset_find_ctx_t *dcp;
	kmutex_t err_lock;

	mutex_init(&err_lock, NULL, MUTEX_DEFAULT, NULL);
	dcp = kmem_alloc(sizeof (*dcp), KM_SLEEP);
	dcp->dc_tq = NULL;
	dcp->dc_dp = dp;
	dcp->dc_ddobj = ddobj;
	dcp->dc_ddname = NULL;
	dcp->dc_func = func;
	dcp->dc_arg = arg;
	dcp->dc_flags = flags;
	dcp->dc_error_lock = &err_lock;
	dcp->dc_error = &error;

	if ((flags & DS_FIND_SERIALIZE) \|\| dsl_pool_config_held_writer(dp)) {
	/*
	* In case a write lock is held we can't make use of
	* parallelism, as down the stack of the worker threads
	* the lock is asserted via dsl_pool_config_held.
	* In case of a read lock this is solved by getting a read
	* lock in each worker thread, which isn't possible in case
	* of a writer lock. So we fall back to the synchronous path
	* here.
	* In the future it might be possible to get some magic into
	* dsl_pool_config_held in a way that it returns true for
	* the worker threads so that a single lock held from this
	* thread suffices. For now, stay single threaded.
	*/
	dmu_objset_find_dp_impl(dcp);
	mutex_destroy(&err_lock);

	return (error);
	}

	ntasks = dmu_find_threads;
	if (ntasks == 0)
	ntasks = vdev_count_leaves(dp->dp_spa) * 4;
	tq = taskq_create("dmu_objset_find", ntasks, maxclsyspri, ntasks,
	INT_MAX, 0);
	if (tq == NULL) {
	kmem_free(dcp, sizeof (*dcp));
	mutex_destroy(&err_lock);

	return (SET_ERROR(ENOMEM));
	}
	dcp->dc_tq = tq;

	/* dcp will be freed by task */
	(void) taskq_dispatch(tq, dmu_objset_find_dp_cb, dcp, TQ_SLEEP);

	/*
	* PORTING: this code relies on the property of taskq_wait to wait
	* until no more tasks are queued and no more tasks are active. As
	* we always queue new tasks from within other tasks, task_wait
	* reliably waits for the full recursion to finish, even though we
	* enqueue new tasks after taskq_wait has been called.
	* On platforms other than illumos, taskq_wait may not have this
	* property.
	*/
	taskq_wait(tq);
	taskq_destroy(tq);
	mutex_destroy(&err_lock);

	return (error);
	}

	/*
	* Find all objsets under name, and for each, call 'func(child_name, arg)'.
	* The dp_config_rwlock must not be held when this is called, and it
	* will not be held when the callback is called.
	* Therefore this function should only be used when the pool is not changing
	* (e.g. in syncing context), or the callback can deal with the possible races.
	*/
	static int
	dmu_objset_find_impl(spa_t spa, const char name,
	int func(const char , void ), void *arg, int flags)
	{
	dsl_dir_t *dd;
	dsl_pool_t *dp = spa_get_dsl(spa);
	dsl_dataset_t *ds;
	zap_cursor_t zc;
	zap_attribute_t *attr;
	char *child;
	uint64_t thisobj;
	int err;

	dsl_pool_config_enter(dp, FTAG);

	err = dsl_dir_hold(dp, name, FTAG, &dd, NULL);
	if (err != 0) {
	dsl_pool_config_exit(dp, FTAG);
	return (err);
	}

	/* Don't visit hidden ($MOS & $ORIGIN) objsets. */
	if (dd->dd_myname[0] == '$') {
	dsl_dir_rele(dd, FTAG);
	dsl_pool_config_exit(dp, FTAG);
	return (0);
	}

	thisobj = dsl_dir_phys(dd)->dd_head_dataset_obj;
	attr = kmem_alloc(sizeof (zap_attribute_t), KM_SLEEP);

	/*
	* Iterate over all children.
	*/
	if (flags & DS_FIND_CHILDREN) {
	for (zap_cursor_init(&zc, dp->dp_meta_objset,
	dsl_dir_phys(dd)->dd_child_dir_zapobj);
	zap_cursor_retrieve(&zc, attr) == 0;
	(void) zap_cursor_advance(&zc)) {
	ASSERT3U(attr->za_integer_length, ==,
	sizeof (uint64_t));
	ASSERT3U(attr->za_num_integers, ==, 1);

	child = kmem_asprintf("%s/%s", name, attr->za_name);
	dsl_pool_config_exit(dp, FTAG);
	err = dmu_objset_find_impl(spa, child,
	func, arg, flags);
	dsl_pool_config_enter(dp, FTAG);
	kmem_strfree(child);
	if (err != 0)
	break;
	}
	zap_cursor_fini(&zc);

	if (err != 0) {
	dsl_dir_rele(dd, FTAG);
	dsl_pool_config_exit(dp, FTAG);
	kmem_free(attr, sizeof (zap_attribute_t));
	return (err);
	}
	}

	/*
	* Iterate over all snapshots.
	*/
	if (flags & DS_FIND_SNAPSHOTS) {
	err = dsl_dataset_hold_obj(dp, thisobj, FTAG, &ds);

	if (err == 0) {
	uint64_t snapobj;

	snapobj = dsl_dataset_phys(ds)->ds_snapnames_zapobj;
	dsl_dataset_rele(ds, FTAG);

	for (zap_cursor_init(&zc, dp->dp_meta_objset, snapobj);
	zap_cursor_retrieve(&zc, attr) == 0;
	(void) zap_cursor_advance(&zc)) {
	ASSERT3U(attr->za_integer_length, ==,
	sizeof (uint64_t));
	ASSERT3U(attr->za_num_integers, ==, 1);

	child = kmem_asprintf("%s@%s",
	name, attr->za_name);
	dsl_pool_config_exit(dp, FTAG);
	err = func(child, arg);
	dsl_pool_config_enter(dp, FTAG);
	kmem_strfree(child);
	if (err != 0)
	break;
	}
	zap_cursor_fini(&zc);
	}
	}

	dsl_dir_rele(dd, FTAG);
	kmem_free(attr, sizeof (zap_attribute_t));
	dsl_pool_config_exit(dp, FTAG);

	if (err != 0)
	return (err);

	/* Apply to self. */
	return (func(name, arg));
	}

	/*
	* See comment above dmu_objset_find_impl().
	*/
	int
	dmu_objset_find(const char name, int func(const char , void ), void arg,
	int flags)
	{
	spa_t *spa;
	int error;

	error = spa_open(name, &spa, FTAG);
	if (error != 0)
	return (error);
	error = dmu_objset_find_impl(spa, name, func, arg, flags);
	spa_close(spa, FTAG);
	return (error);
	}

	boolean_t
	dmu_objset_incompatible_encryption_version(objset_t *os)
	{
	return (dsl_dir_incompatible_encryption_version(
	os->os_dsl_dataset->ds_dir));
	}

	void
	dmu_objset_set_user(objset_t os, void user_ptr)
	{
	ASSERT(MUTEX_HELD(&os->os_user_ptr_lock));
	os->os_user_ptr = user_ptr;
	}

	void *
	dmu_objset_get_user(objset_t *os)
	{
	ASSERT(MUTEX_HELD(&os->os_user_ptr_lock));
	return (os->os_user_ptr);
	}

	/*
	* Determine name of filesystem, given name of snapshot.
	* buf must be at least ZFS_MAX_DATASET_NAME_LEN bytes
	*/
	int
	dmu_fsname(const char snapname, char buf)
	{
	char *atp = strchr(snapname, '@');
	if (atp == NULL)
	return (SET_ERROR(EINVAL));
	if (atp - snapname >= ZFS_MAX_DATASET_NAME_LEN)
	return (SET_ERROR(ENAMETOOLONG));
	(void) strlcpy(buf, snapname, atp - snapname + 1);
	return (0);
	}

	/*
	* Call when we think we're going to write/free space in open context
	* to track the amount of dirty data in the open txg, which is also the
	* amount of memory that can not be evicted until this txg syncs.
	*
	* Note that there are two conditions where this can be called from
	* syncing context:
	*
	* [1] When we just created the dataset, in which case we go on with
	* updating any accounting of dirty data as usual.
	* [2] When we are dirtying MOS data, in which case we only update the
	* pool's accounting of dirty data.
	*/
	void
	dmu_objset_willuse_space(objset_t os, int64_t space, dmu_tx_t tx)
	{
	dsl_dataset_t *ds = os->os_dsl_dataset;
	int64_t aspace = spa_get_worst_case_asize(os->os_spa, space);

	if (ds != NULL) {
	dsl_dir_willuse_space(ds->ds_dir, aspace, tx);
	}

	dsl_pool_dirty_space(dmu_tx_pool(tx), space, tx);
	}

	#if defined(_KERNEL)
	EXPORT_SYMBOL(dmu_objset_zil);
	EXPORT_SYMBOL(dmu_objset_pool);
	EXPORT_SYMBOL(dmu_objset_ds);
	EXPORT_SYMBOL(dmu_objset_type);
	EXPORT_SYMBOL(dmu_objset_name);
	EXPORT_SYMBOL(dmu_objset_hold);
	EXPORT_SYMBOL(dmu_objset_hold_flags);
	EXPORT_SYMBOL(dmu_objset_own);
	EXPORT_SYMBOL(dmu_objset_rele);
	EXPORT_SYMBOL(dmu_objset_rele_flags);
	EXPORT_SYMBOL(dmu_objset_disown);
	EXPORT_SYMBOL(dmu_objset_from_ds);
	EXPORT_SYMBOL(dmu_objset_create);
	EXPORT_SYMBOL(dmu_objset_clone);
	EXPORT_SYMBOL(dmu_objset_stats);
	EXPORT_SYMBOL(dmu_objset_fast_stat);
	EXPORT_SYMBOL(dmu_objset_spa);
	EXPORT_SYMBOL(dmu_objset_space);
	EXPORT_SYMBOL(dmu_objset_fsid_guid);
	EXPORT_SYMBOL(dmu_objset_find);
	EXPORT_SYMBOL(dmu_objset_byteswap);
	EXPORT_SYMBOL(dmu_objset_evict_dbufs);
	EXPORT_SYMBOL(dmu_objset_snap_cmtime);
	EXPORT_SYMBOL(dmu_objset_dnodesize);

	EXPORT_SYMBOL(dmu_objset_sync);
	EXPORT_SYMBOL(dmu_objset_is_dirty);
	EXPORT_SYMBOL(dmu_objset_create_impl_dnstats);
	EXPORT_SYMBOL(dmu_objset_create_impl);
	EXPORT_SYMBOL(dmu_objset_open_impl);
	EXPORT_SYMBOL(dmu_objset_evict);
	EXPORT_SYMBOL(dmu_objset_register_type);
	EXPORT_SYMBOL(dmu_objset_do_userquota_updates);
	EXPORT_SYMBOL(dmu_objset_userquota_get_ids);
	EXPORT_SYMBOL(dmu_objset_userused_enabled);
	EXPORT_SYMBOL(dmu_objset_userspace_upgrade);
	EXPORT_SYMBOL(dmu_objset_userspace_present);
	EXPORT_SYMBOL(dmu_objset_userobjused_enabled);
	EXPORT_SYMBOL(dmu_objset_userobjspace_upgradable);
	EXPORT_SYMBOL(dmu_objset_userobjspace_present);
	EXPORT_SYMBOL(dmu_objset_projectquota_enabled);
	EXPORT_SYMBOL(dmu_objset_projectquota_present);
	EXPORT_SYMBOL(dmu_objset_projectquota_upgradable);
	EXPORT_SYMBOL(dmu_objset_id_quota_upgrade);
	#endif
	Index: vendor-sys/openzfs/dist/module/zfs/dmu_redact.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/zfs/dmu_redact.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/zfs/dmu_redact.c (revision 366774)
	@@ -1,1199 +1,1199 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2017, 2018 by Delphix. All rights reserved.
	*/

	#include <sys/zfs_context.h>
	#include <sys/txg.h>
	#include <sys/dmu_objset.h>
	#include <sys/dmu_traverse.h>
	#include <sys/dmu_redact.h>
	#include <sys/bqueue.h>
	#include <sys/objlist.h>
	#include <sys/dmu_tx.h>
	#ifdef _KERNEL
	#include <sys/zfs_vfsops.h>
	#include <sys/zap.h>
	#include <sys/zfs_znode.h>
	#endif

	/*
	* This controls the number of entries in the buffer the redaction_list_update
	* synctask uses to buffer writes to the redaction list.
	*/
	int redact_sync_bufsize = 1024;

	/*
	* Controls how often to update the redaction list when creating a redaction
	* list.
	*/
	uint64_t redaction_list_update_interval_ns = 1000 * 1000 * 1000ULL; /* NS */

	/*
	* This tunable controls the length of the queues that zfs redact worker threads
	* use to communicate. If the dmu_redact_snap thread is blocking on these
	* queues, this variable may need to be increased. If there is a significant
	* slowdown at the start of a redact operation as these threads consume all the
	* available IO resources, or the queues are consuming too much memory, this
	* variable may need to be decreased.
	*/
	int zfs_redact_queue_length = 1024 * 1024;

	/*
	* These tunables control the fill fraction of the queues by zfs redact. The
	* fill fraction controls the frequency with which threads have to be
	* cv_signaled. If a lot of cpu time is being spent on cv_signal, then these
	* should be tuned down. If the queues empty before the signalled thread can
	* catch up, then these should be tuned up.
	*/
	uint64_t zfs_redact_queue_ff = 20;

	struct redact_record {
	bqueue_node_t ln;
	boolean_t eos_marker; /* Marks the end of the stream */
	uint64_t start_object;
	uint64_t start_blkid;
	uint64_t end_object;
	uint64_t end_blkid;
	uint8_t indblkshift;
	uint32_t datablksz;
	};

	struct redact_thread_arg {
	bqueue_t q;
	objset_t os; / Objset to traverse */
	dsl_dataset_t ds; / Dataset to traverse */
	struct redact_record *current_record;
	int error_code;
	boolean_t cancel;
	zbookmark_phys_t resume;
	objlist_t *deleted_objs;
	uint64_t *num_blocks_visited;
	uint64_t ignore_object; /* ignore further callbacks on this */
	uint64_t txg; /* txg to traverse since */
	};

	/*
	* The redaction node is a wrapper around the redaction record that is used
	* by the redaction merging thread to sort the records and determine overlaps.
	*
	* It contains two nodes; one sorts the records by their start_zb, and the other
	* sorts the records by their end_zb.
	*/
	struct redact_node {
	avl_node_t avl_node_start;
	avl_node_t avl_node_end;
	struct redact_record *record;
	struct redact_thread_arg *rt_arg;
	uint32_t thread_num;
	};

	struct merge_data {
	list_t md_redact_block_pending;
	redact_block_phys_t md_coalesce_block;
	uint64_t md_last_time;
	redact_block_phys_t md_furthest[TXG_SIZE];
	/* Lists of struct redact_block_list_node. */
	list_t md_blocks[TXG_SIZE];
	boolean_t md_synctask_txg[TXG_SIZE];
	uint64_t md_latest_synctask_txg;
	redaction_list_t *md_redaction_list;
	};

	/*
	* A wrapper around struct redact_block so it can be stored in a list_t.
	*/
	struct redact_block_list_node {
	redact_block_phys_t block;
	list_node_t node;
	};

	/*
	* We've found a new redaction candidate. In order to improve performance, we
	* coalesce these blocks when they're adjacent to each other. This function
	* handles that. If the new candidate block range is immediately after the
	* range we're building, coalesce it into the range we're building. Otherwise,
	* put the record we're building on the queue, and update the build pointer to
	* point to the new record.
	*/
	static void
	record_merge_enqueue(bqueue_t q, struct redact_record *build,
	struct redact_record *new)
	{
	if (new->eos_marker) {
	if (*build != NULL)
	bqueue_enqueue(q, build, sizeof (build));
	bqueue_enqueue_flush(q, new, sizeof (*new));
	return;
	}
	if (*build == NULL) {
	*build = new;
	return;
	}
	struct redact_record curbuild = build;
	if ((curbuild->end_object == new->start_object &&
	curbuild->end_blkid + 1 == new->start_blkid &&
	curbuild->end_blkid != UINT64_MAX) \|\|
	(curbuild->end_object + 1 == new->start_object &&
	curbuild->end_blkid == UINT64_MAX && new->start_blkid == 0)) {
	curbuild->end_object = new->end_object;
	curbuild->end_blkid = new->end_blkid;
	kmem_free(new, sizeof (*new));
	} else {
	bqueue_enqueue(q, curbuild, sizeof (*curbuild));
	*build = new;
	}
	}
	#ifdef _KERNEL
	struct objnode {
	avl_node_t node;
	uint64_t obj;
	};

	static int
	objnode_compare(const void o1, const void o2)
	{
	const struct objnode *obj1 = o1;
	const struct objnode *obj2 = o2;
	if (obj1->obj < obj2->obj)
	return (-1);
	if (obj1->obj > obj2->obj)
	return (1);
	return (0);
	}


	static objlist_t *
	zfs_get_deleteq(objset_t *os)
	{
	objlist_t *deleteq_objlist = objlist_create();
	uint64_t deleteq_obj;
	zap_cursor_t zc;
	zap_attribute_t za;
	dmu_object_info_t doi;

	ASSERT3U(os->os_phys->os_type, ==, DMU_OST_ZFS);
	VERIFY0(dmu_object_info(os, MASTER_NODE_OBJ, &doi));
	ASSERT3U(doi.doi_type, ==, DMU_OT_MASTER_NODE);

	VERIFY0(zap_lookup(os, MASTER_NODE_OBJ,
	ZFS_UNLINKED_SET, sizeof (uint64_t), 1, &deleteq_obj));

	/*
	* In order to insert objects into the objlist, they must be in sorted
	* order. We don't know what order we'll get them out of the ZAP in, so
	* we insert them into and remove them from an avl_tree_t to sort them.
	*/
	avl_tree_t at;
	avl_create(&at, objnode_compare, sizeof (struct objnode),
	offsetof(struct objnode, node));

	for (zap_cursor_init(&zc, os, deleteq_obj);
	zap_cursor_retrieve(&zc, &za) == 0; zap_cursor_advance(&zc)) {
	struct objnode obj = kmem_zalloc(sizeof (obj), KM_SLEEP);
	obj->obj = za.za_first_integer;
	avl_add(&at, obj);
	}
	zap_cursor_fini(&zc);

	struct objnode next, found = avl_first(&at);
	while (found != NULL) {
	next = AVL_NEXT(&at, found);
	objlist_insert(deleteq_objlist, found->obj);
	found = next;
	}

	void *cookie = NULL;
	while ((found = avl_destroy_nodes(&at, &cookie)) != NULL)
	kmem_free(found, sizeof (*found));
	avl_destroy(&at);
	return (deleteq_objlist);
	}
	#endif

	/*
	* This is the callback function to traverse_dataset for the redaction threads
	* for dmu_redact_snap. This thread is responsible for creating redaction
	* records for all the data that is modified by the snapshots we're redacting
	* with respect to. Redaction records represent ranges of data that have been
	* modified by one of the redaction snapshots, and are stored in the
	* redact_record struct. We need to create redaction records for three
	* cases:
	*
	* First, if there's a normal write, we need to create a redaction record for
	* that block.
	*
	* Second, if there's a hole, we need to create a redaction record that covers
	* the whole range of the hole. If the hole is in the meta-dnode, it must cover
	* every block in all of the objects in the hole.
	*
	* Third, if there is a deleted object, we need to create a redaction record for
	* all of the blocks in that object.
	*/
	/ARGSUSED/
	static int
	redact_cb(spa_t spa, zilog_t zilog, const blkptr_t *bp,
	const zbookmark_phys_t zb, const struct dnode_phys dnp, void *arg)
	{
	struct redact_thread_arg *rta = arg;
	struct redact_record *record;

	ASSERT(zb->zb_object == DMU_META_DNODE_OBJECT \|\|
	zb->zb_object >= rta->resume.zb_object);

	if (rta->cancel)
	return (SET_ERROR(EINTR));

	if (rta->ignore_object == zb->zb_object)
	return (0);

	/*
	* If we're visiting a dnode, we need to handle the case where the
	* object has been deleted.
	*/
	if (zb->zb_level == ZB_DNODE_LEVEL) {
	ASSERT3U(zb->zb_level, ==, ZB_DNODE_LEVEL);

	if (zb->zb_object == 0)
	return (0);

	/*
	* If the object has been deleted, redact all of the blocks in
	* it.
	*/
	if (dnp->dn_type == DMU_OT_NONE \|\|
	objlist_exists(rta->deleted_objs, zb->zb_object)) {
	rta->ignore_object = zb->zb_object;
	record = kmem_zalloc(sizeof (struct redact_record),
	KM_SLEEP);

	record->eos_marker = B_FALSE;
	record->start_object = record->end_object =
	zb->zb_object;
	record->start_blkid = 0;
	record->end_blkid = UINT64_MAX;
	record_merge_enqueue(&rta->q,
	&rta->current_record, record);
	}
	return (0);
	} else if (zb->zb_level < 0) {
	return (0);
	} else if (zb->zb_level > 0 && !BP_IS_HOLE(bp)) {
	/*
	* If this is an indirect block, but not a hole, it doesn't
	* provide any useful information for redaction, so ignore it.
	*/
	return (0);
	}

	/*
	* At this point, there are two options left for the type of block we're
	* looking at. Either this is a hole (which could be in the dnode or
	* the meta-dnode), or it's a level 0 block of some sort. If it's a
	* hole, we create a redaction record that covers the whole range. If
	* the hole is in a dnode, we need to redact all the blocks in that
	* hole. If the hole is in the meta-dnode, we instead need to redact
	* all blocks in every object covered by that hole. If it's a level 0
	* block, we only need to redact that single block.
	*/
	record = kmem_zalloc(sizeof (struct redact_record), KM_SLEEP);
	record->eos_marker = B_FALSE;

	record->start_object = record->end_object = zb->zb_object;
	if (BP_IS_HOLE(bp)) {
	record->start_blkid = zb->zb_blkid *
	bp_span_in_blocks(dnp->dn_indblkshift, zb->zb_level);

	record->end_blkid = ((zb->zb_blkid + 1) *
	bp_span_in_blocks(dnp->dn_indblkshift, zb->zb_level)) - 1;

	if (zb->zb_object == DMU_META_DNODE_OBJECT) {
	record->start_object = record->start_blkid *
	((SPA_MINBLOCKSIZE * dnp->dn_datablkszsec) /
	sizeof (dnode_phys_t));
	record->start_blkid = 0;
	record->end_object = ((record->end_blkid +
	1) * ((SPA_MINBLOCKSIZE * dnp->dn_datablkszsec) /
	sizeof (dnode_phys_t))) - 1;
	record->end_blkid = UINT64_MAX;
	}
	} else if (zb->zb_level != 0 \|\|
	zb->zb_object == DMU_META_DNODE_OBJECT) {
	kmem_free(record, sizeof (*record));
	return (0);
	} else {
	record->start_blkid = record->end_blkid = zb->zb_blkid;
	}
	record->indblkshift = dnp->dn_indblkshift;
	record->datablksz = dnp->dn_datablkszsec << SPA_MINBLOCKSHIFT;
	record_merge_enqueue(&rta->q, &rta->current_record, record);

	return (0);
	}

	static void
	redact_traverse_thread(void *arg)
	{
	struct redact_thread_arg *rt_arg = arg;
	int err;
	struct redact_record *data;
	#ifdef _KERNEL
	if (rt_arg->os->os_phys->os_type == DMU_OST_ZFS)
	rt_arg->deleted_objs = zfs_get_deleteq(rt_arg->os);
	else
	rt_arg->deleted_objs = objlist_create();
	#else
	rt_arg->deleted_objs = objlist_create();
	#endif

	err = traverse_dataset_resume(rt_arg->ds, rt_arg->txg,
	&rt_arg->resume, TRAVERSE_PRE \| TRAVERSE_PREFETCH_METADATA,
	redact_cb, rt_arg);

	if (err != EINTR)
	rt_arg->error_code = err;
	objlist_destroy(rt_arg->deleted_objs);
	data = kmem_zalloc(sizeof (*data), KM_SLEEP);
	data->eos_marker = B_TRUE;
	record_merge_enqueue(&rt_arg->q, &rt_arg->current_record, data);
	thread_exit();
	}

	static inline void
	create_zbookmark_from_obj_off(zbookmark_phys_t *zb, uint64_t object,
	uint64_t blkid)
	{
	zb->zb_object = object;
	zb->zb_level = 0;
	zb->zb_blkid = blkid;
	}

	/*
	* This is a utility function that can do the comparison for the start or ends
	* of the ranges in a redact_record.
	*/
	static int
	redact_range_compare(uint64_t obj1, uint64_t off1, uint32_t dbss1,
	uint64_t obj2, uint64_t off2, uint32_t dbss2)
	{
	zbookmark_phys_t z1, z2;
	create_zbookmark_from_obj_off(&z1, obj1, off1);
	create_zbookmark_from_obj_off(&z2, obj2, off2);

	return (zbookmark_compare(dbss1 >> SPA_MINBLOCKSHIFT, 0,
	dbss2 >> SPA_MINBLOCKSHIFT, 0, &z1, &z2));
	}

	/*
	* Compare two redaction records by their range's start location. Also makes
	* eos records always compare last. We use the thread number in the redact_node
	* to ensure that records do not compare equal (which is not allowed in our avl
	* trees).
	*/
	static int
	redact_node_compare_start(const void arg1, const void arg2)
	{
	const struct redact_node *rn1 = arg1;
	const struct redact_node *rn2 = arg2;
	const struct redact_record *rr1 = rn1->record;
	const struct redact_record *rr2 = rn2->record;
	if (rr1->eos_marker)
	return (1);
	if (rr2->eos_marker)
	return (-1);

	int cmp = redact_range_compare(rr1->start_object, rr1->start_blkid,
	rr1->datablksz, rr2->start_object, rr2->start_blkid,
	rr2->datablksz);
	if (cmp == 0)
	cmp = (rn1->thread_num < rn2->thread_num ? -1 : 1);
	return (cmp);
	}

	/*
	* Compare two redaction records by their range's end location. Also makes
	* eos records always compare last. We use the thread number in the redact_node
	* to ensure that records do not compare equal (which is not allowed in our avl
	* trees).
	*/
	static int
	redact_node_compare_end(const void arg1, const void arg2)
	{
	const struct redact_node *rn1 = arg1;
	const struct redact_node *rn2 = arg2;
	const struct redact_record *srr1 = rn1->record;
	const struct redact_record *srr2 = rn2->record;
	if (srr1->eos_marker)
	return (1);
	if (srr2->eos_marker)
	return (-1);

	int cmp = redact_range_compare(srr1->end_object, srr1->end_blkid,
	srr1->datablksz, srr2->end_object, srr2->end_blkid,
	srr2->datablksz);
	if (cmp == 0)
	cmp = (rn1->thread_num < rn2->thread_num ? -1 : 1);
	return (cmp);
	}

	/*
	* Utility function that compares two redaction records to determine if any part
	* of the "from" record is before any part of the "to" record. Also causes End
	* of Stream redaction records to compare after all others, so that the
	* redaction merging logic can stay simple.
	*/
	static boolean_t
	redact_record_before(const struct redact_record *from,
	const struct redact_record *to)
	{
	if (from->eos_marker == B_TRUE)
	return (B_FALSE);
	else if (to->eos_marker == B_TRUE)
	return (B_TRUE);
	return (redact_range_compare(from->start_object, from->start_blkid,
	from->datablksz, to->end_object, to->end_blkid,
	to->datablksz) <= 0);
	}

	/*
	* Pop a new redaction record off the queue, check that the records are in the
	* right order, and free the old data.
	*/
	static struct redact_record *
	get_next_redact_record(bqueue_t bq, struct redact_record prev)
	{
	struct redact_record *next = bqueue_dequeue(bq);
	ASSERT(redact_record_before(prev, next));
	kmem_free(prev, sizeof (*prev));
	return (next);
	}

	/*
	* Remove the given redaction node from both trees, pull a new redaction record
	* off the queue, free the old redaction record, update the redaction node, and
	* reinsert the node into the trees.
	*/
	static int
	update_avl_trees(avl_tree_t start_tree, avl_tree_t end_tree,
	struct redact_node *redact_node)
	{
	avl_remove(start_tree, redact_node);
	avl_remove(end_tree, redact_node);
	redact_node->record = get_next_redact_record(&redact_node->rt_arg->q,
	redact_node->record);
	avl_add(end_tree, redact_node);
	avl_add(start_tree, redact_node);
	return (redact_node->rt_arg->error_code);
	}

	/*
	* Synctask for updating redaction lists. We first take this txg's list of
	* redacted blocks and append those to the redaction list. We then update the
	* redaction list's bonus buffer. We store the furthest blocks we visited and
	* the list of snapshots that we're redacting with respect to. We need these so
	* that redacted sends and receives can be correctly resumed.
	*/
	static void
	redaction_list_update_sync(void arg, dmu_tx_t tx)
	{
	struct merge_data *md = arg;
	uint64_t txg = dmu_tx_get_txg(tx);
	list_t *list = &md->md_blocks[txg & TXG_MASK];
	redact_block_phys_t *furthest_visited =
	&md->md_furthest[txg & TXG_MASK];
	objset_t *mos = tx->tx_pool->dp_meta_objset;
	redaction_list_t *rl = md->md_redaction_list;
	int bufsize = redact_sync_bufsize;
	redact_block_phys_t buf = kmem_alloc(bufsize sizeof (*buf),
	KM_SLEEP);
	int index = 0;

	dmu_buf_will_dirty(rl->rl_dbuf, tx);

	for (struct redact_block_list_node *rbln = list_remove_head(list);
	rbln != NULL; rbln = list_remove_head(list)) {
	ASSERT3U(rbln->block.rbp_object, <=,
	furthest_visited->rbp_object);
	ASSERT(rbln->block.rbp_object < furthest_visited->rbp_object \|\|
	rbln->block.rbp_blkid <= furthest_visited->rbp_blkid);
	buf[index] = rbln->block;
	index++;
	if (index == bufsize) {
	dmu_write(mos, rl->rl_object,
	rl->rl_phys->rlp_num_entries * sizeof (*buf),
	bufsize * sizeof (*buf), buf, tx);
	rl->rl_phys->rlp_num_entries += bufsize;
	index = 0;
	}
	kmem_free(rbln, sizeof (*rbln));
	}
	if (index > 0) {
	dmu_write(mos, rl->rl_object, rl->rl_phys->rlp_num_entries *
	sizeof (buf), index sizeof (*buf), buf, tx);
	rl->rl_phys->rlp_num_entries += index;
	}
	kmem_free(buf, bufsize * sizeof (*buf));

	md->md_synctask_txg[txg & TXG_MASK] = B_FALSE;
	rl->rl_phys->rlp_last_object = furthest_visited->rbp_object;
	rl->rl_phys->rlp_last_blkid = furthest_visited->rbp_blkid;
	}

	static void
	commit_rl_updates(objset_t os, struct merge_data md, uint64_t object,
	uint64_t blkid)
	{
	dmu_tx_t *tx = dmu_tx_create_dd(spa_get_dsl(os->os_spa)->dp_mos_dir);
	dmu_tx_hold_space(tx, sizeof (struct redact_block_list_node));
	VERIFY0(dmu_tx_assign(tx, TXG_WAIT));
	uint64_t txg = dmu_tx_get_txg(tx);
	if (!md->md_synctask_txg[txg & TXG_MASK]) {
	dsl_sync_task_nowait(dmu_tx_pool(tx),
	redaction_list_update_sync, md, tx);
	md->md_synctask_txg[txg & TXG_MASK] = B_TRUE;
	md->md_latest_synctask_txg = txg;
	}
	md->md_furthest[txg & TXG_MASK].rbp_object = object;
	md->md_furthest[txg & TXG_MASK].rbp_blkid = blkid;
	list_move_tail(&md->md_blocks[txg & TXG_MASK],
	&md->md_redact_block_pending);
	dmu_tx_commit(tx);
	md->md_last_time = gethrtime();
	}

	/*
	* We want to store the list of blocks that we're redacting in the bookmark's
	* redaction list. However, this list is stored in the MOS, which means it can
	* only be written to in syncing context. To get around this, we create a
	* synctask that will write to the mos for us. We tell it what to write by
	* a linked list for each current transaction group; every time we decide to
	* redact a block, we append it to the transaction group that is currently in
	* open context. We also update some progress information that the synctask
	* will store to enable resumable redacted sends.
	*/
	static void
	update_redaction_list(struct merge_data md, objset_t os,
	uint64_t object, uint64_t blkid, uint64_t endblkid, uint32_t blksz)
	{
	boolean_t enqueue = B_FALSE;
	redact_block_phys_t cur = {0};
	uint64_t count = endblkid - blkid + 1;
	while (count > REDACT_BLOCK_MAX_COUNT) {
	update_redaction_list(md, os, object, blkid,
	blkid + REDACT_BLOCK_MAX_COUNT - 1, blksz);
	blkid += REDACT_BLOCK_MAX_COUNT;
	count -= REDACT_BLOCK_MAX_COUNT;
	}
	redact_block_phys_t *coalesce = &md->md_coalesce_block;
	boolean_t new;
	if (coalesce->rbp_size_count == 0) {
	new = B_TRUE;
	enqueue = B_FALSE;
	} else {
	uint64_t old_count = redact_block_get_count(coalesce);
	if (coalesce->rbp_object == object &&
	coalesce->rbp_blkid + old_count == blkid &&
	old_count + count <= REDACT_BLOCK_MAX_COUNT) {
	ASSERT3U(redact_block_get_size(coalesce), ==, blksz);
	redact_block_set_count(coalesce, old_count + count);
	new = B_FALSE;
	enqueue = B_FALSE;
	} else {
	new = B_TRUE;
	enqueue = B_TRUE;
	}
	}

	if (new) {
	cur = *coalesce;
	coalesce->rbp_blkid = blkid;
	coalesce->rbp_object = object;

	redact_block_set_count(coalesce, count);
	redact_block_set_size(coalesce, blksz);
	}

	if (enqueue && redact_block_get_size(&cur) != 0) {
	struct redact_block_list_node *rbln =
	kmem_alloc(sizeof (struct redact_block_list_node),
	KM_SLEEP);
	rbln->block = cur;
	list_insert_tail(&md->md_redact_block_pending, rbln);
	}

	if (gethrtime() > md->md_last_time +
	redaction_list_update_interval_ns) {
	commit_rl_updates(os, md, object, blkid);
	}
	}

	/*
	* This thread merges all the redaction records provided by the worker threads,
	* and determines which blocks are redacted by all the snapshots. The algorithm
	* for doing so is similar to performing a merge in mergesort with n sub-lists
	* instead of 2, with some added complexity due to the fact that the entries are
	* ranges, not just single blocks. This algorithm relies on the fact that the
	* queues are sorted, which is ensured by the fact that traverse_dataset
	* traverses the dataset in a consistent order. We pull one entry off the front
	* of the queues of each secure dataset traversal thread. Then we repeat the
	* following: each record represents a range of blocks modified by one of the
	* redaction snapshots, and each block in that range may need to be redacted in
	* the send stream. Find the record with the latest start of its range, and the
	* record with the earliest end of its range. If the last start is before the
	* first end, then we know that the blocks in the range [last_start, first_end]
	* are covered by all of the ranges at the front of the queues, which means
	* every thread redacts that whole range. For example, let's say the ranges on
	* each queue look like this:
	*
	* Block Id 1 2 3 4 5 6 7 8 9 10 11
	* Thread 1 \| [====================]
	* Thread 2 \| [========]
	* Thread 3 \| [=================]
	*
	* Thread 3 has the last start (5), and the thread 2 has the last end (6). All
	* three threads modified the range [5,6], so that data should not be sent over
	* the wire. After we've determined whether or not to redact anything, we take
	* the record with the first end. We discard that record, and pull a new one
	* off the front of the queue it came from. In the above example, we would
	* discard Thread 2's record, and pull a new one. Let's say the next record we
	* pulled from Thread 2 covered range [10,11]. The new layout would look like
	* this:
	*
	* Block Id 1 2 3 4 5 6 7 8 9 10 11
	* Thread 1 \| [====================]
	* Thread 2 \| [==]
	* Thread 3 \| [=================]
	*
	* When we compare the last start (10, from Thread 2) and the first end (9, from
	* Thread 1), we see that the last start is greater than the first end.
	* Therefore, we do not redact anything from these records. We'll iterate by
	* replacing the record from Thread 1.
	*
	* We iterate by replacing the record with the lowest end because we know
	* that the record with the lowest end has helped us as much as it can. All the
	* ranges before it that we will ever redact have been redacted. In addition,
	* by replacing the one with the lowest end, we guarantee we catch all ranges
	* that need to be redacted. For example, if in the case above we had replaced
	* the record from Thread 1 instead, we might have ended up with the following:
	*
	* Block Id 1 2 3 4 5 6 7 8 9 10 11 12
	* Thread 1 \| [==]
	* Thread 2 \| [========]
	* Thread 3 \| [=================]
	*
	* If the next record from Thread 2 had been [8,10], for example, we should have
	* redacted part of that range, but because we updated Thread 1's record, we
	* missed it.
	*
	* We implement this algorithm by using two trees. The first sorts the
	* redaction records by their start_zb, and the second sorts them by their
	* end_zb. We use these to find the record with the last start and the record
	* with the first end. We create a record with that start and end, and send it
	* on. The overall runtime of this implementation is O(n log m), where n is the
	* total number of redaction records from all the different redaction snapshots,
	* and m is the number of redaction snapshots.
	*
	* If we redact with respect to zero snapshots, we create a redaction
	* record with the start object and blkid to 0, and the end object and blkid to
	* UINT64_MAX. This will result in us redacting every block.
	*/
	static int
	perform_thread_merge(bqueue_t *q, uint32_t num_threads,
	struct redact_thread_arg thread_args, boolean_t cancel)
	{
	struct redact_node *redact_nodes = NULL;
	avl_tree_t start_tree, end_tree;
	struct redact_record *record;
	struct redact_record *current_record = NULL;
	int err = 0;
	struct merge_data md = { {0} };
	list_create(&md.md_redact_block_pending,
	sizeof (struct redact_block_list_node),
	offsetof(struct redact_block_list_node, node));

	/*
	* If we're redacting with respect to zero snapshots, then no data is
	* permitted to be sent. We enqueue a record that redacts all blocks,
	* and an eos marker.
	*/
	if (num_threads == 0) {
	record = kmem_zalloc(sizeof (struct redact_record),
	KM_SLEEP);
	// We can't redact object 0, so don't try.
	record->start_object = 1;
	record->start_blkid = 0;
	record->end_object = record->end_blkid = UINT64_MAX;
	bqueue_enqueue(q, record, sizeof (*record));
	return (0);
	}
	if (num_threads > 0) {
	redact_nodes = kmem_zalloc(num_threads *
	sizeof (*redact_nodes), KM_SLEEP);
	}

	avl_create(&start_tree, redact_node_compare_start,
	sizeof (struct redact_node),
	offsetof(struct redact_node, avl_node_start));
	avl_create(&end_tree, redact_node_compare_end,
	sizeof (struct redact_node),
	offsetof(struct redact_node, avl_node_end));

	for (int i = 0; i < num_threads; i++) {
	struct redact_node *node = &redact_nodes[i];
	struct redact_thread_arg *targ = &thread_args[i];
	node->record = bqueue_dequeue(&targ->q);
	node->rt_arg = targ;
	node->thread_num = i;
	avl_add(&start_tree, node);
	avl_add(&end_tree, node);
	}

	/*
	* Once the first record in the end tree has returned EOS, every record
	* must be an EOS record, so we should stop.
	*/
	while (err == 0 && !((struct redact_node *)avl_first(&end_tree))->
	record->eos_marker) {
	if (*cancel) {
	err = EINTR;
	break;
	}
	struct redact_node *last_start = avl_last(&start_tree);
	struct redact_node *first_end = avl_first(&end_tree);

	/*
	* If the last start record is before the first end record,
	* then we have blocks that are redacted by all threads.
	* Therefore, we should redact them. Copy the record, and send
	* it to the main thread.
	*/
	if (redact_record_before(last_start->record,
	first_end->record)) {
	record = kmem_zalloc(sizeof (struct redact_record),
	KM_SLEEP);
	record = first_end->record;
	record->start_object = last_start->record->start_object;
	record->start_blkid = last_start->record->start_blkid;
	record_merge_enqueue(q, &current_record,
	record);
	}
	err = update_avl_trees(&start_tree, &end_tree, first_end);
	}

	/*
	* We're done; if we were cancelled, we need to cancel our workers and
	* clear out their queues. Either way, we need to remove every thread's
	* redact_node struct from the avl trees.
	*/
	for (int i = 0; i < num_threads; i++) {
	if (err != 0) {
	thread_args[i].cancel = B_TRUE;
	while (!redact_nodes[i].record->eos_marker) {
	(void) update_avl_trees(&start_tree, &end_tree,
	&redact_nodes[i]);
	}
	}
	avl_remove(&start_tree, &redact_nodes[i]);
	avl_remove(&end_tree, &redact_nodes[i]);
	kmem_free(redact_nodes[i].record,
	sizeof (struct redact_record));
	}

	avl_destroy(&start_tree);
	avl_destroy(&end_tree);
	kmem_free(redact_nodes, num_threads * sizeof (*redact_nodes));
	if (current_record != NULL)
	bqueue_enqueue(q, current_record, sizeof (current_record));
	return (err);
	}

	struct redact_merge_thread_arg {
	bqueue_t q;
	spa_t *spa;
	int numsnaps;
	struct redact_thread_arg *thr_args;
	boolean_t cancel;
	int error_code;
	};

	static void
	redact_merge_thread(void *arg)
	{
	struct redact_merge_thread_arg *rmta = arg;
	rmta->error_code = perform_thread_merge(&rmta->q,
	rmta->numsnaps, rmta->thr_args, &rmta->cancel);
	struct redact_record rec = kmem_zalloc(sizeof (rec), KM_SLEEP);
	rec->eos_marker = B_TRUE;
	bqueue_enqueue_flush(&rmta->q, rec, 1);
	thread_exit();
	}

	/*
	* Find the next object in or after the redaction range passed in, and hold
	* its dnode with the provided tag. Also update *object to contain the new
	* object number.
	*/
	static int
	hold_next_object(objset_t os, struct redact_record rec, void *tag,
	uint64_t object, dnode_t *dn)
	{
	int err = 0;
	if (*dn != NULL)
	dnode_rele(*dn, FTAG);
	*dn = NULL;
	if (*object < rec->start_object) {
	*object = rec->start_object - 1;
	}
	err = dmu_object_next(os, object, B_FALSE, 0);
	if (err != 0)
	return (err);

	err = dnode_hold(os, *object, tag, dn);
	while (err == 0 && (*object < rec->start_object \|\|
	DMU_OT_IS_METADATA((*dn)->dn_type))) {
	dnode_rele(*dn, tag);
	*dn = NULL;
	err = dmu_object_next(os, object, B_FALSE, 0);
	if (err != 0)
	break;
	err = dnode_hold(os, *object, tag, dn);
	}
	return (err);
	}

	static int
	perform_redaction(objset_t os, redaction_list_t rl,
	struct redact_merge_thread_arg *rmta)
	{
	int err = 0;
	bqueue_t *q = &rmta->q;
	struct redact_record *rec = NULL;
	struct merge_data md = { {0} };

	list_create(&md.md_redact_block_pending,
	sizeof (struct redact_block_list_node),
	offsetof(struct redact_block_list_node, node));
	md.md_redaction_list = rl;

	for (int i = 0; i < TXG_SIZE; i++) {
	list_create(&md.md_blocks[i],
	sizeof (struct redact_block_list_node),
	offsetof(struct redact_block_list_node, node));
	}
	dnode_t *dn = NULL;
	uint64_t prev_obj = 0;
	for (rec = bqueue_dequeue(q); !rec->eos_marker && err == 0;
	rec = get_next_redact_record(q, rec)) {
	ASSERT3U(rec->start_object, !=, 0);
	uint64_t object;
	if (prev_obj != rec->start_object) {
	object = rec->start_object - 1;
	err = hold_next_object(os, rec, FTAG, &object, &dn);
	} else {
	object = prev_obj;
	}
	while (err == 0 && object <= rec->end_object) {
	if (issig(JUSTLOOKING) && issig(FORREAL)) {
	err = EINTR;
	break;
	}
	/*
	* Part of the current object is contained somewhere in
	* the range covered by rec.
	*/
	uint64_t startblkid;
	uint64_t endblkid;
	uint64_t maxblkid = dn->dn_phys->dn_maxblkid;

	if (rec->start_object < object)
	startblkid = 0;
	else if (rec->start_blkid > maxblkid)
	break;
	else
	startblkid = rec->start_blkid;

	if (rec->end_object > object \|\| rec->end_blkid >
	maxblkid) {
	endblkid = maxblkid;
	} else {
	endblkid = rec->end_blkid;
	}
	update_redaction_list(&md, os, object, startblkid,
	endblkid, dn->dn_datablksz);

	if (object == rec->end_object)
	break;
	err = hold_next_object(os, rec, FTAG, &object, &dn);
	}
	if (err == ESRCH)
	err = 0;
	if (dn != NULL)
	prev_obj = object;
	}
	if (err == 0 && dn != NULL)
	dnode_rele(dn, FTAG);

	if (err == ESRCH)
	err = 0;
	rmta->cancel = B_TRUE;
	while (!rec->eos_marker)
	rec = get_next_redact_record(q, rec);
	kmem_free(rec, sizeof (*rec));

	/*
	* There may be a block that's being coalesced, sync that out before we
	* return.
	*/
	if (err == 0 && md.md_coalesce_block.rbp_size_count != 0) {
	struct redact_block_list_node *rbln =
	kmem_alloc(sizeof (struct redact_block_list_node),
	KM_SLEEP);
	rbln->block = md.md_coalesce_block;
	list_insert_tail(&md.md_redact_block_pending, rbln);
	}
	commit_rl_updates(os, &md, UINT64_MAX, UINT64_MAX);

	/*
	* Wait for all the redaction info to sync out before we return, so that
	* anyone who attempts to resume this redaction will have all the data
	* they need.
	*/
	dsl_pool_t *dp = spa_get_dsl(os->os_spa);
	if (md.md_latest_synctask_txg != 0)
	txg_wait_synced(dp, md.md_latest_synctask_txg);
	for (int i = 0; i < TXG_SIZE; i++)
	list_destroy(&md.md_blocks[i]);
	return (err);
	}

	static boolean_t
	redact_snaps_contains(uint64_t *snaps, uint64_t num_snaps, uint64_t guid)
	{
	for (int i = 0; i < num_snaps; i++) {
	if (snaps[i] == guid)
	return (B_TRUE);
	}
	return (B_FALSE);
	}

	int
	dmu_redact_snap(const char snapname, nvlist_t redactnvl,
	const char *redactbook)
	{
	int err = 0;
	dsl_pool_t *dp = NULL;
	dsl_dataset_t *ds = NULL;
	int numsnaps = 0;
	objset_t *os;
	struct redact_thread_arg *args = NULL;
	redaction_list_t *new_rl = NULL;
	char *newredactbook;

	if ((err = dsl_pool_hold(snapname, FTAG, &dp)) != 0)
	return (err);

	newredactbook = kmem_zalloc(sizeof (char) * ZFS_MAX_DATASET_NAME_LEN,
	KM_SLEEP);

	if ((err = dsl_dataset_hold_flags(dp, snapname, DS_HOLD_FLAG_DECRYPT,
	FTAG, &ds)) != 0) {
	goto out;
	}
	dsl_dataset_long_hold(ds, FTAG);
	if (!ds->ds_is_snapshot \|\| dmu_objset_from_ds(ds, &os) != 0) {
	err = EINVAL;
	goto out;
	}
	if (dsl_dataset_feature_is_active(ds, SPA_FEATURE_REDACTED_DATASETS)) {
	err = EALREADY;
	goto out;
	}

	numsnaps = fnvlist_num_pairs(redactnvl);
	if (numsnaps > 0)
	args = kmem_zalloc(numsnaps * sizeof (*args), KM_SLEEP);

	nvpair_t *pair = NULL;
	for (int i = 0; i < numsnaps; i++) {
	pair = nvlist_next_nvpair(redactnvl, pair);
	const char *name = nvpair_name(pair);
	struct redact_thread_arg *rta = &args[i];
	err = dsl_dataset_hold_flags(dp, name, DS_HOLD_FLAG_DECRYPT,
	FTAG, &rta->ds);
	if (err != 0)
	break;
	/*
	* We want to do the long hold before we can get any other
	* errors, because the cleanup code will release the long
	* hold if rta->ds is filled in.
	*/
	dsl_dataset_long_hold(rta->ds, FTAG);

	err = dmu_objset_from_ds(rta->ds, &rta->os);
	if (err != 0)
	break;
	if (!dsl_dataset_is_before(rta->ds, ds, 0)) {
	err = EINVAL;
	break;
	}
	if (dsl_dataset_feature_is_active(rta->ds,
	SPA_FEATURE_REDACTED_DATASETS)) {
	err = EALREADY;
	break;

	}
	}
	- VERIFY3P(nvlist_next_nvpair(redactnvl, pair), ==, NULL);
	if (err != 0)
	goto out;
	+ VERIFY3P(nvlist_next_nvpair(redactnvl, pair), ==, NULL);

	boolean_t resuming = B_FALSE;
	zfs_bookmark_phys_t bookmark;

	(void) strlcpy(newredactbook, snapname, ZFS_MAX_DATASET_NAME_LEN);
	char *c = strchr(newredactbook, '@');
	ASSERT3P(c, !=, NULL);
	int n = snprintf(c, ZFS_MAX_DATASET_NAME_LEN - (c - newredactbook),
	"#%s", redactbook);
	if (n >= ZFS_MAX_DATASET_NAME_LEN - (c - newredactbook)) {
	dsl_pool_rele(dp, FTAG);
	kmem_free(newredactbook,
	sizeof (char) * ZFS_MAX_DATASET_NAME_LEN);
	if (args != NULL)
	kmem_free(args, numsnaps * sizeof (*args));
	return (SET_ERROR(ENAMETOOLONG));
	}
	err = dsl_bookmark_lookup(dp, newredactbook, NULL, &bookmark);
	if (err == 0) {
	resuming = B_TRUE;
	if (bookmark.zbm_redaction_obj == 0) {
	err = EEXIST;
	goto out;
	}
	err = dsl_redaction_list_hold_obj(dp,
	bookmark.zbm_redaction_obj, FTAG, &new_rl);
	if (err != 0) {
	err = EIO;
	goto out;
	}
	dsl_redaction_list_long_hold(dp, new_rl, FTAG);
	if (new_rl->rl_phys->rlp_num_snaps != numsnaps) {
	err = ESRCH;
	goto out;
	}
	for (int i = 0; i < numsnaps; i++) {
	struct redact_thread_arg *rta = &args[i];
	if (!redact_snaps_contains(new_rl->rl_phys->rlp_snaps,
	new_rl->rl_phys->rlp_num_snaps,
	dsl_dataset_phys(rta->ds)->ds_guid)) {
	err = ESRCH;
	goto out;
	}
	}
	if (new_rl->rl_phys->rlp_last_blkid == UINT64_MAX &&
	new_rl->rl_phys->rlp_last_object == UINT64_MAX) {
	err = EEXIST;
	goto out;
	}
	dsl_pool_rele(dp, FTAG);
	dp = NULL;
	} else {
	uint64_t *guids = NULL;
	if (numsnaps > 0) {
	guids = kmem_zalloc(numsnaps * sizeof (uint64_t),
	KM_SLEEP);
	}
	for (int i = 0; i < numsnaps; i++) {
	struct redact_thread_arg *rta = &args[i];
	guids[i] = dsl_dataset_phys(rta->ds)->ds_guid;
	}

	dsl_pool_rele(dp, FTAG);
	dp = NULL;
	err = dsl_bookmark_create_redacted(newredactbook, snapname,
	numsnaps, guids, FTAG, &new_rl);
	kmem_free(guids, numsnaps * sizeof (uint64_t));
	if (err != 0) {
	goto out;
	}
	}

	for (int i = 0; i < numsnaps; i++) {
	struct redact_thread_arg *rta = &args[i];
	(void) bqueue_init(&rta->q, zfs_redact_queue_ff,
	zfs_redact_queue_length,
	offsetof(struct redact_record, ln));
	if (resuming) {
	rta->resume.zb_blkid =
	new_rl->rl_phys->rlp_last_blkid;
	rta->resume.zb_object =
	new_rl->rl_phys->rlp_last_object;
	}
	rta->txg = dsl_dataset_phys(ds)->ds_creation_txg;
	(void) thread_create(NULL, 0, redact_traverse_thread, rta,
	0, curproc, TS_RUN, minclsyspri);
	}

	struct redact_merge_thread_arg *rmta;
	rmta = kmem_zalloc(sizeof (struct redact_merge_thread_arg), KM_SLEEP);

	(void) bqueue_init(&rmta->q, zfs_redact_queue_ff,
	zfs_redact_queue_length, offsetof(struct redact_record, ln));
	rmta->numsnaps = numsnaps;
	rmta->spa = os->os_spa;
	rmta->thr_args = args;
	(void) thread_create(NULL, 0, redact_merge_thread, rmta, 0, curproc,
	TS_RUN, minclsyspri);
	err = perform_redaction(os, new_rl, rmta);
	kmem_free(rmta, sizeof (struct redact_merge_thread_arg));

	out:
	kmem_free(newredactbook, sizeof (char) * ZFS_MAX_DATASET_NAME_LEN);

	if (new_rl != NULL) {
	dsl_redaction_list_long_rele(new_rl, FTAG);
	dsl_redaction_list_rele(new_rl, FTAG);
	}
	for (int i = 0; i < numsnaps; i++) {
	struct redact_thread_arg *rta = &args[i];
	/*
	* rta->ds may be NULL if we got an error while filling
	* it in.
	*/
	if (rta->ds != NULL) {
	dsl_dataset_long_rele(rta->ds, FTAG);
	dsl_dataset_rele_flags(rta->ds,
	DS_HOLD_FLAG_DECRYPT, FTAG);
	}
	}

	if (args != NULL)
	kmem_free(args, numsnaps * sizeof (*args));
	if (dp != NULL)
	dsl_pool_rele(dp, FTAG);
	if (ds != NULL) {
	dsl_dataset_long_rele(ds, FTAG);
	dsl_dataset_rele_flags(ds, DS_HOLD_FLAG_DECRYPT, FTAG);
	}
	return (SET_ERROR(err));

	}
	Index: vendor-sys/openzfs/dist/module/zfs/dmu_traverse.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/zfs/dmu_traverse.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/zfs/dmu_traverse.c (revision 366774)
	@@ -1,786 +1,788 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2012, 2018 by Delphix. All rights reserved.
	*/

	#include <sys/zfs_context.h>
	#include <sys/dmu_objset.h>
	#include <sys/dmu_traverse.h>
	#include <sys/dsl_dataset.h>
	#include <sys/dsl_dir.h>
	#include <sys/dsl_pool.h>
	#include <sys/dnode.h>
	#include <sys/spa.h>
	#include <sys/spa_impl.h>
	#include <sys/zio.h>
	#include <sys/dmu_impl.h>
	#include <sys/sa.h>
	#include <sys/sa_impl.h>
	#include <sys/callb.h>
	#include <sys/zfeature.h>

	int32_t zfs_pd_bytes_max = 50 * 1024 * 1024; /* 50MB */
	int32_t send_holes_without_birth_time = 1;

	typedef struct prefetch_data {
	kmutex_t pd_mtx;
	kcondvar_t pd_cv;
	int32_t pd_bytes_fetched;
	int pd_flags;
	boolean_t pd_cancel;
	boolean_t pd_exited;
	zbookmark_phys_t pd_resume;
	} prefetch_data_t;

	typedef struct traverse_data {
	spa_t *td_spa;
	uint64_t td_objset;
	blkptr_t *td_rootbp;
	uint64_t td_min_txg;
	zbookmark_phys_t *td_resume;
	int td_flags;
	prefetch_data_t *td_pfd;
	boolean_t td_paused;
	uint64_t td_hole_birth_enabled_txg;
	blkptr_cb_t *td_func;
	void *td_arg;
	boolean_t td_realloc_possible;
	} traverse_data_t;

	static int traverse_dnode(traverse_data_t td, const blkptr_t bp,
	const dnode_phys_t *dnp, uint64_t objset, uint64_t object);
	static void prefetch_dnode_metadata(traverse_data_t td, const dnode_phys_t ,
	uint64_t objset, uint64_t object);

	static int
	-traverse_zil_block(zilog_t zilog, blkptr_t bp, void *arg, uint64_t claim_txg)
	+traverse_zil_block(zilog_t zilog, const blkptr_t bp, void *arg,
	+ uint64_t claim_txg)
	{
	traverse_data_t *td = arg;
	zbookmark_phys_t zb;

	if (BP_IS_HOLE(bp))
	return (0);

	if (claim_txg == 0 && bp->blk_birth >= spa_min_claim_txg(td->td_spa))
	return (-1);

	SET_BOOKMARK(&zb, td->td_objset, ZB_ZIL_OBJECT, ZB_ZIL_LEVEL,
	bp->blk_cksum.zc_word[ZIL_ZC_SEQ]);

	(void) td->td_func(td->td_spa, zilog, bp, &zb, NULL, td->td_arg);

	return (0);
	}

	static int
	-traverse_zil_record(zilog_t zilog, lr_t lrc, void *arg, uint64_t claim_txg)
	+traverse_zil_record(zilog_t zilog, const lr_t lrc, void *arg,
	+ uint64_t claim_txg)
	{
	traverse_data_t *td = arg;

	if (lrc->lrc_txtype == TX_WRITE) {
	lr_write_t lr = (lr_write_t )lrc;
	blkptr_t *bp = &lr->lr_blkptr;
	zbookmark_phys_t zb;

	if (BP_IS_HOLE(bp))
	return (0);

	if (claim_txg == 0 \|\| bp->blk_birth < claim_txg)
	return (0);

	SET_BOOKMARK(&zb, td->td_objset, lr->lr_foid,
	ZB_ZIL_LEVEL, lr->lr_offset / BP_GET_LSIZE(bp));

	(void) td->td_func(td->td_spa, zilog, bp, &zb, NULL,
	td->td_arg);
	}
	return (0);
	}

	static void
	traverse_zil(traverse_data_t td, zil_header_t zh)
	{
	uint64_t claim_txg = zh->zh_claim_txg;

	/*
	* We only want to visit blocks that have been claimed but not yet
	* replayed; plus blocks that are already stable in read-only mode.
	*/
	if (claim_txg == 0 && spa_writeable(td->td_spa))
	return;

	zilog_t *zilog = zil_alloc(spa_get_dsl(td->td_spa)->dp_meta_objset, zh);
	(void) zil_parse(zilog, traverse_zil_block, traverse_zil_record, td,
	claim_txg, !(td->td_flags & TRAVERSE_NO_DECRYPT));
	zil_free(zilog);
	}

	typedef enum resume_skip {
	RESUME_SKIP_ALL,
	RESUME_SKIP_NONE,
	RESUME_SKIP_CHILDREN
	} resume_skip_t;

	/*
	* Returns RESUME_SKIP_ALL if td indicates that we are resuming a traversal and
	* the block indicated by zb does not need to be visited at all. Returns
	* RESUME_SKIP_CHILDREN if we are resuming a post traversal and we reach the
	* resume point. This indicates that this block should be visited but not its
	* children (since they must have been visited in a previous traversal).
	* Otherwise returns RESUME_SKIP_NONE.
	*/
	static resume_skip_t
	resume_skip_check(traverse_data_t td, const dnode_phys_t dnp,
	const zbookmark_phys_t *zb)
	{
	if (td->td_resume != NULL && !ZB_IS_ZERO(td->td_resume)) {
	/*
	* If we already visited this bp & everything below,
	* don't bother doing it again.
	*/
	if (zbookmark_subtree_completed(dnp, zb, td->td_resume))
	return (RESUME_SKIP_ALL);

	/*
	* If we found the block we're trying to resume from, zero
	* the bookmark out to indicate that we have resumed.
	*/
	if (bcmp(zb, td->td_resume, sizeof (*zb)) == 0) {
	bzero(td->td_resume, sizeof (*zb));
	if (td->td_flags & TRAVERSE_POST)
	return (RESUME_SKIP_CHILDREN);
	}
	}
	return (RESUME_SKIP_NONE);
	}

	static void
	traverse_prefetch_metadata(traverse_data_t *td,
	const blkptr_t bp, const zbookmark_phys_t zb)
	{
	arc_flags_t flags = ARC_FLAG_NOWAIT \| ARC_FLAG_PREFETCH;
	int zio_flags = ZIO_FLAG_CANFAIL \| ZIO_FLAG_SPECULATIVE;

	if (!(td->td_flags & TRAVERSE_PREFETCH_METADATA))
	return;
	/*
	* If we are in the process of resuming, don't prefetch, because
	* some children will not be needed (and in fact may have already
	* been freed).
	*/
	if (td->td_resume != NULL && !ZB_IS_ZERO(td->td_resume))
	return;
	if (BP_IS_HOLE(bp) \|\| bp->blk_birth <= td->td_min_txg)
	return;
	if (BP_GET_LEVEL(bp) == 0 && BP_GET_TYPE(bp) != DMU_OT_DNODE)
	return;
	ASSERT(!BP_IS_REDACTED(bp));

	if ((td->td_flags & TRAVERSE_NO_DECRYPT) && BP_IS_PROTECTED(bp))
	zio_flags \|= ZIO_FLAG_RAW;

	(void) arc_read(NULL, td->td_spa, bp, NULL, NULL,
	ZIO_PRIORITY_ASYNC_READ, zio_flags, &flags, zb);
	}

	static boolean_t
	prefetch_needed(prefetch_data_t pfd, const blkptr_t bp)
	{
	ASSERT(pfd->pd_flags & TRAVERSE_PREFETCH_DATA);
	if (BP_IS_HOLE(bp) \|\| BP_IS_EMBEDDED(bp) \|\|
	BP_GET_TYPE(bp) == DMU_OT_INTENT_LOG \|\| BP_IS_REDACTED(bp))
	return (B_FALSE);
	return (B_TRUE);
	}

	static int
	traverse_visitbp(traverse_data_t td, const dnode_phys_t dnp,
	const blkptr_t bp, const zbookmark_phys_t zb)
	{
	int err = 0;
	arc_buf_t *buf = NULL;
	prefetch_data_t *pd = td->td_pfd;

	switch (resume_skip_check(td, dnp, zb)) {
	case RESUME_SKIP_ALL:
	return (0);
	case RESUME_SKIP_CHILDREN:
	goto post;
	case RESUME_SKIP_NONE:
	break;
	default:
	ASSERT(0);
	}

	if (bp->blk_birth == 0) {
	/*
	* Since this block has a birth time of 0 it must be one of
	* two things: a hole created before the
	* SPA_FEATURE_HOLE_BIRTH feature was enabled, or a hole
	* which has always been a hole in an object.
	*
	* If a file is written sparsely, then the unwritten parts of
	* the file were "always holes" -- that is, they have been
	* holes since this object was allocated. However, we (and
	* our callers) can not necessarily tell when an object was
	* allocated. Therefore, if it's possible that this object
	* was freed and then its object number reused, we need to
	* visit all the holes with birth==0.
	*
	* If it isn't possible that the object number was reused,
	* then if SPA_FEATURE_HOLE_BIRTH was enabled before we wrote
	* all the blocks we will visit as part of this traversal,
	* then this hole must have always existed, so we can skip
	* it. We visit blocks born after (exclusive) td_min_txg.
	*
	* Note that the meta-dnode cannot be reallocated.
	*/
	if (!send_holes_without_birth_time &&
	(!td->td_realloc_possible \|\|
	zb->zb_object == DMU_META_DNODE_OBJECT) &&
	td->td_hole_birth_enabled_txg <= td->td_min_txg)
	return (0);
	} else if (bp->blk_birth <= td->td_min_txg) {
	return (0);
	}

	if (pd != NULL && !pd->pd_exited && prefetch_needed(pd, bp)) {
	uint64_t size = BP_GET_LSIZE(bp);
	mutex_enter(&pd->pd_mtx);
	ASSERT(pd->pd_bytes_fetched >= 0);
	while (pd->pd_bytes_fetched < size && !pd->pd_exited)
	cv_wait_sig(&pd->pd_cv, &pd->pd_mtx);
	pd->pd_bytes_fetched -= size;
	cv_broadcast(&pd->pd_cv);
	mutex_exit(&pd->pd_mtx);
	}

	if (BP_IS_HOLE(bp) \|\| BP_IS_REDACTED(bp)) {
	err = td->td_func(td->td_spa, NULL, bp, zb, dnp, td->td_arg);
	if (err != 0)
	goto post;
	return (0);
	}

	if (td->td_flags & TRAVERSE_PRE) {
	err = td->td_func(td->td_spa, NULL, bp, zb, dnp,
	td->td_arg);
	if (err == TRAVERSE_VISIT_NO_CHILDREN)
	return (0);
	if (err != 0)
	goto post;
	}

	if (BP_GET_LEVEL(bp) > 0) {
	uint32_t flags = ARC_FLAG_WAIT;
	int32_t i;
	int32_t epb = BP_GET_LSIZE(bp) >> SPA_BLKPTRSHIFT;
	zbookmark_phys_t *czb;

	ASSERT(!BP_IS_PROTECTED(bp));

	err = arc_read(NULL, td->td_spa, bp, arc_getbuf_func, &buf,
	ZIO_PRIORITY_ASYNC_READ, ZIO_FLAG_CANFAIL, &flags, zb);
	if (err != 0)
	goto post;

	czb = kmem_alloc(sizeof (zbookmark_phys_t), KM_SLEEP);

	for (i = 0; i < epb; i++) {
	SET_BOOKMARK(czb, zb->zb_objset, zb->zb_object,
	zb->zb_level - 1,
	zb->zb_blkid * epb + i);
	traverse_prefetch_metadata(td,
	&((blkptr_t *)buf->b_data)[i], czb);
	}

	/* recursively visitbp() blocks below this */
	for (i = 0; i < epb; i++) {
	SET_BOOKMARK(czb, zb->zb_objset, zb->zb_object,
	zb->zb_level - 1,
	zb->zb_blkid * epb + i);
	err = traverse_visitbp(td, dnp,
	&((blkptr_t *)buf->b_data)[i], czb);
	if (err != 0)
	break;
	}

	kmem_free(czb, sizeof (zbookmark_phys_t));

	} else if (BP_GET_TYPE(bp) == DMU_OT_DNODE) {
	uint32_t flags = ARC_FLAG_WAIT;
	uint32_t zio_flags = ZIO_FLAG_CANFAIL;
	int32_t i;
	int32_t epb = BP_GET_LSIZE(bp) >> DNODE_SHIFT;
	dnode_phys_t *child_dnp;

	/*
	* dnode blocks might have their bonus buffers encrypted, so
	* we must be careful to honor TRAVERSE_NO_DECRYPT
	*/
	if ((td->td_flags & TRAVERSE_NO_DECRYPT) && BP_IS_PROTECTED(bp))
	zio_flags \|= ZIO_FLAG_RAW;

	err = arc_read(NULL, td->td_spa, bp, arc_getbuf_func, &buf,
	ZIO_PRIORITY_ASYNC_READ, zio_flags, &flags, zb);
	if (err != 0)
	goto post;

	child_dnp = buf->b_data;

	for (i = 0; i < epb; i += child_dnp[i].dn_extra_slots + 1) {
	prefetch_dnode_metadata(td, &child_dnp[i],
	zb->zb_objset, zb->zb_blkid * epb + i);
	}

	/* recursively visitbp() blocks below this */
	for (i = 0; i < epb; i += child_dnp[i].dn_extra_slots + 1) {
	err = traverse_dnode(td, bp, &child_dnp[i],
	zb->zb_objset, zb->zb_blkid * epb + i);
	if (err != 0)
	break;
	}
	} else if (BP_GET_TYPE(bp) == DMU_OT_OBJSET) {
	uint32_t zio_flags = ZIO_FLAG_CANFAIL;
	arc_flags_t flags = ARC_FLAG_WAIT;
	objset_phys_t *osp;

	if ((td->td_flags & TRAVERSE_NO_DECRYPT) && BP_IS_PROTECTED(bp))
	zio_flags \|= ZIO_FLAG_RAW;

	err = arc_read(NULL, td->td_spa, bp, arc_getbuf_func, &buf,
	ZIO_PRIORITY_ASYNC_READ, zio_flags, &flags, zb);
	if (err != 0)
	goto post;

	osp = buf->b_data;
	prefetch_dnode_metadata(td, &osp->os_meta_dnode, zb->zb_objset,
	DMU_META_DNODE_OBJECT);
	/*
	* See the block comment above for the goal of this variable.
	* If the maxblkid of the meta-dnode is 0, then we know that
	* we've never had more than DNODES_PER_BLOCK objects in the
	* dataset, which means we can't have reused any object ids.
	*/
	if (osp->os_meta_dnode.dn_maxblkid == 0)
	td->td_realloc_possible = B_FALSE;

	if (OBJSET_BUF_HAS_USERUSED(buf)) {
	if (OBJSET_BUF_HAS_PROJECTUSED(buf))
	prefetch_dnode_metadata(td,
	&osp->os_projectused_dnode,
	zb->zb_objset, DMU_PROJECTUSED_OBJECT);
	prefetch_dnode_metadata(td, &osp->os_groupused_dnode,
	zb->zb_objset, DMU_GROUPUSED_OBJECT);
	prefetch_dnode_metadata(td, &osp->os_userused_dnode,
	zb->zb_objset, DMU_USERUSED_OBJECT);
	}

	err = traverse_dnode(td, bp, &osp->os_meta_dnode, zb->zb_objset,
	DMU_META_DNODE_OBJECT);
	if (err == 0 && OBJSET_BUF_HAS_USERUSED(buf)) {
	if (OBJSET_BUF_HAS_PROJECTUSED(buf))
	err = traverse_dnode(td, bp,
	&osp->os_projectused_dnode, zb->zb_objset,
	DMU_PROJECTUSED_OBJECT);
	if (err == 0)
	err = traverse_dnode(td, bp,
	&osp->os_groupused_dnode, zb->zb_objset,
	DMU_GROUPUSED_OBJECT);
	if (err == 0)
	err = traverse_dnode(td, bp,
	&osp->os_userused_dnode, zb->zb_objset,
	DMU_USERUSED_OBJECT);
	}
	}

	if (buf)
	arc_buf_destroy(buf, &buf);

	post:
	if (err == 0 && (td->td_flags & TRAVERSE_POST))
	err = td->td_func(td->td_spa, NULL, bp, zb, dnp, td->td_arg);

	if ((td->td_flags & TRAVERSE_HARD) && (err == EIO \|\| err == ECKSUM)) {
	/*
	* Ignore this disk error as requested by the HARD flag,
	* and continue traversal.
	*/
	err = 0;
	}

	/*
	* If we are stopping here, set td_resume.
	*/
	if (td->td_resume != NULL && err != 0 && !td->td_paused) {
	td->td_resume->zb_objset = zb->zb_objset;
	td->td_resume->zb_object = zb->zb_object;
	td->td_resume->zb_level = 0;
	/*
	* If we have stopped on an indirect block (e.g. due to
	* i/o error), we have not visited anything below it.
	* Set the bookmark to the first level-0 block that we need
	* to visit. This way, the resuming code does not need to
	* deal with resuming from indirect blocks.
	*
	* Note, if zb_level <= 0, dnp may be NULL, so we don't want
	* to dereference it.
	*/
	td->td_resume->zb_blkid = zb->zb_blkid;
	if (zb->zb_level > 0) {
	td->td_resume->zb_blkid <<= zb->zb_level *
	(dnp->dn_indblkshift - SPA_BLKPTRSHIFT);
	}
	td->td_paused = B_TRUE;
	}

	return (err);
	}

	static void
	prefetch_dnode_metadata(traverse_data_t td, const dnode_phys_t dnp,
	uint64_t objset, uint64_t object)
	{
	int j;
	zbookmark_phys_t czb;

	for (j = 0; j < dnp->dn_nblkptr; j++) {
	SET_BOOKMARK(&czb, objset, object, dnp->dn_nlevels - 1, j);
	traverse_prefetch_metadata(td, &dnp->dn_blkptr[j], &czb);
	}

	if (dnp->dn_flags & DNODE_FLAG_SPILL_BLKPTR) {
	SET_BOOKMARK(&czb, objset, object, 0, DMU_SPILL_BLKID);
	traverse_prefetch_metadata(td, DN_SPILL_BLKPTR(dnp), &czb);
	}
	}

	static int
	traverse_dnode(traverse_data_t td, const blkptr_t bp, const dnode_phys_t *dnp,
	uint64_t objset, uint64_t object)
	{
	int j, err = 0;
	zbookmark_phys_t czb;

	if (object != DMU_META_DNODE_OBJECT && td->td_resume != NULL &&
	object < td->td_resume->zb_object)
	return (0);

	if (td->td_flags & TRAVERSE_PRE) {
	SET_BOOKMARK(&czb, objset, object, ZB_DNODE_LEVEL,
	ZB_DNODE_BLKID);
	err = td->td_func(td->td_spa, NULL, bp, &czb, dnp,
	td->td_arg);
	if (err == TRAVERSE_VISIT_NO_CHILDREN)
	return (0);
	if (err != 0)
	return (err);
	}

	for (j = 0; j < dnp->dn_nblkptr; j++) {
	SET_BOOKMARK(&czb, objset, object, dnp->dn_nlevels - 1, j);
	err = traverse_visitbp(td, dnp, &dnp->dn_blkptr[j], &czb);
	if (err != 0)
	break;
	}

	if (err == 0 && (dnp->dn_flags & DNODE_FLAG_SPILL_BLKPTR)) {
	SET_BOOKMARK(&czb, objset, object, 0, DMU_SPILL_BLKID);
	err = traverse_visitbp(td, dnp, DN_SPILL_BLKPTR(dnp), &czb);
	}

	if (err == 0 && (td->td_flags & TRAVERSE_POST)) {
	SET_BOOKMARK(&czb, objset, object, ZB_DNODE_LEVEL,
	ZB_DNODE_BLKID);
	err = td->td_func(td->td_spa, NULL, bp, &czb, dnp,
	td->td_arg);
	if (err == TRAVERSE_VISIT_NO_CHILDREN)
	return (0);
	if (err != 0)
	return (err);
	}
	return (err);
	}

	/* ARGSUSED */
	static int
	traverse_prefetcher(spa_t spa, zilog_t zilog, const blkptr_t *bp,
	const zbookmark_phys_t zb, const dnode_phys_t dnp, void *arg)
	{
	prefetch_data_t *pfd = arg;
	int zio_flags = ZIO_FLAG_CANFAIL \| ZIO_FLAG_SPECULATIVE;
	arc_flags_t aflags = ARC_FLAG_NOWAIT \| ARC_FLAG_PREFETCH \|
	ARC_FLAG_PRESCIENT_PREFETCH;

	ASSERT(pfd->pd_bytes_fetched >= 0);
	if (zb->zb_level == ZB_DNODE_LEVEL)
	return (0);
	if (pfd->pd_cancel)
	return (SET_ERROR(EINTR));

	if (!prefetch_needed(pfd, bp))
	return (0);

	mutex_enter(&pfd->pd_mtx);
	while (!pfd->pd_cancel && pfd->pd_bytes_fetched >= zfs_pd_bytes_max)
	cv_wait_sig(&pfd->pd_cv, &pfd->pd_mtx);
	pfd->pd_bytes_fetched += BP_GET_LSIZE(bp);
	cv_broadcast(&pfd->pd_cv);
	mutex_exit(&pfd->pd_mtx);

	if ((pfd->pd_flags & TRAVERSE_NO_DECRYPT) && BP_IS_PROTECTED(bp))
	zio_flags \|= ZIO_FLAG_RAW;

	(void) arc_read(NULL, spa, bp, NULL, NULL, ZIO_PRIORITY_ASYNC_READ,
	zio_flags, &aflags, zb);

	return (0);
	}

	static void
	traverse_prefetch_thread(void *arg)
	{
	traverse_data_t *td_main = arg;
	traverse_data_t td = *td_main;
	zbookmark_phys_t czb;
	fstrans_cookie_t cookie = spl_fstrans_mark();

	td.td_func = traverse_prefetcher;
	td.td_arg = td_main->td_pfd;
	td.td_pfd = NULL;
	td.td_resume = &td_main->td_pfd->pd_resume;

	SET_BOOKMARK(&czb, td.td_objset,
	ZB_ROOT_OBJECT, ZB_ROOT_LEVEL, ZB_ROOT_BLKID);
	(void) traverse_visitbp(&td, NULL, td.td_rootbp, &czb);

	mutex_enter(&td_main->td_pfd->pd_mtx);
	td_main->td_pfd->pd_exited = B_TRUE;
	cv_broadcast(&td_main->td_pfd->pd_cv);
	mutex_exit(&td_main->td_pfd->pd_mtx);
	spl_fstrans_unmark(cookie);
	}

	/*
	* NB: dataset must not be changing on-disk (eg, is a snapshot or we are
	* in syncing context).
	*/
	static int
	traverse_impl(spa_t spa, dsl_dataset_t ds, uint64_t objset, blkptr_t *rootbp,
	uint64_t txg_start, zbookmark_phys_t *resume, int flags,
	blkptr_cb_t func, void *arg)
	{
	traverse_data_t *td;
	prefetch_data_t *pd;
	zbookmark_phys_t *czb;
	int err;

	ASSERT(ds == NULL \|\| objset == ds->ds_object);
	ASSERT(!(flags & TRAVERSE_PRE) \|\| !(flags & TRAVERSE_POST));

	td = kmem_alloc(sizeof (traverse_data_t), KM_SLEEP);
	pd = kmem_zalloc(sizeof (prefetch_data_t), KM_SLEEP);
	czb = kmem_alloc(sizeof (zbookmark_phys_t), KM_SLEEP);

	td->td_spa = spa;
	td->td_objset = objset;
	td->td_rootbp = rootbp;
	td->td_min_txg = txg_start;
	td->td_resume = resume;
	td->td_func = func;
	td->td_arg = arg;
	td->td_pfd = pd;
	td->td_flags = flags;
	td->td_paused = B_FALSE;
	td->td_realloc_possible = (txg_start == 0 ? B_FALSE : B_TRUE);

	if (spa_feature_is_active(spa, SPA_FEATURE_HOLE_BIRTH)) {
	VERIFY(spa_feature_enabled_txg(spa,
	SPA_FEATURE_HOLE_BIRTH, &td->td_hole_birth_enabled_txg));
	} else {
	td->td_hole_birth_enabled_txg = UINT64_MAX;
	}

	pd->pd_flags = flags;
	if (resume != NULL)
	pd->pd_resume = *resume;
	mutex_init(&pd->pd_mtx, NULL, MUTEX_DEFAULT, NULL);
	cv_init(&pd->pd_cv, NULL, CV_DEFAULT, NULL);

	SET_BOOKMARK(czb, td->td_objset,
	ZB_ROOT_OBJECT, ZB_ROOT_LEVEL, ZB_ROOT_BLKID);

	/* See comment on ZIL traversal in dsl_scan_visitds. */
	if (ds != NULL && !ds->ds_is_snapshot && !BP_IS_HOLE(rootbp)) {
	enum zio_flag zio_flags = ZIO_FLAG_CANFAIL;
	uint32_t flags = ARC_FLAG_WAIT;
	objset_phys_t *osp;
	arc_buf_t *buf;
	ASSERT(!BP_IS_REDACTED(rootbp));

	if ((td->td_flags & TRAVERSE_NO_DECRYPT) &&
	BP_IS_PROTECTED(rootbp))
	zio_flags \|= ZIO_FLAG_RAW;

	err = arc_read(NULL, td->td_spa, rootbp, arc_getbuf_func,
	&buf, ZIO_PRIORITY_ASYNC_READ, zio_flags, &flags, czb);
	if (err != 0) {
	/*
	* If both TRAVERSE_HARD and TRAVERSE_PRE are set,
	* continue to visitbp so that td_func can be called
	* in pre stage, and err will reset to zero.
	*/
	if (!(td->td_flags & TRAVERSE_HARD) \|\|
	!(td->td_flags & TRAVERSE_PRE))
	goto out;
	} else {
	osp = buf->b_data;
	traverse_zil(td, &osp->os_zil_header);
	arc_buf_destroy(buf, &buf);
	}
	}

	if (!(flags & TRAVERSE_PREFETCH_DATA) \|\|
	taskq_dispatch(spa->spa_prefetch_taskq, traverse_prefetch_thread,
	td, TQ_NOQUEUE) == TASKQID_INVALID)
	pd->pd_exited = B_TRUE;

	err = traverse_visitbp(td, NULL, rootbp, czb);

	mutex_enter(&pd->pd_mtx);
	pd->pd_cancel = B_TRUE;
	cv_broadcast(&pd->pd_cv);
	while (!pd->pd_exited)
	cv_wait_sig(&pd->pd_cv, &pd->pd_mtx);
	mutex_exit(&pd->pd_mtx);
	out:
	mutex_destroy(&pd->pd_mtx);
	cv_destroy(&pd->pd_cv);

	kmem_free(czb, sizeof (zbookmark_phys_t));
	kmem_free(pd, sizeof (struct prefetch_data));
	kmem_free(td, sizeof (struct traverse_data));

	return (err);
	}

	/*
	* NB: dataset must not be changing on-disk (eg, is a snapshot or we are
	* in syncing context).
	*/
	int
	traverse_dataset_resume(dsl_dataset_t *ds, uint64_t txg_start,
	zbookmark_phys_t *resume,
	int flags, blkptr_cb_t func, void *arg)
	{
	return (traverse_impl(ds->ds_dir->dd_pool->dp_spa, ds, ds->ds_object,
	&dsl_dataset_phys(ds)->ds_bp, txg_start, resume, flags, func, arg));
	}

	int
	traverse_dataset(dsl_dataset_t *ds, uint64_t txg_start,
	int flags, blkptr_cb_t func, void *arg)
	{
	return (traverse_dataset_resume(ds, txg_start, NULL, flags, func, arg));
	}

	int
	traverse_dataset_destroyed(spa_t spa, blkptr_t blkptr,
	uint64_t txg_start, zbookmark_phys_t *resume, int flags,
	blkptr_cb_t func, void *arg)
	{
	return (traverse_impl(spa, NULL, ZB_DESTROYED_OBJSET,
	blkptr, txg_start, resume, flags, func, arg));
	}

	/*
	* NB: pool must not be changing on-disk (eg, from zdb or sync context).
	*/
	int
	traverse_pool(spa_t *spa, uint64_t txg_start, int flags,
	blkptr_cb_t func, void *arg)
	{
	int err;
	dsl_pool_t *dp = spa_get_dsl(spa);
	objset_t *mos = dp->dp_meta_objset;
	boolean_t hard = (flags & TRAVERSE_HARD);

	/* visit the MOS */
	err = traverse_impl(spa, NULL, 0, spa_get_rootblkptr(spa),
	txg_start, NULL, flags, func, arg);
	if (err != 0)
	return (err);

	/* visit each dataset */
	for (uint64_t obj = 1; err == 0;
	err = dmu_object_next(mos, &obj, B_FALSE, txg_start)) {
	dmu_object_info_t doi;

	err = dmu_object_info(mos, obj, &doi);
	if (err != 0) {
	if (hard)
	continue;
	break;
	}

	if (doi.doi_bonus_type == DMU_OT_DSL_DATASET) {
	dsl_dataset_t *ds;
	uint64_t txg = txg_start;

	dsl_pool_config_enter(dp, FTAG);
	err = dsl_dataset_hold_obj(dp, obj, FTAG, &ds);
	dsl_pool_config_exit(dp, FTAG);
	if (err != 0) {
	if (hard)
	continue;
	break;
	}
	if (dsl_dataset_phys(ds)->ds_prev_snap_txg > txg)
	txg = dsl_dataset_phys(ds)->ds_prev_snap_txg;
	err = traverse_dataset(ds, txg, flags, func, arg);
	dsl_dataset_rele(ds, FTAG);
	if (err != 0)
	break;
	}
	}
	if (err == ESRCH)
	err = 0;
	return (err);
	}

	EXPORT_SYMBOL(traverse_dataset);
	EXPORT_SYMBOL(traverse_pool);

	/* BEGIN CSTYLED */
	ZFS_MODULE_PARAM(zfs, zfs_, pd_bytes_max, INT, ZMOD_RW,
	"Max number of bytes to prefetch");

	#if defined(_KERNEL)
	module_param_named(ignore_hole_birth, send_holes_without_birth_time, int, 0644);
	MODULE_PARM_DESC(ignore_hole_birth,
	"Alias for send_holes_without_birth_time");
	#endif

	ZFS_MODULE_PARAM(zfs, , send_holes_without_birth_time, INT, ZMOD_RW,
	"Ignore hole_birth txg for zfs send");
	/* END CSTYLED */
	Index: vendor-sys/openzfs/dist/module/zfs/dmu_zfetch.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/zfs/dmu_zfetch.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/zfs/dmu_zfetch.c (revision 366774)
	@@ -1,384 +1,387 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright 2009 Sun Microsystems, Inc. All rights reserved.
	* Use is subject to license terms.
	*/

	/*
	* Copyright (c) 2013, 2017 by Delphix. All rights reserved.
	*/

	#include <sys/zfs_context.h>
	#include <sys/dnode.h>
	#include <sys/dmu_objset.h>
	#include <sys/dmu_zfetch.h>
	#include <sys/dmu.h>
	#include <sys/dbuf.h>
	#include <sys/kstat.h>

	/*
	* This tunable disables predictive prefetch. Note that it leaves "prescient"
	* prefetch (e.g. prefetch for zfs send) intact. Unlike predictive prefetch,
	* prescient prefetch never issues i/os that end up not being needed,
	* so it can't hurt performance.
	*/

	int zfs_prefetch_disable = B_FALSE;

	/* max # of streams per zfetch */
	unsigned int zfetch_max_streams = 8;
	/* min time before stream reclaim */
	unsigned int zfetch_min_sec_reap = 2;
	/* max bytes to prefetch per stream (default 8MB) */
	unsigned int zfetch_max_distance = 8 * 1024 * 1024;
	/* max bytes to prefetch indirects for per stream (default 64MB) */
	unsigned int zfetch_max_idistance = 64 * 1024 * 1024;
	/* max number of bytes in an array_read in which we allow prefetching (1MB) */
	unsigned long zfetch_array_rd_sz = 1024 * 1024;

	typedef struct zfetch_stats {
	kstat_named_t zfetchstat_hits;
	kstat_named_t zfetchstat_misses;
	kstat_named_t zfetchstat_max_streams;
	} zfetch_stats_t;

	static zfetch_stats_t zfetch_stats = {
	{ "hits", KSTAT_DATA_UINT64 },
	{ "misses", KSTAT_DATA_UINT64 },
	{ "max_streams", KSTAT_DATA_UINT64 },
	};

	#define ZFETCHSTAT_BUMP(stat) \
	atomic_inc_64(&zfetch_stats.stat.value.ui64);

	kstat_t *zfetch_ksp;

	void
	zfetch_init(void)
	{
	zfetch_ksp = kstat_create("zfs", 0, "zfetchstats", "misc",
	KSTAT_TYPE_NAMED, sizeof (zfetch_stats) / sizeof (kstat_named_t),
	KSTAT_FLAG_VIRTUAL);

	if (zfetch_ksp != NULL) {
	zfetch_ksp->ks_data = &zfetch_stats;
	kstat_install(zfetch_ksp);
	}
	}

	void
	zfetch_fini(void)
	{
	if (zfetch_ksp != NULL) {
	kstat_delete(zfetch_ksp);
	zfetch_ksp = NULL;
	}
	}

	/*
	* This takes a pointer to a zfetch structure and a dnode. It performs the
	* necessary setup for the zfetch structure, grokking data from the
	* associated dnode.
	*/
	void
	dmu_zfetch_init(zfetch_t zf, dnode_t dno)
	{
	if (zf == NULL)
	return;

	zf->zf_dnode = dno;

	list_create(&zf->zf_stream, sizeof (zstream_t),
	offsetof(zstream_t, zs_node));

	mutex_init(&zf->zf_lock, NULL, MUTEX_DEFAULT, NULL);
	}

	static void
	dmu_zfetch_stream_remove(zfetch_t zf, zstream_t zs)
	{
	ASSERT(MUTEX_HELD(&zf->zf_lock));
	list_remove(&zf->zf_stream, zs);
	mutex_destroy(&zs->zs_lock);
	kmem_free(zs, sizeof (*zs));
	}

	/*
	* Clean-up state associated with a zfetch structure (e.g. destroy the
	* streams). This doesn't free the zfetch_t itself, that's left to the caller.
	*/
	void
	dmu_zfetch_fini(zfetch_t *zf)
	{
	zstream_t *zs;

	mutex_enter(&zf->zf_lock);
	while ((zs = list_head(&zf->zf_stream)) != NULL)
	dmu_zfetch_stream_remove(zf, zs);
	mutex_exit(&zf->zf_lock);
	list_destroy(&zf->zf_stream);
	mutex_destroy(&zf->zf_lock);

	zf->zf_dnode = NULL;
	}

	/*
	* If there aren't too many streams already, create a new stream.
	* The "blkid" argument is the next block that we expect this stream to access.
	* While we're here, clean up old streams (which haven't been
	* accessed for at least zfetch_min_sec_reap seconds).
	*/
	static void
	dmu_zfetch_stream_create(zfetch_t *zf, uint64_t blkid)
	{
	zstream_t *zs_next;
	int numstreams = 0;

	ASSERT(MUTEX_HELD(&zf->zf_lock));

	/*
	* Clean up old streams.
	*/
	for (zstream_t *zs = list_head(&zf->zf_stream);
	zs != NULL; zs = zs_next) {
	zs_next = list_next(&zf->zf_stream, zs);
	if (((gethrtime() - zs->zs_atime) / NANOSEC) >
	zfetch_min_sec_reap)
	dmu_zfetch_stream_remove(zf, zs);
	else
	numstreams++;
	}

	/*
	* The maximum number of streams is normally zfetch_max_streams,
	* but for small files we lower it such that it's at least possible
	* for all the streams to be non-overlapping.
	*
	* If we are already at the maximum number of streams for this file,
	* even after removing old streams, then don't create this stream.
	*/
	uint32_t max_streams = MAX(1, MIN(zfetch_max_streams,
	zf->zf_dnode->dn_maxblkid * zf->zf_dnode->dn_datablksz /
	zfetch_max_distance));
	if (numstreams >= max_streams) {
	ZFETCHSTAT_BUMP(zfetchstat_max_streams);
	return;
	}

	zstream_t zs = kmem_zalloc(sizeof (zs), KM_SLEEP);
	zs->zs_blkid = blkid;
	zs->zs_pf_blkid = blkid;
	zs->zs_ipf_blkid = blkid;
	zs->zs_atime = gethrtime();
	mutex_init(&zs->zs_lock, NULL, MUTEX_DEFAULT, NULL);

	list_insert_head(&zf->zf_stream, zs);
	}

	/*
	* This is the predictive prefetch entry point. It associates dnode access
	* specified with blkid and nblks arguments with prefetch stream, predicts
	* further accesses based on that stats and initiates speculative prefetch.
	* fetch_data argument specifies whether actual data blocks should be fetched:
	* FALSE -- prefetch only indirect blocks for predicted data blocks;
	* TRUE -- prefetch predicted data blocks plus following indirect blocks.
	*/
	void
	dmu_zfetch(zfetch_t *zf, uint64_t blkid, uint64_t nblks, boolean_t fetch_data,
	boolean_t have_lock)
	{
	zstream_t *zs;
	int64_t pf_start, ipf_start, ipf_istart, ipf_iend;
	int64_t pf_ahead_blks, max_blks;
	int epbs, max_dist_blks, pf_nblks, ipf_nblks;
	uint64_t end_of_access_blkid;
	end_of_access_blkid = blkid + nblks;
	spa_t *spa = zf->zf_dnode->dn_objset->os_spa;

	if (zfs_prefetch_disable)
	return;
	/*
	* If we haven't yet loaded the indirect vdevs' mappings, we
	* can only read from blocks that we carefully ensure are on
	* concrete vdevs (or previously-loaded indirect vdevs). So we
	* can't allow the predictive prefetcher to attempt reads of other
	* blocks (e.g. of the MOS's dnode object).
	*/
	if (!spa_indirect_vdevs_loaded(spa))
	return;

	/*
	* As a fast path for small (single-block) files, ignore access
	* to the first block.
	*/
	if (blkid == 0)
	return;

	if (!have_lock)
	rw_enter(&zf->zf_dnode->dn_struct_rwlock, RW_READER);
	mutex_enter(&zf->zf_lock);

	/*
	* Find matching prefetch stream. Depending on whether the accesses
	* are block-aligned, first block of the new access may either follow
	* the last block of the previous access, or be equal to it.
	*/
	for (zs = list_head(&zf->zf_stream); zs != NULL;
	zs = list_next(&zf->zf_stream, zs)) {
	if (blkid == zs->zs_blkid \|\| blkid + 1 == zs->zs_blkid) {
	mutex_enter(&zs->zs_lock);
	/*
	* zs_blkid could have changed before we
	* acquired zs_lock; re-check them here.
	*/
	if (blkid == zs->zs_blkid) {
	break;
	} else if (blkid + 1 == zs->zs_blkid) {
	blkid++;
	nblks--;
	if (nblks == 0) {
	/* Already prefetched this before. */
	mutex_exit(&zs->zs_lock);
	mutex_exit(&zf->zf_lock);
	if (!have_lock) {
	rw_exit(&zf->zf_dnode->
	dn_struct_rwlock);
	}
	return;
	}
	break;
	}
	mutex_exit(&zs->zs_lock);
	}
	}

	if (zs == NULL) {
	/*
	* This access is not part of any existing stream. Create
	* a new stream for it.
	*/
	ZFETCHSTAT_BUMP(zfetchstat_misses);

	dmu_zfetch_stream_create(zf, end_of_access_blkid);
	mutex_exit(&zf->zf_lock);
	if (!have_lock)
	rw_exit(&zf->zf_dnode->dn_struct_rwlock);
	return;
	}

	/*
	* This access was to a block that we issued a prefetch for on
	* behalf of this stream. Issue further prefetches for this stream.
	*
	* Normally, we start prefetching where we stopped
	* prefetching last (zs_pf_blkid). But when we get our first
	* hit on this stream, zs_pf_blkid == zs_blkid, we don't
	* want to prefetch the block we just accessed. In this case,
	* start just after the block we just accessed.
	*/
	pf_start = MAX(zs->zs_pf_blkid, end_of_access_blkid);

	/*
	* Double our amount of prefetched data, but don't let the
	* prefetch get further ahead than zfetch_max_distance.
	*/
	if (fetch_data) {
	max_dist_blks =
	zfetch_max_distance >> zf->zf_dnode->dn_datablkshift;
	/*
	* Previously, we were (zs_pf_blkid - blkid) ahead. We
	* want to now be double that, so read that amount again,
	* plus the amount we are catching up by (i.e. the amount
	* read just now).
	*/
	pf_ahead_blks = zs->zs_pf_blkid - blkid + nblks;
	max_blks = max_dist_blks - (pf_start - end_of_access_blkid);
	pf_nblks = MIN(pf_ahead_blks, max_blks);
	} else {
	pf_nblks = 0;
	}

	zs->zs_pf_blkid = pf_start + pf_nblks;

	/*
	* Do the same for indirects, starting from where we stopped last,
	* or where we will stop reading data blocks (and the indirects
	* that point to them).
	*/
	ipf_start = MAX(zs->zs_ipf_blkid, zs->zs_pf_blkid);
	max_dist_blks = zfetch_max_idistance >> zf->zf_dnode->dn_datablkshift;
	/*
	* We want to double our distance ahead of the data prefetch
	* (or reader, if we are not prefetching data). Previously, we
	* were (zs_ipf_blkid - blkid) ahead. To double that, we read
	* that amount again, plus the amount we are catching up by
	* (i.e. the amount read now + the amount of data prefetched now).
	*/
	pf_ahead_blks = zs->zs_ipf_blkid - blkid + nblks + pf_nblks;
	max_blks = max_dist_blks - (ipf_start - end_of_access_blkid);
	ipf_nblks = MIN(pf_ahead_blks, max_blks);
	zs->zs_ipf_blkid = ipf_start + ipf_nblks;

	epbs = zf->zf_dnode->dn_indblkshift - SPA_BLKPTRSHIFT;
	ipf_istart = P2ROUNDUP(ipf_start, 1 << epbs) >> epbs;
	ipf_iend = P2ROUNDUP(zs->zs_ipf_blkid, 1 << epbs) >> epbs;

	zs->zs_atime = gethrtime();
	zs->zs_blkid = end_of_access_blkid;
	mutex_exit(&zs->zs_lock);
	mutex_exit(&zf->zf_lock);

	/*
	* dbuf_prefetch() is asynchronous (even when it needs to read
	* indirect blocks), but we still prefer to drop our locks before
	* calling it to reduce the time we hold them.
	*/

	for (int i = 0; i < pf_nblks; i++) {
	dbuf_prefetch(zf->zf_dnode, 0, pf_start + i,
	ZIO_PRIORITY_ASYNC_READ, ARC_FLAG_PREDICTIVE_PREFETCH);
	}
	for (int64_t iblk = ipf_istart; iblk < ipf_iend; iblk++) {
	dbuf_prefetch(zf->zf_dnode, 1, iblk,
	ZIO_PRIORITY_ASYNC_READ, ARC_FLAG_PREDICTIVE_PREFETCH);
	}
	if (!have_lock)
	rw_exit(&zf->zf_dnode->dn_struct_rwlock);
	ZFETCHSTAT_BUMP(zfetchstat_hits);
	}

	/* BEGIN CSTYLED */
	ZFS_MODULE_PARAM(zfs_prefetch, zfs_prefetch_, disable, INT, ZMOD_RW,
	"Disable all ZFS prefetching");

	ZFS_MODULE_PARAM(zfs_prefetch, zfetch_, max_streams, UINT, ZMOD_RW,
	"Max number of streams per zfetch");

	ZFS_MODULE_PARAM(zfs_prefetch, zfetch_, min_sec_reap, UINT, ZMOD_RW,
	"Min time before stream reclaim");

	ZFS_MODULE_PARAM(zfs_prefetch, zfetch_, max_distance, UINT, ZMOD_RW,
	- "Max bytes to prefetch per stream (default 8MB)");
	+ "Max bytes to prefetch per stream");
	+
	+ZFS_MODULE_PARAM(zfs_prefetch, zfetch_, max_idistance, UINT, ZMOD_RW,
	+ "Max bytes to prefetch indirects for per stream");

	ZFS_MODULE_PARAM(zfs_prefetch, zfetch_, array_rd_sz, ULONG, ZMOD_RW,
	"Number of bytes in a array_read");
	/* END CSTYLED */
	Index: vendor-sys/openzfs/dist/module/zfs/dsl_scan.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/zfs/dsl_scan.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/zfs/dsl_scan.c (revision 366774)
	@@ -1,4427 +1,4429 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2008, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2011, 2018 by Delphix. All rights reserved.
	* Copyright 2016 Gary Mills
	* Copyright (c) 2017, 2019, Datto Inc. All rights reserved.
	* Copyright (c) 2015, Nexenta Systems, Inc. All rights reserved.
	* Copyright 2019 Joyent, Inc.
	*/

	#include <sys/dsl_scan.h>
	#include <sys/dsl_pool.h>
	#include <sys/dsl_dataset.h>
	#include <sys/dsl_prop.h>
	#include <sys/dsl_dir.h>
	#include <sys/dsl_synctask.h>
	#include <sys/dnode.h>
	#include <sys/dmu_tx.h>
	#include <sys/dmu_objset.h>
	#include <sys/arc.h>
	#include <sys/zap.h>
	#include <sys/zio.h>
	#include <sys/zfs_context.h>
	#include <sys/fs/zfs.h>
	#include <sys/zfs_znode.h>
	#include <sys/spa_impl.h>
	#include <sys/vdev_impl.h>
	#include <sys/zil_impl.h>
	#include <sys/zio_checksum.h>
	#include <sys/ddt.h>
	#include <sys/sa.h>
	#include <sys/sa_impl.h>
	#include <sys/zfeature.h>
	#include <sys/abd.h>
	#include <sys/range_tree.h>
	#ifdef _KERNEL
	#include <sys/zfs_vfsops.h>
	#endif

	/*
	* Grand theory statement on scan queue sorting
	*
	* Scanning is implemented by recursively traversing all indirection levels
	* in an object and reading all blocks referenced from said objects. This
	* results in us approximately traversing the object from lowest logical
	* offset to the highest. For best performance, we would want the logical
	* blocks to be physically contiguous. However, this is frequently not the
	* case with pools given the allocation patterns of copy-on-write filesystems.
	* So instead, we put the I/Os into a reordering queue and issue them in a
	* way that will most benefit physical disks (LBA-order).
	*
	* Queue management:
	*
	* Ideally, we would want to scan all metadata and queue up all block I/O
	* prior to starting to issue it, because that allows us to do an optimal
	* sorting job. This can however consume large amounts of memory. Therefore
	* we continuously monitor the size of the queues and constrain them to 5%
	* (zfs_scan_mem_lim_fact) of physmem. If the queues grow larger than this
	* limit, we clear out a few of the largest extents at the head of the queues
	* to make room for more scanning. Hopefully, these extents will be fairly
	* large and contiguous, allowing us to approach sequential I/O throughput
	* even without a fully sorted tree.
	*
	* Metadata scanning takes place in dsl_scan_visit(), which is called from
	* dsl_scan_sync() every spa_sync(). If we have either fully scanned all
	* metadata on the pool, or we need to make room in memory because our
	* queues are too large, dsl_scan_visit() is postponed and
	* scan_io_queues_run() is called from dsl_scan_sync() instead. This implies
	* that metadata scanning and queued I/O issuing are mutually exclusive. This
	* allows us to provide maximum sequential I/O throughput for the majority of
	* I/O's issued since sequential I/O performance is significantly negatively
	* impacted if it is interleaved with random I/O.
	*
	* Implementation Notes
	*
	* One side effect of the queued scanning algorithm is that the scanning code
	* needs to be notified whenever a block is freed. This is needed to allow
	* the scanning code to remove these I/Os from the issuing queue. Additionally,
	* we do not attempt to queue gang blocks to be issued sequentially since this
	* is very hard to do and would have an extremely limited performance benefit.
	* Instead, we simply issue gang I/Os as soon as we find them using the legacy
	* algorithm.
	*
	* Backwards compatibility
	*
	* This new algorithm is backwards compatible with the legacy on-disk data
	* structures (and therefore does not require a new feature flag).
	* Periodically during scanning (see zfs_scan_checkpoint_intval), the scan
	* will stop scanning metadata (in logical order) and wait for all outstanding
	* sorted I/O to complete. Once this is done, we write out a checkpoint
	* bookmark, indicating that we have scanned everything logically before it.
	* If the pool is imported on a machine without the new sorting algorithm,
	* the scan simply resumes from the last checkpoint using the legacy algorithm.
	*/

	typedef int (scan_cb_t)(dsl_pool_t , const blkptr_t ,
	const zbookmark_phys_t *);

	static scan_cb_t dsl_scan_scrub_cb;

	static int scan_ds_queue_compare(const void a, const void b);
	static int scan_prefetch_queue_compare(const void a, const void b);
	static void scan_ds_queue_clear(dsl_scan_t *scn);
	static void scan_ds_prefetch_queue_clear(dsl_scan_t *scn);
	static boolean_t scan_ds_queue_contains(dsl_scan_t *scn, uint64_t dsobj,
	uint64_t *txg);
	static void scan_ds_queue_insert(dsl_scan_t *scn, uint64_t dsobj, uint64_t txg);
	static void scan_ds_queue_remove(dsl_scan_t *scn, uint64_t dsobj);
	static void scan_ds_queue_sync(dsl_scan_t scn, dmu_tx_t tx);
	static uint64_t dsl_scan_count_leaves(vdev_t *vd);

	extern int zfs_vdev_async_write_active_min_dirty_percent;

	/*
	* By default zfs will check to ensure it is not over the hard memory
	* limit before each txg. If finer-grained control of this is needed
	* this value can be set to 1 to enable checking before scanning each
	* block.
	*/
	int zfs_scan_strict_mem_lim = B_FALSE;

	/*
	* Maximum number of parallelly executed bytes per leaf vdev. We attempt
	* to strike a balance here between keeping the vdev queues full of I/Os
	* at all times and not overflowing the queues to cause long latency,
	* which would cause long txg sync times. No matter what, we will not
	* overload the drives with I/O, since that is protected by
	* zfs_vdev_scrub_max_active.
	*/
	unsigned long zfs_scan_vdev_limit = 4 << 20;

	int zfs_scan_issue_strategy = 0;
	int zfs_scan_legacy = B_FALSE; /* don't queue & sort zios, go direct */
	unsigned long zfs_scan_max_ext_gap = 2 << 20; /* in bytes */

	/*
	* fill_weight is non-tunable at runtime, so we copy it at module init from
	* zfs_scan_fill_weight. Runtime adjustments to zfs_scan_fill_weight would
	* break queue sorting.
	*/
	int zfs_scan_fill_weight = 3;
	static uint64_t fill_weight;

	/* See dsl_scan_should_clear() for details on the memory limit tunables */
	uint64_t zfs_scan_mem_lim_min = 16 << 20; /* bytes */
	uint64_t zfs_scan_mem_lim_soft_max = 128 << 20; /* bytes */
	int zfs_scan_mem_lim_fact = 20; /* fraction of physmem */
	int zfs_scan_mem_lim_soft_fact = 20; /* fraction of mem lim above */

	int zfs_scrub_min_time_ms = 1000; /* min millisecs to scrub per txg */
	int zfs_obsolete_min_time_ms = 500; /* min millisecs to obsolete per txg */
	int zfs_free_min_time_ms = 1000; /* min millisecs to free per txg */
	int zfs_resilver_min_time_ms = 3000; /* min millisecs to resilver per txg */
	int zfs_scan_checkpoint_intval = 7200; /* in seconds */
	int zfs_scan_suspend_progress = 0; /* set to prevent scans from progressing */
	int zfs_no_scrub_io = B_FALSE; /* set to disable scrub i/o */
	int zfs_no_scrub_prefetch = B_FALSE; /* set to disable scrub prefetch */
	enum ddt_class zfs_scrub_ddt_class_max = DDT_CLASS_DUPLICATE;
	/* max number of blocks to free in a single TXG */
	unsigned long zfs_async_block_max_blocks = ULONG_MAX;
	/* max number of dedup blocks to free in a single TXG */
	unsigned long zfs_max_async_dedup_frees = 100000;

	int zfs_resilver_disable_defer = 0; /* set to disable resilver deferring */

	/*
	* We wait a few txgs after importing a pool to begin scanning so that
	* the import / mounting code isn't held up by scrub / resilver IO.
	* Unfortunately, it is a bit difficult to determine exactly how long
	* this will take since userspace will trigger fs mounts asynchronously
	* and the kernel will create zvol minors asynchronously. As a result,
	* the value provided here is a bit arbitrary, but represents a
	* reasonable estimate of how many txgs it will take to finish fully
	* importing a pool
	*/
	#define SCAN_IMPORT_WAIT_TXGS 5

	#define DSL_SCAN_IS_SCRUB_RESILVER(scn) \
	((scn)->scn_phys.scn_func == POOL_SCAN_SCRUB \|\| \
	(scn)->scn_phys.scn_func == POOL_SCAN_RESILVER)

	/*
	* Enable/disable the processing of the free_bpobj object.
	*/
	int zfs_free_bpobj_enabled = 1;

	/* the order has to match pool_scan_type */
	static scan_cb_t *scan_funcs[POOL_SCAN_FUNCS] = {
	NULL,
	dsl_scan_scrub_cb, /* POOL_SCAN_SCRUB */
	dsl_scan_scrub_cb, /* POOL_SCAN_RESILVER */
	};

	/* In core node for the scn->scn_queue. Represents a dataset to be scanned */
	typedef struct {
	uint64_t sds_dsobj;
	uint64_t sds_txg;
	avl_node_t sds_node;
	} scan_ds_t;

	/*
	* This controls what conditions are placed on dsl_scan_sync_state():
	* SYNC_OPTIONAL) write out scn_phys iff scn_bytes_pending == 0
	* SYNC_MANDATORY) write out scn_phys always. scn_bytes_pending must be 0.
	* SYNC_CACHED) if scn_bytes_pending == 0, write out scn_phys. Otherwise
	* write out the scn_phys_cached version.
	* See dsl_scan_sync_state for details.
	*/
	typedef enum {
	SYNC_OPTIONAL,
	SYNC_MANDATORY,
	SYNC_CACHED
	} state_sync_type_t;

	/*
	* This struct represents the minimum information needed to reconstruct a
	* zio for sequential scanning. This is useful because many of these will
	* accumulate in the sequential IO queues before being issued, so saving
	* memory matters here.
	*/
	typedef struct scan_io {
	/* fields from blkptr_t */
	uint64_t sio_blk_prop;
	uint64_t sio_phys_birth;
	uint64_t sio_birth;
	zio_cksum_t sio_cksum;
	uint32_t sio_nr_dvas;

	/* fields from zio_t */
	uint32_t sio_flags;
	zbookmark_phys_t sio_zb;

	/* members for queue sorting */
	union {
	avl_node_t sio_addr_node; /* link into issuing queue */
	list_node_t sio_list_node; /* link for issuing to disk */
	} sio_nodes;

	/*
	* There may be up to SPA_DVAS_PER_BP DVAs here from the bp,
	* depending on how many were in the original bp. Only the
	* first DVA is really used for sorting and issuing purposes.
	* The other DVAs (if provided) simply exist so that the zio
	* layer can find additional copies to repair from in the
	* event of an error. This array must go at the end of the
	* struct to allow this for the variable number of elements.
	*/
	dva_t sio_dva[0];
	} scan_io_t;

	#define SIO_SET_OFFSET(sio, x) DVA_SET_OFFSET(&(sio)->sio_dva[0], x)
	#define SIO_SET_ASIZE(sio, x) DVA_SET_ASIZE(&(sio)->sio_dva[0], x)
	#define SIO_GET_OFFSET(sio) DVA_GET_OFFSET(&(sio)->sio_dva[0])
	#define SIO_GET_ASIZE(sio) DVA_GET_ASIZE(&(sio)->sio_dva[0])
	#define SIO_GET_END_OFFSET(sio) \
	(SIO_GET_OFFSET(sio) + SIO_GET_ASIZE(sio))
	#define SIO_GET_MUSED(sio) \
	(sizeof (scan_io_t) + ((sio)->sio_nr_dvas * sizeof (dva_t)))

	struct dsl_scan_io_queue {
	dsl_scan_t q_scn; / associated dsl_scan_t */
	vdev_t q_vd; / top-level vdev that this queue represents */

	/* trees used for sorting I/Os and extents of I/Os */
	range_tree_t *q_exts_by_addr;
	zfs_btree_t q_exts_by_size;
	avl_tree_t q_sios_by_addr;
	uint64_t q_sio_memused;

	/* members for zio rate limiting */
	uint64_t q_maxinflight_bytes;
	uint64_t q_inflight_bytes;
	kcondvar_t q_zio_cv; /* used under vd->vdev_scan_io_queue_lock */

	/* per txg statistics */
	uint64_t q_total_seg_size_this_txg;
	uint64_t q_segs_this_txg;
	uint64_t q_total_zio_size_this_txg;
	uint64_t q_zios_this_txg;
	};

	/* private data for dsl_scan_prefetch_cb() */
	typedef struct scan_prefetch_ctx {
	zfs_refcount_t spc_refcnt; /* refcount for memory management */
	dsl_scan_t spc_scn; / dsl_scan_t for the pool */
	boolean_t spc_root; /* is this prefetch for an objset? */
	uint8_t spc_indblkshift; /* dn_indblkshift of current dnode */
	uint16_t spc_datablkszsec; /* dn_idatablkszsec of current dnode */
	} scan_prefetch_ctx_t;

	/* private data for dsl_scan_prefetch() */
	typedef struct scan_prefetch_issue_ctx {
	avl_node_t spic_avl_node; /* link into scn->scn_prefetch_queue */
	scan_prefetch_ctx_t spic_spc; / spc for the callback */
	blkptr_t spic_bp; /* bp to prefetch */
	zbookmark_phys_t spic_zb; /* bookmark to prefetch */
	} scan_prefetch_issue_ctx_t;

	static void scan_exec_io(dsl_pool_t dp, const blkptr_t bp, int zio_flags,
	const zbookmark_phys_t zb, dsl_scan_io_queue_t queue);
	static void scan_io_queue_insert_impl(dsl_scan_io_queue_t *queue,
	scan_io_t *sio);

	static dsl_scan_io_queue_t scan_io_queue_create(vdev_t vd);
	static void scan_io_queues_destroy(dsl_scan_t *scn);

	static kmem_cache_t *sio_cache[SPA_DVAS_PER_BP];

	/* sio->sio_nr_dvas must be set so we know which cache to free from */
	static void
	sio_free(scan_io_t *sio)
	{
	ASSERT3U(sio->sio_nr_dvas, >, 0);
	ASSERT3U(sio->sio_nr_dvas, <=, SPA_DVAS_PER_BP);

	kmem_cache_free(sio_cache[sio->sio_nr_dvas - 1], sio);
	}

	/* It is up to the caller to set sio->sio_nr_dvas for freeing */
	static scan_io_t *
	sio_alloc(unsigned short nr_dvas)
	{
	ASSERT3U(nr_dvas, >, 0);
	ASSERT3U(nr_dvas, <=, SPA_DVAS_PER_BP);

	return (kmem_cache_alloc(sio_cache[nr_dvas - 1], KM_SLEEP));
	}

	void
	scan_init(void)
	{
	/*
	* This is used in ext_size_compare() to weight segments
	* based on how sparse they are. This cannot be changed
	* mid-scan and the tree comparison functions don't currently
	* have a mechanism for passing additional context to the
	* compare functions. Thus we store this value globally and
	* we only allow it to be set at module initialization time
	*/
	fill_weight = zfs_scan_fill_weight;

	for (int i = 0; i < SPA_DVAS_PER_BP; i++) {
	char name[36];

	(void) snprintf(name, sizeof (name), "sio_cache_%d", i);
	sio_cache[i] = kmem_cache_create(name,
	(sizeof (scan_io_t) + ((i + 1) * sizeof (dva_t))),
	0, NULL, NULL, NULL, NULL, NULL, 0);
	}
	}

	void
	scan_fini(void)
	{
	for (int i = 0; i < SPA_DVAS_PER_BP; i++) {
	kmem_cache_destroy(sio_cache[i]);
	}
	}

	static inline boolean_t
	dsl_scan_is_running(const dsl_scan_t *scn)
	{
	return (scn->scn_phys.scn_state == DSS_SCANNING);
	}

	boolean_t
	dsl_scan_resilvering(dsl_pool_t *dp)
	{
	return (dsl_scan_is_running(dp->dp_scan) &&
	dp->dp_scan->scn_phys.scn_func == POOL_SCAN_RESILVER);
	}

	static inline void
	sio2bp(const scan_io_t sio, blkptr_t bp)
	{
	bzero(bp, sizeof (*bp));
	bp->blk_prop = sio->sio_blk_prop;
	bp->blk_phys_birth = sio->sio_phys_birth;
	bp->blk_birth = sio->sio_birth;
	bp->blk_fill = 1; /* we always only work with data pointers */
	bp->blk_cksum = sio->sio_cksum;

	ASSERT3U(sio->sio_nr_dvas, >, 0);
	ASSERT3U(sio->sio_nr_dvas, <=, SPA_DVAS_PER_BP);

	bcopy(sio->sio_dva, bp->blk_dva, sio->sio_nr_dvas * sizeof (dva_t));
	}

	static inline void
	bp2sio(const blkptr_t bp, scan_io_t sio, int dva_i)
	{
	sio->sio_blk_prop = bp->blk_prop;
	sio->sio_phys_birth = bp->blk_phys_birth;
	sio->sio_birth = bp->blk_birth;
	sio->sio_cksum = bp->blk_cksum;
	sio->sio_nr_dvas = BP_GET_NDVAS(bp);

	/*
	* Copy the DVAs to the sio. We need all copies of the block so
	* that the self healing code can use the alternate copies if the
	* first is corrupted. We want the DVA at index dva_i to be first
	* in the sio since this is the primary one that we want to issue.
	*/
	for (int i = 0, j = dva_i; i < sio->sio_nr_dvas; i++, j++) {
	sio->sio_dva[i] = bp->blk_dva[j % sio->sio_nr_dvas];
	}
	}

	int
	dsl_scan_init(dsl_pool_t *dp, uint64_t txg)
	{
	int err;
	dsl_scan_t *scn;
	spa_t *spa = dp->dp_spa;
	uint64_t f;

	scn = dp->dp_scan = kmem_zalloc(sizeof (dsl_scan_t), KM_SLEEP);
	scn->scn_dp = dp;

	/*
	* It's possible that we're resuming a scan after a reboot so
	* make sure that the scan_async_destroying flag is initialized
	* appropriately.
	*/
	ASSERT(!scn->scn_async_destroying);
	scn->scn_async_destroying = spa_feature_is_active(dp->dp_spa,
	SPA_FEATURE_ASYNC_DESTROY);

	/*
	* Calculate the max number of in-flight bytes for pool-wide
	* scanning operations (minimum 1MB). Limits for the issuing
	* phase are done per top-level vdev and are handled separately.
	*/
	scn->scn_maxinflight_bytes = MAX(zfs_scan_vdev_limit *
	dsl_scan_count_leaves(spa->spa_root_vdev), 1ULL << 20);

	avl_create(&scn->scn_queue, scan_ds_queue_compare, sizeof (scan_ds_t),
	offsetof(scan_ds_t, sds_node));
	avl_create(&scn->scn_prefetch_queue, scan_prefetch_queue_compare,
	sizeof (scan_prefetch_issue_ctx_t),
	offsetof(scan_prefetch_issue_ctx_t, spic_avl_node));

	err = zap_lookup(dp->dp_meta_objset, DMU_POOL_DIRECTORY_OBJECT,
	"scrub_func", sizeof (uint64_t), 1, &f);
	if (err == 0) {
	/*
	* There was an old-style scrub in progress. Restart a
	* new-style scrub from the beginning.
	*/
	scn->scn_restart_txg = txg;
	zfs_dbgmsg("old-style scrub was in progress; "
	"restarting new-style scrub in txg %llu",
	(longlong_t)scn->scn_restart_txg);

	/*
	* Load the queue obj from the old location so that it
	* can be freed by dsl_scan_done().
	*/
	(void) zap_lookup(dp->dp_meta_objset, DMU_POOL_DIRECTORY_OBJECT,
	"scrub_queue", sizeof (uint64_t), 1,
	&scn->scn_phys.scn_queue_obj);
	} else {
	err = zap_lookup(dp->dp_meta_objset, DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_SCAN, sizeof (uint64_t), SCAN_PHYS_NUMINTS,
	&scn->scn_phys);
	/*
	* Detect if the pool contains the signature of #2094. If it
	* does properly update the scn->scn_phys structure and notify
	* the administrator by setting an errata for the pool.
	*/
	if (err == EOVERFLOW) {
	uint64_t zaptmp[SCAN_PHYS_NUMINTS + 1];
	VERIFY3S(SCAN_PHYS_NUMINTS, ==, 24);
	VERIFY3S(offsetof(dsl_scan_phys_t, scn_flags), ==,
	(23 * sizeof (uint64_t)));

	err = zap_lookup(dp->dp_meta_objset,
	DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_SCAN,
	sizeof (uint64_t), SCAN_PHYS_NUMINTS + 1, &zaptmp);
	if (err == 0) {
	uint64_t overflow = zaptmp[SCAN_PHYS_NUMINTS];

	if (overflow & ~DSL_SCAN_FLAGS_MASK \|\|
	scn->scn_async_destroying) {
	spa->spa_errata =
	ZPOOL_ERRATA_ZOL_2094_ASYNC_DESTROY;
	return (EOVERFLOW);
	}

	bcopy(zaptmp, &scn->scn_phys,
	SCAN_PHYS_NUMINTS * sizeof (uint64_t));
	scn->scn_phys.scn_flags = overflow;

	/* Required scrub already in progress. */
	if (scn->scn_phys.scn_state == DSS_FINISHED \|\|
	scn->scn_phys.scn_state == DSS_CANCELED)
	spa->spa_errata =
	ZPOOL_ERRATA_ZOL_2094_SCRUB;
	}
	}

	if (err == ENOENT)
	return (0);
	else if (err)
	return (err);

	/*
	* We might be restarting after a reboot, so jump the issued
	* counter to how far we've scanned. We know we're consistent
	* up to here.
	*/
	scn->scn_issued_before_pass = scn->scn_phys.scn_examined;

	if (dsl_scan_is_running(scn) &&
	spa_prev_software_version(dp->dp_spa) < SPA_VERSION_SCAN) {
	/*
	* A new-type scrub was in progress on an old
	* pool, and the pool was accessed by old
	* software. Restart from the beginning, since
	* the old software may have changed the pool in
	* the meantime.
	*/
	scn->scn_restart_txg = txg;
	zfs_dbgmsg("new-style scrub was modified "
	"by old software; restarting in txg %llu",
	(longlong_t)scn->scn_restart_txg);
	} else if (dsl_scan_resilvering(dp)) {
	/*
	* If a resilver is in progress and there are already
	* errors, restart it instead of finishing this scan and
	* then restarting it. If there haven't been any errors
	* then remember that the incore DTL is valid.
	*/
	if (scn->scn_phys.scn_errors > 0) {
	scn->scn_restart_txg = txg;
	zfs_dbgmsg("resilver can't excise DTL_MISSING "
	"when finished; restarting in txg %llu",
	(u_longlong_t)scn->scn_restart_txg);
	} else {
	/* it's safe to excise DTL when finished */
	spa->spa_scrub_started = B_TRUE;
	}
	}
	}

	bcopy(&scn->scn_phys, &scn->scn_phys_cached, sizeof (scn->scn_phys));

	/* reload the queue into the in-core state */
	if (scn->scn_phys.scn_queue_obj != 0) {
	zap_cursor_t zc;
	zap_attribute_t za;

	for (zap_cursor_init(&zc, dp->dp_meta_objset,
	scn->scn_phys.scn_queue_obj);
	zap_cursor_retrieve(&zc, &za) == 0;
	(void) zap_cursor_advance(&zc)) {
	scan_ds_queue_insert(scn,
	zfs_strtonum(za.za_name, NULL),
	za.za_first_integer);
	}
	zap_cursor_fini(&zc);
	}

	spa_scan_stat_init(spa);
	return (0);
	}

	void
	dsl_scan_fini(dsl_pool_t *dp)
	{
	if (dp->dp_scan != NULL) {
	dsl_scan_t *scn = dp->dp_scan;

	if (scn->scn_taskq != NULL)
	taskq_destroy(scn->scn_taskq);

	scan_ds_queue_clear(scn);
	avl_destroy(&scn->scn_queue);
	scan_ds_prefetch_queue_clear(scn);
	avl_destroy(&scn->scn_prefetch_queue);

	kmem_free(dp->dp_scan, sizeof (dsl_scan_t));
	dp->dp_scan = NULL;
	}
	}

	static boolean_t
	dsl_scan_restarting(dsl_scan_t scn, dmu_tx_t tx)
	{
	return (scn->scn_restart_txg != 0 &&
	scn->scn_restart_txg <= tx->tx_txg);
	}

	boolean_t
	dsl_scan_resilver_scheduled(dsl_pool_t *dp)
	{
	return ((dp->dp_scan && dp->dp_scan->scn_restart_txg != 0) \|\|
	(spa_async_tasks(dp->dp_spa) & SPA_ASYNC_RESILVER));
	}

	boolean_t
	dsl_scan_scrubbing(const dsl_pool_t *dp)
	{
	dsl_scan_phys_t *scn_phys = &dp->dp_scan->scn_phys;

	return (scn_phys->scn_state == DSS_SCANNING &&
	scn_phys->scn_func == POOL_SCAN_SCRUB);
	}

	boolean_t
	dsl_scan_is_paused_scrub(const dsl_scan_t *scn)
	{
	return (dsl_scan_scrubbing(scn->scn_dp) &&
	scn->scn_phys.scn_flags & DSF_SCRUB_PAUSED);
	}

	/*
	* Writes out a persistent dsl_scan_phys_t record to the pool directory.
	* Because we can be running in the block sorting algorithm, we do not always
	* want to write out the record, only when it is "safe" to do so. This safety
	* condition is achieved by making sure that the sorting queues are empty
	* (scn_bytes_pending == 0). When this condition is not true, the sync'd state
	* is inconsistent with how much actual scanning progress has been made. The
	* kind of sync to be performed is specified by the sync_type argument. If the
	* sync is optional, we only sync if the queues are empty. If the sync is
	* mandatory, we do a hard ASSERT to make sure that the queues are empty. The
	* third possible state is a "cached" sync. This is done in response to:
	* 1) The dataset that was in the last sync'd dsl_scan_phys_t having been
	* destroyed, so we wouldn't be able to restart scanning from it.
	* 2) The snapshot that was in the last sync'd dsl_scan_phys_t having been
	* superseded by a newer snapshot.
	* 3) The dataset that was in the last sync'd dsl_scan_phys_t having been
	* swapped with its clone.
	* In all cases, a cached sync simply rewrites the last record we've written,
	* just slightly modified. For the modifications that are performed to the
	* last written dsl_scan_phys_t, see dsl_scan_ds_destroyed,
	* dsl_scan_ds_snapshotted and dsl_scan_ds_clone_swapped.
	*/
	static void
	dsl_scan_sync_state(dsl_scan_t scn, dmu_tx_t tx, state_sync_type_t sync_type)
	{
	int i;
	spa_t *spa = scn->scn_dp->dp_spa;

	ASSERT(sync_type != SYNC_MANDATORY \|\| scn->scn_bytes_pending == 0);
	if (scn->scn_bytes_pending == 0) {
	for (i = 0; i < spa->spa_root_vdev->vdev_children; i++) {
	vdev_t *vd = spa->spa_root_vdev->vdev_child[i];
	dsl_scan_io_queue_t *q = vd->vdev_scan_io_queue;

	if (q == NULL)
	continue;

	mutex_enter(&vd->vdev_scan_io_queue_lock);
	ASSERT3P(avl_first(&q->q_sios_by_addr), ==, NULL);
	ASSERT3P(zfs_btree_first(&q->q_exts_by_size, NULL), ==,
	NULL);
	ASSERT3P(range_tree_first(q->q_exts_by_addr), ==, NULL);
	mutex_exit(&vd->vdev_scan_io_queue_lock);
	}

	if (scn->scn_phys.scn_queue_obj != 0)
	scan_ds_queue_sync(scn, tx);
	VERIFY0(zap_update(scn->scn_dp->dp_meta_objset,
	DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_SCAN, sizeof (uint64_t), SCAN_PHYS_NUMINTS,
	&scn->scn_phys, tx));
	bcopy(&scn->scn_phys, &scn->scn_phys_cached,
	sizeof (scn->scn_phys));

	if (scn->scn_checkpointing)
	zfs_dbgmsg("finish scan checkpoint");

	scn->scn_checkpointing = B_FALSE;
	scn->scn_last_checkpoint = ddi_get_lbolt();
	} else if (sync_type == SYNC_CACHED) {
	VERIFY0(zap_update(scn->scn_dp->dp_meta_objset,
	DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_SCAN, sizeof (uint64_t), SCAN_PHYS_NUMINTS,
	&scn->scn_phys_cached, tx));
	}
	}

	/* ARGSUSED */
	static int
	dsl_scan_setup_check(void arg, dmu_tx_t tx)
	{
	dsl_scan_t *scn = dmu_tx_pool(tx)->dp_scan;
	vdev_t *rvd = scn->scn_dp->dp_spa->spa_root_vdev;

	if (dsl_scan_is_running(scn) \|\| vdev_rebuild_active(rvd))
	return (SET_ERROR(EBUSY));

	return (0);
	}

	static void
	dsl_scan_setup_sync(void arg, dmu_tx_t tx)
	{
	dsl_scan_t *scn = dmu_tx_pool(tx)->dp_scan;
	pool_scan_func_t *funcp = arg;
	dmu_object_type_t ot = 0;
	dsl_pool_t *dp = scn->scn_dp;
	spa_t *spa = dp->dp_spa;

	ASSERT(!dsl_scan_is_running(scn));
	ASSERT(funcp > POOL_SCAN_NONE && funcp < POOL_SCAN_FUNCS);
	bzero(&scn->scn_phys, sizeof (scn->scn_phys));
	scn->scn_phys.scn_func = *funcp;
	scn->scn_phys.scn_state = DSS_SCANNING;
	scn->scn_phys.scn_min_txg = 0;
	scn->scn_phys.scn_max_txg = tx->tx_txg;
	scn->scn_phys.scn_ddt_class_max = DDT_CLASSES - 1; /* the entire DDT */
	scn->scn_phys.scn_start_time = gethrestime_sec();
	scn->scn_phys.scn_errors = 0;
	scn->scn_phys.scn_to_examine = spa->spa_root_vdev->vdev_stat.vs_alloc;
	scn->scn_issued_before_pass = 0;
	scn->scn_restart_txg = 0;
	scn->scn_done_txg = 0;
	scn->scn_last_checkpoint = 0;
	scn->scn_checkpointing = B_FALSE;
	spa_scan_stat_init(spa);

	if (DSL_SCAN_IS_SCRUB_RESILVER(scn)) {
	scn->scn_phys.scn_ddt_class_max = zfs_scrub_ddt_class_max;

	/* rewrite all disk labels */
	vdev_config_dirty(spa->spa_root_vdev);

	if (vdev_resilver_needed(spa->spa_root_vdev,
	&scn->scn_phys.scn_min_txg, &scn->scn_phys.scn_max_txg)) {
	nvlist_t *aux = fnvlist_alloc();
	fnvlist_add_string(aux, ZFS_EV_RESILVER_TYPE,
	"healing");
	spa_event_notify(spa, NULL, aux,
	ESC_ZFS_RESILVER_START);
	nvlist_free(aux);
	} else {
	spa_event_notify(spa, NULL, NULL, ESC_ZFS_SCRUB_START);
	}

	spa->spa_scrub_started = B_TRUE;
	/*
	* If this is an incremental scrub, limit the DDT scrub phase
	* to just the auto-ditto class (for correctness); the rest
	* of the scrub should go faster using top-down pruning.
	*/
	if (scn->scn_phys.scn_min_txg > TXG_INITIAL)
	scn->scn_phys.scn_ddt_class_max = DDT_CLASS_DITTO;

	/*
	* When starting a resilver clear any existing rebuild state.
	* This is required to prevent stale rebuild status from
	* being reported when a rebuild is run, then a resilver and
	* finally a scrub. In which case only the scrub status
	* should be reported by 'zpool status'.
	*/
	if (scn->scn_phys.scn_func == POOL_SCAN_RESILVER) {
	vdev_t *rvd = spa->spa_root_vdev;
	for (uint64_t i = 0; i < rvd->vdev_children; i++) {
	vdev_t *vd = rvd->vdev_child[i];
	vdev_rebuild_clear_sync(
	(void *)(uintptr_t)vd->vdev_id, tx);
	}
	}
	}

	/* back to the generic stuff */

	if (dp->dp_blkstats == NULL) {
	dp->dp_blkstats =
	vmem_alloc(sizeof (zfs_all_blkstats_t), KM_SLEEP);
	mutex_init(&dp->dp_blkstats->zab_lock, NULL,
	MUTEX_DEFAULT, NULL);
	}
	bzero(&dp->dp_blkstats->zab_type, sizeof (dp->dp_blkstats->zab_type));

	if (spa_version(spa) < SPA_VERSION_DSL_SCRUB)
	ot = DMU_OT_ZAP_OTHER;

	scn->scn_phys.scn_queue_obj = zap_create(dp->dp_meta_objset,
	ot ? ot : DMU_OT_SCAN_QUEUE, DMU_OT_NONE, 0, tx);

	bcopy(&scn->scn_phys, &scn->scn_phys_cached, sizeof (scn->scn_phys));

	dsl_scan_sync_state(scn, tx, SYNC_MANDATORY);

	spa_history_log_internal(spa, "scan setup", tx,
	"func=%u mintxg=%llu maxtxg=%llu",
	*funcp, (u_longlong_t)scn->scn_phys.scn_min_txg,
	(u_longlong_t)scn->scn_phys.scn_max_txg);
	}

	/*
	* Called by the ZFS_IOC_POOL_SCAN ioctl to start a scrub or resilver.
	* Can also be called to resume a paused scrub.
	*/
	int
	dsl_scan(dsl_pool_t *dp, pool_scan_func_t func)
	{
	spa_t *spa = dp->dp_spa;
	dsl_scan_t *scn = dp->dp_scan;

	/*
	* Purge all vdev caches and probe all devices. We do this here
	* rather than in sync context because this requires a writer lock
	* on the spa_config lock, which we can't do from sync context. The
	* spa_scrub_reopen flag indicates that vdev_open() should not
	* attempt to start another scrub.
	*/
	spa_vdev_state_enter(spa, SCL_NONE);
	spa->spa_scrub_reopen = B_TRUE;
	vdev_reopen(spa->spa_root_vdev);
	spa->spa_scrub_reopen = B_FALSE;
	(void) spa_vdev_state_exit(spa, NULL, 0);

	if (func == POOL_SCAN_RESILVER) {
	dsl_scan_restart_resilver(spa->spa_dsl_pool, 0);
	return (0);
	}

	if (func == POOL_SCAN_SCRUB && dsl_scan_is_paused_scrub(scn)) {
	/* got scrub start cmd, resume paused scrub */
	int err = dsl_scrub_set_pause_resume(scn->scn_dp,
	POOL_SCRUB_NORMAL);
	if (err == 0) {
	spa_event_notify(spa, NULL, NULL, ESC_ZFS_SCRUB_RESUME);
	return (SET_ERROR(ECANCELED));
	}

	return (SET_ERROR(err));
	}

	return (dsl_sync_task(spa_name(spa), dsl_scan_setup_check,
	dsl_scan_setup_sync, &func, 0, ZFS_SPACE_CHECK_EXTRA_RESERVED));
	}

	/* ARGSUSED */
	static void
	dsl_scan_done(dsl_scan_t scn, boolean_t complete, dmu_tx_t tx)
	{
	static const char *old_names[] = {
	"scrub_bookmark",
	"scrub_ddt_bookmark",
	"scrub_ddt_class_max",
	"scrub_queue",
	"scrub_min_txg",
	"scrub_max_txg",
	"scrub_func",
	"scrub_errors",
	NULL
	};

	dsl_pool_t *dp = scn->scn_dp;
	spa_t *spa = dp->dp_spa;
	int i;

	/* Remove any remnants of an old-style scrub. */
	for (i = 0; old_names[i]; i++) {
	(void) zap_remove(dp->dp_meta_objset,
	DMU_POOL_DIRECTORY_OBJECT, old_names[i], tx);
	}

	if (scn->scn_phys.scn_queue_obj != 0) {
	VERIFY0(dmu_object_free(dp->dp_meta_objset,
	scn->scn_phys.scn_queue_obj, tx));
	scn->scn_phys.scn_queue_obj = 0;
	}
	scan_ds_queue_clear(scn);
	scan_ds_prefetch_queue_clear(scn);

	scn->scn_phys.scn_flags &= ~DSF_SCRUB_PAUSED;

	/*
	* If we were "restarted" from a stopped state, don't bother
	* with anything else.
	*/
	if (!dsl_scan_is_running(scn)) {
	ASSERT(!scn->scn_is_sorted);
	return;
	}

	if (scn->scn_is_sorted) {
	scan_io_queues_destroy(scn);
	scn->scn_is_sorted = B_FALSE;

	if (scn->scn_taskq != NULL) {
	taskq_destroy(scn->scn_taskq);
	scn->scn_taskq = NULL;
	}
	}

	scn->scn_phys.scn_state = complete ? DSS_FINISHED : DSS_CANCELED;

	spa_notify_waiters(spa);

	if (dsl_scan_restarting(scn, tx))
	spa_history_log_internal(spa, "scan aborted, restarting", tx,
	"errors=%llu", (u_longlong_t)spa_get_errlog_size(spa));
	else if (!complete)
	spa_history_log_internal(spa, "scan cancelled", tx,
	"errors=%llu", (u_longlong_t)spa_get_errlog_size(spa));
	else
	spa_history_log_internal(spa, "scan done", tx,
	"errors=%llu", (u_longlong_t)spa_get_errlog_size(spa));

	if (DSL_SCAN_IS_SCRUB_RESILVER(scn)) {
	spa->spa_scrub_active = B_FALSE;

	/*
	* If the scrub/resilver completed, update all DTLs to
	* reflect this. Whether it succeeded or not, vacate
	* all temporary scrub DTLs.
	*
	* As the scrub does not currently support traversing
	* data that have been freed but are part of a checkpoint,
	* we don't mark the scrub as done in the DTLs as faults
	* may still exist in those vdevs.
	*/
	if (complete &&
	!spa_feature_is_active(spa, SPA_FEATURE_POOL_CHECKPOINT)) {
	vdev_dtl_reassess(spa->spa_root_vdev, tx->tx_txg,
	scn->scn_phys.scn_max_txg, B_TRUE, B_FALSE);

	if (scn->scn_phys.scn_min_txg) {
	nvlist_t *aux = fnvlist_alloc();
	fnvlist_add_string(aux, ZFS_EV_RESILVER_TYPE,
	"healing");
	spa_event_notify(spa, NULL, aux,
	ESC_ZFS_RESILVER_FINISH);
	nvlist_free(aux);
	} else {
	spa_event_notify(spa, NULL, NULL,
	ESC_ZFS_SCRUB_FINISH);
	}
	} else {
	vdev_dtl_reassess(spa->spa_root_vdev, tx->tx_txg,
	0, B_TRUE, B_FALSE);
	}
	spa_errlog_rotate(spa);

	/*
	* Don't clear flag until after vdev_dtl_reassess to ensure that
	* DTL_MISSING will get updated when possible.
	*/
	spa->spa_scrub_started = B_FALSE;

	/*
	* We may have finished replacing a device.
	* Let the async thread assess this and handle the detach.
	*/
	spa_async_request(spa, SPA_ASYNC_RESILVER_DONE);

	/*
	* Clear any resilver_deferred flags in the config.
	* If there are drives that need resilvering, kick
	* off an asynchronous request to start resilver.
	* vdev_clear_resilver_deferred() may update the config
	* before the resilver can restart. In the event of
	* a crash during this period, the spa loading code
	* will find the drives that need to be resilvered
	* and start the resilver then.
	*/
	if (spa_feature_is_enabled(spa, SPA_FEATURE_RESILVER_DEFER) &&
	vdev_clear_resilver_deferred(spa->spa_root_vdev, tx)) {
	spa_history_log_internal(spa,
	"starting deferred resilver", tx, "errors=%llu",
	(u_longlong_t)spa_get_errlog_size(spa));
	spa_async_request(spa, SPA_ASYNC_RESILVER);
	}
	}

	scn->scn_phys.scn_end_time = gethrestime_sec();

	if (spa->spa_errata == ZPOOL_ERRATA_ZOL_2094_SCRUB)
	spa->spa_errata = 0;

	ASSERT(!dsl_scan_is_running(scn));
	}

	/* ARGSUSED */
	static int
	dsl_scan_cancel_check(void arg, dmu_tx_t tx)
	{
	dsl_scan_t *scn = dmu_tx_pool(tx)->dp_scan;

	if (!dsl_scan_is_running(scn))
	return (SET_ERROR(ENOENT));
	return (0);
	}

	/* ARGSUSED */
	static void
	dsl_scan_cancel_sync(void arg, dmu_tx_t tx)
	{
	dsl_scan_t *scn = dmu_tx_pool(tx)->dp_scan;

	dsl_scan_done(scn, B_FALSE, tx);
	dsl_scan_sync_state(scn, tx, SYNC_MANDATORY);
	spa_event_notify(scn->scn_dp->dp_spa, NULL, NULL, ESC_ZFS_SCRUB_ABORT);
	}

	int
	dsl_scan_cancel(dsl_pool_t *dp)
	{
	return (dsl_sync_task(spa_name(dp->dp_spa), dsl_scan_cancel_check,
	dsl_scan_cancel_sync, NULL, 3, ZFS_SPACE_CHECK_RESERVED));
	}

	static int
	dsl_scrub_pause_resume_check(void arg, dmu_tx_t tx)
	{
	pool_scrub_cmd_t *cmd = arg;
	dsl_pool_t *dp = dmu_tx_pool(tx);
	dsl_scan_t *scn = dp->dp_scan;

	if (*cmd == POOL_SCRUB_PAUSE) {
	/* can't pause a scrub when there is no in-progress scrub */
	if (!dsl_scan_scrubbing(dp))
	return (SET_ERROR(ENOENT));

	/* can't pause a paused scrub */
	if (dsl_scan_is_paused_scrub(scn))
	return (SET_ERROR(EBUSY));
	} else if (*cmd != POOL_SCRUB_NORMAL) {
	return (SET_ERROR(ENOTSUP));
	}

	return (0);
	}

	static void
	dsl_scrub_pause_resume_sync(void arg, dmu_tx_t tx)
	{
	pool_scrub_cmd_t *cmd = arg;
	dsl_pool_t *dp = dmu_tx_pool(tx);
	spa_t *spa = dp->dp_spa;
	dsl_scan_t *scn = dp->dp_scan;

	if (*cmd == POOL_SCRUB_PAUSE) {
	/* can't pause a scrub when there is no in-progress scrub */
	spa->spa_scan_pass_scrub_pause = gethrestime_sec();
	scn->scn_phys.scn_flags \|= DSF_SCRUB_PAUSED;
	scn->scn_phys_cached.scn_flags \|= DSF_SCRUB_PAUSED;
	dsl_scan_sync_state(scn, tx, SYNC_CACHED);
	spa_event_notify(spa, NULL, NULL, ESC_ZFS_SCRUB_PAUSED);
	spa_notify_waiters(spa);
	} else {
	ASSERT3U(*cmd, ==, POOL_SCRUB_NORMAL);
	if (dsl_scan_is_paused_scrub(scn)) {
	/*
	* We need to keep track of how much time we spend
	* paused per pass so that we can adjust the scrub rate
	* shown in the output of 'zpool status'
	*/
	spa->spa_scan_pass_scrub_spent_paused +=
	gethrestime_sec() - spa->spa_scan_pass_scrub_pause;
	spa->spa_scan_pass_scrub_pause = 0;
	scn->scn_phys.scn_flags &= ~DSF_SCRUB_PAUSED;
	scn->scn_phys_cached.scn_flags &= ~DSF_SCRUB_PAUSED;
	dsl_scan_sync_state(scn, tx, SYNC_CACHED);
	}
	}
	}

	/*
	* Set scrub pause/resume state if it makes sense to do so
	*/
	int
	dsl_scrub_set_pause_resume(const dsl_pool_t *dp, pool_scrub_cmd_t cmd)
	{
	return (dsl_sync_task(spa_name(dp->dp_spa),
	dsl_scrub_pause_resume_check, dsl_scrub_pause_resume_sync, &cmd, 3,
	ZFS_SPACE_CHECK_RESERVED));
	}


	/* start a new scan, or restart an existing one. */
	void
	dsl_scan_restart_resilver(dsl_pool_t *dp, uint64_t txg)
	{
	if (txg == 0) {
	dmu_tx_t *tx;
	tx = dmu_tx_create_dd(dp->dp_mos_dir);
	VERIFY(0 == dmu_tx_assign(tx, TXG_WAIT));

	txg = dmu_tx_get_txg(tx);
	dp->dp_scan->scn_restart_txg = txg;
	dmu_tx_commit(tx);
	} else {
	dp->dp_scan->scn_restart_txg = txg;
	}
	zfs_dbgmsg("restarting resilver txg=%llu", (longlong_t)txg);
	}

	void
	dsl_free(dsl_pool_t dp, uint64_t txg, const blkptr_t bp)
	{
	zio_free(dp->dp_spa, txg, bp);
	}

	void
	dsl_free_sync(zio_t pio, dsl_pool_t dp, uint64_t txg, const blkptr_t *bpp)
	{
	ASSERT(dsl_pool_sync_context(dp));
	zio_nowait(zio_free_sync(pio, dp->dp_spa, txg, bpp, pio->io_flags));
	}

	static int
	scan_ds_queue_compare(const void a, const void b)
	{
	const scan_ds_t sds_a = a, sds_b = b;

	if (sds_a->sds_dsobj < sds_b->sds_dsobj)
	return (-1);
	if (sds_a->sds_dsobj == sds_b->sds_dsobj)
	return (0);
	return (1);
	}

	static void
	scan_ds_queue_clear(dsl_scan_t *scn)
	{
	void *cookie = NULL;
	scan_ds_t *sds;
	while ((sds = avl_destroy_nodes(&scn->scn_queue, &cookie)) != NULL) {
	kmem_free(sds, sizeof (*sds));
	}
	}

	static boolean_t
	scan_ds_queue_contains(dsl_scan_t scn, uint64_t dsobj, uint64_t txg)
	{
	scan_ds_t srch, *sds;

	srch.sds_dsobj = dsobj;
	sds = avl_find(&scn->scn_queue, &srch, NULL);
	if (sds != NULL && txg != NULL)
	*txg = sds->sds_txg;
	return (sds != NULL);
	}

	static void
	scan_ds_queue_insert(dsl_scan_t *scn, uint64_t dsobj, uint64_t txg)
	{
	scan_ds_t *sds;
	avl_index_t where;

	sds = kmem_zalloc(sizeof (*sds), KM_SLEEP);
	sds->sds_dsobj = dsobj;
	sds->sds_txg = txg;

	VERIFY3P(avl_find(&scn->scn_queue, sds, &where), ==, NULL);
	avl_insert(&scn->scn_queue, sds, where);
	}

	static void
	scan_ds_queue_remove(dsl_scan_t *scn, uint64_t dsobj)
	{
	scan_ds_t srch, *sds;

	srch.sds_dsobj = dsobj;

	sds = avl_find(&scn->scn_queue, &srch, NULL);
	VERIFY(sds != NULL);
	avl_remove(&scn->scn_queue, sds);
	kmem_free(sds, sizeof (*sds));
	}

	static void
	scan_ds_queue_sync(dsl_scan_t scn, dmu_tx_t tx)
	{
	dsl_pool_t *dp = scn->scn_dp;
	spa_t *spa = dp->dp_spa;
	dmu_object_type_t ot = (spa_version(spa) >= SPA_VERSION_DSL_SCRUB) ?
	DMU_OT_SCAN_QUEUE : DMU_OT_ZAP_OTHER;

	ASSERT0(scn->scn_bytes_pending);
	ASSERT(scn->scn_phys.scn_queue_obj != 0);

	VERIFY0(dmu_object_free(dp->dp_meta_objset,
	scn->scn_phys.scn_queue_obj, tx));
	scn->scn_phys.scn_queue_obj = zap_create(dp->dp_meta_objset, ot,
	DMU_OT_NONE, 0, tx);
	for (scan_ds_t *sds = avl_first(&scn->scn_queue);
	sds != NULL; sds = AVL_NEXT(&scn->scn_queue, sds)) {
	VERIFY0(zap_add_int_key(dp->dp_meta_objset,
	scn->scn_phys.scn_queue_obj, sds->sds_dsobj,
	sds->sds_txg, tx));
	}
	}

	/*
	* Computes the memory limit state that we're currently in. A sorted scan
	* needs quite a bit of memory to hold the sorting queue, so we need to
	* reasonably constrain the size so it doesn't impact overall system
	* performance. We compute two limits:
	* 1) Hard memory limit: if the amount of memory used by the sorting
	* queues on a pool gets above this value, we stop the metadata
	* scanning portion and start issuing the queued up and sorted
	* I/Os to reduce memory usage.
	* This limit is calculated as a fraction of physmem (by default 5%).
	* We constrain the lower bound of the hard limit to an absolute
	* minimum of zfs_scan_mem_lim_min (default: 16 MiB). We also constrain
	* the upper bound to 5% of the total pool size - no chance we'll
	* ever need that much memory, but just to keep the value in check.
	* 2) Soft memory limit: once we hit the hard memory limit, we start
	* issuing I/O to reduce queue memory usage, but we don't want to
	* completely empty out the queues, since we might be able to find I/Os
	* that will fill in the gaps of our non-sequential IOs at some point
	* in the future. So we stop the issuing of I/Os once the amount of
	* memory used drops below the soft limit (at which point we stop issuing
	* I/O and start scanning metadata again).
	*
	* This limit is calculated by subtracting a fraction of the hard
	* limit from the hard limit. By default this fraction is 5%, so
	* the soft limit is 95% of the hard limit. We cap the size of the
	* difference between the hard and soft limits at an absolute
	* maximum of zfs_scan_mem_lim_soft_max (default: 128 MiB) - this is
	* sufficient to not cause too frequent switching between the
	* metadata scan and I/O issue (even at 2k recordsize, 128 MiB's
	* worth of queues is about 1.2 GiB of on-pool data, so scanning
	* that should take at least a decent fraction of a second).
	*/
	static boolean_t
	dsl_scan_should_clear(dsl_scan_t *scn)
	{
	spa_t *spa = scn->scn_dp->dp_spa;
	vdev_t *rvd = scn->scn_dp->dp_spa->spa_root_vdev;
	uint64_t alloc, mlim_hard, mlim_soft, mused;

	alloc = metaslab_class_get_alloc(spa_normal_class(spa));
	alloc += metaslab_class_get_alloc(spa_special_class(spa));
	alloc += metaslab_class_get_alloc(spa_dedup_class(spa));

	mlim_hard = MAX((physmem / zfs_scan_mem_lim_fact) * PAGESIZE,
	zfs_scan_mem_lim_min);
	mlim_hard = MIN(mlim_hard, alloc / 20);
	mlim_soft = mlim_hard - MIN(mlim_hard / zfs_scan_mem_lim_soft_fact,
	zfs_scan_mem_lim_soft_max);
	mused = 0;
	for (uint64_t i = 0; i < rvd->vdev_children; i++) {
	vdev_t *tvd = rvd->vdev_child[i];
	dsl_scan_io_queue_t *queue;

	mutex_enter(&tvd->vdev_scan_io_queue_lock);
	queue = tvd->vdev_scan_io_queue;
	if (queue != NULL) {
	/* # extents in exts_by_size = # in exts_by_addr */
	mused += zfs_btree_numnodes(&queue->q_exts_by_size) *
	sizeof (range_seg_gap_t) + queue->q_sio_memused;
	}
	mutex_exit(&tvd->vdev_scan_io_queue_lock);
	}

	dprintf("current scan memory usage: %llu bytes\n", (longlong_t)mused);

	if (mused == 0)
	ASSERT0(scn->scn_bytes_pending);

	/*
	* If we are above our hard limit, we need to clear out memory.
	* If we are below our soft limit, we need to accumulate sequential IOs.
	* Otherwise, we should keep doing whatever we are currently doing.
	*/
	if (mused >= mlim_hard)
	return (B_TRUE);
	else if (mused < mlim_soft)
	return (B_FALSE);
	else
	return (scn->scn_clearing);
	}

	static boolean_t
	dsl_scan_check_suspend(dsl_scan_t scn, const zbookmark_phys_t zb)
	{
	/* we never skip user/group accounting objects */
	if (zb && (int64_t)zb->zb_object < 0)
	return (B_FALSE);

	if (scn->scn_suspending)
	return (B_TRUE); /* we're already suspending */

	if (!ZB_IS_ZERO(&scn->scn_phys.scn_bookmark))
	return (B_FALSE); /* we're resuming */

	/* We only know how to resume from level-0 and objset blocks. */
	if (zb && (zb->zb_level != 0 && zb->zb_level != ZB_ROOT_LEVEL))
	return (B_FALSE);

	/*
	* We suspend if:
	* - we have scanned for at least the minimum time (default 1 sec
	* for scrub, 3 sec for resilver), and either we have sufficient
	* dirty data that we are starting to write more quickly
	* (default 30%), someone is explicitly waiting for this txg
	* to complete, or we have used up all of the time in the txg
	* timeout (default 5 sec).
	* or
	* - the spa is shutting down because this pool is being exported
	* or the machine is rebooting.
	* or
	* - the scan queue has reached its memory use limit
	*/
	uint64_t curr_time_ns = gethrtime();
	uint64_t scan_time_ns = curr_time_ns - scn->scn_sync_start_time;
	uint64_t sync_time_ns = curr_time_ns -
	scn->scn_dp->dp_spa->spa_sync_starttime;
	int dirty_pct = scn->scn_dp->dp_dirty_total * 100 / zfs_dirty_data_max;
	int mintime = (scn->scn_phys.scn_func == POOL_SCAN_RESILVER) ?
	zfs_resilver_min_time_ms : zfs_scrub_min_time_ms;

	if ((NSEC2MSEC(scan_time_ns) > mintime &&
	(dirty_pct >= zfs_vdev_async_write_active_min_dirty_percent \|\|
	txg_sync_waiting(scn->scn_dp) \|\|
	NSEC2SEC(sync_time_ns) >= zfs_txg_timeout)) \|\|
	spa_shutting_down(scn->scn_dp->dp_spa) \|\|
	(zfs_scan_strict_mem_lim && dsl_scan_should_clear(scn))) {
	if (zb && zb->zb_level == ZB_ROOT_LEVEL) {
	dprintf("suspending at first available bookmark "
	"%llx/%llx/%llx/%llx\n",
	(longlong_t)zb->zb_objset,
	(longlong_t)zb->zb_object,
	(longlong_t)zb->zb_level,
	(longlong_t)zb->zb_blkid);
	SET_BOOKMARK(&scn->scn_phys.scn_bookmark,
	zb->zb_objset, 0, 0, 0);
	} else if (zb != NULL) {
	dprintf("suspending at bookmark %llx/%llx/%llx/%llx\n",
	(longlong_t)zb->zb_objset,
	(longlong_t)zb->zb_object,
	(longlong_t)zb->zb_level,
	(longlong_t)zb->zb_blkid);
	scn->scn_phys.scn_bookmark = *zb;
	} else {
	#ifdef ZFS_DEBUG
	dsl_scan_phys_t *scnp = &scn->scn_phys;
	dprintf("suspending at at DDT bookmark "
	"%llx/%llx/%llx/%llx\n",
	(longlong_t)scnp->scn_ddt_bookmark.ddb_class,
	(longlong_t)scnp->scn_ddt_bookmark.ddb_type,
	(longlong_t)scnp->scn_ddt_bookmark.ddb_checksum,
	(longlong_t)scnp->scn_ddt_bookmark.ddb_cursor);
	#endif
	}
	scn->scn_suspending = B_TRUE;
	return (B_TRUE);
	}
	return (B_FALSE);
	}

	typedef struct zil_scan_arg {
	dsl_pool_t *zsa_dp;
	zil_header_t *zsa_zh;
	} zil_scan_arg_t;

	/* ARGSUSED */
	static int
	-dsl_scan_zil_block(zilog_t zilog, blkptr_t bp, void *arg, uint64_t claim_txg)
	+dsl_scan_zil_block(zilog_t zilog, const blkptr_t bp, void *arg,
	+ uint64_t claim_txg)
	{
	zil_scan_arg_t *zsa = arg;
	dsl_pool_t *dp = zsa->zsa_dp;
	dsl_scan_t *scn = dp->dp_scan;
	zil_header_t *zh = zsa->zsa_zh;
	zbookmark_phys_t zb;

	ASSERT(!BP_IS_REDACTED(bp));
	if (BP_IS_HOLE(bp) \|\| bp->blk_birth <= scn->scn_phys.scn_cur_min_txg)
	return (0);

	/*
	* One block ("stubby") can be allocated a long time ago; we
	* want to visit that one because it has been allocated
	* (on-disk) even if it hasn't been claimed (even though for
	* scrub there's nothing to do to it).
	*/
	if (claim_txg == 0 && bp->blk_birth >= spa_min_claim_txg(dp->dp_spa))
	return (0);

	SET_BOOKMARK(&zb, zh->zh_log.blk_cksum.zc_word[ZIL_ZC_OBJSET],
	ZB_ZIL_OBJECT, ZB_ZIL_LEVEL, bp->blk_cksum.zc_word[ZIL_ZC_SEQ]);

	VERIFY(0 == scan_funcs[scn->scn_phys.scn_func](dp, bp, &zb));
	return (0);
	}

	/* ARGSUSED */
	static int
	-dsl_scan_zil_record(zilog_t zilog, lr_t lrc, void *arg, uint64_t claim_txg)
	+dsl_scan_zil_record(zilog_t zilog, const lr_t lrc, void *arg,
	+ uint64_t claim_txg)
	{
	if (lrc->lrc_txtype == TX_WRITE) {
	zil_scan_arg_t *zsa = arg;
	dsl_pool_t *dp = zsa->zsa_dp;
	dsl_scan_t *scn = dp->dp_scan;
	zil_header_t *zh = zsa->zsa_zh;
	- lr_write_t lr = (lr_write_t )lrc;
	- blkptr_t *bp = &lr->lr_blkptr;
	+ const lr_write_t lr = (const lr_write_t )lrc;
	+ const blkptr_t *bp = &lr->lr_blkptr;
	zbookmark_phys_t zb;

	ASSERT(!BP_IS_REDACTED(bp));
	if (BP_IS_HOLE(bp) \|\|
	bp->blk_birth <= scn->scn_phys.scn_cur_min_txg)
	return (0);

	/*
	* birth can be < claim_txg if this record's txg is
	* already txg sync'ed (but this log block contains
	* other records that are not synced)
	*/
	if (claim_txg == 0 \|\| bp->blk_birth < claim_txg)
	return (0);

	SET_BOOKMARK(&zb, zh->zh_log.blk_cksum.zc_word[ZIL_ZC_OBJSET],
	lr->lr_foid, ZB_ZIL_LEVEL,
	lr->lr_offset / BP_GET_LSIZE(bp));

	VERIFY(0 == scan_funcs[scn->scn_phys.scn_func](dp, bp, &zb));
	}
	return (0);
	}

	static void
	dsl_scan_zil(dsl_pool_t dp, zil_header_t zh)
	{
	uint64_t claim_txg = zh->zh_claim_txg;
	zil_scan_arg_t zsa = { dp, zh };
	zilog_t *zilog;

	ASSERT(spa_writeable(dp->dp_spa));

	/*
	* We only want to visit blocks that have been claimed but not yet
	* replayed (or, in read-only mode, blocks that would be claimed).
	*/
	if (claim_txg == 0)
	return;

	zilog = zil_alloc(dp->dp_meta_objset, zh);

	(void) zil_parse(zilog, dsl_scan_zil_block, dsl_scan_zil_record, &zsa,
	claim_txg, B_FALSE);

	zil_free(zilog);
	}

	/*
	* We compare scan_prefetch_issue_ctx_t's based on their bookmarks. The idea
	* here is to sort the AVL tree by the order each block will be needed.
	*/
	static int
	scan_prefetch_queue_compare(const void a, const void b)
	{
	const scan_prefetch_issue_ctx_t spic_a = a, spic_b = b;
	const scan_prefetch_ctx_t *spc_a = spic_a->spic_spc;
	const scan_prefetch_ctx_t *spc_b = spic_b->spic_spc;

	return (zbookmark_compare(spc_a->spc_datablkszsec,
	spc_a->spc_indblkshift, spc_b->spc_datablkszsec,
	spc_b->spc_indblkshift, &spic_a->spic_zb, &spic_b->spic_zb));
	}

	static void
	scan_prefetch_ctx_rele(scan_prefetch_ctx_t spc, void tag)
	{
	if (zfs_refcount_remove(&spc->spc_refcnt, tag) == 0) {
	zfs_refcount_destroy(&spc->spc_refcnt);
	kmem_free(spc, sizeof (scan_prefetch_ctx_t));
	}
	}

	static scan_prefetch_ctx_t *
	scan_prefetch_ctx_create(dsl_scan_t scn, dnode_phys_t dnp, void *tag)
	{
	scan_prefetch_ctx_t *spc;

	spc = kmem_alloc(sizeof (scan_prefetch_ctx_t), KM_SLEEP);
	zfs_refcount_create(&spc->spc_refcnt);
	zfs_refcount_add(&spc->spc_refcnt, tag);
	spc->spc_scn = scn;
	if (dnp != NULL) {
	spc->spc_datablkszsec = dnp->dn_datablkszsec;
	spc->spc_indblkshift = dnp->dn_indblkshift;
	spc->spc_root = B_FALSE;
	} else {
	spc->spc_datablkszsec = 0;
	spc->spc_indblkshift = 0;
	spc->spc_root = B_TRUE;
	}

	return (spc);
	}

	static void
	scan_prefetch_ctx_add_ref(scan_prefetch_ctx_t spc, void tag)
	{
	zfs_refcount_add(&spc->spc_refcnt, tag);
	}

	static void
	scan_ds_prefetch_queue_clear(dsl_scan_t *scn)
	{
	spa_t *spa = scn->scn_dp->dp_spa;
	void *cookie = NULL;
	scan_prefetch_issue_ctx_t *spic = NULL;

	mutex_enter(&spa->spa_scrub_lock);
	while ((spic = avl_destroy_nodes(&scn->scn_prefetch_queue,
	&cookie)) != NULL) {
	scan_prefetch_ctx_rele(spic->spic_spc, scn);
	kmem_free(spic, sizeof (scan_prefetch_issue_ctx_t));
	}
	mutex_exit(&spa->spa_scrub_lock);
	}

	static boolean_t
	dsl_scan_check_prefetch_resume(scan_prefetch_ctx_t *spc,
	const zbookmark_phys_t *zb)
	{
	zbookmark_phys_t *last_zb = &spc->spc_scn->scn_prefetch_bookmark;
	dnode_phys_t tmp_dnp;
	dnode_phys_t *dnp = (spc->spc_root) ? NULL : &tmp_dnp;

	if (zb->zb_objset != last_zb->zb_objset)
	return (B_TRUE);
	if ((int64_t)zb->zb_object < 0)
	return (B_FALSE);

	tmp_dnp.dn_datablkszsec = spc->spc_datablkszsec;
	tmp_dnp.dn_indblkshift = spc->spc_indblkshift;

	if (zbookmark_subtree_completed(dnp, zb, last_zb))
	return (B_TRUE);

	return (B_FALSE);
	}

	static void
	dsl_scan_prefetch(scan_prefetch_ctx_t spc, blkptr_t bp, zbookmark_phys_t *zb)
	{
	avl_index_t idx;
	dsl_scan_t *scn = spc->spc_scn;
	spa_t *spa = scn->scn_dp->dp_spa;
	scan_prefetch_issue_ctx_t *spic;

	if (zfs_no_scrub_prefetch \|\| BP_IS_REDACTED(bp))
	return;

	if (BP_IS_HOLE(bp) \|\| bp->blk_birth <= scn->scn_phys.scn_cur_min_txg \|\|
	(BP_GET_LEVEL(bp) == 0 && BP_GET_TYPE(bp) != DMU_OT_DNODE &&
	BP_GET_TYPE(bp) != DMU_OT_OBJSET))
	return;

	if (dsl_scan_check_prefetch_resume(spc, zb))
	return;

	scan_prefetch_ctx_add_ref(spc, scn);
	spic = kmem_alloc(sizeof (scan_prefetch_issue_ctx_t), KM_SLEEP);
	spic->spic_spc = spc;
	spic->spic_bp = *bp;
	spic->spic_zb = *zb;

	/*
	* Add the IO to the queue of blocks to prefetch. This allows us to
	* prioritize blocks that we will need first for the main traversal
	* thread.
	*/
	mutex_enter(&spa->spa_scrub_lock);
	if (avl_find(&scn->scn_prefetch_queue, spic, &idx) != NULL) {
	/* this block is already queued for prefetch */
	kmem_free(spic, sizeof (scan_prefetch_issue_ctx_t));
	scan_prefetch_ctx_rele(spc, scn);
	mutex_exit(&spa->spa_scrub_lock);
	return;
	}

	avl_insert(&scn->scn_prefetch_queue, spic, idx);
	cv_broadcast(&spa->spa_scrub_io_cv);
	mutex_exit(&spa->spa_scrub_lock);
	}

	static void
	dsl_scan_prefetch_dnode(dsl_scan_t scn, dnode_phys_t dnp,
	uint64_t objset, uint64_t object)
	{
	int i;
	zbookmark_phys_t zb;
	scan_prefetch_ctx_t *spc;

	if (dnp->dn_nblkptr == 0 && !(dnp->dn_flags & DNODE_FLAG_SPILL_BLKPTR))
	return;

	SET_BOOKMARK(&zb, objset, object, 0, 0);

	spc = scan_prefetch_ctx_create(scn, dnp, FTAG);

	for (i = 0; i < dnp->dn_nblkptr; i++) {
	zb.zb_level = BP_GET_LEVEL(&dnp->dn_blkptr[i]);
	zb.zb_blkid = i;
	dsl_scan_prefetch(spc, &dnp->dn_blkptr[i], &zb);
	}

	if (dnp->dn_flags & DNODE_FLAG_SPILL_BLKPTR) {
	zb.zb_level = 0;
	zb.zb_blkid = DMU_SPILL_BLKID;
	dsl_scan_prefetch(spc, DN_SPILL_BLKPTR(dnp), &zb);
	}

	scan_prefetch_ctx_rele(spc, FTAG);
	}

	static void
	dsl_scan_prefetch_cb(zio_t zio, const zbookmark_phys_t zb, const blkptr_t *bp,
	arc_buf_t buf, void private)
	{
	scan_prefetch_ctx_t *spc = private;
	dsl_scan_t *scn = spc->spc_scn;
	spa_t *spa = scn->scn_dp->dp_spa;

	/* broadcast that the IO has completed for rate limiting purposes */
	mutex_enter(&spa->spa_scrub_lock);
	ASSERT3U(spa->spa_scrub_inflight, >=, BP_GET_PSIZE(bp));
	spa->spa_scrub_inflight -= BP_GET_PSIZE(bp);
	cv_broadcast(&spa->spa_scrub_io_cv);
	mutex_exit(&spa->spa_scrub_lock);

	/* if there was an error or we are done prefetching, just cleanup */
	if (buf == NULL \|\| scn->scn_prefetch_stop)
	goto out;

	if (BP_GET_LEVEL(bp) > 0) {
	int i;
	blkptr_t *cbp;
	int epb = BP_GET_LSIZE(bp) >> SPA_BLKPTRSHIFT;
	zbookmark_phys_t czb;

	for (i = 0, cbp = buf->b_data; i < epb; i++, cbp++) {
	SET_BOOKMARK(&czb, zb->zb_objset, zb->zb_object,
	zb->zb_level - 1, zb->zb_blkid * epb + i);
	dsl_scan_prefetch(spc, cbp, &czb);
	}
	} else if (BP_GET_TYPE(bp) == DMU_OT_DNODE) {
	dnode_phys_t *cdnp;
	int i;
	int epb = BP_GET_LSIZE(bp) >> DNODE_SHIFT;

	for (i = 0, cdnp = buf->b_data; i < epb;
	i += cdnp->dn_extra_slots + 1,
	cdnp += cdnp->dn_extra_slots + 1) {
	dsl_scan_prefetch_dnode(scn, cdnp,
	zb->zb_objset, zb->zb_blkid * epb + i);
	}
	} else if (BP_GET_TYPE(bp) == DMU_OT_OBJSET) {
	objset_phys_t *osp = buf->b_data;

	dsl_scan_prefetch_dnode(scn, &osp->os_meta_dnode,
	zb->zb_objset, DMU_META_DNODE_OBJECT);

	if (OBJSET_BUF_HAS_USERUSED(buf)) {
	dsl_scan_prefetch_dnode(scn,
	&osp->os_groupused_dnode, zb->zb_objset,
	DMU_GROUPUSED_OBJECT);
	dsl_scan_prefetch_dnode(scn,
	&osp->os_userused_dnode, zb->zb_objset,
	DMU_USERUSED_OBJECT);
	}
	}

	out:
	if (buf != NULL)
	arc_buf_destroy(buf, private);
	scan_prefetch_ctx_rele(spc, scn);
	}

	/* ARGSUSED */
	static void
	dsl_scan_prefetch_thread(void *arg)
	{
	dsl_scan_t *scn = arg;
	spa_t *spa = scn->scn_dp->dp_spa;
	scan_prefetch_issue_ctx_t *spic;

	/* loop until we are told to stop */
	while (!scn->scn_prefetch_stop) {
	arc_flags_t flags = ARC_FLAG_NOWAIT \|
	ARC_FLAG_PRESCIENT_PREFETCH \| ARC_FLAG_PREFETCH;
	int zio_flags = ZIO_FLAG_CANFAIL \| ZIO_FLAG_SCAN_THREAD;

	mutex_enter(&spa->spa_scrub_lock);

	/*
	* Wait until we have an IO to issue and are not above our
	* maximum in flight limit.
	*/
	while (!scn->scn_prefetch_stop &&
	(avl_numnodes(&scn->scn_prefetch_queue) == 0 \|\|
	spa->spa_scrub_inflight >= scn->scn_maxinflight_bytes)) {
	cv_wait(&spa->spa_scrub_io_cv, &spa->spa_scrub_lock);
	}

	/* recheck if we should stop since we waited for the cv */
	if (scn->scn_prefetch_stop) {
	mutex_exit(&spa->spa_scrub_lock);
	break;
	}

	/* remove the prefetch IO from the tree */
	spic = avl_first(&scn->scn_prefetch_queue);
	spa->spa_scrub_inflight += BP_GET_PSIZE(&spic->spic_bp);
	avl_remove(&scn->scn_prefetch_queue, spic);

	mutex_exit(&spa->spa_scrub_lock);

	if (BP_IS_PROTECTED(&spic->spic_bp)) {
	ASSERT(BP_GET_TYPE(&spic->spic_bp) == DMU_OT_DNODE \|\|
	BP_GET_TYPE(&spic->spic_bp) == DMU_OT_OBJSET);
	ASSERT3U(BP_GET_LEVEL(&spic->spic_bp), ==, 0);
	zio_flags \|= ZIO_FLAG_RAW;
	}

	/* issue the prefetch asynchronously */
	(void) arc_read(scn->scn_zio_root, scn->scn_dp->dp_spa,
	&spic->spic_bp, dsl_scan_prefetch_cb, spic->spic_spc,
	ZIO_PRIORITY_SCRUB, zio_flags, &flags, &spic->spic_zb);

	kmem_free(spic, sizeof (scan_prefetch_issue_ctx_t));
	}

	ASSERT(scn->scn_prefetch_stop);

	/* free any prefetches we didn't get to complete */
	mutex_enter(&spa->spa_scrub_lock);
	while ((spic = avl_first(&scn->scn_prefetch_queue)) != NULL) {
	avl_remove(&scn->scn_prefetch_queue, spic);
	scan_prefetch_ctx_rele(spic->spic_spc, scn);
	kmem_free(spic, sizeof (scan_prefetch_issue_ctx_t));
	}
	ASSERT0(avl_numnodes(&scn->scn_prefetch_queue));
	mutex_exit(&spa->spa_scrub_lock);
	}

	static boolean_t
	dsl_scan_check_resume(dsl_scan_t scn, const dnode_phys_t dnp,
	const zbookmark_phys_t *zb)
	{
	/*
	* We never skip over user/group accounting objects (obj<0)
	*/
	if (!ZB_IS_ZERO(&scn->scn_phys.scn_bookmark) &&
	(int64_t)zb->zb_object >= 0) {
	/*
	* If we already visited this bp & everything below (in
	* a prior txg sync), don't bother doing it again.
	*/
	if (zbookmark_subtree_completed(dnp, zb,
	&scn->scn_phys.scn_bookmark))
	return (B_TRUE);

	/*
	* If we found the block we're trying to resume from, or
	* we went past it to a different object, zero it out to
	* indicate that it's OK to start checking for suspending
	* again.
	*/
	if (bcmp(zb, &scn->scn_phys.scn_bookmark, sizeof (*zb)) == 0 \|\|
	zb->zb_object > scn->scn_phys.scn_bookmark.zb_object) {
	dprintf("resuming at %llx/%llx/%llx/%llx\n",
	(longlong_t)zb->zb_objset,
	(longlong_t)zb->zb_object,
	(longlong_t)zb->zb_level,
	(longlong_t)zb->zb_blkid);
	bzero(&scn->scn_phys.scn_bookmark, sizeof (*zb));
	}
	}
	return (B_FALSE);
	}

	static void dsl_scan_visitbp(blkptr_t bp, const zbookmark_phys_t zb,
	dnode_phys_t dnp, dsl_dataset_t ds, dsl_scan_t *scn,
	dmu_objset_type_t ostype, dmu_tx_t *tx);
	inline __attribute__((always_inline)) static void dsl_scan_visitdnode(
	dsl_scan_t , dsl_dataset_t ds, dmu_objset_type_t ostype,
	dnode_phys_t dnp, uint64_t object, dmu_tx_t tx);

	/*
	* Return nonzero on i/o error.
	* Return new buf to write out in *bufp.
	*/
	inline __attribute__((always_inline)) static int
	dsl_scan_recurse(dsl_scan_t scn, dsl_dataset_t ds, dmu_objset_type_t ostype,
	dnode_phys_t dnp, const blkptr_t bp,
	const zbookmark_phys_t zb, dmu_tx_t tx)
	{
	dsl_pool_t *dp = scn->scn_dp;
	int zio_flags = ZIO_FLAG_CANFAIL \| ZIO_FLAG_SCAN_THREAD;
	int err;

	ASSERT(!BP_IS_REDACTED(bp));

	if (BP_GET_LEVEL(bp) > 0) {
	arc_flags_t flags = ARC_FLAG_WAIT;
	int i;
	blkptr_t *cbp;
	int epb = BP_GET_LSIZE(bp) >> SPA_BLKPTRSHIFT;
	arc_buf_t *buf;

	err = arc_read(NULL, dp->dp_spa, bp, arc_getbuf_func, &buf,
	ZIO_PRIORITY_SCRUB, zio_flags, &flags, zb);
	if (err) {
	scn->scn_phys.scn_errors++;
	return (err);
	}
	for (i = 0, cbp = buf->b_data; i < epb; i++, cbp++) {
	zbookmark_phys_t czb;

	SET_BOOKMARK(&czb, zb->zb_objset, zb->zb_object,
	zb->zb_level - 1,
	zb->zb_blkid * epb + i);
	dsl_scan_visitbp(cbp, &czb, dnp,
	ds, scn, ostype, tx);
	}
	arc_buf_destroy(buf, &buf);
	} else if (BP_GET_TYPE(bp) == DMU_OT_DNODE) {
	arc_flags_t flags = ARC_FLAG_WAIT;
	dnode_phys_t *cdnp;
	int i;
	int epb = BP_GET_LSIZE(bp) >> DNODE_SHIFT;
	arc_buf_t *buf;

	if (BP_IS_PROTECTED(bp)) {
	ASSERT3U(BP_GET_COMPRESS(bp), ==, ZIO_COMPRESS_OFF);
	zio_flags \|= ZIO_FLAG_RAW;
	}

	err = arc_read(NULL, dp->dp_spa, bp, arc_getbuf_func, &buf,
	ZIO_PRIORITY_SCRUB, zio_flags, &flags, zb);
	if (err) {
	scn->scn_phys.scn_errors++;
	return (err);
	}
	for (i = 0, cdnp = buf->b_data; i < epb;
	i += cdnp->dn_extra_slots + 1,
	cdnp += cdnp->dn_extra_slots + 1) {
	dsl_scan_visitdnode(scn, ds, ostype,
	cdnp, zb->zb_blkid * epb + i, tx);
	}

	arc_buf_destroy(buf, &buf);
	} else if (BP_GET_TYPE(bp) == DMU_OT_OBJSET) {
	arc_flags_t flags = ARC_FLAG_WAIT;
	objset_phys_t *osp;
	arc_buf_t *buf;

	err = arc_read(NULL, dp->dp_spa, bp, arc_getbuf_func, &buf,
	ZIO_PRIORITY_SCRUB, zio_flags, &flags, zb);
	if (err) {
	scn->scn_phys.scn_errors++;
	return (err);
	}

	osp = buf->b_data;

	dsl_scan_visitdnode(scn, ds, osp->os_type,
	&osp->os_meta_dnode, DMU_META_DNODE_OBJECT, tx);

	if (OBJSET_BUF_HAS_USERUSED(buf)) {
	/*
	* We also always visit user/group/project accounting
	* objects, and never skip them, even if we are
	* suspending. This is necessary so that the
	* space deltas from this txg get integrated.
	*/
	if (OBJSET_BUF_HAS_PROJECTUSED(buf))
	dsl_scan_visitdnode(scn, ds, osp->os_type,
	&osp->os_projectused_dnode,
	DMU_PROJECTUSED_OBJECT, tx);
	dsl_scan_visitdnode(scn, ds, osp->os_type,
	&osp->os_groupused_dnode,
	DMU_GROUPUSED_OBJECT, tx);
	dsl_scan_visitdnode(scn, ds, osp->os_type,
	&osp->os_userused_dnode,
	DMU_USERUSED_OBJECT, tx);
	}
	arc_buf_destroy(buf, &buf);
	}

	return (0);
	}

	inline __attribute__((always_inline)) static void
	dsl_scan_visitdnode(dsl_scan_t scn, dsl_dataset_t ds,
	dmu_objset_type_t ostype, dnode_phys_t *dnp,
	uint64_t object, dmu_tx_t *tx)
	{
	int j;

	for (j = 0; j < dnp->dn_nblkptr; j++) {
	zbookmark_phys_t czb;

	SET_BOOKMARK(&czb, ds ? ds->ds_object : 0, object,
	dnp->dn_nlevels - 1, j);
	dsl_scan_visitbp(&dnp->dn_blkptr[j],
	&czb, dnp, ds, scn, ostype, tx);
	}

	if (dnp->dn_flags & DNODE_FLAG_SPILL_BLKPTR) {
	zbookmark_phys_t czb;
	SET_BOOKMARK(&czb, ds ? ds->ds_object : 0, object,
	0, DMU_SPILL_BLKID);
	dsl_scan_visitbp(DN_SPILL_BLKPTR(dnp),
	&czb, dnp, ds, scn, ostype, tx);
	}
	}

	/*
	* The arguments are in this order because mdb can only print the
	* first 5; we want them to be useful.
	*/
	static void
	dsl_scan_visitbp(blkptr_t bp, const zbookmark_phys_t zb,
	dnode_phys_t dnp, dsl_dataset_t ds, dsl_scan_t *scn,
	dmu_objset_type_t ostype, dmu_tx_t *tx)
	{
	dsl_pool_t *dp = scn->scn_dp;
	blkptr_t *bp_toread = NULL;

	if (dsl_scan_check_suspend(scn, zb))
	return;

	if (dsl_scan_check_resume(scn, dnp, zb))
	return;

	scn->scn_visited_this_txg++;

	/*
	* This debugging is commented out to conserve stack space. This
	* function is called recursively and the debugging adds several
	* bytes to the stack for each call. It can be commented back in
	* if required to debug an issue in dsl_scan_visitbp().
	*
	* dprintf_bp(bp,
	* "visiting ds=%p/%llu zb=%llx/%llx/%llx/%llx bp=%p",
	* ds, ds ? ds->ds_object : 0,
	* zb->zb_objset, zb->zb_object, zb->zb_level, zb->zb_blkid,
	* bp);
	*/

	if (BP_IS_HOLE(bp)) {
	scn->scn_holes_this_txg++;
	return;
	}

	if (BP_IS_REDACTED(bp)) {
	ASSERT(dsl_dataset_feature_is_active(ds,
	SPA_FEATURE_REDACTED_DATASETS));
	return;
	}

	if (bp->blk_birth <= scn->scn_phys.scn_cur_min_txg) {
	scn->scn_lt_min_this_txg++;
	return;
	}

	bp_toread = kmem_alloc(sizeof (blkptr_t), KM_SLEEP);
	bp_toread = bp;

	if (dsl_scan_recurse(scn, ds, ostype, dnp, bp_toread, zb, tx) != 0)
	goto out;

	/*
	* If dsl_scan_ddt() has already visited this block, it will have
	* already done any translations or scrubbing, so don't call the
	* callback again.
	*/
	if (ddt_class_contains(dp->dp_spa,
	scn->scn_phys.scn_ddt_class_max, bp)) {
	scn->scn_ddt_contained_this_txg++;
	goto out;
	}

	/*
	* If this block is from the future (after cur_max_txg), then we
	* are doing this on behalf of a deleted snapshot, and we will
	* revisit the future block on the next pass of this dataset.
	* Don't scan it now unless we need to because something
	* under it was modified.
	*/
	if (BP_PHYSICAL_BIRTH(bp) > scn->scn_phys.scn_cur_max_txg) {
	scn->scn_gt_max_this_txg++;
	goto out;
	}

	scan_funcs[scn->scn_phys.scn_func](dp, bp, zb);

	out:
	kmem_free(bp_toread, sizeof (blkptr_t));
	}

	static void
	dsl_scan_visit_rootbp(dsl_scan_t scn, dsl_dataset_t ds, blkptr_t *bp,
	dmu_tx_t *tx)
	{
	zbookmark_phys_t zb;
	scan_prefetch_ctx_t *spc;

	SET_BOOKMARK(&zb, ds ? ds->ds_object : DMU_META_OBJSET,
	ZB_ROOT_OBJECT, ZB_ROOT_LEVEL, ZB_ROOT_BLKID);

	if (ZB_IS_ZERO(&scn->scn_phys.scn_bookmark)) {
	SET_BOOKMARK(&scn->scn_prefetch_bookmark,
	zb.zb_objset, 0, 0, 0);
	} else {
	scn->scn_prefetch_bookmark = scn->scn_phys.scn_bookmark;
	}

	scn->scn_objsets_visited_this_txg++;

	spc = scan_prefetch_ctx_create(scn, NULL, FTAG);
	dsl_scan_prefetch(spc, bp, &zb);
	scan_prefetch_ctx_rele(spc, FTAG);

	dsl_scan_visitbp(bp, &zb, NULL, ds, scn, DMU_OST_NONE, tx);

	dprintf_ds(ds, "finished scan%s", "");
	}

	static void
	ds_destroyed_scn_phys(dsl_dataset_t ds, dsl_scan_phys_t scn_phys)
	{
	if (scn_phys->scn_bookmark.zb_objset == ds->ds_object) {
	if (ds->ds_is_snapshot) {
	/*
	* Note:
	* - scn_cur_{min,max}_txg stays the same.
	* - Setting the flag is not really necessary if
	* scn_cur_max_txg == scn_max_txg, because there
	* is nothing after this snapshot that we care
	* about. However, we set it anyway and then
	* ignore it when we retraverse it in
	* dsl_scan_visitds().
	*/
	scn_phys->scn_bookmark.zb_objset =
	dsl_dataset_phys(ds)->ds_next_snap_obj;
	zfs_dbgmsg("destroying ds %llu; currently traversing; "
	"reset zb_objset to %llu",
	(u_longlong_t)ds->ds_object,
	(u_longlong_t)dsl_dataset_phys(ds)->
	ds_next_snap_obj);
	scn_phys->scn_flags \|= DSF_VISIT_DS_AGAIN;
	} else {
	SET_BOOKMARK(&scn_phys->scn_bookmark,
	ZB_DESTROYED_OBJSET, 0, 0, 0);
	zfs_dbgmsg("destroying ds %llu; currently traversing; "
	"reset bookmark to -1,0,0,0",
	(u_longlong_t)ds->ds_object);
	}
	}
	}

	/*
	* Invoked when a dataset is destroyed. We need to make sure that:
	*
	* 1) If it is the dataset that was currently being scanned, we write
	* a new dsl_scan_phys_t and marking the objset reference in it
	* as destroyed.
	* 2) Remove it from the work queue, if it was present.
	*
	* If the dataset was actually a snapshot, instead of marking the dataset
	* as destroyed, we instead substitute the next snapshot in line.
	*/
	void
	dsl_scan_ds_destroyed(dsl_dataset_t ds, dmu_tx_t tx)
	{
	dsl_pool_t *dp = ds->ds_dir->dd_pool;
	dsl_scan_t *scn = dp->dp_scan;
	uint64_t mintxg;

	if (!dsl_scan_is_running(scn))
	return;

	ds_destroyed_scn_phys(ds, &scn->scn_phys);
	ds_destroyed_scn_phys(ds, &scn->scn_phys_cached);

	if (scan_ds_queue_contains(scn, ds->ds_object, &mintxg)) {
	scan_ds_queue_remove(scn, ds->ds_object);
	if (ds->ds_is_snapshot)
	scan_ds_queue_insert(scn,
	dsl_dataset_phys(ds)->ds_next_snap_obj, mintxg);
	}

	if (zap_lookup_int_key(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj,
	ds->ds_object, &mintxg) == 0) {
	ASSERT3U(dsl_dataset_phys(ds)->ds_num_children, <=, 1);
	VERIFY3U(0, ==, zap_remove_int(dp->dp_meta_objset,
	scn->scn_phys.scn_queue_obj, ds->ds_object, tx));
	if (ds->ds_is_snapshot) {
	/*
	* We keep the same mintxg; it could be >
	* ds_creation_txg if the previous snapshot was
	* deleted too.
	*/
	VERIFY(zap_add_int_key(dp->dp_meta_objset,
	scn->scn_phys.scn_queue_obj,
	dsl_dataset_phys(ds)->ds_next_snap_obj,
	mintxg, tx) == 0);
	zfs_dbgmsg("destroying ds %llu; in queue; "
	"replacing with %llu",
	(u_longlong_t)ds->ds_object,
	(u_longlong_t)dsl_dataset_phys(ds)->
	ds_next_snap_obj);
	} else {
	zfs_dbgmsg("destroying ds %llu; in queue; removing",
	(u_longlong_t)ds->ds_object);
	}
	}

	/*
	* dsl_scan_sync() should be called after this, and should sync
	* out our changed state, but just to be safe, do it here.
	*/
	dsl_scan_sync_state(scn, tx, SYNC_CACHED);
	}

	static void
	ds_snapshotted_bookmark(dsl_dataset_t ds, zbookmark_phys_t scn_bookmark)
	{
	if (scn_bookmark->zb_objset == ds->ds_object) {
	scn_bookmark->zb_objset =
	dsl_dataset_phys(ds)->ds_prev_snap_obj;
	zfs_dbgmsg("snapshotting ds %llu; currently traversing; "
	"reset zb_objset to %llu",
	(u_longlong_t)ds->ds_object,
	(u_longlong_t)dsl_dataset_phys(ds)->ds_prev_snap_obj);
	}
	}

	/*
	* Called when a dataset is snapshotted. If we were currently traversing
	* this snapshot, we reset our bookmark to point at the newly created
	* snapshot. We also modify our work queue to remove the old snapshot and
	* replace with the new one.
	*/
	void
	dsl_scan_ds_snapshotted(dsl_dataset_t ds, dmu_tx_t tx)
	{
	dsl_pool_t *dp = ds->ds_dir->dd_pool;
	dsl_scan_t *scn = dp->dp_scan;
	uint64_t mintxg;

	if (!dsl_scan_is_running(scn))
	return;

	ASSERT(dsl_dataset_phys(ds)->ds_prev_snap_obj != 0);

	ds_snapshotted_bookmark(ds, &scn->scn_phys.scn_bookmark);
	ds_snapshotted_bookmark(ds, &scn->scn_phys_cached.scn_bookmark);

	if (scan_ds_queue_contains(scn, ds->ds_object, &mintxg)) {
	scan_ds_queue_remove(scn, ds->ds_object);
	scan_ds_queue_insert(scn,
	dsl_dataset_phys(ds)->ds_prev_snap_obj, mintxg);
	}

	if (zap_lookup_int_key(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj,
	ds->ds_object, &mintxg) == 0) {
	VERIFY3U(0, ==, zap_remove_int(dp->dp_meta_objset,
	scn->scn_phys.scn_queue_obj, ds->ds_object, tx));
	VERIFY(zap_add_int_key(dp->dp_meta_objset,
	scn->scn_phys.scn_queue_obj,
	dsl_dataset_phys(ds)->ds_prev_snap_obj, mintxg, tx) == 0);
	zfs_dbgmsg("snapshotting ds %llu; in queue; "
	"replacing with %llu",
	(u_longlong_t)ds->ds_object,
	(u_longlong_t)dsl_dataset_phys(ds)->ds_prev_snap_obj);
	}

	dsl_scan_sync_state(scn, tx, SYNC_CACHED);
	}

	static void
	ds_clone_swapped_bookmark(dsl_dataset_t ds1, dsl_dataset_t ds2,
	zbookmark_phys_t *scn_bookmark)
	{
	if (scn_bookmark->zb_objset == ds1->ds_object) {
	scn_bookmark->zb_objset = ds2->ds_object;
	zfs_dbgmsg("clone_swap ds %llu; currently traversing; "
	"reset zb_objset to %llu",
	(u_longlong_t)ds1->ds_object,
	(u_longlong_t)ds2->ds_object);
	} else if (scn_bookmark->zb_objset == ds2->ds_object) {
	scn_bookmark->zb_objset = ds1->ds_object;
	zfs_dbgmsg("clone_swap ds %llu; currently traversing; "
	"reset zb_objset to %llu",
	(u_longlong_t)ds2->ds_object,
	(u_longlong_t)ds1->ds_object);
	}
	}

	/*
	* Called when an origin dataset and its clone are swapped. If we were
	* currently traversing the dataset, we need to switch to traversing the
	* newly promoted clone.
	*/
	void
	dsl_scan_ds_clone_swapped(dsl_dataset_t ds1, dsl_dataset_t ds2, dmu_tx_t *tx)
	{
	dsl_pool_t *dp = ds1->ds_dir->dd_pool;
	dsl_scan_t *scn = dp->dp_scan;
	uint64_t mintxg1, mintxg2;
	boolean_t ds1_queued, ds2_queued;

	if (!dsl_scan_is_running(scn))
	return;

	ds_clone_swapped_bookmark(ds1, ds2, &scn->scn_phys.scn_bookmark);
	ds_clone_swapped_bookmark(ds1, ds2, &scn->scn_phys_cached.scn_bookmark);

	/*
	* Handle the in-memory scan queue.
	*/
	ds1_queued = scan_ds_queue_contains(scn, ds1->ds_object, &mintxg1);
	ds2_queued = scan_ds_queue_contains(scn, ds2->ds_object, &mintxg2);

	/* Sanity checking. */
	if (ds1_queued) {
	ASSERT3U(mintxg1, ==, dsl_dataset_phys(ds1)->ds_prev_snap_txg);
	ASSERT3U(mintxg1, ==, dsl_dataset_phys(ds2)->ds_prev_snap_txg);
	}
	if (ds2_queued) {
	ASSERT3U(mintxg2, ==, dsl_dataset_phys(ds1)->ds_prev_snap_txg);
	ASSERT3U(mintxg2, ==, dsl_dataset_phys(ds2)->ds_prev_snap_txg);
	}

	if (ds1_queued && ds2_queued) {
	/*
	* If both are queued, we don't need to do anything.
	* The swapping code below would not handle this case correctly,
	* since we can't insert ds2 if it is already there. That's
	* because scan_ds_queue_insert() prohibits a duplicate insert
	* and panics.
	*/
	} else if (ds1_queued) {
	scan_ds_queue_remove(scn, ds1->ds_object);
	scan_ds_queue_insert(scn, ds2->ds_object, mintxg1);
	} else if (ds2_queued) {
	scan_ds_queue_remove(scn, ds2->ds_object);
	scan_ds_queue_insert(scn, ds1->ds_object, mintxg2);
	}

	/*
	* Handle the on-disk scan queue.
	* The on-disk state is an out-of-date version of the in-memory state,
	* so the in-memory and on-disk values for ds1_queued and ds2_queued may
	* be different. Therefore we need to apply the swap logic to the
	* on-disk state independently of the in-memory state.
	*/
	ds1_queued = zap_lookup_int_key(dp->dp_meta_objset,
	scn->scn_phys.scn_queue_obj, ds1->ds_object, &mintxg1) == 0;
	ds2_queued = zap_lookup_int_key(dp->dp_meta_objset,
	scn->scn_phys.scn_queue_obj, ds2->ds_object, &mintxg2) == 0;

	/* Sanity checking. */
	if (ds1_queued) {
	ASSERT3U(mintxg1, ==, dsl_dataset_phys(ds1)->ds_prev_snap_txg);
	ASSERT3U(mintxg1, ==, dsl_dataset_phys(ds2)->ds_prev_snap_txg);
	}
	if (ds2_queued) {
	ASSERT3U(mintxg2, ==, dsl_dataset_phys(ds1)->ds_prev_snap_txg);
	ASSERT3U(mintxg2, ==, dsl_dataset_phys(ds2)->ds_prev_snap_txg);
	}

	if (ds1_queued && ds2_queued) {
	/*
	* If both are queued, we don't need to do anything.
	* Alternatively, we could check for EEXIST from
	* zap_add_int_key() and back out to the original state, but
	* that would be more work than checking for this case upfront.
	*/
	} else if (ds1_queued) {
	VERIFY3S(0, ==, zap_remove_int(dp->dp_meta_objset,
	scn->scn_phys.scn_queue_obj, ds1->ds_object, tx));
	VERIFY3S(0, ==, zap_add_int_key(dp->dp_meta_objset,
	scn->scn_phys.scn_queue_obj, ds2->ds_object, mintxg1, tx));
	zfs_dbgmsg("clone_swap ds %llu; in queue; "
	"replacing with %llu",
	(u_longlong_t)ds1->ds_object,
	(u_longlong_t)ds2->ds_object);
	} else if (ds2_queued) {
	VERIFY3S(0, ==, zap_remove_int(dp->dp_meta_objset,
	scn->scn_phys.scn_queue_obj, ds2->ds_object, tx));
	VERIFY3S(0, ==, zap_add_int_key(dp->dp_meta_objset,
	scn->scn_phys.scn_queue_obj, ds1->ds_object, mintxg2, tx));
	zfs_dbgmsg("clone_swap ds %llu; in queue; "
	"replacing with %llu",
	(u_longlong_t)ds2->ds_object,
	(u_longlong_t)ds1->ds_object);
	}

	dsl_scan_sync_state(scn, tx, SYNC_CACHED);
	}

	/* ARGSUSED */
	static int
	enqueue_clones_cb(dsl_pool_t dp, dsl_dataset_t hds, void *arg)
	{
	uint64_t originobj = (uint64_t )arg;
	dsl_dataset_t *ds;
	int err;
	dsl_scan_t *scn = dp->dp_scan;

	if (dsl_dir_phys(hds->ds_dir)->dd_origin_obj != originobj)
	return (0);

	err = dsl_dataset_hold_obj(dp, hds->ds_object, FTAG, &ds);
	if (err)
	return (err);

	while (dsl_dataset_phys(ds)->ds_prev_snap_obj != originobj) {
	dsl_dataset_t *prev;
	err = dsl_dataset_hold_obj(dp,
	dsl_dataset_phys(ds)->ds_prev_snap_obj, FTAG, &prev);

	dsl_dataset_rele(ds, FTAG);
	if (err)
	return (err);
	ds = prev;
	}
	scan_ds_queue_insert(scn, ds->ds_object,
	dsl_dataset_phys(ds)->ds_prev_snap_txg);
	dsl_dataset_rele(ds, FTAG);
	return (0);
	}

	static void
	dsl_scan_visitds(dsl_scan_t scn, uint64_t dsobj, dmu_tx_t tx)
	{
	dsl_pool_t *dp = scn->scn_dp;
	dsl_dataset_t *ds;

	VERIFY3U(0, ==, dsl_dataset_hold_obj(dp, dsobj, FTAG, &ds));

	if (scn->scn_phys.scn_cur_min_txg >=
	scn->scn_phys.scn_max_txg) {
	/*
	* This can happen if this snapshot was created after the
	* scan started, and we already completed a previous snapshot
	* that was created after the scan started. This snapshot
	* only references blocks with:
	*
	* birth < our ds_creation_txg
	* cur_min_txg is no less than ds_creation_txg.
	* We have already visited these blocks.
	* or
	* birth > scn_max_txg
	* The scan requested not to visit these blocks.
	*
	* Subsequent snapshots (and clones) can reference our
	* blocks, or blocks with even higher birth times.
	* Therefore we do not need to visit them either,
	* so we do not add them to the work queue.
	*
	* Note that checking for cur_min_txg >= cur_max_txg
	* is not sufficient, because in that case we may need to
	* visit subsequent snapshots. This happens when min_txg > 0,
	* which raises cur_min_txg. In this case we will visit
	* this dataset but skip all of its blocks, because the
	* rootbp's birth time is < cur_min_txg. Then we will
	* add the next snapshots/clones to the work queue.
	*/
	char *dsname = kmem_alloc(ZFS_MAX_DATASET_NAME_LEN, KM_SLEEP);
	dsl_dataset_name(ds, dsname);
	zfs_dbgmsg("scanning dataset %llu (%s) is unnecessary because "
	"cur_min_txg (%llu) >= max_txg (%llu)",
	(longlong_t)dsobj, dsname,
	(longlong_t)scn->scn_phys.scn_cur_min_txg,
	(longlong_t)scn->scn_phys.scn_max_txg);
	kmem_free(dsname, MAXNAMELEN);

	goto out;
	}

	/*
	* Only the ZIL in the head (non-snapshot) is valid. Even though
	* snapshots can have ZIL block pointers (which may be the same
	* BP as in the head), they must be ignored. In addition, $ORIGIN
	* doesn't have a objset (i.e. its ds_bp is a hole) so we don't
	* need to look for a ZIL in it either. So we traverse the ZIL here,
	* rather than in scan_recurse(), because the regular snapshot
	* block-sharing rules don't apply to it.
	*/
	if (!dsl_dataset_is_snapshot(ds) &&
	(dp->dp_origin_snap == NULL \|\|
	ds->ds_dir != dp->dp_origin_snap->ds_dir)) {
	objset_t *os;
	if (dmu_objset_from_ds(ds, &os) != 0) {
	goto out;
	}
	dsl_scan_zil(dp, &os->os_zil_header);
	}

	/*
	* Iterate over the bps in this ds.
	*/
	dmu_buf_will_dirty(ds->ds_dbuf, tx);
	rrw_enter(&ds->ds_bp_rwlock, RW_READER, FTAG);
	dsl_scan_visit_rootbp(scn, ds, &dsl_dataset_phys(ds)->ds_bp, tx);
	rrw_exit(&ds->ds_bp_rwlock, FTAG);

	char *dsname = kmem_alloc(ZFS_MAX_DATASET_NAME_LEN, KM_SLEEP);
	dsl_dataset_name(ds, dsname);
	zfs_dbgmsg("scanned dataset %llu (%s) with min=%llu max=%llu; "
	"suspending=%u",
	(longlong_t)dsobj, dsname,
	(longlong_t)scn->scn_phys.scn_cur_min_txg,
	(longlong_t)scn->scn_phys.scn_cur_max_txg,
	(int)scn->scn_suspending);
	kmem_free(dsname, ZFS_MAX_DATASET_NAME_LEN);

	if (scn->scn_suspending)
	goto out;

	/*
	* We've finished this pass over this dataset.
	*/

	/*
	* If we did not completely visit this dataset, do another pass.
	*/
	if (scn->scn_phys.scn_flags & DSF_VISIT_DS_AGAIN) {
	zfs_dbgmsg("incomplete pass; visiting again");
	scn->scn_phys.scn_flags &= ~DSF_VISIT_DS_AGAIN;
	scan_ds_queue_insert(scn, ds->ds_object,
	scn->scn_phys.scn_cur_max_txg);
	goto out;
	}

	/*
	* Add descendant datasets to work queue.
	*/
	if (dsl_dataset_phys(ds)->ds_next_snap_obj != 0) {
	scan_ds_queue_insert(scn,
	dsl_dataset_phys(ds)->ds_next_snap_obj,
	dsl_dataset_phys(ds)->ds_creation_txg);
	}
	if (dsl_dataset_phys(ds)->ds_num_children > 1) {
	boolean_t usenext = B_FALSE;
	if (dsl_dataset_phys(ds)->ds_next_clones_obj != 0) {
	uint64_t count;
	/*
	* A bug in a previous version of the code could
	* cause upgrade_clones_cb() to not set
	* ds_next_snap_obj when it should, leading to a
	* missing entry. Therefore we can only use the
	* next_clones_obj when its count is correct.
	*/
	int err = zap_count(dp->dp_meta_objset,
	dsl_dataset_phys(ds)->ds_next_clones_obj, &count);
	if (err == 0 &&
	count == dsl_dataset_phys(ds)->ds_num_children - 1)
	usenext = B_TRUE;
	}

	if (usenext) {
	zap_cursor_t zc;
	zap_attribute_t za;
	for (zap_cursor_init(&zc, dp->dp_meta_objset,
	dsl_dataset_phys(ds)->ds_next_clones_obj);
	zap_cursor_retrieve(&zc, &za) == 0;
	(void) zap_cursor_advance(&zc)) {
	scan_ds_queue_insert(scn,
	zfs_strtonum(za.za_name, NULL),
	dsl_dataset_phys(ds)->ds_creation_txg);
	}
	zap_cursor_fini(&zc);
	} else {
	VERIFY0(dmu_objset_find_dp(dp, dp->dp_root_dir_obj,
	enqueue_clones_cb, &ds->ds_object,
	DS_FIND_CHILDREN));
	}
	}

	out:
	dsl_dataset_rele(ds, FTAG);
	}

	/* ARGSUSED */
	static int
	enqueue_cb(dsl_pool_t dp, dsl_dataset_t hds, void *arg)
	{
	dsl_dataset_t *ds;
	int err;
	dsl_scan_t *scn = dp->dp_scan;

	err = dsl_dataset_hold_obj(dp, hds->ds_object, FTAG, &ds);
	if (err)
	return (err);

	while (dsl_dataset_phys(ds)->ds_prev_snap_obj != 0) {
	dsl_dataset_t *prev;
	err = dsl_dataset_hold_obj(dp,
	dsl_dataset_phys(ds)->ds_prev_snap_obj, FTAG, &prev);
	if (err) {
	dsl_dataset_rele(ds, FTAG);
	return (err);
	}

	/*
	* If this is a clone, we don't need to worry about it for now.
	*/
	if (dsl_dataset_phys(prev)->ds_next_snap_obj != ds->ds_object) {
	dsl_dataset_rele(ds, FTAG);
	dsl_dataset_rele(prev, FTAG);
	return (0);
	}
	dsl_dataset_rele(ds, FTAG);
	ds = prev;
	}

	scan_ds_queue_insert(scn, ds->ds_object,
	dsl_dataset_phys(ds)->ds_prev_snap_txg);
	dsl_dataset_rele(ds, FTAG);
	return (0);
	}

	/* ARGSUSED */
	void
	dsl_scan_ddt_entry(dsl_scan_t *scn, enum zio_checksum checksum,
	ddt_entry_t dde, dmu_tx_t tx)
	{
	const ddt_key_t *ddk = &dde->dde_key;
	ddt_phys_t *ddp = dde->dde_phys;
	blkptr_t bp;
	zbookmark_phys_t zb = { 0 };
	int p;

	if (!dsl_scan_is_running(scn))
	return;

	/*
	* This function is special because it is the only thing
	* that can add scan_io_t's to the vdev scan queues from
	* outside dsl_scan_sync(). For the most part this is ok
	* as long as it is called from within syncing context.
	* However, dsl_scan_sync() expects that no new sio's will
	* be added between when all the work for a scan is done
	* and the next txg when the scan is actually marked as
	* completed. This check ensures we do not issue new sio's
	* during this period.
	*/
	if (scn->scn_done_txg != 0)
	return;

	for (p = 0; p < DDT_PHYS_TYPES; p++, ddp++) {
	if (ddp->ddp_phys_birth == 0 \|\|
	ddp->ddp_phys_birth > scn->scn_phys.scn_max_txg)
	continue;
	ddt_bp_create(checksum, ddk, ddp, &bp);

	scn->scn_visited_this_txg++;
	scan_funcs[scn->scn_phys.scn_func](scn->scn_dp, &bp, &zb);
	}
	}

	/*
	* Scrub/dedup interaction.
	*
	* If there are N references to a deduped block, we don't want to scrub it
	* N times -- ideally, we should scrub it exactly once.
	*
	* We leverage the fact that the dde's replication class (enum ddt_class)
	* is ordered from highest replication class (DDT_CLASS_DITTO) to lowest
	* (DDT_CLASS_UNIQUE) so that we may walk the DDT in that order.
	*
	* To prevent excess scrubbing, the scrub begins by walking the DDT
	* to find all blocks with refcnt > 1, and scrubs each of these once.
	* Since there are two replication classes which contain blocks with
	* refcnt > 1, we scrub the highest replication class (DDT_CLASS_DITTO) first.
	* Finally the top-down scrub begins, only visiting blocks with refcnt == 1.
	*
	* There would be nothing more to say if a block's refcnt couldn't change
	* during a scrub, but of course it can so we must account for changes
	* in a block's replication class.
	*
	* Here's an example of what can occur:
	*
	* If a block has refcnt > 1 during the DDT scrub phase, but has refcnt == 1
	* when visited during the top-down scrub phase, it will be scrubbed twice.
	* This negates our scrub optimization, but is otherwise harmless.
	*
	* If a block has refcnt == 1 during the DDT scrub phase, but has refcnt > 1
	* on each visit during the top-down scrub phase, it will never be scrubbed.
	* To catch this, ddt_sync_entry() notifies the scrub code whenever a block's
	* reference class transitions to a higher level (i.e DDT_CLASS_UNIQUE to
	* DDT_CLASS_DUPLICATE); if it transitions from refcnt == 1 to refcnt > 1
	* while a scrub is in progress, it scrubs the block right then.
	*/
	static void
	dsl_scan_ddt(dsl_scan_t scn, dmu_tx_t tx)
	{
	ddt_bookmark_t *ddb = &scn->scn_phys.scn_ddt_bookmark;
	ddt_entry_t dde;
	int error;
	uint64_t n = 0;

	bzero(&dde, sizeof (ddt_entry_t));

	while ((error = ddt_walk(scn->scn_dp->dp_spa, ddb, &dde)) == 0) {
	ddt_t *ddt;

	if (ddb->ddb_class > scn->scn_phys.scn_ddt_class_max)
	break;
	dprintf("visiting ddb=%llu/%llu/%llu/%llx\n",
	(longlong_t)ddb->ddb_class,
	(longlong_t)ddb->ddb_type,
	(longlong_t)ddb->ddb_checksum,
	(longlong_t)ddb->ddb_cursor);

	/* There should be no pending changes to the dedup table */
	ddt = scn->scn_dp->dp_spa->spa_ddt[ddb->ddb_checksum];
	ASSERT(avl_first(&ddt->ddt_tree) == NULL);

	dsl_scan_ddt_entry(scn, ddb->ddb_checksum, &dde, tx);
	n++;

	if (dsl_scan_check_suspend(scn, NULL))
	break;
	}

	zfs_dbgmsg("scanned %llu ddt entries with class_max = %u; "
	"suspending=%u", (longlong_t)n,
	(int)scn->scn_phys.scn_ddt_class_max, (int)scn->scn_suspending);

	ASSERT(error == 0 \|\| error == ENOENT);
	ASSERT(error != ENOENT \|\|
	ddb->ddb_class > scn->scn_phys.scn_ddt_class_max);
	}

	static uint64_t
	dsl_scan_ds_maxtxg(dsl_dataset_t *ds)
	{
	uint64_t smt = ds->ds_dir->dd_pool->dp_scan->scn_phys.scn_max_txg;
	if (ds->ds_is_snapshot)
	return (MIN(smt, dsl_dataset_phys(ds)->ds_creation_txg));
	return (smt);
	}

	static void
	dsl_scan_visit(dsl_scan_t scn, dmu_tx_t tx)
	{
	scan_ds_t *sds;
	dsl_pool_t *dp = scn->scn_dp;

	if (scn->scn_phys.scn_ddt_bookmark.ddb_class <=
	scn->scn_phys.scn_ddt_class_max) {
	scn->scn_phys.scn_cur_min_txg = scn->scn_phys.scn_min_txg;
	scn->scn_phys.scn_cur_max_txg = scn->scn_phys.scn_max_txg;
	dsl_scan_ddt(scn, tx);
	if (scn->scn_suspending)
	return;
	}

	if (scn->scn_phys.scn_bookmark.zb_objset == DMU_META_OBJSET) {
	/* First do the MOS & ORIGIN */

	scn->scn_phys.scn_cur_min_txg = scn->scn_phys.scn_min_txg;
	scn->scn_phys.scn_cur_max_txg = scn->scn_phys.scn_max_txg;
	dsl_scan_visit_rootbp(scn, NULL,
	&dp->dp_meta_rootbp, tx);
	spa_set_rootblkptr(dp->dp_spa, &dp->dp_meta_rootbp);
	if (scn->scn_suspending)
	return;

	if (spa_version(dp->dp_spa) < SPA_VERSION_DSL_SCRUB) {
	VERIFY0(dmu_objset_find_dp(dp, dp->dp_root_dir_obj,
	enqueue_cb, NULL, DS_FIND_CHILDREN));
	} else {
	dsl_scan_visitds(scn,
	dp->dp_origin_snap->ds_object, tx);
	}
	ASSERT(!scn->scn_suspending);
	} else if (scn->scn_phys.scn_bookmark.zb_objset !=
	ZB_DESTROYED_OBJSET) {
	uint64_t dsobj = scn->scn_phys.scn_bookmark.zb_objset;
	/*
	* If we were suspended, continue from here. Note if the
	* ds we were suspended on was deleted, the zb_objset may
	* be -1, so we will skip this and find a new objset
	* below.
	*/
	dsl_scan_visitds(scn, dsobj, tx);
	if (scn->scn_suspending)
	return;
	}

	/*
	* In case we suspended right at the end of the ds, zero the
	* bookmark so we don't think that we're still trying to resume.
	*/
	bzero(&scn->scn_phys.scn_bookmark, sizeof (zbookmark_phys_t));

	/*
	* Keep pulling things out of the dataset avl queue. Updates to the
	* persistent zap-object-as-queue happen only at checkpoints.
	*/
	while ((sds = avl_first(&scn->scn_queue)) != NULL) {
	dsl_dataset_t *ds;
	uint64_t dsobj = sds->sds_dsobj;
	uint64_t txg = sds->sds_txg;

	/* dequeue and free the ds from the queue */
	scan_ds_queue_remove(scn, dsobj);
	sds = NULL;

	/* set up min / max txg */
	VERIFY3U(0, ==, dsl_dataset_hold_obj(dp, dsobj, FTAG, &ds));
	if (txg != 0) {
	scn->scn_phys.scn_cur_min_txg =
	MAX(scn->scn_phys.scn_min_txg, txg);
	} else {
	scn->scn_phys.scn_cur_min_txg =
	MAX(scn->scn_phys.scn_min_txg,
	dsl_dataset_phys(ds)->ds_prev_snap_txg);
	}
	scn->scn_phys.scn_cur_max_txg = dsl_scan_ds_maxtxg(ds);
	dsl_dataset_rele(ds, FTAG);

	dsl_scan_visitds(scn, dsobj, tx);
	if (scn->scn_suspending)
	return;
	}

	/* No more objsets to fetch, we're done */
	scn->scn_phys.scn_bookmark.zb_objset = ZB_DESTROYED_OBJSET;
	ASSERT0(scn->scn_suspending);
	}

	static uint64_t
	dsl_scan_count_leaves(vdev_t *vd)
	{
	uint64_t i, leaves = 0;

	/* we only count leaves that belong to the main pool and are readable */
	if (vd->vdev_islog \|\| vd->vdev_isspare \|\|
	vd->vdev_isl2cache \|\| !vdev_readable(vd))
	return (0);

	if (vd->vdev_ops->vdev_op_leaf)
	return (1);

	for (i = 0; i < vd->vdev_children; i++) {
	leaves += dsl_scan_count_leaves(vd->vdev_child[i]);
	}

	return (leaves);
	}

	static void
	scan_io_queues_update_zio_stats(dsl_scan_io_queue_t q, const blkptr_t bp)
	{
	int i;
	uint64_t cur_size = 0;

	for (i = 0; i < BP_GET_NDVAS(bp); i++) {
	cur_size += DVA_GET_ASIZE(&bp->blk_dva[i]);
	}

	q->q_total_zio_size_this_txg += cur_size;
	q->q_zios_this_txg++;
	}

	static void
	scan_io_queues_update_seg_stats(dsl_scan_io_queue_t *q, uint64_t start,
	uint64_t end)
	{
	q->q_total_seg_size_this_txg += end - start;
	q->q_segs_this_txg++;
	}

	static boolean_t
	scan_io_queue_check_suspend(dsl_scan_t *scn)
	{
	/* See comment in dsl_scan_check_suspend() */
	uint64_t curr_time_ns = gethrtime();
	uint64_t scan_time_ns = curr_time_ns - scn->scn_sync_start_time;
	uint64_t sync_time_ns = curr_time_ns -
	scn->scn_dp->dp_spa->spa_sync_starttime;
	int dirty_pct = scn->scn_dp->dp_dirty_total * 100 / zfs_dirty_data_max;
	int mintime = (scn->scn_phys.scn_func == POOL_SCAN_RESILVER) ?
	zfs_resilver_min_time_ms : zfs_scrub_min_time_ms;

	return ((NSEC2MSEC(scan_time_ns) > mintime &&
	(dirty_pct >= zfs_vdev_async_write_active_min_dirty_percent \|\|
	txg_sync_waiting(scn->scn_dp) \|\|
	NSEC2SEC(sync_time_ns) >= zfs_txg_timeout)) \|\|
	spa_shutting_down(scn->scn_dp->dp_spa));
	}

	/*
	* Given a list of scan_io_t's in io_list, this issues the I/Os out to
	* disk. This consumes the io_list and frees the scan_io_t's. This is
	* called when emptying queues, either when we're up against the memory
	* limit or when we have finished scanning. Returns B_TRUE if we stopped
	* processing the list before we finished. Any sios that were not issued
	* will remain in the io_list.
	*/
	static boolean_t
	scan_io_queue_issue(dsl_scan_io_queue_t queue, list_t io_list)
	{
	dsl_scan_t *scn = queue->q_scn;
	scan_io_t *sio;
	int64_t bytes_issued = 0;
	boolean_t suspended = B_FALSE;

	while ((sio = list_head(io_list)) != NULL) {
	blkptr_t bp;

	if (scan_io_queue_check_suspend(scn)) {
	suspended = B_TRUE;
	break;
	}

	sio2bp(sio, &bp);
	bytes_issued += SIO_GET_ASIZE(sio);
	scan_exec_io(scn->scn_dp, &bp, sio->sio_flags,
	&sio->sio_zb, queue);
	(void) list_remove_head(io_list);
	scan_io_queues_update_zio_stats(queue, &bp);
	sio_free(sio);
	}

	atomic_add_64(&scn->scn_bytes_pending, -bytes_issued);

	return (suspended);
	}

	/*
	* This function removes sios from an IO queue which reside within a given
	* range_seg_t and inserts them (in offset order) into a list. Note that
	* we only ever return a maximum of 32 sios at once. If there are more sios
	* to process within this segment that did not make it onto the list we
	* return B_TRUE and otherwise B_FALSE.
	*/
	static boolean_t
	scan_io_queue_gather(dsl_scan_io_queue_t queue, range_seg_t rs, list_t *list)
	{
	scan_io_t srch_sio, sio, *next_sio;
	avl_index_t idx;
	uint_t num_sios = 0;
	int64_t bytes_issued = 0;

	ASSERT(rs != NULL);
	ASSERT(MUTEX_HELD(&queue->q_vd->vdev_scan_io_queue_lock));

	srch_sio = sio_alloc(1);
	srch_sio->sio_nr_dvas = 1;
	SIO_SET_OFFSET(srch_sio, rs_get_start(rs, queue->q_exts_by_addr));

	/*
	* The exact start of the extent might not contain any matching zios,
	* so if that's the case, examine the next one in the tree.
	*/
	sio = avl_find(&queue->q_sios_by_addr, srch_sio, &idx);
	sio_free(srch_sio);

	if (sio == NULL)
	sio = avl_nearest(&queue->q_sios_by_addr, idx, AVL_AFTER);

	while (sio != NULL && SIO_GET_OFFSET(sio) < rs_get_end(rs,
	queue->q_exts_by_addr) && num_sios <= 32) {
	ASSERT3U(SIO_GET_OFFSET(sio), >=, rs_get_start(rs,
	queue->q_exts_by_addr));
	ASSERT3U(SIO_GET_END_OFFSET(sio), <=, rs_get_end(rs,
	queue->q_exts_by_addr));

	next_sio = AVL_NEXT(&queue->q_sios_by_addr, sio);
	avl_remove(&queue->q_sios_by_addr, sio);
	queue->q_sio_memused -= SIO_GET_MUSED(sio);

	bytes_issued += SIO_GET_ASIZE(sio);
	num_sios++;
	list_insert_tail(list, sio);
	sio = next_sio;
	}

	/*
	* We limit the number of sios we process at once to 32 to avoid
	* biting off more than we can chew. If we didn't take everything
	* in the segment we update it to reflect the work we were able to
	* complete. Otherwise, we remove it from the range tree entirely.
	*/
	if (sio != NULL && SIO_GET_OFFSET(sio) < rs_get_end(rs,
	queue->q_exts_by_addr)) {
	range_tree_adjust_fill(queue->q_exts_by_addr, rs,
	-bytes_issued);
	range_tree_resize_segment(queue->q_exts_by_addr, rs,
	SIO_GET_OFFSET(sio), rs_get_end(rs,
	queue->q_exts_by_addr) - SIO_GET_OFFSET(sio));

	return (B_TRUE);
	} else {
	uint64_t rstart = rs_get_start(rs, queue->q_exts_by_addr);
	uint64_t rend = rs_get_end(rs, queue->q_exts_by_addr);
	range_tree_remove(queue->q_exts_by_addr, rstart, rend - rstart);
	return (B_FALSE);
	}
	}

	/*
	* This is called from the queue emptying thread and selects the next
	* extent from which we are to issue I/Os. The behavior of this function
	* depends on the state of the scan, the current memory consumption and
	* whether or not we are performing a scan shutdown.
	* 1) We select extents in an elevator algorithm (LBA-order) if the scan
	* needs to perform a checkpoint
	* 2) We select the largest available extent if we are up against the
	* memory limit.
	* 3) Otherwise we don't select any extents.
	*/
	static range_seg_t *
	scan_io_queue_fetch_ext(dsl_scan_io_queue_t *queue)
	{
	dsl_scan_t *scn = queue->q_scn;
	range_tree_t *rt = queue->q_exts_by_addr;

	ASSERT(MUTEX_HELD(&queue->q_vd->vdev_scan_io_queue_lock));
	ASSERT(scn->scn_is_sorted);

	/* handle tunable overrides */
	if (scn->scn_checkpointing \|\| scn->scn_clearing) {
	if (zfs_scan_issue_strategy == 1) {
	return (range_tree_first(rt));
	} else if (zfs_scan_issue_strategy == 2) {
	/*
	* We need to get the original entry in the by_addr
	* tree so we can modify it.
	*/
	range_seg_t *size_rs =
	zfs_btree_first(&queue->q_exts_by_size, NULL);
	if (size_rs == NULL)
	return (NULL);
	uint64_t start = rs_get_start(size_rs, rt);
	uint64_t size = rs_get_end(size_rs, rt) - start;
	range_seg_t *addr_rs = range_tree_find(rt, start,
	size);
	ASSERT3P(addr_rs, !=, NULL);
	ASSERT3U(rs_get_start(size_rs, rt), ==,
	rs_get_start(addr_rs, rt));
	ASSERT3U(rs_get_end(size_rs, rt), ==,
	rs_get_end(addr_rs, rt));
	return (addr_rs);
	}
	}

	/*
	* During normal clearing, we want to issue our largest segments
	* first, keeping IO as sequential as possible, and leaving the
	* smaller extents for later with the hope that they might eventually
	* grow to larger sequential segments. However, when the scan is
	* checkpointing, no new extents will be added to the sorting queue,
	* so the way we are sorted now is as good as it will ever get.
	* In this case, we instead switch to issuing extents in LBA order.
	*/
	if (scn->scn_checkpointing) {
	return (range_tree_first(rt));
	} else if (scn->scn_clearing) {
	/*
	* We need to get the original entry in the by_addr
	* tree so we can modify it.
	*/
	range_seg_t *size_rs = zfs_btree_first(&queue->q_exts_by_size,
	NULL);
	if (size_rs == NULL)
	return (NULL);
	uint64_t start = rs_get_start(size_rs, rt);
	uint64_t size = rs_get_end(size_rs, rt) - start;
	range_seg_t *addr_rs = range_tree_find(rt, start, size);
	ASSERT3P(addr_rs, !=, NULL);
	ASSERT3U(rs_get_start(size_rs, rt), ==, rs_get_start(addr_rs,
	rt));
	ASSERT3U(rs_get_end(size_rs, rt), ==, rs_get_end(addr_rs, rt));
	return (addr_rs);
	} else {
	return (NULL);
	}
	}

	static void
	scan_io_queues_run_one(void *arg)
	{
	dsl_scan_io_queue_t *queue = arg;
	kmutex_t *q_lock = &queue->q_vd->vdev_scan_io_queue_lock;
	boolean_t suspended = B_FALSE;
	range_seg_t *rs = NULL;
	scan_io_t *sio = NULL;
	list_t sio_list;
	uint64_t bytes_per_leaf = zfs_scan_vdev_limit;
	uint64_t nr_leaves = dsl_scan_count_leaves(queue->q_vd);

	ASSERT(queue->q_scn->scn_is_sorted);

	list_create(&sio_list, sizeof (scan_io_t),
	offsetof(scan_io_t, sio_nodes.sio_list_node));
	mutex_enter(q_lock);

	/* calculate maximum in-flight bytes for this txg (min 1MB) */
	queue->q_maxinflight_bytes =
	MAX(nr_leaves * bytes_per_leaf, 1ULL << 20);

	/* reset per-queue scan statistics for this txg */
	queue->q_total_seg_size_this_txg = 0;
	queue->q_segs_this_txg = 0;
	queue->q_total_zio_size_this_txg = 0;
	queue->q_zios_this_txg = 0;

	/* loop until we run out of time or sios */
	while ((rs = scan_io_queue_fetch_ext(queue)) != NULL) {
	uint64_t seg_start = 0, seg_end = 0;
	boolean_t more_left = B_TRUE;

	ASSERT(list_is_empty(&sio_list));

	/* loop while we still have sios left to process in this rs */
	while (more_left) {
	scan_io_t first_sio, last_sio;

	/*
	* We have selected which extent needs to be
	* processed next. Gather up the corresponding sios.
	*/
	more_left = scan_io_queue_gather(queue, rs, &sio_list);
	ASSERT(!list_is_empty(&sio_list));
	first_sio = list_head(&sio_list);
	last_sio = list_tail(&sio_list);

	seg_end = SIO_GET_END_OFFSET(last_sio);
	if (seg_start == 0)
	seg_start = SIO_GET_OFFSET(first_sio);

	/*
	* Issuing sios can take a long time so drop the
	* queue lock. The sio queue won't be updated by
	* other threads since we're in syncing context so
	* we can be sure that our trees will remain exactly
	* as we left them.
	*/
	mutex_exit(q_lock);
	suspended = scan_io_queue_issue(queue, &sio_list);
	mutex_enter(q_lock);

	if (suspended)
	break;
	}

	/* update statistics for debugging purposes */
	scan_io_queues_update_seg_stats(queue, seg_start, seg_end);

	if (suspended)
	break;
	}

	/*
	* If we were suspended in the middle of processing,
	* requeue any unfinished sios and exit.
	*/
	while ((sio = list_head(&sio_list)) != NULL) {
	list_remove(&sio_list, sio);
	scan_io_queue_insert_impl(queue, sio);
	}

	mutex_exit(q_lock);
	list_destroy(&sio_list);
	}

	/*
	* Performs an emptying run on all scan queues in the pool. This just
	* punches out one thread per top-level vdev, each of which processes
	* only that vdev's scan queue. We can parallelize the I/O here because
	* we know that each queue's I/Os only affect its own top-level vdev.
	*
	* This function waits for the queue runs to complete, and must be
	* called from dsl_scan_sync (or in general, syncing context).
	*/
	static void
	scan_io_queues_run(dsl_scan_t *scn)
	{
	spa_t *spa = scn->scn_dp->dp_spa;

	ASSERT(scn->scn_is_sorted);
	ASSERT(spa_config_held(spa, SCL_CONFIG, RW_READER));

	if (scn->scn_bytes_pending == 0)
	return;

	if (scn->scn_taskq == NULL) {
	int nthreads = spa->spa_root_vdev->vdev_children;

	/*
	* We need to make this taskq always execute as many
	* threads in parallel as we have top-level vdevs and no
	* less, otherwise strange serialization of the calls to
	* scan_io_queues_run_one can occur during spa_sync runs
	* and that significantly impacts performance.
	*/
	scn->scn_taskq = taskq_create("dsl_scan_iss", nthreads,
	minclsyspri, nthreads, nthreads, TASKQ_PREPOPULATE);
	}

	for (uint64_t i = 0; i < spa->spa_root_vdev->vdev_children; i++) {
	vdev_t *vd = spa->spa_root_vdev->vdev_child[i];

	mutex_enter(&vd->vdev_scan_io_queue_lock);
	if (vd->vdev_scan_io_queue != NULL) {
	VERIFY(taskq_dispatch(scn->scn_taskq,
	scan_io_queues_run_one, vd->vdev_scan_io_queue,
	TQ_SLEEP) != TASKQID_INVALID);
	}
	mutex_exit(&vd->vdev_scan_io_queue_lock);
	}

	/*
	* Wait for the queues to finish issuing their IOs for this run
	* before we return. There may still be IOs in flight at this
	* point.
	*/
	taskq_wait(scn->scn_taskq);
	}

	static boolean_t
	dsl_scan_async_block_should_pause(dsl_scan_t *scn)
	{
	uint64_t elapsed_nanosecs;

	if (zfs_recover)
	return (B_FALSE);

	if (zfs_async_block_max_blocks != 0 &&
	scn->scn_visited_this_txg >= zfs_async_block_max_blocks) {
	return (B_TRUE);
	}

	if (zfs_max_async_dedup_frees != 0 &&
	scn->scn_dedup_frees_this_txg >= zfs_max_async_dedup_frees) {
	return (B_TRUE);
	}

	elapsed_nanosecs = gethrtime() - scn->scn_sync_start_time;
	return (elapsed_nanosecs / NANOSEC > zfs_txg_timeout \|\|
	(NSEC2MSEC(elapsed_nanosecs) > scn->scn_async_block_min_time_ms &&
	txg_sync_waiting(scn->scn_dp)) \|\|
	spa_shutting_down(scn->scn_dp->dp_spa));
	}

	static int
	dsl_scan_free_block_cb(void arg, const blkptr_t bp, dmu_tx_t *tx)
	{
	dsl_scan_t *scn = arg;

	if (!scn->scn_is_bptree \|\|
	(BP_GET_LEVEL(bp) == 0 && BP_GET_TYPE(bp) != DMU_OT_OBJSET)) {
	if (dsl_scan_async_block_should_pause(scn))
	return (SET_ERROR(ERESTART));
	}

	zio_nowait(zio_free_sync(scn->scn_zio_root, scn->scn_dp->dp_spa,
	dmu_tx_get_txg(tx), bp, 0));
	dsl_dir_diduse_space(tx->tx_pool->dp_free_dir, DD_USED_HEAD,
	-bp_get_dsize_sync(scn->scn_dp->dp_spa, bp),
	-BP_GET_PSIZE(bp), -BP_GET_UCSIZE(bp), tx);
	scn->scn_visited_this_txg++;
	if (BP_GET_DEDUP(bp))
	scn->scn_dedup_frees_this_txg++;
	return (0);
	}

	static void
	dsl_scan_update_stats(dsl_scan_t *scn)
	{
	spa_t *spa = scn->scn_dp->dp_spa;
	uint64_t i;
	uint64_t seg_size_total = 0, zio_size_total = 0;
	uint64_t seg_count_total = 0, zio_count_total = 0;

	for (i = 0; i < spa->spa_root_vdev->vdev_children; i++) {
	vdev_t *vd = spa->spa_root_vdev->vdev_child[i];
	dsl_scan_io_queue_t *queue = vd->vdev_scan_io_queue;

	if (queue == NULL)
	continue;

	seg_size_total += queue->q_total_seg_size_this_txg;
	zio_size_total += queue->q_total_zio_size_this_txg;
	seg_count_total += queue->q_segs_this_txg;
	zio_count_total += queue->q_zios_this_txg;
	}

	if (seg_count_total == 0 \|\| zio_count_total == 0) {
	scn->scn_avg_seg_size_this_txg = 0;
	scn->scn_avg_zio_size_this_txg = 0;
	scn->scn_segs_this_txg = 0;
	scn->scn_zios_this_txg = 0;
	return;
	}

	scn->scn_avg_seg_size_this_txg = seg_size_total / seg_count_total;
	scn->scn_avg_zio_size_this_txg = zio_size_total / zio_count_total;
	scn->scn_segs_this_txg = seg_count_total;
	scn->scn_zios_this_txg = zio_count_total;
	}

	static int
	bpobj_dsl_scan_free_block_cb(void arg, const blkptr_t bp, boolean_t bp_freed,
	dmu_tx_t *tx)
	{
	ASSERT(!bp_freed);
	return (dsl_scan_free_block_cb(arg, bp, tx));
	}

	static int
	dsl_scan_obsolete_block_cb(void arg, const blkptr_t bp, boolean_t bp_freed,
	dmu_tx_t *tx)
	{
	ASSERT(!bp_freed);
	dsl_scan_t *scn = arg;
	const dva_t *dva = &bp->blk_dva[0];

	if (dsl_scan_async_block_should_pause(scn))
	return (SET_ERROR(ERESTART));

	spa_vdev_indirect_mark_obsolete(scn->scn_dp->dp_spa,
	DVA_GET_VDEV(dva), DVA_GET_OFFSET(dva),
	DVA_GET_ASIZE(dva), tx);
	scn->scn_visited_this_txg++;
	return (0);
	}

	boolean_t
	dsl_scan_active(dsl_scan_t *scn)
	{
	spa_t *spa = scn->scn_dp->dp_spa;
	uint64_t used = 0, comp, uncomp;
	boolean_t clones_left;

	if (spa->spa_load_state != SPA_LOAD_NONE)
	return (B_FALSE);
	if (spa_shutting_down(spa))
	return (B_FALSE);
	if ((dsl_scan_is_running(scn) && !dsl_scan_is_paused_scrub(scn)) \|\|
	(scn->scn_async_destroying && !scn->scn_async_stalled))
	return (B_TRUE);

	if (spa_version(scn->scn_dp->dp_spa) >= SPA_VERSION_DEADLISTS) {
	(void) bpobj_space(&scn->scn_dp->dp_free_bpobj,
	&used, &comp, &uncomp);
	}
	clones_left = spa_livelist_delete_check(spa);
	return ((used != 0) \|\| (clones_left));
	}

	static boolean_t
	dsl_scan_check_deferred(vdev_t *vd)
	{
	boolean_t need_resilver = B_FALSE;

	for (int c = 0; c < vd->vdev_children; c++) {
	need_resilver \|=
	dsl_scan_check_deferred(vd->vdev_child[c]);
	}

	if (!vdev_is_concrete(vd) \|\| vd->vdev_aux \|\|
	!vd->vdev_ops->vdev_op_leaf)
	return (need_resilver);

	if (!vd->vdev_resilver_deferred)
	need_resilver = B_TRUE;

	return (need_resilver);
	}

	static boolean_t
	dsl_scan_need_resilver(spa_t spa, const dva_t dva, size_t psize,
	uint64_t phys_birth)
	{
	vdev_t *vd;

	vd = vdev_lookup_top(spa, DVA_GET_VDEV(dva));

	if (vd->vdev_ops == &vdev_indirect_ops) {
	/*
	* The indirect vdev can point to multiple
	* vdevs. For simplicity, always create
	* the resilver zio_t. zio_vdev_io_start()
	* will bypass the child resilver i/o's if
	* they are on vdevs that don't have DTL's.
	*/
	return (B_TRUE);
	}

	if (DVA_GET_GANG(dva)) {
	/*
	* Gang members may be spread across multiple
	* vdevs, so the best estimate we have is the
	* scrub range, which has already been checked.
	* XXX -- it would be better to change our
	* allocation policy to ensure that all
	* gang members reside on the same vdev.
	*/
	return (B_TRUE);
	}

	/*
	* Check if the txg falls within the range which must be
	* resilvered. DVAs outside this range can always be skipped.
	*/
	if (!vdev_dtl_contains(vd, DTL_PARTIAL, phys_birth, 1))
	return (B_FALSE);

	/*
	* Check if the top-level vdev must resilver this offset.
	* When the offset does not intersect with a dirty leaf DTL
	* then it may be possible to skip the resilver IO. The psize
	* is provided instead of asize to simplify the check for RAIDZ.
	*/
	if (!vdev_dtl_need_resilver(vd, DVA_GET_OFFSET(dva), psize))
	return (B_FALSE);

	/*
	* Check that this top-level vdev has a device under it which
	* is resilvering and is not deferred.
	*/
	if (!dsl_scan_check_deferred(vd))
	return (B_FALSE);

	return (B_TRUE);
	}

	static int
	dsl_process_async_destroys(dsl_pool_t dp, dmu_tx_t tx)
	{
	dsl_scan_t *scn = dp->dp_scan;
	spa_t *spa = dp->dp_spa;
	int err = 0;

	if (spa_suspend_async_destroy(spa))
	return (0);

	if (zfs_free_bpobj_enabled &&
	spa_version(spa) >= SPA_VERSION_DEADLISTS) {
	scn->scn_is_bptree = B_FALSE;
	scn->scn_async_block_min_time_ms = zfs_free_min_time_ms;
	scn->scn_zio_root = zio_root(spa, NULL,
	NULL, ZIO_FLAG_MUSTSUCCEED);
	err = bpobj_iterate(&dp->dp_free_bpobj,
	bpobj_dsl_scan_free_block_cb, scn, tx);
	VERIFY0(zio_wait(scn->scn_zio_root));
	scn->scn_zio_root = NULL;

	if (err != 0 && err != ERESTART)
	zfs_panic_recover("error %u from bpobj_iterate()", err);
	}

	if (err == 0 && spa_feature_is_active(spa, SPA_FEATURE_ASYNC_DESTROY)) {
	ASSERT(scn->scn_async_destroying);
	scn->scn_is_bptree = B_TRUE;
	scn->scn_zio_root = zio_root(spa, NULL,
	NULL, ZIO_FLAG_MUSTSUCCEED);
	err = bptree_iterate(dp->dp_meta_objset,
	dp->dp_bptree_obj, B_TRUE, dsl_scan_free_block_cb, scn, tx);
	VERIFY0(zio_wait(scn->scn_zio_root));
	scn->scn_zio_root = NULL;

	if (err == EIO \|\| err == ECKSUM) {
	err = 0;
	} else if (err != 0 && err != ERESTART) {
	zfs_panic_recover("error %u from "
	"traverse_dataset_destroyed()", err);
	}

	if (bptree_is_empty(dp->dp_meta_objset, dp->dp_bptree_obj)) {
	/* finished; deactivate async destroy feature */
	spa_feature_decr(spa, SPA_FEATURE_ASYNC_DESTROY, tx);
	ASSERT(!spa_feature_is_active(spa,
	SPA_FEATURE_ASYNC_DESTROY));
	VERIFY0(zap_remove(dp->dp_meta_objset,
	DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_BPTREE_OBJ, tx));
	VERIFY0(bptree_free(dp->dp_meta_objset,
	dp->dp_bptree_obj, tx));
	dp->dp_bptree_obj = 0;
	scn->scn_async_destroying = B_FALSE;
	scn->scn_async_stalled = B_FALSE;
	} else {
	/*
	* If we didn't make progress, mark the async
	* destroy as stalled, so that we will not initiate
	* a spa_sync() on its behalf. Note that we only
	* check this if we are not finished, because if the
	* bptree had no blocks for us to visit, we can
	* finish without "making progress".
	*/
	scn->scn_async_stalled =
	(scn->scn_visited_this_txg == 0);
	}
	}
	if (scn->scn_visited_this_txg) {
	zfs_dbgmsg("freed %llu blocks in %llums from "
	"free_bpobj/bptree txg %llu; err=%u",
	(longlong_t)scn->scn_visited_this_txg,
	(longlong_t)
	NSEC2MSEC(gethrtime() - scn->scn_sync_start_time),
	(longlong_t)tx->tx_txg, err);
	scn->scn_visited_this_txg = 0;
	scn->scn_dedup_frees_this_txg = 0;

	/*
	* Write out changes to the DDT that may be required as a
	* result of the blocks freed. This ensures that the DDT
	* is clean when a scrub/resilver runs.
	*/
	ddt_sync(spa, tx->tx_txg);
	}
	if (err != 0)
	return (err);
	if (dp->dp_free_dir != NULL && !scn->scn_async_destroying &&
	zfs_free_leak_on_eio &&
	(dsl_dir_phys(dp->dp_free_dir)->dd_used_bytes != 0 \|\|
	dsl_dir_phys(dp->dp_free_dir)->dd_compressed_bytes != 0 \|\|
	dsl_dir_phys(dp->dp_free_dir)->dd_uncompressed_bytes != 0)) {
	/*
	* We have finished background destroying, but there is still
	* some space left in the dp_free_dir. Transfer this leaked
	* space to the dp_leak_dir.
	*/
	if (dp->dp_leak_dir == NULL) {
	rrw_enter(&dp->dp_config_rwlock, RW_WRITER, FTAG);
	(void) dsl_dir_create_sync(dp, dp->dp_root_dir,
	LEAK_DIR_NAME, tx);
	VERIFY0(dsl_pool_open_special_dir(dp,
	LEAK_DIR_NAME, &dp->dp_leak_dir));
	rrw_exit(&dp->dp_config_rwlock, FTAG);
	}
	dsl_dir_diduse_space(dp->dp_leak_dir, DD_USED_HEAD,
	dsl_dir_phys(dp->dp_free_dir)->dd_used_bytes,
	dsl_dir_phys(dp->dp_free_dir)->dd_compressed_bytes,
	dsl_dir_phys(dp->dp_free_dir)->dd_uncompressed_bytes, tx);
	dsl_dir_diduse_space(dp->dp_free_dir, DD_USED_HEAD,
	-dsl_dir_phys(dp->dp_free_dir)->dd_used_bytes,
	-dsl_dir_phys(dp->dp_free_dir)->dd_compressed_bytes,
	-dsl_dir_phys(dp->dp_free_dir)->dd_uncompressed_bytes, tx);
	}

	if (dp->dp_free_dir != NULL && !scn->scn_async_destroying &&
	!spa_livelist_delete_check(spa)) {
	/* finished; verify that space accounting went to zero */
	ASSERT0(dsl_dir_phys(dp->dp_free_dir)->dd_used_bytes);
	ASSERT0(dsl_dir_phys(dp->dp_free_dir)->dd_compressed_bytes);
	ASSERT0(dsl_dir_phys(dp->dp_free_dir)->dd_uncompressed_bytes);
	}

	spa_notify_waiters(spa);

	EQUIV(bpobj_is_open(&dp->dp_obsolete_bpobj),
	0 == zap_contains(dp->dp_meta_objset, DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_OBSOLETE_BPOBJ));
	if (err == 0 && bpobj_is_open(&dp->dp_obsolete_bpobj)) {
	ASSERT(spa_feature_is_active(dp->dp_spa,
	SPA_FEATURE_OBSOLETE_COUNTS));

	scn->scn_is_bptree = B_FALSE;
	scn->scn_async_block_min_time_ms = zfs_obsolete_min_time_ms;
	err = bpobj_iterate(&dp->dp_obsolete_bpobj,
	dsl_scan_obsolete_block_cb, scn, tx);
	if (err != 0 && err != ERESTART)
	zfs_panic_recover("error %u from bpobj_iterate()", err);

	if (bpobj_is_empty(&dp->dp_obsolete_bpobj))
	dsl_pool_destroy_obsolete_bpobj(dp, tx);
	}
	return (0);
	}

	/*
	* This is the primary entry point for scans that is called from syncing
	* context. Scans must happen entirely during syncing context so that we
	* can guarantee that blocks we are currently scanning will not change out
	* from under us. While a scan is active, this function controls how quickly
	* transaction groups proceed, instead of the normal handling provided by
	* txg_sync_thread().
	*/
	void
	dsl_scan_sync(dsl_pool_t dp, dmu_tx_t tx)
	{
	int err = 0;
	dsl_scan_t *scn = dp->dp_scan;
	spa_t *spa = dp->dp_spa;
	state_sync_type_t sync_type = SYNC_OPTIONAL;

	if (spa->spa_resilver_deferred &&
	!spa_feature_is_active(dp->dp_spa, SPA_FEATURE_RESILVER_DEFER))
	spa_feature_incr(spa, SPA_FEATURE_RESILVER_DEFER, tx);

	/*
	* Check for scn_restart_txg before checking spa_load_state, so
	* that we can restart an old-style scan while the pool is being
	* imported (see dsl_scan_init). We also restart scans if there
	* is a deferred resilver and the user has manually disabled
	* deferred resilvers via the tunable.
	*/
	if (dsl_scan_restarting(scn, tx) \|\|
	(spa->spa_resilver_deferred && zfs_resilver_disable_defer)) {
	pool_scan_func_t func = POOL_SCAN_SCRUB;
	dsl_scan_done(scn, B_FALSE, tx);
	if (vdev_resilver_needed(spa->spa_root_vdev, NULL, NULL))
	func = POOL_SCAN_RESILVER;
	zfs_dbgmsg("restarting scan func=%u txg=%llu",
	func, (longlong_t)tx->tx_txg);
	dsl_scan_setup_sync(&func, tx);
	}

	/*
	* Only process scans in sync pass 1.
	*/
	if (spa_sync_pass(spa) > 1)
	return;

	/*
	* If the spa is shutting down, then stop scanning. This will
	* ensure that the scan does not dirty any new data during the
	* shutdown phase.
	*/
	if (spa_shutting_down(spa))
	return;

	/*
	* If the scan is inactive due to a stalled async destroy, try again.
	*/
	if (!scn->scn_async_stalled && !dsl_scan_active(scn))
	return;

	/* reset scan statistics */
	scn->scn_visited_this_txg = 0;
	scn->scn_dedup_frees_this_txg = 0;
	scn->scn_holes_this_txg = 0;
	scn->scn_lt_min_this_txg = 0;
	scn->scn_gt_max_this_txg = 0;
	scn->scn_ddt_contained_this_txg = 0;
	scn->scn_objsets_visited_this_txg = 0;
	scn->scn_avg_seg_size_this_txg = 0;
	scn->scn_segs_this_txg = 0;
	scn->scn_avg_zio_size_this_txg = 0;
	scn->scn_zios_this_txg = 0;
	scn->scn_suspending = B_FALSE;
	scn->scn_sync_start_time = gethrtime();
	spa->spa_scrub_active = B_TRUE;

	/*
	* First process the async destroys. If we suspend, don't do
	* any scrubbing or resilvering. This ensures that there are no
	* async destroys while we are scanning, so the scan code doesn't
	* have to worry about traversing it. It is also faster to free the
	* blocks than to scrub them.
	*/
	err = dsl_process_async_destroys(dp, tx);
	if (err != 0)
	return;

	if (!dsl_scan_is_running(scn) \|\| dsl_scan_is_paused_scrub(scn))
	return;

	/*
	* Wait a few txgs after importing to begin scanning so that
	* we can get the pool imported quickly.
	*/
	if (spa->spa_syncing_txg < spa->spa_first_txg + SCAN_IMPORT_WAIT_TXGS)
	return;

	/*
	* zfs_scan_suspend_progress can be set to disable scan progress.
	* We don't want to spin the txg_sync thread, so we add a delay
	* here to simulate the time spent doing a scan. This is mostly
	* useful for testing and debugging.
	*/
	if (zfs_scan_suspend_progress) {
	uint64_t scan_time_ns = gethrtime() - scn->scn_sync_start_time;
	int mintime = (scn->scn_phys.scn_func == POOL_SCAN_RESILVER) ?
	zfs_resilver_min_time_ms : zfs_scrub_min_time_ms;

	while (zfs_scan_suspend_progress &&
	!txg_sync_waiting(scn->scn_dp) &&
	!spa_shutting_down(scn->scn_dp->dp_spa) &&
	NSEC2MSEC(scan_time_ns) < mintime) {
	delay(hz);
	scan_time_ns = gethrtime() - scn->scn_sync_start_time;
	}
	return;
	}

	/*
	* It is possible to switch from unsorted to sorted at any time,
	* but afterwards the scan will remain sorted unless reloaded from
	* a checkpoint after a reboot.
	*/
	if (!zfs_scan_legacy) {
	scn->scn_is_sorted = B_TRUE;
	if (scn->scn_last_checkpoint == 0)
	scn->scn_last_checkpoint = ddi_get_lbolt();
	}

	/*
	* For sorted scans, determine what kind of work we will be doing
	* this txg based on our memory limitations and whether or not we
	* need to perform a checkpoint.
	*/
	if (scn->scn_is_sorted) {
	/*
	* If we are over our checkpoint interval, set scn_clearing
	* so that we can begin checkpointing immediately. The
	* checkpoint allows us to save a consistent bookmark
	* representing how much data we have scrubbed so far.
	* Otherwise, use the memory limit to determine if we should
	* scan for metadata or start issue scrub IOs. We accumulate
	* metadata until we hit our hard memory limit at which point
	* we issue scrub IOs until we are at our soft memory limit.
	*/
	if (scn->scn_checkpointing \|\|
	ddi_get_lbolt() - scn->scn_last_checkpoint >
	SEC_TO_TICK(zfs_scan_checkpoint_intval)) {
	if (!scn->scn_checkpointing)
	zfs_dbgmsg("begin scan checkpoint");

	scn->scn_checkpointing = B_TRUE;
	scn->scn_clearing = B_TRUE;
	} else {
	boolean_t should_clear = dsl_scan_should_clear(scn);
	if (should_clear && !scn->scn_clearing) {
	zfs_dbgmsg("begin scan clearing");
	scn->scn_clearing = B_TRUE;
	} else if (!should_clear && scn->scn_clearing) {
	zfs_dbgmsg("finish scan clearing");
	scn->scn_clearing = B_FALSE;
	}
	}
	} else {
	ASSERT0(scn->scn_checkpointing);
	ASSERT0(scn->scn_clearing);
	}

	if (!scn->scn_clearing && scn->scn_done_txg == 0) {
	/* Need to scan metadata for more blocks to scrub */
	dsl_scan_phys_t *scnp = &scn->scn_phys;
	taskqid_t prefetch_tqid;
	uint64_t bytes_per_leaf = zfs_scan_vdev_limit;
	uint64_t nr_leaves = dsl_scan_count_leaves(spa->spa_root_vdev);

	/*
	* Recalculate the max number of in-flight bytes for pool-wide
	* scanning operations (minimum 1MB). Limits for the issuing
	* phase are done per top-level vdev and are handled separately.
	*/
	scn->scn_maxinflight_bytes =
	MAX(nr_leaves * bytes_per_leaf, 1ULL << 20);

	if (scnp->scn_ddt_bookmark.ddb_class <=
	scnp->scn_ddt_class_max) {
	ASSERT(ZB_IS_ZERO(&scnp->scn_bookmark));
	zfs_dbgmsg("doing scan sync txg %llu; "
	"ddt bm=%llu/%llu/%llu/%llx",
	(longlong_t)tx->tx_txg,
	(longlong_t)scnp->scn_ddt_bookmark.ddb_class,
	(longlong_t)scnp->scn_ddt_bookmark.ddb_type,
	(longlong_t)scnp->scn_ddt_bookmark.ddb_checksum,
	(longlong_t)scnp->scn_ddt_bookmark.ddb_cursor);
	} else {
	zfs_dbgmsg("doing scan sync txg %llu; "
	"bm=%llu/%llu/%llu/%llu",
	(longlong_t)tx->tx_txg,
	(longlong_t)scnp->scn_bookmark.zb_objset,
	(longlong_t)scnp->scn_bookmark.zb_object,
	(longlong_t)scnp->scn_bookmark.zb_level,
	(longlong_t)scnp->scn_bookmark.zb_blkid);
	}

	scn->scn_zio_root = zio_root(dp->dp_spa, NULL,
	NULL, ZIO_FLAG_CANFAIL);

	scn->scn_prefetch_stop = B_FALSE;
	prefetch_tqid = taskq_dispatch(dp->dp_sync_taskq,
	dsl_scan_prefetch_thread, scn, TQ_SLEEP);
	ASSERT(prefetch_tqid != TASKQID_INVALID);

	dsl_pool_config_enter(dp, FTAG);
	dsl_scan_visit(scn, tx);
	dsl_pool_config_exit(dp, FTAG);

	mutex_enter(&dp->dp_spa->spa_scrub_lock);
	scn->scn_prefetch_stop = B_TRUE;
	cv_broadcast(&spa->spa_scrub_io_cv);
	mutex_exit(&dp->dp_spa->spa_scrub_lock);

	taskq_wait_id(dp->dp_sync_taskq, prefetch_tqid);
	(void) zio_wait(scn->scn_zio_root);
	scn->scn_zio_root = NULL;

	zfs_dbgmsg("scan visited %llu blocks in %llums "
	"(%llu os's, %llu holes, %llu < mintxg, "
	"%llu in ddt, %llu > maxtxg)",
	(longlong_t)scn->scn_visited_this_txg,
	(longlong_t)NSEC2MSEC(gethrtime() -
	scn->scn_sync_start_time),
	(longlong_t)scn->scn_objsets_visited_this_txg,
	(longlong_t)scn->scn_holes_this_txg,
	(longlong_t)scn->scn_lt_min_this_txg,
	(longlong_t)scn->scn_ddt_contained_this_txg,
	(longlong_t)scn->scn_gt_max_this_txg);

	if (!scn->scn_suspending) {
	ASSERT0(avl_numnodes(&scn->scn_queue));
	scn->scn_done_txg = tx->tx_txg + 1;
	if (scn->scn_is_sorted) {
	scn->scn_checkpointing = B_TRUE;
	scn->scn_clearing = B_TRUE;
	}
	zfs_dbgmsg("scan complete txg %llu",
	(longlong_t)tx->tx_txg);
	}
	} else if (scn->scn_is_sorted && scn->scn_bytes_pending != 0) {
	ASSERT(scn->scn_clearing);

	/* need to issue scrubbing IOs from per-vdev queues */
	scn->scn_zio_root = zio_root(dp->dp_spa, NULL,
	NULL, ZIO_FLAG_CANFAIL);
	scan_io_queues_run(scn);
	(void) zio_wait(scn->scn_zio_root);
	scn->scn_zio_root = NULL;

	/* calculate and dprintf the current memory usage */
	(void) dsl_scan_should_clear(scn);
	dsl_scan_update_stats(scn);

	zfs_dbgmsg("scan issued %llu blocks (%llu segs) in %llums "
	"(avg_block_size = %llu, avg_seg_size = %llu)",
	(longlong_t)scn->scn_zios_this_txg,
	(longlong_t)scn->scn_segs_this_txg,
	(longlong_t)NSEC2MSEC(gethrtime() -
	scn->scn_sync_start_time),
	(longlong_t)scn->scn_avg_zio_size_this_txg,
	(longlong_t)scn->scn_avg_seg_size_this_txg);
	} else if (scn->scn_done_txg != 0 && scn->scn_done_txg <= tx->tx_txg) {
	/* Finished with everything. Mark the scrub as complete */
	zfs_dbgmsg("scan issuing complete txg %llu",
	(longlong_t)tx->tx_txg);
	ASSERT3U(scn->scn_done_txg, !=, 0);
	ASSERT0(spa->spa_scrub_inflight);
	ASSERT0(scn->scn_bytes_pending);
	dsl_scan_done(scn, B_TRUE, tx);
	sync_type = SYNC_MANDATORY;
	}

	dsl_scan_sync_state(scn, tx, sync_type);
	}

	static void
	count_block(dsl_scan_t scn, zfs_all_blkstats_t zab, const blkptr_t *bp)
	{
	int i;

	/*
	* Don't count embedded bp's, since we already did the work of
	* scanning these when we scanned the containing block.
	*/
	if (BP_IS_EMBEDDED(bp))
	return;

	/*
	* Update the spa's stats on how many bytes we have issued.
	* Sequential scrubs create a zio for each DVA of the bp. Each
	* of these will include all DVAs for repair purposes, but the
	* zio code will only try the first one unless there is an issue.
	* Therefore, we should only count the first DVA for these IOs.
	*/
	if (scn->scn_is_sorted) {
	atomic_add_64(&scn->scn_dp->dp_spa->spa_scan_pass_issued,
	DVA_GET_ASIZE(&bp->blk_dva[0]));
	} else {
	spa_t *spa = scn->scn_dp->dp_spa;

	for (i = 0; i < BP_GET_NDVAS(bp); i++) {
	atomic_add_64(&spa->spa_scan_pass_issued,
	DVA_GET_ASIZE(&bp->blk_dva[i]));
	}
	}

	/*
	* If we resume after a reboot, zab will be NULL; don't record
	* incomplete stats in that case.
	*/
	if (zab == NULL)
	return;

	mutex_enter(&zab->zab_lock);

	for (i = 0; i < 4; i++) {
	int l = (i < 2) ? BP_GET_LEVEL(bp) : DN_MAX_LEVELS;
	int t = (i & 1) ? BP_GET_TYPE(bp) : DMU_OT_TOTAL;

	if (t & DMU_OT_NEWTYPE)
	t = DMU_OT_OTHER;
	zfs_blkstat_t *zb = &zab->zab_type[l][t];
	int equal;

	zb->zb_count++;
	zb->zb_asize += BP_GET_ASIZE(bp);
	zb->zb_lsize += BP_GET_LSIZE(bp);
	zb->zb_psize += BP_GET_PSIZE(bp);
	zb->zb_gangs += BP_COUNT_GANG(bp);

	switch (BP_GET_NDVAS(bp)) {
	case 2:
	if (DVA_GET_VDEV(&bp->blk_dva[0]) ==
	DVA_GET_VDEV(&bp->blk_dva[1]))
	zb->zb_ditto_2_of_2_samevdev++;
	break;
	case 3:
	equal = (DVA_GET_VDEV(&bp->blk_dva[0]) ==
	DVA_GET_VDEV(&bp->blk_dva[1])) +
	(DVA_GET_VDEV(&bp->blk_dva[0]) ==
	DVA_GET_VDEV(&bp->blk_dva[2])) +
	(DVA_GET_VDEV(&bp->blk_dva[1]) ==
	DVA_GET_VDEV(&bp->blk_dva[2]));
	if (equal == 1)
	zb->zb_ditto_2_of_3_samevdev++;
	else if (equal == 3)
	zb->zb_ditto_3_of_3_samevdev++;
	break;
	}
	}

	mutex_exit(&zab->zab_lock);
	}

	static void
	scan_io_queue_insert_impl(dsl_scan_io_queue_t queue, scan_io_t sio)
	{
	avl_index_t idx;
	int64_t asize = SIO_GET_ASIZE(sio);
	dsl_scan_t *scn = queue->q_scn;

	ASSERT(MUTEX_HELD(&queue->q_vd->vdev_scan_io_queue_lock));

	if (avl_find(&queue->q_sios_by_addr, sio, &idx) != NULL) {
	/* block is already scheduled for reading */
	atomic_add_64(&scn->scn_bytes_pending, -asize);
	sio_free(sio);
	return;
	}
	avl_insert(&queue->q_sios_by_addr, sio, idx);
	queue->q_sio_memused += SIO_GET_MUSED(sio);
	range_tree_add(queue->q_exts_by_addr, SIO_GET_OFFSET(sio), asize);
	}

	/*
	* Given all the info we got from our metadata scanning process, we
	* construct a scan_io_t and insert it into the scan sorting queue. The
	* I/O must already be suitable for us to process. This is controlled
	* by dsl_scan_enqueue().
	*/
	static void
	scan_io_queue_insert(dsl_scan_io_queue_t queue, const blkptr_t bp, int dva_i,
	int zio_flags, const zbookmark_phys_t *zb)
	{
	dsl_scan_t *scn = queue->q_scn;
	scan_io_t *sio = sio_alloc(BP_GET_NDVAS(bp));

	ASSERT0(BP_IS_GANG(bp));
	ASSERT(MUTEX_HELD(&queue->q_vd->vdev_scan_io_queue_lock));

	bp2sio(bp, sio, dva_i);
	sio->sio_flags = zio_flags;
	sio->sio_zb = *zb;

	/*
	* Increment the bytes pending counter now so that we can't
	* get an integer underflow in case the worker processes the
	* zio before we get to incrementing this counter.
	*/
	atomic_add_64(&scn->scn_bytes_pending, SIO_GET_ASIZE(sio));

	scan_io_queue_insert_impl(queue, sio);
	}

	/*
	* Given a set of I/O parameters as discovered by the metadata traversal
	* process, attempts to place the I/O into the sorted queues (if allowed),
	* or immediately executes the I/O.
	*/
	static void
	dsl_scan_enqueue(dsl_pool_t dp, const blkptr_t bp, int zio_flags,
	const zbookmark_phys_t *zb)
	{
	spa_t *spa = dp->dp_spa;

	ASSERT(!BP_IS_EMBEDDED(bp));

	/*
	* Gang blocks are hard to issue sequentially, so we just issue them
	* here immediately instead of queuing them.
	*/
	if (!dp->dp_scan->scn_is_sorted \|\| BP_IS_GANG(bp)) {
	scan_exec_io(dp, bp, zio_flags, zb, NULL);
	return;
	}

	for (int i = 0; i < BP_GET_NDVAS(bp); i++) {
	dva_t dva;
	vdev_t *vdev;

	dva = bp->blk_dva[i];
	vdev = vdev_lookup_top(spa, DVA_GET_VDEV(&dva));
	ASSERT(vdev != NULL);

	mutex_enter(&vdev->vdev_scan_io_queue_lock);
	if (vdev->vdev_scan_io_queue == NULL)
	vdev->vdev_scan_io_queue = scan_io_queue_create(vdev);
	ASSERT(dp->dp_scan != NULL);
	scan_io_queue_insert(vdev->vdev_scan_io_queue, bp,
	i, zio_flags, zb);
	mutex_exit(&vdev->vdev_scan_io_queue_lock);
	}
	}

	static int
	dsl_scan_scrub_cb(dsl_pool_t *dp,
	const blkptr_t bp, const zbookmark_phys_t zb)
	{
	dsl_scan_t *scn = dp->dp_scan;
	spa_t *spa = dp->dp_spa;
	uint64_t phys_birth = BP_PHYSICAL_BIRTH(bp);
	size_t psize = BP_GET_PSIZE(bp);
	boolean_t needs_io = B_FALSE;
	int zio_flags = ZIO_FLAG_SCAN_THREAD \| ZIO_FLAG_RAW \| ZIO_FLAG_CANFAIL;


	if (phys_birth <= scn->scn_phys.scn_min_txg \|\|
	phys_birth >= scn->scn_phys.scn_max_txg) {
	count_block(scn, dp->dp_blkstats, bp);
	return (0);
	}

	/* Embedded BP's have phys_birth==0, so we reject them above. */
	ASSERT(!BP_IS_EMBEDDED(bp));

	ASSERT(DSL_SCAN_IS_SCRUB_RESILVER(scn));
	if (scn->scn_phys.scn_func == POOL_SCAN_SCRUB) {
	zio_flags \|= ZIO_FLAG_SCRUB;
	needs_io = B_TRUE;
	} else {
	ASSERT3U(scn->scn_phys.scn_func, ==, POOL_SCAN_RESILVER);
	zio_flags \|= ZIO_FLAG_RESILVER;
	needs_io = B_FALSE;
	}

	/* If it's an intent log block, failure is expected. */
	if (zb->zb_level == ZB_ZIL_LEVEL)
	zio_flags \|= ZIO_FLAG_SPECULATIVE;

	for (int d = 0; d < BP_GET_NDVAS(bp); d++) {
	const dva_t *dva = &bp->blk_dva[d];

	/*
	* Keep track of how much data we've examined so that
	* zpool(1M) status can make useful progress reports.
	*/
	scn->scn_phys.scn_examined += DVA_GET_ASIZE(dva);
	spa->spa_scan_pass_exam += DVA_GET_ASIZE(dva);

	/* if it's a resilver, this may not be in the target range */
	if (!needs_io)
	needs_io = dsl_scan_need_resilver(spa, dva, psize,
	phys_birth);
	}

	if (needs_io && !zfs_no_scrub_io) {
	dsl_scan_enqueue(dp, bp, zio_flags, zb);
	} else {
	count_block(scn, dp->dp_blkstats, bp);
	}

	/* do not relocate this block */
	return (0);
	}

	static void
	dsl_scan_scrub_done(zio_t *zio)
	{
	spa_t *spa = zio->io_spa;
	blkptr_t *bp = zio->io_bp;
	dsl_scan_io_queue_t *queue = zio->io_private;

	abd_free(zio->io_abd);

	if (queue == NULL) {
	mutex_enter(&spa->spa_scrub_lock);
	ASSERT3U(spa->spa_scrub_inflight, >=, BP_GET_PSIZE(bp));
	spa->spa_scrub_inflight -= BP_GET_PSIZE(bp);
	cv_broadcast(&spa->spa_scrub_io_cv);
	mutex_exit(&spa->spa_scrub_lock);
	} else {
	mutex_enter(&queue->q_vd->vdev_scan_io_queue_lock);
	ASSERT3U(queue->q_inflight_bytes, >=, BP_GET_PSIZE(bp));
	queue->q_inflight_bytes -= BP_GET_PSIZE(bp);
	cv_broadcast(&queue->q_zio_cv);
	mutex_exit(&queue->q_vd->vdev_scan_io_queue_lock);
	}

	if (zio->io_error && (zio->io_error != ECKSUM \|\|
	!(zio->io_flags & ZIO_FLAG_SPECULATIVE))) {
	atomic_inc_64(&spa->spa_dsl_pool->dp_scan->scn_phys.scn_errors);
	}
	}

	/*
	* Given a scanning zio's information, executes the zio. The zio need
	* not necessarily be only sortable, this function simply executes the
	* zio, no matter what it is. The optional queue argument allows the
	* caller to specify that they want per top level vdev IO rate limiting
	* instead of the legacy global limiting.
	*/
	static void
	scan_exec_io(dsl_pool_t dp, const blkptr_t bp, int zio_flags,
	const zbookmark_phys_t zb, dsl_scan_io_queue_t queue)
	{
	spa_t *spa = dp->dp_spa;
	dsl_scan_t *scn = dp->dp_scan;
	size_t size = BP_GET_PSIZE(bp);
	abd_t *data = abd_alloc_for_io(size, B_FALSE);

	ASSERT3U(scn->scn_maxinflight_bytes, >, 0);

	if (queue == NULL) {
	mutex_enter(&spa->spa_scrub_lock);
	while (spa->spa_scrub_inflight >= scn->scn_maxinflight_bytes)
	cv_wait(&spa->spa_scrub_io_cv, &spa->spa_scrub_lock);
	spa->spa_scrub_inflight += BP_GET_PSIZE(bp);
	mutex_exit(&spa->spa_scrub_lock);
	} else {
	kmutex_t *q_lock = &queue->q_vd->vdev_scan_io_queue_lock;

	mutex_enter(q_lock);
	while (queue->q_inflight_bytes >= queue->q_maxinflight_bytes)
	cv_wait(&queue->q_zio_cv, q_lock);
	queue->q_inflight_bytes += BP_GET_PSIZE(bp);
	mutex_exit(q_lock);
	}

	count_block(scn, dp->dp_blkstats, bp);
	zio_nowait(zio_read(scn->scn_zio_root, spa, bp, data, size,
	dsl_scan_scrub_done, queue, ZIO_PRIORITY_SCRUB, zio_flags, zb));
	}

	/*
	* This is the primary extent sorting algorithm. We balance two parameters:
	* 1) how many bytes of I/O are in an extent
	* 2) how well the extent is filled with I/O (as a fraction of its total size)
	* Since we allow extents to have gaps between their constituent I/Os, it's
	* possible to have a fairly large extent that contains the same amount of
	* I/O bytes than a much smaller extent, which just packs the I/O more tightly.
	* The algorithm sorts based on a score calculated from the extent's size,
	* the relative fill volume (in %) and a "fill weight" parameter that controls
	* the split between whether we prefer larger extents or more well populated
	* extents:
	*
	* SCORE = FILL_IN_BYTES + (FILL_IN_PERCENT * FILL_IN_BYTES * FILL_WEIGHT)
	*
	* Example:
	* 1) assume extsz = 64 MiB
	* 2) assume fill = 32 MiB (extent is half full)
	* 3) assume fill_weight = 3
	* 4) SCORE = 32M + (((32M * 100) / 64M) * 3 * 32M) / 100
	* SCORE = 32M + (50 * 3 * 32M) / 100
	* SCORE = 32M + (4800M / 100)
	* SCORE = 32M + 48M
	* ^ ^
	* \| +--- final total relative fill-based score
	* +--------- final total fill-based score
	* SCORE = 80M
	*
	* As can be seen, at fill_ratio=3, the algorithm is slightly biased towards
	* extents that are more completely filled (in a 3:2 ratio) vs just larger.
	* Note that as an optimization, we replace multiplication and division by
	* 100 with bitshifting by 7 (which effectively multiplies and divides by 128).
	*/
	static int
	ext_size_compare(const void x, const void y)
	{
	const range_seg_gap_t rsa = x, rsb = y;

	uint64_t sa = rsa->rs_end - rsa->rs_start;
	uint64_t sb = rsb->rs_end - rsb->rs_start;
	uint64_t score_a, score_b;

	score_a = rsa->rs_fill + ((((rsa->rs_fill << 7) / sa) *
	fill_weight * rsa->rs_fill) >> 7);
	score_b = rsb->rs_fill + ((((rsb->rs_fill << 7) / sb) *
	fill_weight * rsb->rs_fill) >> 7);

	if (score_a > score_b)
	return (-1);
	if (score_a == score_b) {
	if (rsa->rs_start < rsb->rs_start)
	return (-1);
	if (rsa->rs_start == rsb->rs_start)
	return (0);
	return (1);
	}
	return (1);
	}

	/*
	* Comparator for the q_sios_by_addr tree. Sorting is simply performed
	* based on LBA-order (from lowest to highest).
	*/
	static int
	sio_addr_compare(const void x, const void y)
	{
	const scan_io_t a = x, b = y;

	return (TREE_CMP(SIO_GET_OFFSET(a), SIO_GET_OFFSET(b)));
	}

	/* IO queues are created on demand when they are needed. */
	static dsl_scan_io_queue_t *
	scan_io_queue_create(vdev_t *vd)
	{
	dsl_scan_t *scn = vd->vdev_spa->spa_dsl_pool->dp_scan;
	dsl_scan_io_queue_t q = kmem_zalloc(sizeof (q), KM_SLEEP);

	q->q_scn = scn;
	q->q_vd = vd;
	q->q_sio_memused = 0;
	cv_init(&q->q_zio_cv, NULL, CV_DEFAULT, NULL);
	q->q_exts_by_addr = range_tree_create_impl(&rt_btree_ops, RANGE_SEG_GAP,
	&q->q_exts_by_size, 0, 0, ext_size_compare, zfs_scan_max_ext_gap);
	avl_create(&q->q_sios_by_addr, sio_addr_compare,
	sizeof (scan_io_t), offsetof(scan_io_t, sio_nodes.sio_addr_node));

	return (q);
	}

	/*
	* Destroys a scan queue and all segments and scan_io_t's contained in it.
	* No further execution of I/O occurs, anything pending in the queue is
	* simply freed without being executed.
	*/
	void
	dsl_scan_io_queue_destroy(dsl_scan_io_queue_t *queue)
	{
	dsl_scan_t *scn = queue->q_scn;
	scan_io_t *sio;
	void *cookie = NULL;
	int64_t bytes_dequeued = 0;

	ASSERT(MUTEX_HELD(&queue->q_vd->vdev_scan_io_queue_lock));

	while ((sio = avl_destroy_nodes(&queue->q_sios_by_addr, &cookie)) !=
	NULL) {
	ASSERT(range_tree_contains(queue->q_exts_by_addr,
	SIO_GET_OFFSET(sio), SIO_GET_ASIZE(sio)));
	bytes_dequeued += SIO_GET_ASIZE(sio);
	queue->q_sio_memused -= SIO_GET_MUSED(sio);
	sio_free(sio);
	}

	ASSERT0(queue->q_sio_memused);
	atomic_add_64(&scn->scn_bytes_pending, -bytes_dequeued);
	range_tree_vacate(queue->q_exts_by_addr, NULL, queue);
	range_tree_destroy(queue->q_exts_by_addr);
	avl_destroy(&queue->q_sios_by_addr);
	cv_destroy(&queue->q_zio_cv);

	kmem_free(queue, sizeof (*queue));
	}

	/*
	* Properly transfers a dsl_scan_queue_t from `svd' to `tvd'. This is
	* called on behalf of vdev_top_transfer when creating or destroying
	* a mirror vdev due to zpool attach/detach.
	*/
	void
	dsl_scan_io_queue_vdev_xfer(vdev_t svd, vdev_t tvd)
	{
	mutex_enter(&svd->vdev_scan_io_queue_lock);
	mutex_enter(&tvd->vdev_scan_io_queue_lock);

	VERIFY3P(tvd->vdev_scan_io_queue, ==, NULL);
	tvd->vdev_scan_io_queue = svd->vdev_scan_io_queue;
	svd->vdev_scan_io_queue = NULL;
	if (tvd->vdev_scan_io_queue != NULL)
	tvd->vdev_scan_io_queue->q_vd = tvd;

	mutex_exit(&tvd->vdev_scan_io_queue_lock);
	mutex_exit(&svd->vdev_scan_io_queue_lock);
	}

	static void
	scan_io_queues_destroy(dsl_scan_t *scn)
	{
	vdev_t *rvd = scn->scn_dp->dp_spa->spa_root_vdev;

	for (uint64_t i = 0; i < rvd->vdev_children; i++) {
	vdev_t *tvd = rvd->vdev_child[i];

	mutex_enter(&tvd->vdev_scan_io_queue_lock);
	if (tvd->vdev_scan_io_queue != NULL)
	dsl_scan_io_queue_destroy(tvd->vdev_scan_io_queue);
	tvd->vdev_scan_io_queue = NULL;
	mutex_exit(&tvd->vdev_scan_io_queue_lock);
	}
	}

	static void
	dsl_scan_freed_dva(spa_t spa, const blkptr_t bp, int dva_i)
	{
	dsl_pool_t *dp = spa->spa_dsl_pool;
	dsl_scan_t *scn = dp->dp_scan;
	vdev_t *vdev;
	kmutex_t *q_lock;
	dsl_scan_io_queue_t *queue;
	scan_io_t srch_sio, sio;
	avl_index_t idx;
	uint64_t start, size;

	vdev = vdev_lookup_top(spa, DVA_GET_VDEV(&bp->blk_dva[dva_i]));
	ASSERT(vdev != NULL);
	q_lock = &vdev->vdev_scan_io_queue_lock;
	queue = vdev->vdev_scan_io_queue;

	mutex_enter(q_lock);
	if (queue == NULL) {
	mutex_exit(q_lock);
	return;
	}

	srch_sio = sio_alloc(BP_GET_NDVAS(bp));
	bp2sio(bp, srch_sio, dva_i);
	start = SIO_GET_OFFSET(srch_sio);
	size = SIO_GET_ASIZE(srch_sio);

	/*
	* We can find the zio in two states:
	* 1) Cold, just sitting in the queue of zio's to be issued at
	* some point in the future. In this case, all we do is
	* remove the zio from the q_sios_by_addr tree, decrement
	* its data volume from the containing range_seg_t and
	* resort the q_exts_by_size tree to reflect that the
	* range_seg_t has lost some of its 'fill'. We don't shorten
	* the range_seg_t - this is usually rare enough not to be
	* worth the extra hassle of trying keep track of precise
	* extent boundaries.
	* 2) Hot, where the zio is currently in-flight in
	* dsl_scan_issue_ios. In this case, we can't simply
	* reach in and stop the in-flight zio's, so we instead
	* block the caller. Eventually, dsl_scan_issue_ios will
	* be done with issuing the zio's it gathered and will
	* signal us.
	*/
	sio = avl_find(&queue->q_sios_by_addr, srch_sio, &idx);
	sio_free(srch_sio);

	if (sio != NULL) {
	int64_t asize = SIO_GET_ASIZE(sio);
	blkptr_t tmpbp;

	/* Got it while it was cold in the queue */
	ASSERT3U(start, ==, SIO_GET_OFFSET(sio));
	ASSERT3U(size, ==, asize);
	avl_remove(&queue->q_sios_by_addr, sio);
	queue->q_sio_memused -= SIO_GET_MUSED(sio);

	ASSERT(range_tree_contains(queue->q_exts_by_addr, start, size));
	range_tree_remove_fill(queue->q_exts_by_addr, start, size);

	/*
	* We only update scn_bytes_pending in the cold path,
	* otherwise it will already have been accounted for as
	* part of the zio's execution.
	*/
	atomic_add_64(&scn->scn_bytes_pending, -asize);

	/* count the block as though we issued it */
	sio2bp(sio, &tmpbp);
	count_block(scn, dp->dp_blkstats, &tmpbp);

	sio_free(sio);
	}
	mutex_exit(q_lock);
	}

	/*
	* Callback invoked when a zio_free() zio is executing. This needs to be
	* intercepted to prevent the zio from deallocating a particular portion
	* of disk space and it then getting reallocated and written to, while we
	* still have it queued up for processing.
	*/
	void
	dsl_scan_freed(spa_t spa, const blkptr_t bp)
	{
	dsl_pool_t *dp = spa->spa_dsl_pool;
	dsl_scan_t *scn = dp->dp_scan;

	ASSERT(!BP_IS_EMBEDDED(bp));
	ASSERT(scn != NULL);
	if (!dsl_scan_is_running(scn))
	return;

	for (int i = 0; i < BP_GET_NDVAS(bp); i++)
	dsl_scan_freed_dva(spa, bp, i);
	}

	/*
	* Check if a vdev needs resilvering (non-empty DTL), if so, and resilver has
	* not started, start it. Otherwise, only restart if max txg in DTL range is
	* greater than the max txg in the current scan. If the DTL max is less than
	* the scan max, then the vdev has not missed any new data since the resilver
	* started, so a restart is not needed.
	*/
	void
	dsl_scan_assess_vdev(dsl_pool_t dp, vdev_t vd)
	{
	uint64_t min, max;

	if (!vdev_resilver_needed(vd, &min, &max))
	return;

	if (!dsl_scan_resilvering(dp)) {
	spa_async_request(dp->dp_spa, SPA_ASYNC_RESILVER);
	return;
	}

	if (max <= dp->dp_scan->scn_phys.scn_max_txg)
	return;

	/* restart is needed, check if it can be deferred */
	if (spa_feature_is_enabled(dp->dp_spa, SPA_FEATURE_RESILVER_DEFER))
	vdev_defer_resilver(vd);
	else
	spa_async_request(dp->dp_spa, SPA_ASYNC_RESILVER);
	}

	/* BEGIN CSTYLED */
	ZFS_MODULE_PARAM(zfs, zfs_, scan_vdev_limit, ULONG, ZMOD_RW,
	"Max bytes in flight per leaf vdev for scrubs and resilvers");

	ZFS_MODULE_PARAM(zfs, zfs_, scrub_min_time_ms, INT, ZMOD_RW,
	"Min millisecs to scrub per txg");

	ZFS_MODULE_PARAM(zfs, zfs_, obsolete_min_time_ms, INT, ZMOD_RW,
	"Min millisecs to obsolete per txg");

	ZFS_MODULE_PARAM(zfs, zfs_, free_min_time_ms, INT, ZMOD_RW,
	"Min millisecs to free per txg");

	ZFS_MODULE_PARAM(zfs, zfs_, resilver_min_time_ms, INT, ZMOD_RW,
	"Min millisecs to resilver per txg");

	ZFS_MODULE_PARAM(zfs, zfs_, scan_suspend_progress, INT, ZMOD_RW,
	"Set to prevent scans from progressing");

	ZFS_MODULE_PARAM(zfs, zfs_, no_scrub_io, INT, ZMOD_RW,
	"Set to disable scrub I/O");

	ZFS_MODULE_PARAM(zfs, zfs_, no_scrub_prefetch, INT, ZMOD_RW,
	"Set to disable scrub prefetching");

	ZFS_MODULE_PARAM(zfs, zfs_, async_block_max_blocks, ULONG, ZMOD_RW,
	"Max number of blocks freed in one txg");

	ZFS_MODULE_PARAM(zfs, zfs_, max_async_dedup_frees, ULONG, ZMOD_RW,
	"Max number of dedup blocks freed in one txg");

	ZFS_MODULE_PARAM(zfs, zfs_, free_bpobj_enabled, INT, ZMOD_RW,
	"Enable processing of the free_bpobj");

	ZFS_MODULE_PARAM(zfs, zfs_, scan_mem_lim_fact, INT, ZMOD_RW,
	"Fraction of RAM for scan hard limit");

	ZFS_MODULE_PARAM(zfs, zfs_, scan_issue_strategy, INT, ZMOD_RW,
	"IO issuing strategy during scrubbing. "
	"0 = default, 1 = LBA, 2 = size");

	ZFS_MODULE_PARAM(zfs, zfs_, scan_legacy, INT, ZMOD_RW,
	"Scrub using legacy non-sequential method");

	ZFS_MODULE_PARAM(zfs, zfs_, scan_checkpoint_intval, INT, ZMOD_RW,
	"Scan progress on-disk checkpointing interval");

	ZFS_MODULE_PARAM(zfs, zfs_, scan_max_ext_gap, ULONG, ZMOD_RW,
	"Max gap in bytes between sequential scrub / resilver I/Os");

	ZFS_MODULE_PARAM(zfs, zfs_, scan_mem_lim_soft_fact, INT, ZMOD_RW,
	"Fraction of hard limit used as soft limit");

	ZFS_MODULE_PARAM(zfs, zfs_, scan_strict_mem_lim, INT, ZMOD_RW,
	"Tunable to attempt to reduce lock contention");

	ZFS_MODULE_PARAM(zfs, zfs_, scan_fill_weight, INT, ZMOD_RW,
	"Tunable to adjust bias towards more filled segments during scans");

	ZFS_MODULE_PARAM(zfs, zfs_, resilver_disable_defer, INT, ZMOD_RW,
	"Process all resilvers immediately");
	/* END CSTYLED */
	Index: vendor-sys/openzfs/dist/module/zfs/pathname.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/zfs/pathname.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/zfs/pathname.c (revision 366774)
	@@ -1,96 +1,88 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright 2007 Sun Microsystems, Inc. All rights reserved.
	* Use is subject to license terms.
	*/

	/* Copyright (c) 1983, 1984, 1985, 1986, 1987, 1988, 1989 AT&T */
	/* All Rights Reserved */

	/*
	* University Copyright- Copyright (c) 1982, 1986, 1988
	* The Regents of the University of California
	* All Rights Reserved
	*
	* University Acknowledgment- Portions of this document are derived from
	* software developed by the University of California, Berkeley, and its
	* contributors.
	*/


	#include <sys/types.h>
	#include <sys/pathname.h>
	#include <sys/kmem.h>
	#include <sys/sysmacros.h>

	/*
	* Pathname utilities.
	*
	* In translating file names we copy each argument file
	* name into a pathname structure where we operate on it.
	* Each pathname structure can hold "pn_bufsize" characters
	* including a terminating null, and operations here support
	* allocating and freeing pathname structures, fetching
	* strings from user space, getting the next character from
	* a pathname, combining two pathnames (used in symbolic
	* link processing), and peeling off the first component
	* of a pathname.
	*/

	/*
	* Allocate contents of pathname structure. Structure is typically
	* an automatic variable in calling routine for convenience.
	*
	* May sleep in the call to kmem_alloc() and so must not be called
	* from interrupt level.
	*/
	void
	pn_alloc(struct pathname *pnp)
	{
	pn_alloc_sz(pnp, MAXPATHLEN);
	}
	void
	pn_alloc_sz(struct pathname *pnp, size_t sz)
	{
	pnp->pn_buf = kmem_alloc(sz, KM_SLEEP);
	pnp->pn_bufsize = sz;
	-#if 0 /* unused in ZoL */
	- pnp->pn_path = pnp->pn_buf;
	- pnp->pn_pathlen = 0;
	-#endif
	}

	/*
	* Free pathname resources.
	*/
	void
	pn_free(struct pathname *pnp)
	{
	/* pn_bufsize is usually MAXPATHLEN, but may not be */
	kmem_free(pnp->pn_buf, pnp->pn_bufsize);
	pnp->pn_buf = NULL;
	pnp->pn_bufsize = 0;
	-#if 0 /* unused in ZoL */
	- pnp->pn_path = NULL;
	- pnp->pn_pathlen = 0;
	-#endif
	}
	Index: vendor-sys/openzfs/dist/module/zfs/range_tree.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/zfs/range_tree.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/zfs/range_tree.c (revision 366774)
	@@ -1,922 +1,922 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright 2009 Sun Microsystems, Inc. All rights reserved.
	* Use is subject to license terms.
	*/
	/*
	* Copyright (c) 2013, 2019 by Delphix. All rights reserved.
	* Copyright (c) 2015, Nexenta Systems, Inc. All rights reserved.
	*/

	#include <sys/zfs_context.h>
	#include <sys/spa.h>
	#include <sys/dmu.h>
	#include <sys/dnode.h>
	#include <sys/zio.h>
	#include <sys/range_tree.h>

	/*
	* Range trees are tree-based data structures that can be used to
	* track free space or generally any space allocation information.
	* A range tree keeps track of individual segments and automatically
	* provides facilities such as adjacent extent merging and extent
	* splitting in response to range add/remove requests.
	*
	* A range tree starts out completely empty, with no segments in it.
	* Adding an allocation via range_tree_add to the range tree can either:
	* 1) create a new extent
	* 2) extend an adjacent extent
	* 3) merge two adjacent extents
	* Conversely, removing an allocation via range_tree_remove can:
	* 1) completely remove an extent
	* 2) shorten an extent (if the allocation was near one of its ends)
	* 3) split an extent into two extents, in effect punching a hole
	*
	* A range tree is also capable of 'bridging' gaps when adding
	* allocations. This is useful for cases when close proximity of
	* allocations is an important detail that needs to be represented
	* in the range tree. See range_tree_set_gap(). The default behavior
	* is not to bridge gaps (i.e. the maximum allowed gap size is 0).
	*
	* In order to traverse a range tree, use either the range_tree_walk()
	* or range_tree_vacate() functions.
	*
	* To obtain more accurate information on individual segment
	* operations that the range tree performs "under the hood", you can
	* specify a set of callbacks by passing a range_tree_ops_t structure
	* to the range_tree_create function. Any callbacks that are non-NULL
	* are then called at the appropriate times.
	*
	* The range tree code also supports a special variant of range trees
	* that can bridge small gaps between segments. This kind of tree is used
	* by the dsl scanning code to group I/Os into mostly sequential chunks to
	* optimize disk performance. The code here attempts to do this with as
	* little memory and computational overhead as possible. One limitation of
	* this implementation is that segments of range trees with gaps can only
	* support removing complete segments.
	*/

	static inline void
	rs_copy(range_seg_t src, range_seg_t dest, range_tree_t *rt)
	{
	ASSERT3U(rt->rt_type, <=, RANGE_SEG_NUM_TYPES);
	size_t size = 0;
	switch (rt->rt_type) {
	case RANGE_SEG32:
	size = sizeof (range_seg32_t);
	break;
	case RANGE_SEG64:
	size = sizeof (range_seg64_t);
	break;
	case RANGE_SEG_GAP:
	size = sizeof (range_seg_gap_t);
	break;
	default:
	VERIFY(0);
	}
	bcopy(src, dest, size);
	}

	void
	range_tree_stat_verify(range_tree_t *rt)
	{
	range_seg_t *rs;
	zfs_btree_index_t where;
	uint64_t hist[RANGE_TREE_HISTOGRAM_SIZE] = { 0 };
	int i;

	for (rs = zfs_btree_first(&rt->rt_root, &where); rs != NULL;
	rs = zfs_btree_next(&rt->rt_root, &where, &where)) {
	uint64_t size = rs_get_end(rs, rt) - rs_get_start(rs, rt);
	int idx = highbit64(size) - 1;

	hist[idx]++;
	ASSERT3U(hist[idx], !=, 0);
	}

	for (i = 0; i < RANGE_TREE_HISTOGRAM_SIZE; i++) {
	if (hist[i] != rt->rt_histogram[i]) {
	zfs_dbgmsg("i=%d, hist=%px, hist=%llu, rt_hist=%llu",
	i, hist, hist[i], rt->rt_histogram[i]);
	}
	VERIFY3U(hist[i], ==, rt->rt_histogram[i]);
	}
	}

	static void
	range_tree_stat_incr(range_tree_t rt, range_seg_t rs)
	{
	uint64_t size = rs_get_end(rs, rt) - rs_get_start(rs, rt);
	int idx = highbit64(size) - 1;

	ASSERT(size != 0);
	ASSERT3U(idx, <,
	sizeof (rt->rt_histogram) / sizeof (*rt->rt_histogram));

	rt->rt_histogram[idx]++;
	ASSERT3U(rt->rt_histogram[idx], !=, 0);
	}

	static void
	range_tree_stat_decr(range_tree_t rt, range_seg_t rs)
	{
	uint64_t size = rs_get_end(rs, rt) - rs_get_start(rs, rt);
	int idx = highbit64(size) - 1;

	ASSERT(size != 0);
	ASSERT3U(idx, <,
	sizeof (rt->rt_histogram) / sizeof (*rt->rt_histogram));

	ASSERT3U(rt->rt_histogram[idx], !=, 0);
	rt->rt_histogram[idx]--;
	}

	static int
	range_tree_seg32_compare(const void x1, const void x2)
	{
	const range_seg32_t *r1 = x1;
	const range_seg32_t *r2 = x2;

	ASSERT3U(r1->rs_start, <=, r1->rs_end);
	ASSERT3U(r2->rs_start, <=, r2->rs_end);

	return ((r1->rs_start >= r2->rs_end) - (r1->rs_end <= r2->rs_start));
	}

	static int
	range_tree_seg64_compare(const void x1, const void x2)
	{
	const range_seg64_t *r1 = x1;
	const range_seg64_t *r2 = x2;

	ASSERT3U(r1->rs_start, <=, r1->rs_end);
	ASSERT3U(r2->rs_start, <=, r2->rs_end);

	return ((r1->rs_start >= r2->rs_end) - (r1->rs_end <= r2->rs_start));
	}

	static int
	range_tree_seg_gap_compare(const void x1, const void x2)
	{
	const range_seg_gap_t *r1 = x1;
	const range_seg_gap_t *r2 = x2;

	ASSERT3U(r1->rs_start, <=, r1->rs_end);
	ASSERT3U(r2->rs_start, <=, r2->rs_end);

	return ((r1->rs_start >= r2->rs_end) - (r1->rs_end <= r2->rs_start));
	}

	range_tree_t *
	range_tree_create_impl(range_tree_ops_t ops, range_seg_type_t type, void arg,
	uint64_t start, uint64_t shift,
	int (zfs_btree_compare) (const void , const void *),
	uint64_t gap)
	{
	range_tree_t *rt = kmem_zalloc(sizeof (range_tree_t), KM_SLEEP);

	ASSERT3U(shift, <, 64);
	ASSERT3U(type, <=, RANGE_SEG_NUM_TYPES);
	size_t size;
	int (compare) (const void , const void *);
	switch (type) {
	case RANGE_SEG32:
	size = sizeof (range_seg32_t);
	compare = range_tree_seg32_compare;
	break;
	case RANGE_SEG64:
	size = sizeof (range_seg64_t);
	compare = range_tree_seg64_compare;
	break;
	case RANGE_SEG_GAP:
	size = sizeof (range_seg_gap_t);
	compare = range_tree_seg_gap_compare;
	break;
	default:
	panic("Invalid range seg type %d", type);
	}
	zfs_btree_create(&rt->rt_root, compare, size);

	rt->rt_ops = ops;
	rt->rt_gap = gap;
	rt->rt_arg = arg;
	rt->rt_type = type;
	rt->rt_start = start;
	rt->rt_shift = shift;
	rt->rt_btree_compare = zfs_btree_compare;

	if (rt->rt_ops != NULL && rt->rt_ops->rtop_create != NULL)
	rt->rt_ops->rtop_create(rt, rt->rt_arg);

	return (rt);
	}

	range_tree_t *
	range_tree_create(range_tree_ops_t *ops, range_seg_type_t type,
	void *arg, uint64_t start, uint64_t shift)
	{
	return (range_tree_create_impl(ops, type, arg, start, shift, NULL, 0));
	}

	void
	range_tree_destroy(range_tree_t *rt)
	{
	VERIFY0(rt->rt_space);

	if (rt->rt_ops != NULL && rt->rt_ops->rtop_destroy != NULL)
	rt->rt_ops->rtop_destroy(rt, rt->rt_arg);

	zfs_btree_destroy(&rt->rt_root);
	kmem_free(rt, sizeof (*rt));
	}

	void
	range_tree_adjust_fill(range_tree_t rt, range_seg_t rs, int64_t delta)
	{
	if (delta < 0 && delta * -1 >= rs_get_fill(rs, rt)) {
	zfs_panic_recover("zfs: attempting to decrease fill to or "
	"below 0; probable double remove in segment [%llx:%llx]",
	(longlong_t)rs_get_start(rs, rt),
	(longlong_t)rs_get_end(rs, rt));
	}
	if (rs_get_fill(rs, rt) + delta > rs_get_end(rs, rt) -
	rs_get_start(rs, rt)) {
	zfs_panic_recover("zfs: attempting to increase fill beyond "
	"max; probable double add in segment [%llx:%llx]",
	(longlong_t)rs_get_start(rs, rt),
	(longlong_t)rs_get_end(rs, rt));
	}

	if (rt->rt_ops != NULL && rt->rt_ops->rtop_remove != NULL)
	rt->rt_ops->rtop_remove(rt, rs, rt->rt_arg);
	rs_set_fill(rs, rt, rs_get_fill(rs, rt) + delta);
	if (rt->rt_ops != NULL && rt->rt_ops->rtop_add != NULL)
	rt->rt_ops->rtop_add(rt, rs, rt->rt_arg);
	}

	static void
	range_tree_add_impl(void *arg, uint64_t start, uint64_t size, uint64_t fill)
	{
	range_tree_t *rt = arg;
	zfs_btree_index_t where;
	range_seg_t rs_before, rs_after, *rs;
	range_seg_max_t tmp, rsearch;
	uint64_t end = start + size, gap = rt->rt_gap;
	uint64_t bridge_size = 0;
	boolean_t merge_before, merge_after;

	ASSERT3U(size, !=, 0);
	ASSERT3U(fill, <=, size);
	ASSERT3U(start + size, >, start);

	rs_set_start(&rsearch, rt, start);
	rs_set_end(&rsearch, rt, end);
	rs = zfs_btree_find(&rt->rt_root, &rsearch, &where);

	/*
	* If this is a gap-supporting range tree, it is possible that we
	* are inserting into an existing segment. In this case simply
	* bump the fill count and call the remove / add callbacks. If the
	* new range will extend an existing segment, we remove the
	* existing one, apply the new extent to it and re-insert it using
	* the normal code paths.
	*/
	if (rs != NULL) {
	if (gap == 0) {
	zfs_panic_recover("zfs: adding existent segment to "
	"range tree (offset=%llx size=%llx)",
	(longlong_t)start, (longlong_t)size);
	return;
	}
	uint64_t rstart = rs_get_start(rs, rt);
	uint64_t rend = rs_get_end(rs, rt);
	if (rstart <= start && rend >= end) {
	range_tree_adjust_fill(rt, rs, fill);
	return;
	}

	- zfs_btree_remove(&rt->rt_root, rs);
	if (rt->rt_ops != NULL && rt->rt_ops->rtop_remove != NULL)
	rt->rt_ops->rtop_remove(rt, rs, rt->rt_arg);

	range_tree_stat_decr(rt, rs);
	rt->rt_space -= rend - rstart;

	fill += rs_get_fill(rs, rt);
	start = MIN(start, rstart);
	end = MAX(end, rend);
	size = end - start;

	+ zfs_btree_remove(&rt->rt_root, rs);
	range_tree_add_impl(rt, start, size, fill);
	return;
	}

	ASSERT3P(rs, ==, NULL);

	/*
	* Determine whether or not we will have to merge with our neighbors.
	* If gap != 0, we might need to merge with our neighbors even if we
	* aren't directly touching.
	*/
	zfs_btree_index_t where_before, where_after;
	rs_before = zfs_btree_prev(&rt->rt_root, &where, &where_before);
	rs_after = zfs_btree_next(&rt->rt_root, &where, &where_after);

	merge_before = (rs_before != NULL && rs_get_end(rs_before, rt) >=
	start - gap);
	merge_after = (rs_after != NULL && rs_get_start(rs_after, rt) <= end +
	gap);

	if (merge_before && gap != 0)
	bridge_size += start - rs_get_end(rs_before, rt);
	if (merge_after && gap != 0)
	bridge_size += rs_get_start(rs_after, rt) - end;

	if (merge_before && merge_after) {
	if (rt->rt_ops != NULL && rt->rt_ops->rtop_remove != NULL) {
	rt->rt_ops->rtop_remove(rt, rs_before, rt->rt_arg);
	rt->rt_ops->rtop_remove(rt, rs_after, rt->rt_arg);
	}

	range_tree_stat_decr(rt, rs_before);
	range_tree_stat_decr(rt, rs_after);

	rs_copy(rs_after, &tmp, rt);
	uint64_t before_start = rs_get_start_raw(rs_before, rt);
	uint64_t before_fill = rs_get_fill(rs_before, rt);
	uint64_t after_fill = rs_get_fill(rs_after, rt);
	zfs_btree_remove_idx(&rt->rt_root, &where_before);

	/*
	* We have to re-find the node because our old reference is
	* invalid as soon as we do any mutating btree operations.
	*/
	rs_after = zfs_btree_find(&rt->rt_root, &tmp, &where_after);
	rs_set_start_raw(rs_after, rt, before_start);
	rs_set_fill(rs_after, rt, after_fill + before_fill + fill);
	rs = rs_after;
	} else if (merge_before) {
	if (rt->rt_ops != NULL && rt->rt_ops->rtop_remove != NULL)
	rt->rt_ops->rtop_remove(rt, rs_before, rt->rt_arg);

	range_tree_stat_decr(rt, rs_before);

	uint64_t before_fill = rs_get_fill(rs_before, rt);
	rs_set_end(rs_before, rt, end);
	rs_set_fill(rs_before, rt, before_fill + fill);
	rs = rs_before;
	} else if (merge_after) {
	if (rt->rt_ops != NULL && rt->rt_ops->rtop_remove != NULL)
	rt->rt_ops->rtop_remove(rt, rs_after, rt->rt_arg);

	range_tree_stat_decr(rt, rs_after);

	uint64_t after_fill = rs_get_fill(rs_after, rt);
	rs_set_start(rs_after, rt, start);
	rs_set_fill(rs_after, rt, after_fill + fill);
	rs = rs_after;
	} else {
	rs = &tmp;

	rs_set_start(rs, rt, start);
	rs_set_end(rs, rt, end);
	rs_set_fill(rs, rt, fill);
	zfs_btree_add_idx(&rt->rt_root, rs, &where);
	}

	if (gap != 0) {
	ASSERT3U(rs_get_fill(rs, rt), <=, rs_get_end(rs, rt) -
	rs_get_start(rs, rt));
	} else {
	ASSERT3U(rs_get_fill(rs, rt), ==, rs_get_end(rs, rt) -
	rs_get_start(rs, rt));
	}

	if (rt->rt_ops != NULL && rt->rt_ops->rtop_add != NULL)
	rt->rt_ops->rtop_add(rt, rs, rt->rt_arg);

	range_tree_stat_incr(rt, rs);
	rt->rt_space += size + bridge_size;
	}

	void
	range_tree_add(void *arg, uint64_t start, uint64_t size)
	{
	range_tree_add_impl(arg, start, size, size);
	}

	static void
	range_tree_remove_impl(range_tree_t *rt, uint64_t start, uint64_t size,
	boolean_t do_fill)
	{
	zfs_btree_index_t where;
	range_seg_t *rs;
	range_seg_max_t rsearch, rs_tmp;
	uint64_t end = start + size;
	boolean_t left_over, right_over;

	VERIFY3U(size, !=, 0);
	VERIFY3U(size, <=, rt->rt_space);
	if (rt->rt_type == RANGE_SEG64)
	ASSERT3U(start + size, >, start);

	rs_set_start(&rsearch, rt, start);
	rs_set_end(&rsearch, rt, end);
	rs = zfs_btree_find(&rt->rt_root, &rsearch, &where);

	/* Make sure we completely overlap with someone */
	if (rs == NULL) {
	zfs_panic_recover("zfs: removing nonexistent segment from "
	"range tree (offset=%llx size=%llx)",
	(longlong_t)start, (longlong_t)size);
	return;
	}

	/*
	* Range trees with gap support must only remove complete segments
	* from the tree. This allows us to maintain accurate fill accounting
	* and to ensure that bridged sections are not leaked. If we need to
	* remove less than the full segment, we can only adjust the fill count.
	*/
	if (rt->rt_gap != 0) {
	if (do_fill) {
	if (rs_get_fill(rs, rt) == size) {
	start = rs_get_start(rs, rt);
	end = rs_get_end(rs, rt);
	size = end - start;
	} else {
	range_tree_adjust_fill(rt, rs, -size);
	return;
	}
	} else if (rs_get_start(rs, rt) != start \|\|
	rs_get_end(rs, rt) != end) {
	zfs_panic_recover("zfs: freeing partial segment of "
	"gap tree (offset=%llx size=%llx) of "
	"(offset=%llx size=%llx)",
	(longlong_t)start, (longlong_t)size,
	(longlong_t)rs_get_start(rs, rt),
	(longlong_t)rs_get_end(rs, rt) - rs_get_start(rs,
	rt));
	return;
	}
	}

	VERIFY3U(rs_get_start(rs, rt), <=, start);
	VERIFY3U(rs_get_end(rs, rt), >=, end);

	left_over = (rs_get_start(rs, rt) != start);
	right_over = (rs_get_end(rs, rt) != end);

	range_tree_stat_decr(rt, rs);

	if (rt->rt_ops != NULL && rt->rt_ops->rtop_remove != NULL)
	rt->rt_ops->rtop_remove(rt, rs, rt->rt_arg);

	if (left_over && right_over) {
	range_seg_max_t newseg;
	rs_set_start(&newseg, rt, end);
	rs_set_end_raw(&newseg, rt, rs_get_end_raw(rs, rt));
	rs_set_fill(&newseg, rt, rs_get_end(rs, rt) - end);
	range_tree_stat_incr(rt, &newseg);

	// This modifies the buffer already inside the range tree
	rs_set_end(rs, rt, start);

	rs_copy(rs, &rs_tmp, rt);
	if (zfs_btree_next(&rt->rt_root, &where, &where) != NULL)
	zfs_btree_add_idx(&rt->rt_root, &newseg, &where);
	else
	zfs_btree_add(&rt->rt_root, &newseg);

	if (rt->rt_ops != NULL && rt->rt_ops->rtop_add != NULL)
	rt->rt_ops->rtop_add(rt, &newseg, rt->rt_arg);
	} else if (left_over) {
	// This modifies the buffer already inside the range tree
	rs_set_end(rs, rt, start);
	rs_copy(rs, &rs_tmp, rt);
	} else if (right_over) {
	// This modifies the buffer already inside the range tree
	rs_set_start(rs, rt, end);
	rs_copy(rs, &rs_tmp, rt);
	} else {
	zfs_btree_remove_idx(&rt->rt_root, &where);
	rs = NULL;
	}

	if (rs != NULL) {
	/*
	* The fill of the leftover segment will always be equal to
	* the size, since we do not support removing partial segments
	* of range trees with gaps.
	*/
	rs_set_fill_raw(rs, rt, rs_get_end_raw(rs, rt) -
	rs_get_start_raw(rs, rt));
	range_tree_stat_incr(rt, &rs_tmp);

	if (rt->rt_ops != NULL && rt->rt_ops->rtop_add != NULL)
	rt->rt_ops->rtop_add(rt, &rs_tmp, rt->rt_arg);
	}

	rt->rt_space -= size;
	}

	void
	range_tree_remove(void *arg, uint64_t start, uint64_t size)
	{
	range_tree_remove_impl(arg, start, size, B_FALSE);
	}

	void
	range_tree_remove_fill(range_tree_t *rt, uint64_t start, uint64_t size)
	{
	range_tree_remove_impl(rt, start, size, B_TRUE);
	}

	void
	range_tree_resize_segment(range_tree_t rt, range_seg_t rs,
	uint64_t newstart, uint64_t newsize)
	{
	int64_t delta = newsize - (rs_get_end(rs, rt) - rs_get_start(rs, rt));

	range_tree_stat_decr(rt, rs);
	if (rt->rt_ops != NULL && rt->rt_ops->rtop_remove != NULL)
	rt->rt_ops->rtop_remove(rt, rs, rt->rt_arg);

	rs_set_start(rs, rt, newstart);
	rs_set_end(rs, rt, newstart + newsize);

	range_tree_stat_incr(rt, rs);
	if (rt->rt_ops != NULL && rt->rt_ops->rtop_add != NULL)
	rt->rt_ops->rtop_add(rt, rs, rt->rt_arg);

	rt->rt_space += delta;
	}

	static range_seg_t *
	range_tree_find_impl(range_tree_t *rt, uint64_t start, uint64_t size)
	{
	range_seg_max_t rsearch;
	uint64_t end = start + size;

	VERIFY(size != 0);

	rs_set_start(&rsearch, rt, start);
	rs_set_end(&rsearch, rt, end);
	return (zfs_btree_find(&rt->rt_root, &rsearch, NULL));
	}

	range_seg_t *
	range_tree_find(range_tree_t *rt, uint64_t start, uint64_t size)
	{
	if (rt->rt_type == RANGE_SEG64)
	ASSERT3U(start + size, >, start);

	range_seg_t *rs = range_tree_find_impl(rt, start, size);
	if (rs != NULL && rs_get_start(rs, rt) <= start &&
	rs_get_end(rs, rt) >= start + size) {
	return (rs);
	}
	return (NULL);
	}

	void
	range_tree_verify_not_present(range_tree_t *rt, uint64_t off, uint64_t size)
	{
	range_seg_t *rs = range_tree_find(rt, off, size);
	if (rs != NULL)
	panic("segment already in tree; rs=%p", (void *)rs);
	}

	boolean_t
	range_tree_contains(range_tree_t *rt, uint64_t start, uint64_t size)
	{
	return (range_tree_find(rt, start, size) != NULL);
	}

	/*
	* Returns the first subset of the given range which overlaps with the range
	* tree. Returns true if there is a segment in the range, and false if there
	* isn't.
	*/
	boolean_t
	range_tree_find_in(range_tree_t *rt, uint64_t start, uint64_t size,
	uint64_t ostart, uint64_t osize)
	{
	if (rt->rt_type == RANGE_SEG64)
	ASSERT3U(start + size, >, start);

	range_seg_max_t rsearch;
	rs_set_start(&rsearch, rt, start);
	rs_set_end_raw(&rsearch, rt, rs_get_start_raw(&rsearch, rt) + 1);

	zfs_btree_index_t where;
	range_seg_t *rs = zfs_btree_find(&rt->rt_root, &rsearch, &where);
	if (rs != NULL) {
	*ostart = start;
	*osize = MIN(size, rs_get_end(rs, rt) - start);
	return (B_TRUE);
	}

	rs = zfs_btree_next(&rt->rt_root, &where, &where);
	if (rs == NULL \|\| rs_get_start(rs, rt) > start + size)
	return (B_FALSE);

	*ostart = rs_get_start(rs, rt);
	*osize = MIN(start + size, rs_get_end(rs, rt)) -
	rs_get_start(rs, rt);
	return (B_TRUE);
	}

	/*
	* Ensure that this range is not in the tree, regardless of whether
	* it is currently in the tree.
	*/
	void
	range_tree_clear(range_tree_t *rt, uint64_t start, uint64_t size)
	{
	range_seg_t *rs;

	if (size == 0)
	return;

	if (rt->rt_type == RANGE_SEG64)
	ASSERT3U(start + size, >, start);

	while ((rs = range_tree_find_impl(rt, start, size)) != NULL) {
	uint64_t free_start = MAX(rs_get_start(rs, rt), start);
	uint64_t free_end = MIN(rs_get_end(rs, rt), start + size);
	range_tree_remove(rt, free_start, free_end - free_start);
	}
	}

	void
	range_tree_swap(range_tree_t rtsrc, range_tree_t rtdst)
	{
	range_tree_t *rt;

	ASSERT0(range_tree_space(*rtdst));
	ASSERT0(zfs_btree_numnodes(&(*rtdst)->rt_root));

	rt = *rtsrc;
	rtsrc = rtdst;
	*rtdst = rt;
	}

	void
	range_tree_vacate(range_tree_t rt, range_tree_func_t func, void *arg)
	{
	if (rt->rt_ops != NULL && rt->rt_ops->rtop_vacate != NULL)
	rt->rt_ops->rtop_vacate(rt, rt->rt_arg);

	if (func != NULL) {
	range_seg_t *rs;
	zfs_btree_index_t *cookie = NULL;

	while ((rs = zfs_btree_destroy_nodes(&rt->rt_root, &cookie)) !=
	NULL) {
	func(arg, rs_get_start(rs, rt), rs_get_end(rs, rt) -
	rs_get_start(rs, rt));
	}
	} else {
	zfs_btree_clear(&rt->rt_root);
	}

	bzero(rt->rt_histogram, sizeof (rt->rt_histogram));
	rt->rt_space = 0;
	}

	void
	range_tree_walk(range_tree_t rt, range_tree_func_t func, void *arg)
	{
	zfs_btree_index_t where;
	for (range_seg_t *rs = zfs_btree_first(&rt->rt_root, &where);
	rs != NULL; rs = zfs_btree_next(&rt->rt_root, &where, &where)) {
	func(arg, rs_get_start(rs, rt), rs_get_end(rs, rt) -
	rs_get_start(rs, rt));
	}
	}

	range_seg_t *
	range_tree_first(range_tree_t *rt)
	{
	return (zfs_btree_first(&rt->rt_root, NULL));
	}

	uint64_t
	range_tree_space(range_tree_t *rt)
	{
	return (rt->rt_space);
	}

	uint64_t
	range_tree_numsegs(range_tree_t *rt)
	{
	return ((rt == NULL) ? 0 : zfs_btree_numnodes(&rt->rt_root));
	}

	boolean_t
	range_tree_is_empty(range_tree_t *rt)
	{
	ASSERT(rt != NULL);
	return (range_tree_space(rt) == 0);
	}

	/* ARGSUSED */
	void
	rt_btree_create(range_tree_t rt, void arg)
	{
	zfs_btree_t *size_tree = arg;

	size_t size;
	switch (rt->rt_type) {
	case RANGE_SEG32:
	size = sizeof (range_seg32_t);
	break;
	case RANGE_SEG64:
	size = sizeof (range_seg64_t);
	break;
	case RANGE_SEG_GAP:
	size = sizeof (range_seg_gap_t);
	break;
	default:
	panic("Invalid range seg type %d", rt->rt_type);
	}
	zfs_btree_create(size_tree, rt->rt_btree_compare, size);
	}

	/* ARGSUSED */
	void
	rt_btree_destroy(range_tree_t rt, void arg)
	{
	zfs_btree_t *size_tree = arg;
	ASSERT0(zfs_btree_numnodes(size_tree));

	zfs_btree_destroy(size_tree);
	}

	/* ARGSUSED */
	void
	rt_btree_add(range_tree_t rt, range_seg_t rs, void *arg)
	{
	zfs_btree_t *size_tree = arg;

	zfs_btree_add(size_tree, rs);
	}

	/* ARGSUSED */
	void
	rt_btree_remove(range_tree_t rt, range_seg_t rs, void *arg)
	{
	zfs_btree_t *size_tree = arg;

	zfs_btree_remove(size_tree, rs);
	}

	/* ARGSUSED */
	void
	rt_btree_vacate(range_tree_t rt, void arg)
	{
	zfs_btree_t *size_tree = arg;
	zfs_btree_clear(size_tree);
	zfs_btree_destroy(size_tree);

	rt_btree_create(rt, arg);
	}

	range_tree_ops_t rt_btree_ops = {
	.rtop_create = rt_btree_create,
	.rtop_destroy = rt_btree_destroy,
	.rtop_add = rt_btree_add,
	.rtop_remove = rt_btree_remove,
	.rtop_vacate = rt_btree_vacate
	};

	/*
	* Remove any overlapping ranges between the given segment [start, end)
	* from removefrom. Add non-overlapping leftovers to addto.
	*/
	void
	range_tree_remove_xor_add_segment(uint64_t start, uint64_t end,
	range_tree_t removefrom, range_tree_t addto)
	{
	zfs_btree_index_t where;
	range_seg_max_t starting_rs;
	rs_set_start(&starting_rs, removefrom, start);
	rs_set_end_raw(&starting_rs, removefrom, rs_get_start_raw(&starting_rs,
	removefrom) + 1);

	range_seg_t *curr = zfs_btree_find(&removefrom->rt_root,
	&starting_rs, &where);

	if (curr == NULL)
	curr = zfs_btree_next(&removefrom->rt_root, &where, &where);

	range_seg_t *next;
	for (; curr != NULL; curr = next) {
	if (start == end)
	return;
	VERIFY3U(start, <, end);

	/* there is no overlap */
	if (end <= rs_get_start(curr, removefrom)) {
	range_tree_add(addto, start, end - start);
	return;
	}

	uint64_t overlap_start = MAX(rs_get_start(curr, removefrom),
	start);
	uint64_t overlap_end = MIN(rs_get_end(curr, removefrom),
	end);
	uint64_t overlap_size = overlap_end - overlap_start;
	ASSERT3S(overlap_size, >, 0);
	range_seg_max_t rs;
	rs_copy(curr, &rs, removefrom);

	range_tree_remove(removefrom, overlap_start, overlap_size);

	if (start < overlap_start)
	range_tree_add(addto, start, overlap_start - start);

	start = overlap_end;
	next = zfs_btree_find(&removefrom->rt_root, &rs, &where);
	/*
	* If we find something here, we only removed part of the
	* curr segment. Either there's some left at the end
	* because we've reached the end of the range we're removing,
	* or there's some left at the start because we started
	* partway through the range. Either way, we continue with
	* the loop. If it's the former, we'll return at the start of
	* the loop, and if it's the latter we'll see if there is more
	* area to process.
	*/
	if (next != NULL) {
	ASSERT(start == end \|\| start == rs_get_end(&rs,
	removefrom));
	}

	next = zfs_btree_next(&removefrom->rt_root, &where, &where);
	}
	VERIFY3P(curr, ==, NULL);

	if (start != end) {
	VERIFY3U(start, <, end);
	range_tree_add(addto, start, end - start);
	} else {
	VERIFY3U(start, ==, end);
	}
	}

	/*
	* For each entry in rt, if it exists in removefrom, remove it
	* from removefrom. Otherwise, add it to addto.
	*/
	void
	range_tree_remove_xor_add(range_tree_t rt, range_tree_t removefrom,
	range_tree_t *addto)
	{
	zfs_btree_index_t where;
	for (range_seg_t *rs = zfs_btree_first(&rt->rt_root, &where); rs;
	rs = zfs_btree_next(&rt->rt_root, &where, &where)) {
	range_tree_remove_xor_add_segment(rs_get_start(rs, rt),
	rs_get_end(rs, rt), removefrom, addto);
	}
	}

	uint64_t
	range_tree_min(range_tree_t *rt)
	{
	range_seg_t *rs = zfs_btree_first(&rt->rt_root, NULL);
	return (rs != NULL ? rs_get_start(rs, rt) : 0);
	}

	uint64_t
	range_tree_max(range_tree_t *rt)
	{
	range_seg_t *rs = zfs_btree_last(&rt->rt_root, NULL);
	return (rs != NULL ? rs_get_end(rs, rt) : 0);
	}

	uint64_t
	range_tree_span(range_tree_t *rt)
	{
	return (range_tree_max(rt) - range_tree_min(rt));
	}
	Index: vendor-sys/openzfs/dist/module/zfs/spa.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/zfs/spa.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/zfs/spa.c (revision 366774)
	@@ -1,9768 +1,9768 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/

	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2011, 2020 by Delphix. All rights reserved.
	* Copyright (c) 2018, Nexenta Systems, Inc. All rights reserved.
	* Copyright (c) 2014 Spectra Logic Corporation, All rights reserved.
	* Copyright 2013 Saso Kiselkov. All rights reserved.
	* Copyright (c) 2014 Integros [integros.com]
	* Copyright 2016 Toomas Soome <tsoome@me.com>
	* Copyright (c) 2016 Actifio, Inc. All rights reserved.
	* Copyright 2018 Joyent, Inc.
	* Copyright (c) 2017, 2019, Datto Inc. All rights reserved.
	* Copyright 2017 Joyent, Inc.
	* Copyright (c) 2017, Intel Corporation.
	*/

	/*
	* SPA: Storage Pool Allocator
	*
	* This file contains all the routines used when modifying on-disk SPA state.
	* This includes opening, importing, destroying, exporting a pool, and syncing a
	* pool.
	*/

	#include <sys/zfs_context.h>
	#include <sys/fm/fs/zfs.h>
	#include <sys/spa_impl.h>
	#include <sys/zio.h>
	#include <sys/zio_checksum.h>
	#include <sys/dmu.h>
	#include <sys/dmu_tx.h>
	#include <sys/zap.h>
	#include <sys/zil.h>
	#include <sys/ddt.h>
	#include <sys/vdev_impl.h>
	#include <sys/vdev_removal.h>
	#include <sys/vdev_indirect_mapping.h>
	#include <sys/vdev_indirect_births.h>
	#include <sys/vdev_initialize.h>
	#include <sys/vdev_rebuild.h>
	#include <sys/vdev_trim.h>
	#include <sys/vdev_disk.h>
	#include <sys/metaslab.h>
	#include <sys/metaslab_impl.h>
	#include <sys/mmp.h>
	#include <sys/uberblock_impl.h>
	#include <sys/txg.h>
	#include <sys/avl.h>
	#include <sys/bpobj.h>
	#include <sys/dmu_traverse.h>
	#include <sys/dmu_objset.h>
	#include <sys/unique.h>
	#include <sys/dsl_pool.h>
	#include <sys/dsl_dataset.h>
	#include <sys/dsl_dir.h>
	#include <sys/dsl_prop.h>
	#include <sys/dsl_synctask.h>
	#include <sys/fs/zfs.h>
	#include <sys/arc.h>
	#include <sys/callb.h>
	#include <sys/systeminfo.h>
	#include <sys/spa_boot.h>
	#include <sys/zfs_ioctl.h>
	#include <sys/dsl_scan.h>
	#include <sys/zfeature.h>
	#include <sys/dsl_destroy.h>
	#include <sys/zvol.h>

	#ifdef _KERNEL
	#include <sys/fm/protocol.h>
	#include <sys/fm/util.h>
	#include <sys/callb.h>
	#include <sys/zone.h>
	#include <sys/vmsystm.h>
	#endif /* _KERNEL */

	#include "zfs_prop.h"
	#include "zfs_comutil.h"

	/*
	* The interval, in seconds, at which failed configuration cache file writes
	* should be retried.
	*/
	int zfs_ccw_retry_interval = 300;

	typedef enum zti_modes {
	ZTI_MODE_FIXED, /* value is # of threads (min 1) */
	ZTI_MODE_BATCH, /* cpu-intensive; value is ignored */
	ZTI_MODE_NULL, /* don't create a taskq */
	ZTI_NMODES
	} zti_modes_t;

	#define ZTI_P(n, q) { ZTI_MODE_FIXED, (n), (q) }
	#define ZTI_PCT(n) { ZTI_MODE_ONLINE_PERCENT, (n), 1 }
	#define ZTI_BATCH { ZTI_MODE_BATCH, 0, 1 }
	#define ZTI_NULL { ZTI_MODE_NULL, 0, 0 }

	#define ZTI_N(n) ZTI_P(n, 1)
	#define ZTI_ONE ZTI_N(1)

	typedef struct zio_taskq_info {
	zti_modes_t zti_mode;
	uint_t zti_value;
	uint_t zti_count;
	} zio_taskq_info_t;

	static const char *const zio_taskq_types[ZIO_TASKQ_TYPES] = {
	"iss", "iss_h", "int", "int_h"
	};

	/*
	* This table defines the taskq settings for each ZFS I/O type. When
	* initializing a pool, we use this table to create an appropriately sized
	* taskq. Some operations are low volume and therefore have a small, static
	* number of threads assigned to their taskqs using the ZTI_N(#) or ZTI_ONE
	* macros. Other operations process a large amount of data; the ZTI_BATCH
	* macro causes us to create a taskq oriented for throughput. Some operations
	* are so high frequency and short-lived that the taskq itself can become a
	* point of lock contention. The ZTI_P(#, #) macro indicates that we need an
	* additional degree of parallelism specified by the number of threads per-
	* taskq and the number of taskqs; when dispatching an event in this case, the
	* particular taskq is chosen at random.
	*
	* The different taskq priorities are to handle the different contexts (issue
	* and interrupt) and then to reserve threads for ZIO_PRIORITY_NOW I/Os that
	* need to be handled with minimum delay.
	*/
	const zio_taskq_info_t zio_taskqs[ZIO_TYPES][ZIO_TASKQ_TYPES] = {
	/* ISSUE ISSUE_HIGH INTR INTR_HIGH */
	{ ZTI_ONE, ZTI_NULL, ZTI_ONE, ZTI_NULL }, /* NULL */
	{ ZTI_N(8), ZTI_NULL, ZTI_P(12, 8), ZTI_NULL }, /* READ */
	{ ZTI_BATCH, ZTI_N(5), ZTI_P(12, 8), ZTI_N(5) }, /* WRITE */
	{ ZTI_P(12, 8), ZTI_NULL, ZTI_ONE, ZTI_NULL }, /* FREE */
	{ ZTI_ONE, ZTI_NULL, ZTI_ONE, ZTI_NULL }, /* CLAIM */
	{ ZTI_ONE, ZTI_NULL, ZTI_ONE, ZTI_NULL }, /* IOCTL */
	{ ZTI_N(4), ZTI_NULL, ZTI_ONE, ZTI_NULL }, /* TRIM */
	};

	static void spa_sync_version(void arg, dmu_tx_t tx);
	static void spa_sync_props(void arg, dmu_tx_t tx);
	static boolean_t spa_has_active_shared_spare(spa_t *spa);
	static int spa_load_impl(spa_t spa, spa_import_type_t type, char *ereport);
	static void spa_vdev_resilver_done(spa_t *spa);

	uint_t zio_taskq_batch_pct = 75; /* 1 thread per cpu in pset */
	boolean_t zio_taskq_sysdc = B_TRUE; /* use SDC scheduling class */
	uint_t zio_taskq_basedc = 80; /* base duty cycle */

	boolean_t spa_create_process = B_TRUE; /* no process ==> no sysdc */

	/*
	* Report any spa_load_verify errors found, but do not fail spa_load.
	* This is used by zdb to analyze non-idle pools.
	*/
	boolean_t spa_load_verify_dryrun = B_FALSE;

	/*
	* This (illegal) pool name is used when temporarily importing a spa_t in order
	* to get the vdev stats associated with the imported devices.
	*/
	#define TRYIMPORT_NAME "$import"

	/*
	* For debugging purposes: print out vdev tree during pool import.
	*/
	int spa_load_print_vdev_tree = B_FALSE;

	/*
	* A non-zero value for zfs_max_missing_tvds means that we allow importing
	* pools with missing top-level vdevs. This is strictly intended for advanced
	* pool recovery cases since missing data is almost inevitable. Pools with
	* missing devices can only be imported read-only for safety reasons, and their
	* fail-mode will be automatically set to "continue".
	*
	* With 1 missing vdev we should be able to import the pool and mount all
	* datasets. User data that was not modified after the missing device has been
	* added should be recoverable. This means that snapshots created prior to the
	* addition of that device should be completely intact.
	*
	* With 2 missing vdevs, some datasets may fail to mount since there are
	* dataset statistics that are stored as regular metadata. Some data might be
	* recoverable if those vdevs were added recently.
	*
	* With 3 or more missing vdevs, the pool is severely damaged and MOS entries
	* may be missing entirely. Chances of data recovery are very low. Note that
	* there are also risks of performing an inadvertent rewind as we might be
	* missing all the vdevs with the latest uberblocks.
	*/
	unsigned long zfs_max_missing_tvds = 0;

	/*
	* The parameters below are similar to zfs_max_missing_tvds but are only
	* intended for a preliminary open of the pool with an untrusted config which
	* might be incomplete or out-dated.
	*
	* We are more tolerant for pools opened from a cachefile since we could have
	* an out-dated cachefile where a device removal was not registered.
	* We could have set the limit arbitrarily high but in the case where devices
	* are really missing we would want to return the proper error codes; we chose
	* SPA_DVAS_PER_BP - 1 so that some copies of the MOS would still be available
	* and we get a chance to retrieve the trusted config.
	*/
	uint64_t zfs_max_missing_tvds_cachefile = SPA_DVAS_PER_BP - 1;

	/*
	* In the case where config was assembled by scanning device paths (/dev/dsks
	* by default) we are less tolerant since all the existing devices should have
	* been detected and we want spa_load to return the right error codes.
	*/
	uint64_t zfs_max_missing_tvds_scan = 0;

	/*
	* Debugging aid that pauses spa_sync() towards the end.
	*/
	boolean_t zfs_pause_spa_sync = B_FALSE;

	/*
	* Variables to indicate the livelist condense zthr func should wait at certain
	* points for the livelist to be removed - used to test condense/destroy races
	*/
	int zfs_livelist_condense_zthr_pause = 0;
	int zfs_livelist_condense_sync_pause = 0;

	/*
	* Variables to track whether or not condense cancellation has been
	* triggered in testing.
	*/
	int zfs_livelist_condense_sync_cancel = 0;
	int zfs_livelist_condense_zthr_cancel = 0;

	/*
	* Variable to track whether or not extra ALLOC blkptrs were added to a
	* livelist entry while it was being condensed (caused by the way we track
	* remapped blkptrs in dbuf_remap_impl)
	*/
	int zfs_livelist_condense_new_alloc = 0;

	/*
	* ==========================================================================
	* SPA properties routines
	* ==========================================================================
	*/

	/*
	* Add a (source=src, propname=propval) list to an nvlist.
	*/
	static void
	spa_prop_add_list(nvlist_t nvl, zpool_prop_t prop, char strval,
	uint64_t intval, zprop_source_t src)
	{
	const char *propname = zpool_prop_to_name(prop);
	nvlist_t *propval;

	VERIFY(nvlist_alloc(&propval, NV_UNIQUE_NAME, KM_SLEEP) == 0);
	VERIFY(nvlist_add_uint64(propval, ZPROP_SOURCE, src) == 0);

	if (strval != NULL)
	VERIFY(nvlist_add_string(propval, ZPROP_VALUE, strval) == 0);
	else
	VERIFY(nvlist_add_uint64(propval, ZPROP_VALUE, intval) == 0);

	VERIFY(nvlist_add_nvlist(nvl, propname, propval) == 0);
	nvlist_free(propval);
	}

	/*
	* Get property values from the spa configuration.
	*/
	static void
	spa_prop_get_config(spa_t spa, nvlist_t *nvp)
	{
	vdev_t *rvd = spa->spa_root_vdev;
	dsl_pool_t *pool = spa->spa_dsl_pool;
	uint64_t size, alloc, cap, version;
	const zprop_source_t src = ZPROP_SRC_NONE;
	spa_config_dirent_t *dp;
	metaslab_class_t *mc = spa_normal_class(spa);

	ASSERT(MUTEX_HELD(&spa->spa_props_lock));

	if (rvd != NULL) {
	alloc = metaslab_class_get_alloc(mc);
	alloc += metaslab_class_get_alloc(spa_special_class(spa));
	alloc += metaslab_class_get_alloc(spa_dedup_class(spa));

	size = metaslab_class_get_space(mc);
	size += metaslab_class_get_space(spa_special_class(spa));
	size += metaslab_class_get_space(spa_dedup_class(spa));

	spa_prop_add_list(*nvp, ZPOOL_PROP_NAME, spa_name(spa), 0, src);
	spa_prop_add_list(*nvp, ZPOOL_PROP_SIZE, NULL, size, src);
	spa_prop_add_list(*nvp, ZPOOL_PROP_ALLOCATED, NULL, alloc, src);
	spa_prop_add_list(*nvp, ZPOOL_PROP_FREE, NULL,
	size - alloc, src);
	spa_prop_add_list(*nvp, ZPOOL_PROP_CHECKPOINT, NULL,
	spa->spa_checkpoint_info.sci_dspace, src);

	spa_prop_add_list(*nvp, ZPOOL_PROP_FRAGMENTATION, NULL,
	metaslab_class_fragmentation(mc), src);
	spa_prop_add_list(*nvp, ZPOOL_PROP_EXPANDSZ, NULL,
	metaslab_class_expandable_space(mc), src);
	spa_prop_add_list(*nvp, ZPOOL_PROP_READONLY, NULL,
	(spa_mode(spa) == SPA_MODE_READ), src);

	cap = (size == 0) ? 0 : (alloc * 100 / size);
	spa_prop_add_list(*nvp, ZPOOL_PROP_CAPACITY, NULL, cap, src);

	spa_prop_add_list(*nvp, ZPOOL_PROP_DEDUPRATIO, NULL,
	ddt_get_pool_dedup_ratio(spa), src);

	spa_prop_add_list(*nvp, ZPOOL_PROP_HEALTH, NULL,
	rvd->vdev_state, src);

	version = spa_version(spa);
	if (version == zpool_prop_default_numeric(ZPOOL_PROP_VERSION)) {
	spa_prop_add_list(*nvp, ZPOOL_PROP_VERSION, NULL,
	version, ZPROP_SRC_DEFAULT);
	} else {
	spa_prop_add_list(*nvp, ZPOOL_PROP_VERSION, NULL,
	version, ZPROP_SRC_LOCAL);
	}
	spa_prop_add_list(*nvp, ZPOOL_PROP_LOAD_GUID,
	NULL, spa_load_guid(spa), src);
	}

	if (pool != NULL) {
	/*
	* The $FREE directory was introduced in SPA_VERSION_DEADLISTS,
	* when opening pools before this version freedir will be NULL.
	*/
	if (pool->dp_free_dir != NULL) {
	spa_prop_add_list(*nvp, ZPOOL_PROP_FREEING, NULL,
	dsl_dir_phys(pool->dp_free_dir)->dd_used_bytes,
	src);
	} else {
	spa_prop_add_list(*nvp, ZPOOL_PROP_FREEING,
	NULL, 0, src);
	}

	if (pool->dp_leak_dir != NULL) {
	spa_prop_add_list(*nvp, ZPOOL_PROP_LEAKED, NULL,
	dsl_dir_phys(pool->dp_leak_dir)->dd_used_bytes,
	src);
	} else {
	spa_prop_add_list(*nvp, ZPOOL_PROP_LEAKED,
	NULL, 0, src);
	}
	}

	spa_prop_add_list(*nvp, ZPOOL_PROP_GUID, NULL, spa_guid(spa), src);

	if (spa->spa_comment != NULL) {
	spa_prop_add_list(*nvp, ZPOOL_PROP_COMMENT, spa->spa_comment,
	0, ZPROP_SRC_LOCAL);
	}

	if (spa->spa_root != NULL)
	spa_prop_add_list(*nvp, ZPOOL_PROP_ALTROOT, spa->spa_root,
	0, ZPROP_SRC_LOCAL);

	if (spa_feature_is_enabled(spa, SPA_FEATURE_LARGE_BLOCKS)) {
	spa_prop_add_list(*nvp, ZPOOL_PROP_MAXBLOCKSIZE, NULL,
	MIN(zfs_max_recordsize, SPA_MAXBLOCKSIZE), ZPROP_SRC_NONE);
	} else {
	spa_prop_add_list(*nvp, ZPOOL_PROP_MAXBLOCKSIZE, NULL,
	SPA_OLD_MAXBLOCKSIZE, ZPROP_SRC_NONE);
	}

	if (spa_feature_is_enabled(spa, SPA_FEATURE_LARGE_DNODE)) {
	spa_prop_add_list(*nvp, ZPOOL_PROP_MAXDNODESIZE, NULL,
	DNODE_MAX_SIZE, ZPROP_SRC_NONE);
	} else {
	spa_prop_add_list(*nvp, ZPOOL_PROP_MAXDNODESIZE, NULL,
	DNODE_MIN_SIZE, ZPROP_SRC_NONE);
	}

	if ((dp = list_head(&spa->spa_config_list)) != NULL) {
	if (dp->scd_path == NULL) {
	spa_prop_add_list(*nvp, ZPOOL_PROP_CACHEFILE,
	"none", 0, ZPROP_SRC_LOCAL);
	} else if (strcmp(dp->scd_path, spa_config_path) != 0) {
	spa_prop_add_list(*nvp, ZPOOL_PROP_CACHEFILE,
	dp->scd_path, 0, ZPROP_SRC_LOCAL);
	}
	}
	}

	/*
	* Get zpool property values.
	*/
	int
	spa_prop_get(spa_t spa, nvlist_t *nvp)
	{
	objset_t *mos = spa->spa_meta_objset;
	zap_cursor_t zc;
	zap_attribute_t za;
	dsl_pool_t *dp;
	int err;

	err = nvlist_alloc(nvp, NV_UNIQUE_NAME, KM_SLEEP);
	if (err)
	return (err);

	dp = spa_get_dsl(spa);
	dsl_pool_config_enter(dp, FTAG);
	mutex_enter(&spa->spa_props_lock);

	/*
	* Get properties from the spa config.
	*/
	spa_prop_get_config(spa, nvp);

	/* If no pool property object, no more prop to get. */
	if (mos == NULL \|\| spa->spa_pool_props_object == 0)
	goto out;

	/*
	* Get properties from the MOS pool property object.
	*/
	for (zap_cursor_init(&zc, mos, spa->spa_pool_props_object);
	(err = zap_cursor_retrieve(&zc, &za)) == 0;
	zap_cursor_advance(&zc)) {
	uint64_t intval = 0;
	char *strval = NULL;
	zprop_source_t src = ZPROP_SRC_DEFAULT;
	zpool_prop_t prop;

	if ((prop = zpool_name_to_prop(za.za_name)) == ZPOOL_PROP_INVAL)
	continue;

	switch (za.za_integer_length) {
	case 8:
	/* integer property */
	if (za.za_first_integer !=
	zpool_prop_default_numeric(prop))
	src = ZPROP_SRC_LOCAL;

	if (prop == ZPOOL_PROP_BOOTFS) {
	dsl_dataset_t *ds = NULL;

	err = dsl_dataset_hold_obj(dp,
	za.za_first_integer, FTAG, &ds);
	if (err != 0)
	break;

	strval = kmem_alloc(ZFS_MAX_DATASET_NAME_LEN,
	KM_SLEEP);
	dsl_dataset_name(ds, strval);
	dsl_dataset_rele(ds, FTAG);
	} else {
	strval = NULL;
	intval = za.za_first_integer;
	}

	spa_prop_add_list(*nvp, prop, strval, intval, src);

	if (strval != NULL)
	kmem_free(strval, ZFS_MAX_DATASET_NAME_LEN);

	break;

	case 1:
	/* string property */
	strval = kmem_alloc(za.za_num_integers, KM_SLEEP);
	err = zap_lookup(mos, spa->spa_pool_props_object,
	za.za_name, 1, za.za_num_integers, strval);
	if (err) {
	kmem_free(strval, za.za_num_integers);
	break;
	}
	spa_prop_add_list(*nvp, prop, strval, 0, src);
	kmem_free(strval, za.za_num_integers);
	break;

	default:
	break;
	}
	}
	zap_cursor_fini(&zc);
	out:
	mutex_exit(&spa->spa_props_lock);
	dsl_pool_config_exit(dp, FTAG);
	if (err && err != ENOENT) {
	nvlist_free(*nvp);
	*nvp = NULL;
	return (err);
	}

	return (0);
	}

	/*
	* Validate the given pool properties nvlist and modify the list
	* for the property values to be set.
	*/
	static int
	spa_prop_validate(spa_t spa, nvlist_t props)
	{
	nvpair_t *elem;
	int error = 0, reset_bootfs = 0;
	uint64_t objnum = 0;
	boolean_t has_feature = B_FALSE;

	elem = NULL;
	while ((elem = nvlist_next_nvpair(props, elem)) != NULL) {
	uint64_t intval;
	char strval, slash, check, fname;
	const char *propname = nvpair_name(elem);
	zpool_prop_t prop = zpool_name_to_prop(propname);

	switch (prop) {
	case ZPOOL_PROP_INVAL:
	if (!zpool_prop_feature(propname)) {
	error = SET_ERROR(EINVAL);
	break;
	}

	/*
	* Sanitize the input.
	*/
	if (nvpair_type(elem) != DATA_TYPE_UINT64) {
	error = SET_ERROR(EINVAL);
	break;
	}

	if (nvpair_value_uint64(elem, &intval) != 0) {
	error = SET_ERROR(EINVAL);
	break;
	}

	if (intval != 0) {
	error = SET_ERROR(EINVAL);
	break;
	}

	fname = strchr(propname, '@') + 1;
	if (zfeature_lookup_name(fname, NULL) != 0) {
	error = SET_ERROR(EINVAL);
	break;
	}

	has_feature = B_TRUE;
	break;

	case ZPOOL_PROP_VERSION:
	error = nvpair_value_uint64(elem, &intval);
	if (!error &&
	(intval < spa_version(spa) \|\|
	intval > SPA_VERSION_BEFORE_FEATURES \|\|
	has_feature))
	error = SET_ERROR(EINVAL);
	break;

	case ZPOOL_PROP_DELEGATION:
	case ZPOOL_PROP_AUTOREPLACE:
	case ZPOOL_PROP_LISTSNAPS:
	case ZPOOL_PROP_AUTOEXPAND:
	case ZPOOL_PROP_AUTOTRIM:
	error = nvpair_value_uint64(elem, &intval);
	if (!error && intval > 1)
	error = SET_ERROR(EINVAL);
	break;

	case ZPOOL_PROP_MULTIHOST:
	error = nvpair_value_uint64(elem, &intval);
	if (!error && intval > 1)
	error = SET_ERROR(EINVAL);

	if (!error) {
	uint32_t hostid = zone_get_hostid(NULL);
	if (hostid)
	spa->spa_hostid = hostid;
	else
	error = SET_ERROR(ENOTSUP);
	}

	break;

	case ZPOOL_PROP_BOOTFS:
	/*
	* If the pool version is less than SPA_VERSION_BOOTFS,
	* or the pool is still being created (version == 0),
	* the bootfs property cannot be set.
	*/
	if (spa_version(spa) < SPA_VERSION_BOOTFS) {
	error = SET_ERROR(ENOTSUP);
	break;
	}

	/*
	* Make sure the vdev config is bootable
	*/
	if (!vdev_is_bootable(spa->spa_root_vdev)) {
	error = SET_ERROR(ENOTSUP);
	break;
	}

	reset_bootfs = 1;

	error = nvpair_value_string(elem, &strval);

	if (!error) {
	objset_t *os;

	if (strval == NULL \|\| strval[0] == '\0') {
	objnum = zpool_prop_default_numeric(
	ZPOOL_PROP_BOOTFS);
	break;
	}

	error = dmu_objset_hold(strval, FTAG, &os);
	if (error != 0)
	break;

	/* Must be ZPL. */
	if (dmu_objset_type(os) != DMU_OST_ZFS) {
	error = SET_ERROR(ENOTSUP);
	} else {
	objnum = dmu_objset_id(os);
	}
	dmu_objset_rele(os, FTAG);
	}
	break;

	case ZPOOL_PROP_FAILUREMODE:
	error = nvpair_value_uint64(elem, &intval);
	if (!error && intval > ZIO_FAILURE_MODE_PANIC)
	error = SET_ERROR(EINVAL);

	/*
	* This is a special case which only occurs when
	* the pool has completely failed. This allows
	* the user to change the in-core failmode property
	* without syncing it out to disk (I/Os might
	* currently be blocked). We do this by returning
	* EIO to the caller (spa_prop_set) to trick it
	* into thinking we encountered a property validation
	* error.
	*/
	if (!error && spa_suspended(spa)) {
	spa->spa_failmode = intval;
	error = SET_ERROR(EIO);
	}
	break;

	case ZPOOL_PROP_CACHEFILE:
	if ((error = nvpair_value_string(elem, &strval)) != 0)
	break;

	if (strval[0] == '\0')
	break;

	if (strcmp(strval, "none") == 0)
	break;

	if (strval[0] != '/') {
	error = SET_ERROR(EINVAL);
	break;
	}

	slash = strrchr(strval, '/');
	ASSERT(slash != NULL);

	if (slash[1] == '\0' \|\| strcmp(slash, "/.") == 0 \|\|
	strcmp(slash, "/..") == 0)
	error = SET_ERROR(EINVAL);
	break;

	case ZPOOL_PROP_COMMENT:
	if ((error = nvpair_value_string(elem, &strval)) != 0)
	break;
	for (check = strval; *check != '\0'; check++) {
	if (!isprint(*check)) {
	error = SET_ERROR(EINVAL);
	break;
	}
	}
	if (strlen(strval) > ZPROP_MAX_COMMENT)
	error = SET_ERROR(E2BIG);
	break;

	default:
	break;
	}

	if (error)
	break;
	}

	(void) nvlist_remove_all(props,
	zpool_prop_to_name(ZPOOL_PROP_DEDUPDITTO));

	if (!error && reset_bootfs) {
	error = nvlist_remove(props,
	zpool_prop_to_name(ZPOOL_PROP_BOOTFS), DATA_TYPE_STRING);

	if (!error) {
	error = nvlist_add_uint64(props,
	zpool_prop_to_name(ZPOOL_PROP_BOOTFS), objnum);
	}
	}

	return (error);
	}

	void
	spa_configfile_set(spa_t spa, nvlist_t nvp, boolean_t need_sync)
	{
	char *cachefile;
	spa_config_dirent_t *dp;

	if (nvlist_lookup_string(nvp, zpool_prop_to_name(ZPOOL_PROP_CACHEFILE),
	&cachefile) != 0)
	return;

	dp = kmem_alloc(sizeof (spa_config_dirent_t),
	KM_SLEEP);

	if (cachefile[0] == '\0')
	dp->scd_path = spa_strdup(spa_config_path);
	else if (strcmp(cachefile, "none") == 0)
	dp->scd_path = NULL;
	else
	dp->scd_path = spa_strdup(cachefile);

	list_insert_head(&spa->spa_config_list, dp);
	if (need_sync)
	spa_async_request(spa, SPA_ASYNC_CONFIG_UPDATE);
	}

	int
	spa_prop_set(spa_t spa, nvlist_t nvp)
	{
	int error;
	nvpair_t *elem = NULL;
	boolean_t need_sync = B_FALSE;

	if ((error = spa_prop_validate(spa, nvp)) != 0)
	return (error);

	while ((elem = nvlist_next_nvpair(nvp, elem)) != NULL) {
	zpool_prop_t prop = zpool_name_to_prop(nvpair_name(elem));

	if (prop == ZPOOL_PROP_CACHEFILE \|\|
	prop == ZPOOL_PROP_ALTROOT \|\|
	prop == ZPOOL_PROP_READONLY)
	continue;

	if (prop == ZPOOL_PROP_VERSION \|\| prop == ZPOOL_PROP_INVAL) {
	uint64_t ver;

	if (prop == ZPOOL_PROP_VERSION) {
	VERIFY(nvpair_value_uint64(elem, &ver) == 0);
	} else {
	ASSERT(zpool_prop_feature(nvpair_name(elem)));
	ver = SPA_VERSION_FEATURES;
	need_sync = B_TRUE;
	}

	/* Save time if the version is already set. */
	if (ver == spa_version(spa))
	continue;

	/*
	* In addition to the pool directory object, we might
	* create the pool properties object, the features for
	* read object, the features for write object, or the
	* feature descriptions object.
	*/
	error = dsl_sync_task(spa->spa_name, NULL,
	spa_sync_version, &ver,
	6, ZFS_SPACE_CHECK_RESERVED);
	if (error)
	return (error);
	continue;
	}

	need_sync = B_TRUE;
	break;
	}

	if (need_sync) {
	return (dsl_sync_task(spa->spa_name, NULL, spa_sync_props,
	nvp, 6, ZFS_SPACE_CHECK_RESERVED));
	}

	return (0);
	}

	/*
	* If the bootfs property value is dsobj, clear it.
	*/
	void
	spa_prop_clear_bootfs(spa_t spa, uint64_t dsobj, dmu_tx_t tx)
	{
	if (spa->spa_bootfs == dsobj && spa->spa_pool_props_object != 0) {
	VERIFY(zap_remove(spa->spa_meta_objset,
	spa->spa_pool_props_object,
	zpool_prop_to_name(ZPOOL_PROP_BOOTFS), tx) == 0);
	spa->spa_bootfs = 0;
	}
	}

	/ARGSUSED/
	static int
	spa_change_guid_check(void arg, dmu_tx_t tx)
	{
	uint64_t *newguid __maybe_unused = arg;
	spa_t *spa = dmu_tx_pool(tx)->dp_spa;
	vdev_t *rvd = spa->spa_root_vdev;
	uint64_t vdev_state;

	if (spa_feature_is_active(spa, SPA_FEATURE_POOL_CHECKPOINT)) {
	int error = (spa_has_checkpoint(spa)) ?
	ZFS_ERR_CHECKPOINT_EXISTS : ZFS_ERR_DISCARDING_CHECKPOINT;
	return (SET_ERROR(error));
	}

	spa_config_enter(spa, SCL_STATE, FTAG, RW_READER);
	vdev_state = rvd->vdev_state;
	spa_config_exit(spa, SCL_STATE, FTAG);

	if (vdev_state != VDEV_STATE_HEALTHY)
	return (SET_ERROR(ENXIO));

	ASSERT3U(spa_guid(spa), !=, *newguid);

	return (0);
	}

	static void
	spa_change_guid_sync(void arg, dmu_tx_t tx)
	{
	uint64_t *newguid = arg;
	spa_t *spa = dmu_tx_pool(tx)->dp_spa;
	uint64_t oldguid;
	vdev_t *rvd = spa->spa_root_vdev;

	oldguid = spa_guid(spa);

	spa_config_enter(spa, SCL_STATE, FTAG, RW_READER);
	rvd->vdev_guid = *newguid;
	rvd->vdev_guid_sum += (*newguid - oldguid);
	vdev_config_dirty(rvd);
	spa_config_exit(spa, SCL_STATE, FTAG);

	spa_history_log_internal(spa, "guid change", tx, "old=%llu new=%llu",
	(u_longlong_t)oldguid, (u_longlong_t)*newguid);
	}

	/*
	* Change the GUID for the pool. This is done so that we can later
	* re-import a pool built from a clone of our own vdevs. We will modify
	* the root vdev's guid, our own pool guid, and then mark all of our
	* vdevs dirty. Note that we must make sure that all our vdevs are
	* online when we do this, or else any vdevs that weren't present
	* would be orphaned from our pool. We are also going to issue a
	* sysevent to update any watchers.
	*/
	int
	spa_change_guid(spa_t *spa)
	{
	int error;
	uint64_t guid;

	mutex_enter(&spa->spa_vdev_top_lock);
	mutex_enter(&spa_namespace_lock);
	guid = spa_generate_guid(NULL);

	error = dsl_sync_task(spa->spa_name, spa_change_guid_check,
	spa_change_guid_sync, &guid, 5, ZFS_SPACE_CHECK_RESERVED);

	if (error == 0) {
	spa_write_cachefile(spa, B_FALSE, B_TRUE);
	spa_event_notify(spa, NULL, NULL, ESC_ZFS_POOL_REGUID);
	}

	mutex_exit(&spa_namespace_lock);
	mutex_exit(&spa->spa_vdev_top_lock);

	return (error);
	}

	/*
	* ==========================================================================
	* SPA state manipulation (open/create/destroy/import/export)
	* ==========================================================================
	*/

	static int
	spa_error_entry_compare(const void a, const void b)
	{
	const spa_error_entry_t sa = (const spa_error_entry_t )a;
	const spa_error_entry_t sb = (const spa_error_entry_t )b;
	int ret;

	ret = memcmp(&sa->se_bookmark, &sb->se_bookmark,
	sizeof (zbookmark_phys_t));

	return (TREE_ISIGN(ret));
	}

	/*
	* Utility function which retrieves copies of the current logs and
	* re-initializes them in the process.
	*/
	void
	spa_get_errlists(spa_t spa, avl_tree_t last, avl_tree_t *scrub)
	{
	ASSERT(MUTEX_HELD(&spa->spa_errlist_lock));

	bcopy(&spa->spa_errlist_last, last, sizeof (avl_tree_t));
	bcopy(&spa->spa_errlist_scrub, scrub, sizeof (avl_tree_t));

	avl_create(&spa->spa_errlist_scrub,
	spa_error_entry_compare, sizeof (spa_error_entry_t),
	offsetof(spa_error_entry_t, se_avl));
	avl_create(&spa->spa_errlist_last,
	spa_error_entry_compare, sizeof (spa_error_entry_t),
	offsetof(spa_error_entry_t, se_avl));
	}

	static void
	spa_taskqs_init(spa_t *spa, zio_type_t t, zio_taskq_type_t q)
	{
	const zio_taskq_info_t *ztip = &zio_taskqs[t][q];
	enum zti_modes mode = ztip->zti_mode;
	uint_t value = ztip->zti_value;
	uint_t count = ztip->zti_count;
	spa_taskqs_t *tqs = &spa->spa_zio_taskq[t][q];
	uint_t flags = 0;
	boolean_t batch = B_FALSE;

	if (mode == ZTI_MODE_NULL) {
	tqs->stqs_count = 0;
	tqs->stqs_taskq = NULL;
	return;
	}

	ASSERT3U(count, >, 0);

	tqs->stqs_count = count;
	tqs->stqs_taskq = kmem_alloc(count * sizeof (taskq_t *), KM_SLEEP);

	switch (mode) {
	case ZTI_MODE_FIXED:
	ASSERT3U(value, >=, 1);
	value = MAX(value, 1);
	flags \|= TASKQ_DYNAMIC;
	break;

	case ZTI_MODE_BATCH:
	batch = B_TRUE;
	flags \|= TASKQ_THREADS_CPU_PCT;
	value = MIN(zio_taskq_batch_pct, 100);
	break;

	default:
	panic("unrecognized mode for %s_%s taskq (%u:%u) in "
	"spa_activate()",
	zio_type_name[t], zio_taskq_types[q], mode, value);
	break;
	}

	for (uint_t i = 0; i < count; i++) {
	taskq_t *tq;
	char name[32];

	(void) snprintf(name, sizeof (name), "%s_%s",
	zio_type_name[t], zio_taskq_types[q]);

	if (zio_taskq_sysdc && spa->spa_proc != &p0) {
	if (batch)
	flags \|= TASKQ_DC_BATCH;

	tq = taskq_create_sysdc(name, value, 50, INT_MAX,
	spa->spa_proc, zio_taskq_basedc, flags);
	} else {
	pri_t pri = maxclsyspri;
	/*
	* The write issue taskq can be extremely CPU
	* intensive. Run it at slightly less important
	* priority than the other taskqs.
	*
	* Under Linux and FreeBSD this means incrementing
	* the priority value as opposed to platforms like
	* illumos where it should be decremented.
	*
	* On FreeBSD, if priorities divided by four (RQ_PPQ)
	* are equal then a difference between them is
	* insignificant.
	*/
	if (t == ZIO_TYPE_WRITE && q == ZIO_TASKQ_ISSUE) {
	#if defined(__linux__)
	pri++;
	#elif defined(__FreeBSD__)
	pri += 4;
	#else
	#error "unknown OS"
	#endif
	}
	tq = taskq_create_proc(name, value, pri, 50,
	INT_MAX, spa->spa_proc, flags);
	}

	tqs->stqs_taskq[i] = tq;
	}
	}

	static void
	spa_taskqs_fini(spa_t *spa, zio_type_t t, zio_taskq_type_t q)
	{
	spa_taskqs_t *tqs = &spa->spa_zio_taskq[t][q];

	if (tqs->stqs_taskq == NULL) {
	ASSERT3U(tqs->stqs_count, ==, 0);
	return;
	}

	for (uint_t i = 0; i < tqs->stqs_count; i++) {
	ASSERT3P(tqs->stqs_taskq[i], !=, NULL);
	taskq_destroy(tqs->stqs_taskq[i]);
	}

	kmem_free(tqs->stqs_taskq, tqs->stqs_count * sizeof (taskq_t *));
	tqs->stqs_taskq = NULL;
	}

	/*
	* Dispatch a task to the appropriate taskq for the ZFS I/O type and priority.
	* Note that a type may have multiple discrete taskqs to avoid lock contention
	* on the taskq itself. In that case we choose which taskq at random by using
	* the low bits of gethrtime().
	*/
	void
	spa_taskq_dispatch_ent(spa_t *spa, zio_type_t t, zio_taskq_type_t q,
	task_func_t func, void arg, uint_t flags, taskq_ent_t *ent)
	{
	spa_taskqs_t *tqs = &spa->spa_zio_taskq[t][q];
	taskq_t *tq;

	ASSERT3P(tqs->stqs_taskq, !=, NULL);
	ASSERT3U(tqs->stqs_count, !=, 0);

	if (tqs->stqs_count == 1) {
	tq = tqs->stqs_taskq[0];
	} else {
	tq = tqs->stqs_taskq[((uint64_t)gethrtime()) % tqs->stqs_count];
	}

	taskq_dispatch_ent(tq, func, arg, flags, ent);
	}

	/*
	* Same as spa_taskq_dispatch_ent() but block on the task until completion.
	*/
	void
	spa_taskq_dispatch_sync(spa_t *spa, zio_type_t t, zio_taskq_type_t q,
	task_func_t func, void arg, uint_t flags)
	{
	spa_taskqs_t *tqs = &spa->spa_zio_taskq[t][q];
	taskq_t *tq;
	taskqid_t id;

	ASSERT3P(tqs->stqs_taskq, !=, NULL);
	ASSERT3U(tqs->stqs_count, !=, 0);

	if (tqs->stqs_count == 1) {
	tq = tqs->stqs_taskq[0];
	} else {
	tq = tqs->stqs_taskq[((uint64_t)gethrtime()) % tqs->stqs_count];
	}

	id = taskq_dispatch(tq, func, arg, flags);
	if (id)
	taskq_wait_id(tq, id);
	}

	static void
	spa_create_zio_taskqs(spa_t *spa)
	{
	for (int t = 0; t < ZIO_TYPES; t++) {
	for (int q = 0; q < ZIO_TASKQ_TYPES; q++) {
	spa_taskqs_init(spa, t, q);
	}
	}
	}

	/*
	* Disabled until spa_thread() can be adapted for Linux.
	*/
	#undef HAVE_SPA_THREAD

	#if defined(_KERNEL) && defined(HAVE_SPA_THREAD)
	static void
	spa_thread(void *arg)
	{
	psetid_t zio_taskq_psrset_bind = PS_NONE;
	callb_cpr_t cprinfo;

	spa_t *spa = arg;
	user_t *pu = PTOU(curproc);

	CALLB_CPR_INIT(&cprinfo, &spa->spa_proc_lock, callb_generic_cpr,
	spa->spa_name);

	ASSERT(curproc != &p0);
	(void) snprintf(pu->u_psargs, sizeof (pu->u_psargs),
	"zpool-%s", spa->spa_name);
	(void) strlcpy(pu->u_comm, pu->u_psargs, sizeof (pu->u_comm));

	/* bind this thread to the requested psrset */
	if (zio_taskq_psrset_bind != PS_NONE) {
	pool_lock();
	mutex_enter(&cpu_lock);
	mutex_enter(&pidlock);
	mutex_enter(&curproc->p_lock);

	if (cpupart_bind_thread(curthread, zio_taskq_psrset_bind,
	0, NULL, NULL) == 0) {
	curthread->t_bind_pset = zio_taskq_psrset_bind;
	} else {
	cmn_err(CE_WARN,
	"Couldn't bind process for zfs pool \"%s\" to "
	"pset %d\n", spa->spa_name, zio_taskq_psrset_bind);
	}

	mutex_exit(&curproc->p_lock);
	mutex_exit(&pidlock);
	mutex_exit(&cpu_lock);
	pool_unlock();
	}

	if (zio_taskq_sysdc) {
	sysdc_thread_enter(curthread, 100, 0);
	}

	spa->spa_proc = curproc;
	spa->spa_did = curthread->t_did;

	spa_create_zio_taskqs(spa);

	mutex_enter(&spa->spa_proc_lock);
	ASSERT(spa->spa_proc_state == SPA_PROC_CREATED);

	spa->spa_proc_state = SPA_PROC_ACTIVE;
	cv_broadcast(&spa->spa_proc_cv);

	CALLB_CPR_SAFE_BEGIN(&cprinfo);
	while (spa->spa_proc_state == SPA_PROC_ACTIVE)
	cv_wait(&spa->spa_proc_cv, &spa->spa_proc_lock);
	CALLB_CPR_SAFE_END(&cprinfo, &spa->spa_proc_lock);

	ASSERT(spa->spa_proc_state == SPA_PROC_DEACTIVATE);
	spa->spa_proc_state = SPA_PROC_GONE;
	spa->spa_proc = &p0;
	cv_broadcast(&spa->spa_proc_cv);
	CALLB_CPR_EXIT(&cprinfo); /* drops spa_proc_lock */

	mutex_enter(&curproc->p_lock);
	lwp_exit();
	}
	#endif

	/*
	* Activate an uninitialized pool.
	*/
	static void
	spa_activate(spa_t *spa, spa_mode_t mode)
	{
	ASSERT(spa->spa_state == POOL_STATE_UNINITIALIZED);

	spa->spa_state = POOL_STATE_ACTIVE;
	spa->spa_mode = mode;

	spa->spa_normal_class = metaslab_class_create(spa, zfs_metaslab_ops);
	spa->spa_log_class = metaslab_class_create(spa, zfs_metaslab_ops);
	spa->spa_special_class = metaslab_class_create(spa, zfs_metaslab_ops);
	spa->spa_dedup_class = metaslab_class_create(spa, zfs_metaslab_ops);

	/* Try to create a covering process */
	mutex_enter(&spa->spa_proc_lock);
	ASSERT(spa->spa_proc_state == SPA_PROC_NONE);
	ASSERT(spa->spa_proc == &p0);
	spa->spa_did = 0;

	#ifdef HAVE_SPA_THREAD
	/* Only create a process if we're going to be around a while. */
	if (spa_create_process && strcmp(spa->spa_name, TRYIMPORT_NAME) != 0) {
	if (newproc(spa_thread, (caddr_t)spa, syscid, maxclsyspri,
	NULL, 0) == 0) {
	spa->spa_proc_state = SPA_PROC_CREATED;
	while (spa->spa_proc_state == SPA_PROC_CREATED) {
	cv_wait(&spa->spa_proc_cv,
	&spa->spa_proc_lock);
	}
	ASSERT(spa->spa_proc_state == SPA_PROC_ACTIVE);
	ASSERT(spa->spa_proc != &p0);
	ASSERT(spa->spa_did != 0);
	} else {
	#ifdef _KERNEL
	cmn_err(CE_WARN,
	"Couldn't create process for zfs pool \"%s\"\n",
	spa->spa_name);
	#endif
	}
	}
	#endif /* HAVE_SPA_THREAD */
	mutex_exit(&spa->spa_proc_lock);

	/* If we didn't create a process, we need to create our taskqs. */
	if (spa->spa_proc == &p0) {
	spa_create_zio_taskqs(spa);
	}

	for (size_t i = 0; i < TXG_SIZE; i++) {
	spa->spa_txg_zio[i] = zio_root(spa, NULL, NULL,
	ZIO_FLAG_CANFAIL);
	}

	list_create(&spa->spa_config_dirty_list, sizeof (vdev_t),
	offsetof(vdev_t, vdev_config_dirty_node));
	list_create(&spa->spa_evicting_os_list, sizeof (objset_t),
	offsetof(objset_t, os_evicting_node));
	list_create(&spa->spa_state_dirty_list, sizeof (vdev_t),
	offsetof(vdev_t, vdev_state_dirty_node));

	txg_list_create(&spa->spa_vdev_txg_list, spa,
	offsetof(struct vdev, vdev_txg_node));

	avl_create(&spa->spa_errlist_scrub,
	spa_error_entry_compare, sizeof (spa_error_entry_t),
	offsetof(spa_error_entry_t, se_avl));
	avl_create(&spa->spa_errlist_last,
	spa_error_entry_compare, sizeof (spa_error_entry_t),
	offsetof(spa_error_entry_t, se_avl));

	spa_keystore_init(&spa->spa_keystore);

	/*
	* This taskq is used to perform zvol-minor-related tasks
	* asynchronously. This has several advantages, including easy
	- * resolution of various deadlocks (zfsonlinux bug #3681).
	+ * resolution of various deadlocks.
	*
	* The taskq must be single threaded to ensure tasks are always
	* processed in the order in which they were dispatched.
	*
	* A taskq per pool allows one to keep the pools independent.
	* This way if one pool is suspended, it will not impact another.
	*
	* The preferred location to dispatch a zvol minor task is a sync
	* task. In this context, there is easy access to the spa_t and minimal
	* error handling is required because the sync task must succeed.
	*/
	spa->spa_zvol_taskq = taskq_create("z_zvol", 1, defclsyspri,
	1, INT_MAX, 0);

	/*
	* Taskq dedicated to prefetcher threads: this is used to prevent the
	* pool traverse code from monopolizing the global (and limited)
	* system_taskq by inappropriately scheduling long running tasks on it.
	*/
	spa->spa_prefetch_taskq = taskq_create("z_prefetch", boot_ncpus,
	defclsyspri, 1, INT_MAX, TASKQ_DYNAMIC);

	/*
	* The taskq to upgrade datasets in this pool. Currently used by
	* feature SPA_FEATURE_USEROBJ_ACCOUNTING/SPA_FEATURE_PROJECT_QUOTA.
	*/
	spa->spa_upgrade_taskq = taskq_create("z_upgrade", boot_ncpus,
	defclsyspri, 1, INT_MAX, TASKQ_DYNAMIC);
	}

	/*
	* Opposite of spa_activate().
	*/
	static void
	spa_deactivate(spa_t *spa)
	{
	ASSERT(spa->spa_sync_on == B_FALSE);
	ASSERT(spa->spa_dsl_pool == NULL);
	ASSERT(spa->spa_root_vdev == NULL);
	ASSERT(spa->spa_async_zio_root == NULL);
	ASSERT(spa->spa_state != POOL_STATE_UNINITIALIZED);

	spa_evicting_os_wait(spa);

	if (spa->spa_zvol_taskq) {
	taskq_destroy(spa->spa_zvol_taskq);
	spa->spa_zvol_taskq = NULL;
	}

	if (spa->spa_prefetch_taskq) {
	taskq_destroy(spa->spa_prefetch_taskq);
	spa->spa_prefetch_taskq = NULL;
	}

	if (spa->spa_upgrade_taskq) {
	taskq_destroy(spa->spa_upgrade_taskq);
	spa->spa_upgrade_taskq = NULL;
	}

	txg_list_destroy(&spa->spa_vdev_txg_list);

	list_destroy(&spa->spa_config_dirty_list);
	list_destroy(&spa->spa_evicting_os_list);
	list_destroy(&spa->spa_state_dirty_list);

	taskq_cancel_id(system_delay_taskq, spa->spa_deadman_tqid);

	for (int t = 0; t < ZIO_TYPES; t++) {
	for (int q = 0; q < ZIO_TASKQ_TYPES; q++) {
	spa_taskqs_fini(spa, t, q);
	}
	}

	for (size_t i = 0; i < TXG_SIZE; i++) {
	ASSERT3P(spa->spa_txg_zio[i], !=, NULL);
	VERIFY0(zio_wait(spa->spa_txg_zio[i]));
	spa->spa_txg_zio[i] = NULL;
	}

	metaslab_class_destroy(spa->spa_normal_class);
	spa->spa_normal_class = NULL;

	metaslab_class_destroy(spa->spa_log_class);
	spa->spa_log_class = NULL;

	metaslab_class_destroy(spa->spa_special_class);
	spa->spa_special_class = NULL;

	metaslab_class_destroy(spa->spa_dedup_class);
	spa->spa_dedup_class = NULL;

	/*
	* If this was part of an import or the open otherwise failed, we may
	* still have errors left in the queues. Empty them just in case.
	*/
	spa_errlog_drain(spa);
	avl_destroy(&spa->spa_errlist_scrub);
	avl_destroy(&spa->spa_errlist_last);

	spa_keystore_fini(&spa->spa_keystore);

	spa->spa_state = POOL_STATE_UNINITIALIZED;

	mutex_enter(&spa->spa_proc_lock);
	if (spa->spa_proc_state != SPA_PROC_NONE) {
	ASSERT(spa->spa_proc_state == SPA_PROC_ACTIVE);
	spa->spa_proc_state = SPA_PROC_DEACTIVATE;
	cv_broadcast(&spa->spa_proc_cv);
	while (spa->spa_proc_state == SPA_PROC_DEACTIVATE) {
	ASSERT(spa->spa_proc != &p0);
	cv_wait(&spa->spa_proc_cv, &spa->spa_proc_lock);
	}
	ASSERT(spa->spa_proc_state == SPA_PROC_GONE);
	spa->spa_proc_state = SPA_PROC_NONE;
	}
	ASSERT(spa->spa_proc == &p0);
	mutex_exit(&spa->spa_proc_lock);

	/*
	* We want to make sure spa_thread() has actually exited the ZFS
	* module, so that the module can't be unloaded out from underneath
	* it.
	*/
	if (spa->spa_did != 0) {
	thread_join(spa->spa_did);
	spa->spa_did = 0;
	}
	}

	/*
	* Verify a pool configuration, and construct the vdev tree appropriately. This
	* will create all the necessary vdevs in the appropriate layout, with each vdev
	* in the CLOSED state. This will prep the pool before open/creation/import.
	* All vdev validation is done by the vdev_alloc() routine.
	*/
	int
	spa_config_parse(spa_t spa, vdev_t vdp, nvlist_t nv, vdev_t *parent,
	uint_t id, int atype)
	{
	nvlist_t **child;
	uint_t children;
	int error;

	if ((error = vdev_alloc(spa, vdp, nv, parent, id, atype)) != 0)
	return (error);

	if ((*vdp)->vdev_ops->vdev_op_leaf)
	return (0);

	error = nvlist_lookup_nvlist_array(nv, ZPOOL_CONFIG_CHILDREN,
	&child, &children);

	if (error == ENOENT)
	return (0);

	if (error) {
	vdev_free(*vdp);
	*vdp = NULL;
	return (SET_ERROR(EINVAL));
	}

	for (int c = 0; c < children; c++) {
	vdev_t *vd;
	if ((error = spa_config_parse(spa, &vd, child[c], *vdp, c,
	atype)) != 0) {
	vdev_free(*vdp);
	*vdp = NULL;
	return (error);
	}
	}

	ASSERT(*vdp != NULL);

	return (0);
	}

	static boolean_t
	spa_should_flush_logs_on_unload(spa_t *spa)
	{
	if (!spa_feature_is_active(spa, SPA_FEATURE_LOG_SPACEMAP))
	return (B_FALSE);

	if (!spa_writeable(spa))
	return (B_FALSE);

	if (!spa->spa_sync_on)
	return (B_FALSE);

	if (spa_state(spa) != POOL_STATE_EXPORTED)
	return (B_FALSE);

	if (zfs_keep_log_spacemaps_at_export)
	return (B_FALSE);

	return (B_TRUE);
	}

	/*
	* Opens a transaction that will set the flag that will instruct
	* spa_sync to attempt to flush all the metaslabs for that txg.
	*/
	static void
	spa_unload_log_sm_flush_all(spa_t *spa)
	{
	dmu_tx_t *tx = dmu_tx_create_dd(spa_get_dsl(spa)->dp_mos_dir);
	VERIFY0(dmu_tx_assign(tx, TXG_WAIT));

	ASSERT3U(spa->spa_log_flushall_txg, ==, 0);
	spa->spa_log_flushall_txg = dmu_tx_get_txg(tx);

	dmu_tx_commit(tx);
	txg_wait_synced(spa_get_dsl(spa), spa->spa_log_flushall_txg);
	}

	static void
	spa_unload_log_sm_metadata(spa_t *spa)
	{
	void *cookie = NULL;
	spa_log_sm_t *sls;
	while ((sls = avl_destroy_nodes(&spa->spa_sm_logs_by_txg,
	&cookie)) != NULL) {
	VERIFY0(sls->sls_mscount);
	kmem_free(sls, sizeof (spa_log_sm_t));
	}

	for (log_summary_entry_t *e = list_head(&spa->spa_log_summary);
	e != NULL; e = list_head(&spa->spa_log_summary)) {
	VERIFY0(e->lse_mscount);
	list_remove(&spa->spa_log_summary, e);
	kmem_free(e, sizeof (log_summary_entry_t));
	}

	spa->spa_unflushed_stats.sus_nblocks = 0;
	spa->spa_unflushed_stats.sus_memused = 0;
	spa->spa_unflushed_stats.sus_blocklimit = 0;
	}

	static void
	spa_destroy_aux_threads(spa_t *spa)
	{
	if (spa->spa_condense_zthr != NULL) {
	zthr_destroy(spa->spa_condense_zthr);
	spa->spa_condense_zthr = NULL;
	}
	if (spa->spa_checkpoint_discard_zthr != NULL) {
	zthr_destroy(spa->spa_checkpoint_discard_zthr);
	spa->spa_checkpoint_discard_zthr = NULL;
	}
	if (spa->spa_livelist_delete_zthr != NULL) {
	zthr_destroy(spa->spa_livelist_delete_zthr);
	spa->spa_livelist_delete_zthr = NULL;
	}
	if (spa->spa_livelist_condense_zthr != NULL) {
	zthr_destroy(spa->spa_livelist_condense_zthr);
	spa->spa_livelist_condense_zthr = NULL;
	}
	}

	/*
	* Opposite of spa_load().
	*/
	static void
	spa_unload(spa_t *spa)
	{
	ASSERT(MUTEX_HELD(&spa_namespace_lock));
	ASSERT(spa_state(spa) != POOL_STATE_UNINITIALIZED);

	spa_import_progress_remove(spa_guid(spa));
	spa_load_note(spa, "UNLOADING");

	spa_wake_waiters(spa);

	/*
	* If the log space map feature is enabled and the pool is getting
	* exported (but not destroyed), we want to spend some time flushing
	* as many metaslabs as we can in an attempt to destroy log space
	* maps and save import time.
	*/
	if (spa_should_flush_logs_on_unload(spa))
	spa_unload_log_sm_flush_all(spa);

	/*
	* Stop async tasks.
	*/
	spa_async_suspend(spa);

	if (spa->spa_root_vdev) {
	vdev_t *root_vdev = spa->spa_root_vdev;
	vdev_initialize_stop_all(root_vdev, VDEV_INITIALIZE_ACTIVE);
	vdev_trim_stop_all(root_vdev, VDEV_TRIM_ACTIVE);
	vdev_autotrim_stop_all(spa);
	vdev_rebuild_stop_all(spa);
	}

	/*
	* Stop syncing.
	*/
	if (spa->spa_sync_on) {
	txg_sync_stop(spa->spa_dsl_pool);
	spa->spa_sync_on = B_FALSE;
	}

	/*
	* This ensures that there is no async metaslab prefetching
	* while we attempt to unload the spa.
	*/
	if (spa->spa_root_vdev != NULL) {
	for (int c = 0; c < spa->spa_root_vdev->vdev_children; c++) {
	vdev_t *vc = spa->spa_root_vdev->vdev_child[c];
	if (vc->vdev_mg != NULL)
	taskq_wait(vc->vdev_mg->mg_taskq);
	}
	}

	if (spa->spa_mmp.mmp_thread)
	mmp_thread_stop(spa);

	/*
	* Wait for any outstanding async I/O to complete.
	*/
	if (spa->spa_async_zio_root != NULL) {
	for (int i = 0; i < max_ncpus; i++)
	(void) zio_wait(spa->spa_async_zio_root[i]);
	kmem_free(spa->spa_async_zio_root, max_ncpus * sizeof (void *));
	spa->spa_async_zio_root = NULL;
	}

	if (spa->spa_vdev_removal != NULL) {
	spa_vdev_removal_destroy(spa->spa_vdev_removal);
	spa->spa_vdev_removal = NULL;
	}

	spa_destroy_aux_threads(spa);

	spa_condense_fini(spa);

	bpobj_close(&spa->spa_deferred_bpobj);

	spa_config_enter(spa, SCL_ALL, spa, RW_WRITER);

	/*
	* Close all vdevs.
	*/
	if (spa->spa_root_vdev)
	vdev_free(spa->spa_root_vdev);
	ASSERT(spa->spa_root_vdev == NULL);

	/*
	* Close the dsl pool.
	*/
	if (spa->spa_dsl_pool) {
	dsl_pool_close(spa->spa_dsl_pool);
	spa->spa_dsl_pool = NULL;
	spa->spa_meta_objset = NULL;
	}

	ddt_unload(spa);
	spa_unload_log_sm_metadata(spa);

	/*
	* Drop and purge level 2 cache
	*/
	spa_l2cache_drop(spa);

	for (int i = 0; i < spa->spa_spares.sav_count; i++)
	vdev_free(spa->spa_spares.sav_vdevs[i]);
	if (spa->spa_spares.sav_vdevs) {
	kmem_free(spa->spa_spares.sav_vdevs,
	spa->spa_spares.sav_count * sizeof (void *));
	spa->spa_spares.sav_vdevs = NULL;
	}
	if (spa->spa_spares.sav_config) {
	nvlist_free(spa->spa_spares.sav_config);
	spa->spa_spares.sav_config = NULL;
	}
	spa->spa_spares.sav_count = 0;

	for (int i = 0; i < spa->spa_l2cache.sav_count; i++) {
	vdev_clear_stats(spa->spa_l2cache.sav_vdevs[i]);
	vdev_free(spa->spa_l2cache.sav_vdevs[i]);
	}
	if (spa->spa_l2cache.sav_vdevs) {
	kmem_free(spa->spa_l2cache.sav_vdevs,
	spa->spa_l2cache.sav_count * sizeof (void *));
	spa->spa_l2cache.sav_vdevs = NULL;
	}
	if (spa->spa_l2cache.sav_config) {
	nvlist_free(spa->spa_l2cache.sav_config);
	spa->spa_l2cache.sav_config = NULL;
	}
	spa->spa_l2cache.sav_count = 0;

	spa->spa_async_suspended = 0;

	spa->spa_indirect_vdevs_loaded = B_FALSE;

	if (spa->spa_comment != NULL) {
	spa_strfree(spa->spa_comment);
	spa->spa_comment = NULL;
	}

	spa_config_exit(spa, SCL_ALL, spa);
	}

	/*
	* Load (or re-load) the current list of vdevs describing the active spares for
	* this pool. When this is called, we have some form of basic information in
	* 'spa_spares.sav_config'. We parse this into vdevs, try to open them, and
	* then re-generate a more complete list including status information.
	*/
	void
	spa_load_spares(spa_t *spa)
	{
	nvlist_t **spares;
	uint_t nspares;
	int i;
	vdev_t vd, tvd;

	#ifndef _KERNEL
	/*
	* zdb opens both the current state of the pool and the
	* checkpointed state (if present), with a different spa_t.
	*
	* As spare vdevs are shared among open pools, we skip loading
	* them when we load the checkpointed state of the pool.
	*/
	if (!spa_writeable(spa))
	return;
	#endif

	ASSERT(spa_config_held(spa, SCL_ALL, RW_WRITER) == SCL_ALL);

	/*
	* First, close and free any existing spare vdevs.
	*/
	for (i = 0; i < spa->spa_spares.sav_count; i++) {
	vd = spa->spa_spares.sav_vdevs[i];

	/* Undo the call to spa_activate() below */
	if ((tvd = spa_lookup_by_guid(spa, vd->vdev_guid,
	B_FALSE)) != NULL && tvd->vdev_isspare)
	spa_spare_remove(tvd);
	vdev_close(vd);
	vdev_free(vd);
	}

	if (spa->spa_spares.sav_vdevs)
	kmem_free(spa->spa_spares.sav_vdevs,
	spa->spa_spares.sav_count * sizeof (void *));

	if (spa->spa_spares.sav_config == NULL)
	nspares = 0;
	else
	VERIFY(nvlist_lookup_nvlist_array(spa->spa_spares.sav_config,
	ZPOOL_CONFIG_SPARES, &spares, &nspares) == 0);

	spa->spa_spares.sav_count = (int)nspares;
	spa->spa_spares.sav_vdevs = NULL;

	if (nspares == 0)
	return;

	/*
	* Construct the array of vdevs, opening them to get status in the
	* process. For each spare, there is potentially two different vdev_t
	* structures associated with it: one in the list of spares (used only
	* for basic validation purposes) and one in the active vdev
	* configuration (if it's spared in). During this phase we open and
	* validate each vdev on the spare list. If the vdev also exists in the
	* active configuration, then we also mark this vdev as an active spare.
	*/
	spa->spa_spares.sav_vdevs = kmem_zalloc(nspares * sizeof (void *),
	KM_SLEEP);
	for (i = 0; i < spa->spa_spares.sav_count; i++) {
	VERIFY(spa_config_parse(spa, &vd, spares[i], NULL, 0,
	VDEV_ALLOC_SPARE) == 0);
	ASSERT(vd != NULL);

	spa->spa_spares.sav_vdevs[i] = vd;

	if ((tvd = spa_lookup_by_guid(spa, vd->vdev_guid,
	B_FALSE)) != NULL) {
	if (!tvd->vdev_isspare)
	spa_spare_add(tvd);

	/*
	* We only mark the spare active if we were successfully
	* able to load the vdev. Otherwise, importing a pool
	* with a bad active spare would result in strange
	* behavior, because multiple pool would think the spare
	* is actively in use.
	*
	* There is a vulnerability here to an equally bizarre
	* circumstance, where a dead active spare is later
	* brought back to life (onlined or otherwise). Given
	* the rarity of this scenario, and the extra complexity
	* it adds, we ignore the possibility.
	*/
	if (!vdev_is_dead(tvd))
	spa_spare_activate(tvd);
	}

	vd->vdev_top = vd;
	vd->vdev_aux = &spa->spa_spares;

	if (vdev_open(vd) != 0)
	continue;

	if (vdev_validate_aux(vd) == 0)
	spa_spare_add(vd);
	}

	/*
	* Recompute the stashed list of spares, with status information
	* this time.
	*/
	VERIFY(nvlist_remove(spa->spa_spares.sav_config, ZPOOL_CONFIG_SPARES,
	DATA_TYPE_NVLIST_ARRAY) == 0);

	spares = kmem_alloc(spa->spa_spares.sav_count * sizeof (void *),
	KM_SLEEP);
	for (i = 0; i < spa->spa_spares.sav_count; i++)
	spares[i] = vdev_config_generate(spa,
	spa->spa_spares.sav_vdevs[i], B_TRUE, VDEV_CONFIG_SPARE);
	VERIFY(nvlist_add_nvlist_array(spa->spa_spares.sav_config,
	ZPOOL_CONFIG_SPARES, spares, spa->spa_spares.sav_count) == 0);
	for (i = 0; i < spa->spa_spares.sav_count; i++)
	nvlist_free(spares[i]);
	kmem_free(spares, spa->spa_spares.sav_count * sizeof (void *));
	}

	/*
	* Load (or re-load) the current list of vdevs describing the active l2cache for
	* this pool. When this is called, we have some form of basic information in
	* 'spa_l2cache.sav_config'. We parse this into vdevs, try to open them, and
	* then re-generate a more complete list including status information.
	* Devices which are already active have their details maintained, and are
	* not re-opened.
	*/
	void
	spa_load_l2cache(spa_t *spa)
	{
	nvlist_t **l2cache = NULL;
	uint_t nl2cache;
	int i, j, oldnvdevs;
	uint64_t guid;
	vdev_t vd, oldvdevs, *newvdevs;
	spa_aux_vdev_t *sav = &spa->spa_l2cache;

	#ifndef _KERNEL
	/*
	* zdb opens both the current state of the pool and the
	* checkpointed state (if present), with a different spa_t.
	*
	* As L2 caches are part of the ARC which is shared among open
	* pools, we skip loading them when we load the checkpointed
	* state of the pool.
	*/
	if (!spa_writeable(spa))
	return;
	#endif

	ASSERT(spa_config_held(spa, SCL_ALL, RW_WRITER) == SCL_ALL);

	oldvdevs = sav->sav_vdevs;
	oldnvdevs = sav->sav_count;
	sav->sav_vdevs = NULL;
	sav->sav_count = 0;

	if (sav->sav_config == NULL) {
	nl2cache = 0;
	newvdevs = NULL;
	goto out;
	}

	VERIFY(nvlist_lookup_nvlist_array(sav->sav_config,
	ZPOOL_CONFIG_L2CACHE, &l2cache, &nl2cache) == 0);
	newvdevs = kmem_alloc(nl2cache * sizeof (void *), KM_SLEEP);

	/*
	* Process new nvlist of vdevs.
	*/
	for (i = 0; i < nl2cache; i++) {
	VERIFY(nvlist_lookup_uint64(l2cache[i], ZPOOL_CONFIG_GUID,
	&guid) == 0);

	newvdevs[i] = NULL;
	for (j = 0; j < oldnvdevs; j++) {
	vd = oldvdevs[j];
	if (vd != NULL && guid == vd->vdev_guid) {
	/*
	* Retain previous vdev for add/remove ops.
	*/
	newvdevs[i] = vd;
	oldvdevs[j] = NULL;
	break;
	}
	}

	if (newvdevs[i] == NULL) {
	/*
	* Create new vdev
	*/
	VERIFY(spa_config_parse(spa, &vd, l2cache[i], NULL, 0,
	VDEV_ALLOC_L2CACHE) == 0);
	ASSERT(vd != NULL);
	newvdevs[i] = vd;

	/*
	* Commit this vdev as an l2cache device,
	* even if it fails to open.
	*/
	spa_l2cache_add(vd);

	vd->vdev_top = vd;
	vd->vdev_aux = sav;

	spa_l2cache_activate(vd);

	if (vdev_open(vd) != 0)
	continue;

	(void) vdev_validate_aux(vd);

	if (!vdev_is_dead(vd))
	l2arc_add_vdev(spa, vd);

	/*
	* Upon cache device addition to a pool or pool
	* creation with a cache device or if the header
	* of the device is invalid we issue an async
	* TRIM command for the whole device which will
	* execute if l2arc_trim_ahead > 0.
	*/
	spa_async_request(spa, SPA_ASYNC_L2CACHE_TRIM);
	}
	}

	sav->sav_vdevs = newvdevs;
	sav->sav_count = (int)nl2cache;

	/*
	* Recompute the stashed list of l2cache devices, with status
	* information this time.
	*/
	VERIFY(nvlist_remove(sav->sav_config, ZPOOL_CONFIG_L2CACHE,
	DATA_TYPE_NVLIST_ARRAY) == 0);

	if (sav->sav_count > 0)
	l2cache = kmem_alloc(sav->sav_count * sizeof (void *),
	KM_SLEEP);
	for (i = 0; i < sav->sav_count; i++)
	l2cache[i] = vdev_config_generate(spa,
	sav->sav_vdevs[i], B_TRUE, VDEV_CONFIG_L2CACHE);
	VERIFY(nvlist_add_nvlist_array(sav->sav_config,
	ZPOOL_CONFIG_L2CACHE, l2cache, sav->sav_count) == 0);

	out:
	/*
	* Purge vdevs that were dropped
	*/
	for (i = 0; i < oldnvdevs; i++) {
	uint64_t pool;

	vd = oldvdevs[i];
	if (vd != NULL) {
	ASSERT(vd->vdev_isl2cache);

	if (spa_l2cache_exists(vd->vdev_guid, &pool) &&
	pool != 0ULL && l2arc_vdev_present(vd))
	l2arc_remove_vdev(vd);
	vdev_clear_stats(vd);
	vdev_free(vd);
	}
	}

	if (oldvdevs)
	kmem_free(oldvdevs, oldnvdevs * sizeof (void *));

	for (i = 0; i < sav->sav_count; i++)
	nvlist_free(l2cache[i]);
	if (sav->sav_count)
	kmem_free(l2cache, sav->sav_count * sizeof (void *));
	}

	static int
	load_nvlist(spa_t spa, uint64_t obj, nvlist_t *value)
	{
	dmu_buf_t *db;
	char *packed = NULL;
	size_t nvsize = 0;
	int error;
	*value = NULL;

	error = dmu_bonus_hold(spa->spa_meta_objset, obj, FTAG, &db);
	if (error)
	return (error);

	nvsize = (uint64_t )db->db_data;
	dmu_buf_rele(db, FTAG);

	packed = vmem_alloc(nvsize, KM_SLEEP);
	error = dmu_read(spa->spa_meta_objset, obj, 0, nvsize, packed,
	DMU_READ_PREFETCH);
	if (error == 0)
	error = nvlist_unpack(packed, nvsize, value, 0);
	vmem_free(packed, nvsize);

	return (error);
	}

	/*
	* Concrete top-level vdevs that are not missing and are not logs. At every
	* spa_sync we write new uberblocks to at least SPA_SYNC_MIN_VDEVS core tvds.
	*/
	static uint64_t
	spa_healthy_core_tvds(spa_t *spa)
	{
	vdev_t *rvd = spa->spa_root_vdev;
	uint64_t tvds = 0;

	for (uint64_t i = 0; i < rvd->vdev_children; i++) {
	vdev_t *vd = rvd->vdev_child[i];
	if (vd->vdev_islog)
	continue;
	if (vdev_is_concrete(vd) && !vdev_is_dead(vd))
	tvds++;
	}

	return (tvds);
	}

	/*
	* Checks to see if the given vdev could not be opened, in which case we post a
	* sysevent to notify the autoreplace code that the device has been removed.
	*/
	static void
	spa_check_removed(vdev_t *vd)
	{
	for (uint64_t c = 0; c < vd->vdev_children; c++)
	spa_check_removed(vd->vdev_child[c]);

	if (vd->vdev_ops->vdev_op_leaf && vdev_is_dead(vd) &&
	vdev_is_concrete(vd)) {
	zfs_post_autoreplace(vd->vdev_spa, vd);
	spa_event_notify(vd->vdev_spa, vd, NULL, ESC_ZFS_VDEV_CHECK);
	}
	}

	static int
	spa_check_for_missing_logs(spa_t *spa)
	{
	vdev_t *rvd = spa->spa_root_vdev;

	/*
	* If we're doing a normal import, then build up any additional
	* diagnostic information about missing log devices.
	* We'll pass this up to the user for further processing.
	*/
	if (!(spa->spa_import_flags & ZFS_IMPORT_MISSING_LOG)) {
	nvlist_t *child, nv;
	uint64_t idx = 0;

	child = kmem_alloc(rvd->vdev_children * sizeof (nvlist_t *),
	KM_SLEEP);
	VERIFY(nvlist_alloc(&nv, NV_UNIQUE_NAME, KM_SLEEP) == 0);

	for (uint64_t c = 0; c < rvd->vdev_children; c++) {
	vdev_t *tvd = rvd->vdev_child[c];

	/*
	* We consider a device as missing only if it failed
	* to open (i.e. offline or faulted is not considered
	* as missing).
	*/
	if (tvd->vdev_islog &&
	tvd->vdev_state == VDEV_STATE_CANT_OPEN) {
	child[idx++] = vdev_config_generate(spa, tvd,
	B_FALSE, VDEV_CONFIG_MISSING);
	}
	}

	if (idx > 0) {
	fnvlist_add_nvlist_array(nv,
	ZPOOL_CONFIG_CHILDREN, child, idx);
	fnvlist_add_nvlist(spa->spa_load_info,
	ZPOOL_CONFIG_MISSING_DEVICES, nv);

	for (uint64_t i = 0; i < idx; i++)
	nvlist_free(child[i]);
	}
	nvlist_free(nv);
	kmem_free(child, rvd->vdev_children * sizeof (char **));

	if (idx > 0) {
	spa_load_failed(spa, "some log devices are missing");
	vdev_dbgmsg_print_tree(rvd, 2);
	return (SET_ERROR(ENXIO));
	}
	} else {
	for (uint64_t c = 0; c < rvd->vdev_children; c++) {
	vdev_t *tvd = rvd->vdev_child[c];

	if (tvd->vdev_islog &&
	tvd->vdev_state == VDEV_STATE_CANT_OPEN) {
	spa_set_log_state(spa, SPA_LOG_CLEAR);
	spa_load_note(spa, "some log devices are "
	"missing, ZIL is dropped.");
	vdev_dbgmsg_print_tree(rvd, 2);
	break;
	}
	}
	}

	return (0);
	}

	/*
	* Check for missing log devices
	*/
	static boolean_t
	spa_check_logs(spa_t *spa)
	{
	boolean_t rv = B_FALSE;
	dsl_pool_t *dp = spa_get_dsl(spa);

	switch (spa->spa_log_state) {
	default:
	break;
	case SPA_LOG_MISSING:
	/* need to recheck in case slog has been restored */
	case SPA_LOG_UNKNOWN:
	rv = (dmu_objset_find_dp(dp, dp->dp_root_dir_obj,
	zil_check_log_chain, NULL, DS_FIND_CHILDREN) != 0);
	if (rv)
	spa_set_log_state(spa, SPA_LOG_MISSING);
	break;
	}
	return (rv);
	}

	static boolean_t
	spa_passivate_log(spa_t *spa)
	{
	vdev_t *rvd = spa->spa_root_vdev;
	boolean_t slog_found = B_FALSE;

	ASSERT(spa_config_held(spa, SCL_ALLOC, RW_WRITER));

	if (!spa_has_slogs(spa))
	return (B_FALSE);

	for (int c = 0; c < rvd->vdev_children; c++) {
	vdev_t *tvd = rvd->vdev_child[c];
	metaslab_group_t *mg = tvd->vdev_mg;

	if (tvd->vdev_islog) {
	metaslab_group_passivate(mg);
	slog_found = B_TRUE;
	}
	}

	return (slog_found);
	}

	static void
	spa_activate_log(spa_t *spa)
	{
	vdev_t *rvd = spa->spa_root_vdev;

	ASSERT(spa_config_held(spa, SCL_ALLOC, RW_WRITER));

	for (int c = 0; c < rvd->vdev_children; c++) {
	vdev_t *tvd = rvd->vdev_child[c];
	metaslab_group_t *mg = tvd->vdev_mg;

	if (tvd->vdev_islog)
	metaslab_group_activate(mg);
	}
	}

	int
	spa_reset_logs(spa_t *spa)
	{
	int error;

	error = dmu_objset_find(spa_name(spa), zil_reset,
	NULL, DS_FIND_CHILDREN);
	if (error == 0) {
	/*
	* We successfully offlined the log device, sync out the
	* current txg so that the "stubby" block can be removed
	* by zil_sync().
	*/
	txg_wait_synced(spa->spa_dsl_pool, 0);
	}
	return (error);
	}

	static void
	spa_aux_check_removed(spa_aux_vdev_t *sav)
	{
	for (int i = 0; i < sav->sav_count; i++)
	spa_check_removed(sav->sav_vdevs[i]);
	}

	void
	spa_claim_notify(zio_t *zio)
	{
	spa_t *spa = zio->io_spa;

	if (zio->io_error)
	return;

	mutex_enter(&spa->spa_props_lock); /* any mutex will do */
	if (spa->spa_claim_max_txg < zio->io_bp->blk_birth)
	spa->spa_claim_max_txg = zio->io_bp->blk_birth;
	mutex_exit(&spa->spa_props_lock);
	}

	typedef struct spa_load_error {
	uint64_t sle_meta_count;
	uint64_t sle_data_count;
	} spa_load_error_t;

	static void
	spa_load_verify_done(zio_t *zio)
	{
	blkptr_t *bp = zio->io_bp;
	spa_load_error_t *sle = zio->io_private;
	dmu_object_type_t type = BP_GET_TYPE(bp);
	int error = zio->io_error;
	spa_t *spa = zio->io_spa;

	abd_free(zio->io_abd);
	if (error) {
	if ((BP_GET_LEVEL(bp) != 0 \|\| DMU_OT_IS_METADATA(type)) &&
	type != DMU_OT_INTENT_LOG)
	atomic_inc_64(&sle->sle_meta_count);
	else
	atomic_inc_64(&sle->sle_data_count);
	}

	mutex_enter(&spa->spa_scrub_lock);
	spa->spa_load_verify_bytes -= BP_GET_PSIZE(bp);
	cv_broadcast(&spa->spa_scrub_io_cv);
	mutex_exit(&spa->spa_scrub_lock);
	}

	/*
	* Maximum number of inflight bytes is the log2 fraction of the arc size.
	* By default, we set it to 1/16th of the arc.
	*/
	int spa_load_verify_shift = 4;
	int spa_load_verify_metadata = B_TRUE;
	int spa_load_verify_data = B_TRUE;

	/ARGSUSED/
	static int
	spa_load_verify_cb(spa_t spa, zilog_t zilog, const blkptr_t *bp,
	const zbookmark_phys_t zb, const dnode_phys_t dnp, void *arg)
	{
	if (zb->zb_level == ZB_DNODE_LEVEL \|\| BP_IS_HOLE(bp) \|\|
	BP_IS_EMBEDDED(bp) \|\| BP_IS_REDACTED(bp))
	return (0);
	/*
	* Note: normally this routine will not be called if
	* spa_load_verify_metadata is not set. However, it may be useful
	* to manually set the flag after the traversal has begun.
	*/
	if (!spa_load_verify_metadata)
	return (0);
	if (!BP_IS_METADATA(bp) && !spa_load_verify_data)
	return (0);

	uint64_t maxinflight_bytes =
	arc_target_bytes() >> spa_load_verify_shift;
	zio_t *rio = arg;
	size_t size = BP_GET_PSIZE(bp);

	mutex_enter(&spa->spa_scrub_lock);
	while (spa->spa_load_verify_bytes >= maxinflight_bytes)
	cv_wait(&spa->spa_scrub_io_cv, &spa->spa_scrub_lock);
	spa->spa_load_verify_bytes += size;
	mutex_exit(&spa->spa_scrub_lock);

	zio_nowait(zio_read(rio, spa, bp, abd_alloc_for_io(size, B_FALSE), size,
	spa_load_verify_done, rio->io_private, ZIO_PRIORITY_SCRUB,
	ZIO_FLAG_SPECULATIVE \| ZIO_FLAG_CANFAIL \|
	ZIO_FLAG_SCRUB \| ZIO_FLAG_RAW, zb));
	return (0);
	}

	/* ARGSUSED */
	static int
	verify_dataset_name_len(dsl_pool_t dp, dsl_dataset_t ds, void *arg)
	{
	if (dsl_dataset_namelen(ds) >= ZFS_MAX_DATASET_NAME_LEN)
	return (SET_ERROR(ENAMETOOLONG));

	return (0);
	}

	static int
	spa_load_verify(spa_t *spa)
	{
	zio_t *rio;
	spa_load_error_t sle = { 0 };
	zpool_load_policy_t policy;
	boolean_t verify_ok = B_FALSE;
	int error = 0;

	zpool_get_load_policy(spa->spa_config, &policy);

	if (policy.zlp_rewind & ZPOOL_NEVER_REWIND)
	return (0);

	dsl_pool_config_enter(spa->spa_dsl_pool, FTAG);
	error = dmu_objset_find_dp(spa->spa_dsl_pool,
	spa->spa_dsl_pool->dp_root_dir_obj, verify_dataset_name_len, NULL,
	DS_FIND_CHILDREN);
	dsl_pool_config_exit(spa->spa_dsl_pool, FTAG);
	if (error != 0)
	return (error);

	rio = zio_root(spa, NULL, &sle,
	ZIO_FLAG_CANFAIL \| ZIO_FLAG_SPECULATIVE);

	if (spa_load_verify_metadata) {
	if (spa->spa_extreme_rewind) {
	spa_load_note(spa, "performing a complete scan of the "
	"pool since extreme rewind is on. This may take "
	"a very long time.\n (spa_load_verify_data=%u, "
	"spa_load_verify_metadata=%u)",
	spa_load_verify_data, spa_load_verify_metadata);
	}

	error = traverse_pool(spa, spa->spa_verify_min_txg,
	TRAVERSE_PRE \| TRAVERSE_PREFETCH_METADATA \|
	TRAVERSE_NO_DECRYPT, spa_load_verify_cb, rio);
	}

	(void) zio_wait(rio);
	ASSERT0(spa->spa_load_verify_bytes);

	spa->spa_load_meta_errors = sle.sle_meta_count;
	spa->spa_load_data_errors = sle.sle_data_count;

	if (sle.sle_meta_count != 0 \|\| sle.sle_data_count != 0) {
	spa_load_note(spa, "spa_load_verify found %llu metadata errors "
	"and %llu data errors", (u_longlong_t)sle.sle_meta_count,
	(u_longlong_t)sle.sle_data_count);
	}

	if (spa_load_verify_dryrun \|\|
	(!error && sle.sle_meta_count <= policy.zlp_maxmeta &&
	sle.sle_data_count <= policy.zlp_maxdata)) {
	int64_t loss = 0;

	verify_ok = B_TRUE;
	spa->spa_load_txg = spa->spa_uberblock.ub_txg;
	spa->spa_load_txg_ts = spa->spa_uberblock.ub_timestamp;

	loss = spa->spa_last_ubsync_txg_ts - spa->spa_load_txg_ts;
	VERIFY(nvlist_add_uint64(spa->spa_load_info,
	ZPOOL_CONFIG_LOAD_TIME, spa->spa_load_txg_ts) == 0);
	VERIFY(nvlist_add_int64(spa->spa_load_info,
	ZPOOL_CONFIG_REWIND_TIME, loss) == 0);
	VERIFY(nvlist_add_uint64(spa->spa_load_info,
	ZPOOL_CONFIG_LOAD_DATA_ERRORS, sle.sle_data_count) == 0);
	} else {
	spa->spa_load_max_txg = spa->spa_uberblock.ub_txg;
	}

	if (spa_load_verify_dryrun)
	return (0);

	if (error) {
	if (error != ENXIO && error != EIO)
	error = SET_ERROR(EIO);
	return (error);
	}

	return (verify_ok ? 0 : EIO);
	}

	/*
	* Find a value in the pool props object.
	*/
	static void
	spa_prop_find(spa_t spa, zpool_prop_t prop, uint64_t val)
	{
	(void) zap_lookup(spa->spa_meta_objset, spa->spa_pool_props_object,
	zpool_prop_to_name(prop), sizeof (uint64_t), 1, val);
	}

	/*
	* Find a value in the pool directory object.
	*/
	static int
	spa_dir_prop(spa_t spa, const char name, uint64_t *val, boolean_t log_enoent)
	{
	int error = zap_lookup(spa->spa_meta_objset, DMU_POOL_DIRECTORY_OBJECT,
	name, sizeof (uint64_t), 1, val);

	if (error != 0 && (error != ENOENT \|\| log_enoent)) {
	spa_load_failed(spa, "couldn't get '%s' value in MOS directory "
	"[error=%d]", name, error);
	}

	return (error);
	}

	static int
	spa_vdev_err(vdev_t *vdev, vdev_aux_t aux, int err)
	{
	vdev_set_state(vdev, B_TRUE, VDEV_STATE_CANT_OPEN, aux);
	return (SET_ERROR(err));
	}

	boolean_t
	spa_livelist_delete_check(spa_t *spa)
	{
	return (spa->spa_livelists_to_delete != 0);
	}

	/* ARGSUSED */
	static boolean_t
	spa_livelist_delete_cb_check(void arg, zthr_t z)
	{
	spa_t *spa = arg;
	return (spa_livelist_delete_check(spa));
	}

	static int
	delete_blkptr_cb(void arg, const blkptr_t bp, dmu_tx_t *tx)
	{
	spa_t *spa = arg;
	zio_free(spa, tx->tx_txg, bp);
	dsl_dir_diduse_space(tx->tx_pool->dp_free_dir, DD_USED_HEAD,
	-bp_get_dsize_sync(spa, bp),
	-BP_GET_PSIZE(bp), -BP_GET_UCSIZE(bp), tx);
	return (0);
	}

	static int
	dsl_get_next_livelist_obj(objset_t os, uint64_t zap_obj, uint64_t llp)
	{
	int err;
	zap_cursor_t zc;
	zap_attribute_t za;
	zap_cursor_init(&zc, os, zap_obj);
	err = zap_cursor_retrieve(&zc, &za);
	zap_cursor_fini(&zc);
	if (err == 0)
	*llp = za.za_first_integer;
	return (err);
	}

	/*
	* Components of livelist deletion that must be performed in syncing
	* context: freeing block pointers and updating the pool-wide data
	* structures to indicate how much work is left to do
	*/
	typedef struct sublist_delete_arg {
	spa_t *spa;
	dsl_deadlist_t *ll;
	uint64_t key;
	bplist_t *to_free;
	} sublist_delete_arg_t;

	static void
	sublist_delete_sync(void arg, dmu_tx_t tx)
	{
	sublist_delete_arg_t *sda = arg;
	spa_t *spa = sda->spa;
	dsl_deadlist_t *ll = sda->ll;
	uint64_t key = sda->key;
	bplist_t *to_free = sda->to_free;

	bplist_iterate(to_free, delete_blkptr_cb, spa, tx);
	dsl_deadlist_remove_entry(ll, key, tx);
	}

	typedef struct livelist_delete_arg {
	spa_t *spa;
	uint64_t ll_obj;
	uint64_t zap_obj;
	} livelist_delete_arg_t;

	static void
	livelist_delete_sync(void arg, dmu_tx_t tx)
	{
	livelist_delete_arg_t *lda = arg;
	spa_t *spa = lda->spa;
	uint64_t ll_obj = lda->ll_obj;
	uint64_t zap_obj = lda->zap_obj;
	objset_t *mos = spa->spa_meta_objset;
	uint64_t count;

	/* free the livelist and decrement the feature count */
	VERIFY0(zap_remove_int(mos, zap_obj, ll_obj, tx));
	dsl_deadlist_free(mos, ll_obj, tx);
	spa_feature_decr(spa, SPA_FEATURE_LIVELIST, tx);
	VERIFY0(zap_count(mos, zap_obj, &count));
	if (count == 0) {
	/* no more livelists to delete */
	VERIFY0(zap_remove(mos, DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_DELETED_CLONES, tx));
	VERIFY0(zap_destroy(mos, zap_obj, tx));
	spa->spa_livelists_to_delete = 0;
	spa_notify_waiters(spa);
	}
	}

	/*
	* Load in the value for the livelist to be removed and open it. Then,
	* load its first sublist and determine which block pointers should actually
	* be freed. Then, call a synctask which performs the actual frees and updates
	* the pool-wide livelist data.
	*/
	/* ARGSUSED */
	static void
	spa_livelist_delete_cb(void arg, zthr_t z)
	{
	spa_t *spa = arg;
	uint64_t ll_obj = 0, count;
	objset_t *mos = spa->spa_meta_objset;
	uint64_t zap_obj = spa->spa_livelists_to_delete;
	/*
	* Determine the next livelist to delete. This function should only
	* be called if there is at least one deleted clone.
	*/
	VERIFY0(dsl_get_next_livelist_obj(mos, zap_obj, &ll_obj));
	VERIFY0(zap_count(mos, ll_obj, &count));
	if (count > 0) {
	dsl_deadlist_t *ll;
	dsl_deadlist_entry_t *dle;
	bplist_t to_free;
	ll = kmem_zalloc(sizeof (dsl_deadlist_t), KM_SLEEP);
	dsl_deadlist_open(ll, mos, ll_obj);
	dle = dsl_deadlist_first(ll);
	ASSERT3P(dle, !=, NULL);
	bplist_create(&to_free);
	int err = dsl_process_sub_livelist(&dle->dle_bpobj, &to_free,
	z, NULL);
	if (err == 0) {
	sublist_delete_arg_t sync_arg = {
	.spa = spa,
	.ll = ll,
	.key = dle->dle_mintxg,
	.to_free = &to_free
	};
	zfs_dbgmsg("deleting sublist (id %llu) from"
	" livelist %llu, %d remaining",
	dle->dle_bpobj.bpo_object, ll_obj, count - 1);
	VERIFY0(dsl_sync_task(spa_name(spa), NULL,
	sublist_delete_sync, &sync_arg, 0,
	ZFS_SPACE_CHECK_DESTROY));
	} else {
	VERIFY3U(err, ==, EINTR);
	}
	bplist_clear(&to_free);
	bplist_destroy(&to_free);
	dsl_deadlist_close(ll);
	kmem_free(ll, sizeof (dsl_deadlist_t));
	} else {
	livelist_delete_arg_t sync_arg = {
	.spa = spa,
	.ll_obj = ll_obj,
	.zap_obj = zap_obj
	};
	zfs_dbgmsg("deletion of livelist %llu completed", ll_obj);
	VERIFY0(dsl_sync_task(spa_name(spa), NULL, livelist_delete_sync,
	&sync_arg, 0, ZFS_SPACE_CHECK_DESTROY));
	}
	}

	static void
	spa_start_livelist_destroy_thread(spa_t *spa)
	{
	ASSERT3P(spa->spa_livelist_delete_zthr, ==, NULL);
	spa->spa_livelist_delete_zthr =
	zthr_create("z_livelist_destroy",
	spa_livelist_delete_cb_check, spa_livelist_delete_cb, spa);
	}

	typedef struct livelist_new_arg {
	bplist_t *allocs;
	bplist_t *frees;
	} livelist_new_arg_t;

	static int
	livelist_track_new_cb(void arg, const blkptr_t bp, boolean_t bp_freed,
	dmu_tx_t *tx)
	{
	ASSERT(tx == NULL);
	livelist_new_arg_t *lna = arg;
	if (bp_freed) {
	bplist_append(lna->frees, bp);
	} else {
	bplist_append(lna->allocs, bp);
	zfs_livelist_condense_new_alloc++;
	}
	return (0);
	}

	typedef struct livelist_condense_arg {
	spa_t *spa;
	bplist_t to_keep;
	uint64_t first_size;
	uint64_t next_size;
	} livelist_condense_arg_t;

	static void
	spa_livelist_condense_sync(void arg, dmu_tx_t tx)
	{
	livelist_condense_arg_t *lca = arg;
	spa_t *spa = lca->spa;
	bplist_t new_frees;
	dsl_dataset_t *ds = spa->spa_to_condense.ds;

	/* Have we been cancelled? */
	if (spa->spa_to_condense.cancelled) {
	zfs_livelist_condense_sync_cancel++;
	goto out;
	}

	dsl_deadlist_entry_t *first = spa->spa_to_condense.first;
	dsl_deadlist_entry_t *next = spa->spa_to_condense.next;
	dsl_deadlist_t *ll = &ds->ds_dir->dd_livelist;

	/*
	* It's possible that the livelist was changed while the zthr was
	* running. Therefore, we need to check for new blkptrs in the two
	* entries being condensed and continue to track them in the livelist.
	* Because of the way we handle remapped blkptrs (see dbuf_remap_impl),
	* it's possible that the newly added blkptrs are FREEs or ALLOCs so
	* we need to sort them into two different bplists.
	*/
	uint64_t first_obj = first->dle_bpobj.bpo_object;
	uint64_t next_obj = next->dle_bpobj.bpo_object;
	uint64_t cur_first_size = first->dle_bpobj.bpo_phys->bpo_num_blkptrs;
	uint64_t cur_next_size = next->dle_bpobj.bpo_phys->bpo_num_blkptrs;

	bplist_create(&new_frees);
	livelist_new_arg_t new_bps = {
	.allocs = &lca->to_keep,
	.frees = &new_frees,
	};

	if (cur_first_size > lca->first_size) {
	VERIFY0(livelist_bpobj_iterate_from_nofree(&first->dle_bpobj,
	livelist_track_new_cb, &new_bps, lca->first_size));
	}
	if (cur_next_size > lca->next_size) {
	VERIFY0(livelist_bpobj_iterate_from_nofree(&next->dle_bpobj,
	livelist_track_new_cb, &new_bps, lca->next_size));
	}

	dsl_deadlist_clear_entry(first, ll, tx);
	ASSERT(bpobj_is_empty(&first->dle_bpobj));
	dsl_deadlist_remove_entry(ll, next->dle_mintxg, tx);

	bplist_iterate(&lca->to_keep, dsl_deadlist_insert_alloc_cb, ll, tx);
	bplist_iterate(&new_frees, dsl_deadlist_insert_free_cb, ll, tx);
	bplist_destroy(&new_frees);

	char dsname[ZFS_MAX_DATASET_NAME_LEN];
	dsl_dataset_name(ds, dsname);
	zfs_dbgmsg("txg %llu condensing livelist of %s (id %llu), bpobj %llu "
	"(%llu blkptrs) and bpobj %llu (%llu blkptrs) -> bpobj %llu "
	"(%llu blkptrs)", tx->tx_txg, dsname, ds->ds_object, first_obj,
	cur_first_size, next_obj, cur_next_size,
	first->dle_bpobj.bpo_object,
	first->dle_bpobj.bpo_phys->bpo_num_blkptrs);
	out:
	dmu_buf_rele(ds->ds_dbuf, spa);
	spa->spa_to_condense.ds = NULL;
	bplist_clear(&lca->to_keep);
	bplist_destroy(&lca->to_keep);
	kmem_free(lca, sizeof (livelist_condense_arg_t));
	spa->spa_to_condense.syncing = B_FALSE;
	}

	static void
	spa_livelist_condense_cb(void arg, zthr_t t)
	{
	while (zfs_livelist_condense_zthr_pause &&
	!(zthr_has_waiters(t) \|\| zthr_iscancelled(t)))
	delay(1);

	spa_t *spa = arg;
	dsl_deadlist_entry_t *first = spa->spa_to_condense.first;
	dsl_deadlist_entry_t *next = spa->spa_to_condense.next;
	uint64_t first_size, next_size;

	livelist_condense_arg_t *lca =
	kmem_alloc(sizeof (livelist_condense_arg_t), KM_SLEEP);
	bplist_create(&lca->to_keep);

	/*
	* Process the livelists (matching FREEs and ALLOCs) in open context
	* so we have minimal work in syncing context to condense.
	*
	* We save bpobj sizes (first_size and next_size) to use later in
	* syncing context to determine if entries were added to these sublists
	* while in open context. This is possible because the clone is still
	* active and open for normal writes and we want to make sure the new,
	* unprocessed blockpointers are inserted into the livelist normally.
	*
	* Note that dsl_process_sub_livelist() both stores the size number of
	* blockpointers and iterates over them while the bpobj's lock held, so
	* the sizes returned to us are consistent which what was actually
	* processed.
	*/
	int err = dsl_process_sub_livelist(&first->dle_bpobj, &lca->to_keep, t,
	&first_size);
	if (err == 0)
	err = dsl_process_sub_livelist(&next->dle_bpobj, &lca->to_keep,
	t, &next_size);

	if (err == 0) {
	while (zfs_livelist_condense_sync_pause &&
	!(zthr_has_waiters(t) \|\| zthr_iscancelled(t)))
	delay(1);

	dmu_tx_t *tx = dmu_tx_create_dd(spa_get_dsl(spa)->dp_mos_dir);
	dmu_tx_mark_netfree(tx);
	dmu_tx_hold_space(tx, 1);
	err = dmu_tx_assign(tx, TXG_NOWAIT \| TXG_NOTHROTTLE);
	if (err == 0) {
	/*
	* Prevent the condense zthr restarting before
	* the synctask completes.
	*/
	spa->spa_to_condense.syncing = B_TRUE;
	lca->spa = spa;
	lca->first_size = first_size;
	lca->next_size = next_size;
	dsl_sync_task_nowait(spa_get_dsl(spa),
	spa_livelist_condense_sync, lca, tx);
	dmu_tx_commit(tx);
	return;
	}
	}
	/*
	* Condensing can not continue: either it was externally stopped or
	* we were unable to assign to a tx because the pool has run out of
	* space. In the second case, we'll just end up trying to condense
	* again in a later txg.
	*/
	ASSERT(err != 0);
	bplist_clear(&lca->to_keep);
	bplist_destroy(&lca->to_keep);
	kmem_free(lca, sizeof (livelist_condense_arg_t));
	dmu_buf_rele(spa->spa_to_condense.ds->ds_dbuf, spa);
	spa->spa_to_condense.ds = NULL;
	if (err == EINTR)
	zfs_livelist_condense_zthr_cancel++;
	}

	/* ARGSUSED */
	/*
	* Check that there is something to condense but that a condense is not
	* already in progress and that condensing has not been cancelled.
	*/
	static boolean_t
	spa_livelist_condense_cb_check(void arg, zthr_t z)
	{
	spa_t *spa = arg;
	if ((spa->spa_to_condense.ds != NULL) &&
	(spa->spa_to_condense.syncing == B_FALSE) &&
	(spa->spa_to_condense.cancelled == B_FALSE)) {
	return (B_TRUE);
	}
	return (B_FALSE);
	}

	static void
	spa_start_livelist_condensing_thread(spa_t *spa)
	{
	spa->spa_to_condense.ds = NULL;
	spa->spa_to_condense.first = NULL;
	spa->spa_to_condense.next = NULL;
	spa->spa_to_condense.syncing = B_FALSE;
	spa->spa_to_condense.cancelled = B_FALSE;

	ASSERT3P(spa->spa_livelist_condense_zthr, ==, NULL);
	spa->spa_livelist_condense_zthr =
	zthr_create("z_livelist_condense",
	spa_livelist_condense_cb_check,
	spa_livelist_condense_cb, spa);
	}

	static void
	spa_spawn_aux_threads(spa_t *spa)
	{
	ASSERT(spa_writeable(spa));

	ASSERT(MUTEX_HELD(&spa_namespace_lock));

	spa_start_indirect_condensing_thread(spa);
	spa_start_livelist_destroy_thread(spa);
	spa_start_livelist_condensing_thread(spa);

	ASSERT3P(spa->spa_checkpoint_discard_zthr, ==, NULL);
	spa->spa_checkpoint_discard_zthr =
	zthr_create("z_checkpoint_discard",
	spa_checkpoint_discard_thread_check,
	spa_checkpoint_discard_thread, spa);
	}

	/*
	* Fix up config after a partly-completed split. This is done with the
	* ZPOOL_CONFIG_SPLIT nvlist. Both the splitting pool and the split-off
	* pool have that entry in their config, but only the splitting one contains
	* a list of all the guids of the vdevs that are being split off.
	*
	* This function determines what to do with that list: either rejoin
	* all the disks to the pool, or complete the splitting process. To attempt
	* the rejoin, each disk that is offlined is marked online again, and
	* we do a reopen() call. If the vdev label for every disk that was
	* marked online indicates it was successfully split off (VDEV_AUX_SPLIT_POOL)
	* then we call vdev_split() on each disk, and complete the split.
	*
	* Otherwise we leave the config alone, with all the vdevs in place in
	* the original pool.
	*/
	static void
	spa_try_repair(spa_t spa, nvlist_t config)
	{
	uint_t extracted;
	uint64_t *glist;
	uint_t i, gcount;
	nvlist_t *nvl;
	vdev_t **vd;
	boolean_t attempt_reopen;

	if (nvlist_lookup_nvlist(config, ZPOOL_CONFIG_SPLIT, &nvl) != 0)
	return;

	/* check that the config is complete */
	if (nvlist_lookup_uint64_array(nvl, ZPOOL_CONFIG_SPLIT_LIST,
	&glist, &gcount) != 0)
	return;

	vd = kmem_zalloc(gcount * sizeof (vdev_t *), KM_SLEEP);

	/* attempt to online all the vdevs & validate */
	attempt_reopen = B_TRUE;
	for (i = 0; i < gcount; i++) {
	if (glist[i] == 0) /* vdev is hole */
	continue;

	vd[i] = spa_lookup_by_guid(spa, glist[i], B_FALSE);
	if (vd[i] == NULL) {
	/*
	* Don't bother attempting to reopen the disks;
	* just do the split.
	*/
	attempt_reopen = B_FALSE;
	} else {
	/* attempt to re-online it */
	vd[i]->vdev_offline = B_FALSE;
	}
	}

	if (attempt_reopen) {
	vdev_reopen(spa->spa_root_vdev);

	/* check each device to see what state it's in */
	for (extracted = 0, i = 0; i < gcount; i++) {
	if (vd[i] != NULL &&
	vd[i]->vdev_stat.vs_aux != VDEV_AUX_SPLIT_POOL)
	break;
	++extracted;
	}
	}

	/*
	* If every disk has been moved to the new pool, or if we never
	* even attempted to look at them, then we split them off for
	* good.
	*/
	if (!attempt_reopen \|\| gcount == extracted) {
	for (i = 0; i < gcount; i++)
	if (vd[i] != NULL)
	vdev_split(vd[i]);
	vdev_reopen(spa->spa_root_vdev);
	}

	kmem_free(vd, gcount * sizeof (vdev_t *));
	}

	static int
	spa_load(spa_t *spa, spa_load_state_t state, spa_import_type_t type)
	{
	char *ereport = FM_EREPORT_ZFS_POOL;
	int error;

	spa->spa_load_state = state;
	(void) spa_import_progress_set_state(spa_guid(spa),
	spa_load_state(spa));

	gethrestime(&spa->spa_loaded_ts);
	error = spa_load_impl(spa, type, &ereport);

	/*
	* Don't count references from objsets that are already closed
	* and are making their way through the eviction process.
	*/
	spa_evicting_os_wait(spa);
	spa->spa_minref = zfs_refcount_count(&spa->spa_refcount);
	if (error) {
	if (error != EEXIST) {
	spa->spa_loaded_ts.tv_sec = 0;
	spa->spa_loaded_ts.tv_nsec = 0;
	}
	if (error != EBADF) {
	(void) zfs_ereport_post(ereport, spa,
	NULL, NULL, NULL, 0);
	}
	}
	spa->spa_load_state = error ? SPA_LOAD_ERROR : SPA_LOAD_NONE;
	spa->spa_ena = 0;

	(void) spa_import_progress_set_state(spa_guid(spa),
	spa_load_state(spa));

	return (error);
	}

	#ifdef ZFS_DEBUG
	/*
	* Count the number of per-vdev ZAPs associated with all of the vdevs in the
	* vdev tree rooted in the given vd, and ensure that each ZAP is present in the
	* spa's per-vdev ZAP list.
	*/
	static uint64_t
	vdev_count_verify_zaps(vdev_t *vd)
	{
	spa_t *spa = vd->vdev_spa;
	uint64_t total = 0;

	if (vd->vdev_top_zap != 0) {
	total++;
	ASSERT0(zap_lookup_int(spa->spa_meta_objset,
	spa->spa_all_vdev_zaps, vd->vdev_top_zap));
	}
	if (vd->vdev_leaf_zap != 0) {
	total++;
	ASSERT0(zap_lookup_int(spa->spa_meta_objset,
	spa->spa_all_vdev_zaps, vd->vdev_leaf_zap));
	}

	for (uint64_t i = 0; i < vd->vdev_children; i++) {
	total += vdev_count_verify_zaps(vd->vdev_child[i]);
	}

	return (total);
	}
	#endif

	/*
	* Determine whether the activity check is required.
	*/
	static boolean_t
	spa_activity_check_required(spa_t spa, uberblock_t ub, nvlist_t *label,
	nvlist_t *config)
	{
	uint64_t state = 0;
	uint64_t hostid = 0;
	uint64_t tryconfig_txg = 0;
	uint64_t tryconfig_timestamp = 0;
	uint16_t tryconfig_mmp_seq = 0;
	nvlist_t *nvinfo;

	if (nvlist_exists(config, ZPOOL_CONFIG_LOAD_INFO)) {
	nvinfo = fnvlist_lookup_nvlist(config, ZPOOL_CONFIG_LOAD_INFO);
	(void) nvlist_lookup_uint64(nvinfo, ZPOOL_CONFIG_MMP_TXG,
	&tryconfig_txg);
	(void) nvlist_lookup_uint64(config, ZPOOL_CONFIG_TIMESTAMP,
	&tryconfig_timestamp);
	(void) nvlist_lookup_uint16(nvinfo, ZPOOL_CONFIG_MMP_SEQ,
	&tryconfig_mmp_seq);
	}

	(void) nvlist_lookup_uint64(config, ZPOOL_CONFIG_POOL_STATE, &state);

	/*
	* Disable the MMP activity check - This is used by zdb which
	* is intended to be used on potentially active pools.
	*/
	if (spa->spa_import_flags & ZFS_IMPORT_SKIP_MMP)
	return (B_FALSE);

	/*
	* Skip the activity check when the MMP feature is disabled.
	*/
	if (ub->ub_mmp_magic == MMP_MAGIC && ub->ub_mmp_delay == 0)
	return (B_FALSE);

	/*
	* If the tryconfig_ values are nonzero, they are the results of an
	* earlier tryimport. If they all match the uberblock we just found,
	* then the pool has not changed and we return false so we do not test
	* a second time.
	*/
	if (tryconfig_txg && tryconfig_txg == ub->ub_txg &&
	tryconfig_timestamp && tryconfig_timestamp == ub->ub_timestamp &&
	tryconfig_mmp_seq && tryconfig_mmp_seq ==
	(MMP_SEQ_VALID(ub) ? MMP_SEQ(ub) : 0))
	return (B_FALSE);

	/*
	* Allow the activity check to be skipped when importing the pool
	* on the same host which last imported it. Since the hostid from
	* configuration may be stale use the one read from the label.
	*/
	if (nvlist_exists(label, ZPOOL_CONFIG_HOSTID))
	hostid = fnvlist_lookup_uint64(label, ZPOOL_CONFIG_HOSTID);

	if (hostid == spa_get_hostid(spa))
	return (B_FALSE);

	/*
	* Skip the activity test when the pool was cleanly exported.
	*/
	if (state != POOL_STATE_ACTIVE)
	return (B_FALSE);

	return (B_TRUE);
	}

	/*
	* Nanoseconds the activity check must watch for changes on-disk.
	*/
	static uint64_t
	spa_activity_check_duration(spa_t spa, uberblock_t ub)
	{
	uint64_t import_intervals = MAX(zfs_multihost_import_intervals, 1);
	uint64_t multihost_interval = MSEC2NSEC(
	MMP_INTERVAL_OK(zfs_multihost_interval));
	uint64_t import_delay = MAX(NANOSEC, import_intervals *
	multihost_interval);

	/*
	* Local tunables determine a minimum duration except for the case
	* where we know when the remote host will suspend the pool if MMP
	* writes do not land.
	*
	* See Big Theory comment at the top of mmp.c for the reasoning behind
	* these cases and times.
	*/

	ASSERT(MMP_IMPORT_SAFETY_FACTOR >= 100);

	if (MMP_INTERVAL_VALID(ub) && MMP_FAIL_INT_VALID(ub) &&
	MMP_FAIL_INT(ub) > 0) {

	/* MMP on remote host will suspend pool after failed writes */
	import_delay = MMP_FAIL_INT(ub) * MSEC2NSEC(MMP_INTERVAL(ub)) *
	MMP_IMPORT_SAFETY_FACTOR / 100;

	zfs_dbgmsg("fail_intvals>0 import_delay=%llu ub_mmp "
	"mmp_fails=%llu ub_mmp mmp_interval=%llu "
	"import_intervals=%u", import_delay, MMP_FAIL_INT(ub),
	MMP_INTERVAL(ub), import_intervals);

	} else if (MMP_INTERVAL_VALID(ub) && MMP_FAIL_INT_VALID(ub) &&
	MMP_FAIL_INT(ub) == 0) {

	/* MMP on remote host will never suspend pool */
	import_delay = MAX(import_delay, (MSEC2NSEC(MMP_INTERVAL(ub)) +
	ub->ub_mmp_delay) * import_intervals);

	zfs_dbgmsg("fail_intvals=0 import_delay=%llu ub_mmp "
	"mmp_interval=%llu ub_mmp_delay=%llu "
	"import_intervals=%u", import_delay, MMP_INTERVAL(ub),
	ub->ub_mmp_delay, import_intervals);

	} else if (MMP_VALID(ub)) {
	/*
	* zfs-0.7 compatibility case
	*/

	import_delay = MAX(import_delay, (multihost_interval +
	ub->ub_mmp_delay) * import_intervals);

	zfs_dbgmsg("import_delay=%llu ub_mmp_delay=%llu "
	"import_intervals=%u leaves=%u", import_delay,
	ub->ub_mmp_delay, import_intervals,
	vdev_count_leaves(spa));
	} else {
	/* Using local tunings is the only reasonable option */
	zfs_dbgmsg("pool last imported on non-MMP aware "
	"host using import_delay=%llu multihost_interval=%llu "
	"import_intervals=%u", import_delay, multihost_interval,
	import_intervals);
	}

	return (import_delay);
	}

	/*
	* Perform the import activity check. If the user canceled the import or
	* we detected activity then fail.
	*/
	static int
	spa_activity_check(spa_t spa, uberblock_t ub, nvlist_t *config)
	{
	uint64_t txg = ub->ub_txg;
	uint64_t timestamp = ub->ub_timestamp;
	uint64_t mmp_config = ub->ub_mmp_config;
	uint16_t mmp_seq = MMP_SEQ_VALID(ub) ? MMP_SEQ(ub) : 0;
	uint64_t import_delay;
	hrtime_t import_expire;
	nvlist_t *mmp_label = NULL;
	vdev_t *rvd = spa->spa_root_vdev;
	kcondvar_t cv;
	kmutex_t mtx;
	int error = 0;

	cv_init(&cv, NULL, CV_DEFAULT, NULL);
	mutex_init(&mtx, NULL, MUTEX_DEFAULT, NULL);
	mutex_enter(&mtx);

	/*
	* If ZPOOL_CONFIG_MMP_TXG is present an activity check was performed
	* during the earlier tryimport. If the txg recorded there is 0 then
	* the pool is known to be active on another host.
	*
	* Otherwise, the pool might be in use on another host. Check for
	* changes in the uberblocks on disk if necessary.
	*/
	if (nvlist_exists(config, ZPOOL_CONFIG_LOAD_INFO)) {
	nvlist_t *nvinfo = fnvlist_lookup_nvlist(config,
	ZPOOL_CONFIG_LOAD_INFO);

	if (nvlist_exists(nvinfo, ZPOOL_CONFIG_MMP_TXG) &&
	fnvlist_lookup_uint64(nvinfo, ZPOOL_CONFIG_MMP_TXG) == 0) {
	vdev_uberblock_load(rvd, ub, &mmp_label);
	error = SET_ERROR(EREMOTEIO);
	goto out;
	}
	}

	import_delay = spa_activity_check_duration(spa, ub);

	/* Add a small random factor in case of simultaneous imports (0-25%) */
	import_delay += import_delay * spa_get_random(250) / 1000;

	import_expire = gethrtime() + import_delay;

	while (gethrtime() < import_expire) {
	(void) spa_import_progress_set_mmp_check(spa_guid(spa),
	NSEC2SEC(import_expire - gethrtime()));

	vdev_uberblock_load(rvd, ub, &mmp_label);

	if (txg != ub->ub_txg \|\| timestamp != ub->ub_timestamp \|\|
	mmp_seq != (MMP_SEQ_VALID(ub) ? MMP_SEQ(ub) : 0)) {
	zfs_dbgmsg("multihost activity detected "
	"txg %llu ub_txg %llu "
	"timestamp %llu ub_timestamp %llu "
	"mmp_config %#llx ub_mmp_config %#llx",
	txg, ub->ub_txg, timestamp, ub->ub_timestamp,
	mmp_config, ub->ub_mmp_config);

	error = SET_ERROR(EREMOTEIO);
	break;
	}

	if (mmp_label) {
	nvlist_free(mmp_label);
	mmp_label = NULL;
	}

	error = cv_timedwait_sig(&cv, &mtx, ddi_get_lbolt() + hz);
	if (error != -1) {
	error = SET_ERROR(EINTR);
	break;
	}
	error = 0;
	}

	out:
	mutex_exit(&mtx);
	mutex_destroy(&mtx);
	cv_destroy(&cv);

	/*
	* If the pool is determined to be active store the status in the
	* spa->spa_load_info nvlist. If the remote hostname or hostid are
	* available from configuration read from disk store them as well.
	* This allows 'zpool import' to generate a more useful message.
	*
	* ZPOOL_CONFIG_MMP_STATE - observed pool status (mandatory)
	* ZPOOL_CONFIG_MMP_HOSTNAME - hostname from the active pool
	* ZPOOL_CONFIG_MMP_HOSTID - hostid from the active pool
	*/
	if (error == EREMOTEIO) {
	char *hostname = "<unknown>";
	uint64_t hostid = 0;

	if (mmp_label) {
	if (nvlist_exists(mmp_label, ZPOOL_CONFIG_HOSTNAME)) {
	hostname = fnvlist_lookup_string(mmp_label,
	ZPOOL_CONFIG_HOSTNAME);
	fnvlist_add_string(spa->spa_load_info,
	ZPOOL_CONFIG_MMP_HOSTNAME, hostname);
	}

	if (nvlist_exists(mmp_label, ZPOOL_CONFIG_HOSTID)) {
	hostid = fnvlist_lookup_uint64(mmp_label,
	ZPOOL_CONFIG_HOSTID);
	fnvlist_add_uint64(spa->spa_load_info,
	ZPOOL_CONFIG_MMP_HOSTID, hostid);
	}
	}

	fnvlist_add_uint64(spa->spa_load_info,
	ZPOOL_CONFIG_MMP_STATE, MMP_STATE_ACTIVE);
	fnvlist_add_uint64(spa->spa_load_info,
	ZPOOL_CONFIG_MMP_TXG, 0);

	error = spa_vdev_err(rvd, VDEV_AUX_ACTIVE, EREMOTEIO);
	}

	if (mmp_label)
	nvlist_free(mmp_label);

	return (error);
	}

	static int
	spa_verify_host(spa_t spa, nvlist_t mos_config)
	{
	uint64_t hostid;
	char *hostname;
	uint64_t myhostid = 0;

	if (!spa_is_root(spa) && nvlist_lookup_uint64(mos_config,
	ZPOOL_CONFIG_HOSTID, &hostid) == 0) {
	hostname = fnvlist_lookup_string(mos_config,
	ZPOOL_CONFIG_HOSTNAME);

	myhostid = zone_get_hostid(NULL);

	if (hostid != 0 && myhostid != 0 && hostid != myhostid) {
	cmn_err(CE_WARN, "pool '%s' could not be "
	"loaded as it was last accessed by "
	"another system (host: %s hostid: 0x%llx). "
	"See: https://openzfs.github.io/openzfs-docs/msg/"
	"ZFS-8000-EY",
	spa_name(spa), hostname, (u_longlong_t)hostid);
	spa_load_failed(spa, "hostid verification failed: pool "
	"last accessed by host: %s (hostid: 0x%llx)",
	hostname, (u_longlong_t)hostid);
	return (SET_ERROR(EBADF));
	}
	}

	return (0);
	}

	static int
	spa_ld_parse_config(spa_t *spa, spa_import_type_t type)
	{
	int error = 0;
	nvlist_t nvtree, nvl, *config = spa->spa_config;
	int parse;
	vdev_t *rvd;
	uint64_t pool_guid;
	char *comment;

	/*
	* Versioning wasn't explicitly added to the label until later, so if
	* it's not present treat it as the initial version.
	*/
	if (nvlist_lookup_uint64(config, ZPOOL_CONFIG_VERSION,
	&spa->spa_ubsync.ub_version) != 0)
	spa->spa_ubsync.ub_version = SPA_VERSION_INITIAL;

	if (nvlist_lookup_uint64(config, ZPOOL_CONFIG_POOL_GUID, &pool_guid)) {
	spa_load_failed(spa, "invalid config provided: '%s' missing",
	ZPOOL_CONFIG_POOL_GUID);
	return (SET_ERROR(EINVAL));
	}

	/*
	* If we are doing an import, ensure that the pool is not already
	* imported by checking if its pool guid already exists in the
	* spa namespace.
	*
	* The only case that we allow an already imported pool to be
	* imported again, is when the pool is checkpointed and we want to
	* look at its checkpointed state from userland tools like zdb.
	*/
	#ifdef _KERNEL
	if ((spa->spa_load_state == SPA_LOAD_IMPORT \|\|
	spa->spa_load_state == SPA_LOAD_TRYIMPORT) &&
	spa_guid_exists(pool_guid, 0)) {
	#else
	if ((spa->spa_load_state == SPA_LOAD_IMPORT \|\|
	spa->spa_load_state == SPA_LOAD_TRYIMPORT) &&
	spa_guid_exists(pool_guid, 0) &&
	!spa_importing_readonly_checkpoint(spa)) {
	#endif
	spa_load_failed(spa, "a pool with guid %llu is already open",
	(u_longlong_t)pool_guid);
	return (SET_ERROR(EEXIST));
	}

	spa->spa_config_guid = pool_guid;

	nvlist_free(spa->spa_load_info);
	spa->spa_load_info = fnvlist_alloc();

	ASSERT(spa->spa_comment == NULL);
	if (nvlist_lookup_string(config, ZPOOL_CONFIG_COMMENT, &comment) == 0)
	spa->spa_comment = spa_strdup(comment);

	(void) nvlist_lookup_uint64(config, ZPOOL_CONFIG_POOL_TXG,
	&spa->spa_config_txg);

	if (nvlist_lookup_nvlist(config, ZPOOL_CONFIG_SPLIT, &nvl) == 0)
	spa->spa_config_splitting = fnvlist_dup(nvl);

	if (nvlist_lookup_nvlist(config, ZPOOL_CONFIG_VDEV_TREE, &nvtree)) {
	spa_load_failed(spa, "invalid config provided: '%s' missing",
	ZPOOL_CONFIG_VDEV_TREE);
	return (SET_ERROR(EINVAL));
	}

	/*
	* Create "The Godfather" zio to hold all async IOs
	*/
	spa->spa_async_zio_root = kmem_alloc(max_ncpus * sizeof (void *),
	KM_SLEEP);
	for (int i = 0; i < max_ncpus; i++) {
	spa->spa_async_zio_root[i] = zio_root(spa, NULL, NULL,
	ZIO_FLAG_CANFAIL \| ZIO_FLAG_SPECULATIVE \|
	ZIO_FLAG_GODFATHER);
	}

	/*
	* Parse the configuration into a vdev tree. We explicitly set the
	* value that will be returned by spa_version() since parsing the
	* configuration requires knowing the version number.
	*/
	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);
	parse = (type == SPA_IMPORT_EXISTING ?
	VDEV_ALLOC_LOAD : VDEV_ALLOC_SPLIT);
	error = spa_config_parse(spa, &rvd, nvtree, NULL, 0, parse);
	spa_config_exit(spa, SCL_ALL, FTAG);

	if (error != 0) {
	spa_load_failed(spa, "unable to parse config [error=%d]",
	error);
	return (error);
	}

	ASSERT(spa->spa_root_vdev == rvd);
	ASSERT3U(spa->spa_min_ashift, >=, SPA_MINBLOCKSHIFT);
	ASSERT3U(spa->spa_max_ashift, <=, SPA_MAXBLOCKSHIFT);

	if (type != SPA_IMPORT_ASSEMBLE) {
	ASSERT(spa_guid(spa) == pool_guid);
	}

	return (0);
	}

	/*
	* Recursively open all vdevs in the vdev tree. This function is called twice:
	* first with the untrusted config, then with the trusted config.
	*/
	static int
	spa_ld_open_vdevs(spa_t *spa)
	{
	int error = 0;

	/*
	* spa_missing_tvds_allowed defines how many top-level vdevs can be
	* missing/unopenable for the root vdev to be still considered openable.
	*/
	if (spa->spa_trust_config) {
	spa->spa_missing_tvds_allowed = zfs_max_missing_tvds;
	} else if (spa->spa_config_source == SPA_CONFIG_SRC_CACHEFILE) {
	spa->spa_missing_tvds_allowed = zfs_max_missing_tvds_cachefile;
	} else if (spa->spa_config_source == SPA_CONFIG_SRC_SCAN) {
	spa->spa_missing_tvds_allowed = zfs_max_missing_tvds_scan;
	} else {
	spa->spa_missing_tvds_allowed = 0;
	}

	spa->spa_missing_tvds_allowed =
	MAX(zfs_max_missing_tvds, spa->spa_missing_tvds_allowed);

	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);
	error = vdev_open(spa->spa_root_vdev);
	spa_config_exit(spa, SCL_ALL, FTAG);

	if (spa->spa_missing_tvds != 0) {
	spa_load_note(spa, "vdev tree has %lld missing top-level "
	"vdevs.", (u_longlong_t)spa->spa_missing_tvds);
	if (spa->spa_trust_config && (spa->spa_mode & SPA_MODE_WRITE)) {
	/*
	* Although theoretically we could allow users to open
	* incomplete pools in RW mode, we'd need to add a lot
	* of extra logic (e.g. adjust pool space to account
	* for missing vdevs).
	* This limitation also prevents users from accidentally
	* opening the pool in RW mode during data recovery and
	* damaging it further.
	*/
	spa_load_note(spa, "pools with missing top-level "
	"vdevs can only be opened in read-only mode.");
	error = SET_ERROR(ENXIO);
	} else {
	spa_load_note(spa, "current settings allow for maximum "
	"%lld missing top-level vdevs at this stage.",
	(u_longlong_t)spa->spa_missing_tvds_allowed);
	}
	}
	if (error != 0) {
	spa_load_failed(spa, "unable to open vdev tree [error=%d]",
	error);
	}
	if (spa->spa_missing_tvds != 0 \|\| error != 0)
	vdev_dbgmsg_print_tree(spa->spa_root_vdev, 2);

	return (error);
	}

	/*
	* We need to validate the vdev labels against the configuration that
	* we have in hand. This function is called twice: first with an untrusted
	* config, then with a trusted config. The validation is more strict when the
	* config is trusted.
	*/
	static int
	spa_ld_validate_vdevs(spa_t *spa)
	{
	int error = 0;
	vdev_t *rvd = spa->spa_root_vdev;

	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);
	error = vdev_validate(rvd);
	spa_config_exit(spa, SCL_ALL, FTAG);

	if (error != 0) {
	spa_load_failed(spa, "vdev_validate failed [error=%d]", error);
	return (error);
	}

	if (rvd->vdev_state <= VDEV_STATE_CANT_OPEN) {
	spa_load_failed(spa, "cannot open vdev tree after invalidating "
	"some vdevs");
	vdev_dbgmsg_print_tree(rvd, 2);
	return (SET_ERROR(ENXIO));
	}

	return (0);
	}

	static void
	spa_ld_select_uberblock_done(spa_t spa, uberblock_t ub)
	{
	spa->spa_state = POOL_STATE_ACTIVE;
	spa->spa_ubsync = spa->spa_uberblock;
	spa->spa_verify_min_txg = spa->spa_extreme_rewind ?
	TXG_INITIAL - 1 : spa_last_synced_txg(spa) - TXG_DEFER_SIZE - 1;
	spa->spa_first_txg = spa->spa_last_ubsync_txg ?
	spa->spa_last_ubsync_txg : spa_last_synced_txg(spa) + 1;
	spa->spa_claim_max_txg = spa->spa_first_txg;
	spa->spa_prev_software_version = ub->ub_software_version;
	}

	static int
	spa_ld_select_uberblock(spa_t *spa, spa_import_type_t type)
	{
	vdev_t *rvd = spa->spa_root_vdev;
	nvlist_t *label;
	uberblock_t *ub = &spa->spa_uberblock;
	boolean_t activity_check = B_FALSE;

	/*
	* If we are opening the checkpointed state of the pool by
	* rewinding to it, at this point we will have written the
	* checkpointed uberblock to the vdev labels, so searching
	* the labels will find the right uberblock. However, if
	* we are opening the checkpointed state read-only, we have
	* not modified the labels. Therefore, we must ignore the
	* labels and continue using the spa_uberblock that was set
	* by spa_ld_checkpoint_rewind.
	*
	* Note that it would be fine to ignore the labels when
	* rewinding (opening writeable) as well. However, if we
	* crash just after writing the labels, we will end up
	* searching the labels. Doing so in the common case means
	* that this code path gets exercised normally, rather than
	* just in the edge case.
	*/
	if (ub->ub_checkpoint_txg != 0 &&
	spa_importing_readonly_checkpoint(spa)) {
	spa_ld_select_uberblock_done(spa, ub);
	return (0);
	}

	/*
	* Find the best uberblock.
	*/
	vdev_uberblock_load(rvd, ub, &label);

	/*
	* If we weren't able to find a single valid uberblock, return failure.
	*/
	if (ub->ub_txg == 0) {
	nvlist_free(label);
	spa_load_failed(spa, "no valid uberblock found");
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, ENXIO));
	}

	if (spa->spa_load_max_txg != UINT64_MAX) {
	(void) spa_import_progress_set_max_txg(spa_guid(spa),
	(u_longlong_t)spa->spa_load_max_txg);
	}
	spa_load_note(spa, "using uberblock with txg=%llu",
	(u_longlong_t)ub->ub_txg);


	/*
	* For pools which have the multihost property on determine if the
	* pool is truly inactive and can be safely imported. Prevent
	* hosts which don't have a hostid set from importing the pool.
	*/
	activity_check = spa_activity_check_required(spa, ub, label,
	spa->spa_config);
	if (activity_check) {
	if (ub->ub_mmp_magic == MMP_MAGIC && ub->ub_mmp_delay &&
	spa_get_hostid(spa) == 0) {
	nvlist_free(label);
	fnvlist_add_uint64(spa->spa_load_info,
	ZPOOL_CONFIG_MMP_STATE, MMP_STATE_NO_HOSTID);
	return (spa_vdev_err(rvd, VDEV_AUX_ACTIVE, EREMOTEIO));
	}

	int error = spa_activity_check(spa, ub, spa->spa_config);
	if (error) {
	nvlist_free(label);
	return (error);
	}

	fnvlist_add_uint64(spa->spa_load_info,
	ZPOOL_CONFIG_MMP_STATE, MMP_STATE_INACTIVE);
	fnvlist_add_uint64(spa->spa_load_info,
	ZPOOL_CONFIG_MMP_TXG, ub->ub_txg);
	fnvlist_add_uint16(spa->spa_load_info,
	ZPOOL_CONFIG_MMP_SEQ,
	(MMP_SEQ_VALID(ub) ? MMP_SEQ(ub) : 0));
	}

	/*
	* If the pool has an unsupported version we can't open it.
	*/
	if (!SPA_VERSION_IS_SUPPORTED(ub->ub_version)) {
	nvlist_free(label);
	spa_load_failed(spa, "version %llu is not supported",
	(u_longlong_t)ub->ub_version);
	return (spa_vdev_err(rvd, VDEV_AUX_VERSION_NEWER, ENOTSUP));
	}

	if (ub->ub_version >= SPA_VERSION_FEATURES) {
	nvlist_t *features;

	/*
	* If we weren't able to find what's necessary for reading the
	* MOS in the label, return failure.
	*/
	if (label == NULL) {
	spa_load_failed(spa, "label config unavailable");
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA,
	ENXIO));
	}

	if (nvlist_lookup_nvlist(label, ZPOOL_CONFIG_FEATURES_FOR_READ,
	&features) != 0) {
	nvlist_free(label);
	spa_load_failed(spa, "invalid label: '%s' missing",
	ZPOOL_CONFIG_FEATURES_FOR_READ);
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA,
	ENXIO));
	}

	/*
	* Update our in-core representation with the definitive values
	* from the label.
	*/
	nvlist_free(spa->spa_label_features);
	VERIFY(nvlist_dup(features, &spa->spa_label_features, 0) == 0);
	}

	nvlist_free(label);

	/*
	* Look through entries in the label nvlist's features_for_read. If
	* there is a feature listed there which we don't understand then we
	* cannot open a pool.
	*/
	if (ub->ub_version >= SPA_VERSION_FEATURES) {
	nvlist_t *unsup_feat;

	VERIFY(nvlist_alloc(&unsup_feat, NV_UNIQUE_NAME, KM_SLEEP) ==
	0);

	for (nvpair_t *nvp = nvlist_next_nvpair(spa->spa_label_features,
	NULL); nvp != NULL;
	nvp = nvlist_next_nvpair(spa->spa_label_features, nvp)) {
	if (!zfeature_is_supported(nvpair_name(nvp))) {
	VERIFY(nvlist_add_string(unsup_feat,
	nvpair_name(nvp), "") == 0);
	}
	}

	if (!nvlist_empty(unsup_feat)) {
	VERIFY(nvlist_add_nvlist(spa->spa_load_info,
	ZPOOL_CONFIG_UNSUP_FEAT, unsup_feat) == 0);
	nvlist_free(unsup_feat);
	spa_load_failed(spa, "some features are unsupported");
	return (spa_vdev_err(rvd, VDEV_AUX_UNSUP_FEAT,
	ENOTSUP));
	}

	nvlist_free(unsup_feat);
	}

	if (type != SPA_IMPORT_ASSEMBLE && spa->spa_config_splitting) {
	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);
	spa_try_repair(spa, spa->spa_config);
	spa_config_exit(spa, SCL_ALL, FTAG);
	nvlist_free(spa->spa_config_splitting);
	spa->spa_config_splitting = NULL;
	}

	/*
	* Initialize internal SPA structures.
	*/
	spa_ld_select_uberblock_done(spa, ub);

	return (0);
	}

	static int
	spa_ld_open_rootbp(spa_t *spa)
	{
	int error = 0;
	vdev_t *rvd = spa->spa_root_vdev;

	error = dsl_pool_init(spa, spa->spa_first_txg, &spa->spa_dsl_pool);
	if (error != 0) {
	spa_load_failed(spa, "unable to open rootbp in dsl_pool_init "
	"[error=%d]", error);
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));
	}
	spa->spa_meta_objset = spa->spa_dsl_pool->dp_meta_objset;

	return (0);
	}

	static int
	spa_ld_trusted_config(spa_t *spa, spa_import_type_t type,
	boolean_t reloading)
	{
	vdev_t mrvd, rvd = spa->spa_root_vdev;
	nvlist_t nv, mos_config, *policy;
	int error = 0, copy_error;
	uint64_t healthy_tvds, healthy_tvds_mos;
	uint64_t mos_config_txg;

	if (spa_dir_prop(spa, DMU_POOL_CONFIG, &spa->spa_config_object, B_TRUE)
	!= 0)
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));

	/*
	* If we're assembling a pool from a split, the config provided is
	* already trusted so there is nothing to do.
	*/
	if (type == SPA_IMPORT_ASSEMBLE)
	return (0);

	healthy_tvds = spa_healthy_core_tvds(spa);

	if (load_nvlist(spa, spa->spa_config_object, &mos_config)
	!= 0) {
	spa_load_failed(spa, "unable to retrieve MOS config");
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));
	}

	/*
	* If we are doing an open, pool owner wasn't verified yet, thus do
	* the verification here.
	*/
	if (spa->spa_load_state == SPA_LOAD_OPEN) {
	error = spa_verify_host(spa, mos_config);
	if (error != 0) {
	nvlist_free(mos_config);
	return (error);
	}
	}

	nv = fnvlist_lookup_nvlist(mos_config, ZPOOL_CONFIG_VDEV_TREE);

	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);

	/*
	* Build a new vdev tree from the trusted config
	*/
	VERIFY(spa_config_parse(spa, &mrvd, nv, NULL, 0, VDEV_ALLOC_LOAD) == 0);

	/*
	* Vdev paths in the MOS may be obsolete. If the untrusted config was
	* obtained by scanning /dev/dsk, then it will have the right vdev
	* paths. We update the trusted MOS config with this information.
	* We first try to copy the paths with vdev_copy_path_strict, which
	* succeeds only when both configs have exactly the same vdev tree.
	* If that fails, we fall back to a more flexible method that has a
	* best effort policy.
	*/
	copy_error = vdev_copy_path_strict(rvd, mrvd);
	if (copy_error != 0 \|\| spa_load_print_vdev_tree) {
	spa_load_note(spa, "provided vdev tree:");
	vdev_dbgmsg_print_tree(rvd, 2);
	spa_load_note(spa, "MOS vdev tree:");
	vdev_dbgmsg_print_tree(mrvd, 2);
	}
	if (copy_error != 0) {
	spa_load_note(spa, "vdev_copy_path_strict failed, falling "
	"back to vdev_copy_path_relaxed");
	vdev_copy_path_relaxed(rvd, mrvd);
	}

	vdev_close(rvd);
	vdev_free(rvd);
	spa->spa_root_vdev = mrvd;
	rvd = mrvd;
	spa_config_exit(spa, SCL_ALL, FTAG);

	/*
	* We will use spa_config if we decide to reload the spa or if spa_load
	* fails and we rewind. We must thus regenerate the config using the
	* MOS information with the updated paths. ZPOOL_LOAD_POLICY is used to
	* pass settings on how to load the pool and is not stored in the MOS.
	* We copy it over to our new, trusted config.
	*/
	mos_config_txg = fnvlist_lookup_uint64(mos_config,
	ZPOOL_CONFIG_POOL_TXG);
	nvlist_free(mos_config);
	mos_config = spa_config_generate(spa, NULL, mos_config_txg, B_FALSE);
	if (nvlist_lookup_nvlist(spa->spa_config, ZPOOL_LOAD_POLICY,
	&policy) == 0)
	fnvlist_add_nvlist(mos_config, ZPOOL_LOAD_POLICY, policy);
	spa_config_set(spa, mos_config);
	spa->spa_config_source = SPA_CONFIG_SRC_MOS;

	/*
	* Now that we got the config from the MOS, we should be more strict
	* in checking blkptrs and can make assumptions about the consistency
	* of the vdev tree. spa_trust_config must be set to true before opening
	* vdevs in order for them to be writeable.
	*/
	spa->spa_trust_config = B_TRUE;

	/*
	* Open and validate the new vdev tree
	*/
	error = spa_ld_open_vdevs(spa);
	if (error != 0)
	return (error);

	error = spa_ld_validate_vdevs(spa);
	if (error != 0)
	return (error);

	if (copy_error != 0 \|\| spa_load_print_vdev_tree) {
	spa_load_note(spa, "final vdev tree:");
	vdev_dbgmsg_print_tree(rvd, 2);
	}

	if (spa->spa_load_state != SPA_LOAD_TRYIMPORT &&
	!spa->spa_extreme_rewind && zfs_max_missing_tvds == 0) {
	/*
	* Sanity check to make sure that we are indeed loading the
	* latest uberblock. If we missed SPA_SYNC_MIN_VDEVS tvds
	* in the config provided and they happened to be the only ones
	* to have the latest uberblock, we could involuntarily perform
	* an extreme rewind.
	*/
	healthy_tvds_mos = spa_healthy_core_tvds(spa);
	if (healthy_tvds_mos - healthy_tvds >=
	SPA_SYNC_MIN_VDEVS) {
	spa_load_note(spa, "config provided misses too many "
	"top-level vdevs compared to MOS (%lld vs %lld). ",
	(u_longlong_t)healthy_tvds,
	(u_longlong_t)healthy_tvds_mos);
	spa_load_note(spa, "vdev tree:");
	vdev_dbgmsg_print_tree(rvd, 2);
	if (reloading) {
	spa_load_failed(spa, "config was already "
	"provided from MOS. Aborting.");
	return (spa_vdev_err(rvd,
	VDEV_AUX_CORRUPT_DATA, EIO));
	}
	spa_load_note(spa, "spa must be reloaded using MOS "
	"config");
	return (SET_ERROR(EAGAIN));
	}
	}

	error = spa_check_for_missing_logs(spa);
	if (error != 0)
	return (spa_vdev_err(rvd, VDEV_AUX_BAD_GUID_SUM, ENXIO));

	if (rvd->vdev_guid_sum != spa->spa_uberblock.ub_guid_sum) {
	spa_load_failed(spa, "uberblock guid sum doesn't match MOS "
	"guid sum (%llu != %llu)",
	(u_longlong_t)spa->spa_uberblock.ub_guid_sum,
	(u_longlong_t)rvd->vdev_guid_sum);
	return (spa_vdev_err(rvd, VDEV_AUX_BAD_GUID_SUM,
	ENXIO));
	}

	return (0);
	}

	static int
	spa_ld_open_indirect_vdev_metadata(spa_t *spa)
	{
	int error = 0;
	vdev_t *rvd = spa->spa_root_vdev;

	/*
	* Everything that we read before spa_remove_init() must be stored
	* on concreted vdevs. Therefore we do this as early as possible.
	*/
	error = spa_remove_init(spa);
	if (error != 0) {
	spa_load_failed(spa, "spa_remove_init failed [error=%d]",
	error);
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));
	}

	/*
	* Retrieve information needed to condense indirect vdev mappings.
	*/
	error = spa_condense_init(spa);
	if (error != 0) {
	spa_load_failed(spa, "spa_condense_init failed [error=%d]",
	error);
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, error));
	}

	return (0);
	}

	static int
	spa_ld_check_features(spa_t spa, boolean_t missing_feat_writep)
	{
	int error = 0;
	vdev_t *rvd = spa->spa_root_vdev;

	if (spa_version(spa) >= SPA_VERSION_FEATURES) {
	boolean_t missing_feat_read = B_FALSE;
	nvlist_t unsup_feat, enabled_feat;

	if (spa_dir_prop(spa, DMU_POOL_FEATURES_FOR_READ,
	&spa->spa_feat_for_read_obj, B_TRUE) != 0) {
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));
	}

	if (spa_dir_prop(spa, DMU_POOL_FEATURES_FOR_WRITE,
	&spa->spa_feat_for_write_obj, B_TRUE) != 0) {
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));
	}

	if (spa_dir_prop(spa, DMU_POOL_FEATURE_DESCRIPTIONS,
	&spa->spa_feat_desc_obj, B_TRUE) != 0) {
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));
	}

	enabled_feat = fnvlist_alloc();
	unsup_feat = fnvlist_alloc();

	if (!spa_features_check(spa, B_FALSE,
	unsup_feat, enabled_feat))
	missing_feat_read = B_TRUE;

	if (spa_writeable(spa) \|\|
	spa->spa_load_state == SPA_LOAD_TRYIMPORT) {
	if (!spa_features_check(spa, B_TRUE,
	unsup_feat, enabled_feat)) {
	*missing_feat_writep = B_TRUE;
	}
	}

	fnvlist_add_nvlist(spa->spa_load_info,
	ZPOOL_CONFIG_ENABLED_FEAT, enabled_feat);

	if (!nvlist_empty(unsup_feat)) {
	fnvlist_add_nvlist(spa->spa_load_info,
	ZPOOL_CONFIG_UNSUP_FEAT, unsup_feat);
	}

	fnvlist_free(enabled_feat);
	fnvlist_free(unsup_feat);

	if (!missing_feat_read) {
	fnvlist_add_boolean(spa->spa_load_info,
	ZPOOL_CONFIG_CAN_RDONLY);
	}

	/*
	* If the state is SPA_LOAD_TRYIMPORT, our objective is
	* twofold: to determine whether the pool is available for
	* import in read-write mode and (if it is not) whether the
	* pool is available for import in read-only mode. If the pool
	* is available for import in read-write mode, it is displayed
	* as available in userland; if it is not available for import
	* in read-only mode, it is displayed as unavailable in
	* userland. If the pool is available for import in read-only
	* mode but not read-write mode, it is displayed as unavailable
	* in userland with a special note that the pool is actually
	* available for open in read-only mode.
	*
	* As a result, if the state is SPA_LOAD_TRYIMPORT and we are
	* missing a feature for write, we must first determine whether
	* the pool can be opened read-only before returning to
	* userland in order to know whether to display the
	* abovementioned note.
	*/
	if (missing_feat_read \|\| (*missing_feat_writep &&
	spa_writeable(spa))) {
	spa_load_failed(spa, "pool uses unsupported features");
	return (spa_vdev_err(rvd, VDEV_AUX_UNSUP_FEAT,
	ENOTSUP));
	}

	/*
	* Load refcounts for ZFS features from disk into an in-memory
	* cache during SPA initialization.
	*/
	for (spa_feature_t i = 0; i < SPA_FEATURES; i++) {
	uint64_t refcount;

	error = feature_get_refcount_from_disk(spa,
	&spa_feature_table[i], &refcount);
	if (error == 0) {
	spa->spa_feat_refcount_cache[i] = refcount;
	} else if (error == ENOTSUP) {
	spa->spa_feat_refcount_cache[i] =
	SPA_FEATURE_DISABLED;
	} else {
	spa_load_failed(spa, "error getting refcount "
	"for feature %s [error=%d]",
	spa_feature_table[i].fi_guid, error);
	return (spa_vdev_err(rvd,
	VDEV_AUX_CORRUPT_DATA, EIO));
	}
	}
	}

	if (spa_feature_is_active(spa, SPA_FEATURE_ENABLED_TXG)) {
	if (spa_dir_prop(spa, DMU_POOL_FEATURE_ENABLED_TXG,
	&spa->spa_feat_enabled_txg_obj, B_TRUE) != 0)
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));
	}

	/*
	* Encryption was added before bookmark_v2, even though bookmark_v2
	* is now a dependency. If this pool has encryption enabled without
	* bookmark_v2, trigger an errata message.
	*/
	if (spa_feature_is_enabled(spa, SPA_FEATURE_ENCRYPTION) &&
	!spa_feature_is_enabled(spa, SPA_FEATURE_BOOKMARK_V2)) {
	spa->spa_errata = ZPOOL_ERRATA_ZOL_8308_ENCRYPTION;
	}

	return (0);
	}

	static int
	spa_ld_load_special_directories(spa_t *spa)
	{
	int error = 0;
	vdev_t *rvd = spa->spa_root_vdev;

	spa->spa_is_initializing = B_TRUE;
	error = dsl_pool_open(spa->spa_dsl_pool);
	spa->spa_is_initializing = B_FALSE;
	if (error != 0) {
	spa_load_failed(spa, "dsl_pool_open failed [error=%d]", error);
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));
	}

	return (0);
	}

	static int
	spa_ld_get_props(spa_t *spa)
	{
	int error = 0;
	uint64_t obj;
	vdev_t *rvd = spa->spa_root_vdev;

	/* Grab the checksum salt from the MOS. */
	error = zap_lookup(spa->spa_meta_objset, DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_CHECKSUM_SALT, 1,
	sizeof (spa->spa_cksum_salt.zcs_bytes),
	spa->spa_cksum_salt.zcs_bytes);
	if (error == ENOENT) {
	/* Generate a new salt for subsequent use */
	(void) random_get_pseudo_bytes(spa->spa_cksum_salt.zcs_bytes,
	sizeof (spa->spa_cksum_salt.zcs_bytes));
	} else if (error != 0) {
	spa_load_failed(spa, "unable to retrieve checksum salt from "
	"MOS [error=%d]", error);
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));
	}

	if (spa_dir_prop(spa, DMU_POOL_SYNC_BPOBJ, &obj, B_TRUE) != 0)
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));
	error = bpobj_open(&spa->spa_deferred_bpobj, spa->spa_meta_objset, obj);
	if (error != 0) {
	spa_load_failed(spa, "error opening deferred-frees bpobj "
	"[error=%d]", error);
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));
	}

	/*
	* Load the bit that tells us to use the new accounting function
	* (raid-z deflation). If we have an older pool, this will not
	* be present.
	*/
	error = spa_dir_prop(spa, DMU_POOL_DEFLATE, &spa->spa_deflate, B_FALSE);
	if (error != 0 && error != ENOENT)
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));

	error = spa_dir_prop(spa, DMU_POOL_CREATION_VERSION,
	&spa->spa_creation_version, B_FALSE);
	if (error != 0 && error != ENOENT)
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));

	/*
	* Load the persistent error log. If we have an older pool, this will
	* not be present.
	*/
	error = spa_dir_prop(spa, DMU_POOL_ERRLOG_LAST, &spa->spa_errlog_last,
	B_FALSE);
	if (error != 0 && error != ENOENT)
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));

	error = spa_dir_prop(spa, DMU_POOL_ERRLOG_SCRUB,
	&spa->spa_errlog_scrub, B_FALSE);
	if (error != 0 && error != ENOENT)
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));

	/*
	* Load the livelist deletion field. If a livelist is queued for
	* deletion, indicate that in the spa
	*/
	error = spa_dir_prop(spa, DMU_POOL_DELETED_CLONES,
	&spa->spa_livelists_to_delete, B_FALSE);
	if (error != 0 && error != ENOENT)
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));

	/*
	* Load the history object. If we have an older pool, this
	* will not be present.
	*/
	error = spa_dir_prop(spa, DMU_POOL_HISTORY, &spa->spa_history, B_FALSE);
	if (error != 0 && error != ENOENT)
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));

	/*
	* Load the per-vdev ZAP map. If we have an older pool, this will not
	* be present; in this case, defer its creation to a later time to
	* avoid dirtying the MOS this early / out of sync context. See
	* spa_sync_config_object.
	*/

	/* The sentinel is only available in the MOS config. */
	nvlist_t *mos_config;
	if (load_nvlist(spa, spa->spa_config_object, &mos_config) != 0) {
	spa_load_failed(spa, "unable to retrieve MOS config");
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));
	}

	error = spa_dir_prop(spa, DMU_POOL_VDEV_ZAP_MAP,
	&spa->spa_all_vdev_zaps, B_FALSE);

	if (error == ENOENT) {
	VERIFY(!nvlist_exists(mos_config,
	ZPOOL_CONFIG_HAS_PER_VDEV_ZAPS));
	spa->spa_avz_action = AVZ_ACTION_INITIALIZE;
	ASSERT0(vdev_count_verify_zaps(spa->spa_root_vdev));
	} else if (error != 0) {
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));
	} else if (!nvlist_exists(mos_config, ZPOOL_CONFIG_HAS_PER_VDEV_ZAPS)) {
	/*
	* An older version of ZFS overwrote the sentinel value, so
	* we have orphaned per-vdev ZAPs in the MOS. Defer their
	* destruction to later; see spa_sync_config_object.
	*/
	spa->spa_avz_action = AVZ_ACTION_DESTROY;
	/*
	* We're assuming that no vdevs have had their ZAPs created
	* before this. Better be sure of it.
	*/
	ASSERT0(vdev_count_verify_zaps(spa->spa_root_vdev));
	}
	nvlist_free(mos_config);

	spa->spa_delegation = zpool_prop_default_numeric(ZPOOL_PROP_DELEGATION);

	error = spa_dir_prop(spa, DMU_POOL_PROPS, &spa->spa_pool_props_object,
	B_FALSE);
	if (error && error != ENOENT)
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));

	if (error == 0) {
	uint64_t autoreplace;

	spa_prop_find(spa, ZPOOL_PROP_BOOTFS, &spa->spa_bootfs);
	spa_prop_find(spa, ZPOOL_PROP_AUTOREPLACE, &autoreplace);
	spa_prop_find(spa, ZPOOL_PROP_DELEGATION, &spa->spa_delegation);
	spa_prop_find(spa, ZPOOL_PROP_FAILUREMODE, &spa->spa_failmode);
	spa_prop_find(spa, ZPOOL_PROP_AUTOEXPAND, &spa->spa_autoexpand);
	spa_prop_find(spa, ZPOOL_PROP_MULTIHOST, &spa->spa_multihost);
	spa_prop_find(spa, ZPOOL_PROP_AUTOTRIM, &spa->spa_autotrim);
	spa->spa_autoreplace = (autoreplace != 0);
	}

	/*
	* If we are importing a pool with missing top-level vdevs,
	* we enforce that the pool doesn't panic or get suspended on
	* error since the likelihood of missing data is extremely high.
	*/
	if (spa->spa_missing_tvds > 0 &&
	spa->spa_failmode != ZIO_FAILURE_MODE_CONTINUE &&
	spa->spa_load_state != SPA_LOAD_TRYIMPORT) {
	spa_load_note(spa, "forcing failmode to 'continue' "
	"as some top level vdevs are missing");
	spa->spa_failmode = ZIO_FAILURE_MODE_CONTINUE;
	}

	return (0);
	}

	static int
	spa_ld_open_aux_vdevs(spa_t *spa, spa_import_type_t type)
	{
	int error = 0;
	vdev_t *rvd = spa->spa_root_vdev;

	/*
	* If we're assembling the pool from the split-off vdevs of
	* an existing pool, we don't want to attach the spares & cache
	* devices.
	*/

	/*
	* Load any hot spares for this pool.
	*/
	error = spa_dir_prop(spa, DMU_POOL_SPARES, &spa->spa_spares.sav_object,
	B_FALSE);
	if (error != 0 && error != ENOENT)
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));
	if (error == 0 && type != SPA_IMPORT_ASSEMBLE) {
	ASSERT(spa_version(spa) >= SPA_VERSION_SPARES);
	if (load_nvlist(spa, spa->spa_spares.sav_object,
	&spa->spa_spares.sav_config) != 0) {
	spa_load_failed(spa, "error loading spares nvlist");
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));
	}

	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);
	spa_load_spares(spa);
	spa_config_exit(spa, SCL_ALL, FTAG);
	} else if (error == 0) {
	spa->spa_spares.sav_sync = B_TRUE;
	}

	/*
	* Load any level 2 ARC devices for this pool.
	*/
	error = spa_dir_prop(spa, DMU_POOL_L2CACHE,
	&spa->spa_l2cache.sav_object, B_FALSE);
	if (error != 0 && error != ENOENT)
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));
	if (error == 0 && type != SPA_IMPORT_ASSEMBLE) {
	ASSERT(spa_version(spa) >= SPA_VERSION_L2CACHE);
	if (load_nvlist(spa, spa->spa_l2cache.sav_object,
	&spa->spa_l2cache.sav_config) != 0) {
	spa_load_failed(spa, "error loading l2cache nvlist");
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));
	}

	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);
	spa_load_l2cache(spa);
	spa_config_exit(spa, SCL_ALL, FTAG);
	} else if (error == 0) {
	spa->spa_l2cache.sav_sync = B_TRUE;
	}

	return (0);
	}

	static int
	spa_ld_load_vdev_metadata(spa_t *spa)
	{
	int error = 0;
	vdev_t *rvd = spa->spa_root_vdev;

	/*
	* If the 'multihost' property is set, then never allow a pool to
	* be imported when the system hostid is zero. The exception to
	* this rule is zdb which is always allowed to access pools.
	*/
	if (spa_multihost(spa) && spa_get_hostid(spa) == 0 &&
	(spa->spa_import_flags & ZFS_IMPORT_SKIP_MMP) == 0) {
	fnvlist_add_uint64(spa->spa_load_info,
	ZPOOL_CONFIG_MMP_STATE, MMP_STATE_NO_HOSTID);
	return (spa_vdev_err(rvd, VDEV_AUX_ACTIVE, EREMOTEIO));
	}

	/*
	* If the 'autoreplace' property is set, then post a resource notifying
	* the ZFS DE that it should not issue any faults for unopenable
	* devices. We also iterate over the vdevs, and post a sysevent for any
	* unopenable vdevs so that the normal autoreplace handler can take
	* over.
	*/
	if (spa->spa_autoreplace && spa->spa_load_state != SPA_LOAD_TRYIMPORT) {
	spa_check_removed(spa->spa_root_vdev);
	/*
	* For the import case, this is done in spa_import(), because
	* at this point we're using the spare definitions from
	* the MOS config, not necessarily from the userland config.
	*/
	if (spa->spa_load_state != SPA_LOAD_IMPORT) {
	spa_aux_check_removed(&spa->spa_spares);
	spa_aux_check_removed(&spa->spa_l2cache);
	}
	}

	/*
	* Load the vdev metadata such as metaslabs, DTLs, spacemap object, etc.
	*/
	error = vdev_load(rvd);
	if (error != 0) {
	spa_load_failed(spa, "vdev_load failed [error=%d]", error);
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, error));
	}

	error = spa_ld_log_spacemaps(spa);
	if (error != 0) {
	spa_load_failed(spa, "spa_ld_log_sm_data failed [error=%d]",
	error);
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, error));
	}

	/*
	* Propagate the leaf DTLs we just loaded all the way up the vdev tree.
	*/
	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);
	vdev_dtl_reassess(rvd, 0, 0, B_FALSE, B_FALSE);
	spa_config_exit(spa, SCL_ALL, FTAG);

	return (0);
	}

	static int
	spa_ld_load_dedup_tables(spa_t *spa)
	{
	int error = 0;
	vdev_t *rvd = spa->spa_root_vdev;

	error = ddt_load(spa);
	if (error != 0) {
	spa_load_failed(spa, "ddt_load failed [error=%d]", error);
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, EIO));
	}

	return (0);
	}

	static int
	spa_ld_verify_logs(spa_t spa, spa_import_type_t type, char *ereport)
	{
	vdev_t *rvd = spa->spa_root_vdev;

	if (type != SPA_IMPORT_ASSEMBLE && spa_writeable(spa)) {
	boolean_t missing = spa_check_logs(spa);
	if (missing) {
	if (spa->spa_missing_tvds != 0) {
	spa_load_note(spa, "spa_check_logs failed "
	"so dropping the logs");
	} else {
	*ereport = FM_EREPORT_ZFS_LOG_REPLAY;
	spa_load_failed(spa, "spa_check_logs failed");
	return (spa_vdev_err(rvd, VDEV_AUX_BAD_LOG,
	ENXIO));
	}
	}
	}

	return (0);
	}

	static int
	spa_ld_verify_pool_data(spa_t *spa)
	{
	int error = 0;
	vdev_t *rvd = spa->spa_root_vdev;

	/*
	* We've successfully opened the pool, verify that we're ready
	* to start pushing transactions.
	*/
	if (spa->spa_load_state != SPA_LOAD_TRYIMPORT) {
	error = spa_load_verify(spa);
	if (error != 0) {
	spa_load_failed(spa, "spa_load_verify failed "
	"[error=%d]", error);
	return (spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA,
	error));
	}
	}

	return (0);
	}

	static void
	spa_ld_claim_log_blocks(spa_t *spa)
	{
	dmu_tx_t *tx;
	dsl_pool_t *dp = spa_get_dsl(spa);

	/*
	* Claim log blocks that haven't been committed yet.
	* This must all happen in a single txg.
	* Note: spa_claim_max_txg is updated by spa_claim_notify(),
	* invoked from zil_claim_log_block()'s i/o done callback.
	* Price of rollback is that we abandon the log.
	*/
	spa->spa_claiming = B_TRUE;

	tx = dmu_tx_create_assigned(dp, spa_first_txg(spa));
	(void) dmu_objset_find_dp(dp, dp->dp_root_dir_obj,
	zil_claim, tx, DS_FIND_CHILDREN);
	dmu_tx_commit(tx);

	spa->spa_claiming = B_FALSE;

	spa_set_log_state(spa, SPA_LOG_GOOD);
	}

	static void
	spa_ld_check_for_config_update(spa_t *spa, uint64_t config_cache_txg,
	boolean_t update_config_cache)
	{
	vdev_t *rvd = spa->spa_root_vdev;
	int need_update = B_FALSE;

	/*
	* If the config cache is stale, or we have uninitialized
	* metaslabs (see spa_vdev_add()), then update the config.
	*
	* If this is a verbatim import, trust the current
	* in-core spa_config and update the disk labels.
	*/
	if (update_config_cache \|\| config_cache_txg != spa->spa_config_txg \|\|
	spa->spa_load_state == SPA_LOAD_IMPORT \|\|
	spa->spa_load_state == SPA_LOAD_RECOVER \|\|
	(spa->spa_import_flags & ZFS_IMPORT_VERBATIM))
	need_update = B_TRUE;

	for (int c = 0; c < rvd->vdev_children; c++)
	if (rvd->vdev_child[c]->vdev_ms_array == 0)
	need_update = B_TRUE;

	/*
	* Update the config cache asynchronously in case we're the
	* root pool, in which case the config cache isn't writable yet.
	*/
	if (need_update)
	spa_async_request(spa, SPA_ASYNC_CONFIG_UPDATE);
	}

	static void
	spa_ld_prepare_for_reload(spa_t *spa)
	{
	spa_mode_t mode = spa->spa_mode;
	int async_suspended = spa->spa_async_suspended;

	spa_unload(spa);
	spa_deactivate(spa);
	spa_activate(spa, mode);

	/*
	* We save the value of spa_async_suspended as it gets reset to 0 by
	* spa_unload(). We want to restore it back to the original value before
	* returning as we might be calling spa_async_resume() later.
	*/
	spa->spa_async_suspended = async_suspended;
	}

	static int
	spa_ld_read_checkpoint_txg(spa_t *spa)
	{
	uberblock_t checkpoint;
	int error = 0;

	ASSERT0(spa->spa_checkpoint_txg);
	ASSERT(MUTEX_HELD(&spa_namespace_lock));

	error = zap_lookup(spa->spa_meta_objset, DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_ZPOOL_CHECKPOINT, sizeof (uint64_t),
	sizeof (uberblock_t) / sizeof (uint64_t), &checkpoint);

	if (error == ENOENT)
	return (0);

	if (error != 0)
	return (error);

	ASSERT3U(checkpoint.ub_txg, !=, 0);
	ASSERT3U(checkpoint.ub_checkpoint_txg, !=, 0);
	ASSERT3U(checkpoint.ub_timestamp, !=, 0);
	spa->spa_checkpoint_txg = checkpoint.ub_txg;
	spa->spa_checkpoint_info.sci_timestamp = checkpoint.ub_timestamp;

	return (0);
	}

	static int
	spa_ld_mos_init(spa_t *spa, spa_import_type_t type)
	{
	int error = 0;

	ASSERT(MUTEX_HELD(&spa_namespace_lock));
	ASSERT(spa->spa_config_source != SPA_CONFIG_SRC_NONE);

	/*
	* Never trust the config that is provided unless we are assembling
	* a pool following a split.
	* This means don't trust blkptrs and the vdev tree in general. This
	* also effectively puts the spa in read-only mode since
	* spa_writeable() checks for spa_trust_config to be true.
	* We will later load a trusted config from the MOS.
	*/
	if (type != SPA_IMPORT_ASSEMBLE)
	spa->spa_trust_config = B_FALSE;

	/*
	* Parse the config provided to create a vdev tree.
	*/
	error = spa_ld_parse_config(spa, type);
	if (error != 0)
	return (error);

	spa_import_progress_add(spa);

	/*
	* Now that we have the vdev tree, try to open each vdev. This involves
	* opening the underlying physical device, retrieving its geometry and
	* probing the vdev with a dummy I/O. The state of each vdev will be set
	* based on the success of those operations. After this we'll be ready
	* to read from the vdevs.
	*/
	error = spa_ld_open_vdevs(spa);
	if (error != 0)
	return (error);

	/*
	* Read the label of each vdev and make sure that the GUIDs stored
	* there match the GUIDs in the config provided.
	* If we're assembling a new pool that's been split off from an
	* existing pool, the labels haven't yet been updated so we skip
	* validation for now.
	*/
	if (type != SPA_IMPORT_ASSEMBLE) {
	error = spa_ld_validate_vdevs(spa);
	if (error != 0)
	return (error);
	}

	/*
	* Read all vdev labels to find the best uberblock (i.e. latest,
	* unless spa_load_max_txg is set) and store it in spa_uberblock. We
	* get the list of features required to read blkptrs in the MOS from
	* the vdev label with the best uberblock and verify that our version
	* of zfs supports them all.
	*/
	error = spa_ld_select_uberblock(spa, type);
	if (error != 0)
	return (error);

	/*
	* Pass that uberblock to the dsl_pool layer which will open the root
	* blkptr. This blkptr points to the latest version of the MOS and will
	* allow us to read its contents.
	*/
	error = spa_ld_open_rootbp(spa);
	if (error != 0)
	return (error);

	return (0);
	}

	static int
	spa_ld_checkpoint_rewind(spa_t *spa)
	{
	uberblock_t checkpoint;
	int error = 0;

	ASSERT(MUTEX_HELD(&spa_namespace_lock));
	ASSERT(spa->spa_import_flags & ZFS_IMPORT_CHECKPOINT);

	error = zap_lookup(spa->spa_meta_objset, DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_ZPOOL_CHECKPOINT, sizeof (uint64_t),
	sizeof (uberblock_t) / sizeof (uint64_t), &checkpoint);

	if (error != 0) {
	spa_load_failed(spa, "unable to retrieve checkpointed "
	"uberblock from the MOS config [error=%d]", error);

	if (error == ENOENT)
	error = ZFS_ERR_NO_CHECKPOINT;

	return (error);
	}

	ASSERT3U(checkpoint.ub_txg, <, spa->spa_uberblock.ub_txg);
	ASSERT3U(checkpoint.ub_txg, ==, checkpoint.ub_checkpoint_txg);

	/*
	* We need to update the txg and timestamp of the checkpointed
	* uberblock to be higher than the latest one. This ensures that
	* the checkpointed uberblock is selected if we were to close and
	* reopen the pool right after we've written it in the vdev labels.
	* (also see block comment in vdev_uberblock_compare)
	*/
	checkpoint.ub_txg = spa->spa_uberblock.ub_txg + 1;
	checkpoint.ub_timestamp = gethrestime_sec();

	/*
	* Set current uberblock to be the checkpointed uberblock.
	*/
	spa->spa_uberblock = checkpoint;

	/*
	* If we are doing a normal rewind, then the pool is open for
	* writing and we sync the "updated" checkpointed uberblock to
	* disk. Once this is done, we've basically rewound the whole
	* pool and there is no way back.
	*
	* There are cases when we don't want to attempt and sync the
	* checkpointed uberblock to disk because we are opening a
	* pool as read-only. Specifically, verifying the checkpointed
	* state with zdb, and importing the checkpointed state to get
	* a "preview" of its content.
	*/
	if (spa_writeable(spa)) {
	vdev_t *rvd = spa->spa_root_vdev;

	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);
	vdev_t *svd[SPA_SYNC_MIN_VDEVS] = { NULL };
	int svdcount = 0;
	int children = rvd->vdev_children;
	int c0 = spa_get_random(children);

	for (int c = 0; c < children; c++) {
	vdev_t *vd = rvd->vdev_child[(c0 + c) % children];

	/* Stop when revisiting the first vdev */
	if (c > 0 && svd[0] == vd)
	break;

	if (vd->vdev_ms_array == 0 \|\| vd->vdev_islog \|\|
	!vdev_is_concrete(vd))
	continue;

	svd[svdcount++] = vd;
	if (svdcount == SPA_SYNC_MIN_VDEVS)
	break;
	}
	error = vdev_config_sync(svd, svdcount, spa->spa_first_txg);
	if (error == 0)
	spa->spa_last_synced_guid = rvd->vdev_guid;
	spa_config_exit(spa, SCL_ALL, FTAG);

	if (error != 0) {
	spa_load_failed(spa, "failed to write checkpointed "
	"uberblock to the vdev labels [error=%d]", error);
	return (error);
	}
	}

	return (0);
	}

	static int
	spa_ld_mos_with_trusted_config(spa_t *spa, spa_import_type_t type,
	boolean_t *update_config_cache)
	{
	int error;

	/*
	* Parse the config for pool, open and validate vdevs,
	* select an uberblock, and use that uberblock to open
	* the MOS.
	*/
	error = spa_ld_mos_init(spa, type);
	if (error != 0)
	return (error);

	/*
	* Retrieve the trusted config stored in the MOS and use it to create
	* a new, exact version of the vdev tree, then reopen all vdevs.
	*/
	error = spa_ld_trusted_config(spa, type, B_FALSE);
	if (error == EAGAIN) {
	if (update_config_cache != NULL)
	*update_config_cache = B_TRUE;

	/*
	* Redo the loading process with the trusted config if it is
	* too different from the untrusted config.
	*/
	spa_ld_prepare_for_reload(spa);
	spa_load_note(spa, "RELOADING");
	error = spa_ld_mos_init(spa, type);
	if (error != 0)
	return (error);

	error = spa_ld_trusted_config(spa, type, B_TRUE);
	if (error != 0)
	return (error);

	} else if (error != 0) {
	return (error);
	}

	return (0);
	}

	/*
	* Load an existing storage pool, using the config provided. This config
	* describes which vdevs are part of the pool and is later validated against
	* partial configs present in each vdev's label and an entire copy of the
	* config stored in the MOS.
	*/
	static int
	spa_load_impl(spa_t spa, spa_import_type_t type, char *ereport)
	{
	int error = 0;
	boolean_t missing_feat_write = B_FALSE;
	boolean_t checkpoint_rewind =
	(spa->spa_import_flags & ZFS_IMPORT_CHECKPOINT);
	boolean_t update_config_cache = B_FALSE;

	ASSERT(MUTEX_HELD(&spa_namespace_lock));
	ASSERT(spa->spa_config_source != SPA_CONFIG_SRC_NONE);

	spa_load_note(spa, "LOADING");

	error = spa_ld_mos_with_trusted_config(spa, type, &update_config_cache);
	if (error != 0)
	return (error);

	/*
	* If we are rewinding to the checkpoint then we need to repeat
	* everything we've done so far in this function but this time
	* selecting the checkpointed uberblock and using that to open
	* the MOS.
	*/
	if (checkpoint_rewind) {
	/*
	* If we are rewinding to the checkpoint update config cache
	* anyway.
	*/
	update_config_cache = B_TRUE;

	/*
	* Extract the checkpointed uberblock from the current MOS
	* and use this as the pool's uberblock from now on. If the
	* pool is imported as writeable we also write the checkpoint
	* uberblock to the labels, making the rewind permanent.
	*/
	error = spa_ld_checkpoint_rewind(spa);
	if (error != 0)
	return (error);

	/*
	* Redo the loading process again with the
	* checkpointed uberblock.
	*/
	spa_ld_prepare_for_reload(spa);
	spa_load_note(spa, "LOADING checkpointed uberblock");
	error = spa_ld_mos_with_trusted_config(spa, type, NULL);
	if (error != 0)
	return (error);
	}

	/*
	* Retrieve the checkpoint txg if the pool has a checkpoint.
	*/
	error = spa_ld_read_checkpoint_txg(spa);
	if (error != 0)
	return (error);

	/*
	* Retrieve the mapping of indirect vdevs. Those vdevs were removed
	* from the pool and their contents were re-mapped to other vdevs. Note
	* that everything that we read before this step must have been
	* rewritten on concrete vdevs after the last device removal was
	* initiated. Otherwise we could be reading from indirect vdevs before
	* we have loaded their mappings.
	*/
	error = spa_ld_open_indirect_vdev_metadata(spa);
	if (error != 0)
	return (error);

	/*
	* Retrieve the full list of active features from the MOS and check if
	* they are all supported.
	*/
	error = spa_ld_check_features(spa, &missing_feat_write);
	if (error != 0)
	return (error);

	/*
	* Load several special directories from the MOS needed by the dsl_pool
	* layer.
	*/
	error = spa_ld_load_special_directories(spa);
	if (error != 0)
	return (error);

	/*
	* Retrieve pool properties from the MOS.
	*/
	error = spa_ld_get_props(spa);
	if (error != 0)
	return (error);

	/*
	* Retrieve the list of auxiliary devices - cache devices and spares -
	* and open them.
	*/
	error = spa_ld_open_aux_vdevs(spa, type);
	if (error != 0)
	return (error);

	/*
	* Load the metadata for all vdevs. Also check if unopenable devices
	* should be autoreplaced.
	*/
	error = spa_ld_load_vdev_metadata(spa);
	if (error != 0)
	return (error);

	error = spa_ld_load_dedup_tables(spa);
	if (error != 0)
	return (error);

	/*
	* Verify the logs now to make sure we don't have any unexpected errors
	* when we claim log blocks later.
	*/
	error = spa_ld_verify_logs(spa, type, ereport);
	if (error != 0)
	return (error);

	if (missing_feat_write) {
	ASSERT(spa->spa_load_state == SPA_LOAD_TRYIMPORT);

	/*
	* At this point, we know that we can open the pool in
	* read-only mode but not read-write mode. We now have enough
	* information and can return to userland.
	*/
	return (spa_vdev_err(spa->spa_root_vdev, VDEV_AUX_UNSUP_FEAT,
	ENOTSUP));
	}

	/*
	* Traverse the last txgs to make sure the pool was left off in a safe
	* state. When performing an extreme rewind, we verify the whole pool,
	* which can take a very long time.
	*/
	error = spa_ld_verify_pool_data(spa);
	if (error != 0)
	return (error);

	/*
	* Calculate the deflated space for the pool. This must be done before
	* we write anything to the pool because we'd need to update the space
	* accounting using the deflated sizes.
	*/
	spa_update_dspace(spa);

	/*
	* We have now retrieved all the information we needed to open the
	* pool. If we are importing the pool in read-write mode, a few
	* additional steps must be performed to finish the import.
	*/
	if (spa_writeable(spa) && (spa->spa_load_state == SPA_LOAD_RECOVER \|\|
	spa->spa_load_max_txg == UINT64_MAX)) {
	uint64_t config_cache_txg = spa->spa_config_txg;

	ASSERT(spa->spa_load_state != SPA_LOAD_TRYIMPORT);

	/*
	* In case of a checkpoint rewind, log the original txg
	* of the checkpointed uberblock.
	*/
	if (checkpoint_rewind) {
	spa_history_log_internal(spa, "checkpoint rewind",
	NULL, "rewound state to txg=%llu",
	(u_longlong_t)spa->spa_uberblock.ub_checkpoint_txg);
	}

	/*
	* Traverse the ZIL and claim all blocks.
	*/
	spa_ld_claim_log_blocks(spa);

	/*
	* Kick-off the syncing thread.
	*/
	spa->spa_sync_on = B_TRUE;
	txg_sync_start(spa->spa_dsl_pool);
	mmp_thread_start(spa);

	/*
	* Wait for all claims to sync. We sync up to the highest
	* claimed log block birth time so that claimed log blocks
	* don't appear to be from the future. spa_claim_max_txg
	* will have been set for us by ZIL traversal operations
	* performed above.
	*/
	txg_wait_synced(spa->spa_dsl_pool, spa->spa_claim_max_txg);

	/*
	* Check if we need to request an update of the config. On the
	* next sync, we would update the config stored in vdev labels
	* and the cachefile (by default /etc/zfs/zpool.cache).
	*/
	spa_ld_check_for_config_update(spa, config_cache_txg,
	update_config_cache);

	/*
	* Check if a rebuild was in progress and if so resume it.
	* Then check all DTLs to see if anything needs resilvering.
	* The resilver will be deferred if a rebuild was started.
	*/
	if (vdev_rebuild_active(spa->spa_root_vdev)) {
	vdev_rebuild_restart(spa);
	} else if (!dsl_scan_resilvering(spa->spa_dsl_pool) &&
	vdev_resilver_needed(spa->spa_root_vdev, NULL, NULL)) {
	spa_async_request(spa, SPA_ASYNC_RESILVER);
	}

	/*
	* Log the fact that we booted up (so that we can detect if
	* we rebooted in the middle of an operation).
	*/
	spa_history_log_version(spa, "open", NULL);

	spa_restart_removal(spa);
	spa_spawn_aux_threads(spa);

	/*
	* Delete any inconsistent datasets.
	*
	* Note:
	* Since we may be issuing deletes for clones here,
	* we make sure to do so after we've spawned all the
	* auxiliary threads above (from which the livelist
	* deletion zthr is part of).
	*/
	(void) dmu_objset_find(spa_name(spa),
	dsl_destroy_inconsistent, NULL, DS_FIND_CHILDREN);

	/*
	* Clean up any stale temporary dataset userrefs.
	*/
	dsl_pool_clean_tmp_userrefs(spa->spa_dsl_pool);

	spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER);
	vdev_initialize_restart(spa->spa_root_vdev);
	vdev_trim_restart(spa->spa_root_vdev);
	vdev_autotrim_restart(spa);
	spa_config_exit(spa, SCL_CONFIG, FTAG);
	}

	spa_import_progress_remove(spa_guid(spa));
	spa_async_request(spa, SPA_ASYNC_L2CACHE_REBUILD);

	spa_load_note(spa, "LOADED");

	return (0);
	}

	static int
	spa_load_retry(spa_t *spa, spa_load_state_t state)
	{
	spa_mode_t mode = spa->spa_mode;

	spa_unload(spa);
	spa_deactivate(spa);

	spa->spa_load_max_txg = spa->spa_uberblock.ub_txg - 1;

	spa_activate(spa, mode);
	spa_async_suspend(spa);

	spa_load_note(spa, "spa_load_retry: rewind, max txg: %llu",
	(u_longlong_t)spa->spa_load_max_txg);

	return (spa_load(spa, state, SPA_IMPORT_EXISTING));
	}

	/*
	* If spa_load() fails this function will try loading prior txg's. If
	* 'state' is SPA_LOAD_RECOVER and one of these loads succeeds the pool
	* will be rewound to that txg. If 'state' is not SPA_LOAD_RECOVER this
	* function will not rewind the pool and will return the same error as
	* spa_load().
	*/
	static int
	spa_load_best(spa_t *spa, spa_load_state_t state, uint64_t max_request,
	int rewind_flags)
	{
	nvlist_t *loadinfo = NULL;
	nvlist_t *config = NULL;
	int load_error, rewind_error;
	uint64_t safe_rewind_txg;
	uint64_t min_txg;

	if (spa->spa_load_txg && state == SPA_LOAD_RECOVER) {
	spa->spa_load_max_txg = spa->spa_load_txg;
	spa_set_log_state(spa, SPA_LOG_CLEAR);
	} else {
	spa->spa_load_max_txg = max_request;
	if (max_request != UINT64_MAX)
	spa->spa_extreme_rewind = B_TRUE;
	}

	load_error = rewind_error = spa_load(spa, state, SPA_IMPORT_EXISTING);
	if (load_error == 0)
	return (0);
	if (load_error == ZFS_ERR_NO_CHECKPOINT) {
	/*
	* When attempting checkpoint-rewind on a pool with no
	* checkpoint, we should not attempt to load uberblocks
	* from previous txgs when spa_load fails.
	*/
	ASSERT(spa->spa_import_flags & ZFS_IMPORT_CHECKPOINT);
	spa_import_progress_remove(spa_guid(spa));
	return (load_error);
	}

	if (spa->spa_root_vdev != NULL)
	config = spa_config_generate(spa, NULL, -1ULL, B_TRUE);

	spa->spa_last_ubsync_txg = spa->spa_uberblock.ub_txg;
	spa->spa_last_ubsync_txg_ts = spa->spa_uberblock.ub_timestamp;

	if (rewind_flags & ZPOOL_NEVER_REWIND) {
	nvlist_free(config);
	spa_import_progress_remove(spa_guid(spa));
	return (load_error);
	}

	if (state == SPA_LOAD_RECOVER) {
	/* Price of rolling back is discarding txgs, including log */
	spa_set_log_state(spa, SPA_LOG_CLEAR);
	} else {
	/*
	* If we aren't rolling back save the load info from our first
	* import attempt so that we can restore it after attempting
	* to rewind.
	*/
	loadinfo = spa->spa_load_info;
	spa->spa_load_info = fnvlist_alloc();
	}

	spa->spa_load_max_txg = spa->spa_last_ubsync_txg;
	safe_rewind_txg = spa->spa_last_ubsync_txg - TXG_DEFER_SIZE;
	min_txg = (rewind_flags & ZPOOL_EXTREME_REWIND) ?
	TXG_INITIAL : safe_rewind_txg;

	/*
	* Continue as long as we're finding errors, we're still within
	* the acceptable rewind range, and we're still finding uberblocks
	*/
	while (rewind_error && spa->spa_uberblock.ub_txg >= min_txg &&
	spa->spa_uberblock.ub_txg <= spa->spa_load_max_txg) {
	if (spa->spa_load_max_txg < safe_rewind_txg)
	spa->spa_extreme_rewind = B_TRUE;
	rewind_error = spa_load_retry(spa, state);
	}

	spa->spa_extreme_rewind = B_FALSE;
	spa->spa_load_max_txg = UINT64_MAX;

	if (config && (rewind_error \|\| state != SPA_LOAD_RECOVER))
	spa_config_set(spa, config);
	else
	nvlist_free(config);

	if (state == SPA_LOAD_RECOVER) {
	ASSERT3P(loadinfo, ==, NULL);
	spa_import_progress_remove(spa_guid(spa));
	return (rewind_error);
	} else {
	/* Store the rewind info as part of the initial load info */
	fnvlist_add_nvlist(loadinfo, ZPOOL_CONFIG_REWIND_INFO,
	spa->spa_load_info);

	/* Restore the initial load info */
	fnvlist_free(spa->spa_load_info);
	spa->spa_load_info = loadinfo;

	spa_import_progress_remove(spa_guid(spa));
	return (load_error);
	}
	}

	/*
	* Pool Open/Import
	*
	* The import case is identical to an open except that the configuration is sent
	* down from userland, instead of grabbed from the configuration cache. For the
	* case of an open, the pool configuration will exist in the
	* POOL_STATE_UNINITIALIZED state.
	*
	* The stats information (gen/count/ustats) is used to gather vdev statistics at
	* the same time open the pool, without having to keep around the spa_t in some
	* ambiguous state.
	*/
	static int
	spa_open_common(const char pool, spa_t spapp, void tag, nvlist_t *nvpolicy,
	nvlist_t **config)
	{
	spa_t *spa;
	spa_load_state_t state = SPA_LOAD_OPEN;
	int error;
	int locked = B_FALSE;
	int firstopen = B_FALSE;

	*spapp = NULL;

	/*
	* As disgusting as this is, we need to support recursive calls to this
	* function because dsl_dir_open() is called during spa_load(), and ends
	* up calling spa_open() again. The real fix is to figure out how to
	* avoid dsl_dir_open() calling this in the first place.
	*/
	if (MUTEX_NOT_HELD(&spa_namespace_lock)) {
	mutex_enter(&spa_namespace_lock);
	locked = B_TRUE;
	}

	if ((spa = spa_lookup(pool)) == NULL) {
	if (locked)
	mutex_exit(&spa_namespace_lock);
	return (SET_ERROR(ENOENT));
	}

	if (spa->spa_state == POOL_STATE_UNINITIALIZED) {
	zpool_load_policy_t policy;

	firstopen = B_TRUE;

	zpool_get_load_policy(nvpolicy ? nvpolicy : spa->spa_config,
	&policy);
	if (policy.zlp_rewind & ZPOOL_DO_REWIND)
	state = SPA_LOAD_RECOVER;

	spa_activate(spa, spa_mode_global);

	if (state != SPA_LOAD_RECOVER)
	spa->spa_last_ubsync_txg = spa->spa_load_txg = 0;
	spa->spa_config_source = SPA_CONFIG_SRC_CACHEFILE;

	zfs_dbgmsg("spa_open_common: opening %s", pool);
	error = spa_load_best(spa, state, policy.zlp_txg,
	policy.zlp_rewind);

	if (error == EBADF) {
	/*
	* If vdev_validate() returns failure (indicated by
	* EBADF), it indicates that one of the vdevs indicates
	* that the pool has been exported or destroyed. If
	* this is the case, the config cache is out of sync and
	* we should remove the pool from the namespace.
	*/
	spa_unload(spa);
	spa_deactivate(spa);
	spa_write_cachefile(spa, B_TRUE, B_TRUE);
	spa_remove(spa);
	if (locked)
	mutex_exit(&spa_namespace_lock);
	return (SET_ERROR(ENOENT));
	}

	if (error) {
	/*
	* We can't open the pool, but we still have useful
	* information: the state of each vdev after the
	* attempted vdev_open(). Return this to the user.
	*/
	if (config != NULL && spa->spa_config) {
	VERIFY(nvlist_dup(spa->spa_config, config,
	KM_SLEEP) == 0);
	VERIFY(nvlist_add_nvlist(*config,
	ZPOOL_CONFIG_LOAD_INFO,
	spa->spa_load_info) == 0);
	}
	spa_unload(spa);
	spa_deactivate(spa);
	spa->spa_last_open_failed = error;
	if (locked)
	mutex_exit(&spa_namespace_lock);
	*spapp = NULL;
	return (error);
	}
	}

	spa_open_ref(spa, tag);

	if (config != NULL)
	*config = spa_config_generate(spa, NULL, -1ULL, B_TRUE);

	/*
	* If we've recovered the pool, pass back any information we
	* gathered while doing the load.
	*/
	if (state == SPA_LOAD_RECOVER) {
	VERIFY(nvlist_add_nvlist(*config, ZPOOL_CONFIG_LOAD_INFO,
	spa->spa_load_info) == 0);
	}

	if (locked) {
	spa->spa_last_open_failed = 0;
	spa->spa_last_ubsync_txg = 0;
	spa->spa_load_txg = 0;
	mutex_exit(&spa_namespace_lock);
	}

	if (firstopen)
	zvol_create_minors_recursive(spa_name(spa));

	*spapp = spa;

	return (0);
	}

	int
	spa_open_rewind(const char name, spa_t spapp, void tag, nvlist_t *policy,
	nvlist_t **config)
	{
	return (spa_open_common(name, spapp, tag, policy, config));
	}

	int
	spa_open(const char name, spa_t spapp, void tag)
	{
	return (spa_open_common(name, spapp, tag, NULL, NULL));
	}

	/*
	* Lookup the given spa_t, incrementing the inject count in the process,
	* preventing it from being exported or destroyed.
	*/
	spa_t *
	spa_inject_addref(char *name)
	{
	spa_t *spa;

	mutex_enter(&spa_namespace_lock);
	if ((spa = spa_lookup(name)) == NULL) {
	mutex_exit(&spa_namespace_lock);
	return (NULL);
	}
	spa->spa_inject_ref++;
	mutex_exit(&spa_namespace_lock);

	return (spa);
	}

	void
	spa_inject_delref(spa_t *spa)
	{
	mutex_enter(&spa_namespace_lock);
	spa->spa_inject_ref--;
	mutex_exit(&spa_namespace_lock);
	}

	/*
	* Add spares device information to the nvlist.
	*/
	static void
	spa_add_spares(spa_t spa, nvlist_t config)
	{
	nvlist_t **spares;
	uint_t i, nspares;
	nvlist_t *nvroot;
	uint64_t guid;
	vdev_stat_t *vs;
	uint_t vsc;
	uint64_t pool;

	ASSERT(spa_config_held(spa, SCL_CONFIG, RW_READER));

	if (spa->spa_spares.sav_count == 0)
	return;

	VERIFY(nvlist_lookup_nvlist(config,
	ZPOOL_CONFIG_VDEV_TREE, &nvroot) == 0);
	VERIFY(nvlist_lookup_nvlist_array(spa->spa_spares.sav_config,
	ZPOOL_CONFIG_SPARES, &spares, &nspares) == 0);
	if (nspares != 0) {
	VERIFY(nvlist_add_nvlist_array(nvroot,
	ZPOOL_CONFIG_SPARES, spares, nspares) == 0);
	VERIFY(nvlist_lookup_nvlist_array(nvroot,
	ZPOOL_CONFIG_SPARES, &spares, &nspares) == 0);

	/*
	* Go through and find any spares which have since been
	* repurposed as an active spare. If this is the case, update
	* their status appropriately.
	*/
	for (i = 0; i < nspares; i++) {
	VERIFY(nvlist_lookup_uint64(spares[i],
	ZPOOL_CONFIG_GUID, &guid) == 0);
	if (spa_spare_exists(guid, &pool, NULL) &&
	pool != 0ULL) {
	VERIFY(nvlist_lookup_uint64_array(
	spares[i], ZPOOL_CONFIG_VDEV_STATS,
	(uint64_t **)&vs, &vsc) == 0);
	vs->vs_state = VDEV_STATE_CANT_OPEN;
	vs->vs_aux = VDEV_AUX_SPARED;
	}
	}
	}
	}

	/*
	* Add l2cache device information to the nvlist, including vdev stats.
	*/
	static void
	spa_add_l2cache(spa_t spa, nvlist_t config)
	{
	nvlist_t **l2cache;
	uint_t i, j, nl2cache;
	nvlist_t *nvroot;
	uint64_t guid;
	vdev_t *vd;
	vdev_stat_t *vs;
	uint_t vsc;

	ASSERT(spa_config_held(spa, SCL_CONFIG, RW_READER));

	if (spa->spa_l2cache.sav_count == 0)
	return;

	VERIFY(nvlist_lookup_nvlist(config,
	ZPOOL_CONFIG_VDEV_TREE, &nvroot) == 0);
	VERIFY(nvlist_lookup_nvlist_array(spa->spa_l2cache.sav_config,
	ZPOOL_CONFIG_L2CACHE, &l2cache, &nl2cache) == 0);
	if (nl2cache != 0) {
	VERIFY(nvlist_add_nvlist_array(nvroot,
	ZPOOL_CONFIG_L2CACHE, l2cache, nl2cache) == 0);
	VERIFY(nvlist_lookup_nvlist_array(nvroot,
	ZPOOL_CONFIG_L2CACHE, &l2cache, &nl2cache) == 0);

	/*
	* Update level 2 cache device stats.
	*/

	for (i = 0; i < nl2cache; i++) {
	VERIFY(nvlist_lookup_uint64(l2cache[i],
	ZPOOL_CONFIG_GUID, &guid) == 0);

	vd = NULL;
	for (j = 0; j < spa->spa_l2cache.sav_count; j++) {
	if (guid ==
	spa->spa_l2cache.sav_vdevs[j]->vdev_guid) {
	vd = spa->spa_l2cache.sav_vdevs[j];
	break;
	}
	}
	ASSERT(vd != NULL);

	VERIFY(nvlist_lookup_uint64_array(l2cache[i],
	ZPOOL_CONFIG_VDEV_STATS, (uint64_t **)&vs, &vsc)
	== 0);
	vdev_get_stats(vd, vs);
	vdev_config_generate_stats(vd, l2cache[i]);

	}
	}
	}

	static void
	spa_feature_stats_from_disk(spa_t spa, nvlist_t features)
	{
	zap_cursor_t zc;
	zap_attribute_t za;

	if (spa->spa_feat_for_read_obj != 0) {
	for (zap_cursor_init(&zc, spa->spa_meta_objset,
	spa->spa_feat_for_read_obj);
	zap_cursor_retrieve(&zc, &za) == 0;
	zap_cursor_advance(&zc)) {
	ASSERT(za.za_integer_length == sizeof (uint64_t) &&
	za.za_num_integers == 1);
	VERIFY0(nvlist_add_uint64(features, za.za_name,
	za.za_first_integer));
	}
	zap_cursor_fini(&zc);
	}

	if (spa->spa_feat_for_write_obj != 0) {
	for (zap_cursor_init(&zc, spa->spa_meta_objset,
	spa->spa_feat_for_write_obj);
	zap_cursor_retrieve(&zc, &za) == 0;
	zap_cursor_advance(&zc)) {
	ASSERT(za.za_integer_length == sizeof (uint64_t) &&
	za.za_num_integers == 1);
	VERIFY0(nvlist_add_uint64(features, za.za_name,
	za.za_first_integer));
	}
	zap_cursor_fini(&zc);
	}
	}

	static void
	spa_feature_stats_from_cache(spa_t spa, nvlist_t features)
	{
	int i;

	for (i = 0; i < SPA_FEATURES; i++) {
	zfeature_info_t feature = spa_feature_table[i];
	uint64_t refcount;

	if (feature_get_refcount(spa, &feature, &refcount) != 0)
	continue;

	VERIFY0(nvlist_add_uint64(features, feature.fi_guid, refcount));
	}
	}

	/*
	* Store a list of pool features and their reference counts in the
	* config.
	*
	* The first time this is called on a spa, allocate a new nvlist, fetch
	* the pool features and reference counts from disk, then save the list
	* in the spa. In subsequent calls on the same spa use the saved nvlist
	* and refresh its values from the cached reference counts. This
	* ensures we don't block here on I/O on a suspended pool so 'zpool
	* clear' can resume the pool.
	*/
	static void
	spa_add_feature_stats(spa_t spa, nvlist_t config)
	{
	nvlist_t *features;

	ASSERT(spa_config_held(spa, SCL_CONFIG, RW_READER));

	mutex_enter(&spa->spa_feat_stats_lock);
	features = spa->spa_feat_stats;

	if (features != NULL) {
	spa_feature_stats_from_cache(spa, features);
	} else {
	VERIFY0(nvlist_alloc(&features, NV_UNIQUE_NAME, KM_SLEEP));
	spa->spa_feat_stats = features;
	spa_feature_stats_from_disk(spa, features);
	}

	VERIFY0(nvlist_add_nvlist(config, ZPOOL_CONFIG_FEATURE_STATS,
	features));

	mutex_exit(&spa->spa_feat_stats_lock);
	}

	int
	spa_get_stats(const char name, nvlist_t *config,
	char *altroot, size_t buflen)
	{
	int error;
	spa_t *spa;

	*config = NULL;
	error = spa_open_common(name, &spa, FTAG, NULL, config);

	if (spa != NULL) {
	/*
	* This still leaves a window of inconsistency where the spares
	* or l2cache devices could change and the config would be
	* self-inconsistent.
	*/
	spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER);

	if (*config != NULL) {
	uint64_t loadtimes[2];

	loadtimes[0] = spa->spa_loaded_ts.tv_sec;
	loadtimes[1] = spa->spa_loaded_ts.tv_nsec;
	VERIFY(nvlist_add_uint64_array(*config,
	ZPOOL_CONFIG_LOADED_TIME, loadtimes, 2) == 0);

	VERIFY(nvlist_add_uint64(*config,
	ZPOOL_CONFIG_ERRCOUNT,
	spa_get_errlog_size(spa)) == 0);

	if (spa_suspended(spa)) {
	VERIFY(nvlist_add_uint64(*config,
	ZPOOL_CONFIG_SUSPENDED,
	spa->spa_failmode) == 0);
	VERIFY(nvlist_add_uint64(*config,
	ZPOOL_CONFIG_SUSPENDED_REASON,
	spa->spa_suspended) == 0);
	}

	spa_add_spares(spa, *config);
	spa_add_l2cache(spa, *config);
	spa_add_feature_stats(spa, *config);
	}
	}

	/*
	* We want to get the alternate root even for faulted pools, so we cheat
	* and call spa_lookup() directly.
	*/
	if (altroot) {
	if (spa == NULL) {
	mutex_enter(&spa_namespace_lock);
	spa = spa_lookup(name);
	if (spa)
	spa_altroot(spa, altroot, buflen);
	else
	altroot[0] = '\0';
	spa = NULL;
	mutex_exit(&spa_namespace_lock);
	} else {
	spa_altroot(spa, altroot, buflen);
	}
	}

	if (spa != NULL) {
	spa_config_exit(spa, SCL_CONFIG, FTAG);
	spa_close(spa, FTAG);
	}

	return (error);
	}

	/*
	* Validate that the auxiliary device array is well formed. We must have an
	* array of nvlists, each which describes a valid leaf vdev. If this is an
	* import (mode is VDEV_ALLOC_SPARE), then we allow corrupted spares to be
	* specified, as long as they are well-formed.
	*/
	static int
	spa_validate_aux_devs(spa_t spa, nvlist_t nvroot, uint64_t crtxg, int mode,
	spa_aux_vdev_t sav, const char config, uint64_t version,
	vdev_labeltype_t label)
	{
	nvlist_t **dev;
	uint_t i, ndev;
	vdev_t *vd;
	int error;

	ASSERT(spa_config_held(spa, SCL_ALL, RW_WRITER) == SCL_ALL);

	/*
	* It's acceptable to have no devs specified.
	*/
	if (nvlist_lookup_nvlist_array(nvroot, config, &dev, &ndev) != 0)
	return (0);

	if (ndev == 0)
	return (SET_ERROR(EINVAL));

	/*
	* Make sure the pool is formatted with a version that supports this
	* device type.
	*/
	if (spa_version(spa) < version)
	return (SET_ERROR(ENOTSUP));

	/*
	* Set the pending device list so we correctly handle device in-use
	* checking.
	*/
	sav->sav_pending = dev;
	sav->sav_npending = ndev;

	for (i = 0; i < ndev; i++) {
	if ((error = spa_config_parse(spa, &vd, dev[i], NULL, 0,
	mode)) != 0)
	goto out;

	if (!vd->vdev_ops->vdev_op_leaf) {
	vdev_free(vd);
	error = SET_ERROR(EINVAL);
	goto out;
	}

	vd->vdev_top = vd;

	if ((error = vdev_open(vd)) == 0 &&
	(error = vdev_label_init(vd, crtxg, label)) == 0) {
	VERIFY(nvlist_add_uint64(dev[i], ZPOOL_CONFIG_GUID,
	vd->vdev_guid) == 0);
	}

	vdev_free(vd);

	if (error &&
	(mode != VDEV_ALLOC_SPARE && mode != VDEV_ALLOC_L2CACHE))
	goto out;
	else
	error = 0;
	}

	out:
	sav->sav_pending = NULL;
	sav->sav_npending = 0;
	return (error);
	}

	static int
	spa_validate_aux(spa_t spa, nvlist_t nvroot, uint64_t crtxg, int mode)
	{
	int error;

	ASSERT(spa_config_held(spa, SCL_ALL, RW_WRITER) == SCL_ALL);

	if ((error = spa_validate_aux_devs(spa, nvroot, crtxg, mode,
	&spa->spa_spares, ZPOOL_CONFIG_SPARES, SPA_VERSION_SPARES,
	VDEV_LABEL_SPARE)) != 0) {
	return (error);
	}

	return (spa_validate_aux_devs(spa, nvroot, crtxg, mode,
	&spa->spa_l2cache, ZPOOL_CONFIG_L2CACHE, SPA_VERSION_L2CACHE,
	VDEV_LABEL_L2CACHE));
	}

	static void
	spa_set_aux_vdevs(spa_aux_vdev_t sav, nvlist_t *devs, int ndevs,
	const char *config)
	{
	int i;

	if (sav->sav_config != NULL) {
	nvlist_t **olddevs;
	uint_t oldndevs;
	nvlist_t **newdevs;

	/*
	* Generate new dev list by concatenating with the
	* current dev list.
	*/
	VERIFY(nvlist_lookup_nvlist_array(sav->sav_config, config,
	&olddevs, &oldndevs) == 0);

	newdevs = kmem_alloc(sizeof (void )
	(ndevs + oldndevs), KM_SLEEP);
	for (i = 0; i < oldndevs; i++)
	VERIFY(nvlist_dup(olddevs[i], &newdevs[i],
	KM_SLEEP) == 0);
	for (i = 0; i < ndevs; i++)
	VERIFY(nvlist_dup(devs[i], &newdevs[i + oldndevs],
	KM_SLEEP) == 0);

	VERIFY(nvlist_remove(sav->sav_config, config,
	DATA_TYPE_NVLIST_ARRAY) == 0);

	VERIFY(nvlist_add_nvlist_array(sav->sav_config,
	config, newdevs, ndevs + oldndevs) == 0);
	for (i = 0; i < oldndevs + ndevs; i++)
	nvlist_free(newdevs[i]);
	kmem_free(newdevs, (oldndevs + ndevs) * sizeof (void *));
	} else {
	/*
	* Generate a new dev list.
	*/
	VERIFY(nvlist_alloc(&sav->sav_config, NV_UNIQUE_NAME,
	KM_SLEEP) == 0);
	VERIFY(nvlist_add_nvlist_array(sav->sav_config, config,
	devs, ndevs) == 0);
	}
	}

	/*
	* Stop and drop level 2 ARC devices
	*/
	void
	spa_l2cache_drop(spa_t *spa)
	{
	vdev_t *vd;
	int i;
	spa_aux_vdev_t *sav = &spa->spa_l2cache;

	for (i = 0; i < sav->sav_count; i++) {
	uint64_t pool;

	vd = sav->sav_vdevs[i];
	ASSERT(vd != NULL);

	if (spa_l2cache_exists(vd->vdev_guid, &pool) &&
	pool != 0ULL && l2arc_vdev_present(vd))
	l2arc_remove_vdev(vd);
	}
	}

	/*
	* Verify encryption parameters for spa creation. If we are encrypting, we must
	* have the encryption feature flag enabled.
	*/
	static int
	spa_create_check_encryption_params(dsl_crypto_params_t *dcp,
	boolean_t has_encryption)
	{
	if (dcp->cp_crypt != ZIO_CRYPT_OFF &&
	dcp->cp_crypt != ZIO_CRYPT_INHERIT &&
	!has_encryption)
	return (SET_ERROR(ENOTSUP));

	return (dmu_objset_create_crypt_check(NULL, dcp, NULL));
	}

	/*
	* Pool Creation
	*/
	int
	spa_create(const char pool, nvlist_t nvroot, nvlist_t *props,
	nvlist_t zplprops, dsl_crypto_params_t dcp)
	{
	spa_t *spa;
	char *altroot = NULL;
	vdev_t *rvd;
	dsl_pool_t *dp;
	dmu_tx_t *tx;
	int error = 0;
	uint64_t txg = TXG_INITIAL;
	nvlist_t spares, l2cache;
	uint_t nspares, nl2cache;
	uint64_t version, obj;
	boolean_t has_features;
	boolean_t has_encryption;
	boolean_t has_allocclass;
	spa_feature_t feat;
	char *feat_name;
	char *poolname;
	nvlist_t *nvl;

	if (props == NULL \|\|
	nvlist_lookup_string(props, "tname", &poolname) != 0)
	poolname = (char *)pool;

	/*
	* If this pool already exists, return failure.
	*/
	mutex_enter(&spa_namespace_lock);
	if (spa_lookup(poolname) != NULL) {
	mutex_exit(&spa_namespace_lock);
	return (SET_ERROR(EEXIST));
	}

	/*
	* Allocate a new spa_t structure.
	*/
	nvl = fnvlist_alloc();
	fnvlist_add_string(nvl, ZPOOL_CONFIG_POOL_NAME, pool);
	(void) nvlist_lookup_string(props,
	zpool_prop_to_name(ZPOOL_PROP_ALTROOT), &altroot);
	spa = spa_add(poolname, nvl, altroot);
	fnvlist_free(nvl);
	spa_activate(spa, spa_mode_global);

	if (props && (error = spa_prop_validate(spa, props))) {
	spa_deactivate(spa);
	spa_remove(spa);
	mutex_exit(&spa_namespace_lock);
	return (error);
	}

	/*
	* Temporary pool names should never be written to disk.
	*/
	if (poolname != pool)
	spa->spa_import_flags \|= ZFS_IMPORT_TEMP_NAME;

	has_features = B_FALSE;
	has_encryption = B_FALSE;
	has_allocclass = B_FALSE;
	for (nvpair_t *elem = nvlist_next_nvpair(props, NULL);
	elem != NULL; elem = nvlist_next_nvpair(props, elem)) {
	if (zpool_prop_feature(nvpair_name(elem))) {
	has_features = B_TRUE;

	feat_name = strchr(nvpair_name(elem), '@') + 1;
	VERIFY0(zfeature_lookup_name(feat_name, &feat));
	if (feat == SPA_FEATURE_ENCRYPTION)
	has_encryption = B_TRUE;
	if (feat == SPA_FEATURE_ALLOCATION_CLASSES)
	has_allocclass = B_TRUE;
	}
	}

	/* verify encryption params, if they were provided */
	if (dcp != NULL) {
	error = spa_create_check_encryption_params(dcp, has_encryption);
	if (error != 0) {
	spa_deactivate(spa);
	spa_remove(spa);
	mutex_exit(&spa_namespace_lock);
	return (error);
	}
	}
	if (!has_allocclass && zfs_special_devs(nvroot, NULL)) {
	spa_deactivate(spa);
	spa_remove(spa);
	mutex_exit(&spa_namespace_lock);
	return (ENOTSUP);
	}

	if (has_features \|\| nvlist_lookup_uint64(props,
	zpool_prop_to_name(ZPOOL_PROP_VERSION), &version) != 0) {
	version = SPA_VERSION;
	}
	ASSERT(SPA_VERSION_IS_SUPPORTED(version));

	spa->spa_first_txg = txg;
	spa->spa_uberblock.ub_txg = txg - 1;
	spa->spa_uberblock.ub_version = version;
	spa->spa_ubsync = spa->spa_uberblock;
	spa->spa_load_state = SPA_LOAD_CREATE;
	spa->spa_removing_phys.sr_state = DSS_NONE;
	spa->spa_removing_phys.sr_removing_vdev = -1;
	spa->spa_removing_phys.sr_prev_indirect_vdev = -1;
	spa->spa_indirect_vdevs_loaded = B_TRUE;

	/*
	* Create "The Godfather" zio to hold all async IOs
	*/
	spa->spa_async_zio_root = kmem_alloc(max_ncpus * sizeof (void *),
	KM_SLEEP);
	for (int i = 0; i < max_ncpus; i++) {
	spa->spa_async_zio_root[i] = zio_root(spa, NULL, NULL,
	ZIO_FLAG_CANFAIL \| ZIO_FLAG_SPECULATIVE \|
	ZIO_FLAG_GODFATHER);
	}

	/*
	* Create the root vdev.
	*/
	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);

	error = spa_config_parse(spa, &rvd, nvroot, NULL, 0, VDEV_ALLOC_ADD);

	ASSERT(error != 0 \|\| rvd != NULL);
	ASSERT(error != 0 \|\| spa->spa_root_vdev == rvd);

	if (error == 0 && !zfs_allocatable_devs(nvroot))
	error = SET_ERROR(EINVAL);

	if (error == 0 &&
	(error = vdev_create(rvd, txg, B_FALSE)) == 0 &&
	(error = spa_validate_aux(spa, nvroot, txg,
	VDEV_ALLOC_ADD)) == 0) {
	/*
	* instantiate the metaslab groups (this will dirty the vdevs)
	* we can no longer error exit past this point
	*/
	for (int c = 0; error == 0 && c < rvd->vdev_children; c++) {
	vdev_t *vd = rvd->vdev_child[c];

	vdev_metaslab_set_size(vd);
	vdev_expand(vd, txg);
	}
	}

	spa_config_exit(spa, SCL_ALL, FTAG);

	if (error != 0) {
	spa_unload(spa);
	spa_deactivate(spa);
	spa_remove(spa);
	mutex_exit(&spa_namespace_lock);
	return (error);
	}

	/*
	* Get the list of spares, if specified.
	*/
	if (nvlist_lookup_nvlist_array(nvroot, ZPOOL_CONFIG_SPARES,
	&spares, &nspares) == 0) {
	VERIFY(nvlist_alloc(&spa->spa_spares.sav_config, NV_UNIQUE_NAME,
	KM_SLEEP) == 0);
	VERIFY(nvlist_add_nvlist_array(spa->spa_spares.sav_config,
	ZPOOL_CONFIG_SPARES, spares, nspares) == 0);
	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);
	spa_load_spares(spa);
	spa_config_exit(spa, SCL_ALL, FTAG);
	spa->spa_spares.sav_sync = B_TRUE;
	}

	/*
	* Get the list of level 2 cache devices, if specified.
	*/
	if (nvlist_lookup_nvlist_array(nvroot, ZPOOL_CONFIG_L2CACHE,
	&l2cache, &nl2cache) == 0) {
	VERIFY(nvlist_alloc(&spa->spa_l2cache.sav_config,
	NV_UNIQUE_NAME, KM_SLEEP) == 0);
	VERIFY(nvlist_add_nvlist_array(spa->spa_l2cache.sav_config,
	ZPOOL_CONFIG_L2CACHE, l2cache, nl2cache) == 0);
	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);
	spa_load_l2cache(spa);
	spa_config_exit(spa, SCL_ALL, FTAG);
	spa->spa_l2cache.sav_sync = B_TRUE;
	}

	spa->spa_is_initializing = B_TRUE;
	spa->spa_dsl_pool = dp = dsl_pool_create(spa, zplprops, dcp, txg);
	spa->spa_is_initializing = B_FALSE;

	/*
	* Create DDTs (dedup tables).
	*/
	ddt_create(spa);

	spa_update_dspace(spa);

	tx = dmu_tx_create_assigned(dp, txg);

	/*
	* Create the pool's history object.
	*/
	if (version >= SPA_VERSION_ZPOOL_HISTORY && !spa->spa_history)
	spa_history_create_obj(spa, tx);

	spa_event_notify(spa, NULL, NULL, ESC_ZFS_POOL_CREATE);
	spa_history_log_version(spa, "create", tx);

	/*
	* Create the pool config object.
	*/
	spa->spa_config_object = dmu_object_alloc(spa->spa_meta_objset,
	DMU_OT_PACKED_NVLIST, SPA_CONFIG_BLOCKSIZE,
	DMU_OT_PACKED_NVLIST_SIZE, sizeof (uint64_t), tx);

	if (zap_add(spa->spa_meta_objset,
	DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_CONFIG,
	sizeof (uint64_t), 1, &spa->spa_config_object, tx) != 0) {
	cmn_err(CE_PANIC, "failed to add pool config");
	}

	if (zap_add(spa->spa_meta_objset,
	DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_CREATION_VERSION,
	sizeof (uint64_t), 1, &version, tx) != 0) {
	cmn_err(CE_PANIC, "failed to add pool version");
	}

	/* Newly created pools with the right version are always deflated. */
	if (version >= SPA_VERSION_RAIDZ_DEFLATE) {
	spa->spa_deflate = TRUE;
	if (zap_add(spa->spa_meta_objset,
	DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_DEFLATE,
	sizeof (uint64_t), 1, &spa->spa_deflate, tx) != 0) {
	cmn_err(CE_PANIC, "failed to add deflate");
	}
	}

	/*
	* Create the deferred-free bpobj. Turn off compression
	* because sync-to-convergence takes longer if the blocksize
	* keeps changing.
	*/
	obj = bpobj_alloc(spa->spa_meta_objset, 1 << 14, tx);
	dmu_object_set_compress(spa->spa_meta_objset, obj,
	ZIO_COMPRESS_OFF, tx);
	if (zap_add(spa->spa_meta_objset,
	DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_SYNC_BPOBJ,
	sizeof (uint64_t), 1, &obj, tx) != 0) {
	cmn_err(CE_PANIC, "failed to add bpobj");
	}
	VERIFY3U(0, ==, bpobj_open(&spa->spa_deferred_bpobj,
	spa->spa_meta_objset, obj));

	/*
	* Generate some random noise for salted checksums to operate on.
	*/
	(void) random_get_pseudo_bytes(spa->spa_cksum_salt.zcs_bytes,
	sizeof (spa->spa_cksum_salt.zcs_bytes));

	/*
	* Set pool properties.
	*/
	spa->spa_bootfs = zpool_prop_default_numeric(ZPOOL_PROP_BOOTFS);
	spa->spa_delegation = zpool_prop_default_numeric(ZPOOL_PROP_DELEGATION);
	spa->spa_failmode = zpool_prop_default_numeric(ZPOOL_PROP_FAILUREMODE);
	spa->spa_autoexpand = zpool_prop_default_numeric(ZPOOL_PROP_AUTOEXPAND);
	spa->spa_multihost = zpool_prop_default_numeric(ZPOOL_PROP_MULTIHOST);
	spa->spa_autotrim = zpool_prop_default_numeric(ZPOOL_PROP_AUTOTRIM);

	if (props != NULL) {
	spa_configfile_set(spa, props, B_FALSE);
	spa_sync_props(props, tx);
	}

	dmu_tx_commit(tx);

	spa->spa_sync_on = B_TRUE;
	txg_sync_start(dp);
	mmp_thread_start(spa);
	txg_wait_synced(dp, txg);

	spa_spawn_aux_threads(spa);

	spa_write_cachefile(spa, B_FALSE, B_TRUE);

	/*
	* Don't count references from objsets that are already closed
	* and are making their way through the eviction process.
	*/
	spa_evicting_os_wait(spa);
	spa->spa_minref = zfs_refcount_count(&spa->spa_refcount);
	spa->spa_load_state = SPA_LOAD_NONE;

	mutex_exit(&spa_namespace_lock);

	return (0);
	}

	/*
	* Import a non-root pool into the system.
	*/
	int
	spa_import(char pool, nvlist_t config, nvlist_t *props, uint64_t flags)
	{
	spa_t *spa;
	char *altroot = NULL;
	spa_load_state_t state = SPA_LOAD_IMPORT;
	zpool_load_policy_t policy;
	spa_mode_t mode = spa_mode_global;
	uint64_t readonly = B_FALSE;
	int error;
	nvlist_t *nvroot;
	nvlist_t spares, l2cache;
	uint_t nspares, nl2cache;

	/*
	* If a pool with this name exists, return failure.
	*/
	mutex_enter(&spa_namespace_lock);
	if (spa_lookup(pool) != NULL) {
	mutex_exit(&spa_namespace_lock);
	return (SET_ERROR(EEXIST));
	}

	/*
	* Create and initialize the spa structure.
	*/
	(void) nvlist_lookup_string(props,
	zpool_prop_to_name(ZPOOL_PROP_ALTROOT), &altroot);
	(void) nvlist_lookup_uint64(props,
	zpool_prop_to_name(ZPOOL_PROP_READONLY), &readonly);
	if (readonly)
	mode = SPA_MODE_READ;
	spa = spa_add(pool, config, altroot);
	spa->spa_import_flags = flags;

	/*
	* Verbatim import - Take a pool and insert it into the namespace
	* as if it had been loaded at boot.
	*/
	if (spa->spa_import_flags & ZFS_IMPORT_VERBATIM) {
	if (props != NULL)
	spa_configfile_set(spa, props, B_FALSE);

	spa_write_cachefile(spa, B_FALSE, B_TRUE);
	spa_event_notify(spa, NULL, NULL, ESC_ZFS_POOL_IMPORT);
	zfs_dbgmsg("spa_import: verbatim import of %s", pool);
	mutex_exit(&spa_namespace_lock);
	return (0);
	}

	spa_activate(spa, mode);

	/*
	* Don't start async tasks until we know everything is healthy.
	*/
	spa_async_suspend(spa);

	zpool_get_load_policy(config, &policy);
	if (policy.zlp_rewind & ZPOOL_DO_REWIND)
	state = SPA_LOAD_RECOVER;

	spa->spa_config_source = SPA_CONFIG_SRC_TRYIMPORT;

	if (state != SPA_LOAD_RECOVER) {
	spa->spa_last_ubsync_txg = spa->spa_load_txg = 0;
	zfs_dbgmsg("spa_import: importing %s", pool);
	} else {
	zfs_dbgmsg("spa_import: importing %s, max_txg=%lld "
	"(RECOVERY MODE)", pool, (longlong_t)policy.zlp_txg);
	}
	error = spa_load_best(spa, state, policy.zlp_txg, policy.zlp_rewind);

	/*
	* Propagate anything learned while loading the pool and pass it
	* back to caller (i.e. rewind info, missing devices, etc).
	*/
	VERIFY(nvlist_add_nvlist(config, ZPOOL_CONFIG_LOAD_INFO,
	spa->spa_load_info) == 0);

	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);
	/*
	* Toss any existing sparelist, as it doesn't have any validity
	* anymore, and conflicts with spa_has_spare().
	*/
	if (spa->spa_spares.sav_config) {
	nvlist_free(spa->spa_spares.sav_config);
	spa->spa_spares.sav_config = NULL;
	spa_load_spares(spa);
	}
	if (spa->spa_l2cache.sav_config) {
	nvlist_free(spa->spa_l2cache.sav_config);
	spa->spa_l2cache.sav_config = NULL;
	spa_load_l2cache(spa);
	}

	VERIFY(nvlist_lookup_nvlist(config, ZPOOL_CONFIG_VDEV_TREE,
	&nvroot) == 0);
	spa_config_exit(spa, SCL_ALL, FTAG);

	if (props != NULL)
	spa_configfile_set(spa, props, B_FALSE);

	if (error != 0 \|\| (props && spa_writeable(spa) &&
	(error = spa_prop_set(spa, props)))) {
	spa_unload(spa);
	spa_deactivate(spa);
	spa_remove(spa);
	mutex_exit(&spa_namespace_lock);
	return (error);
	}

	spa_async_resume(spa);

	/*
	* Override any spares and level 2 cache devices as specified by
	* the user, as these may have correct device names/devids, etc.
	*/
	if (nvlist_lookup_nvlist_array(nvroot, ZPOOL_CONFIG_SPARES,
	&spares, &nspares) == 0) {
	if (spa->spa_spares.sav_config)
	VERIFY(nvlist_remove(spa->spa_spares.sav_config,
	ZPOOL_CONFIG_SPARES, DATA_TYPE_NVLIST_ARRAY) == 0);
	else
	VERIFY(nvlist_alloc(&spa->spa_spares.sav_config,
	NV_UNIQUE_NAME, KM_SLEEP) == 0);
	VERIFY(nvlist_add_nvlist_array(spa->spa_spares.sav_config,
	ZPOOL_CONFIG_SPARES, spares, nspares) == 0);
	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);
	spa_load_spares(spa);
	spa_config_exit(spa, SCL_ALL, FTAG);
	spa->spa_spares.sav_sync = B_TRUE;
	}
	if (nvlist_lookup_nvlist_array(nvroot, ZPOOL_CONFIG_L2CACHE,
	&l2cache, &nl2cache) == 0) {
	if (spa->spa_l2cache.sav_config)
	VERIFY(nvlist_remove(spa->spa_l2cache.sav_config,
	ZPOOL_CONFIG_L2CACHE, DATA_TYPE_NVLIST_ARRAY) == 0);
	else
	VERIFY(nvlist_alloc(&spa->spa_l2cache.sav_config,
	NV_UNIQUE_NAME, KM_SLEEP) == 0);
	VERIFY(nvlist_add_nvlist_array(spa->spa_l2cache.sav_config,
	ZPOOL_CONFIG_L2CACHE, l2cache, nl2cache) == 0);
	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);
	spa_load_l2cache(spa);
	spa_config_exit(spa, SCL_ALL, FTAG);
	spa->spa_l2cache.sav_sync = B_TRUE;
	}

	/*
	* Check for any removed devices.
	*/
	if (spa->spa_autoreplace) {
	spa_aux_check_removed(&spa->spa_spares);
	spa_aux_check_removed(&spa->spa_l2cache);
	}

	if (spa_writeable(spa)) {
	/*
	* Update the config cache to include the newly-imported pool.
	*/
	spa_config_update(spa, SPA_CONFIG_UPDATE_POOL);
	}

	/*
	* It's possible that the pool was expanded while it was exported.
	* We kick off an async task to handle this for us.
	*/
	spa_async_request(spa, SPA_ASYNC_AUTOEXPAND);

	spa_history_log_version(spa, "import", NULL);

	spa_event_notify(spa, NULL, NULL, ESC_ZFS_POOL_IMPORT);

	mutex_exit(&spa_namespace_lock);

	zvol_create_minors_recursive(pool);

	return (0);
	}

	nvlist_t *
	spa_tryimport(nvlist_t *tryconfig)
	{
	nvlist_t *config = NULL;
	char poolname, cachefile;
	spa_t *spa;
	uint64_t state;
	int error;
	zpool_load_policy_t policy;

	if (nvlist_lookup_string(tryconfig, ZPOOL_CONFIG_POOL_NAME, &poolname))
	return (NULL);

	if (nvlist_lookup_uint64(tryconfig, ZPOOL_CONFIG_POOL_STATE, &state))
	return (NULL);

	/*
	* Create and initialize the spa structure.
	*/
	mutex_enter(&spa_namespace_lock);
	spa = spa_add(TRYIMPORT_NAME, tryconfig, NULL);
	spa_activate(spa, SPA_MODE_READ);

	/*
	* Rewind pool if a max txg was provided.
	*/
	zpool_get_load_policy(spa->spa_config, &policy);
	if (policy.zlp_txg != UINT64_MAX) {
	spa->spa_load_max_txg = policy.zlp_txg;
	spa->spa_extreme_rewind = B_TRUE;
	zfs_dbgmsg("spa_tryimport: importing %s, max_txg=%lld",
	poolname, (longlong_t)policy.zlp_txg);
	} else {
	zfs_dbgmsg("spa_tryimport: importing %s", poolname);
	}

	if (nvlist_lookup_string(tryconfig, ZPOOL_CONFIG_CACHEFILE, &cachefile)
	== 0) {
	zfs_dbgmsg("spa_tryimport: using cachefile '%s'", cachefile);
	spa->spa_config_source = SPA_CONFIG_SRC_CACHEFILE;
	} else {
	spa->spa_config_source = SPA_CONFIG_SRC_SCAN;
	}

	error = spa_load(spa, SPA_LOAD_TRYIMPORT, SPA_IMPORT_EXISTING);

	/*
	* If 'tryconfig' was at least parsable, return the current config.
	*/
	if (spa->spa_root_vdev != NULL) {
	config = spa_config_generate(spa, NULL, -1ULL, B_TRUE);
	VERIFY(nvlist_add_string(config, ZPOOL_CONFIG_POOL_NAME,
	poolname) == 0);
	VERIFY(nvlist_add_uint64(config, ZPOOL_CONFIG_POOL_STATE,
	state) == 0);
	VERIFY(nvlist_add_uint64(config, ZPOOL_CONFIG_TIMESTAMP,
	spa->spa_uberblock.ub_timestamp) == 0);
	VERIFY(nvlist_add_nvlist(config, ZPOOL_CONFIG_LOAD_INFO,
	spa->spa_load_info) == 0);
	VERIFY(nvlist_add_uint64(config, ZPOOL_CONFIG_ERRATA,
	spa->spa_errata) == 0);

	/*
	* If the bootfs property exists on this pool then we
	* copy it out so that external consumers can tell which
	* pools are bootable.
	*/
	if ((!error \|\| error == EEXIST) && spa->spa_bootfs) {
	char *tmpname = kmem_alloc(MAXPATHLEN, KM_SLEEP);

	/*
	* We have to play games with the name since the
	* pool was opened as TRYIMPORT_NAME.
	*/
	if (dsl_dsobj_to_dsname(spa_name(spa),
	spa->spa_bootfs, tmpname) == 0) {
	char *cp;
	char *dsname;

	dsname = kmem_alloc(MAXPATHLEN, KM_SLEEP);

	cp = strchr(tmpname, '/');
	if (cp == NULL) {
	(void) strlcpy(dsname, tmpname,
	MAXPATHLEN);
	} else {
	(void) snprintf(dsname, MAXPATHLEN,
	"%s/%s", poolname, ++cp);
	}
	VERIFY(nvlist_add_string(config,
	ZPOOL_CONFIG_BOOTFS, dsname) == 0);
	kmem_free(dsname, MAXPATHLEN);
	}
	kmem_free(tmpname, MAXPATHLEN);
	}

	/*
	* Add the list of hot spares and level 2 cache devices.
	*/
	spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER);
	spa_add_spares(spa, config);
	spa_add_l2cache(spa, config);
	spa_config_exit(spa, SCL_CONFIG, FTAG);
	}

	spa_unload(spa);
	spa_deactivate(spa);
	spa_remove(spa);
	mutex_exit(&spa_namespace_lock);

	return (config);
	}

	/*
	* Pool export/destroy
	*
	* The act of destroying or exporting a pool is very simple. We make sure there
	* is no more pending I/O and any references to the pool are gone. Then, we
	* update the pool state and sync all the labels to disk, removing the
	* configuration from the cache afterwards. If the 'hardforce' flag is set, then
	* we don't sync the labels or remove the configuration cache.
	*/
	static int
	-spa_export_common(char pool, int new_state, nvlist_t *oldconfig,
	+spa_export_common(const char pool, int new_state, nvlist_t *oldconfig,
	boolean_t force, boolean_t hardforce)
	{
	spa_t *spa;

	if (oldconfig)
	*oldconfig = NULL;

	if (!(spa_mode_global & SPA_MODE_WRITE))
	return (SET_ERROR(EROFS));

	mutex_enter(&spa_namespace_lock);
	if ((spa = spa_lookup(pool)) == NULL) {
	mutex_exit(&spa_namespace_lock);
	return (SET_ERROR(ENOENT));
	}

	if (spa->spa_is_exporting) {
	/* the pool is being exported by another thread */
	mutex_exit(&spa_namespace_lock);
	return (SET_ERROR(ZFS_ERR_EXPORT_IN_PROGRESS));
	}
	spa->spa_is_exporting = B_TRUE;

	/*
	* Put a hold on the pool, drop the namespace lock, stop async tasks,
	* reacquire the namespace lock, and see if we can export.
	*/
	spa_open_ref(spa, FTAG);
	mutex_exit(&spa_namespace_lock);
	spa_async_suspend(spa);
	if (spa->spa_zvol_taskq) {
	zvol_remove_minors(spa, spa_name(spa), B_TRUE);
	taskq_wait(spa->spa_zvol_taskq);
	}
	mutex_enter(&spa_namespace_lock);
	spa_close(spa, FTAG);

	if (spa->spa_state == POOL_STATE_UNINITIALIZED)
	goto export_spa;
	/*
	* The pool will be in core if it's openable, in which case we can
	* modify its state. Objsets may be open only because they're dirty,
	* so we have to force it to sync before checking spa_refcnt.
	*/
	if (spa->spa_sync_on) {
	txg_wait_synced(spa->spa_dsl_pool, 0);
	spa_evicting_os_wait(spa);
	}

	/*
	* A pool cannot be exported or destroyed if there are active
	* references. If we are resetting a pool, allow references by
	* fault injection handlers.
	*/
	if (!spa_refcount_zero(spa) \|\|
	(spa->spa_inject_ref != 0 &&
	new_state != POOL_STATE_UNINITIALIZED)) {
	spa_async_resume(spa);
	spa->spa_is_exporting = B_FALSE;
	mutex_exit(&spa_namespace_lock);
	return (SET_ERROR(EBUSY));
	}

	if (spa->spa_sync_on) {
	/*
	* A pool cannot be exported if it has an active shared spare.
	* This is to prevent other pools stealing the active spare
	* from an exported pool. At user's own will, such pool can
	* be forcedly exported.
	*/
	if (!force && new_state == POOL_STATE_EXPORTED &&
	spa_has_active_shared_spare(spa)) {
	spa_async_resume(spa);
	spa->spa_is_exporting = B_FALSE;
	mutex_exit(&spa_namespace_lock);
	return (SET_ERROR(EXDEV));
	}

	/*
	* We're about to export or destroy this pool. Make sure
	* we stop all initialization and trim activity here before
	* we set the spa_final_txg. This will ensure that all
	* dirty data resulting from the initialization is
	* committed to disk before we unload the pool.
	*/
	if (spa->spa_root_vdev != NULL) {
	vdev_t *rvd = spa->spa_root_vdev;
	vdev_initialize_stop_all(rvd, VDEV_INITIALIZE_ACTIVE);
	vdev_trim_stop_all(rvd, VDEV_TRIM_ACTIVE);
	vdev_autotrim_stop_all(spa);
	vdev_rebuild_stop_all(spa);
	}

	/*
	* We want this to be reflected on every label,
	* so mark them all dirty. spa_unload() will do the
	* final sync that pushes these changes out.
	*/
	if (new_state != POOL_STATE_UNINITIALIZED && !hardforce) {
	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);
	spa->spa_state = new_state;
	spa->spa_final_txg = spa_last_synced_txg(spa) +
	TXG_DEFER_SIZE + 1;
	vdev_config_dirty(spa->spa_root_vdev);
	spa_config_exit(spa, SCL_ALL, FTAG);
	}
	}

	export_spa:
	if (new_state == POOL_STATE_DESTROYED)
	spa_event_notify(spa, NULL, NULL, ESC_ZFS_POOL_DESTROY);
	else if (new_state == POOL_STATE_EXPORTED)
	spa_event_notify(spa, NULL, NULL, ESC_ZFS_POOL_EXPORT);

	if (spa->spa_state != POOL_STATE_UNINITIALIZED) {
	spa_unload(spa);
	spa_deactivate(spa);
	}

	if (oldconfig && spa->spa_config)
	VERIFY(nvlist_dup(spa->spa_config, oldconfig, 0) == 0);

	if (new_state != POOL_STATE_UNINITIALIZED) {
	if (!hardforce)
	spa_write_cachefile(spa, B_TRUE, B_TRUE);
	spa_remove(spa);
	} else {
	/*
	* If spa_remove() is not called for this spa_t and
	* there is any possibility that it can be reused,
	* we make sure to reset the exporting flag.
	*/
	spa->spa_is_exporting = B_FALSE;
	}

	mutex_exit(&spa_namespace_lock);
	return (0);
	}

	/*
	* Destroy a storage pool.
	*/
	int
	-spa_destroy(char *pool)
	+spa_destroy(const char *pool)
	{
	return (spa_export_common(pool, POOL_STATE_DESTROYED, NULL,
	B_FALSE, B_FALSE));
	}

	/*
	* Export a storage pool.
	*/
	int
	-spa_export(char pool, nvlist_t *oldconfig, boolean_t force,
	+spa_export(const char pool, nvlist_t *oldconfig, boolean_t force,
	boolean_t hardforce)
	{
	return (spa_export_common(pool, POOL_STATE_EXPORTED, oldconfig,
	force, hardforce));
	}

	/*
	* Similar to spa_export(), this unloads the spa_t without actually removing it
	* from the namespace in any way.
	*/
	int
	-spa_reset(char *pool)
	+spa_reset(const char *pool)
	{
	return (spa_export_common(pool, POOL_STATE_UNINITIALIZED, NULL,
	B_FALSE, B_FALSE));
	}

	/*
	* ==========================================================================
	* Device manipulation
	* ==========================================================================
	*/

	/*
	* Add a device to a storage pool.
	*/
	int
	spa_vdev_add(spa_t spa, nvlist_t nvroot)
	{
	uint64_t txg;
	int error;
	vdev_t *rvd = spa->spa_root_vdev;
	vdev_t vd, tvd;
	nvlist_t spares, l2cache;
	uint_t nspares, nl2cache;

	ASSERT(spa_writeable(spa));

	txg = spa_vdev_enter(spa);

	if ((error = spa_config_parse(spa, &vd, nvroot, NULL, 0,
	VDEV_ALLOC_ADD)) != 0)
	return (spa_vdev_exit(spa, NULL, txg, error));

	spa->spa_pending_vdev = vd; /* spa_vdev_exit() will clear this */

	if (nvlist_lookup_nvlist_array(nvroot, ZPOOL_CONFIG_SPARES, &spares,
	&nspares) != 0)
	nspares = 0;

	if (nvlist_lookup_nvlist_array(nvroot, ZPOOL_CONFIG_L2CACHE, &l2cache,
	&nl2cache) != 0)
	nl2cache = 0;

	if (vd->vdev_children == 0 && nspares == 0 && nl2cache == 0)
	return (spa_vdev_exit(spa, vd, txg, EINVAL));

	if (vd->vdev_children != 0 &&
	(error = vdev_create(vd, txg, B_FALSE)) != 0)
	return (spa_vdev_exit(spa, vd, txg, error));

	/*
	* We must validate the spares and l2cache devices after checking the
	* children. Otherwise, vdev_inuse() will blindly overwrite the spare.
	*/
	if ((error = spa_validate_aux(spa, nvroot, txg, VDEV_ALLOC_ADD)) != 0)
	return (spa_vdev_exit(spa, vd, txg, error));

	/*
	* If we are in the middle of a device removal, we can only add
	* devices which match the existing devices in the pool.
	* If we are in the middle of a removal, or have some indirect
	* vdevs, we can not add raidz toplevels.
	*/
	if (spa->spa_vdev_removal != NULL \|\|
	spa->spa_removing_phys.sr_prev_indirect_vdev != -1) {
	for (int c = 0; c < vd->vdev_children; c++) {
	tvd = vd->vdev_child[c];
	if (spa->spa_vdev_removal != NULL &&
	tvd->vdev_ashift != spa->spa_max_ashift) {
	return (spa_vdev_exit(spa, vd, txg, EINVAL));
	}
	/* Fail if top level vdev is raidz */
	if (tvd->vdev_ops == &vdev_raidz_ops) {
	return (spa_vdev_exit(spa, vd, txg, EINVAL));
	}
	/*
	* Need the top level mirror to be
	* a mirror of leaf vdevs only
	*/
	if (tvd->vdev_ops == &vdev_mirror_ops) {
	for (uint64_t cid = 0;
	cid < tvd->vdev_children; cid++) {
	vdev_t *cvd = tvd->vdev_child[cid];
	if (!cvd->vdev_ops->vdev_op_leaf) {
	return (spa_vdev_exit(spa, vd,
	txg, EINVAL));
	}
	}
	}
	}
	}

	for (int c = 0; c < vd->vdev_children; c++) {
	tvd = vd->vdev_child[c];
	vdev_remove_child(vd, tvd);
	tvd->vdev_id = rvd->vdev_children;
	vdev_add_child(rvd, tvd);
	vdev_config_dirty(tvd);
	}

	if (nspares != 0) {
	spa_set_aux_vdevs(&spa->spa_spares, spares, nspares,
	ZPOOL_CONFIG_SPARES);
	spa_load_spares(spa);
	spa->spa_spares.sav_sync = B_TRUE;
	}

	if (nl2cache != 0) {
	spa_set_aux_vdevs(&spa->spa_l2cache, l2cache, nl2cache,
	ZPOOL_CONFIG_L2CACHE);
	spa_load_l2cache(spa);
	spa->spa_l2cache.sav_sync = B_TRUE;
	}

	/*
	* We have to be careful when adding new vdevs to an existing pool.
	* If other threads start allocating from these vdevs before we
	* sync the config cache, and we lose power, then upon reboot we may
	* fail to open the pool because there are DVAs that the config cache
	* can't translate. Therefore, we first add the vdevs without
	* initializing metaslabs; sync the config cache (via spa_vdev_exit());
	* and then let spa_config_update() initialize the new metaslabs.
	*
	* spa_load() checks for added-but-not-initialized vdevs, so that
	* if we lose power at any point in this sequence, the remaining
	* steps will be completed the next time we load the pool.
	*/
	(void) spa_vdev_exit(spa, vd, txg, 0);

	mutex_enter(&spa_namespace_lock);
	spa_config_update(spa, SPA_CONFIG_UPDATE_POOL);
	spa_event_notify(spa, NULL, NULL, ESC_ZFS_VDEV_ADD);
	mutex_exit(&spa_namespace_lock);

	return (0);
	}

	/*
	* Attach a device to a mirror. The arguments are the path to any device
	* in the mirror, and the nvroot for the new device. If the path specifies
	* a device that is not mirrored, we automatically insert the mirror vdev.
	*
	* If 'replacing' is specified, the new device is intended to replace the
	* existing device; in this case the two devices are made into their own
	* mirror using the 'replacing' vdev, which is functionally identical to
	* the mirror vdev (it actually reuses all the same ops) but has a few
	* extra rules: you can't attach to it after it's been created, and upon
	* completion of resilvering, the first disk (the one being replaced)
	* is automatically detached.
	*
	* If 'rebuild' is specified, then sequential reconstruction (a.ka. rebuild)
	* should be performed instead of traditional healing reconstruction. From
	* an administrators perspective these are both resilver operations.
	*/
	int
	spa_vdev_attach(spa_t spa, uint64_t guid, nvlist_t nvroot, int replacing,
	int rebuild)
	{
	uint64_t txg, dtl_max_txg;
	vdev_t *rvd = spa->spa_root_vdev;
	vdev_t oldvd, newvd, newrootvd, pvd, *tvd;
	vdev_ops_t *pvops;
	char oldvdpath, newvdpath;
	int newvd_isspare;
	int error;

	ASSERT(spa_writeable(spa));

	txg = spa_vdev_enter(spa);

	oldvd = spa_lookup_by_guid(spa, guid, B_FALSE);

	ASSERT(MUTEX_HELD(&spa_namespace_lock));
	if (spa_feature_is_active(spa, SPA_FEATURE_POOL_CHECKPOINT)) {
	error = (spa_has_checkpoint(spa)) ?
	ZFS_ERR_CHECKPOINT_EXISTS : ZFS_ERR_DISCARDING_CHECKPOINT;
	return (spa_vdev_exit(spa, NULL, txg, error));
	}

	if (rebuild) {
	if (!spa_feature_is_enabled(spa, SPA_FEATURE_DEVICE_REBUILD))
	return (spa_vdev_exit(spa, NULL, txg, ENOTSUP));

	if (dsl_scan_resilvering(spa_get_dsl(spa)))
	return (spa_vdev_exit(spa, NULL, txg,
	ZFS_ERR_RESILVER_IN_PROGRESS));
	} else {
	if (vdev_rebuild_active(rvd))
	return (spa_vdev_exit(spa, NULL, txg,
	ZFS_ERR_REBUILD_IN_PROGRESS));
	}

	if (spa->spa_vdev_removal != NULL)
	return (spa_vdev_exit(spa, NULL, txg, EBUSY));

	if (oldvd == NULL)
	return (spa_vdev_exit(spa, NULL, txg, ENODEV));

	if (!oldvd->vdev_ops->vdev_op_leaf)
	return (spa_vdev_exit(spa, NULL, txg, ENOTSUP));

	pvd = oldvd->vdev_parent;

	if ((error = spa_config_parse(spa, &newrootvd, nvroot, NULL, 0,
	VDEV_ALLOC_ATTACH)) != 0)
	return (spa_vdev_exit(spa, NULL, txg, EINVAL));

	if (newrootvd->vdev_children != 1)
	return (spa_vdev_exit(spa, newrootvd, txg, EINVAL));

	newvd = newrootvd->vdev_child[0];

	if (!newvd->vdev_ops->vdev_op_leaf)
	return (spa_vdev_exit(spa, newrootvd, txg, EINVAL));

	if ((error = vdev_create(newrootvd, txg, replacing)) != 0)
	return (spa_vdev_exit(spa, newrootvd, txg, error));

	/*
	* Spares can't replace logs
	*/
	if (oldvd->vdev_top->vdev_islog && newvd->vdev_isspare)
	return (spa_vdev_exit(spa, newrootvd, txg, ENOTSUP));

	if (rebuild) {
	/*
	* For rebuilds, the parent vdev must support reconstruction
	* using only space maps. This means the only allowable
	* parents are the root vdev or a mirror vdev.
	*/
	if (pvd->vdev_ops != &vdev_mirror_ops &&
	pvd->vdev_ops != &vdev_root_ops) {
	return (spa_vdev_exit(spa, newrootvd, txg, ENOTSUP));
	}
	}

	if (!replacing) {
	/*
	* For attach, the only allowable parent is a mirror or the root
	* vdev.
	*/
	if (pvd->vdev_ops != &vdev_mirror_ops &&
	pvd->vdev_ops != &vdev_root_ops)
	return (spa_vdev_exit(spa, newrootvd, txg, ENOTSUP));

	pvops = &vdev_mirror_ops;
	} else {
	/*
	* Active hot spares can only be replaced by inactive hot
	* spares.
	*/
	if (pvd->vdev_ops == &vdev_spare_ops &&
	oldvd->vdev_isspare &&
	!spa_has_spare(spa, newvd->vdev_guid))
	return (spa_vdev_exit(spa, newrootvd, txg, ENOTSUP));

	/*
	* If the source is a hot spare, and the parent isn't already a
	* spare, then we want to create a new hot spare. Otherwise, we
	* want to create a replacing vdev. The user is not allowed to
	* attach to a spared vdev child unless the 'isspare' state is
	* the same (spare replaces spare, non-spare replaces
	* non-spare).
	*/
	if (pvd->vdev_ops == &vdev_replacing_ops &&
	spa_version(spa) < SPA_VERSION_MULTI_REPLACE) {
	return (spa_vdev_exit(spa, newrootvd, txg, ENOTSUP));
	} else if (pvd->vdev_ops == &vdev_spare_ops &&
	newvd->vdev_isspare != oldvd->vdev_isspare) {
	return (spa_vdev_exit(spa, newrootvd, txg, ENOTSUP));
	}

	if (newvd->vdev_isspare)
	pvops = &vdev_spare_ops;
	else
	pvops = &vdev_replacing_ops;
	}

	/*
	* Make sure the new device is big enough.
	*/
	if (newvd->vdev_asize < vdev_get_min_asize(oldvd))
	return (spa_vdev_exit(spa, newrootvd, txg, EOVERFLOW));

	/*
	* The new device cannot have a higher alignment requirement
	* than the top-level vdev.
	*/
	if (newvd->vdev_ashift > oldvd->vdev_top->vdev_ashift)
	return (spa_vdev_exit(spa, newrootvd, txg, ENOTSUP));

	/*
	* If this is an in-place replacement, update oldvd's path and devid
	* to make it distinguishable from newvd, and unopenable from now on.
	*/
	if (strcmp(oldvd->vdev_path, newvd->vdev_path) == 0) {
	spa_strfree(oldvd->vdev_path);
	oldvd->vdev_path = kmem_alloc(strlen(newvd->vdev_path) + 5,
	KM_SLEEP);
	(void) snprintf(oldvd->vdev_path, strlen(newvd->vdev_path) + 5,
	"%s/%s", newvd->vdev_path, "old");
	if (oldvd->vdev_devid != NULL) {
	spa_strfree(oldvd->vdev_devid);
	oldvd->vdev_devid = NULL;
	}
	}

	/*
	* If the parent is not a mirror, or if we're replacing, insert the new
	* mirror/replacing/spare vdev above oldvd.
	*/
	if (pvd->vdev_ops != pvops)
	pvd = vdev_add_parent(oldvd, pvops);

	ASSERT(pvd->vdev_top->vdev_parent == rvd);
	ASSERT(pvd->vdev_ops == pvops);
	ASSERT(oldvd->vdev_parent == pvd);

	/*
	* Extract the new device from its root and add it to pvd.
	*/
	vdev_remove_child(newrootvd, newvd);
	newvd->vdev_id = pvd->vdev_children;
	newvd->vdev_crtxg = oldvd->vdev_crtxg;
	vdev_add_child(pvd, newvd);

	/*
	* Reevaluate the parent vdev state.
	*/
	vdev_propagate_state(pvd);

	tvd = newvd->vdev_top;
	ASSERT(pvd->vdev_top == tvd);
	ASSERT(tvd->vdev_parent == rvd);

	vdev_config_dirty(tvd);

	/*
	* Set newvd's DTL to [TXG_INITIAL, dtl_max_txg) so that we account
	* for any dmu_sync-ed blocks. It will propagate upward when
	* spa_vdev_exit() calls vdev_dtl_reassess().
	*/
	dtl_max_txg = txg + TXG_CONCURRENT_STATES;

	vdev_dtl_dirty(newvd, DTL_MISSING,
	TXG_INITIAL, dtl_max_txg - TXG_INITIAL);

	if (newvd->vdev_isspare) {
	spa_spare_activate(newvd);
	spa_event_notify(spa, newvd, NULL, ESC_ZFS_VDEV_SPARE);
	}

	oldvdpath = spa_strdup(oldvd->vdev_path);
	newvdpath = spa_strdup(newvd->vdev_path);
	newvd_isspare = newvd->vdev_isspare;

	/*
	* Mark newvd's DTL dirty in this txg.
	*/
	vdev_dirty(tvd, VDD_DTL, newvd, txg);

	/*
	* Schedule the resilver or rebuild to restart in the future. We do
	* this to ensure that dmu_sync-ed blocks have been stitched into the
	* respective datasets.
	*/
	if (rebuild) {
	newvd->vdev_rebuild_txg = txg;

	vdev_rebuild(tvd);
	} else {
	newvd->vdev_resilver_txg = txg;

	if (dsl_scan_resilvering(spa_get_dsl(spa)) &&
	spa_feature_is_enabled(spa, SPA_FEATURE_RESILVER_DEFER)) {
	vdev_defer_resilver(newvd);
	} else {
	dsl_scan_restart_resilver(spa->spa_dsl_pool,
	dtl_max_txg);
	}
	}

	if (spa->spa_bootfs)
	spa_event_notify(spa, newvd, NULL, ESC_ZFS_BOOTFS_VDEV_ATTACH);

	spa_event_notify(spa, newvd, NULL, ESC_ZFS_VDEV_ATTACH);

	/*
	* Commit the config
	*/
	(void) spa_vdev_exit(spa, newrootvd, dtl_max_txg, 0);

	spa_history_log_internal(spa, "vdev attach", NULL,
	"%s vdev=%s %s vdev=%s",
	replacing && newvd_isspare ? "spare in" :
	replacing ? "replace" : "attach", newvdpath,
	replacing ? "for" : "to", oldvdpath);

	spa_strfree(oldvdpath);
	spa_strfree(newvdpath);

	return (0);
	}

	/*
	* Detach a device from a mirror or replacing vdev.
	*
	* If 'replace_done' is specified, only detach if the parent
	* is a replacing vdev.
	*/
	int
	spa_vdev_detach(spa_t *spa, uint64_t guid, uint64_t pguid, int replace_done)
	{
	uint64_t txg;
	int error;
	vdev_t *rvd __maybe_unused = spa->spa_root_vdev;
	vdev_t vd, pvd, cvd, tvd;
	boolean_t unspare = B_FALSE;
	uint64_t unspare_guid = 0;
	char *vdpath;

	ASSERT(spa_writeable(spa));

	txg = spa_vdev_detach_enter(spa, guid);

	vd = spa_lookup_by_guid(spa, guid, B_FALSE);

	/*
	* Besides being called directly from the userland through the
	* ioctl interface, spa_vdev_detach() can be potentially called
	* at the end of spa_vdev_resilver_done().
	*
	* In the regular case, when we have a checkpoint this shouldn't
	* happen as we never empty the DTLs of a vdev during the scrub
	* [see comment in dsl_scan_done()]. Thus spa_vdev_resilvering_done()
	* should never get here when we have a checkpoint.
	*
	* That said, even in a case when we checkpoint the pool exactly
	* as spa_vdev_resilver_done() calls this function everything
	* should be fine as the resilver will return right away.
	*/
	ASSERT(MUTEX_HELD(&spa_namespace_lock));
	if (spa_feature_is_active(spa, SPA_FEATURE_POOL_CHECKPOINT)) {
	error = (spa_has_checkpoint(spa)) ?
	ZFS_ERR_CHECKPOINT_EXISTS : ZFS_ERR_DISCARDING_CHECKPOINT;
	return (spa_vdev_exit(spa, NULL, txg, error));
	}

	if (vd == NULL)
	return (spa_vdev_exit(spa, NULL, txg, ENODEV));

	if (!vd->vdev_ops->vdev_op_leaf)
	return (spa_vdev_exit(spa, NULL, txg, ENOTSUP));

	pvd = vd->vdev_parent;

	/*
	* If the parent/child relationship is not as expected, don't do it.
	* Consider M(A,R(B,C)) -- that is, a mirror of A with a replacing
	* vdev that's replacing B with C. The user's intent in replacing
	* is to go from M(A,B) to M(A,C). If the user decides to cancel
	* the replace by detaching C, the expected behavior is to end up
	* M(A,B). But suppose that right after deciding to detach C,
	* the replacement of B completes. We would have M(A,C), and then
	* ask to detach C, which would leave us with just A -- not what
	* the user wanted. To prevent this, we make sure that the
	* parent/child relationship hasn't changed -- in this example,
	* that C's parent is still the replacing vdev R.
	*/
	if (pvd->vdev_guid != pguid && pguid != 0)
	return (spa_vdev_exit(spa, NULL, txg, EBUSY));

	/*
	* Only 'replacing' or 'spare' vdevs can be replaced.
	*/
	if (replace_done && pvd->vdev_ops != &vdev_replacing_ops &&
	pvd->vdev_ops != &vdev_spare_ops)
	return (spa_vdev_exit(spa, NULL, txg, ENOTSUP));

	ASSERT(pvd->vdev_ops != &vdev_spare_ops \|\|
	spa_version(spa) >= SPA_VERSION_SPARES);

	/*
	* Only mirror, replacing, and spare vdevs support detach.
	*/
	if (pvd->vdev_ops != &vdev_replacing_ops &&
	pvd->vdev_ops != &vdev_mirror_ops &&
	pvd->vdev_ops != &vdev_spare_ops)
	return (spa_vdev_exit(spa, NULL, txg, ENOTSUP));

	/*
	* If this device has the only valid copy of some data,
	* we cannot safely detach it.
	*/
	if (vdev_dtl_required(vd))
	return (spa_vdev_exit(spa, NULL, txg, EBUSY));

	ASSERT(pvd->vdev_children >= 2);

	/*
	* If we are detaching the second disk from a replacing vdev, then
	* check to see if we changed the original vdev's path to have "/old"
	* at the end in spa_vdev_attach(). If so, undo that change now.
	*/
	if (pvd->vdev_ops == &vdev_replacing_ops && vd->vdev_id > 0 &&
	vd->vdev_path != NULL) {
	size_t len = strlen(vd->vdev_path);

	for (int c = 0; c < pvd->vdev_children; c++) {
	cvd = pvd->vdev_child[c];

	if (cvd == vd \|\| cvd->vdev_path == NULL)
	continue;

	if (strncmp(cvd->vdev_path, vd->vdev_path, len) == 0 &&
	strcmp(cvd->vdev_path + len, "/old") == 0) {
	spa_strfree(cvd->vdev_path);
	cvd->vdev_path = spa_strdup(vd->vdev_path);
	break;
	}
	}
	}

	/*
	* If we are detaching the original disk from a spare, then it implies
	* that the spare should become a real disk, and be removed from the
	* active spare list for the pool.
	*/
	if (pvd->vdev_ops == &vdev_spare_ops &&
	vd->vdev_id == 0 &&
	pvd->vdev_child[pvd->vdev_children - 1]->vdev_isspare)
	unspare = B_TRUE;

	/*
	* Erase the disk labels so the disk can be used for other things.
	* This must be done after all other error cases are handled,
	* but before we disembowel vd (so we can still do I/O to it).
	* But if we can't do it, don't treat the error as fatal --
	* it may be that the unwritability of the disk is the reason
	* it's being detached!
	*/
	error = vdev_label_init(vd, 0, VDEV_LABEL_REMOVE);

	/*
	* Remove vd from its parent and compact the parent's children.
	*/
	vdev_remove_child(pvd, vd);
	vdev_compact_children(pvd);

	/*
	* Remember one of the remaining children so we can get tvd below.
	*/
	cvd = pvd->vdev_child[pvd->vdev_children - 1];

	/*
	* If we need to remove the remaining child from the list of hot spares,
	* do it now, marking the vdev as no longer a spare in the process.
	* We must do this before vdev_remove_parent(), because that can
	* change the GUID if it creates a new toplevel GUID. For a similar
	* reason, we must remove the spare now, in the same txg as the detach;
	* otherwise someone could attach a new sibling, change the GUID, and
	* the subsequent attempt to spa_vdev_remove(unspare_guid) would fail.
	*/
	if (unspare) {
	ASSERT(cvd->vdev_isspare);
	spa_spare_remove(cvd);
	unspare_guid = cvd->vdev_guid;
	(void) spa_vdev_remove(spa, unspare_guid, B_TRUE);
	cvd->vdev_unspare = B_TRUE;
	}

	/*
	* If the parent mirror/replacing vdev only has one child,
	* the parent is no longer needed. Remove it from the tree.
	*/
	if (pvd->vdev_children == 1) {
	if (pvd->vdev_ops == &vdev_spare_ops)
	cvd->vdev_unspare = B_FALSE;
	vdev_remove_parent(cvd);
	}

	/*
	* We don't set tvd until now because the parent we just removed
	* may have been the previous top-level vdev.
	*/
	tvd = cvd->vdev_top;
	ASSERT(tvd->vdev_parent == rvd);

	/*
	* Reevaluate the parent vdev state.
	*/
	vdev_propagate_state(cvd);

	/*
	* If the 'autoexpand' property is set on the pool then automatically
	* try to expand the size of the pool. For example if the device we
	* just detached was smaller than the others, it may be possible to
	* add metaslabs (i.e. grow the pool). We need to reopen the vdev
	* first so that we can obtain the updated sizes of the leaf vdevs.
	*/
	if (spa->spa_autoexpand) {
	vdev_reopen(tvd);
	vdev_expand(tvd, txg);
	}

	vdev_config_dirty(tvd);

	/*
	* Mark vd's DTL as dirty in this txg. vdev_dtl_sync() will see that
	* vd->vdev_detached is set and free vd's DTL object in syncing context.
	* But first make sure we're not on any other txg's DTL list, to
	* prevent vd from being accessed after it's freed.
	*/
	vdpath = spa_strdup(vd->vdev_path ? vd->vdev_path : "none");
	for (int t = 0; t < TXG_SIZE; t++)
	(void) txg_list_remove_this(&tvd->vdev_dtl_list, vd, t);
	vd->vdev_detached = B_TRUE;
	vdev_dirty(tvd, VDD_DTL, vd, txg);

	spa_event_notify(spa, vd, NULL, ESC_ZFS_VDEV_REMOVE);
	spa_notify_waiters(spa);

	/* hang on to the spa before we release the lock */
	spa_open_ref(spa, FTAG);

	error = spa_vdev_exit(spa, vd, txg, 0);

	spa_history_log_internal(spa, "detach", NULL,
	"vdev=%s", vdpath);
	spa_strfree(vdpath);

	/*
	* If this was the removal of the original device in a hot spare vdev,
	* then we want to go through and remove the device from the hot spare
	* list of every other pool.
	*/
	if (unspare) {
	spa_t *altspa = NULL;

	mutex_enter(&spa_namespace_lock);
	while ((altspa = spa_next(altspa)) != NULL) {
	if (altspa->spa_state != POOL_STATE_ACTIVE \|\|
	altspa == spa)
	continue;

	spa_open_ref(altspa, FTAG);
	mutex_exit(&spa_namespace_lock);
	(void) spa_vdev_remove(altspa, unspare_guid, B_TRUE);
	mutex_enter(&spa_namespace_lock);
	spa_close(altspa, FTAG);
	}
	mutex_exit(&spa_namespace_lock);

	/* search the rest of the vdevs for spares to remove */
	spa_vdev_resilver_done(spa);
	}

	/* all done with the spa; OK to release */
	mutex_enter(&spa_namespace_lock);
	spa_close(spa, FTAG);
	mutex_exit(&spa_namespace_lock);

	return (error);
	}

	static int
	spa_vdev_initialize_impl(spa_t *spa, uint64_t guid, uint64_t cmd_type,
	list_t *vd_list)
	{
	ASSERT(MUTEX_HELD(&spa_namespace_lock));

	spa_config_enter(spa, SCL_CONFIG \| SCL_STATE, FTAG, RW_READER);

	/* Look up vdev and ensure it's a leaf. */
	vdev_t *vd = spa_lookup_by_guid(spa, guid, B_FALSE);
	if (vd == NULL \|\| vd->vdev_detached) {
	spa_config_exit(spa, SCL_CONFIG \| SCL_STATE, FTAG);
	return (SET_ERROR(ENODEV));
	} else if (!vd->vdev_ops->vdev_op_leaf \|\| !vdev_is_concrete(vd)) {
	spa_config_exit(spa, SCL_CONFIG \| SCL_STATE, FTAG);
	return (SET_ERROR(EINVAL));
	} else if (!vdev_writeable(vd)) {
	spa_config_exit(spa, SCL_CONFIG \| SCL_STATE, FTAG);
	return (SET_ERROR(EROFS));
	}
	mutex_enter(&vd->vdev_initialize_lock);
	spa_config_exit(spa, SCL_CONFIG \| SCL_STATE, FTAG);

	/*
	* When we activate an initialize action we check to see
	* if the vdev_initialize_thread is NULL. We do this instead
	* of using the vdev_initialize_state since there might be
	* a previous initialization process which has completed but
	* the thread is not exited.
	*/
	if (cmd_type == POOL_INITIALIZE_START &&
	(vd->vdev_initialize_thread != NULL \|\|
	vd->vdev_top->vdev_removing)) {
	mutex_exit(&vd->vdev_initialize_lock);
	return (SET_ERROR(EBUSY));
	} else if (cmd_type == POOL_INITIALIZE_CANCEL &&
	(vd->vdev_initialize_state != VDEV_INITIALIZE_ACTIVE &&
	vd->vdev_initialize_state != VDEV_INITIALIZE_SUSPENDED)) {
	mutex_exit(&vd->vdev_initialize_lock);
	return (SET_ERROR(ESRCH));
	} else if (cmd_type == POOL_INITIALIZE_SUSPEND &&
	vd->vdev_initialize_state != VDEV_INITIALIZE_ACTIVE) {
	mutex_exit(&vd->vdev_initialize_lock);
	return (SET_ERROR(ESRCH));
	}

	switch (cmd_type) {
	case POOL_INITIALIZE_START:
	vdev_initialize(vd);
	break;
	case POOL_INITIALIZE_CANCEL:
	vdev_initialize_stop(vd, VDEV_INITIALIZE_CANCELED, vd_list);
	break;
	case POOL_INITIALIZE_SUSPEND:
	vdev_initialize_stop(vd, VDEV_INITIALIZE_SUSPENDED, vd_list);
	break;
	default:
	panic("invalid cmd_type %llu", (unsigned long long)cmd_type);
	}
	mutex_exit(&vd->vdev_initialize_lock);

	return (0);
	}

	int
	spa_vdev_initialize(spa_t spa, nvlist_t nv, uint64_t cmd_type,
	nvlist_t *vdev_errlist)
	{
	int total_errors = 0;
	list_t vd_list;

	list_create(&vd_list, sizeof (vdev_t),
	offsetof(vdev_t, vdev_initialize_node));

	/*
	* We hold the namespace lock through the whole function
	* to prevent any changes to the pool while we're starting or
	* stopping initialization. The config and state locks are held so that
	* we can properly assess the vdev state before we commit to
	* the initializing operation.
	*/
	mutex_enter(&spa_namespace_lock);

	for (nvpair_t *pair = nvlist_next_nvpair(nv, NULL);
	pair != NULL; pair = nvlist_next_nvpair(nv, pair)) {
	uint64_t vdev_guid = fnvpair_value_uint64(pair);

	int error = spa_vdev_initialize_impl(spa, vdev_guid, cmd_type,
	&vd_list);
	if (error != 0) {
	char guid_as_str[MAXNAMELEN];

	(void) snprintf(guid_as_str, sizeof (guid_as_str),
	"%llu", (unsigned long long)vdev_guid);
	fnvlist_add_int64(vdev_errlist, guid_as_str, error);
	total_errors++;
	}
	}

	/* Wait for all initialize threads to stop. */
	vdev_initialize_stop_wait(spa, &vd_list);

	/* Sync out the initializing state */
	txg_wait_synced(spa->spa_dsl_pool, 0);
	mutex_exit(&spa_namespace_lock);

	list_destroy(&vd_list);

	return (total_errors);
	}

	static int
	spa_vdev_trim_impl(spa_t *spa, uint64_t guid, uint64_t cmd_type,
	uint64_t rate, boolean_t partial, boolean_t secure, list_t *vd_list)
	{
	ASSERT(MUTEX_HELD(&spa_namespace_lock));

	spa_config_enter(spa, SCL_CONFIG \| SCL_STATE, FTAG, RW_READER);

	/* Look up vdev and ensure it's a leaf. */
	vdev_t *vd = spa_lookup_by_guid(spa, guid, B_FALSE);
	if (vd == NULL \|\| vd->vdev_detached) {
	spa_config_exit(spa, SCL_CONFIG \| SCL_STATE, FTAG);
	return (SET_ERROR(ENODEV));
	} else if (!vd->vdev_ops->vdev_op_leaf \|\| !vdev_is_concrete(vd)) {
	spa_config_exit(spa, SCL_CONFIG \| SCL_STATE, FTAG);
	return (SET_ERROR(EINVAL));
	} else if (!vdev_writeable(vd)) {
	spa_config_exit(spa, SCL_CONFIG \| SCL_STATE, FTAG);
	return (SET_ERROR(EROFS));
	} else if (!vd->vdev_has_trim) {
	spa_config_exit(spa, SCL_CONFIG \| SCL_STATE, FTAG);
	return (SET_ERROR(EOPNOTSUPP));
	} else if (secure && !vd->vdev_has_securetrim) {
	spa_config_exit(spa, SCL_CONFIG \| SCL_STATE, FTAG);
	return (SET_ERROR(EOPNOTSUPP));
	}
	mutex_enter(&vd->vdev_trim_lock);
	spa_config_exit(spa, SCL_CONFIG \| SCL_STATE, FTAG);

	/*
	* When we activate a TRIM action we check to see if the
	* vdev_trim_thread is NULL. We do this instead of using the
	* vdev_trim_state since there might be a previous TRIM process
	* which has completed but the thread is not exited.
	*/
	if (cmd_type == POOL_TRIM_START &&
	(vd->vdev_trim_thread != NULL \|\| vd->vdev_top->vdev_removing)) {
	mutex_exit(&vd->vdev_trim_lock);
	return (SET_ERROR(EBUSY));
	} else if (cmd_type == POOL_TRIM_CANCEL &&
	(vd->vdev_trim_state != VDEV_TRIM_ACTIVE &&
	vd->vdev_trim_state != VDEV_TRIM_SUSPENDED)) {
	mutex_exit(&vd->vdev_trim_lock);
	return (SET_ERROR(ESRCH));
	} else if (cmd_type == POOL_TRIM_SUSPEND &&
	vd->vdev_trim_state != VDEV_TRIM_ACTIVE) {
	mutex_exit(&vd->vdev_trim_lock);
	return (SET_ERROR(ESRCH));
	}

	switch (cmd_type) {
	case POOL_TRIM_START:
	vdev_trim(vd, rate, partial, secure);
	break;
	case POOL_TRIM_CANCEL:
	vdev_trim_stop(vd, VDEV_TRIM_CANCELED, vd_list);
	break;
	case POOL_TRIM_SUSPEND:
	vdev_trim_stop(vd, VDEV_TRIM_SUSPENDED, vd_list);
	break;
	default:
	panic("invalid cmd_type %llu", (unsigned long long)cmd_type);
	}
	mutex_exit(&vd->vdev_trim_lock);

	return (0);
	}

	/*
	* Initiates a manual TRIM for the requested vdevs. This kicks off individual
	* TRIM threads for each child vdev. These threads pass over all of the free
	* space in the vdev's metaslabs and issues TRIM commands for that space.
	*/
	int
	spa_vdev_trim(spa_t spa, nvlist_t nv, uint64_t cmd_type, uint64_t rate,
	boolean_t partial, boolean_t secure, nvlist_t *vdev_errlist)
	{
	int total_errors = 0;
	list_t vd_list;

	list_create(&vd_list, sizeof (vdev_t),
	offsetof(vdev_t, vdev_trim_node));

	/*
	* We hold the namespace lock through the whole function
	* to prevent any changes to the pool while we're starting or
	* stopping TRIM. The config and state locks are held so that
	* we can properly assess the vdev state before we commit to
	* the TRIM operation.
	*/
	mutex_enter(&spa_namespace_lock);

	for (nvpair_t *pair = nvlist_next_nvpair(nv, NULL);
	pair != NULL; pair = nvlist_next_nvpair(nv, pair)) {
	uint64_t vdev_guid = fnvpair_value_uint64(pair);

	int error = spa_vdev_trim_impl(spa, vdev_guid, cmd_type,
	rate, partial, secure, &vd_list);
	if (error != 0) {
	char guid_as_str[MAXNAMELEN];

	(void) snprintf(guid_as_str, sizeof (guid_as_str),
	"%llu", (unsigned long long)vdev_guid);
	fnvlist_add_int64(vdev_errlist, guid_as_str, error);
	total_errors++;
	}
	}

	/* Wait for all TRIM threads to stop. */
	vdev_trim_stop_wait(spa, &vd_list);

	/* Sync out the TRIM state */
	txg_wait_synced(spa->spa_dsl_pool, 0);
	mutex_exit(&spa_namespace_lock);

	list_destroy(&vd_list);

	return (total_errors);
	}

	/*
	* Split a set of devices from their mirrors, and create a new pool from them.
	*/
	int
	spa_vdev_split_mirror(spa_t spa, char newname, nvlist_t *config,
	nvlist_t *props, boolean_t exp)
	{
	int error = 0;
	uint64_t txg, *glist;
	spa_t *newspa;
	uint_t c, children, lastlog;
	nvlist_t *child, nvl, *tmp;
	dmu_tx_t *tx;
	char *altroot = NULL;
	vdev_t rvd, vml = NULL; / vdev modify list */
	boolean_t activate_slog;

	ASSERT(spa_writeable(spa));

	txg = spa_vdev_enter(spa);

	ASSERT(MUTEX_HELD(&spa_namespace_lock));
	if (spa_feature_is_active(spa, SPA_FEATURE_POOL_CHECKPOINT)) {
	error = (spa_has_checkpoint(spa)) ?
	ZFS_ERR_CHECKPOINT_EXISTS : ZFS_ERR_DISCARDING_CHECKPOINT;
	return (spa_vdev_exit(spa, NULL, txg, error));
	}

	/* clear the log and flush everything up to now */
	activate_slog = spa_passivate_log(spa);
	(void) spa_vdev_config_exit(spa, NULL, txg, 0, FTAG);
	error = spa_reset_logs(spa);
	txg = spa_vdev_config_enter(spa);

	if (activate_slog)
	spa_activate_log(spa);

	if (error != 0)
	return (spa_vdev_exit(spa, NULL, txg, error));

	/* check new spa name before going any further */
	if (spa_lookup(newname) != NULL)
	return (spa_vdev_exit(spa, NULL, txg, EEXIST));

	/*
	* scan through all the children to ensure they're all mirrors
	*/
	if (nvlist_lookup_nvlist(config, ZPOOL_CONFIG_VDEV_TREE, &nvl) != 0 \|\|
	nvlist_lookup_nvlist_array(nvl, ZPOOL_CONFIG_CHILDREN, &child,
	&children) != 0)
	return (spa_vdev_exit(spa, NULL, txg, EINVAL));

	/* first, check to ensure we've got the right child count */
	rvd = spa->spa_root_vdev;
	lastlog = 0;
	for (c = 0; c < rvd->vdev_children; c++) {
	vdev_t *vd = rvd->vdev_child[c];

	/* don't count the holes & logs as children */
	if (vd->vdev_islog \|\| (vd->vdev_ops != &vdev_indirect_ops &&
	!vdev_is_concrete(vd))) {
	if (lastlog == 0)
	lastlog = c;
	continue;
	}

	lastlog = 0;
	}
	if (children != (lastlog != 0 ? lastlog : rvd->vdev_children))
	return (spa_vdev_exit(spa, NULL, txg, EINVAL));

	/* next, ensure no spare or cache devices are part of the split */
	if (nvlist_lookup_nvlist(nvl, ZPOOL_CONFIG_SPARES, &tmp) == 0 \|\|
	nvlist_lookup_nvlist(nvl, ZPOOL_CONFIG_L2CACHE, &tmp) == 0)
	return (spa_vdev_exit(spa, NULL, txg, EINVAL));

	vml = kmem_zalloc(children * sizeof (vdev_t *), KM_SLEEP);
	glist = kmem_zalloc(children * sizeof (uint64_t), KM_SLEEP);

	/* then, loop over each vdev and validate it */
	for (c = 0; c < children; c++) {
	uint64_t is_hole = 0;

	(void) nvlist_lookup_uint64(child[c], ZPOOL_CONFIG_IS_HOLE,
	&is_hole);

	if (is_hole != 0) {
	if (spa->spa_root_vdev->vdev_child[c]->vdev_ishole \|\|
	spa->spa_root_vdev->vdev_child[c]->vdev_islog) {
	continue;
	} else {
	error = SET_ERROR(EINVAL);
	break;
	}
	}

	/* deal with indirect vdevs */
	if (spa->spa_root_vdev->vdev_child[c]->vdev_ops ==
	&vdev_indirect_ops)
	continue;

	/* which disk is going to be split? */
	if (nvlist_lookup_uint64(child[c], ZPOOL_CONFIG_GUID,
	&glist[c]) != 0) {
	error = SET_ERROR(EINVAL);
	break;
	}

	/* look it up in the spa */
	vml[c] = spa_lookup_by_guid(spa, glist[c], B_FALSE);
	if (vml[c] == NULL) {
	error = SET_ERROR(ENODEV);
	break;
	}

	/* make sure there's nothing stopping the split */
	if (vml[c]->vdev_parent->vdev_ops != &vdev_mirror_ops \|\|
	vml[c]->vdev_islog \|\|
	!vdev_is_concrete(vml[c]) \|\|
	vml[c]->vdev_isspare \|\|
	vml[c]->vdev_isl2cache \|\|
	!vdev_writeable(vml[c]) \|\|
	vml[c]->vdev_children != 0 \|\|
	vml[c]->vdev_state != VDEV_STATE_HEALTHY \|\|
	c != spa->spa_root_vdev->vdev_child[c]->vdev_id) {
	error = SET_ERROR(EINVAL);
	break;
	}

	if (vdev_dtl_required(vml[c]) \|\|
	vdev_resilver_needed(vml[c], NULL, NULL)) {
	error = SET_ERROR(EBUSY);
	break;
	}

	/* we need certain info from the top level */
	VERIFY(nvlist_add_uint64(child[c], ZPOOL_CONFIG_METASLAB_ARRAY,
	vml[c]->vdev_top->vdev_ms_array) == 0);
	VERIFY(nvlist_add_uint64(child[c], ZPOOL_CONFIG_METASLAB_SHIFT,
	vml[c]->vdev_top->vdev_ms_shift) == 0);
	VERIFY(nvlist_add_uint64(child[c], ZPOOL_CONFIG_ASIZE,
	vml[c]->vdev_top->vdev_asize) == 0);
	VERIFY(nvlist_add_uint64(child[c], ZPOOL_CONFIG_ASHIFT,
	vml[c]->vdev_top->vdev_ashift) == 0);

	/* transfer per-vdev ZAPs */
	ASSERT3U(vml[c]->vdev_leaf_zap, !=, 0);
	VERIFY0(nvlist_add_uint64(child[c],
	ZPOOL_CONFIG_VDEV_LEAF_ZAP, vml[c]->vdev_leaf_zap));

	ASSERT3U(vml[c]->vdev_top->vdev_top_zap, !=, 0);
	VERIFY0(nvlist_add_uint64(child[c],
	ZPOOL_CONFIG_VDEV_TOP_ZAP,
	vml[c]->vdev_parent->vdev_top_zap));
	}

	if (error != 0) {
	kmem_free(vml, children * sizeof (vdev_t *));
	kmem_free(glist, children * sizeof (uint64_t));
	return (spa_vdev_exit(spa, NULL, txg, error));
	}

	/* stop writers from using the disks */
	for (c = 0; c < children; c++) {
	if (vml[c] != NULL)
	vml[c]->vdev_offline = B_TRUE;
	}
	vdev_reopen(spa->spa_root_vdev);

	/*
	* Temporarily record the splitting vdevs in the spa config. This
	* will disappear once the config is regenerated.
	*/
	VERIFY(nvlist_alloc(&nvl, NV_UNIQUE_NAME, KM_SLEEP) == 0);
	VERIFY(nvlist_add_uint64_array(nvl, ZPOOL_CONFIG_SPLIT_LIST,
	glist, children) == 0);
	kmem_free(glist, children * sizeof (uint64_t));

	mutex_enter(&spa->spa_props_lock);
	VERIFY(nvlist_add_nvlist(spa->spa_config, ZPOOL_CONFIG_SPLIT,
	nvl) == 0);
	mutex_exit(&spa->spa_props_lock);
	spa->spa_config_splitting = nvl;
	vdev_config_dirty(spa->spa_root_vdev);

	/* configure and create the new pool */
	VERIFY(nvlist_add_string(config, ZPOOL_CONFIG_POOL_NAME, newname) == 0);
	VERIFY(nvlist_add_uint64(config, ZPOOL_CONFIG_POOL_STATE,
	exp ? POOL_STATE_EXPORTED : POOL_STATE_ACTIVE) == 0);
	VERIFY(nvlist_add_uint64(config, ZPOOL_CONFIG_VERSION,
	spa_version(spa)) == 0);
	VERIFY(nvlist_add_uint64(config, ZPOOL_CONFIG_POOL_TXG,
	spa->spa_config_txg) == 0);
	VERIFY(nvlist_add_uint64(config, ZPOOL_CONFIG_POOL_GUID,
	spa_generate_guid(NULL)) == 0);
	VERIFY0(nvlist_add_boolean(config, ZPOOL_CONFIG_HAS_PER_VDEV_ZAPS));
	(void) nvlist_lookup_string(props,
	zpool_prop_to_name(ZPOOL_PROP_ALTROOT), &altroot);

	/* add the new pool to the namespace */
	newspa = spa_add(newname, config, altroot);
	newspa->spa_avz_action = AVZ_ACTION_REBUILD;
	newspa->spa_config_txg = spa->spa_config_txg;
	spa_set_log_state(newspa, SPA_LOG_CLEAR);

	/* release the spa config lock, retaining the namespace lock */
	spa_vdev_config_exit(spa, NULL, txg, 0, FTAG);

	if (zio_injection_enabled)
	zio_handle_panic_injection(spa, FTAG, 1);

	spa_activate(newspa, spa_mode_global);
	spa_async_suspend(newspa);

	/*
	* Temporarily stop the initializing and TRIM activity. We set the
	* state to ACTIVE so that we know to resume initializing or TRIM
	* once the split has completed.
	*/
	list_t vd_initialize_list;
	list_create(&vd_initialize_list, sizeof (vdev_t),
	offsetof(vdev_t, vdev_initialize_node));

	list_t vd_trim_list;
	list_create(&vd_trim_list, sizeof (vdev_t),
	offsetof(vdev_t, vdev_trim_node));

	for (c = 0; c < children; c++) {
	if (vml[c] != NULL && vml[c]->vdev_ops != &vdev_indirect_ops) {
	mutex_enter(&vml[c]->vdev_initialize_lock);
	vdev_initialize_stop(vml[c],
	VDEV_INITIALIZE_ACTIVE, &vd_initialize_list);
	mutex_exit(&vml[c]->vdev_initialize_lock);

	mutex_enter(&vml[c]->vdev_trim_lock);
	vdev_trim_stop(vml[c], VDEV_TRIM_ACTIVE, &vd_trim_list);
	mutex_exit(&vml[c]->vdev_trim_lock);
	}
	}

	vdev_initialize_stop_wait(spa, &vd_initialize_list);
	vdev_trim_stop_wait(spa, &vd_trim_list);

	list_destroy(&vd_initialize_list);
	list_destroy(&vd_trim_list);

	newspa->spa_config_source = SPA_CONFIG_SRC_SPLIT;
	newspa->spa_is_splitting = B_TRUE;

	/* create the new pool from the disks of the original pool */
	error = spa_load(newspa, SPA_LOAD_IMPORT, SPA_IMPORT_ASSEMBLE);
	if (error)
	goto out;

	/* if that worked, generate a real config for the new pool */
	if (newspa->spa_root_vdev != NULL) {
	VERIFY(nvlist_alloc(&newspa->spa_config_splitting,
	NV_UNIQUE_NAME, KM_SLEEP) == 0);
	VERIFY(nvlist_add_uint64(newspa->spa_config_splitting,
	ZPOOL_CONFIG_SPLIT_GUID, spa_guid(spa)) == 0);
	spa_config_set(newspa, spa_config_generate(newspa, NULL, -1ULL,
	B_TRUE));
	}

	/* set the props */
	if (props != NULL) {
	spa_configfile_set(newspa, props, B_FALSE);
	error = spa_prop_set(newspa, props);
	if (error)
	goto out;
	}

	/* flush everything */
	txg = spa_vdev_config_enter(newspa);
	vdev_config_dirty(newspa->spa_root_vdev);
	(void) spa_vdev_config_exit(newspa, NULL, txg, 0, FTAG);

	if (zio_injection_enabled)
	zio_handle_panic_injection(spa, FTAG, 2);

	spa_async_resume(newspa);

	/* finally, update the original pool's config */
	txg = spa_vdev_config_enter(spa);
	tx = dmu_tx_create_dd(spa_get_dsl(spa)->dp_mos_dir);
	error = dmu_tx_assign(tx, TXG_WAIT);
	if (error != 0)
	dmu_tx_abort(tx);
	for (c = 0; c < children; c++) {
	if (vml[c] != NULL && vml[c]->vdev_ops != &vdev_indirect_ops) {
	vdev_t *tvd = vml[c]->vdev_top;

	/*
	* Need to be sure the detachable VDEV is not
	* on any other txg's DTL list to prevent it
	* from being accessed after it's freed.
	*/
	for (int t = 0; t < TXG_SIZE; t++) {
	(void) txg_list_remove_this(
	&tvd->vdev_dtl_list, vml[c], t);
	}

	vdev_split(vml[c]);
	if (error == 0)
	spa_history_log_internal(spa, "detach", tx,
	"vdev=%s", vml[c]->vdev_path);

	vdev_free(vml[c]);
	}
	}
	spa->spa_avz_action = AVZ_ACTION_REBUILD;
	vdev_config_dirty(spa->spa_root_vdev);
	spa->spa_config_splitting = NULL;
	nvlist_free(nvl);
	if (error == 0)
	dmu_tx_commit(tx);
	(void) spa_vdev_exit(spa, NULL, txg, 0);

	if (zio_injection_enabled)
	zio_handle_panic_injection(spa, FTAG, 3);

	/* split is complete; log a history record */
	spa_history_log_internal(newspa, "split", NULL,
	"from pool %s", spa_name(spa));

	newspa->spa_is_splitting = B_FALSE;
	kmem_free(vml, children * sizeof (vdev_t *));

	/* if we're not going to mount the filesystems in userland, export */
	if (exp)
	error = spa_export_common(newname, POOL_STATE_EXPORTED, NULL,
	B_FALSE, B_FALSE);

	return (error);

	out:
	spa_unload(newspa);
	spa_deactivate(newspa);
	spa_remove(newspa);

	txg = spa_vdev_config_enter(spa);

	/* re-online all offlined disks */
	for (c = 0; c < children; c++) {
	if (vml[c] != NULL)
	vml[c]->vdev_offline = B_FALSE;
	}

	/* restart initializing or trimming disks as necessary */
	spa_async_request(spa, SPA_ASYNC_INITIALIZE_RESTART);
	spa_async_request(spa, SPA_ASYNC_TRIM_RESTART);
	spa_async_request(spa, SPA_ASYNC_AUTOTRIM_RESTART);

	vdev_reopen(spa->spa_root_vdev);

	nvlist_free(spa->spa_config_splitting);
	spa->spa_config_splitting = NULL;
	(void) spa_vdev_exit(spa, NULL, txg, error);

	kmem_free(vml, children * sizeof (vdev_t *));
	return (error);
	}

	/*
	* Find any device that's done replacing, or a vdev marked 'unspare' that's
	* currently spared, so we can detach it.
	*/
	static vdev_t *
	spa_vdev_resilver_done_hunt(vdev_t *vd)
	{
	vdev_t newvd, oldvd;

	for (int c = 0; c < vd->vdev_children; c++) {
	oldvd = spa_vdev_resilver_done_hunt(vd->vdev_child[c]);
	if (oldvd != NULL)
	return (oldvd);
	}

	/*
	* Check for a completed replacement. We always consider the first
	* vdev in the list to be the oldest vdev, and the last one to be
	* the newest (see spa_vdev_attach() for how that works). In
	* the case where the newest vdev is faulted, we will not automatically
	* remove it after a resilver completes. This is OK as it will require
	* user intervention to determine which disk the admin wishes to keep.
	*/
	if (vd->vdev_ops == &vdev_replacing_ops) {
	ASSERT(vd->vdev_children > 1);

	newvd = vd->vdev_child[vd->vdev_children - 1];
	oldvd = vd->vdev_child[0];

	if (vdev_dtl_empty(newvd, DTL_MISSING) &&
	vdev_dtl_empty(newvd, DTL_OUTAGE) &&
	!vdev_dtl_required(oldvd))
	return (oldvd);
	}

	/*
	* Check for a completed resilver with the 'unspare' flag set.
	* Also potentially update faulted state.
	*/
	if (vd->vdev_ops == &vdev_spare_ops) {
	vdev_t *first = vd->vdev_child[0];
	vdev_t *last = vd->vdev_child[vd->vdev_children - 1];

	if (last->vdev_unspare) {
	oldvd = first;
	newvd = last;
	} else if (first->vdev_unspare) {
	oldvd = last;
	newvd = first;
	} else {
	oldvd = NULL;
	}

	if (oldvd != NULL &&
	vdev_dtl_empty(newvd, DTL_MISSING) &&
	vdev_dtl_empty(newvd, DTL_OUTAGE) &&
	!vdev_dtl_required(oldvd))
	return (oldvd);

	vdev_propagate_state(vd);

	/*
	* If there are more than two spares attached to a disk,
	* and those spares are not required, then we want to
	* attempt to free them up now so that they can be used
	* by other pools. Once we're back down to a single
	* disk+spare, we stop removing them.
	*/
	if (vd->vdev_children > 2) {
	newvd = vd->vdev_child[1];

	if (newvd->vdev_isspare && last->vdev_isspare &&
	vdev_dtl_empty(last, DTL_MISSING) &&
	vdev_dtl_empty(last, DTL_OUTAGE) &&
	!vdev_dtl_required(newvd))
	return (newvd);
	}
	}

	return (NULL);
	}

	static void
	spa_vdev_resilver_done(spa_t *spa)
	{
	vdev_t vd, pvd, *ppvd;
	uint64_t guid, sguid, pguid, ppguid;

	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);

	while ((vd = spa_vdev_resilver_done_hunt(spa->spa_root_vdev)) != NULL) {
	pvd = vd->vdev_parent;
	ppvd = pvd->vdev_parent;
	guid = vd->vdev_guid;
	pguid = pvd->vdev_guid;
	ppguid = ppvd->vdev_guid;
	sguid = 0;
	/*
	* If we have just finished replacing a hot spared device, then
	* we need to detach the parent's first child (the original hot
	* spare) as well.
	*/
	if (ppvd->vdev_ops == &vdev_spare_ops && pvd->vdev_id == 0 &&
	ppvd->vdev_children == 2) {
	ASSERT(pvd->vdev_ops == &vdev_replacing_ops);
	sguid = ppvd->vdev_child[1]->vdev_guid;
	}
	ASSERT(vd->vdev_resilver_txg == 0 \|\| !vdev_dtl_required(vd));

	spa_config_exit(spa, SCL_ALL, FTAG);
	if (spa_vdev_detach(spa, guid, pguid, B_TRUE) != 0)
	return;
	if (sguid && spa_vdev_detach(spa, sguid, ppguid, B_TRUE) != 0)
	return;
	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);
	}

	spa_config_exit(spa, SCL_ALL, FTAG);

	/*
	* If a detach was not performed above replace waiters will not have
	* been notified. In which case we must do so now.
	*/
	spa_notify_waiters(spa);
	}

	/*
	* Update the stored path or FRU for this vdev.
	*/
	static int
	spa_vdev_set_common(spa_t spa, uint64_t guid, const char value,
	boolean_t ispath)
	{
	vdev_t *vd;
	boolean_t sync = B_FALSE;

	ASSERT(spa_writeable(spa));

	spa_vdev_state_enter(spa, SCL_ALL);

	if ((vd = spa_lookup_by_guid(spa, guid, B_TRUE)) == NULL)
	return (spa_vdev_state_exit(spa, NULL, ENOENT));

	if (!vd->vdev_ops->vdev_op_leaf)
	return (spa_vdev_state_exit(spa, NULL, ENOTSUP));

	if (ispath) {
	if (strcmp(value, vd->vdev_path) != 0) {
	spa_strfree(vd->vdev_path);
	vd->vdev_path = spa_strdup(value);
	sync = B_TRUE;
	}
	} else {
	if (vd->vdev_fru == NULL) {
	vd->vdev_fru = spa_strdup(value);
	sync = B_TRUE;
	} else if (strcmp(value, vd->vdev_fru) != 0) {
	spa_strfree(vd->vdev_fru);
	vd->vdev_fru = spa_strdup(value);
	sync = B_TRUE;
	}
	}

	return (spa_vdev_state_exit(spa, sync ? vd : NULL, 0));
	}

	int
	spa_vdev_setpath(spa_t spa, uint64_t guid, const char newpath)
	{
	return (spa_vdev_set_common(spa, guid, newpath, B_TRUE));
	}

	int
	spa_vdev_setfru(spa_t spa, uint64_t guid, const char newfru)
	{
	return (spa_vdev_set_common(spa, guid, newfru, B_FALSE));
	}

	/*
	* ==========================================================================
	* SPA Scanning
	* ==========================================================================
	*/
	int
	spa_scrub_pause_resume(spa_t *spa, pool_scrub_cmd_t cmd)
	{
	ASSERT(spa_config_held(spa, SCL_ALL, RW_WRITER) == 0);

	if (dsl_scan_resilvering(spa->spa_dsl_pool))
	return (SET_ERROR(EBUSY));

	return (dsl_scrub_set_pause_resume(spa->spa_dsl_pool, cmd));
	}

	int
	spa_scan_stop(spa_t *spa)
	{
	ASSERT(spa_config_held(spa, SCL_ALL, RW_WRITER) == 0);
	if (dsl_scan_resilvering(spa->spa_dsl_pool))
	return (SET_ERROR(EBUSY));
	return (dsl_scan_cancel(spa->spa_dsl_pool));
	}

	int
	spa_scan(spa_t *spa, pool_scan_func_t func)
	{
	ASSERT(spa_config_held(spa, SCL_ALL, RW_WRITER) == 0);

	if (func >= POOL_SCAN_FUNCS \|\| func == POOL_SCAN_NONE)
	return (SET_ERROR(ENOTSUP));

	if (func == POOL_SCAN_RESILVER &&
	!spa_feature_is_enabled(spa, SPA_FEATURE_RESILVER_DEFER))
	return (SET_ERROR(ENOTSUP));

	/*
	* If a resilver was requested, but there is no DTL on a
	* writeable leaf device, we have nothing to do.
	*/
	if (func == POOL_SCAN_RESILVER &&
	!vdev_resilver_needed(spa->spa_root_vdev, NULL, NULL)) {
	spa_async_request(spa, SPA_ASYNC_RESILVER_DONE);
	return (0);
	}

	return (dsl_scan(spa->spa_dsl_pool, func));
	}

	/*
	* ==========================================================================
	* SPA async task processing
	* ==========================================================================
	*/

	static void
	spa_async_remove(spa_t spa, vdev_t vd)
	{
	if (vd->vdev_remove_wanted) {
	vd->vdev_remove_wanted = B_FALSE;
	vd->vdev_delayed_close = B_FALSE;
	vdev_set_state(vd, B_FALSE, VDEV_STATE_REMOVED, VDEV_AUX_NONE);

	/*
	* We want to clear the stats, but we don't want to do a full
	* vdev_clear() as that will cause us to throw away
	* degraded/faulted state as well as attempt to reopen the
	* device, all of which is a waste.
	*/
	vd->vdev_stat.vs_read_errors = 0;
	vd->vdev_stat.vs_write_errors = 0;
	vd->vdev_stat.vs_checksum_errors = 0;

	vdev_state_dirty(vd->vdev_top);
	}

	for (int c = 0; c < vd->vdev_children; c++)
	spa_async_remove(spa, vd->vdev_child[c]);
	}

	static void
	spa_async_probe(spa_t spa, vdev_t vd)
	{
	if (vd->vdev_probe_wanted) {
	vd->vdev_probe_wanted = B_FALSE;
	vdev_reopen(vd); /* vdev_open() does the actual probe */
	}

	for (int c = 0; c < vd->vdev_children; c++)
	spa_async_probe(spa, vd->vdev_child[c]);
	}

	static void
	spa_async_autoexpand(spa_t spa, vdev_t vd)
	{
	if (!spa->spa_autoexpand)
	return;

	for (int c = 0; c < vd->vdev_children; c++) {
	vdev_t *cvd = vd->vdev_child[c];
	spa_async_autoexpand(spa, cvd);
	}

	if (!vd->vdev_ops->vdev_op_leaf \|\| vd->vdev_physpath == NULL)
	return;

	spa_event_notify(vd->vdev_spa, vd, NULL, ESC_ZFS_VDEV_AUTOEXPAND);
	}

	static void
	spa_async_thread(void *arg)
	{
	spa_t spa = (spa_t )arg;
	dsl_pool_t *dp = spa->spa_dsl_pool;
	int tasks;

	ASSERT(spa->spa_sync_on);

	mutex_enter(&spa->spa_async_lock);
	tasks = spa->spa_async_tasks;
	spa->spa_async_tasks = 0;
	mutex_exit(&spa->spa_async_lock);

	/*
	* See if the config needs to be updated.
	*/
	if (tasks & SPA_ASYNC_CONFIG_UPDATE) {
	uint64_t old_space, new_space;

	mutex_enter(&spa_namespace_lock);
	old_space = metaslab_class_get_space(spa_normal_class(spa));
	old_space += metaslab_class_get_space(spa_special_class(spa));
	old_space += metaslab_class_get_space(spa_dedup_class(spa));

	spa_config_update(spa, SPA_CONFIG_UPDATE_POOL);

	new_space = metaslab_class_get_space(spa_normal_class(spa));
	new_space += metaslab_class_get_space(spa_special_class(spa));
	new_space += metaslab_class_get_space(spa_dedup_class(spa));
	mutex_exit(&spa_namespace_lock);

	/*
	* If the pool grew as a result of the config update,
	* then log an internal history event.
	*/
	if (new_space != old_space) {
	spa_history_log_internal(spa, "vdev online", NULL,
	"pool '%s' size: %llu(+%llu)",
	spa_name(spa), (u_longlong_t)new_space,
	(u_longlong_t)(new_space - old_space));
	}
	}

	/*
	* See if any devices need to be marked REMOVED.
	*/
	if (tasks & SPA_ASYNC_REMOVE) {
	spa_vdev_state_enter(spa, SCL_NONE);
	spa_async_remove(spa, spa->spa_root_vdev);
	for (int i = 0; i < spa->spa_l2cache.sav_count; i++)
	spa_async_remove(spa, spa->spa_l2cache.sav_vdevs[i]);
	for (int i = 0; i < spa->spa_spares.sav_count; i++)
	spa_async_remove(spa, spa->spa_spares.sav_vdevs[i]);
	(void) spa_vdev_state_exit(spa, NULL, 0);
	}

	if ((tasks & SPA_ASYNC_AUTOEXPAND) && !spa_suspended(spa)) {
	spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER);
	spa_async_autoexpand(spa, spa->spa_root_vdev);
	spa_config_exit(spa, SCL_CONFIG, FTAG);
	}

	/*
	* See if any devices need to be probed.
	*/
	if (tasks & SPA_ASYNC_PROBE) {
	spa_vdev_state_enter(spa, SCL_NONE);
	spa_async_probe(spa, spa->spa_root_vdev);
	(void) spa_vdev_state_exit(spa, NULL, 0);
	}

	/*
	* If any devices are done replacing, detach them.
	*/
	if (tasks & SPA_ASYNC_RESILVER_DONE)
	spa_vdev_resilver_done(spa);

	/*
	* If any devices are done replacing, detach them. Then if no
	* top-level vdevs are rebuilding attempt to kick off a scrub.
	*/
	if (tasks & SPA_ASYNC_REBUILD_DONE) {
	spa_vdev_resilver_done(spa);

	if (!vdev_rebuild_active(spa->spa_root_vdev))
	(void) dsl_scan(spa->spa_dsl_pool, POOL_SCAN_SCRUB);
	}

	/*
	* Kick off a resilver.
	*/
	if (tasks & SPA_ASYNC_RESILVER &&
	!vdev_rebuild_active(spa->spa_root_vdev) &&
	(!dsl_scan_resilvering(dp) \|\|
	!spa_feature_is_enabled(dp->dp_spa, SPA_FEATURE_RESILVER_DEFER)))
	dsl_scan_restart_resilver(dp, 0);

	if (tasks & SPA_ASYNC_INITIALIZE_RESTART) {
	mutex_enter(&spa_namespace_lock);
	spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER);
	vdev_initialize_restart(spa->spa_root_vdev);
	spa_config_exit(spa, SCL_CONFIG, FTAG);
	mutex_exit(&spa_namespace_lock);
	}

	if (tasks & SPA_ASYNC_TRIM_RESTART) {
	mutex_enter(&spa_namespace_lock);
	spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER);
	vdev_trim_restart(spa->spa_root_vdev);
	spa_config_exit(spa, SCL_CONFIG, FTAG);
	mutex_exit(&spa_namespace_lock);
	}

	if (tasks & SPA_ASYNC_AUTOTRIM_RESTART) {
	mutex_enter(&spa_namespace_lock);
	spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER);
	vdev_autotrim_restart(spa);
	spa_config_exit(spa, SCL_CONFIG, FTAG);
	mutex_exit(&spa_namespace_lock);
	}

	/*
	* Kick off L2 cache whole device TRIM.
	*/
	if (tasks & SPA_ASYNC_L2CACHE_TRIM) {
	mutex_enter(&spa_namespace_lock);
	spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER);
	vdev_trim_l2arc(spa);
	spa_config_exit(spa, SCL_CONFIG, FTAG);
	mutex_exit(&spa_namespace_lock);
	}

	/*
	* Kick off L2 cache rebuilding.
	*/
	if (tasks & SPA_ASYNC_L2CACHE_REBUILD) {
	mutex_enter(&spa_namespace_lock);
	spa_config_enter(spa, SCL_L2ARC, FTAG, RW_READER);
	l2arc_spa_rebuild_start(spa);
	spa_config_exit(spa, SCL_L2ARC, FTAG);
	mutex_exit(&spa_namespace_lock);
	}

	/*
	* Let the world know that we're done.
	*/
	mutex_enter(&spa->spa_async_lock);
	spa->spa_async_thread = NULL;
	cv_broadcast(&spa->spa_async_cv);
	mutex_exit(&spa->spa_async_lock);
	thread_exit();
	}

	void
	spa_async_suspend(spa_t *spa)
	{
	mutex_enter(&spa->spa_async_lock);
	spa->spa_async_suspended++;
	while (spa->spa_async_thread != NULL)
	cv_wait(&spa->spa_async_cv, &spa->spa_async_lock);
	mutex_exit(&spa->spa_async_lock);

	spa_vdev_remove_suspend(spa);

	zthr_t *condense_thread = spa->spa_condense_zthr;
	if (condense_thread != NULL)
	zthr_cancel(condense_thread);

	zthr_t *discard_thread = spa->spa_checkpoint_discard_zthr;
	if (discard_thread != NULL)
	zthr_cancel(discard_thread);

	zthr_t *ll_delete_thread = spa->spa_livelist_delete_zthr;
	if (ll_delete_thread != NULL)
	zthr_cancel(ll_delete_thread);

	zthr_t *ll_condense_thread = spa->spa_livelist_condense_zthr;
	if (ll_condense_thread != NULL)
	zthr_cancel(ll_condense_thread);
	}

	void
	spa_async_resume(spa_t *spa)
	{
	mutex_enter(&spa->spa_async_lock);
	ASSERT(spa->spa_async_suspended != 0);
	spa->spa_async_suspended--;
	mutex_exit(&spa->spa_async_lock);
	spa_restart_removal(spa);

	zthr_t *condense_thread = spa->spa_condense_zthr;
	if (condense_thread != NULL)
	zthr_resume(condense_thread);

	zthr_t *discard_thread = spa->spa_checkpoint_discard_zthr;
	if (discard_thread != NULL)
	zthr_resume(discard_thread);

	zthr_t *ll_delete_thread = spa->spa_livelist_delete_zthr;
	if (ll_delete_thread != NULL)
	zthr_resume(ll_delete_thread);

	zthr_t *ll_condense_thread = spa->spa_livelist_condense_zthr;
	if (ll_condense_thread != NULL)
	zthr_resume(ll_condense_thread);
	}

	static boolean_t
	spa_async_tasks_pending(spa_t *spa)
	{
	uint_t non_config_tasks;
	uint_t config_task;
	boolean_t config_task_suspended;

	non_config_tasks = spa->spa_async_tasks & ~SPA_ASYNC_CONFIG_UPDATE;
	config_task = spa->spa_async_tasks & SPA_ASYNC_CONFIG_UPDATE;
	if (spa->spa_ccw_fail_time == 0) {
	config_task_suspended = B_FALSE;
	} else {
	config_task_suspended =
	(gethrtime() - spa->spa_ccw_fail_time) <
	((hrtime_t)zfs_ccw_retry_interval * NANOSEC);
	}

	return (non_config_tasks \|\| (config_task && !config_task_suspended));
	}

	static void
	spa_async_dispatch(spa_t *spa)
	{
	mutex_enter(&spa->spa_async_lock);
	if (spa_async_tasks_pending(spa) &&
	!spa->spa_async_suspended &&
	spa->spa_async_thread == NULL)
	spa->spa_async_thread = thread_create(NULL, 0,
	spa_async_thread, spa, 0, &p0, TS_RUN, maxclsyspri);
	mutex_exit(&spa->spa_async_lock);
	}

	void
	spa_async_request(spa_t *spa, int task)
	{
	zfs_dbgmsg("spa=%s async request task=%u", spa->spa_name, task);
	mutex_enter(&spa->spa_async_lock);
	spa->spa_async_tasks \|= task;
	mutex_exit(&spa->spa_async_lock);
	}

	int
	spa_async_tasks(spa_t *spa)
	{
	return (spa->spa_async_tasks);
	}

	/*
	* ==========================================================================
	* SPA syncing routines
	* ==========================================================================
	*/


	static int
	bpobj_enqueue_cb(void arg, const blkptr_t bp, boolean_t bp_freed,
	dmu_tx_t *tx)
	{
	bpobj_t *bpo = arg;
	bpobj_enqueue(bpo, bp, bp_freed, tx);
	return (0);
	}

	int
	bpobj_enqueue_alloc_cb(void arg, const blkptr_t bp, dmu_tx_t *tx)
	{
	return (bpobj_enqueue_cb(arg, bp, B_FALSE, tx));
	}

	int
	bpobj_enqueue_free_cb(void arg, const blkptr_t bp, dmu_tx_t *tx)
	{
	return (bpobj_enqueue_cb(arg, bp, B_TRUE, tx));
	}

	static int
	spa_free_sync_cb(void arg, const blkptr_t bp, dmu_tx_t *tx)
	{
	zio_t *pio = arg;

	zio_nowait(zio_free_sync(pio, pio->io_spa, dmu_tx_get_txg(tx), bp,
	pio->io_flags));
	return (0);
	}

	static int
	bpobj_spa_free_sync_cb(void arg, const blkptr_t bp, boolean_t bp_freed,
	dmu_tx_t *tx)
	{
	ASSERT(!bp_freed);
	return (spa_free_sync_cb(arg, bp, tx));
	}

	/*
	* Note: this simple function is not inlined to make it easier to dtrace the
	* amount of time spent syncing frees.
	*/
	static void
	spa_sync_frees(spa_t spa, bplist_t bpl, dmu_tx_t *tx)
	{
	zio_t *zio = zio_root(spa, NULL, NULL, 0);
	bplist_iterate(bpl, spa_free_sync_cb, zio, tx);
	VERIFY(zio_wait(zio) == 0);
	}

	/*
	* Note: this simple function is not inlined to make it easier to dtrace the
	* amount of time spent syncing deferred frees.
	*/
	static void
	spa_sync_deferred_frees(spa_t spa, dmu_tx_t tx)
	{
	if (spa_sync_pass(spa) != 1)
	return;

	/*
	* Note:
	* If the log space map feature is active, we stop deferring
	* frees to the next TXG and therefore running this function
	* would be considered a no-op as spa_deferred_bpobj should
	* not have any entries.
	*
	* That said we run this function anyway (instead of returning
	* immediately) for the edge-case scenario where we just
	* activated the log space map feature in this TXG but we have
	* deferred frees from the previous TXG.
	*/
	zio_t *zio = zio_root(spa, NULL, NULL, 0);
	VERIFY3U(bpobj_iterate(&spa->spa_deferred_bpobj,
	bpobj_spa_free_sync_cb, zio, tx), ==, 0);
	VERIFY0(zio_wait(zio));
	}

	static void
	spa_sync_nvlist(spa_t spa, uint64_t obj, nvlist_t nv, dmu_tx_t *tx)
	{
	char *packed = NULL;
	size_t bufsize;
	size_t nvsize = 0;
	dmu_buf_t *db;

	VERIFY(nvlist_size(nv, &nvsize, NV_ENCODE_XDR) == 0);

	/*
	* Write full (SPA_CONFIG_BLOCKSIZE) blocks of configuration
	* information. This avoids the dmu_buf_will_dirty() path and
	* saves us a pre-read to get data we don't actually care about.
	*/
	bufsize = P2ROUNDUP((uint64_t)nvsize, SPA_CONFIG_BLOCKSIZE);
	packed = vmem_alloc(bufsize, KM_SLEEP);

	VERIFY(nvlist_pack(nv, &packed, &nvsize, NV_ENCODE_XDR,
	KM_SLEEP) == 0);
	bzero(packed + nvsize, bufsize - nvsize);

	dmu_write(spa->spa_meta_objset, obj, 0, bufsize, packed, tx);

	vmem_free(packed, bufsize);

	VERIFY(0 == dmu_bonus_hold(spa->spa_meta_objset, obj, FTAG, &db));
	dmu_buf_will_dirty(db, tx);
	(uint64_t )db->db_data = nvsize;
	dmu_buf_rele(db, FTAG);
	}

	static void
	spa_sync_aux_dev(spa_t spa, spa_aux_vdev_t sav, dmu_tx_t *tx,
	const char config, const char entry)
	{
	nvlist_t *nvroot;
	nvlist_t **list;
	int i;

	if (!sav->sav_sync)
	return;

	/*
	* Update the MOS nvlist describing the list of available devices.
	* spa_validate_aux() will have already made sure this nvlist is
	* valid and the vdevs are labeled appropriately.
	*/
	if (sav->sav_object == 0) {
	sav->sav_object = dmu_object_alloc(spa->spa_meta_objset,
	DMU_OT_PACKED_NVLIST, 1 << 14, DMU_OT_PACKED_NVLIST_SIZE,
	sizeof (uint64_t), tx);
	VERIFY(zap_update(spa->spa_meta_objset,
	DMU_POOL_DIRECTORY_OBJECT, entry, sizeof (uint64_t), 1,
	&sav->sav_object, tx) == 0);
	}

	VERIFY(nvlist_alloc(&nvroot, NV_UNIQUE_NAME, KM_SLEEP) == 0);
	if (sav->sav_count == 0) {
	VERIFY(nvlist_add_nvlist_array(nvroot, config, NULL, 0) == 0);
	} else {
	list = kmem_alloc(sav->sav_countsizeof (void ), KM_SLEEP);
	for (i = 0; i < sav->sav_count; i++)
	list[i] = vdev_config_generate(spa, sav->sav_vdevs[i],
	B_FALSE, VDEV_CONFIG_L2CACHE);
	VERIFY(nvlist_add_nvlist_array(nvroot, config, list,
	sav->sav_count) == 0);
	for (i = 0; i < sav->sav_count; i++)
	nvlist_free(list[i]);
	kmem_free(list, sav->sav_count * sizeof (void *));
	}

	spa_sync_nvlist(spa, sav->sav_object, nvroot, tx);
	nvlist_free(nvroot);

	sav->sav_sync = B_FALSE;
	}

	/*
	* Rebuild spa's all-vdev ZAP from the vdev ZAPs indicated in each vdev_t.
	* The all-vdev ZAP must be empty.
	*/
	static void
	spa_avz_build(vdev_t vd, uint64_t avz, dmu_tx_t tx)
	{
	spa_t *spa = vd->vdev_spa;

	if (vd->vdev_top_zap != 0) {
	VERIFY0(zap_add_int(spa->spa_meta_objset, avz,
	vd->vdev_top_zap, tx));
	}
	if (vd->vdev_leaf_zap != 0) {
	VERIFY0(zap_add_int(spa->spa_meta_objset, avz,
	vd->vdev_leaf_zap, tx));
	}
	for (uint64_t i = 0; i < vd->vdev_children; i++) {
	spa_avz_build(vd->vdev_child[i], avz, tx);
	}
	}

	static void
	spa_sync_config_object(spa_t spa, dmu_tx_t tx)
	{
	nvlist_t *config;

	/*
	* If the pool is being imported from a pre-per-vdev-ZAP version of ZFS,
	* its config may not be dirty but we still need to build per-vdev ZAPs.
	* Similarly, if the pool is being assembled (e.g. after a split), we
	* need to rebuild the AVZ although the config may not be dirty.
	*/
	if (list_is_empty(&spa->spa_config_dirty_list) &&
	spa->spa_avz_action == AVZ_ACTION_NONE)
	return;

	spa_config_enter(spa, SCL_STATE, FTAG, RW_READER);

	ASSERT(spa->spa_avz_action == AVZ_ACTION_NONE \|\|
	spa->spa_avz_action == AVZ_ACTION_INITIALIZE \|\|
	spa->spa_all_vdev_zaps != 0);

	if (spa->spa_avz_action == AVZ_ACTION_REBUILD) {
	/* Make and build the new AVZ */
	uint64_t new_avz = zap_create(spa->spa_meta_objset,
	DMU_OTN_ZAP_METADATA, DMU_OT_NONE, 0, tx);
	spa_avz_build(spa->spa_root_vdev, new_avz, tx);

	/* Diff old AVZ with new one */
	zap_cursor_t zc;
	zap_attribute_t za;

	for (zap_cursor_init(&zc, spa->spa_meta_objset,
	spa->spa_all_vdev_zaps);
	zap_cursor_retrieve(&zc, &za) == 0;
	zap_cursor_advance(&zc)) {
	uint64_t vdzap = za.za_first_integer;
	if (zap_lookup_int(spa->spa_meta_objset, new_avz,
	vdzap) == ENOENT) {
	/*
	* ZAP is listed in old AVZ but not in new one;
	* destroy it
	*/
	VERIFY0(zap_destroy(spa->spa_meta_objset, vdzap,
	tx));
	}
	}

	zap_cursor_fini(&zc);

	/* Destroy the old AVZ */
	VERIFY0(zap_destroy(spa->spa_meta_objset,
	spa->spa_all_vdev_zaps, tx));

	/* Replace the old AVZ in the dir obj with the new one */
	VERIFY0(zap_update(spa->spa_meta_objset,
	DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_VDEV_ZAP_MAP,
	sizeof (new_avz), 1, &new_avz, tx));

	spa->spa_all_vdev_zaps = new_avz;
	} else if (spa->spa_avz_action == AVZ_ACTION_DESTROY) {
	zap_cursor_t zc;
	zap_attribute_t za;

	/* Walk through the AVZ and destroy all listed ZAPs */
	for (zap_cursor_init(&zc, spa->spa_meta_objset,
	spa->spa_all_vdev_zaps);
	zap_cursor_retrieve(&zc, &za) == 0;
	zap_cursor_advance(&zc)) {
	uint64_t zap = za.za_first_integer;
	VERIFY0(zap_destroy(spa->spa_meta_objset, zap, tx));
	}

	zap_cursor_fini(&zc);

	/* Destroy and unlink the AVZ itself */
	VERIFY0(zap_destroy(spa->spa_meta_objset,
	spa->spa_all_vdev_zaps, tx));
	VERIFY0(zap_remove(spa->spa_meta_objset,
	DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_VDEV_ZAP_MAP, tx));
	spa->spa_all_vdev_zaps = 0;
	}

	if (spa->spa_all_vdev_zaps == 0) {
	spa->spa_all_vdev_zaps = zap_create_link(spa->spa_meta_objset,
	DMU_OTN_ZAP_METADATA, DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_VDEV_ZAP_MAP, tx);
	}
	spa->spa_avz_action = AVZ_ACTION_NONE;

	/* Create ZAPs for vdevs that don't have them. */
	vdev_construct_zaps(spa->spa_root_vdev, tx);

	config = spa_config_generate(spa, spa->spa_root_vdev,
	dmu_tx_get_txg(tx), B_FALSE);

	/*
	* If we're upgrading the spa version then make sure that
	* the config object gets updated with the correct version.
	*/
	if (spa->spa_ubsync.ub_version < spa->spa_uberblock.ub_version)
	fnvlist_add_uint64(config, ZPOOL_CONFIG_VERSION,
	spa->spa_uberblock.ub_version);

	spa_config_exit(spa, SCL_STATE, FTAG);

	nvlist_free(spa->spa_config_syncing);
	spa->spa_config_syncing = config;

	spa_sync_nvlist(spa, spa->spa_config_object, config, tx);
	}

	static void
	spa_sync_version(void arg, dmu_tx_t tx)
	{
	uint64_t *versionp = arg;
	uint64_t version = *versionp;
	spa_t *spa = dmu_tx_pool(tx)->dp_spa;

	/*
	* Setting the version is special cased when first creating the pool.
	*/
	ASSERT(tx->tx_txg != TXG_INITIAL);

	ASSERT(SPA_VERSION_IS_SUPPORTED(version));
	ASSERT(version >= spa_version(spa));

	spa->spa_uberblock.ub_version = version;
	vdev_config_dirty(spa->spa_root_vdev);
	spa_history_log_internal(spa, "set", tx, "version=%lld",
	(longlong_t)version);
	}

	/*
	* Set zpool properties.
	*/
	static void
	spa_sync_props(void arg, dmu_tx_t tx)
	{
	nvlist_t *nvp = arg;
	spa_t *spa = dmu_tx_pool(tx)->dp_spa;
	objset_t *mos = spa->spa_meta_objset;
	nvpair_t *elem = NULL;

	mutex_enter(&spa->spa_props_lock);

	while ((elem = nvlist_next_nvpair(nvp, elem))) {
	uint64_t intval;
	char strval, fname;
	zpool_prop_t prop;
	const char *propname;
	zprop_type_t proptype;
	spa_feature_t fid;

	switch (prop = zpool_name_to_prop(nvpair_name(elem))) {
	case ZPOOL_PROP_INVAL:
	/*
	* We checked this earlier in spa_prop_validate().
	*/
	ASSERT(zpool_prop_feature(nvpair_name(elem)));

	fname = strchr(nvpair_name(elem), '@') + 1;
	VERIFY0(zfeature_lookup_name(fname, &fid));

	spa_feature_enable(spa, fid, tx);
	spa_history_log_internal(spa, "set", tx,
	"%s=enabled", nvpair_name(elem));
	break;

	case ZPOOL_PROP_VERSION:
	intval = fnvpair_value_uint64(elem);
	/*
	* The version is synced separately before other
	* properties and should be correct by now.
	*/
	ASSERT3U(spa_version(spa), >=, intval);
	break;

	case ZPOOL_PROP_ALTROOT:
	/*
	* 'altroot' is a non-persistent property. It should
	* have been set temporarily at creation or import time.
	*/
	ASSERT(spa->spa_root != NULL);
	break;

	case ZPOOL_PROP_READONLY:
	case ZPOOL_PROP_CACHEFILE:
	/*
	* 'readonly' and 'cachefile' are also non-persistent
	* properties.
	*/
	break;
	case ZPOOL_PROP_COMMENT:
	strval = fnvpair_value_string(elem);
	if (spa->spa_comment != NULL)
	spa_strfree(spa->spa_comment);
	spa->spa_comment = spa_strdup(strval);
	/*
	* We need to dirty the configuration on all the vdevs
	* so that their labels get updated. It's unnecessary
	* to do this for pool creation since the vdev's
	* configuration has already been dirtied.
	*/
	if (tx->tx_txg != TXG_INITIAL)
	vdev_config_dirty(spa->spa_root_vdev);
	spa_history_log_internal(spa, "set", tx,
	"%s=%s", nvpair_name(elem), strval);
	break;
	default:
	/*
	* Set pool property values in the poolprops mos object.
	*/
	if (spa->spa_pool_props_object == 0) {
	spa->spa_pool_props_object =
	zap_create_link(mos, DMU_OT_POOL_PROPS,
	DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_PROPS,
	tx);
	}

	/* normalize the property name */
	propname = zpool_prop_to_name(prop);
	proptype = zpool_prop_get_type(prop);

	if (nvpair_type(elem) == DATA_TYPE_STRING) {
	ASSERT(proptype == PROP_TYPE_STRING);
	strval = fnvpair_value_string(elem);
	VERIFY0(zap_update(mos,
	spa->spa_pool_props_object, propname,
	1, strlen(strval) + 1, strval, tx));
	spa_history_log_internal(spa, "set", tx,
	"%s=%s", nvpair_name(elem), strval);
	} else if (nvpair_type(elem) == DATA_TYPE_UINT64) {
	intval = fnvpair_value_uint64(elem);

	if (proptype == PROP_TYPE_INDEX) {
	const char *unused;
	VERIFY0(zpool_prop_index_to_string(
	prop, intval, &unused));
	}
	VERIFY0(zap_update(mos,
	spa->spa_pool_props_object, propname,
	8, 1, &intval, tx));
	spa_history_log_internal(spa, "set", tx,
	"%s=%lld", nvpair_name(elem),
	(longlong_t)intval);
	} else {
	ASSERT(0); /* not allowed */
	}

	switch (prop) {
	case ZPOOL_PROP_DELEGATION:
	spa->spa_delegation = intval;
	break;
	case ZPOOL_PROP_BOOTFS:
	spa->spa_bootfs = intval;
	break;
	case ZPOOL_PROP_FAILUREMODE:
	spa->spa_failmode = intval;
	break;
	case ZPOOL_PROP_AUTOTRIM:
	spa->spa_autotrim = intval;
	spa_async_request(spa,
	SPA_ASYNC_AUTOTRIM_RESTART);
	break;
	case ZPOOL_PROP_AUTOEXPAND:
	spa->spa_autoexpand = intval;
	if (tx->tx_txg != TXG_INITIAL)
	spa_async_request(spa,
	SPA_ASYNC_AUTOEXPAND);
	break;
	case ZPOOL_PROP_MULTIHOST:
	spa->spa_multihost = intval;
	break;
	default:
	break;
	}
	}

	}

	mutex_exit(&spa->spa_props_lock);
	}

	/*
	* Perform one-time upgrade on-disk changes. spa_version() does not
	* reflect the new version this txg, so there must be no changes this
	* txg to anything that the upgrade code depends on after it executes.
	* Therefore this must be called after dsl_pool_sync() does the sync
	* tasks.
	*/
	static void
	spa_sync_upgrades(spa_t spa, dmu_tx_t tx)
	{
	if (spa_sync_pass(spa) != 1)
	return;

	dsl_pool_t *dp = spa->spa_dsl_pool;
	rrw_enter(&dp->dp_config_rwlock, RW_WRITER, FTAG);

	if (spa->spa_ubsync.ub_version < SPA_VERSION_ORIGIN &&
	spa->spa_uberblock.ub_version >= SPA_VERSION_ORIGIN) {
	dsl_pool_create_origin(dp, tx);

	/* Keeping the origin open increases spa_minref */
	spa->spa_minref += 3;
	}

	if (spa->spa_ubsync.ub_version < SPA_VERSION_NEXT_CLONES &&
	spa->spa_uberblock.ub_version >= SPA_VERSION_NEXT_CLONES) {
	dsl_pool_upgrade_clones(dp, tx);
	}

	if (spa->spa_ubsync.ub_version < SPA_VERSION_DIR_CLONES &&
	spa->spa_uberblock.ub_version >= SPA_VERSION_DIR_CLONES) {
	dsl_pool_upgrade_dir_clones(dp, tx);

	/* Keeping the freedir open increases spa_minref */
	spa->spa_minref += 3;
	}

	if (spa->spa_ubsync.ub_version < SPA_VERSION_FEATURES &&
	spa->spa_uberblock.ub_version >= SPA_VERSION_FEATURES) {
	spa_feature_create_zap_objects(spa, tx);
	}

	/*
	* LZ4_COMPRESS feature's behaviour was changed to activate_on_enable
	* when possibility to use lz4 compression for metadata was added
	* Old pools that have this feature enabled must be upgraded to have
	* this feature active
	*/
	if (spa->spa_uberblock.ub_version >= SPA_VERSION_FEATURES) {
	boolean_t lz4_en = spa_feature_is_enabled(spa,
	SPA_FEATURE_LZ4_COMPRESS);
	boolean_t lz4_ac = spa_feature_is_active(spa,
	SPA_FEATURE_LZ4_COMPRESS);

	if (lz4_en && !lz4_ac)
	spa_feature_incr(spa, SPA_FEATURE_LZ4_COMPRESS, tx);
	}

	/*
	* If we haven't written the salt, do so now. Note that the
	* feature may not be activated yet, but that's fine since
	* the presence of this ZAP entry is backwards compatible.
	*/
	if (zap_contains(spa->spa_meta_objset, DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_CHECKSUM_SALT) == ENOENT) {
	VERIFY0(zap_add(spa->spa_meta_objset,
	DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_CHECKSUM_SALT, 1,
	sizeof (spa->spa_cksum_salt.zcs_bytes),
	spa->spa_cksum_salt.zcs_bytes, tx));
	}

	rrw_exit(&dp->dp_config_rwlock, FTAG);
	}

	static void
	vdev_indirect_state_sync_verify(vdev_t *vd)
	{
	vdev_indirect_mapping_t *vim __maybe_unused = vd->vdev_indirect_mapping;
	vdev_indirect_births_t *vib __maybe_unused = vd->vdev_indirect_births;

	if (vd->vdev_ops == &vdev_indirect_ops) {
	ASSERT(vim != NULL);
	ASSERT(vib != NULL);
	}

	uint64_t obsolete_sm_object = 0;
	ASSERT0(vdev_obsolete_sm_object(vd, &obsolete_sm_object));
	if (obsolete_sm_object != 0) {
	ASSERT(vd->vdev_obsolete_sm != NULL);
	ASSERT(vd->vdev_removing \|\|
	vd->vdev_ops == &vdev_indirect_ops);
	ASSERT(vdev_indirect_mapping_num_entries(vim) > 0);
	ASSERT(vdev_indirect_mapping_bytes_mapped(vim) > 0);
	ASSERT3U(obsolete_sm_object, ==,
	space_map_object(vd->vdev_obsolete_sm));
	ASSERT3U(vdev_indirect_mapping_bytes_mapped(vim), >=,
	space_map_allocated(vd->vdev_obsolete_sm));
	}
	ASSERT(vd->vdev_obsolete_segments != NULL);

	/*
	* Since frees / remaps to an indirect vdev can only
	* happen in syncing context, the obsolete segments
	* tree must be empty when we start syncing.
	*/
	ASSERT0(range_tree_space(vd->vdev_obsolete_segments));
	}

	/*
	* Set the top-level vdev's max queue depth. Evaluate each top-level's
	* async write queue depth in case it changed. The max queue depth will
	* not change in the middle of syncing out this txg.
	*/
	static void
	spa_sync_adjust_vdev_max_queue_depth(spa_t *spa)
	{
	ASSERT(spa_writeable(spa));

	vdev_t *rvd = spa->spa_root_vdev;
	uint32_t max_queue_depth = zfs_vdev_async_write_max_active *
	zfs_vdev_queue_depth_pct / 100;
	metaslab_class_t *normal = spa_normal_class(spa);
	metaslab_class_t *special = spa_special_class(spa);
	metaslab_class_t *dedup = spa_dedup_class(spa);

	uint64_t slots_per_allocator = 0;
	for (int c = 0; c < rvd->vdev_children; c++) {
	vdev_t *tvd = rvd->vdev_child[c];

	metaslab_group_t *mg = tvd->vdev_mg;
	if (mg == NULL \|\| !metaslab_group_initialized(mg))
	continue;

	metaslab_class_t *mc = mg->mg_class;
	if (mc != normal && mc != special && mc != dedup)
	continue;

	/*
	* It is safe to do a lock-free check here because only async
	* allocations look at mg_max_alloc_queue_depth, and async
	* allocations all happen from spa_sync().
	*/
	for (int i = 0; i < mg->mg_allocators; i++) {
	ASSERT0(zfs_refcount_count(
	&(mg->mg_allocator[i].mga_alloc_queue_depth)));
	}
	mg->mg_max_alloc_queue_depth = max_queue_depth;

	for (int i = 0; i < mg->mg_allocators; i++) {
	mg->mg_allocator[i].mga_cur_max_alloc_queue_depth =
	zfs_vdev_def_queue_depth;
	}
	slots_per_allocator += zfs_vdev_def_queue_depth;
	}

	for (int i = 0; i < spa->spa_alloc_count; i++) {
	ASSERT0(zfs_refcount_count(&normal->mc_alloc_slots[i]));
	ASSERT0(zfs_refcount_count(&special->mc_alloc_slots[i]));
	ASSERT0(zfs_refcount_count(&dedup->mc_alloc_slots[i]));
	normal->mc_alloc_max_slots[i] = slots_per_allocator;
	special->mc_alloc_max_slots[i] = slots_per_allocator;
	dedup->mc_alloc_max_slots[i] = slots_per_allocator;
	}
	normal->mc_alloc_throttle_enabled = zio_dva_throttle_enabled;
	special->mc_alloc_throttle_enabled = zio_dva_throttle_enabled;
	dedup->mc_alloc_throttle_enabled = zio_dva_throttle_enabled;
	}

	static void
	spa_sync_condense_indirect(spa_t spa, dmu_tx_t tx)
	{
	ASSERT(spa_writeable(spa));

	vdev_t *rvd = spa->spa_root_vdev;
	for (int c = 0; c < rvd->vdev_children; c++) {
	vdev_t *vd = rvd->vdev_child[c];
	vdev_indirect_state_sync_verify(vd);

	if (vdev_indirect_should_condense(vd)) {
	spa_condense_indirect_start_sync(vd, tx);
	break;
	}
	}
	}

	static void
	spa_sync_iterate_to_convergence(spa_t spa, dmu_tx_t tx)
	{
	objset_t *mos = spa->spa_meta_objset;
	dsl_pool_t *dp = spa->spa_dsl_pool;
	uint64_t txg = tx->tx_txg;
	bplist_t *free_bpl = &spa->spa_free_bplist[txg & TXG_MASK];

	do {
	int pass = ++spa->spa_sync_pass;

	spa_sync_config_object(spa, tx);
	spa_sync_aux_dev(spa, &spa->spa_spares, tx,
	ZPOOL_CONFIG_SPARES, DMU_POOL_SPARES);
	spa_sync_aux_dev(spa, &spa->spa_l2cache, tx,
	ZPOOL_CONFIG_L2CACHE, DMU_POOL_L2CACHE);
	spa_errlog_sync(spa, txg);
	dsl_pool_sync(dp, txg);

	if (pass < zfs_sync_pass_deferred_free \|\|
	spa_feature_is_active(spa, SPA_FEATURE_LOG_SPACEMAP)) {
	/*
	* If the log space map feature is active we don't
	* care about deferred frees and the deferred bpobj
	* as the log space map should effectively have the
	* same results (i.e. appending only to one object).
	*/
	spa_sync_frees(spa, free_bpl, tx);
	} else {
	/*
	* We can not defer frees in pass 1, because
	* we sync the deferred frees later in pass 1.
	*/
	ASSERT3U(pass, >, 1);
	bplist_iterate(free_bpl, bpobj_enqueue_alloc_cb,
	&spa->spa_deferred_bpobj, tx);
	}

	ddt_sync(spa, txg);
	dsl_scan_sync(dp, tx);
	svr_sync(spa, tx);
	spa_sync_upgrades(spa, tx);

	spa_flush_metaslabs(spa, tx);

	vdev_t *vd = NULL;
	while ((vd = txg_list_remove(&spa->spa_vdev_txg_list, txg))
	!= NULL)
	vdev_sync(vd, txg);

	/*
	* Note: We need to check if the MOS is dirty because we could
	* have marked the MOS dirty without updating the uberblock
	* (e.g. if we have sync tasks but no dirty user data). We need
	* to check the uberblock's rootbp because it is updated if we
	* have synced out dirty data (though in this case the MOS will
	* most likely also be dirty due to second order effects, we
	* don't want to rely on that here).
	*/
	if (pass == 1 &&
	spa->spa_uberblock.ub_rootbp.blk_birth < txg &&
	!dmu_objset_is_dirty(mos, txg)) {
	/*
	* Nothing changed on the first pass, therefore this
	* TXG is a no-op. Avoid syncing deferred frees, so
	* that we can keep this TXG as a no-op.
	*/
	ASSERT(txg_list_empty(&dp->dp_dirty_datasets, txg));
	ASSERT(txg_list_empty(&dp->dp_dirty_dirs, txg));
	ASSERT(txg_list_empty(&dp->dp_sync_tasks, txg));
	ASSERT(txg_list_empty(&dp->dp_early_sync_tasks, txg));
	break;
	}

	spa_sync_deferred_frees(spa, tx);
	} while (dmu_objset_is_dirty(mos, txg));
	}

	/*
	* Rewrite the vdev configuration (which includes the uberblock) to
	* commit the transaction group.
	*
	* If there are no dirty vdevs, we sync the uberblock to a few random
	* top-level vdevs that are known to be visible in the config cache
	* (see spa_vdev_add() for a complete description). If there are dirty
	* vdevs, sync the uberblock to all vdevs.
	*/
	static void
	spa_sync_rewrite_vdev_config(spa_t spa, dmu_tx_t tx)
	{
	vdev_t *rvd = spa->spa_root_vdev;
	uint64_t txg = tx->tx_txg;

	for (;;) {
	int error = 0;

	/*
	* We hold SCL_STATE to prevent vdev open/close/etc.
	* while we're attempting to write the vdev labels.
	*/
	spa_config_enter(spa, SCL_STATE, FTAG, RW_READER);

	if (list_is_empty(&spa->spa_config_dirty_list)) {
	vdev_t *svd[SPA_SYNC_MIN_VDEVS] = { NULL };
	int svdcount = 0;
	int children = rvd->vdev_children;
	int c0 = spa_get_random(children);

	for (int c = 0; c < children; c++) {
	vdev_t *vd =
	rvd->vdev_child[(c0 + c) % children];

	/* Stop when revisiting the first vdev */
	if (c > 0 && svd[0] == vd)
	break;

	if (vd->vdev_ms_array == 0 \|\|
	vd->vdev_islog \|\|
	!vdev_is_concrete(vd))
	continue;

	svd[svdcount++] = vd;
	if (svdcount == SPA_SYNC_MIN_VDEVS)
	break;
	}
	error = vdev_config_sync(svd, svdcount, txg);
	} else {
	error = vdev_config_sync(rvd->vdev_child,
	rvd->vdev_children, txg);
	}

	if (error == 0)
	spa->spa_last_synced_guid = rvd->vdev_guid;

	spa_config_exit(spa, SCL_STATE, FTAG);

	if (error == 0)
	break;
	zio_suspend(spa, NULL, ZIO_SUSPEND_IOERR);
	zio_resume_wait(spa);
	}
	}

	/*
	* Sync the specified transaction group. New blocks may be dirtied as
	* part of the process, so we iterate until it converges.
	*/
	void
	spa_sync(spa_t *spa, uint64_t txg)
	{
	vdev_t *vd = NULL;

	VERIFY(spa_writeable(spa));

	/*
	* Wait for i/os issued in open context that need to complete
	* before this txg syncs.
	*/
	(void) zio_wait(spa->spa_txg_zio[txg & TXG_MASK]);
	spa->spa_txg_zio[txg & TXG_MASK] = zio_root(spa, NULL, NULL,
	ZIO_FLAG_CANFAIL);

	/*
	* Lock out configuration changes.
	*/
	spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER);

	spa->spa_syncing_txg = txg;
	spa->spa_sync_pass = 0;

	for (int i = 0; i < spa->spa_alloc_count; i++) {
	mutex_enter(&spa->spa_alloc_locks[i]);
	VERIFY0(avl_numnodes(&spa->spa_alloc_trees[i]));
	mutex_exit(&spa->spa_alloc_locks[i]);
	}

	/*
	* If there are any pending vdev state changes, convert them
	* into config changes that go out with this transaction group.
	*/
	spa_config_enter(spa, SCL_STATE, FTAG, RW_READER);
	while (list_head(&spa->spa_state_dirty_list) != NULL) {
	/*
	* We need the write lock here because, for aux vdevs,
	* calling vdev_config_dirty() modifies sav_config.
	* This is ugly and will become unnecessary when we
	* eliminate the aux vdev wart by integrating all vdevs
	* into the root vdev tree.
	*/
	spa_config_exit(spa, SCL_CONFIG \| SCL_STATE, FTAG);
	spa_config_enter(spa, SCL_CONFIG \| SCL_STATE, FTAG, RW_WRITER);
	while ((vd = list_head(&spa->spa_state_dirty_list)) != NULL) {
	vdev_state_clean(vd);
	vdev_config_dirty(vd);
	}
	spa_config_exit(spa, SCL_CONFIG \| SCL_STATE, FTAG);
	spa_config_enter(spa, SCL_CONFIG \| SCL_STATE, FTAG, RW_READER);
	}
	spa_config_exit(spa, SCL_STATE, FTAG);

	dsl_pool_t *dp = spa->spa_dsl_pool;
	dmu_tx_t *tx = dmu_tx_create_assigned(dp, txg);

	spa->spa_sync_starttime = gethrtime();
	taskq_cancel_id(system_delay_taskq, spa->spa_deadman_tqid);
	spa->spa_deadman_tqid = taskq_dispatch_delay(system_delay_taskq,
	spa_deadman, spa, TQ_SLEEP, ddi_get_lbolt() +
	NSEC_TO_TICK(spa->spa_deadman_synctime));

	/*
	* If we are upgrading to SPA_VERSION_RAIDZ_DEFLATE this txg,
	* set spa_deflate if we have no raid-z vdevs.
	*/
	if (spa->spa_ubsync.ub_version < SPA_VERSION_RAIDZ_DEFLATE &&
	spa->spa_uberblock.ub_version >= SPA_VERSION_RAIDZ_DEFLATE) {
	vdev_t *rvd = spa->spa_root_vdev;

	int i;
	for (i = 0; i < rvd->vdev_children; i++) {
	vd = rvd->vdev_child[i];
	if (vd->vdev_deflate_ratio != SPA_MINBLOCKSIZE)
	break;
	}
	if (i == rvd->vdev_children) {
	spa->spa_deflate = TRUE;
	VERIFY0(zap_add(spa->spa_meta_objset,
	DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_DEFLATE,
	sizeof (uint64_t), 1, &spa->spa_deflate, tx));
	}
	}

	spa_sync_adjust_vdev_max_queue_depth(spa);

	spa_sync_condense_indirect(spa, tx);

	spa_sync_iterate_to_convergence(spa, tx);

	#ifdef ZFS_DEBUG
	if (!list_is_empty(&spa->spa_config_dirty_list)) {
	/*
	* Make sure that the number of ZAPs for all the vdevs matches
	* the number of ZAPs in the per-vdev ZAP list. This only gets
	* called if the config is dirty; otherwise there may be
	* outstanding AVZ operations that weren't completed in
	* spa_sync_config_object.
	*/
	uint64_t all_vdev_zap_entry_count;
	ASSERT0(zap_count(spa->spa_meta_objset,
	spa->spa_all_vdev_zaps, &all_vdev_zap_entry_count));
	ASSERT3U(vdev_count_verify_zaps(spa->spa_root_vdev), ==,
	all_vdev_zap_entry_count);
	}
	#endif

	if (spa->spa_vdev_removal != NULL) {
	ASSERT0(spa->spa_vdev_removal->svr_bytes_done[txg & TXG_MASK]);
	}

	spa_sync_rewrite_vdev_config(spa, tx);
	dmu_tx_commit(tx);

	taskq_cancel_id(system_delay_taskq, spa->spa_deadman_tqid);
	spa->spa_deadman_tqid = 0;

	/*
	* Clear the dirty config list.
	*/
	while ((vd = list_head(&spa->spa_config_dirty_list)) != NULL)
	vdev_config_clean(vd);

	/*
	* Now that the new config has synced transactionally,
	* let it become visible to the config cache.
	*/
	if (spa->spa_config_syncing != NULL) {
	spa_config_set(spa, spa->spa_config_syncing);
	spa->spa_config_txg = txg;
	spa->spa_config_syncing = NULL;
	}

	dsl_pool_sync_done(dp, txg);

	for (int i = 0; i < spa->spa_alloc_count; i++) {
	mutex_enter(&spa->spa_alloc_locks[i]);
	VERIFY0(avl_numnodes(&spa->spa_alloc_trees[i]));
	mutex_exit(&spa->spa_alloc_locks[i]);
	}

	/*
	* Update usable space statistics.
	*/
	while ((vd = txg_list_remove(&spa->spa_vdev_txg_list, TXG_CLEAN(txg)))
	!= NULL)
	vdev_sync_done(vd, txg);

	metaslab_class_evict_old(spa->spa_normal_class, txg);
	metaslab_class_evict_old(spa->spa_log_class, txg);

	spa_sync_close_syncing_log_sm(spa);

	spa_update_dspace(spa);

	/*
	* It had better be the case that we didn't dirty anything
	* since vdev_config_sync().
	*/
	ASSERT(txg_list_empty(&dp->dp_dirty_datasets, txg));
	ASSERT(txg_list_empty(&dp->dp_dirty_dirs, txg));
	ASSERT(txg_list_empty(&spa->spa_vdev_txg_list, txg));

	while (zfs_pause_spa_sync)
	delay(1);

	spa->spa_sync_pass = 0;

	/*
	* Update the last synced uberblock here. We want to do this at
	* the end of spa_sync() so that consumers of spa_last_synced_txg()
	* will be guaranteed that all the processing associated with
	* that txg has been completed.
	*/
	spa->spa_ubsync = spa->spa_uberblock;
	spa_config_exit(spa, SCL_CONFIG, FTAG);

	spa_handle_ignored_writes(spa);

	/*
	* If any async tasks have been requested, kick them off.
	*/
	spa_async_dispatch(spa);
	}

	/*
	* Sync all pools. We don't want to hold the namespace lock across these
	* operations, so we take a reference on the spa_t and drop the lock during the
	* sync.
	*/
	void
	spa_sync_allpools(void)
	{
	spa_t *spa = NULL;
	mutex_enter(&spa_namespace_lock);
	while ((spa = spa_next(spa)) != NULL) {
	if (spa_state(spa) != POOL_STATE_ACTIVE \|\|
	!spa_writeable(spa) \|\| spa_suspended(spa))
	continue;
	spa_open_ref(spa, FTAG);
	mutex_exit(&spa_namespace_lock);
	txg_wait_synced(spa_get_dsl(spa), 0);
	mutex_enter(&spa_namespace_lock);
	spa_close(spa, FTAG);
	}
	mutex_exit(&spa_namespace_lock);
	}

	/*
	* ==========================================================================
	* Miscellaneous routines
	* ==========================================================================
	*/

	/*
	* Remove all pools in the system.
	*/
	void
	spa_evict_all(void)
	{
	spa_t *spa;

	/*
	* Remove all cached state. All pools should be closed now,
	* so every spa in the AVL tree should be unreferenced.
	*/
	mutex_enter(&spa_namespace_lock);
	while ((spa = spa_next(NULL)) != NULL) {
	/*
	* Stop async tasks. The async thread may need to detach
	* a device that's been replaced, which requires grabbing
	* spa_namespace_lock, so we must drop it here.
	*/
	spa_open_ref(spa, FTAG);
	mutex_exit(&spa_namespace_lock);
	spa_async_suspend(spa);
	mutex_enter(&spa_namespace_lock);
	spa_close(spa, FTAG);

	if (spa->spa_state != POOL_STATE_UNINITIALIZED) {
	spa_unload(spa);
	spa_deactivate(spa);
	}
	spa_remove(spa);
	}
	mutex_exit(&spa_namespace_lock);
	}

	vdev_t *
	spa_lookup_by_guid(spa_t *spa, uint64_t guid, boolean_t aux)
	{
	vdev_t *vd;
	int i;

	if ((vd = vdev_lookup_by_guid(spa->spa_root_vdev, guid)) != NULL)
	return (vd);

	if (aux) {
	for (i = 0; i < spa->spa_l2cache.sav_count; i++) {
	vd = spa->spa_l2cache.sav_vdevs[i];
	if (vd->vdev_guid == guid)
	return (vd);
	}

	for (i = 0; i < spa->spa_spares.sav_count; i++) {
	vd = spa->spa_spares.sav_vdevs[i];
	if (vd->vdev_guid == guid)
	return (vd);
	}
	}

	return (NULL);
	}

	void
	spa_upgrade(spa_t *spa, uint64_t version)
	{
	ASSERT(spa_writeable(spa));

	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);

	/*
	* This should only be called for a non-faulted pool, and since a
	* future version would result in an unopenable pool, this shouldn't be
	* possible.
	*/
	ASSERT(SPA_VERSION_IS_SUPPORTED(spa->spa_uberblock.ub_version));
	ASSERT3U(version, >=, spa->spa_uberblock.ub_version);

	spa->spa_uberblock.ub_version = version;
	vdev_config_dirty(spa->spa_root_vdev);

	spa_config_exit(spa, SCL_ALL, FTAG);

	txg_wait_synced(spa_get_dsl(spa), 0);
	}

	boolean_t
	spa_has_spare(spa_t *spa, uint64_t guid)
	{
	int i;
	uint64_t spareguid;
	spa_aux_vdev_t *sav = &spa->spa_spares;

	for (i = 0; i < sav->sav_count; i++)
	if (sav->sav_vdevs[i]->vdev_guid == guid)
	return (B_TRUE);

	for (i = 0; i < sav->sav_npending; i++) {
	if (nvlist_lookup_uint64(sav->sav_pending[i], ZPOOL_CONFIG_GUID,
	&spareguid) == 0 && spareguid == guid)
	return (B_TRUE);
	}

	return (B_FALSE);
	}

	/*
	* Check if a pool has an active shared spare device.
	* Note: reference count of an active spare is 2, as a spare and as a replace
	*/
	static boolean_t
	spa_has_active_shared_spare(spa_t *spa)
	{
	int i, refcnt;
	uint64_t pool;
	spa_aux_vdev_t *sav = &spa->spa_spares;

	for (i = 0; i < sav->sav_count; i++) {
	if (spa_spare_exists(sav->sav_vdevs[i]->vdev_guid, &pool,
	&refcnt) && pool != 0ULL && pool == spa_guid(spa) &&
	refcnt > 2)
	return (B_TRUE);
	}

	return (B_FALSE);
	}

	uint64_t
	spa_total_metaslabs(spa_t *spa)
	{
	vdev_t *rvd = spa->spa_root_vdev;

	uint64_t m = 0;
	for (uint64_t c = 0; c < rvd->vdev_children; c++) {
	vdev_t *vd = rvd->vdev_child[c];
	if (!vdev_is_concrete(vd))
	continue;
	m += vd->vdev_ms_count;
	}
	return (m);
	}

	/*
	* Notify any waiting threads that some activity has switched from being in-
	* progress to not-in-progress so that the thread can wake up and determine
	* whether it is finished waiting.
	*/
	void
	spa_notify_waiters(spa_t *spa)
	{
	/*
	* Acquiring spa_activities_lock here prevents the cv_broadcast from
	* happening between the waiting thread's check and cv_wait.
	*/
	mutex_enter(&spa->spa_activities_lock);
	cv_broadcast(&spa->spa_activities_cv);
	mutex_exit(&spa->spa_activities_lock);
	}

	/*
	* Notify any waiting threads that the pool is exporting, and then block until
	* they are finished using the spa_t.
	*/
	void
	spa_wake_waiters(spa_t *spa)
	{
	mutex_enter(&spa->spa_activities_lock);
	spa->spa_waiters_cancel = B_TRUE;
	cv_broadcast(&spa->spa_activities_cv);
	while (spa->spa_waiters != 0)
	cv_wait(&spa->spa_waiters_cv, &spa->spa_activities_lock);
	spa->spa_waiters_cancel = B_FALSE;
	mutex_exit(&spa->spa_activities_lock);
	}

	/* Whether the vdev or any of its descendants are being initialized/trimmed. */
	static boolean_t
	spa_vdev_activity_in_progress_impl(vdev_t *vd, zpool_wait_activity_t activity)
	{
	spa_t *spa = vd->vdev_spa;

	ASSERT(spa_config_held(spa, SCL_CONFIG \| SCL_STATE, RW_READER));
	ASSERT(MUTEX_HELD(&spa->spa_activities_lock));
	ASSERT(activity == ZPOOL_WAIT_INITIALIZE \|\|
	activity == ZPOOL_WAIT_TRIM);

	kmutex_t *lock = activity == ZPOOL_WAIT_INITIALIZE ?
	&vd->vdev_initialize_lock : &vd->vdev_trim_lock;

	mutex_exit(&spa->spa_activities_lock);
	mutex_enter(lock);
	mutex_enter(&spa->spa_activities_lock);

	boolean_t in_progress = (activity == ZPOOL_WAIT_INITIALIZE) ?
	(vd->vdev_initialize_state == VDEV_INITIALIZE_ACTIVE) :
	(vd->vdev_trim_state == VDEV_TRIM_ACTIVE);
	mutex_exit(lock);

	if (in_progress)
	return (B_TRUE);

	for (int i = 0; i < vd->vdev_children; i++) {
	if (spa_vdev_activity_in_progress_impl(vd->vdev_child[i],
	activity))
	return (B_TRUE);
	}

	return (B_FALSE);
	}

	/*
	* If use_guid is true, this checks whether the vdev specified by guid is
	* being initialized/trimmed. Otherwise, it checks whether any vdev in the pool
	* is being initialized/trimmed. The caller must hold the config lock and
	* spa_activities_lock.
	*/
	static int
	spa_vdev_activity_in_progress(spa_t *spa, boolean_t use_guid, uint64_t guid,
	zpool_wait_activity_t activity, boolean_t *in_progress)
	{
	mutex_exit(&spa->spa_activities_lock);
	spa_config_enter(spa, SCL_CONFIG \| SCL_STATE, FTAG, RW_READER);
	mutex_enter(&spa->spa_activities_lock);

	vdev_t *vd;
	if (use_guid) {
	vd = spa_lookup_by_guid(spa, guid, B_FALSE);
	if (vd == NULL \|\| !vd->vdev_ops->vdev_op_leaf) {
	spa_config_exit(spa, SCL_CONFIG \| SCL_STATE, FTAG);
	return (EINVAL);
	}
	} else {
	vd = spa->spa_root_vdev;
	}

	*in_progress = spa_vdev_activity_in_progress_impl(vd, activity);

	spa_config_exit(spa, SCL_CONFIG \| SCL_STATE, FTAG);
	return (0);
	}

	/*
	* Locking for waiting threads
	* ---------------------------
	*
	* Waiting threads need a way to check whether a given activity is in progress,
	* and then, if it is, wait for it to complete. Each activity will have some
	* in-memory representation of the relevant on-disk state which can be used to
	* determine whether or not the activity is in progress. The in-memory state and
	* the locking used to protect it will be different for each activity, and may
	* not be suitable for use with a cvar (e.g., some state is protected by the
	* config lock). To allow waiting threads to wait without any races, another
	* lock, spa_activities_lock, is used.
	*
	* When the state is checked, both the activity-specific lock (if there is one)
	* and spa_activities_lock are held. In some cases, the activity-specific lock
	* is acquired explicitly (e.g. the config lock). In others, the locking is
	* internal to some check (e.g. bpobj_is_empty). After checking, the waiting
	* thread releases the activity-specific lock and, if the activity is in
	* progress, then cv_waits using spa_activities_lock.
	*
	* The waiting thread is woken when another thread, one completing some
	* activity, updates the state of the activity and then calls
	* spa_notify_waiters, which will cv_broadcast. This 'completing' thread only
	* needs to hold its activity-specific lock when updating the state, and this
	* lock can (but doesn't have to) be dropped before calling spa_notify_waiters.
	*
	* Because spa_notify_waiters acquires spa_activities_lock before broadcasting,
	* and because it is held when the waiting thread checks the state of the
	* activity, it can never be the case that the completing thread both updates
	* the activity state and cv_broadcasts in between the waiting thread's check
	* and cv_wait. Thus, a waiting thread can never miss a wakeup.
	*
	* In order to prevent deadlock, when the waiting thread does its check, in some
	* cases it will temporarily drop spa_activities_lock in order to acquire the
	* activity-specific lock. The order in which spa_activities_lock and the
	* activity specific lock are acquired in the waiting thread is determined by
	* the order in which they are acquired in the completing thread; if the
	* completing thread calls spa_notify_waiters with the activity-specific lock
	* held, then the waiting thread must also acquire the activity-specific lock
	* first.
	*/

	static int
	spa_activity_in_progress(spa_t *spa, zpool_wait_activity_t activity,
	boolean_t use_tag, uint64_t tag, boolean_t *in_progress)
	{
	int error = 0;

	ASSERT(MUTEX_HELD(&spa->spa_activities_lock));

	switch (activity) {
	case ZPOOL_WAIT_CKPT_DISCARD:
	*in_progress =
	(spa_feature_is_active(spa, SPA_FEATURE_POOL_CHECKPOINT) &&
	zap_contains(spa_meta_objset(spa),
	DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_ZPOOL_CHECKPOINT) ==
	ENOENT);
	break;
	case ZPOOL_WAIT_FREE:
	*in_progress = ((spa_version(spa) >= SPA_VERSION_DEADLISTS &&
	!bpobj_is_empty(&spa->spa_dsl_pool->dp_free_bpobj)) \|\|
	spa_feature_is_active(spa, SPA_FEATURE_ASYNC_DESTROY) \|\|
	spa_livelist_delete_check(spa));
	break;
	case ZPOOL_WAIT_INITIALIZE:
	case ZPOOL_WAIT_TRIM:
	error = spa_vdev_activity_in_progress(spa, use_tag, tag,
	activity, in_progress);
	break;
	case ZPOOL_WAIT_REPLACE:
	mutex_exit(&spa->spa_activities_lock);
	spa_config_enter(spa, SCL_CONFIG \| SCL_STATE, FTAG, RW_READER);
	mutex_enter(&spa->spa_activities_lock);

	*in_progress = vdev_replace_in_progress(spa->spa_root_vdev);
	spa_config_exit(spa, SCL_CONFIG \| SCL_STATE, FTAG);
	break;
	case ZPOOL_WAIT_REMOVE:
	*in_progress = (spa->spa_removing_phys.sr_state ==
	DSS_SCANNING);
	break;
	case ZPOOL_WAIT_RESILVER:
	if ((*in_progress = vdev_rebuild_active(spa->spa_root_vdev)))
	break;
	/* fall through */
	case ZPOOL_WAIT_SCRUB:
	{
	boolean_t scanning, paused, is_scrub;
	dsl_scan_t *scn = spa->spa_dsl_pool->dp_scan;

	is_scrub = (scn->scn_phys.scn_func == POOL_SCAN_SCRUB);
	scanning = (scn->scn_phys.scn_state == DSS_SCANNING);
	paused = dsl_scan_is_paused_scrub(scn);
	*in_progress = (scanning && !paused &&
	is_scrub == (activity == ZPOOL_WAIT_SCRUB));
	break;
	}
	default:
	panic("unrecognized value for activity %d", activity);
	}

	return (error);
	}

	static int
	spa_wait_common(const char *pool, zpool_wait_activity_t activity,
	boolean_t use_tag, uint64_t tag, boolean_t *waited)
	{
	/*
	* The tag is used to distinguish between instances of an activity.
	* 'initialize' and 'trim' are the only activities that we use this for.
	* The other activities can only have a single instance in progress in a
	* pool at one time, making the tag unnecessary.
	*
	* There can be multiple devices being replaced at once, but since they
	* all finish once resilvering finishes, we don't bother keeping track
	* of them individually, we just wait for them all to finish.
	*/
	if (use_tag && activity != ZPOOL_WAIT_INITIALIZE &&
	activity != ZPOOL_WAIT_TRIM)
	return (EINVAL);

	if (activity < 0 \|\| activity >= ZPOOL_WAIT_NUM_ACTIVITIES)
	return (EINVAL);

	spa_t *spa;
	int error = spa_open(pool, &spa, FTAG);
	if (error != 0)
	return (error);

	/*
	* Increment the spa's waiter count so that we can call spa_close and
	* still ensure that the spa_t doesn't get freed before this thread is
	* finished with it when the pool is exported. We want to call spa_close
	* before we start waiting because otherwise the additional ref would
	* prevent the pool from being exported or destroyed throughout the
	* potentially long wait.
	*/
	mutex_enter(&spa->spa_activities_lock);
	spa->spa_waiters++;
	spa_close(spa, FTAG);

	*waited = B_FALSE;
	for (;;) {
	boolean_t in_progress;
	error = spa_activity_in_progress(spa, activity, use_tag, tag,
	&in_progress);

	if (error \|\| !in_progress \|\| spa->spa_waiters_cancel)
	break;

	*waited = B_TRUE;

	if (cv_wait_sig(&spa->spa_activities_cv,
	&spa->spa_activities_lock) == 0) {
	error = EINTR;
	break;
	}
	}

	spa->spa_waiters--;
	cv_signal(&spa->spa_waiters_cv);
	mutex_exit(&spa->spa_activities_lock);

	return (error);
	}

	/*
	* Wait for a particular instance of the specified activity to complete, where
	* the instance is identified by 'tag'
	*/
	int
	spa_wait_tag(const char *pool, zpool_wait_activity_t activity, uint64_t tag,
	boolean_t *waited)
	{
	return (spa_wait_common(pool, activity, B_TRUE, tag, waited));
	}

	/*
	* Wait for all instances of the specified activity complete
	*/
	int
	spa_wait(const char pool, zpool_wait_activity_t activity, boolean_t waited)
	{

	return (spa_wait_common(pool, activity, B_FALSE, 0, waited));
	}

	sysevent_t *
	spa_event_create(spa_t spa, vdev_t vd, nvlist_t hist_nvl, const char name)
	{
	sysevent_t *ev = NULL;
	#ifdef _KERNEL
	nvlist_t *resource;

	resource = zfs_event_create(spa, vd, FM_SYSEVENT_CLASS, name, hist_nvl);
	if (resource) {
	ev = kmem_alloc(sizeof (sysevent_t), KM_SLEEP);
	ev->resource = resource;
	}
	#endif
	return (ev);
	}

	void
	spa_event_post(sysevent_t *ev)
	{
	#ifdef _KERNEL
	if (ev) {
	zfs_zevent_post(ev->resource, NULL, zfs_zevent_post_cb);
	kmem_free(ev, sizeof (*ev));
	}
	#endif
	}

	/*
	* Post a zevent corresponding to the given sysevent. The 'name' must be one
	* of the event definitions in sys/sysevent/eventdefs.h. The payload will be
	* filled in from the spa and (optionally) the vdev. This doesn't do anything
	* in the userland libzpool, as we don't want consumers to misinterpret ztest
	* or zdb as real changes.
	*/
	void
	spa_event_notify(spa_t spa, vdev_t vd, nvlist_t hist_nvl, const char name)
	{
	spa_event_post(spa_event_create(spa, vd, hist_nvl, name));
	}

	/* state manipulation functions */
	EXPORT_SYMBOL(spa_open);
	EXPORT_SYMBOL(spa_open_rewind);
	EXPORT_SYMBOL(spa_get_stats);
	EXPORT_SYMBOL(spa_create);
	EXPORT_SYMBOL(spa_import);
	EXPORT_SYMBOL(spa_tryimport);
	EXPORT_SYMBOL(spa_destroy);
	EXPORT_SYMBOL(spa_export);
	EXPORT_SYMBOL(spa_reset);
	EXPORT_SYMBOL(spa_async_request);
	EXPORT_SYMBOL(spa_async_suspend);
	EXPORT_SYMBOL(spa_async_resume);
	EXPORT_SYMBOL(spa_inject_addref);
	EXPORT_SYMBOL(spa_inject_delref);
	EXPORT_SYMBOL(spa_scan_stat_init);
	EXPORT_SYMBOL(spa_scan_get_stats);

	/* device manipulation */
	EXPORT_SYMBOL(spa_vdev_add);
	EXPORT_SYMBOL(spa_vdev_attach);
	EXPORT_SYMBOL(spa_vdev_detach);
	EXPORT_SYMBOL(spa_vdev_setpath);
	EXPORT_SYMBOL(spa_vdev_setfru);
	EXPORT_SYMBOL(spa_vdev_split_mirror);

	/* spare statech is global across all pools) */
	EXPORT_SYMBOL(spa_spare_add);
	EXPORT_SYMBOL(spa_spare_remove);
	EXPORT_SYMBOL(spa_spare_exists);
	EXPORT_SYMBOL(spa_spare_activate);

	/* L2ARC statech is global across all pools) */
	EXPORT_SYMBOL(spa_l2cache_add);
	EXPORT_SYMBOL(spa_l2cache_remove);
	EXPORT_SYMBOL(spa_l2cache_exists);
	EXPORT_SYMBOL(spa_l2cache_activate);
	EXPORT_SYMBOL(spa_l2cache_drop);

	/* scanning */
	EXPORT_SYMBOL(spa_scan);
	EXPORT_SYMBOL(spa_scan_stop);

	/* spa syncing */
	EXPORT_SYMBOL(spa_sync); /* only for DMU use */
	EXPORT_SYMBOL(spa_sync_allpools);

	/* properties */
	EXPORT_SYMBOL(spa_prop_set);
	EXPORT_SYMBOL(spa_prop_get);
	EXPORT_SYMBOL(spa_prop_clear_bootfs);

	/* asynchronous event notification */
	EXPORT_SYMBOL(spa_event_notify);

	/* BEGIN CSTYLED */
	ZFS_MODULE_PARAM(zfs_spa, spa_, load_verify_shift, INT, ZMOD_RW,
	"log2(fraction of arc that can be used by inflight I/Os when "
	"verifying pool during import");

	ZFS_MODULE_PARAM(zfs_spa, spa_, load_verify_metadata, INT, ZMOD_RW,
	"Set to traverse metadata on pool import");

	ZFS_MODULE_PARAM(zfs_spa, spa_, load_verify_data, INT, ZMOD_RW,
	"Set to traverse data on pool import");

	ZFS_MODULE_PARAM(zfs_spa, spa_, load_print_vdev_tree, INT, ZMOD_RW,
	"Print vdev tree to zfs_dbgmsg during pool import");

	ZFS_MODULE_PARAM(zfs_zio, zio_, taskq_batch_pct, UINT, ZMOD_RD,
	"Percentage of CPUs to run an IO worker thread");

	ZFS_MODULE_PARAM(zfs, zfs_, max_missing_tvds, ULONG, ZMOD_RW,
	"Allow importing pool with up to this number of missing top-level "
	"vdevs (in read-only mode)");

	ZFS_MODULE_PARAM(zfs_livelist_condense, zfs_livelist_condense_, zthr_pause, INT, ZMOD_RW,
	"Set the livelist condense zthr to pause");

	ZFS_MODULE_PARAM(zfs_livelist_condense, zfs_livelist_condense_, sync_pause, INT, ZMOD_RW,
	"Set the livelist condense synctask to pause");

	ZFS_MODULE_PARAM(zfs_livelist_condense, zfs_livelist_condense_, sync_cancel, INT, ZMOD_RW,
	"Whether livelist condensing was canceled in the synctask");

	ZFS_MODULE_PARAM(zfs_livelist_condense, zfs_livelist_condense_, zthr_cancel, INT, ZMOD_RW,
	"Whether livelist condensing was canceled in the zthr function");

	ZFS_MODULE_PARAM(zfs_livelist_condense, zfs_livelist_condense_, new_alloc, INT, ZMOD_RW,
	"Whether extra ALLOC blkptrs were added to a livelist entry while it "
	"was being condensed");
	/* END CSTYLED */
	Index: vendor-sys/openzfs/dist/module/zfs/vdev.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/zfs/vdev.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/zfs/vdev.c (revision 366774)
	@@ -1,5112 +1,5100 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/

	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2011, 2020 by Delphix. All rights reserved.
	* Copyright 2017 Nexenta Systems, Inc.
	* Copyright (c) 2014 Integros [integros.com]
	* Copyright 2016 Toomas Soome <tsoome@me.com>
	* Copyright 2017 Joyent, Inc.
	* Copyright (c) 2017, Intel Corporation.
	* Copyright (c) 2019, Datto Inc. All rights reserved.
	*/

	#include <sys/zfs_context.h>
	#include <sys/fm/fs/zfs.h>
	#include <sys/spa.h>
	#include <sys/spa_impl.h>
	#include <sys/bpobj.h>
	#include <sys/dmu.h>
	#include <sys/dmu_tx.h>
	#include <sys/dsl_dir.h>
	#include <sys/vdev_impl.h>
	#include <sys/vdev_rebuild.h>
	#include <sys/uberblock_impl.h>
	#include <sys/metaslab.h>
	#include <sys/metaslab_impl.h>
	#include <sys/space_map.h>
	#include <sys/space_reftree.h>
	#include <sys/zio.h>
	#include <sys/zap.h>
	#include <sys/fs/zfs.h>
	#include <sys/arc.h>
	#include <sys/zil.h>
	#include <sys/dsl_scan.h>
	#include <sys/abd.h>
	#include <sys/vdev_initialize.h>
	#include <sys/vdev_trim.h>
	#include <sys/zvol.h>
	#include <sys/zfs_ratelimit.h>

	/* default target for number of metaslabs per top-level vdev */
	int zfs_vdev_default_ms_count = 200;

	/* minimum number of metaslabs per top-level vdev */
	int zfs_vdev_min_ms_count = 16;

	/* practical upper limit of total metaslabs per top-level vdev */
	int zfs_vdev_ms_count_limit = 1ULL << 17;

	/* lower limit for metaslab size (512M) */
	int zfs_vdev_default_ms_shift = 29;

	/* upper limit for metaslab size (16G) */
	int zfs_vdev_max_ms_shift = 34;

	int vdev_validate_skip = B_FALSE;

	/*
	* Since the DTL space map of a vdev is not expected to have a lot of
	* entries, we default its block size to 4K.
	*/
	int zfs_vdev_dtl_sm_blksz = (1 << 12);

	/*
	* Rate limit slow IO (delay) events to this many per second.
	*/
	unsigned int zfs_slow_io_events_per_second = 20;

	/*
	* Rate limit checksum events after this many checksum errors per second.
	*/
	unsigned int zfs_checksum_events_per_second = 20;

	/*
	* Ignore errors during scrub/resilver. Allows to work around resilver
	* upon import when there are pool errors.
	*/
	int zfs_scan_ignore_errors = 0;

	/*
	* vdev-wide space maps that have lots of entries written to them at
	* the end of each transaction can benefit from a higher I/O bandwidth
	* (e.g. vdev_obsolete_sm), thus we default their block size to 128K.
	*/
	int zfs_vdev_standard_sm_blksz = (1 << 17);

	/*
	* Tunable parameter for debugging or performance analysis. Setting this
	* will cause pool corruption on power loss if a volatile out-of-order
	* write cache is enabled.
	*/
	int zfs_nocacheflush = 0;

	uint64_t zfs_vdev_max_auto_ashift = ASHIFT_MAX;
	uint64_t zfs_vdev_min_auto_ashift = ASHIFT_MIN;

	/PRINTFLIKE2/
	void
	vdev_dbgmsg(vdev_t vd, const char fmt, ...)
	{
	va_list adx;
	char buf[256];

	va_start(adx, fmt);
	(void) vsnprintf(buf, sizeof (buf), fmt, adx);
	va_end(adx);

	if (vd->vdev_path != NULL) {
	zfs_dbgmsg("%s vdev '%s': %s", vd->vdev_ops->vdev_op_type,
	vd->vdev_path, buf);
	} else {
	zfs_dbgmsg("%s-%llu vdev (guid %llu): %s",
	vd->vdev_ops->vdev_op_type,
	(u_longlong_t)vd->vdev_id,
	(u_longlong_t)vd->vdev_guid, buf);
	}
	}

	void
	vdev_dbgmsg_print_tree(vdev_t *vd, int indent)
	{
	char state[20];

	if (vd->vdev_ishole \|\| vd->vdev_ops == &vdev_missing_ops) {
	zfs_dbgmsg("%*svdev %u: %s", indent, "", vd->vdev_id,
	vd->vdev_ops->vdev_op_type);
	return;
	}

	switch (vd->vdev_state) {
	case VDEV_STATE_UNKNOWN:
	(void) snprintf(state, sizeof (state), "unknown");
	break;
	case VDEV_STATE_CLOSED:
	(void) snprintf(state, sizeof (state), "closed");
	break;
	case VDEV_STATE_OFFLINE:
	(void) snprintf(state, sizeof (state), "offline");
	break;
	case VDEV_STATE_REMOVED:
	(void) snprintf(state, sizeof (state), "removed");
	break;
	case VDEV_STATE_CANT_OPEN:
	(void) snprintf(state, sizeof (state), "can't open");
	break;
	case VDEV_STATE_FAULTED:
	(void) snprintf(state, sizeof (state), "faulted");
	break;
	case VDEV_STATE_DEGRADED:
	(void) snprintf(state, sizeof (state), "degraded");
	break;
	case VDEV_STATE_HEALTHY:
	(void) snprintf(state, sizeof (state), "healthy");
	break;
	default:
	(void) snprintf(state, sizeof (state), "<state %u>",
	(uint_t)vd->vdev_state);
	}

	zfs_dbgmsg("%*svdev %u: %s%s, guid: %llu, path: %s, %s", indent,
	"", (int)vd->vdev_id, vd->vdev_ops->vdev_op_type,
	vd->vdev_islog ? " (log)" : "",
	(u_longlong_t)vd->vdev_guid,
	vd->vdev_path ? vd->vdev_path : "N/A", state);

	for (uint64_t i = 0; i < vd->vdev_children; i++)
	vdev_dbgmsg_print_tree(vd->vdev_child[i], indent + 2);
	}

	/*
	* Virtual device management.
	*/

	static vdev_ops_t *vdev_ops_table[] = {
	&vdev_root_ops,
	&vdev_raidz_ops,
	&vdev_mirror_ops,
	&vdev_replacing_ops,
	&vdev_spare_ops,
	&vdev_disk_ops,
	&vdev_file_ops,
	&vdev_missing_ops,
	&vdev_hole_ops,
	&vdev_indirect_ops,
	NULL
	};

	/*
	* Given a vdev type, return the appropriate ops vector.
	*/
	static vdev_ops_t *
	vdev_getops(const char *type)
	{
	vdev_ops_t ops, *opspp;

	for (opspp = vdev_ops_table; (ops = *opspp) != NULL; opspp++)
	if (strcmp(ops->vdev_op_type, type) == 0)
	break;

	return (ops);
	}

	/* ARGSUSED */
	void
	vdev_default_xlate(vdev_t vd, const range_seg64_t in, range_seg64_t *res)
	{
	res->rs_start = in->rs_start;
	res->rs_end = in->rs_end;
	}

	/*
	* Derive the enumerated allocation bias from string input.
	* String origin is either the per-vdev zap or zpool(1M).
	*/
	static vdev_alloc_bias_t
	vdev_derive_alloc_bias(const char *bias)
	{
	vdev_alloc_bias_t alloc_bias = VDEV_BIAS_NONE;

	if (strcmp(bias, VDEV_ALLOC_BIAS_LOG) == 0)
	alloc_bias = VDEV_BIAS_LOG;
	else if (strcmp(bias, VDEV_ALLOC_BIAS_SPECIAL) == 0)
	alloc_bias = VDEV_BIAS_SPECIAL;
	else if (strcmp(bias, VDEV_ALLOC_BIAS_DEDUP) == 0)
	alloc_bias = VDEV_BIAS_DEDUP;

	return (alloc_bias);
	}

	/*
	* Default asize function: return the MAX of psize with the asize of
	* all children. This is what's used by anything other than RAID-Z.
	*/
	uint64_t
	vdev_default_asize(vdev_t *vd, uint64_t psize)
	{
	uint64_t asize = P2ROUNDUP(psize, 1ULL << vd->vdev_top->vdev_ashift);
	uint64_t csize;

	for (int c = 0; c < vd->vdev_children; c++) {
	csize = vdev_psize_to_asize(vd->vdev_child[c], psize);
	asize = MAX(asize, csize);
	}

	return (asize);
	}

	/*
	* Get the minimum allocatable size. We define the allocatable size as
	* the vdev's asize rounded to the nearest metaslab. This allows us to
	* replace or attach devices which don't have the same physical size but
	* can still satisfy the same number of allocations.
	*/
	uint64_t
	vdev_get_min_asize(vdev_t *vd)
	{
	vdev_t *pvd = vd->vdev_parent;

	/*
	* If our parent is NULL (inactive spare or cache) or is the root,
	* just return our own asize.
	*/
	if (pvd == NULL)
	return (vd->vdev_asize);

	/*
	* The top-level vdev just returns the allocatable size rounded
	* to the nearest metaslab.
	*/
	if (vd == vd->vdev_top)
	return (P2ALIGN(vd->vdev_asize, 1ULL << vd->vdev_ms_shift));

	/*
	* The allocatable space for a raidz vdev is N * sizeof(smallest child),
	* so each child must provide at least 1/Nth of its asize.
	*/
	if (pvd->vdev_ops == &vdev_raidz_ops)
	return ((pvd->vdev_min_asize + pvd->vdev_children - 1) /
	pvd->vdev_children);

	return (pvd->vdev_min_asize);
	}

	void
	vdev_set_min_asize(vdev_t *vd)
	{
	vd->vdev_min_asize = vdev_get_min_asize(vd);

	for (int c = 0; c < vd->vdev_children; c++)
	vdev_set_min_asize(vd->vdev_child[c]);
	}

	vdev_t *
	vdev_lookup_top(spa_t *spa, uint64_t vdev)
	{
	vdev_t *rvd = spa->spa_root_vdev;

	ASSERT(spa_config_held(spa, SCL_ALL, RW_READER) != 0);

	if (vdev < rvd->vdev_children) {
	ASSERT(rvd->vdev_child[vdev] != NULL);
	return (rvd->vdev_child[vdev]);
	}

	return (NULL);
	}

	vdev_t *
	vdev_lookup_by_guid(vdev_t *vd, uint64_t guid)
	{
	vdev_t *mvd;

	if (vd->vdev_guid == guid)
	return (vd);

	for (int c = 0; c < vd->vdev_children; c++)
	if ((mvd = vdev_lookup_by_guid(vd->vdev_child[c], guid)) !=
	NULL)
	return (mvd);

	return (NULL);
	}

	static int
	vdev_count_leaves_impl(vdev_t *vd)
	{
	int n = 0;

	if (vd->vdev_ops->vdev_op_leaf)
	return (1);

	for (int c = 0; c < vd->vdev_children; c++)
	n += vdev_count_leaves_impl(vd->vdev_child[c]);

	return (n);
	}

	int
	vdev_count_leaves(spa_t *spa)
	{
	int rc;

	spa_config_enter(spa, SCL_VDEV, FTAG, RW_READER);
	rc = vdev_count_leaves_impl(spa->spa_root_vdev);
	spa_config_exit(spa, SCL_VDEV, FTAG);

	return (rc);
	}

	void
	vdev_add_child(vdev_t pvd, vdev_t cvd)
	{
	size_t oldsize, newsize;
	uint64_t id = cvd->vdev_id;
	vdev_t **newchild;

	ASSERT(spa_config_held(cvd->vdev_spa, SCL_ALL, RW_WRITER) == SCL_ALL);
	ASSERT(cvd->vdev_parent == NULL);

	cvd->vdev_parent = pvd;

	if (pvd == NULL)
	return;

	ASSERT(id >= pvd->vdev_children \|\| pvd->vdev_child[id] == NULL);

	oldsize = pvd->vdev_children * sizeof (vdev_t *);
	pvd->vdev_children = MAX(pvd->vdev_children, id + 1);
	newsize = pvd->vdev_children * sizeof (vdev_t *);

	newchild = kmem_alloc(newsize, KM_SLEEP);
	if (pvd->vdev_child != NULL) {
	bcopy(pvd->vdev_child, newchild, oldsize);
	kmem_free(pvd->vdev_child, oldsize);
	}

	pvd->vdev_child = newchild;
	pvd->vdev_child[id] = cvd;

	cvd->vdev_top = (pvd->vdev_top ? pvd->vdev_top: cvd);
	ASSERT(cvd->vdev_top->vdev_parent->vdev_parent == NULL);

	/*
	* Walk up all ancestors to update guid sum.
	*/
	for (; pvd != NULL; pvd = pvd->vdev_parent)
	pvd->vdev_guid_sum += cvd->vdev_guid_sum;

	if (cvd->vdev_ops->vdev_op_leaf) {
	list_insert_head(&cvd->vdev_spa->spa_leaf_list, cvd);
	cvd->vdev_spa->spa_leaf_list_gen++;
	}
	}

	void
	vdev_remove_child(vdev_t pvd, vdev_t cvd)
	{
	int c;
	uint_t id = cvd->vdev_id;

	ASSERT(cvd->vdev_parent == pvd);

	if (pvd == NULL)
	return;

	ASSERT(id < pvd->vdev_children);
	ASSERT(pvd->vdev_child[id] == cvd);

	pvd->vdev_child[id] = NULL;
	cvd->vdev_parent = NULL;

	for (c = 0; c < pvd->vdev_children; c++)
	if (pvd->vdev_child[c])
	break;

	if (c == pvd->vdev_children) {
	kmem_free(pvd->vdev_child, c * sizeof (vdev_t *));
	pvd->vdev_child = NULL;
	pvd->vdev_children = 0;
	}

	if (cvd->vdev_ops->vdev_op_leaf) {
	spa_t *spa = cvd->vdev_spa;
	list_remove(&spa->spa_leaf_list, cvd);
	spa->spa_leaf_list_gen++;
	}

	/*
	* Walk up all ancestors to update guid sum.
	*/
	for (; pvd != NULL; pvd = pvd->vdev_parent)
	pvd->vdev_guid_sum -= cvd->vdev_guid_sum;
	}

	/*
	* Remove any holes in the child array.
	*/
	void
	vdev_compact_children(vdev_t *pvd)
	{
	vdev_t *newchild, cvd;
	int oldc = pvd->vdev_children;
	int newc;

	ASSERT(spa_config_held(pvd->vdev_spa, SCL_ALL, RW_WRITER) == SCL_ALL);

	if (oldc == 0)
	return;

	for (int c = newc = 0; c < oldc; c++)
	if (pvd->vdev_child[c])
	newc++;

	if (newc > 0) {
	newchild = kmem_zalloc(newc * sizeof (vdev_t *), KM_SLEEP);

	for (int c = newc = 0; c < oldc; c++) {
	if ((cvd = pvd->vdev_child[c]) != NULL) {
	newchild[newc] = cvd;
	cvd->vdev_id = newc++;
	}
	}
	} else {
	newchild = NULL;
	}

	kmem_free(pvd->vdev_child, oldc * sizeof (vdev_t *));
	pvd->vdev_child = newchild;
	pvd->vdev_children = newc;
	}

	/*
	* Allocate and minimally initialize a vdev_t.
	*/
	vdev_t *
	vdev_alloc_common(spa_t spa, uint_t id, uint64_t guid, vdev_ops_t ops)
	{
	vdev_t *vd;
	vdev_indirect_config_t *vic;

	vd = kmem_zalloc(sizeof (vdev_t), KM_SLEEP);
	vic = &vd->vdev_indirect_config;

	if (spa->spa_root_vdev == NULL) {
	ASSERT(ops == &vdev_root_ops);
	spa->spa_root_vdev = vd;
	spa->spa_load_guid = spa_generate_guid(NULL);
	}

	if (guid == 0 && ops != &vdev_hole_ops) {
	if (spa->spa_root_vdev == vd) {
	/*
	* The root vdev's guid will also be the pool guid,
	* which must be unique among all pools.
	*/
	guid = spa_generate_guid(NULL);
	} else {
	/*
	* Any other vdev's guid must be unique within the pool.
	*/
	guid = spa_generate_guid(spa);
	}
	ASSERT(!spa_guid_exists(spa_guid(spa), guid));
	}

	vd->vdev_spa = spa;
	vd->vdev_id = id;
	vd->vdev_guid = guid;
	vd->vdev_guid_sum = guid;
	vd->vdev_ops = ops;
	vd->vdev_state = VDEV_STATE_CLOSED;
	vd->vdev_ishole = (ops == &vdev_hole_ops);
	vic->vic_prev_indirect_vdev = UINT64_MAX;

	rw_init(&vd->vdev_indirect_rwlock, NULL, RW_DEFAULT, NULL);
	mutex_init(&vd->vdev_obsolete_lock, NULL, MUTEX_DEFAULT, NULL);
	vd->vdev_obsolete_segments = range_tree_create(NULL, RANGE_SEG64, NULL,
	0, 0);

	/*
	* Initialize rate limit structs for events. We rate limit ZIO delay
	* and checksum events so that we don't overwhelm ZED with thousands
	* of events when a disk is acting up.
	*/
	zfs_ratelimit_init(&vd->vdev_delay_rl, &zfs_slow_io_events_per_second,
	1);
	zfs_ratelimit_init(&vd->vdev_checksum_rl,
	&zfs_checksum_events_per_second, 1);

	list_link_init(&vd->vdev_config_dirty_node);
	list_link_init(&vd->vdev_state_dirty_node);
	list_link_init(&vd->vdev_initialize_node);
	list_link_init(&vd->vdev_leaf_node);
	list_link_init(&vd->vdev_trim_node);
	mutex_init(&vd->vdev_dtl_lock, NULL, MUTEX_NOLOCKDEP, NULL);
	mutex_init(&vd->vdev_stat_lock, NULL, MUTEX_DEFAULT, NULL);
	mutex_init(&vd->vdev_probe_lock, NULL, MUTEX_DEFAULT, NULL);
	mutex_init(&vd->vdev_scan_io_queue_lock, NULL, MUTEX_DEFAULT, NULL);

	mutex_init(&vd->vdev_initialize_lock, NULL, MUTEX_DEFAULT, NULL);
	mutex_init(&vd->vdev_initialize_io_lock, NULL, MUTEX_DEFAULT, NULL);
	cv_init(&vd->vdev_initialize_cv, NULL, CV_DEFAULT, NULL);
	cv_init(&vd->vdev_initialize_io_cv, NULL, CV_DEFAULT, NULL);

	mutex_init(&vd->vdev_trim_lock, NULL, MUTEX_DEFAULT, NULL);
	mutex_init(&vd->vdev_autotrim_lock, NULL, MUTEX_DEFAULT, NULL);
	mutex_init(&vd->vdev_trim_io_lock, NULL, MUTEX_DEFAULT, NULL);
	cv_init(&vd->vdev_trim_cv, NULL, CV_DEFAULT, NULL);
	cv_init(&vd->vdev_autotrim_cv, NULL, CV_DEFAULT, NULL);
	cv_init(&vd->vdev_trim_io_cv, NULL, CV_DEFAULT, NULL);

	mutex_init(&vd->vdev_rebuild_lock, NULL, MUTEX_DEFAULT, NULL);
	mutex_init(&vd->vdev_rebuild_io_lock, NULL, MUTEX_DEFAULT, NULL);
	cv_init(&vd->vdev_rebuild_cv, NULL, CV_DEFAULT, NULL);
	cv_init(&vd->vdev_rebuild_io_cv, NULL, CV_DEFAULT, NULL);

	for (int t = 0; t < DTL_TYPES; t++) {
	vd->vdev_dtl[t] = range_tree_create(NULL, RANGE_SEG64, NULL, 0,
	0);
	}

	txg_list_create(&vd->vdev_ms_list, spa,
	offsetof(struct metaslab, ms_txg_node));
	txg_list_create(&vd->vdev_dtl_list, spa,
	offsetof(struct vdev, vdev_dtl_node));
	vd->vdev_stat.vs_timestamp = gethrtime();
	vdev_queue_init(vd);
	vdev_cache_init(vd);

	return (vd);
	}

	/*
	* Allocate a new vdev. The 'alloctype' is used to control whether we are
	* creating a new vdev or loading an existing one - the behavior is slightly
	* different for each case.
	*/
	int
	vdev_alloc(spa_t spa, vdev_t vdp, nvlist_t nv, vdev_t *parent, uint_t id,
	int alloctype)
	{
	vdev_ops_t *ops;
	char *type;
	uint64_t guid = 0, islog, nparity;
	vdev_t *vd;
	vdev_indirect_config_t *vic;
	char *tmp = NULL;
	int rc;
	vdev_alloc_bias_t alloc_bias = VDEV_BIAS_NONE;
	boolean_t top_level = (parent && !parent->vdev_parent);

	ASSERT(spa_config_held(spa, SCL_ALL, RW_WRITER) == SCL_ALL);

	if (nvlist_lookup_string(nv, ZPOOL_CONFIG_TYPE, &type) != 0)
	return (SET_ERROR(EINVAL));

	if ((ops = vdev_getops(type)) == NULL)
	return (SET_ERROR(EINVAL));

	/*
	* If this is a load, get the vdev guid from the nvlist.
	* Otherwise, vdev_alloc_common() will generate one for us.
	*/
	if (alloctype == VDEV_ALLOC_LOAD) {
	uint64_t label_id;

	if (nvlist_lookup_uint64(nv, ZPOOL_CONFIG_ID, &label_id) \|\|
	label_id != id)
	return (SET_ERROR(EINVAL));

	if (nvlist_lookup_uint64(nv, ZPOOL_CONFIG_GUID, &guid) != 0)
	return (SET_ERROR(EINVAL));
	} else if (alloctype == VDEV_ALLOC_SPARE) {
	if (nvlist_lookup_uint64(nv, ZPOOL_CONFIG_GUID, &guid) != 0)
	return (SET_ERROR(EINVAL));
	} else if (alloctype == VDEV_ALLOC_L2CACHE) {
	if (nvlist_lookup_uint64(nv, ZPOOL_CONFIG_GUID, &guid) != 0)
	return (SET_ERROR(EINVAL));
	} else if (alloctype == VDEV_ALLOC_ROOTPOOL) {
	if (nvlist_lookup_uint64(nv, ZPOOL_CONFIG_GUID, &guid) != 0)
	return (SET_ERROR(EINVAL));
	}

	/*
	* The first allocated vdev must be of type 'root'.
	*/
	if (ops != &vdev_root_ops && spa->spa_root_vdev == NULL)
	return (SET_ERROR(EINVAL));

	/*
	* Determine whether we're a log vdev.
	*/
	islog = 0;
	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_IS_LOG, &islog);
	if (islog && spa_version(spa) < SPA_VERSION_SLOGS)
	return (SET_ERROR(ENOTSUP));

	if (ops == &vdev_hole_ops && spa_version(spa) < SPA_VERSION_HOLES)
	return (SET_ERROR(ENOTSUP));

	/*
	* Set the nparity property for RAID-Z vdevs.
	*/
	nparity = -1ULL;
	if (ops == &vdev_raidz_ops) {
	if (nvlist_lookup_uint64(nv, ZPOOL_CONFIG_NPARITY,
	&nparity) == 0) {
	if (nparity == 0 \|\| nparity > VDEV_RAIDZ_MAXPARITY)
	return (SET_ERROR(EINVAL));
	/*
	* Previous versions could only support 1 or 2 parity
	* device.
	*/
	if (nparity > 1 &&
	spa_version(spa) < SPA_VERSION_RAIDZ2)
	return (SET_ERROR(ENOTSUP));
	if (nparity > 2 &&
	spa_version(spa) < SPA_VERSION_RAIDZ3)
	return (SET_ERROR(ENOTSUP));
	} else {
	/*
	* We require the parity to be specified for SPAs that
	* support multiple parity levels.
	*/
	if (spa_version(spa) >= SPA_VERSION_RAIDZ2)
	return (SET_ERROR(EINVAL));
	/*
	* Otherwise, we default to 1 parity device for RAID-Z.
	*/
	nparity = 1;
	}
	} else {
	nparity = 0;
	}
	ASSERT(nparity != -1ULL);

	/*
	* If creating a top-level vdev, check for allocation classes input
	*/
	if (top_level && alloctype == VDEV_ALLOC_ADD) {
	char *bias;

	if (nvlist_lookup_string(nv, ZPOOL_CONFIG_ALLOCATION_BIAS,
	&bias) == 0) {
	alloc_bias = vdev_derive_alloc_bias(bias);

	/* spa_vdev_add() expects feature to be enabled */
	if (spa->spa_load_state != SPA_LOAD_CREATE &&
	!spa_feature_is_enabled(spa,
	SPA_FEATURE_ALLOCATION_CLASSES)) {
	return (SET_ERROR(ENOTSUP));
	}
	}
	}

	vd = vdev_alloc_common(spa, id, guid, ops);
	vic = &vd->vdev_indirect_config;

	vd->vdev_islog = islog;
	vd->vdev_nparity = nparity;
	if (top_level && alloc_bias != VDEV_BIAS_NONE)
	vd->vdev_alloc_bias = alloc_bias;

	if (nvlist_lookup_string(nv, ZPOOL_CONFIG_PATH, &vd->vdev_path) == 0)
	vd->vdev_path = spa_strdup(vd->vdev_path);

	/*
	* ZPOOL_CONFIG_AUX_STATE = "external" means we previously forced a
	* fault on a vdev and want it to persist across imports (like with
	* zpool offline -f).
	*/
	rc = nvlist_lookup_string(nv, ZPOOL_CONFIG_AUX_STATE, &tmp);
	if (rc == 0 && tmp != NULL && strcmp(tmp, "external") == 0) {
	vd->vdev_stat.vs_aux = VDEV_AUX_EXTERNAL;
	vd->vdev_faulted = 1;
	vd->vdev_label_aux = VDEV_AUX_EXTERNAL;
	}

	if (nvlist_lookup_string(nv, ZPOOL_CONFIG_DEVID, &vd->vdev_devid) == 0)
	vd->vdev_devid = spa_strdup(vd->vdev_devid);
	if (nvlist_lookup_string(nv, ZPOOL_CONFIG_PHYS_PATH,
	&vd->vdev_physpath) == 0)
	vd->vdev_physpath = spa_strdup(vd->vdev_physpath);

	if (nvlist_lookup_string(nv, ZPOOL_CONFIG_VDEV_ENC_SYSFS_PATH,
	&vd->vdev_enc_sysfs_path) == 0)
	vd->vdev_enc_sysfs_path = spa_strdup(vd->vdev_enc_sysfs_path);

	if (nvlist_lookup_string(nv, ZPOOL_CONFIG_FRU, &vd->vdev_fru) == 0)
	vd->vdev_fru = spa_strdup(vd->vdev_fru);

	/*
	* Set the whole_disk property. If it's not specified, leave the value
	* as -1.
	*/
	if (nvlist_lookup_uint64(nv, ZPOOL_CONFIG_WHOLE_DISK,
	&vd->vdev_wholedisk) != 0)
	vd->vdev_wholedisk = -1ULL;

	ASSERT0(vic->vic_mapping_object);
	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_INDIRECT_OBJECT,
	&vic->vic_mapping_object);
	ASSERT0(vic->vic_births_object);
	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_INDIRECT_BIRTHS,
	&vic->vic_births_object);
	ASSERT3U(vic->vic_prev_indirect_vdev, ==, UINT64_MAX);
	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_PREV_INDIRECT_VDEV,
	&vic->vic_prev_indirect_vdev);

	/*
	* Look for the 'not present' flag. This will only be set if the device
	* was not present at the time of import.
	*/
	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_NOT_PRESENT,
	&vd->vdev_not_present);

	/*
	* Get the alignment requirement.
	*/
	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_ASHIFT, &vd->vdev_ashift);

	/*
	* Retrieve the vdev creation time.
	*/
	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_CREATE_TXG,
	&vd->vdev_crtxg);

	/*
	* If we're a top-level vdev, try to load the allocation parameters.
	*/
	if (top_level &&
	(alloctype == VDEV_ALLOC_LOAD \|\| alloctype == VDEV_ALLOC_SPLIT)) {
	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_METASLAB_ARRAY,
	&vd->vdev_ms_array);
	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_METASLAB_SHIFT,
	&vd->vdev_ms_shift);
	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_ASIZE,
	&vd->vdev_asize);
	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_REMOVING,
	&vd->vdev_removing);
	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_VDEV_TOP_ZAP,
	&vd->vdev_top_zap);
	} else {
	ASSERT0(vd->vdev_top_zap);
	}

	if (top_level && alloctype != VDEV_ALLOC_ATTACH) {
	ASSERT(alloctype == VDEV_ALLOC_LOAD \|\|
	alloctype == VDEV_ALLOC_ADD \|\|
	alloctype == VDEV_ALLOC_SPLIT \|\|
	alloctype == VDEV_ALLOC_ROOTPOOL);
	/* Note: metaslab_group_create() is now deferred */
	}

	if (vd->vdev_ops->vdev_op_leaf &&
	(alloctype == VDEV_ALLOC_LOAD \|\| alloctype == VDEV_ALLOC_SPLIT)) {
	(void) nvlist_lookup_uint64(nv,
	ZPOOL_CONFIG_VDEV_LEAF_ZAP, &vd->vdev_leaf_zap);
	} else {
	ASSERT0(vd->vdev_leaf_zap);
	}

	/*
	* If we're a leaf vdev, try to load the DTL object and other state.
	*/

	if (vd->vdev_ops->vdev_op_leaf &&
	(alloctype == VDEV_ALLOC_LOAD \|\| alloctype == VDEV_ALLOC_L2CACHE \|\|
	alloctype == VDEV_ALLOC_ROOTPOOL)) {
	if (alloctype == VDEV_ALLOC_LOAD) {
	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_DTL,
	&vd->vdev_dtl_object);
	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_UNSPARE,
	&vd->vdev_unspare);
	}

	if (alloctype == VDEV_ALLOC_ROOTPOOL) {
	uint64_t spare = 0;

	if (nvlist_lookup_uint64(nv, ZPOOL_CONFIG_IS_SPARE,
	&spare) == 0 && spare)
	spa_spare_add(vd);
	}

	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_OFFLINE,
	&vd->vdev_offline);

	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_RESILVER_TXG,
	&vd->vdev_resilver_txg);

	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_REBUILD_TXG,
	&vd->vdev_rebuild_txg);

	if (nvlist_exists(nv, ZPOOL_CONFIG_RESILVER_DEFER))
	vdev_defer_resilver(vd);

	/*
	* In general, when importing a pool we want to ignore the
	* persistent fault state, as the diagnosis made on another
	* system may not be valid in the current context. The only
	* exception is if we forced a vdev to a persistently faulted
	* state with 'zpool offline -f'. The persistent fault will
	* remain across imports until cleared.
	*
	* Local vdevs will remain in the faulted state.
	*/
	if (spa_load_state(spa) == SPA_LOAD_OPEN \|\|
	spa_load_state(spa) == SPA_LOAD_IMPORT) {
	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_FAULTED,
	&vd->vdev_faulted);
	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_DEGRADED,
	&vd->vdev_degraded);
	(void) nvlist_lookup_uint64(nv, ZPOOL_CONFIG_REMOVED,
	&vd->vdev_removed);

	if (vd->vdev_faulted \|\| vd->vdev_degraded) {
	char *aux;

	vd->vdev_label_aux =
	VDEV_AUX_ERR_EXCEEDED;
	if (nvlist_lookup_string(nv,
	ZPOOL_CONFIG_AUX_STATE, &aux) == 0 &&
	strcmp(aux, "external") == 0)
	vd->vdev_label_aux = VDEV_AUX_EXTERNAL;
	else
	vd->vdev_faulted = 0ULL;
	}
	}
	}

	/*
	* Add ourselves to the parent's list of children.
	*/
	vdev_add_child(parent, vd);

	*vdp = vd;

	return (0);
	}

	void
	vdev_free(vdev_t *vd)
	{
	spa_t *spa = vd->vdev_spa;

	ASSERT3P(vd->vdev_initialize_thread, ==, NULL);
	ASSERT3P(vd->vdev_trim_thread, ==, NULL);
	ASSERT3P(vd->vdev_autotrim_thread, ==, NULL);
	ASSERT3P(vd->vdev_rebuild_thread, ==, NULL);

	/*
	* Scan queues are normally destroyed at the end of a scan. If the
	* queue exists here, that implies the vdev is being removed while
	* the scan is still running.
	*/
	if (vd->vdev_scan_io_queue != NULL) {
	mutex_enter(&vd->vdev_scan_io_queue_lock);
	dsl_scan_io_queue_destroy(vd->vdev_scan_io_queue);
	vd->vdev_scan_io_queue = NULL;
	mutex_exit(&vd->vdev_scan_io_queue_lock);
	}

	/*
	* vdev_free() implies closing the vdev first. This is simpler than
	* trying to ensure complicated semantics for all callers.
	*/
	vdev_close(vd);

	ASSERT(!list_link_active(&vd->vdev_config_dirty_node));
	ASSERT(!list_link_active(&vd->vdev_state_dirty_node));

	/*
	* Free all children.
	*/
	for (int c = 0; c < vd->vdev_children; c++)
	vdev_free(vd->vdev_child[c]);

	ASSERT(vd->vdev_child == NULL);
	ASSERT(vd->vdev_guid_sum == vd->vdev_guid);

	/*
	* Discard allocation state.
	*/
	if (vd->vdev_mg != NULL) {
	vdev_metaslab_fini(vd);
	metaslab_group_destroy(vd->vdev_mg);
	vd->vdev_mg = NULL;
	}

	ASSERT0(vd->vdev_stat.vs_space);
	ASSERT0(vd->vdev_stat.vs_dspace);
	ASSERT0(vd->vdev_stat.vs_alloc);

	/*
	* Remove this vdev from its parent's child list.
	*/
	vdev_remove_child(vd->vdev_parent, vd);

	ASSERT(vd->vdev_parent == NULL);
	ASSERT(!list_link_active(&vd->vdev_leaf_node));

	/*
	* Clean up vdev structure.
	*/
	vdev_queue_fini(vd);
	vdev_cache_fini(vd);

	if (vd->vdev_path)
	spa_strfree(vd->vdev_path);
	if (vd->vdev_devid)
	spa_strfree(vd->vdev_devid);
	if (vd->vdev_physpath)
	spa_strfree(vd->vdev_physpath);

	if (vd->vdev_enc_sysfs_path)
	spa_strfree(vd->vdev_enc_sysfs_path);

	if (vd->vdev_fru)
	spa_strfree(vd->vdev_fru);

	if (vd->vdev_isspare)
	spa_spare_remove(vd);
	if (vd->vdev_isl2cache)
	spa_l2cache_remove(vd);

	txg_list_destroy(&vd->vdev_ms_list);
	txg_list_destroy(&vd->vdev_dtl_list);

	mutex_enter(&vd->vdev_dtl_lock);
	space_map_close(vd->vdev_dtl_sm);
	for (int t = 0; t < DTL_TYPES; t++) {
	range_tree_vacate(vd->vdev_dtl[t], NULL, NULL);
	range_tree_destroy(vd->vdev_dtl[t]);
	}
	mutex_exit(&vd->vdev_dtl_lock);

	EQUIV(vd->vdev_indirect_births != NULL,
	vd->vdev_indirect_mapping != NULL);
	if (vd->vdev_indirect_births != NULL) {
	vdev_indirect_mapping_close(vd->vdev_indirect_mapping);
	vdev_indirect_births_close(vd->vdev_indirect_births);
	}

	if (vd->vdev_obsolete_sm != NULL) {
	ASSERT(vd->vdev_removing \|\|
	vd->vdev_ops == &vdev_indirect_ops);
	space_map_close(vd->vdev_obsolete_sm);
	vd->vdev_obsolete_sm = NULL;
	}
	range_tree_destroy(vd->vdev_obsolete_segments);
	rw_destroy(&vd->vdev_indirect_rwlock);
	mutex_destroy(&vd->vdev_obsolete_lock);

	mutex_destroy(&vd->vdev_dtl_lock);
	mutex_destroy(&vd->vdev_stat_lock);
	mutex_destroy(&vd->vdev_probe_lock);
	mutex_destroy(&vd->vdev_scan_io_queue_lock);

	mutex_destroy(&vd->vdev_initialize_lock);
	mutex_destroy(&vd->vdev_initialize_io_lock);
	cv_destroy(&vd->vdev_initialize_io_cv);
	cv_destroy(&vd->vdev_initialize_cv);

	mutex_destroy(&vd->vdev_trim_lock);
	mutex_destroy(&vd->vdev_autotrim_lock);
	mutex_destroy(&vd->vdev_trim_io_lock);
	cv_destroy(&vd->vdev_trim_cv);
	cv_destroy(&vd->vdev_autotrim_cv);
	cv_destroy(&vd->vdev_trim_io_cv);

	mutex_destroy(&vd->vdev_rebuild_lock);
	mutex_destroy(&vd->vdev_rebuild_io_lock);
	cv_destroy(&vd->vdev_rebuild_cv);
	cv_destroy(&vd->vdev_rebuild_io_cv);

	zfs_ratelimit_fini(&vd->vdev_delay_rl);
	zfs_ratelimit_fini(&vd->vdev_checksum_rl);

	if (vd == spa->spa_root_vdev)
	spa->spa_root_vdev = NULL;

	kmem_free(vd, sizeof (vdev_t));
	}

	/*
	* Transfer top-level vdev state from svd to tvd.
	*/
	static void
	vdev_top_transfer(vdev_t svd, vdev_t tvd)
	{
	spa_t *spa = svd->vdev_spa;
	metaslab_t *msp;
	vdev_t *vd;
	int t;

	ASSERT(tvd == tvd->vdev_top);

	tvd->vdev_pending_fastwrite = svd->vdev_pending_fastwrite;
	tvd->vdev_ms_array = svd->vdev_ms_array;
	tvd->vdev_ms_shift = svd->vdev_ms_shift;
	tvd->vdev_ms_count = svd->vdev_ms_count;
	tvd->vdev_top_zap = svd->vdev_top_zap;

	svd->vdev_ms_array = 0;
	svd->vdev_ms_shift = 0;
	svd->vdev_ms_count = 0;
	svd->vdev_top_zap = 0;

	if (tvd->vdev_mg)
	ASSERT3P(tvd->vdev_mg, ==, svd->vdev_mg);
	tvd->vdev_mg = svd->vdev_mg;
	tvd->vdev_ms = svd->vdev_ms;

	svd->vdev_mg = NULL;
	svd->vdev_ms = NULL;

	if (tvd->vdev_mg != NULL)
	tvd->vdev_mg->mg_vd = tvd;

	tvd->vdev_checkpoint_sm = svd->vdev_checkpoint_sm;
	svd->vdev_checkpoint_sm = NULL;

	tvd->vdev_alloc_bias = svd->vdev_alloc_bias;
	svd->vdev_alloc_bias = VDEV_BIAS_NONE;

	tvd->vdev_stat.vs_alloc = svd->vdev_stat.vs_alloc;
	tvd->vdev_stat.vs_space = svd->vdev_stat.vs_space;
	tvd->vdev_stat.vs_dspace = svd->vdev_stat.vs_dspace;

	svd->vdev_stat.vs_alloc = 0;
	svd->vdev_stat.vs_space = 0;
	svd->vdev_stat.vs_dspace = 0;

	/*
	* State which may be set on a top-level vdev that's in the
	* process of being removed.
	*/
	ASSERT0(tvd->vdev_indirect_config.vic_births_object);
	ASSERT0(tvd->vdev_indirect_config.vic_mapping_object);
	ASSERT3U(tvd->vdev_indirect_config.vic_prev_indirect_vdev, ==, -1ULL);
	ASSERT3P(tvd->vdev_indirect_mapping, ==, NULL);
	ASSERT3P(tvd->vdev_indirect_births, ==, NULL);
	ASSERT3P(tvd->vdev_obsolete_sm, ==, NULL);
	ASSERT0(tvd->vdev_removing);
	ASSERT0(tvd->vdev_rebuilding);
	tvd->vdev_removing = svd->vdev_removing;
	tvd->vdev_rebuilding = svd->vdev_rebuilding;
	tvd->vdev_rebuild_config = svd->vdev_rebuild_config;
	tvd->vdev_indirect_config = svd->vdev_indirect_config;
	tvd->vdev_indirect_mapping = svd->vdev_indirect_mapping;
	tvd->vdev_indirect_births = svd->vdev_indirect_births;
	range_tree_swap(&svd->vdev_obsolete_segments,
	&tvd->vdev_obsolete_segments);
	tvd->vdev_obsolete_sm = svd->vdev_obsolete_sm;
	svd->vdev_indirect_config.vic_mapping_object = 0;
	svd->vdev_indirect_config.vic_births_object = 0;
	svd->vdev_indirect_config.vic_prev_indirect_vdev = -1ULL;
	svd->vdev_indirect_mapping = NULL;
	svd->vdev_indirect_births = NULL;
	svd->vdev_obsolete_sm = NULL;
	svd->vdev_removing = 0;
	svd->vdev_rebuilding = 0;

	for (t = 0; t < TXG_SIZE; t++) {
	while ((msp = txg_list_remove(&svd->vdev_ms_list, t)) != NULL)
	(void) txg_list_add(&tvd->vdev_ms_list, msp, t);
	while ((vd = txg_list_remove(&svd->vdev_dtl_list, t)) != NULL)
	(void) txg_list_add(&tvd->vdev_dtl_list, vd, t);
	if (txg_list_remove_this(&spa->spa_vdev_txg_list, svd, t))
	(void) txg_list_add(&spa->spa_vdev_txg_list, tvd, t);
	}

	if (list_link_active(&svd->vdev_config_dirty_node)) {
	vdev_config_clean(svd);
	vdev_config_dirty(tvd);
	}

	if (list_link_active(&svd->vdev_state_dirty_node)) {
	vdev_state_clean(svd);
	vdev_state_dirty(tvd);
	}

	tvd->vdev_deflate_ratio = svd->vdev_deflate_ratio;
	svd->vdev_deflate_ratio = 0;

	tvd->vdev_islog = svd->vdev_islog;
	svd->vdev_islog = 0;

	dsl_scan_io_queue_vdev_xfer(svd, tvd);
	}

	static void
	vdev_top_update(vdev_t tvd, vdev_t vd)
	{
	if (vd == NULL)
	return;

	vd->vdev_top = tvd;

	for (int c = 0; c < vd->vdev_children; c++)
	vdev_top_update(tvd, vd->vdev_child[c]);
	}

	/*
	* Add a mirror/replacing vdev above an existing vdev.
	*/
	vdev_t *
	vdev_add_parent(vdev_t cvd, vdev_ops_t ops)
	{
	spa_t *spa = cvd->vdev_spa;
	vdev_t *pvd = cvd->vdev_parent;
	vdev_t *mvd;

	ASSERT(spa_config_held(spa, SCL_ALL, RW_WRITER) == SCL_ALL);

	mvd = vdev_alloc_common(spa, cvd->vdev_id, 0, ops);

	mvd->vdev_asize = cvd->vdev_asize;
	mvd->vdev_min_asize = cvd->vdev_min_asize;
	mvd->vdev_max_asize = cvd->vdev_max_asize;
	mvd->vdev_psize = cvd->vdev_psize;
	mvd->vdev_ashift = cvd->vdev_ashift;
	mvd->vdev_logical_ashift = cvd->vdev_logical_ashift;
	mvd->vdev_physical_ashift = cvd->vdev_physical_ashift;
	mvd->vdev_state = cvd->vdev_state;
	mvd->vdev_crtxg = cvd->vdev_crtxg;

	vdev_remove_child(pvd, cvd);
	vdev_add_child(pvd, mvd);
	cvd->vdev_id = mvd->vdev_children;
	vdev_add_child(mvd, cvd);
	vdev_top_update(cvd->vdev_top, cvd->vdev_top);

	if (mvd == mvd->vdev_top)
	vdev_top_transfer(cvd, mvd);

	return (mvd);
	}

	/*
	* Remove a 1-way mirror/replacing vdev from the tree.
	*/
	void
	vdev_remove_parent(vdev_t *cvd)
	{
	vdev_t *mvd = cvd->vdev_parent;
	vdev_t *pvd = mvd->vdev_parent;

	ASSERT(spa_config_held(cvd->vdev_spa, SCL_ALL, RW_WRITER) == SCL_ALL);

	ASSERT(mvd->vdev_children == 1);
	ASSERT(mvd->vdev_ops == &vdev_mirror_ops \|\|
	mvd->vdev_ops == &vdev_replacing_ops \|\|
	mvd->vdev_ops == &vdev_spare_ops);
	cvd->vdev_ashift = mvd->vdev_ashift;
	cvd->vdev_logical_ashift = mvd->vdev_logical_ashift;
	cvd->vdev_physical_ashift = mvd->vdev_physical_ashift;
	vdev_remove_child(mvd, cvd);
	vdev_remove_child(pvd, mvd);

	/*
	* If cvd will replace mvd as a top-level vdev, preserve mvd's guid.
	* Otherwise, we could have detached an offline device, and when we
	* go to import the pool we'll think we have two top-level vdevs,
	* instead of a different version of the same top-level vdev.
	*/
	if (mvd->vdev_top == mvd) {
	uint64_t guid_delta = mvd->vdev_guid - cvd->vdev_guid;
	cvd->vdev_orig_guid = cvd->vdev_guid;
	cvd->vdev_guid += guid_delta;
	cvd->vdev_guid_sum += guid_delta;

	/*
	* If pool not set for autoexpand, we need to also preserve
	* mvd's asize to prevent automatic expansion of cvd.
	* Otherwise if we are adjusting the mirror by attaching and
	* detaching children of non-uniform sizes, the mirror could
	* autoexpand, unexpectedly requiring larger devices to
	* re-establish the mirror.
	*/
	if (!cvd->vdev_spa->spa_autoexpand)
	cvd->vdev_asize = mvd->vdev_asize;
	}
	cvd->vdev_id = mvd->vdev_id;
	vdev_add_child(pvd, cvd);
	vdev_top_update(cvd->vdev_top, cvd->vdev_top);

	if (cvd == cvd->vdev_top)
	vdev_top_transfer(mvd, cvd);

	ASSERT(mvd->vdev_children == 0);
	vdev_free(mvd);
	}

	static void
	vdev_metaslab_group_create(vdev_t *vd)
	{
	spa_t *spa = vd->vdev_spa;

	/*
	* metaslab_group_create was delayed until allocation bias was available
	*/
	if (vd->vdev_mg == NULL) {
	metaslab_class_t *mc;

	if (vd->vdev_islog && vd->vdev_alloc_bias == VDEV_BIAS_NONE)
	vd->vdev_alloc_bias = VDEV_BIAS_LOG;

	ASSERT3U(vd->vdev_islog, ==,
	(vd->vdev_alloc_bias == VDEV_BIAS_LOG));

	switch (vd->vdev_alloc_bias) {
	case VDEV_BIAS_LOG:
	mc = spa_log_class(spa);
	break;
	case VDEV_BIAS_SPECIAL:
	mc = spa_special_class(spa);
	break;
	case VDEV_BIAS_DEDUP:
	mc = spa_dedup_class(spa);
	break;
	default:
	mc = spa_normal_class(spa);
	}

	vd->vdev_mg = metaslab_group_create(mc, vd,
	spa->spa_alloc_count);

	/*
	- * The spa ashift values currently only reflect the
	- * general vdev classes. Class destination is late
	- * binding so ashift checking had to wait until now
	+ * The spa ashift min/max only apply for the normal metaslab
	+ * class. Class destination is late binding so ashift boundry
	+ * setting had to wait until now.
	*/
	if (vd->vdev_top == vd && vd->vdev_ashift != 0 &&
	mc == spa_normal_class(spa) && vd->vdev_aux == NULL) {
	if (vd->vdev_ashift > spa->spa_max_ashift)
	spa->spa_max_ashift = vd->vdev_ashift;
	if (vd->vdev_ashift < spa->spa_min_ashift)
	spa->spa_min_ashift = vd->vdev_ashift;
	}
	}
	}

	int
	vdev_metaslab_init(vdev_t *vd, uint64_t txg)
	{
	spa_t *spa = vd->vdev_spa;
	objset_t *mos = spa->spa_meta_objset;
	uint64_t m;
	uint64_t oldc = vd->vdev_ms_count;
	uint64_t newc = vd->vdev_asize >> vd->vdev_ms_shift;
	metaslab_t **mspp;
	int error;
	boolean_t expanding = (oldc != 0);

	ASSERT(txg == 0 \|\| spa_config_held(spa, SCL_ALLOC, RW_WRITER));

	/*
	* This vdev is not being allocated from yet or is a hole.
	*/
	if (vd->vdev_ms_shift == 0)
	return (0);

	ASSERT(!vd->vdev_ishole);

	ASSERT(oldc <= newc);

	mspp = vmem_zalloc(newc * sizeof (*mspp), KM_SLEEP);

	if (expanding) {
	bcopy(vd->vdev_ms, mspp, oldc * sizeof (*mspp));
	vmem_free(vd->vdev_ms, oldc * sizeof (*mspp));
	}

	vd->vdev_ms = mspp;
	vd->vdev_ms_count = newc;
	for (m = oldc; m < newc; m++) {
	uint64_t object = 0;

	/*
	* vdev_ms_array may be 0 if we are creating the "fake"
	* metaslabs for an indirect vdev for zdb's leak detection.
	* See zdb_leak_init().
	*/
	if (txg == 0 && vd->vdev_ms_array != 0) {
	error = dmu_read(mos, vd->vdev_ms_array,
	m * sizeof (uint64_t), sizeof (uint64_t), &object,
	DMU_READ_PREFETCH);
	if (error != 0) {
	vdev_dbgmsg(vd, "unable to read the metaslab "
	"array [error=%d]", error);
	return (error);
	}
	}

	#ifndef _KERNEL
	/*
	* To accommodate zdb_leak_init() fake indirect
	* metaslabs, we allocate a metaslab group for
	* indirect vdevs which normally don't have one.
	*/
	if (vd->vdev_mg == NULL) {
	ASSERT0(vdev_is_concrete(vd));
	vdev_metaslab_group_create(vd);
	}
	#endif
	error = metaslab_init(vd->vdev_mg, m, object, txg,
	&(vd->vdev_ms[m]));
	if (error != 0) {
	vdev_dbgmsg(vd, "metaslab_init failed [error=%d]",
	error);
	return (error);
	}
	}

	if (txg == 0)
	spa_config_enter(spa, SCL_ALLOC, FTAG, RW_WRITER);

	/*
	* If the vdev is being removed we don't activate
	* the metaslabs since we want to ensure that no new
	* allocations are performed on this device.
	*/
	if (!expanding && !vd->vdev_removing) {
	metaslab_group_activate(vd->vdev_mg);
	}

	if (txg == 0)
	spa_config_exit(spa, SCL_ALLOC, FTAG);

	/*
	* Regardless whether this vdev was just added or it is being
	* expanded, the metaslab count has changed. Recalculate the
	* block limit.
	*/
	spa_log_sm_set_blocklimit(spa);

	return (0);
	}

	void
	vdev_metaslab_fini(vdev_t *vd)
	{
	if (vd->vdev_checkpoint_sm != NULL) {
	ASSERT(spa_feature_is_active(vd->vdev_spa,
	SPA_FEATURE_POOL_CHECKPOINT));
	space_map_close(vd->vdev_checkpoint_sm);
	/*
	* Even though we close the space map, we need to set its
	* pointer to NULL. The reason is that vdev_metaslab_fini()
	* may be called multiple times for certain operations
	* (i.e. when destroying a pool) so we need to ensure that
	* this clause never executes twice. This logic is similar
	* to the one used for the vdev_ms clause below.
	*/
	vd->vdev_checkpoint_sm = NULL;
	}

	if (vd->vdev_ms != NULL) {
	metaslab_group_t *mg = vd->vdev_mg;
	metaslab_group_passivate(mg);

	uint64_t count = vd->vdev_ms_count;
	for (uint64_t m = 0; m < count; m++) {
	metaslab_t *msp = vd->vdev_ms[m];
	if (msp != NULL)
	metaslab_fini(msp);
	}
	vmem_free(vd->vdev_ms, count * sizeof (metaslab_t *));
	vd->vdev_ms = NULL;

	vd->vdev_ms_count = 0;

	for (int i = 0; i < RANGE_TREE_HISTOGRAM_SIZE; i++)
	ASSERT0(mg->mg_histogram[i]);
	}
	ASSERT0(vd->vdev_ms_count);
	ASSERT3U(vd->vdev_pending_fastwrite, ==, 0);
	}

	typedef struct vdev_probe_stats {
	boolean_t vps_readable;
	boolean_t vps_writeable;
	int vps_flags;
	} vdev_probe_stats_t;

	static void
	vdev_probe_done(zio_t *zio)
	{
	spa_t *spa = zio->io_spa;
	vdev_t *vd = zio->io_vd;
	vdev_probe_stats_t *vps = zio->io_private;

	ASSERT(vd->vdev_probe_zio != NULL);

	if (zio->io_type == ZIO_TYPE_READ) {
	if (zio->io_error == 0)
	vps->vps_readable = 1;
	if (zio->io_error == 0 && spa_writeable(spa)) {
	zio_nowait(zio_write_phys(vd->vdev_probe_zio, vd,
	zio->io_offset, zio->io_size, zio->io_abd,
	ZIO_CHECKSUM_OFF, vdev_probe_done, vps,
	ZIO_PRIORITY_SYNC_WRITE, vps->vps_flags, B_TRUE));
	} else {
	abd_free(zio->io_abd);
	}
	} else if (zio->io_type == ZIO_TYPE_WRITE) {
	if (zio->io_error == 0)
	vps->vps_writeable = 1;
	abd_free(zio->io_abd);
	} else if (zio->io_type == ZIO_TYPE_NULL) {
	zio_t *pio;
	zio_link_t *zl;

	vd->vdev_cant_read \|= !vps->vps_readable;
	vd->vdev_cant_write \|= !vps->vps_writeable;

	if (vdev_readable(vd) &&
	(vdev_writeable(vd) \|\| !spa_writeable(spa))) {
	zio->io_error = 0;
	} else {
	ASSERT(zio->io_error != 0);
	vdev_dbgmsg(vd, "failed probe");
	(void) zfs_ereport_post(FM_EREPORT_ZFS_PROBE_FAILURE,
	spa, vd, NULL, NULL, 0);
	zio->io_error = SET_ERROR(ENXIO);
	}

	mutex_enter(&vd->vdev_probe_lock);
	ASSERT(vd->vdev_probe_zio == zio);
	vd->vdev_probe_zio = NULL;
	mutex_exit(&vd->vdev_probe_lock);

	zl = NULL;
	while ((pio = zio_walk_parents(zio, &zl)) != NULL)
	if (!vdev_accessible(vd, pio))
	pio->io_error = SET_ERROR(ENXIO);

	kmem_free(vps, sizeof (*vps));
	}
	}

	/*
	* Determine whether this device is accessible.
	*
	* Read and write to several known locations: the pad regions of each
	* vdev label but the first, which we leave alone in case it contains
	* a VTOC.
	*/
	zio_t *
	vdev_probe(vdev_t vd, zio_t zio)
	{
	spa_t *spa = vd->vdev_spa;
	vdev_probe_stats_t *vps = NULL;
	zio_t *pio;

	ASSERT(vd->vdev_ops->vdev_op_leaf);

	/*
	* Don't probe the probe.
	*/
	if (zio && (zio->io_flags & ZIO_FLAG_PROBE))
	return (NULL);

	/*
	* To prevent 'probe storms' when a device fails, we create
	* just one probe i/o at a time. All zios that want to probe
	* this vdev will become parents of the probe io.
	*/
	mutex_enter(&vd->vdev_probe_lock);

	if ((pio = vd->vdev_probe_zio) == NULL) {
	vps = kmem_zalloc(sizeof (*vps), KM_SLEEP);

	vps->vps_flags = ZIO_FLAG_CANFAIL \| ZIO_FLAG_PROBE \|
	ZIO_FLAG_DONT_CACHE \| ZIO_FLAG_DONT_AGGREGATE \|
	ZIO_FLAG_TRYHARD;

	if (spa_config_held(spa, SCL_ZIO, RW_WRITER)) {
	/*
	* vdev_cant_read and vdev_cant_write can only
	* transition from TRUE to FALSE when we have the
	* SCL_ZIO lock as writer; otherwise they can only
	* transition from FALSE to TRUE. This ensures that
	* any zio looking at these values can assume that
	* failures persist for the life of the I/O. That's
	* important because when a device has intermittent
	* connectivity problems, we want to ensure that
	* they're ascribed to the device (ENXIO) and not
	* the zio (EIO).
	*
	* Since we hold SCL_ZIO as writer here, clear both
	* values so the probe can reevaluate from first
	* principles.
	*/
	vps->vps_flags \|= ZIO_FLAG_CONFIG_WRITER;
	vd->vdev_cant_read = B_FALSE;
	vd->vdev_cant_write = B_FALSE;
	}

	vd->vdev_probe_zio = pio = zio_null(NULL, spa, vd,
	vdev_probe_done, vps,
	vps->vps_flags \| ZIO_FLAG_DONT_PROPAGATE);

	/*
	* We can't change the vdev state in this context, so we
	* kick off an async task to do it on our behalf.
	*/
	if (zio != NULL) {
	vd->vdev_probe_wanted = B_TRUE;
	spa_async_request(spa, SPA_ASYNC_PROBE);
	}
	}

	if (zio != NULL)
	zio_add_child(zio, pio);

	mutex_exit(&vd->vdev_probe_lock);

	if (vps == NULL) {
	ASSERT(zio != NULL);
	return (NULL);
	}

	for (int l = 1; l < VDEV_LABELS; l++) {
	zio_nowait(zio_read_phys(pio, vd,
	vdev_label_offset(vd->vdev_psize, l,
	offsetof(vdev_label_t, vl_be)), VDEV_PAD_SIZE,
	abd_alloc_for_io(VDEV_PAD_SIZE, B_TRUE),
	ZIO_CHECKSUM_OFF, vdev_probe_done, vps,
	ZIO_PRIORITY_SYNC_READ, vps->vps_flags, B_TRUE));
	}

	if (zio == NULL)
	return (pio);

	zio_nowait(pio);
	return (NULL);
	}

	static void
	vdev_open_child(void *arg)
	{
	vdev_t *vd = arg;

	vd->vdev_open_thread = curthread;
	vd->vdev_open_error = vdev_open(vd);
	vd->vdev_open_thread = NULL;
	}

	static boolean_t
	vdev_uses_zvols(vdev_t *vd)
	{
	#ifdef _KERNEL
	if (zvol_is_zvol(vd->vdev_path))
	return (B_TRUE);
	#endif

	for (int c = 0; c < vd->vdev_children; c++)
	if (vdev_uses_zvols(vd->vdev_child[c]))
	return (B_TRUE);

	return (B_FALSE);
	}

	void
	vdev_open_children(vdev_t *vd)
	{
	taskq_t *tq;
	int children = vd->vdev_children;

	/*
	* in order to handle pools on top of zvols, do the opens
	* in a single thread so that the same thread holds the
	* spa_namespace_lock
	*/
	if (vdev_uses_zvols(vd)) {
	retry_sync:
	for (int c = 0; c < children; c++)
	vd->vdev_child[c]->vdev_open_error =
	vdev_open(vd->vdev_child[c]);
	} else {
	tq = taskq_create("vdev_open", children, minclsyspri,
	children, children, TASKQ_PREPOPULATE);
	if (tq == NULL)
	goto retry_sync;

	for (int c = 0; c < children; c++)
	VERIFY(taskq_dispatch(tq, vdev_open_child,
	vd->vdev_child[c], TQ_SLEEP) != TASKQID_INVALID);

	taskq_destroy(tq);
	}

	vd->vdev_nonrot = B_TRUE;

	for (int c = 0; c < children; c++)
	vd->vdev_nonrot &= vd->vdev_child[c]->vdev_nonrot;
	}

	/*
	* Compute the raidz-deflation ratio. Note, we hard-code
	* in 128k (1 << 17) because it is the "typical" blocksize.
	* Even though SPA_MAXBLOCKSIZE changed, this algorithm can not change,
	* otherwise it would inconsistently account for existing bp's.
	*/
	static void
	vdev_set_deflate_ratio(vdev_t *vd)
	{
	if (vd == vd->vdev_top && !vd->vdev_ishole && vd->vdev_ashift != 0) {
	vd->vdev_deflate_ratio = (1 << 17) /
	(vdev_psize_to_asize(vd, 1 << 17) >> SPA_MINBLOCKSHIFT);
	}
	}

	/*
	* Maximize performance by inflating the configured ashift for top level
	* vdevs to be as close to the physical ashift as possible while maintaining
	* administrator defined limits and ensuring it doesn't go below the
	* logical ashift.
	*/
	static void
	vdev_ashift_optimize(vdev_t *vd)
	{
	ASSERT(vd == vd->vdev_top);

	if (vd->vdev_ashift < vd->vdev_physical_ashift) {
	vd->vdev_ashift = MIN(
	MAX(zfs_vdev_max_auto_ashift, vd->vdev_ashift),
	MAX(zfs_vdev_min_auto_ashift,
	vd->vdev_physical_ashift));
	} else {
	/*
	* If the logical and physical ashifts are the same, then
	* we ensure that the top-level vdev's ashift is not smaller
	* than our minimum ashift value. For the unusual case
	* where logical ashift > physical ashift, we can't cap
	* the calculated ashift based on max ashift as that
	* would cause failures.
	* We still check if we need to increase it to match
	* the min ashift.
	*/
	vd->vdev_ashift = MAX(zfs_vdev_min_auto_ashift,
	vd->vdev_ashift);
	}
	}

	/*
	* Prepare a virtual device for access.
	*/
	int
	vdev_open(vdev_t *vd)
	{
	spa_t *spa = vd->vdev_spa;
	int error;
	uint64_t osize = 0;
	uint64_t max_osize = 0;
	uint64_t asize, max_asize, psize;
	uint64_t logical_ashift = 0;
	uint64_t physical_ashift = 0;

	ASSERT(vd->vdev_open_thread == curthread \|\|
	spa_config_held(spa, SCL_STATE_ALL, RW_WRITER) == SCL_STATE_ALL);
	ASSERT(vd->vdev_state == VDEV_STATE_CLOSED \|\|
	vd->vdev_state == VDEV_STATE_CANT_OPEN \|\|
	vd->vdev_state == VDEV_STATE_OFFLINE);

	vd->vdev_stat.vs_aux = VDEV_AUX_NONE;
	vd->vdev_cant_read = B_FALSE;
	vd->vdev_cant_write = B_FALSE;
	vd->vdev_min_asize = vdev_get_min_asize(vd);

	/*
	* If this vdev is not removed, check its fault status. If it's
	* faulted, bail out of the open.
	*/
	if (!vd->vdev_removed && vd->vdev_faulted) {
	ASSERT(vd->vdev_children == 0);
	ASSERT(vd->vdev_label_aux == VDEV_AUX_ERR_EXCEEDED \|\|
	vd->vdev_label_aux == VDEV_AUX_EXTERNAL);
	vdev_set_state(vd, B_TRUE, VDEV_STATE_FAULTED,
	vd->vdev_label_aux);
	return (SET_ERROR(ENXIO));
	} else if (vd->vdev_offline) {
	ASSERT(vd->vdev_children == 0);
	vdev_set_state(vd, B_TRUE, VDEV_STATE_OFFLINE, VDEV_AUX_NONE);
	return (SET_ERROR(ENXIO));
	}

	error = vd->vdev_ops->vdev_op_open(vd, &osize, &max_osize,
	&logical_ashift, &physical_ashift);
	/*
	* Physical volume size should never be larger than its max size, unless
	* the disk has shrunk while we were reading it or the device is buggy
	* or damaged: either way it's not safe for use, bail out of the open.
	*/
	if (osize > max_osize) {
	vdev_set_state(vd, B_TRUE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_OPEN_FAILED);
	return (SET_ERROR(ENXIO));
	}

	/*
	* Reset the vdev_reopening flag so that we actually close
	* the vdev on error.
	*/
	vd->vdev_reopening = B_FALSE;
	if (zio_injection_enabled && error == 0)
	error = zio_handle_device_injection(vd, NULL, SET_ERROR(ENXIO));

	if (error) {
	if (vd->vdev_removed &&
	vd->vdev_stat.vs_aux != VDEV_AUX_OPEN_FAILED)
	vd->vdev_removed = B_FALSE;

	if (vd->vdev_stat.vs_aux == VDEV_AUX_CHILDREN_OFFLINE) {
	vdev_set_state(vd, B_TRUE, VDEV_STATE_OFFLINE,
	vd->vdev_stat.vs_aux);
	} else {
	vdev_set_state(vd, B_TRUE, VDEV_STATE_CANT_OPEN,
	vd->vdev_stat.vs_aux);
	}
	return (error);
	}

	vd->vdev_removed = B_FALSE;

	/*
	* Recheck the faulted flag now that we have confirmed that
	* the vdev is accessible. If we're faulted, bail.
	*/
	if (vd->vdev_faulted) {
	ASSERT(vd->vdev_children == 0);
	ASSERT(vd->vdev_label_aux == VDEV_AUX_ERR_EXCEEDED \|\|
	vd->vdev_label_aux == VDEV_AUX_EXTERNAL);
	vdev_set_state(vd, B_TRUE, VDEV_STATE_FAULTED,
	vd->vdev_label_aux);
	return (SET_ERROR(ENXIO));
	}

	if (vd->vdev_degraded) {
	ASSERT(vd->vdev_children == 0);
	vdev_set_state(vd, B_TRUE, VDEV_STATE_DEGRADED,
	VDEV_AUX_ERR_EXCEEDED);
	} else {
	vdev_set_state(vd, B_TRUE, VDEV_STATE_HEALTHY, 0);
	}

	/*
	* For hole or missing vdevs we just return success.
	*/
	if (vd->vdev_ishole \|\| vd->vdev_ops == &vdev_missing_ops)
	return (0);

	for (int c = 0; c < vd->vdev_children; c++) {
	if (vd->vdev_child[c]->vdev_state != VDEV_STATE_HEALTHY) {
	vdev_set_state(vd, B_TRUE, VDEV_STATE_DEGRADED,
	VDEV_AUX_NONE);
	break;
	}
	}

	osize = P2ALIGN(osize, (uint64_t)sizeof (vdev_label_t));
	max_osize = P2ALIGN(max_osize, (uint64_t)sizeof (vdev_label_t));

	if (vd->vdev_children == 0) {
	if (osize < SPA_MINDEVSIZE) {
	vdev_set_state(vd, B_TRUE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_TOO_SMALL);
	return (SET_ERROR(EOVERFLOW));
	}
	psize = osize;
	asize = osize - (VDEV_LABEL_START_SIZE + VDEV_LABEL_END_SIZE);
	max_asize = max_osize - (VDEV_LABEL_START_SIZE +
	VDEV_LABEL_END_SIZE);
	} else {
	if (vd->vdev_parent != NULL && osize < SPA_MINDEVSIZE -
	(VDEV_LABEL_START_SIZE + VDEV_LABEL_END_SIZE)) {
	vdev_set_state(vd, B_TRUE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_TOO_SMALL);
	return (SET_ERROR(EOVERFLOW));
	}
	psize = 0;
	asize = osize;
	max_asize = max_osize;
	}

	/*
	* If the vdev was expanded, record this so that we can re-create the
	* uberblock rings in labels {2,3}, during the next sync.
	*/
	if ((psize > vd->vdev_psize) && (vd->vdev_psize != 0))
	vd->vdev_copy_uberblocks = B_TRUE;

	vd->vdev_psize = psize;

	/*
	* Make sure the allocatable size hasn't shrunk too much.
	*/
	if (asize < vd->vdev_min_asize) {
	vdev_set_state(vd, B_TRUE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_BAD_LABEL);
	return (SET_ERROR(EINVAL));
	}

	/*
	* We can always set the logical/physical ashift members since
	* their values are only used to calculate the vdev_ashift when
	* the device is first added to the config. These values should
	* not be used for anything else since they may change whenever
	* the device is reopened and we don't store them in the label.
	*/
	vd->vdev_physical_ashift =
	MAX(physical_ashift, vd->vdev_physical_ashift);
	vd->vdev_logical_ashift = MAX(logical_ashift,
	vd->vdev_logical_ashift);

	if (vd->vdev_asize == 0) {
	/*
	* This is the first-ever open, so use the computed values.
	* For compatibility, a different ashift can be requested.
	*/
	vd->vdev_asize = asize;
	vd->vdev_max_asize = max_asize;

	/*
	* If the vdev_ashift was not overriden at creation time,
	* then set it the logical ashift and optimize the ashift.
	*/
	if (vd->vdev_ashift == 0) {
	vd->vdev_ashift = vd->vdev_logical_ashift;

	if (vd->vdev_logical_ashift > ASHIFT_MAX) {
	vdev_set_state(vd, B_TRUE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_ASHIFT_TOO_BIG);
	return (SET_ERROR(EDOM));
	}

	if (vd->vdev_top == vd) {
	vdev_ashift_optimize(vd);
	}
	}
	if (vd->vdev_ashift != 0 && (vd->vdev_ashift < ASHIFT_MIN \|\|
	vd->vdev_ashift > ASHIFT_MAX)) {
	vdev_set_state(vd, B_TRUE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_BAD_ASHIFT);
	return (SET_ERROR(EDOM));
	}
	} else {
	/*
	* Make sure the alignment required hasn't increased.
	*/
	if (vd->vdev_ashift > vd->vdev_top->vdev_ashift &&
	vd->vdev_ops->vdev_op_leaf) {
	(void) zfs_ereport_post(
	FM_EREPORT_ZFS_DEVICE_BAD_ASHIFT,
	spa, vd, NULL, NULL, 0);
	vdev_set_state(vd, B_TRUE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_BAD_LABEL);
	return (SET_ERROR(EDOM));
	}
	vd->vdev_max_asize = max_asize;
	}

	/*
	* If all children are healthy we update asize if either:
	* The asize has increased, due to a device expansion caused by dynamic
	* LUN growth or vdev replacement, and automatic expansion is enabled;
	* making the additional space available.
	*
	* The asize has decreased, due to a device shrink usually caused by a
	* vdev replace with a smaller device. This ensures that calculations
	* based of max_asize and asize e.g. esize are always valid. It's safe
	* to do this as we've already validated that asize is greater than
	* vdev_min_asize.
	*/
	if (vd->vdev_state == VDEV_STATE_HEALTHY &&
	((asize > vd->vdev_asize &&
	(vd->vdev_expanding \|\| spa->spa_autoexpand)) \|\|
	(asize < vd->vdev_asize)))
	vd->vdev_asize = asize;

	vdev_set_min_asize(vd);

	/*
	* Ensure we can issue some IO before declaring the
	* vdev open for business.
	*/
	if (vd->vdev_ops->vdev_op_leaf &&
	(error = zio_wait(vdev_probe(vd, NULL))) != 0) {
	vdev_set_state(vd, B_TRUE, VDEV_STATE_FAULTED,
	VDEV_AUX_ERR_EXCEEDED);
	return (error);
	- }
	-
	- /*
	- * Track the min and max ashift values for normal data devices.
	- */
	- if (vd->vdev_top == vd && vd->vdev_ashift != 0 &&
	- vd->vdev_alloc_bias == VDEV_BIAS_NONE &&
	- vd->vdev_islog == 0 && vd->vdev_aux == NULL) {
	- if (vd->vdev_ashift > spa->spa_max_ashift)
	- spa->spa_max_ashift = vd->vdev_ashift;
	- if (vd->vdev_ashift < spa->spa_min_ashift)
	- spa->spa_min_ashift = vd->vdev_ashift;
	}

	/*
	* If this is a leaf vdev, assess whether a resilver is needed.
	* But don't do this if we are doing a reopen for a scrub, since
	* this would just restart the scrub we are already doing.
	*/
	if (vd->vdev_ops->vdev_op_leaf && !spa->spa_scrub_reopen)
	dsl_scan_assess_vdev(spa->spa_dsl_pool, vd);

	return (0);
	}

	/*
	* Called once the vdevs are all opened, this routine validates the label
	* contents. This needs to be done before vdev_load() so that we don't
	* inadvertently do repair I/Os to the wrong device.
	*
	* This function will only return failure if one of the vdevs indicates that it
	* has since been destroyed or exported. This is only possible if
	* /etc/zfs/zpool.cache was readonly at the time. Otherwise, the vdev state
	* will be updated but the function will return 0.
	*/
	int
	vdev_validate(vdev_t *vd)
	{
	spa_t *spa = vd->vdev_spa;
	nvlist_t *label;
	uint64_t guid = 0, aux_guid = 0, top_guid;
	uint64_t state;
	nvlist_t *nvl;
	uint64_t txg;

	if (vdev_validate_skip)
	return (0);

	for (uint64_t c = 0; c < vd->vdev_children; c++)
	if (vdev_validate(vd->vdev_child[c]) != 0)
	return (SET_ERROR(EBADF));

	/*
	* If the device has already failed, or was marked offline, don't do
	* any further validation. Otherwise, label I/O will fail and we will
	* overwrite the previous state.
	*/
	if (!vd->vdev_ops->vdev_op_leaf \|\| !vdev_readable(vd))
	return (0);

	/*
	* If we are performing an extreme rewind, we allow for a label that
	* was modified at a point after the current txg.
	* If config lock is not held do not check for the txg. spa_sync could
	* be updating the vdev's label before updating spa_last_synced_txg.
	*/
	if (spa->spa_extreme_rewind \|\| spa_last_synced_txg(spa) == 0 \|\|
	spa_config_held(spa, SCL_CONFIG, RW_WRITER) != SCL_CONFIG)
	txg = UINT64_MAX;
	else
	txg = spa_last_synced_txg(spa);

	if ((label = vdev_label_read_config(vd, txg)) == NULL) {
	vdev_set_state(vd, B_TRUE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_BAD_LABEL);
	vdev_dbgmsg(vd, "vdev_validate: failed reading config for "
	"txg %llu", (u_longlong_t)txg);
	return (0);
	}

	/*
	* Determine if this vdev has been split off into another
	* pool. If so, then refuse to open it.
	*/
	if (nvlist_lookup_uint64(label, ZPOOL_CONFIG_SPLIT_GUID,
	&aux_guid) == 0 && aux_guid == spa_guid(spa)) {
	vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_SPLIT_POOL);
	nvlist_free(label);
	vdev_dbgmsg(vd, "vdev_validate: vdev split into other pool");
	return (0);
	}

	if (nvlist_lookup_uint64(label, ZPOOL_CONFIG_POOL_GUID, &guid) != 0) {
	vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_CORRUPT_DATA);
	nvlist_free(label);
	vdev_dbgmsg(vd, "vdev_validate: '%s' missing from label",
	ZPOOL_CONFIG_POOL_GUID);
	return (0);
	}

	/*
	* If config is not trusted then ignore the spa guid check. This is
	* necessary because if the machine crashed during a re-guid the new
	* guid might have been written to all of the vdev labels, but not the
	* cached config. The check will be performed again once we have the
	* trusted config from the MOS.
	*/
	if (spa->spa_trust_config && guid != spa_guid(spa)) {
	vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_CORRUPT_DATA);
	nvlist_free(label);
	vdev_dbgmsg(vd, "vdev_validate: vdev label pool_guid doesn't "
	"match config (%llu != %llu)", (u_longlong_t)guid,
	(u_longlong_t)spa_guid(spa));
	return (0);
	}

	if (nvlist_lookup_nvlist(label, ZPOOL_CONFIG_VDEV_TREE, &nvl)
	!= 0 \|\| nvlist_lookup_uint64(nvl, ZPOOL_CONFIG_ORIG_GUID,
	&aux_guid) != 0)
	aux_guid = 0;

	if (nvlist_lookup_uint64(label, ZPOOL_CONFIG_GUID, &guid) != 0) {
	vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_CORRUPT_DATA);
	nvlist_free(label);
	vdev_dbgmsg(vd, "vdev_validate: '%s' missing from label",
	ZPOOL_CONFIG_GUID);
	return (0);
	}

	if (nvlist_lookup_uint64(label, ZPOOL_CONFIG_TOP_GUID, &top_guid)
	!= 0) {
	vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_CORRUPT_DATA);
	nvlist_free(label);
	vdev_dbgmsg(vd, "vdev_validate: '%s' missing from label",
	ZPOOL_CONFIG_TOP_GUID);
	return (0);
	}

	/*
	* If this vdev just became a top-level vdev because its sibling was
	* detached, it will have adopted the parent's vdev guid -- but the
	* label may or may not be on disk yet. Fortunately, either version
	* of the label will have the same top guid, so if we're a top-level
	* vdev, we can safely compare to that instead.
	* However, if the config comes from a cachefile that failed to update
	* after the detach, a top-level vdev will appear as a non top-level
	* vdev in the config. Also relax the constraints if we perform an
	* extreme rewind.
	*
	* If we split this vdev off instead, then we also check the
	* original pool's guid. We don't want to consider the vdev
	* corrupt if it is partway through a split operation.
	*/
	if (vd->vdev_guid != guid && vd->vdev_guid != aux_guid) {
	boolean_t mismatch = B_FALSE;
	if (spa->spa_trust_config && !spa->spa_extreme_rewind) {
	if (vd != vd->vdev_top \|\| vd->vdev_guid != top_guid)
	mismatch = B_TRUE;
	} else {
	if (vd->vdev_guid != top_guid &&
	vd->vdev_top->vdev_guid != guid)
	mismatch = B_TRUE;
	}

	if (mismatch) {
	vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_CORRUPT_DATA);
	nvlist_free(label);
	vdev_dbgmsg(vd, "vdev_validate: config guid "
	"doesn't match label guid");
	vdev_dbgmsg(vd, "CONFIG: guid %llu, top_guid %llu",
	(u_longlong_t)vd->vdev_guid,
	(u_longlong_t)vd->vdev_top->vdev_guid);
	vdev_dbgmsg(vd, "LABEL: guid %llu, top_guid %llu, "
	"aux_guid %llu", (u_longlong_t)guid,
	(u_longlong_t)top_guid, (u_longlong_t)aux_guid);
	return (0);
	}
	}

	if (nvlist_lookup_uint64(label, ZPOOL_CONFIG_POOL_STATE,
	&state) != 0) {
	vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_CORRUPT_DATA);
	nvlist_free(label);
	vdev_dbgmsg(vd, "vdev_validate: '%s' missing from label",
	ZPOOL_CONFIG_POOL_STATE);
	return (0);
	}

	nvlist_free(label);

	/*
	* If this is a verbatim import, no need to check the
	* state of the pool.
	*/
	if (!(spa->spa_import_flags & ZFS_IMPORT_VERBATIM) &&
	spa_load_state(spa) == SPA_LOAD_OPEN &&
	state != POOL_STATE_ACTIVE) {
	vdev_dbgmsg(vd, "vdev_validate: invalid pool state (%llu) "
	"for spa %s", (u_longlong_t)state, spa->spa_name);
	return (SET_ERROR(EBADF));
	}

	/*
	* If we were able to open and validate a vdev that was
	* previously marked permanently unavailable, clear that state
	* now.
	*/
	if (vd->vdev_not_present)
	vd->vdev_not_present = 0;

	return (0);
	}

	static void
	vdev_copy_path_impl(vdev_t svd, vdev_t dvd)
	{
	if (svd->vdev_path != NULL && dvd->vdev_path != NULL) {
	if (strcmp(svd->vdev_path, dvd->vdev_path) != 0) {
	zfs_dbgmsg("vdev_copy_path: vdev %llu: path changed "
	"from '%s' to '%s'", (u_longlong_t)dvd->vdev_guid,
	dvd->vdev_path, svd->vdev_path);
	spa_strfree(dvd->vdev_path);
	dvd->vdev_path = spa_strdup(svd->vdev_path);
	}
	} else if (svd->vdev_path != NULL) {
	dvd->vdev_path = spa_strdup(svd->vdev_path);
	zfs_dbgmsg("vdev_copy_path: vdev %llu: path set to '%s'",
	(u_longlong_t)dvd->vdev_guid, dvd->vdev_path);
	}
	}

	/*
	* Recursively copy vdev paths from one vdev to another. Source and destination
	* vdev trees must have same geometry otherwise return error. Intended to copy
	* paths from userland config into MOS config.
	*/
	int
	vdev_copy_path_strict(vdev_t svd, vdev_t dvd)
	{
	if ((svd->vdev_ops == &vdev_missing_ops) \|\|
	(svd->vdev_ishole && dvd->vdev_ishole) \|\|
	(dvd->vdev_ops == &vdev_indirect_ops))
	return (0);

	if (svd->vdev_ops != dvd->vdev_ops) {
	vdev_dbgmsg(svd, "vdev_copy_path: vdev type mismatch: %s != %s",
	svd->vdev_ops->vdev_op_type, dvd->vdev_ops->vdev_op_type);
	return (SET_ERROR(EINVAL));
	}

	if (svd->vdev_guid != dvd->vdev_guid) {
	vdev_dbgmsg(svd, "vdev_copy_path: guids mismatch (%llu != "
	"%llu)", (u_longlong_t)svd->vdev_guid,
	(u_longlong_t)dvd->vdev_guid);
	return (SET_ERROR(EINVAL));
	}

	if (svd->vdev_children != dvd->vdev_children) {
	vdev_dbgmsg(svd, "vdev_copy_path: children count mismatch: "
	"%llu != %llu", (u_longlong_t)svd->vdev_children,
	(u_longlong_t)dvd->vdev_children);
	return (SET_ERROR(EINVAL));
	}

	for (uint64_t i = 0; i < svd->vdev_children; i++) {
	int error = vdev_copy_path_strict(svd->vdev_child[i],
	dvd->vdev_child[i]);
	if (error != 0)
	return (error);
	}

	if (svd->vdev_ops->vdev_op_leaf)
	vdev_copy_path_impl(svd, dvd);

	return (0);
	}

	static void
	vdev_copy_path_search(vdev_t stvd, vdev_t dvd)
	{
	ASSERT(stvd->vdev_top == stvd);
	ASSERT3U(stvd->vdev_id, ==, dvd->vdev_top->vdev_id);

	for (uint64_t i = 0; i < dvd->vdev_children; i++) {
	vdev_copy_path_search(stvd, dvd->vdev_child[i]);
	}

	if (!dvd->vdev_ops->vdev_op_leaf \|\| !vdev_is_concrete(dvd))
	return;

	/*
	* The idea here is that while a vdev can shift positions within
	* a top vdev (when replacing, attaching mirror, etc.) it cannot
	* step outside of it.
	*/
	vdev_t *vd = vdev_lookup_by_guid(stvd, dvd->vdev_guid);

	if (vd == NULL \|\| vd->vdev_ops != dvd->vdev_ops)
	return;

	ASSERT(vd->vdev_ops->vdev_op_leaf);

	vdev_copy_path_impl(vd, dvd);
	}

	/*
	* Recursively copy vdev paths from one root vdev to another. Source and
	* destination vdev trees may differ in geometry. For each destination leaf
	* vdev, search a vdev with the same guid and top vdev id in the source.
	* Intended to copy paths from userland config into MOS config.
	*/
	void
	vdev_copy_path_relaxed(vdev_t srvd, vdev_t drvd)
	{
	uint64_t children = MIN(srvd->vdev_children, drvd->vdev_children);
	ASSERT(srvd->vdev_ops == &vdev_root_ops);
	ASSERT(drvd->vdev_ops == &vdev_root_ops);

	for (uint64_t i = 0; i < children; i++) {
	vdev_copy_path_search(srvd->vdev_child[i],
	drvd->vdev_child[i]);
	}
	}

	/*
	* Close a virtual device.
	*/
	void
	vdev_close(vdev_t *vd)
	{
	vdev_t *pvd = vd->vdev_parent;
	spa_t *spa __maybe_unused = vd->vdev_spa;

	ASSERT(spa_config_held(spa, SCL_STATE_ALL, RW_WRITER) == SCL_STATE_ALL);

	/*
	* If our parent is reopening, then we are as well, unless we are
	* going offline.
	*/
	if (pvd != NULL && pvd->vdev_reopening)
	vd->vdev_reopening = (pvd->vdev_reopening && !vd->vdev_offline);

	vd->vdev_ops->vdev_op_close(vd);

	vdev_cache_purge(vd);

	/*
	* We record the previous state before we close it, so that if we are
	* doing a reopen(), we don't generate FMA ereports if we notice that
	* it's still faulted.
	*/
	vd->vdev_prevstate = vd->vdev_state;

	if (vd->vdev_offline)
	vd->vdev_state = VDEV_STATE_OFFLINE;
	else
	vd->vdev_state = VDEV_STATE_CLOSED;
	vd->vdev_stat.vs_aux = VDEV_AUX_NONE;
	}

	void
	vdev_hold(vdev_t *vd)
	{
	spa_t *spa = vd->vdev_spa;

	ASSERT(spa_is_root(spa));
	if (spa->spa_state == POOL_STATE_UNINITIALIZED)
	return;

	for (int c = 0; c < vd->vdev_children; c++)
	vdev_hold(vd->vdev_child[c]);

	if (vd->vdev_ops->vdev_op_leaf)
	vd->vdev_ops->vdev_op_hold(vd);
	}

	void
	vdev_rele(vdev_t *vd)
	{
	ASSERT(spa_is_root(vd->vdev_spa));
	for (int c = 0; c < vd->vdev_children; c++)
	vdev_rele(vd->vdev_child[c]);

	if (vd->vdev_ops->vdev_op_leaf)
	vd->vdev_ops->vdev_op_rele(vd);
	}

	/*
	* Reopen all interior vdevs and any unopened leaves. We don't actually
	* reopen leaf vdevs which had previously been opened as they might deadlock
	* on the spa_config_lock. Instead we only obtain the leaf's physical size.
	* If the leaf has never been opened then open it, as usual.
	*/
	void
	vdev_reopen(vdev_t *vd)
	{
	spa_t *spa = vd->vdev_spa;

	ASSERT(spa_config_held(spa, SCL_STATE_ALL, RW_WRITER) == SCL_STATE_ALL);

	/* set the reopening flag unless we're taking the vdev offline */
	vd->vdev_reopening = !vd->vdev_offline;
	vdev_close(vd);
	(void) vdev_open(vd);

	/*
	* Call vdev_validate() here to make sure we have the same device.
	* Otherwise, a device with an invalid label could be successfully
	* opened in response to vdev_reopen().
	*/
	if (vd->vdev_aux) {
	(void) vdev_validate_aux(vd);
	if (vdev_readable(vd) && vdev_writeable(vd) &&
	vd->vdev_aux == &spa->spa_l2cache) {
	/*
	* In case the vdev is present we should evict all ARC
	* buffers and pointers to log blocks and reclaim their
	* space before restoring its contents to L2ARC.
	*/
	if (l2arc_vdev_present(vd)) {
	l2arc_rebuild_vdev(vd, B_TRUE);
	} else {
	l2arc_add_vdev(spa, vd);
	}
	spa_async_request(spa, SPA_ASYNC_L2CACHE_REBUILD);
	spa_async_request(spa, SPA_ASYNC_L2CACHE_TRIM);
	}
	} else {
	(void) vdev_validate(vd);
	}

	/*
	* Reassess parent vdev's health.
	*/
	vdev_propagate_state(vd);
	}

	int
	vdev_create(vdev_t *vd, uint64_t txg, boolean_t isreplacing)
	{
	int error;

	/*
	* Normally, partial opens (e.g. of a mirror) are allowed.
	* For a create, however, we want to fail the request if
	* there are any components we can't open.
	*/
	error = vdev_open(vd);

	if (error \|\| vd->vdev_state != VDEV_STATE_HEALTHY) {
	vdev_close(vd);
	return (error ? error : SET_ERROR(ENXIO));
	}

	/*
	* Recursively load DTLs and initialize all labels.
	*/
	if ((error = vdev_dtl_load(vd)) != 0 \|\|
	(error = vdev_label_init(vd, txg, isreplacing ?
	VDEV_LABEL_REPLACE : VDEV_LABEL_CREATE)) != 0) {
	vdev_close(vd);
	return (error);
	}

	return (0);
	}

	void
	vdev_metaslab_set_size(vdev_t *vd)
	{
	uint64_t asize = vd->vdev_asize;
	uint64_t ms_count = asize >> zfs_vdev_default_ms_shift;
	uint64_t ms_shift;

	/*
	* There are two dimensions to the metaslab sizing calculation:
	* the size of the metaslab and the count of metaslabs per vdev.
	*
	* The default values used below are a good balance between memory
	* usage (larger metaslab size means more memory needed for loaded
	* metaslabs; more metaslabs means more memory needed for the
	* metaslab_t structs), metaslab load time (larger metaslabs take
	* longer to load), and metaslab sync time (more metaslabs means
	* more time spent syncing all of them).
	*
	* In general, we aim for zfs_vdev_default_ms_count (200) metaslabs.
	* The range of the dimensions are as follows:
	*
	* 2^29 <= ms_size <= 2^34
	* 16 <= ms_count <= 131,072
	*
	* On the lower end of vdev sizes, we aim for metaslabs sizes of
	* at least 512MB (2^29) to minimize fragmentation effects when
	* testing with smaller devices. However, the count constraint
	* of at least 16 metaslabs will override this minimum size goal.
	*
	* On the upper end of vdev sizes, we aim for a maximum metaslab
	* size of 16GB. However, we will cap the total count to 2^17
	* metaslabs to keep our memory footprint in check and let the
	* metaslab size grow from there if that limit is hit.
	*
	* The net effect of applying above constrains is summarized below.
	*
	* vdev size metaslab count
	* --------------\|-----------------
	* < 8GB ~16
	* 8GB - 100GB one per 512MB
	* 100GB - 3TB ~200
	* 3TB - 2PB one per 16GB
	* > 2PB ~131,072
	* --------------------------------
	*
	* Finally, note that all of the above calculate the initial
	* number of metaslabs. Expanding a top-level vdev will result
	* in additional metaslabs being allocated making it possible
	* to exceed the zfs_vdev_ms_count_limit.
	*/

	if (ms_count < zfs_vdev_min_ms_count)
	ms_shift = highbit64(asize / zfs_vdev_min_ms_count);
	else if (ms_count > zfs_vdev_default_ms_count)
	ms_shift = highbit64(asize / zfs_vdev_default_ms_count);
	else
	ms_shift = zfs_vdev_default_ms_shift;

	if (ms_shift < SPA_MAXBLOCKSHIFT) {
	ms_shift = SPA_MAXBLOCKSHIFT;
	} else if (ms_shift > zfs_vdev_max_ms_shift) {
	ms_shift = zfs_vdev_max_ms_shift;
	/* cap the total count to constrain memory footprint */
	if ((asize >> ms_shift) > zfs_vdev_ms_count_limit)
	ms_shift = highbit64(asize / zfs_vdev_ms_count_limit);
	}

	vd->vdev_ms_shift = ms_shift;
	ASSERT3U(vd->vdev_ms_shift, >=, SPA_MAXBLOCKSHIFT);
	}

	void
	vdev_dirty(vdev_t vd, int flags, void arg, uint64_t txg)
	{
	ASSERT(vd == vd->vdev_top);
	/* indirect vdevs don't have metaslabs or dtls */
	ASSERT(vdev_is_concrete(vd) \|\| flags == 0);
	ASSERT(ISP2(flags));
	ASSERT(spa_writeable(vd->vdev_spa));

	if (flags & VDD_METASLAB)
	(void) txg_list_add(&vd->vdev_ms_list, arg, txg);

	if (flags & VDD_DTL)
	(void) txg_list_add(&vd->vdev_dtl_list, arg, txg);

	(void) txg_list_add(&vd->vdev_spa->spa_vdev_txg_list, vd, txg);
	}

	void
	vdev_dirty_leaves(vdev_t *vd, int flags, uint64_t txg)
	{
	for (int c = 0; c < vd->vdev_children; c++)
	vdev_dirty_leaves(vd->vdev_child[c], flags, txg);

	if (vd->vdev_ops->vdev_op_leaf)
	vdev_dirty(vd->vdev_top, flags, vd, txg);
	}

	/*
	* DTLs.
	*
	* A vdev's DTL (dirty time log) is the set of transaction groups for which
	* the vdev has less than perfect replication. There are four kinds of DTL:
	*
	* DTL_MISSING: txgs for which the vdev has no valid copies of the data
	*
	* DTL_PARTIAL: txgs for which data is available, but not fully replicated
	*
	* DTL_SCRUB: the txgs that could not be repaired by the last scrub; upon
	* scrub completion, DTL_SCRUB replaces DTL_MISSING in the range of
	* txgs that was scrubbed.
	*
	* DTL_OUTAGE: txgs which cannot currently be read, whether due to
	* persistent errors or just some device being offline.
	* Unlike the other three, the DTL_OUTAGE map is not generally
	* maintained; it's only computed when needed, typically to
	* determine whether a device can be detached.
	*
	* For leaf vdevs, DTL_MISSING and DTL_PARTIAL are identical: the device
	* either has the data or it doesn't.
	*
	* For interior vdevs such as mirror and RAID-Z the picture is more complex.
	* A vdev's DTL_PARTIAL is the union of its children's DTL_PARTIALs, because
	* if any child is less than fully replicated, then so is its parent.
	* A vdev's DTL_MISSING is a modified union of its children's DTL_MISSINGs,
	* comprising only those txgs which appear in 'maxfaults' or more children;
	* those are the txgs we don't have enough replication to read. For example,
	* double-parity RAID-Z can tolerate up to two missing devices (maxfaults == 2);
	* thus, its DTL_MISSING consists of the set of txgs that appear in more than
	* two child DTL_MISSING maps.
	*
	* It should be clear from the above that to compute the DTLs and outage maps
	* for all vdevs, it suffices to know just the leaf vdevs' DTL_MISSING maps.
	* Therefore, that is all we keep on disk. When loading the pool, or after
	* a configuration change, we generate all other DTLs from first principles.
	*/
	void
	vdev_dtl_dirty(vdev_t *vd, vdev_dtl_type_t t, uint64_t txg, uint64_t size)
	{
	range_tree_t *rt = vd->vdev_dtl[t];

	ASSERT(t < DTL_TYPES);
	ASSERT(vd != vd->vdev_spa->spa_root_vdev);
	ASSERT(spa_writeable(vd->vdev_spa));

	mutex_enter(&vd->vdev_dtl_lock);
	if (!range_tree_contains(rt, txg, size))
	range_tree_add(rt, txg, size);
	mutex_exit(&vd->vdev_dtl_lock);
	}

	boolean_t
	vdev_dtl_contains(vdev_t *vd, vdev_dtl_type_t t, uint64_t txg, uint64_t size)
	{
	range_tree_t *rt = vd->vdev_dtl[t];
	boolean_t dirty = B_FALSE;

	ASSERT(t < DTL_TYPES);
	ASSERT(vd != vd->vdev_spa->spa_root_vdev);

	/*
	* While we are loading the pool, the DTLs have not been loaded yet.
	* Ignore the DTLs and try all devices. This avoids a recursive
	* mutex enter on the vdev_dtl_lock, and also makes us try hard
	* when loading the pool (relying on the checksum to ensure that
	* we get the right data -- note that we while loading, we are
	* only reading the MOS, which is always checksummed).
	*/
	if (vd->vdev_spa->spa_load_state != SPA_LOAD_NONE)
	return (B_FALSE);

	mutex_enter(&vd->vdev_dtl_lock);
	if (!range_tree_is_empty(rt))
	dirty = range_tree_contains(rt, txg, size);
	mutex_exit(&vd->vdev_dtl_lock);

	return (dirty);
	}

	boolean_t
	vdev_dtl_empty(vdev_t *vd, vdev_dtl_type_t t)
	{
	range_tree_t *rt = vd->vdev_dtl[t];
	boolean_t empty;

	mutex_enter(&vd->vdev_dtl_lock);
	empty = range_tree_is_empty(rt);
	mutex_exit(&vd->vdev_dtl_lock);

	return (empty);
	}

	/*
	* Returns B_TRUE if vdev determines offset needs to be resilvered.
	*/
	boolean_t
	vdev_dtl_need_resilver(vdev_t *vd, uint64_t offset, size_t psize)
	{
	ASSERT(vd != vd->vdev_spa->spa_root_vdev);

	if (vd->vdev_ops->vdev_op_need_resilver == NULL \|\|
	vd->vdev_ops->vdev_op_leaf)
	return (B_TRUE);

	return (vd->vdev_ops->vdev_op_need_resilver(vd, offset, psize));
	}

	/*
	* Returns the lowest txg in the DTL range.
	*/
	static uint64_t
	vdev_dtl_min(vdev_t *vd)
	{
	ASSERT(MUTEX_HELD(&vd->vdev_dtl_lock));
	ASSERT3U(range_tree_space(vd->vdev_dtl[DTL_MISSING]), !=, 0);
	ASSERT0(vd->vdev_children);

	return (range_tree_min(vd->vdev_dtl[DTL_MISSING]) - 1);
	}

	/*
	* Returns the highest txg in the DTL.
	*/
	static uint64_t
	vdev_dtl_max(vdev_t *vd)
	{
	ASSERT(MUTEX_HELD(&vd->vdev_dtl_lock));
	ASSERT3U(range_tree_space(vd->vdev_dtl[DTL_MISSING]), !=, 0);
	ASSERT0(vd->vdev_children);

	return (range_tree_max(vd->vdev_dtl[DTL_MISSING]));
	}

	/*
	* Determine if a resilvering vdev should remove any DTL entries from
	* its range. If the vdev was resilvering for the entire duration of the
	* scan then it should excise that range from its DTLs. Otherwise, this
	* vdev is considered partially resilvered and should leave its DTL
	* entries intact. The comment in vdev_dtl_reassess() describes how we
	* excise the DTLs.
	*/
	static boolean_t
	vdev_dtl_should_excise(vdev_t *vd, boolean_t rebuild_done)
	{
	ASSERT0(vd->vdev_children);

	if (vd->vdev_state < VDEV_STATE_DEGRADED)
	return (B_FALSE);

	if (vd->vdev_resilver_deferred)
	return (B_FALSE);

	if (range_tree_is_empty(vd->vdev_dtl[DTL_MISSING]))
	return (B_TRUE);

	if (rebuild_done) {
	vdev_rebuild_t *vr = &vd->vdev_top->vdev_rebuild_config;
	vdev_rebuild_phys_t *vrp = &vr->vr_rebuild_phys;

	/* Rebuild not initiated by attach */
	if (vd->vdev_rebuild_txg == 0)
	return (B_TRUE);

	/*
	* When a rebuild completes without error then all missing data
	* up to the rebuild max txg has been reconstructed and the DTL
	* is eligible for excision.
	*/
	if (vrp->vrp_rebuild_state == VDEV_REBUILD_COMPLETE &&
	vdev_dtl_max(vd) <= vrp->vrp_max_txg) {
	ASSERT3U(vrp->vrp_min_txg, <=, vdev_dtl_min(vd));
	ASSERT3U(vrp->vrp_min_txg, <, vd->vdev_rebuild_txg);
	ASSERT3U(vd->vdev_rebuild_txg, <=, vrp->vrp_max_txg);
	return (B_TRUE);
	}
	} else {
	dsl_scan_t *scn = vd->vdev_spa->spa_dsl_pool->dp_scan;
	dsl_scan_phys_t *scnp __maybe_unused = &scn->scn_phys;

	/* Resilver not initiated by attach */
	if (vd->vdev_resilver_txg == 0)
	return (B_TRUE);

	/*
	* When a resilver is initiated the scan will assign the
	* scn_max_txg value to the highest txg value that exists
	* in all DTLs. If this device's max DTL is not part of this
	* scan (i.e. it is not in the range (scn_min_txg, scn_max_txg]
	* then it is not eligible for excision.
	*/
	if (vdev_dtl_max(vd) <= scn->scn_phys.scn_max_txg) {
	ASSERT3U(scnp->scn_min_txg, <=, vdev_dtl_min(vd));
	ASSERT3U(scnp->scn_min_txg, <, vd->vdev_resilver_txg);
	ASSERT3U(vd->vdev_resilver_txg, <=, scnp->scn_max_txg);
	return (B_TRUE);
	}
	}

	return (B_FALSE);
	}

	/*
	* Reassess DTLs after a config change or scrub completion. If txg == 0 no
	* write operations will be issued to the pool.
	*/
	void
	vdev_dtl_reassess(vdev_t *vd, uint64_t txg, uint64_t scrub_txg,
	boolean_t scrub_done, boolean_t rebuild_done)
	{
	spa_t *spa = vd->vdev_spa;
	avl_tree_t reftree;
	int minref;

	ASSERT(spa_config_held(spa, SCL_ALL, RW_READER) != 0);

	for (int c = 0; c < vd->vdev_children; c++)
	vdev_dtl_reassess(vd->vdev_child[c], txg,
	scrub_txg, scrub_done, rebuild_done);

	if (vd == spa->spa_root_vdev \|\| !vdev_is_concrete(vd) \|\| vd->vdev_aux)
	return;

	if (vd->vdev_ops->vdev_op_leaf) {
	dsl_scan_t *scn = spa->spa_dsl_pool->dp_scan;
	vdev_rebuild_t *vr = &vd->vdev_top->vdev_rebuild_config;
	boolean_t check_excise = B_FALSE;
	boolean_t wasempty = B_TRUE;

	mutex_enter(&vd->vdev_dtl_lock);

	/*
	* If requested, pretend the scan or rebuild completed cleanly.
	*/
	if (zfs_scan_ignore_errors) {
	if (scn != NULL)
	scn->scn_phys.scn_errors = 0;
	if (vr != NULL)
	vr->vr_rebuild_phys.vrp_errors = 0;
	}

	if (scrub_txg != 0 &&
	!range_tree_is_empty(vd->vdev_dtl[DTL_MISSING])) {
	wasempty = B_FALSE;
	zfs_dbgmsg("guid:%llu txg:%llu scrub:%llu started:%d "
	"dtl:%llu/%llu errors:%llu",
	(u_longlong_t)vd->vdev_guid, (u_longlong_t)txg,
	(u_longlong_t)scrub_txg, spa->spa_scrub_started,
	(u_longlong_t)vdev_dtl_min(vd),
	(u_longlong_t)vdev_dtl_max(vd),
	(u_longlong_t)(scn ? scn->scn_phys.scn_errors : 0));
	}

	/*
	* If we've completed a scrub/resilver or a rebuild cleanly
	* then determine if this vdev should remove any DTLs. We
	* only want to excise regions on vdevs that were available
	* during the entire duration of this scan.
	*/
	if (rebuild_done &&
	vr != NULL && vr->vr_rebuild_phys.vrp_errors == 0) {
	check_excise = B_TRUE;
	} else {
	if (spa->spa_scrub_started \|\|
	(scn != NULL && scn->scn_phys.scn_errors == 0)) {
	check_excise = B_TRUE;
	}
	}

	if (scrub_txg && check_excise &&
	vdev_dtl_should_excise(vd, rebuild_done)) {
	/*
	* We completed a scrub, resilver or rebuild up to
	* scrub_txg. If we did it without rebooting, then
	* the scrub dtl will be valid, so excise the old
	* region and fold in the scrub dtl. Otherwise,
	* leave the dtl as-is if there was an error.
	*
	* There's little trick here: to excise the beginning
	* of the DTL_MISSING map, we put it into a reference
	* tree and then add a segment with refcnt -1 that
	* covers the range [0, scrub_txg). This means
	* that each txg in that range has refcnt -1 or 0.
	* We then add DTL_SCRUB with a refcnt of 2, so that
	* entries in the range [0, scrub_txg) will have a
	* positive refcnt -- either 1 or 2. We then convert
	* the reference tree into the new DTL_MISSING map.
	*/
	space_reftree_create(&reftree);
	space_reftree_add_map(&reftree,
	vd->vdev_dtl[DTL_MISSING], 1);
	space_reftree_add_seg(&reftree, 0, scrub_txg, -1);
	space_reftree_add_map(&reftree,
	vd->vdev_dtl[DTL_SCRUB], 2);
	space_reftree_generate_map(&reftree,
	vd->vdev_dtl[DTL_MISSING], 1);
	space_reftree_destroy(&reftree);

	if (!range_tree_is_empty(vd->vdev_dtl[DTL_MISSING])) {
	zfs_dbgmsg("update DTL_MISSING:%llu/%llu",
	(u_longlong_t)vdev_dtl_min(vd),
	(u_longlong_t)vdev_dtl_max(vd));
	} else if (!wasempty) {
	zfs_dbgmsg("DTL_MISSING is now empty");
	}
	}
	range_tree_vacate(vd->vdev_dtl[DTL_PARTIAL], NULL, NULL);
	range_tree_walk(vd->vdev_dtl[DTL_MISSING],
	range_tree_add, vd->vdev_dtl[DTL_PARTIAL]);
	if (scrub_done)
	range_tree_vacate(vd->vdev_dtl[DTL_SCRUB], NULL, NULL);
	range_tree_vacate(vd->vdev_dtl[DTL_OUTAGE], NULL, NULL);
	if (!vdev_readable(vd))
	range_tree_add(vd->vdev_dtl[DTL_OUTAGE], 0, -1ULL);
	else
	range_tree_walk(vd->vdev_dtl[DTL_MISSING],
	range_tree_add, vd->vdev_dtl[DTL_OUTAGE]);

	/*
	* If the vdev was resilvering or rebuilding and no longer
	* has any DTLs then reset the appropriate flag and dirty
	* the top level so that we persist the change.
	*/
	if (txg != 0 &&
	range_tree_is_empty(vd->vdev_dtl[DTL_MISSING]) &&
	range_tree_is_empty(vd->vdev_dtl[DTL_OUTAGE])) {
	if (vd->vdev_rebuild_txg != 0) {
	vd->vdev_rebuild_txg = 0;
	vdev_config_dirty(vd->vdev_top);
	} else if (vd->vdev_resilver_txg != 0) {
	vd->vdev_resilver_txg = 0;
	vdev_config_dirty(vd->vdev_top);
	}
	}

	mutex_exit(&vd->vdev_dtl_lock);

	if (txg != 0)
	vdev_dirty(vd->vdev_top, VDD_DTL, vd, txg);
	return;
	}

	mutex_enter(&vd->vdev_dtl_lock);
	for (int t = 0; t < DTL_TYPES; t++) {
	/* account for child's outage in parent's missing map */
	int s = (t == DTL_MISSING) ? DTL_OUTAGE: t;
	if (t == DTL_SCRUB)
	continue; /* leaf vdevs only */
	if (t == DTL_PARTIAL)
	minref = 1; /* i.e. non-zero */
	else if (vd->vdev_nparity != 0)
	minref = vd->vdev_nparity + 1; /* RAID-Z */
	else
	minref = vd->vdev_children; /* any kind of mirror */
	space_reftree_create(&reftree);
	for (int c = 0; c < vd->vdev_children; c++) {
	vdev_t *cvd = vd->vdev_child[c];
	mutex_enter(&cvd->vdev_dtl_lock);
	space_reftree_add_map(&reftree, cvd->vdev_dtl[s], 1);
	mutex_exit(&cvd->vdev_dtl_lock);
	}
	space_reftree_generate_map(&reftree, vd->vdev_dtl[t], minref);
	space_reftree_destroy(&reftree);
	}
	mutex_exit(&vd->vdev_dtl_lock);
	}

	int
	vdev_dtl_load(vdev_t *vd)
	{
	spa_t *spa = vd->vdev_spa;
	objset_t *mos = spa->spa_meta_objset;
	int error = 0;

	if (vd->vdev_ops->vdev_op_leaf && vd->vdev_dtl_object != 0) {
	ASSERT(vdev_is_concrete(vd));

	error = space_map_open(&vd->vdev_dtl_sm, mos,
	vd->vdev_dtl_object, 0, -1ULL, 0);
	if (error)
	return (error);
	ASSERT(vd->vdev_dtl_sm != NULL);

	mutex_enter(&vd->vdev_dtl_lock);
	error = space_map_load(vd->vdev_dtl_sm,
	vd->vdev_dtl[DTL_MISSING], SM_ALLOC);
	mutex_exit(&vd->vdev_dtl_lock);

	return (error);
	}

	for (int c = 0; c < vd->vdev_children; c++) {
	error = vdev_dtl_load(vd->vdev_child[c]);
	if (error != 0)
	break;
	}

	return (error);
	}

	static void
	vdev_zap_allocation_data(vdev_t vd, dmu_tx_t tx)
	{
	spa_t *spa = vd->vdev_spa;
	objset_t *mos = spa->spa_meta_objset;
	vdev_alloc_bias_t alloc_bias = vd->vdev_alloc_bias;
	const char *string;

	ASSERT(alloc_bias != VDEV_BIAS_NONE);

	string =
	(alloc_bias == VDEV_BIAS_LOG) ? VDEV_ALLOC_BIAS_LOG :
	(alloc_bias == VDEV_BIAS_SPECIAL) ? VDEV_ALLOC_BIAS_SPECIAL :
	(alloc_bias == VDEV_BIAS_DEDUP) ? VDEV_ALLOC_BIAS_DEDUP : NULL;

	ASSERT(string != NULL);
	VERIFY0(zap_add(mos, vd->vdev_top_zap, VDEV_TOP_ZAP_ALLOCATION_BIAS,
	1, strlen(string) + 1, string, tx));

	if (alloc_bias == VDEV_BIAS_SPECIAL \|\| alloc_bias == VDEV_BIAS_DEDUP) {
	spa_activate_allocation_classes(spa, tx);
	}
	}

	void
	vdev_destroy_unlink_zap(vdev_t vd, uint64_t zapobj, dmu_tx_t tx)
	{
	spa_t *spa = vd->vdev_spa;

	VERIFY0(zap_destroy(spa->spa_meta_objset, zapobj, tx));
	VERIFY0(zap_remove_int(spa->spa_meta_objset, spa->spa_all_vdev_zaps,
	zapobj, tx));
	}

	uint64_t
	vdev_create_link_zap(vdev_t vd, dmu_tx_t tx)
	{
	spa_t *spa = vd->vdev_spa;
	uint64_t zap = zap_create(spa->spa_meta_objset, DMU_OTN_ZAP_METADATA,
	DMU_OT_NONE, 0, tx);

	ASSERT(zap != 0);
	VERIFY0(zap_add_int(spa->spa_meta_objset, spa->spa_all_vdev_zaps,
	zap, tx));

	return (zap);
	}

	void
	vdev_construct_zaps(vdev_t vd, dmu_tx_t tx)
	{
	if (vd->vdev_ops != &vdev_hole_ops &&
	vd->vdev_ops != &vdev_missing_ops &&
	vd->vdev_ops != &vdev_root_ops &&
	!vd->vdev_top->vdev_removing) {
	if (vd->vdev_ops->vdev_op_leaf && vd->vdev_leaf_zap == 0) {
	vd->vdev_leaf_zap = vdev_create_link_zap(vd, tx);
	}
	if (vd == vd->vdev_top && vd->vdev_top_zap == 0) {
	vd->vdev_top_zap = vdev_create_link_zap(vd, tx);
	if (vd->vdev_alloc_bias != VDEV_BIAS_NONE)
	vdev_zap_allocation_data(vd, tx);
	}
	}

	for (uint64_t i = 0; i < vd->vdev_children; i++) {
	vdev_construct_zaps(vd->vdev_child[i], tx);
	}
	}

	static void
	vdev_dtl_sync(vdev_t *vd, uint64_t txg)
	{
	spa_t *spa = vd->vdev_spa;
	range_tree_t *rt = vd->vdev_dtl[DTL_MISSING];
	objset_t *mos = spa->spa_meta_objset;
	range_tree_t *rtsync;
	dmu_tx_t *tx;
	uint64_t object = space_map_object(vd->vdev_dtl_sm);

	ASSERT(vdev_is_concrete(vd));
	ASSERT(vd->vdev_ops->vdev_op_leaf);

	tx = dmu_tx_create_assigned(spa->spa_dsl_pool, txg);

	if (vd->vdev_detached \|\| vd->vdev_top->vdev_removing) {
	mutex_enter(&vd->vdev_dtl_lock);
	space_map_free(vd->vdev_dtl_sm, tx);
	space_map_close(vd->vdev_dtl_sm);
	vd->vdev_dtl_sm = NULL;
	mutex_exit(&vd->vdev_dtl_lock);

	/*
	* We only destroy the leaf ZAP for detached leaves or for
	* removed log devices. Removed data devices handle leaf ZAP
	* cleanup later, once cancellation is no longer possible.
	*/
	if (vd->vdev_leaf_zap != 0 && (vd->vdev_detached \|\|
	vd->vdev_top->vdev_islog)) {
	vdev_destroy_unlink_zap(vd, vd->vdev_leaf_zap, tx);
	vd->vdev_leaf_zap = 0;
	}

	dmu_tx_commit(tx);
	return;
	}

	if (vd->vdev_dtl_sm == NULL) {
	uint64_t new_object;

	new_object = space_map_alloc(mos, zfs_vdev_dtl_sm_blksz, tx);
	VERIFY3U(new_object, !=, 0);

	VERIFY0(space_map_open(&vd->vdev_dtl_sm, mos, new_object,
	0, -1ULL, 0));
	ASSERT(vd->vdev_dtl_sm != NULL);
	}

	rtsync = range_tree_create(NULL, RANGE_SEG64, NULL, 0, 0);

	mutex_enter(&vd->vdev_dtl_lock);
	range_tree_walk(rt, range_tree_add, rtsync);
	mutex_exit(&vd->vdev_dtl_lock);

	space_map_truncate(vd->vdev_dtl_sm, zfs_vdev_dtl_sm_blksz, tx);
	space_map_write(vd->vdev_dtl_sm, rtsync, SM_ALLOC, SM_NO_VDEVID, tx);
	range_tree_vacate(rtsync, NULL, NULL);

	range_tree_destroy(rtsync);

	/*
	* If the object for the space map has changed then dirty
	* the top level so that we update the config.
	*/
	if (object != space_map_object(vd->vdev_dtl_sm)) {
	vdev_dbgmsg(vd, "txg %llu, spa %s, DTL old object %llu, "
	"new object %llu", (u_longlong_t)txg, spa_name(spa),
	(u_longlong_t)object,
	(u_longlong_t)space_map_object(vd->vdev_dtl_sm));
	vdev_config_dirty(vd->vdev_top);
	}

	dmu_tx_commit(tx);
	}

	/*
	* Determine whether the specified vdev can be offlined/detached/removed
	* without losing data.
	*/
	boolean_t
	vdev_dtl_required(vdev_t *vd)
	{
	spa_t *spa = vd->vdev_spa;
	vdev_t *tvd = vd->vdev_top;
	uint8_t cant_read = vd->vdev_cant_read;
	boolean_t required;

	ASSERT(spa_config_held(spa, SCL_STATE_ALL, RW_WRITER) == SCL_STATE_ALL);

	if (vd == spa->spa_root_vdev \|\| vd == tvd)
	return (B_TRUE);

	/*
	* Temporarily mark the device as unreadable, and then determine
	* whether this results in any DTL outages in the top-level vdev.
	* If not, we can safely offline/detach/remove the device.
	*/
	vd->vdev_cant_read = B_TRUE;
	vdev_dtl_reassess(tvd, 0, 0, B_FALSE, B_FALSE);
	required = !vdev_dtl_empty(tvd, DTL_OUTAGE);
	vd->vdev_cant_read = cant_read;
	vdev_dtl_reassess(tvd, 0, 0, B_FALSE, B_FALSE);

	if (!required && zio_injection_enabled) {
	required = !!zio_handle_device_injection(vd, NULL,
	SET_ERROR(ECHILD));
	}

	return (required);
	}

	/*
	* Determine if resilver is needed, and if so the txg range.
	*/
	boolean_t
	vdev_resilver_needed(vdev_t vd, uint64_t minp, uint64_t *maxp)
	{
	boolean_t needed = B_FALSE;
	uint64_t thismin = UINT64_MAX;
	uint64_t thismax = 0;

	if (vd->vdev_children == 0) {
	mutex_enter(&vd->vdev_dtl_lock);
	if (!range_tree_is_empty(vd->vdev_dtl[DTL_MISSING]) &&
	vdev_writeable(vd)) {

	thismin = vdev_dtl_min(vd);
	thismax = vdev_dtl_max(vd);
	needed = B_TRUE;
	}
	mutex_exit(&vd->vdev_dtl_lock);
	} else {
	for (int c = 0; c < vd->vdev_children; c++) {
	vdev_t *cvd = vd->vdev_child[c];
	uint64_t cmin, cmax;

	if (vdev_resilver_needed(cvd, &cmin, &cmax)) {
	thismin = MIN(thismin, cmin);
	thismax = MAX(thismax, cmax);
	needed = B_TRUE;
	}
	}
	}

	if (needed && minp) {
	*minp = thismin;
	*maxp = thismax;
	}
	return (needed);
	}

	/*
	* Gets the checkpoint space map object from the vdev's ZAP. On success sm_obj
	* will contain either the checkpoint spacemap object or zero if none exists.
	* All other errors are returned to the caller.
	*/
	int
	vdev_checkpoint_sm_object(vdev_t vd, uint64_t sm_obj)
	{
	ASSERT0(spa_config_held(vd->vdev_spa, SCL_ALL, RW_WRITER));

	if (vd->vdev_top_zap == 0) {
	*sm_obj = 0;
	return (0);
	}

	int error = zap_lookup(spa_meta_objset(vd->vdev_spa), vd->vdev_top_zap,
	VDEV_TOP_ZAP_POOL_CHECKPOINT_SM, sizeof (uint64_t), 1, sm_obj);
	if (error == ENOENT) {
	*sm_obj = 0;
	error = 0;
	}

	return (error);
	}

	int
	vdev_load(vdev_t *vd)
	{
	int error = 0;

	/*
	* Recursively load all children.
	*/
	for (int c = 0; c < vd->vdev_children; c++) {
	error = vdev_load(vd->vdev_child[c]);
	if (error != 0) {
	return (error);
	}
	}

	vdev_set_deflate_ratio(vd);

	/*
	* On spa_load path, grab the allocation bias from our zap
	*/
	if (vd == vd->vdev_top && vd->vdev_top_zap != 0) {
	spa_t *spa = vd->vdev_spa;
	char bias_str[64];

	error = zap_lookup(spa->spa_meta_objset, vd->vdev_top_zap,
	VDEV_TOP_ZAP_ALLOCATION_BIAS, 1, sizeof (bias_str),
	bias_str);
	if (error == 0) {
	ASSERT(vd->vdev_alloc_bias == VDEV_BIAS_NONE);
	vd->vdev_alloc_bias = vdev_derive_alloc_bias(bias_str);
	} else if (error != ENOENT) {
	vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_CORRUPT_DATA);
	vdev_dbgmsg(vd, "vdev_load: zap_lookup(top_zap=%llu) "
	"failed [error=%d]", vd->vdev_top_zap, error);
	return (error);
	}
	}

	/*
	* Load any rebuild state from the top-level vdev zap.
	*/
	if (vd == vd->vdev_top && vd->vdev_top_zap != 0) {
	error = vdev_rebuild_load(vd);
	if (error && error != ENOTSUP) {
	vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_CORRUPT_DATA);
	vdev_dbgmsg(vd, "vdev_load: vdev_rebuild_load "
	"failed [error=%d]", error);
	return (error);
	}
	}

	/*
	* If this is a top-level vdev, initialize its metaslabs.
	*/
	if (vd == vd->vdev_top && vdev_is_concrete(vd)) {
	vdev_metaslab_group_create(vd);

	if (vd->vdev_ashift == 0 \|\| vd->vdev_asize == 0) {
	vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_CORRUPT_DATA);
	vdev_dbgmsg(vd, "vdev_load: invalid size. ashift=%llu, "
	"asize=%llu", (u_longlong_t)vd->vdev_ashift,
	(u_longlong_t)vd->vdev_asize);
	return (SET_ERROR(ENXIO));
	}

	error = vdev_metaslab_init(vd, 0);
	if (error != 0) {
	vdev_dbgmsg(vd, "vdev_load: metaslab_init failed "
	"[error=%d]", error);
	vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_CORRUPT_DATA);
	return (error);
	}

	uint64_t checkpoint_sm_obj;
	error = vdev_checkpoint_sm_object(vd, &checkpoint_sm_obj);
	if (error == 0 && checkpoint_sm_obj != 0) {
	objset_t *mos = spa_meta_objset(vd->vdev_spa);
	ASSERT(vd->vdev_asize != 0);
	ASSERT3P(vd->vdev_checkpoint_sm, ==, NULL);

	error = space_map_open(&vd->vdev_checkpoint_sm,
	mos, checkpoint_sm_obj, 0, vd->vdev_asize,
	vd->vdev_ashift);
	if (error != 0) {
	vdev_dbgmsg(vd, "vdev_load: space_map_open "
	"failed for checkpoint spacemap (obj %llu) "
	"[error=%d]",
	(u_longlong_t)checkpoint_sm_obj, error);
	return (error);
	}
	ASSERT3P(vd->vdev_checkpoint_sm, !=, NULL);

	/*
	* Since the checkpoint_sm contains free entries
	* exclusively we can use space_map_allocated() to
	* indicate the cumulative checkpointed space that
	* has been freed.
	*/
	vd->vdev_stat.vs_checkpoint_space =
	-space_map_allocated(vd->vdev_checkpoint_sm);
	vd->vdev_spa->spa_checkpoint_info.sci_dspace +=
	vd->vdev_stat.vs_checkpoint_space;
	} else if (error != 0) {
	vdev_dbgmsg(vd, "vdev_load: failed to retrieve "
	"checkpoint space map object from vdev ZAP "
	"[error=%d]", error);
	return (error);
	}
	}

	/*
	* If this is a leaf vdev, load its DTL.
	*/
	if (vd->vdev_ops->vdev_op_leaf && (error = vdev_dtl_load(vd)) != 0) {
	vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_CORRUPT_DATA);
	vdev_dbgmsg(vd, "vdev_load: vdev_dtl_load failed "
	"[error=%d]", error);
	return (error);
	}

	uint64_t obsolete_sm_object;
	error = vdev_obsolete_sm_object(vd, &obsolete_sm_object);
	if (error == 0 && obsolete_sm_object != 0) {
	objset_t *mos = vd->vdev_spa->spa_meta_objset;
	ASSERT(vd->vdev_asize != 0);
	ASSERT3P(vd->vdev_obsolete_sm, ==, NULL);

	if ((error = space_map_open(&vd->vdev_obsolete_sm, mos,
	obsolete_sm_object, 0, vd->vdev_asize, 0))) {
	vdev_set_state(vd, B_FALSE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_CORRUPT_DATA);
	vdev_dbgmsg(vd, "vdev_load: space_map_open failed for "
	"obsolete spacemap (obj %llu) [error=%d]",
	(u_longlong_t)obsolete_sm_object, error);
	return (error);
	}
	} else if (error != 0) {
	vdev_dbgmsg(vd, "vdev_load: failed to retrieve obsolete "
	"space map object from vdev ZAP [error=%d]", error);
	return (error);
	}

	return (0);
	}

	/*
	* The special vdev case is used for hot spares and l2cache devices. Its
	* sole purpose it to set the vdev state for the associated vdev. To do this,
	* we make sure that we can open the underlying device, then try to read the
	* label, and make sure that the label is sane and that it hasn't been
	* repurposed to another pool.
	*/
	int
	vdev_validate_aux(vdev_t *vd)
	{
	nvlist_t *label;
	uint64_t guid, version;
	uint64_t state;

	if (!vdev_readable(vd))
	return (0);

	if ((label = vdev_label_read_config(vd, -1ULL)) == NULL) {
	vdev_set_state(vd, B_TRUE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_CORRUPT_DATA);
	return (-1);
	}

	if (nvlist_lookup_uint64(label, ZPOOL_CONFIG_VERSION, &version) != 0 \|\|
	!SPA_VERSION_IS_SUPPORTED(version) \|\|
	nvlist_lookup_uint64(label, ZPOOL_CONFIG_GUID, &guid) != 0 \|\|
	guid != vd->vdev_guid \|\|
	nvlist_lookup_uint64(label, ZPOOL_CONFIG_POOL_STATE, &state) != 0) {
	vdev_set_state(vd, B_TRUE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_CORRUPT_DATA);
	nvlist_free(label);
	return (-1);
	}

	/*
	* We don't actually check the pool state here. If it's in fact in
	* use by another pool, we update this fact on the fly when requested.
	*/
	nvlist_free(label);
	return (0);
	}

	static void
	vdev_destroy_ms_flush_data(vdev_t vd, dmu_tx_t tx)
	{
	objset_t *mos = spa_meta_objset(vd->vdev_spa);

	if (vd->vdev_top_zap == 0)
	return;

	uint64_t object = 0;
	int err = zap_lookup(mos, vd->vdev_top_zap,
	VDEV_TOP_ZAP_MS_UNFLUSHED_PHYS_TXGS, sizeof (uint64_t), 1, &object);
	if (err == ENOENT)
	return;
	VERIFY0(err);

	VERIFY0(dmu_object_free(mos, object, tx));
	VERIFY0(zap_remove(mos, vd->vdev_top_zap,
	VDEV_TOP_ZAP_MS_UNFLUSHED_PHYS_TXGS, tx));
	}

	/*
	* Free the objects used to store this vdev's spacemaps, and the array
	* that points to them.
	*/
	void
	vdev_destroy_spacemaps(vdev_t vd, dmu_tx_t tx)
	{
	if (vd->vdev_ms_array == 0)
	return;

	objset_t *mos = vd->vdev_spa->spa_meta_objset;
	uint64_t array_count = vd->vdev_asize >> vd->vdev_ms_shift;
	size_t array_bytes = array_count * sizeof (uint64_t);
	uint64_t *smobj_array = kmem_alloc(array_bytes, KM_SLEEP);
	VERIFY0(dmu_read(mos, vd->vdev_ms_array, 0,
	array_bytes, smobj_array, 0));

	for (uint64_t i = 0; i < array_count; i++) {
	uint64_t smobj = smobj_array[i];
	if (smobj == 0)
	continue;

	space_map_free_obj(mos, smobj, tx);
	}

	kmem_free(smobj_array, array_bytes);
	VERIFY0(dmu_object_free(mos, vd->vdev_ms_array, tx));
	vdev_destroy_ms_flush_data(vd, tx);
	vd->vdev_ms_array = 0;
	}

	static void
	vdev_remove_empty_log(vdev_t *vd, uint64_t txg)
	{
	spa_t *spa = vd->vdev_spa;

	ASSERT(vd->vdev_islog);
	ASSERT(vd == vd->vdev_top);
	ASSERT3U(txg, ==, spa_syncing_txg(spa));

	dmu_tx_t *tx = dmu_tx_create_assigned(spa_get_dsl(spa), txg);

	vdev_destroy_spacemaps(vd, tx);
	if (vd->vdev_top_zap != 0) {
	vdev_destroy_unlink_zap(vd, vd->vdev_top_zap, tx);
	vd->vdev_top_zap = 0;
	}

	dmu_tx_commit(tx);
	}

	void
	vdev_sync_done(vdev_t *vd, uint64_t txg)
	{
	metaslab_t *msp;
	boolean_t reassess = !txg_list_empty(&vd->vdev_ms_list, TXG_CLEAN(txg));

	ASSERT(vdev_is_concrete(vd));

	while ((msp = txg_list_remove(&vd->vdev_ms_list, TXG_CLEAN(txg)))
	!= NULL)
	metaslab_sync_done(msp, txg);

	if (reassess)
	metaslab_sync_reassess(vd->vdev_mg);
	}

	void
	vdev_sync(vdev_t *vd, uint64_t txg)
	{
	spa_t *spa = vd->vdev_spa;
	vdev_t *lvd;
	metaslab_t *msp;

	ASSERT3U(txg, ==, spa->spa_syncing_txg);
	dmu_tx_t *tx = dmu_tx_create_assigned(spa->spa_dsl_pool, txg);
	if (range_tree_space(vd->vdev_obsolete_segments) > 0) {
	ASSERT(vd->vdev_removing \|\|
	vd->vdev_ops == &vdev_indirect_ops);

	vdev_indirect_sync_obsolete(vd, tx);

	/*
	* If the vdev is indirect, it can't have dirty
	* metaslabs or DTLs.
	*/
	if (vd->vdev_ops == &vdev_indirect_ops) {
	ASSERT(txg_list_empty(&vd->vdev_ms_list, txg));
	ASSERT(txg_list_empty(&vd->vdev_dtl_list, txg));
	dmu_tx_commit(tx);
	return;
	}
	}

	ASSERT(vdev_is_concrete(vd));

	if (vd->vdev_ms_array == 0 && vd->vdev_ms_shift != 0 &&
	!vd->vdev_removing) {
	ASSERT(vd == vd->vdev_top);
	ASSERT0(vd->vdev_indirect_config.vic_mapping_object);
	vd->vdev_ms_array = dmu_object_alloc(spa->spa_meta_objset,
	DMU_OT_OBJECT_ARRAY, 0, DMU_OT_NONE, 0, tx);
	ASSERT(vd->vdev_ms_array != 0);
	vdev_config_dirty(vd);
	}

	while ((msp = txg_list_remove(&vd->vdev_ms_list, txg)) != NULL) {
	metaslab_sync(msp, txg);
	(void) txg_list_add(&vd->vdev_ms_list, msp, TXG_CLEAN(txg));
	}

	while ((lvd = txg_list_remove(&vd->vdev_dtl_list, txg)) != NULL)
	vdev_dtl_sync(lvd, txg);

	/*
	* If this is an empty log device being removed, destroy the
	* metadata associated with it.
	*/
	if (vd->vdev_islog && vd->vdev_stat.vs_alloc == 0 && vd->vdev_removing)
	vdev_remove_empty_log(vd, txg);

	(void) txg_list_add(&spa->spa_vdev_txg_list, vd, TXG_CLEAN(txg));
	dmu_tx_commit(tx);
	}

	uint64_t
	vdev_psize_to_asize(vdev_t *vd, uint64_t psize)
	{
	return (vd->vdev_ops->vdev_op_asize(vd, psize));
	}

	/*
	* Mark the given vdev faulted. A faulted vdev behaves as if the device could
	* not be opened, and no I/O is attempted.
	*/
	int
	vdev_fault(spa_t *spa, uint64_t guid, vdev_aux_t aux)
	{
	vdev_t vd, tvd;

	spa_vdev_state_enter(spa, SCL_NONE);

	if ((vd = spa_lookup_by_guid(spa, guid, B_TRUE)) == NULL)
	return (spa_vdev_state_exit(spa, NULL, SET_ERROR(ENODEV)));

	if (!vd->vdev_ops->vdev_op_leaf)
	return (spa_vdev_state_exit(spa, NULL, SET_ERROR(ENOTSUP)));

	tvd = vd->vdev_top;

	/*
	* If user did a 'zpool offline -f' then make the fault persist across
	* reboots.
	*/
	if (aux == VDEV_AUX_EXTERNAL_PERSIST) {
	/*
	* There are two kinds of forced faults: temporary and
	* persistent. Temporary faults go away at pool import, while
	* persistent faults stay set. Both types of faults can be
	* cleared with a zpool clear.
	*
	* We tell if a vdev is persistently faulted by looking at the
	* ZPOOL_CONFIG_AUX_STATE nvpair. If it's set to "external" at
	* import then it's a persistent fault. Otherwise, it's
	* temporary. We get ZPOOL_CONFIG_AUX_STATE set to "external"
	* by setting vd.vdev_stat.vs_aux to VDEV_AUX_EXTERNAL. This
	* tells vdev_config_generate() (which gets run later) to set
	* ZPOOL_CONFIG_AUX_STATE to "external" in the nvlist.
	*/
	vd->vdev_stat.vs_aux = VDEV_AUX_EXTERNAL;
	vd->vdev_tmpoffline = B_FALSE;
	aux = VDEV_AUX_EXTERNAL;
	} else {
	vd->vdev_tmpoffline = B_TRUE;
	}

	/*
	* We don't directly use the aux state here, but if we do a
	* vdev_reopen(), we need this value to be present to remember why we
	* were faulted.
	*/
	vd->vdev_label_aux = aux;

	/*
	* Faulted state takes precedence over degraded.
	*/
	vd->vdev_delayed_close = B_FALSE;
	vd->vdev_faulted = 1ULL;
	vd->vdev_degraded = 0ULL;
	vdev_set_state(vd, B_FALSE, VDEV_STATE_FAULTED, aux);

	/*
	* If this device has the only valid copy of the data, then
	* back off and simply mark the vdev as degraded instead.
	*/
	if (!tvd->vdev_islog && vd->vdev_aux == NULL && vdev_dtl_required(vd)) {
	vd->vdev_degraded = 1ULL;
	vd->vdev_faulted = 0ULL;

	/*
	* If we reopen the device and it's not dead, only then do we
	* mark it degraded.
	*/
	vdev_reopen(tvd);

	if (vdev_readable(vd))
	vdev_set_state(vd, B_FALSE, VDEV_STATE_DEGRADED, aux);
	}

	return (spa_vdev_state_exit(spa, vd, 0));
	}

	/*
	* Mark the given vdev degraded. A degraded vdev is purely an indication to the
	* user that something is wrong. The vdev continues to operate as normal as far
	* as I/O is concerned.
	*/
	int
	vdev_degrade(spa_t *spa, uint64_t guid, vdev_aux_t aux)
	{
	vdev_t *vd;

	spa_vdev_state_enter(spa, SCL_NONE);

	if ((vd = spa_lookup_by_guid(spa, guid, B_TRUE)) == NULL)
	return (spa_vdev_state_exit(spa, NULL, SET_ERROR(ENODEV)));

	if (!vd->vdev_ops->vdev_op_leaf)
	return (spa_vdev_state_exit(spa, NULL, SET_ERROR(ENOTSUP)));

	/*
	* If the vdev is already faulted, then don't do anything.
	*/
	if (vd->vdev_faulted \|\| vd->vdev_degraded)
	return (spa_vdev_state_exit(spa, NULL, 0));

	vd->vdev_degraded = 1ULL;
	if (!vdev_is_dead(vd))
	vdev_set_state(vd, B_FALSE, VDEV_STATE_DEGRADED,
	aux);

	return (spa_vdev_state_exit(spa, vd, 0));
	}

	/*
	* Online the given vdev.
	*
	* If 'ZFS_ONLINE_UNSPARE' is set, it implies two things. First, any attached
	* spare device should be detached when the device finishes resilvering.
	* Second, the online should be treated like a 'test' online case, so no FMA
	* events are generated if the device fails to open.
	*/
	int
	vdev_online(spa_t spa, uint64_t guid, uint64_t flags, vdev_state_t newstate)
	{
	vdev_t vd, tvd, pvd, rvd = spa->spa_root_vdev;
	boolean_t wasoffline;
	vdev_state_t oldstate;

	spa_vdev_state_enter(spa, SCL_NONE);

	if ((vd = spa_lookup_by_guid(spa, guid, B_TRUE)) == NULL)
	return (spa_vdev_state_exit(spa, NULL, SET_ERROR(ENODEV)));

	if (!vd->vdev_ops->vdev_op_leaf)
	return (spa_vdev_state_exit(spa, NULL, SET_ERROR(ENOTSUP)));

	wasoffline = (vd->vdev_offline \|\| vd->vdev_tmpoffline);
	oldstate = vd->vdev_state;

	tvd = vd->vdev_top;
	vd->vdev_offline = B_FALSE;
	vd->vdev_tmpoffline = B_FALSE;
	vd->vdev_checkremove = !!(flags & ZFS_ONLINE_CHECKREMOVE);
	vd->vdev_forcefault = !!(flags & ZFS_ONLINE_FORCEFAULT);

	/* XXX - L2ARC 1.0 does not support expansion */
	if (!vd->vdev_aux) {
	for (pvd = vd; pvd != rvd; pvd = pvd->vdev_parent)
	pvd->vdev_expanding = !!((flags & ZFS_ONLINE_EXPAND) \|\|
	spa->spa_autoexpand);
	vd->vdev_expansion_time = gethrestime_sec();
	}

	vdev_reopen(tvd);
	vd->vdev_checkremove = vd->vdev_forcefault = B_FALSE;

	if (!vd->vdev_aux) {
	for (pvd = vd; pvd != rvd; pvd = pvd->vdev_parent)
	pvd->vdev_expanding = B_FALSE;
	}

	if (newstate)
	*newstate = vd->vdev_state;
	if ((flags & ZFS_ONLINE_UNSPARE) &&
	!vdev_is_dead(vd) && vd->vdev_parent &&
	vd->vdev_parent->vdev_ops == &vdev_spare_ops &&
	vd->vdev_parent->vdev_child[0] == vd)
	vd->vdev_unspare = B_TRUE;

	if ((flags & ZFS_ONLINE_EXPAND) \|\| spa->spa_autoexpand) {

	/* XXX - L2ARC 1.0 does not support expansion */
	if (vd->vdev_aux)
	return (spa_vdev_state_exit(spa, vd, ENOTSUP));
	spa_async_request(spa, SPA_ASYNC_CONFIG_UPDATE);
	}

	/* Restart initializing if necessary */
	mutex_enter(&vd->vdev_initialize_lock);
	if (vdev_writeable(vd) &&
	vd->vdev_initialize_thread == NULL &&
	vd->vdev_initialize_state == VDEV_INITIALIZE_ACTIVE) {
	(void) vdev_initialize(vd);
	}
	mutex_exit(&vd->vdev_initialize_lock);

	/*
	* Restart trimming if necessary. We do not restart trimming for cache
	* devices here. This is triggered by l2arc_rebuild_vdev()
	* asynchronously for the whole device or in l2arc_evict() as it evicts
	* space for upcoming writes.
	*/
	mutex_enter(&vd->vdev_trim_lock);
	if (vdev_writeable(vd) && !vd->vdev_isl2cache &&
	vd->vdev_trim_thread == NULL &&
	vd->vdev_trim_state == VDEV_TRIM_ACTIVE) {
	(void) vdev_trim(vd, vd->vdev_trim_rate, vd->vdev_trim_partial,
	vd->vdev_trim_secure);
	}
	mutex_exit(&vd->vdev_trim_lock);

	if (wasoffline \|\|
	(oldstate < VDEV_STATE_DEGRADED &&
	vd->vdev_state >= VDEV_STATE_DEGRADED))
	spa_event_notify(spa, vd, NULL, ESC_ZFS_VDEV_ONLINE);

	return (spa_vdev_state_exit(spa, vd, 0));
	}

	static int
	vdev_offline_locked(spa_t *spa, uint64_t guid, uint64_t flags)
	{
	vdev_t vd, tvd;
	int error = 0;
	uint64_t generation;
	metaslab_group_t *mg;

	top:
	spa_vdev_state_enter(spa, SCL_ALLOC);

	if ((vd = spa_lookup_by_guid(spa, guid, B_TRUE)) == NULL)
	return (spa_vdev_state_exit(spa, NULL, SET_ERROR(ENODEV)));

	if (!vd->vdev_ops->vdev_op_leaf)
	return (spa_vdev_state_exit(spa, NULL, SET_ERROR(ENOTSUP)));

	tvd = vd->vdev_top;
	mg = tvd->vdev_mg;
	generation = spa->spa_config_generation + 1;

	/*
	* If the device isn't already offline, try to offline it.
	*/
	if (!vd->vdev_offline) {
	/*
	* If this device has the only valid copy of some data,
	* don't allow it to be offlined. Log devices are always
	* expendable.
	*/
	if (!tvd->vdev_islog && vd->vdev_aux == NULL &&
	vdev_dtl_required(vd))
	return (spa_vdev_state_exit(spa, NULL,
	SET_ERROR(EBUSY)));

	/*
	* If the top-level is a slog and it has had allocations
	* then proceed. We check that the vdev's metaslab group
	* is not NULL since it's possible that we may have just
	* added this vdev but not yet initialized its metaslabs.
	*/
	if (tvd->vdev_islog && mg != NULL) {
	/*
	* Prevent any future allocations.
	*/
	metaslab_group_passivate(mg);
	(void) spa_vdev_state_exit(spa, vd, 0);

	error = spa_reset_logs(spa);

	/*
	* If the log device was successfully reset but has
	* checkpointed data, do not offline it.
	*/
	if (error == 0 &&
	tvd->vdev_checkpoint_sm != NULL) {
	ASSERT3U(space_map_allocated(
	tvd->vdev_checkpoint_sm), !=, 0);
	error = ZFS_ERR_CHECKPOINT_EXISTS;
	}

	spa_vdev_state_enter(spa, SCL_ALLOC);

	/*
	* Check to see if the config has changed.
	*/
	if (error \|\| generation != spa->spa_config_generation) {
	metaslab_group_activate(mg);
	if (error)
	return (spa_vdev_state_exit(spa,
	vd, error));
	(void) spa_vdev_state_exit(spa, vd, 0);
	goto top;
	}
	ASSERT0(tvd->vdev_stat.vs_alloc);
	}

	/*
	* Offline this device and reopen its top-level vdev.
	* If the top-level vdev is a log device then just offline
	* it. Otherwise, if this action results in the top-level
	* vdev becoming unusable, undo it and fail the request.
	*/
	vd->vdev_offline = B_TRUE;
	vdev_reopen(tvd);

	if (!tvd->vdev_islog && vd->vdev_aux == NULL &&
	vdev_is_dead(tvd)) {
	vd->vdev_offline = B_FALSE;
	vdev_reopen(tvd);
	return (spa_vdev_state_exit(spa, NULL,
	SET_ERROR(EBUSY)));
	}

	/*
	* Add the device back into the metaslab rotor so that
	* once we online the device it's open for business.
	*/
	if (tvd->vdev_islog && mg != NULL)
	metaslab_group_activate(mg);
	}

	vd->vdev_tmpoffline = !!(flags & ZFS_OFFLINE_TEMPORARY);

	return (spa_vdev_state_exit(spa, vd, 0));
	}

	int
	vdev_offline(spa_t *spa, uint64_t guid, uint64_t flags)
	{
	int error;

	mutex_enter(&spa->spa_vdev_top_lock);
	error = vdev_offline_locked(spa, guid, flags);
	mutex_exit(&spa->spa_vdev_top_lock);

	return (error);
	}

	/*
	* Clear the error counts associated with this vdev. Unlike vdev_online() and
	* vdev_offline(), we assume the spa config is locked. We also clear all
	* children. If 'vd' is NULL, then the user wants to clear all vdevs.
	*/
	void
	vdev_clear(spa_t spa, vdev_t vd)
	{
	vdev_t *rvd = spa->spa_root_vdev;

	ASSERT(spa_config_held(spa, SCL_STATE_ALL, RW_WRITER) == SCL_STATE_ALL);

	if (vd == NULL)
	vd = rvd;

	vd->vdev_stat.vs_read_errors = 0;
	vd->vdev_stat.vs_write_errors = 0;
	vd->vdev_stat.vs_checksum_errors = 0;
	vd->vdev_stat.vs_slow_ios = 0;

	for (int c = 0; c < vd->vdev_children; c++)
	vdev_clear(spa, vd->vdev_child[c]);

	/*
	* It makes no sense to "clear" an indirect vdev.
	*/
	if (!vdev_is_concrete(vd))
	return;

	/*
	* If we're in the FAULTED state or have experienced failed I/O, then
	* clear the persistent state and attempt to reopen the device. We
	* also mark the vdev config dirty, so that the new faulted state is
	* written out to disk.
	*/
	if (vd->vdev_faulted \|\| vd->vdev_degraded \|\|
	!vdev_readable(vd) \|\| !vdev_writeable(vd)) {
	/*
	* When reopening in response to a clear event, it may be due to
	* a fmadm repair request. In this case, if the device is
	* still broken, we want to still post the ereport again.
	*/
	vd->vdev_forcefault = B_TRUE;

	vd->vdev_faulted = vd->vdev_degraded = 0ULL;
	vd->vdev_cant_read = B_FALSE;
	vd->vdev_cant_write = B_FALSE;
	vd->vdev_stat.vs_aux = 0;

	vdev_reopen(vd == rvd ? rvd : vd->vdev_top);

	vd->vdev_forcefault = B_FALSE;

	if (vd != rvd && vdev_writeable(vd->vdev_top))
	vdev_state_dirty(vd->vdev_top);

	/* If a resilver isn't required, check if vdevs can be culled */
	if (vd->vdev_aux == NULL && !vdev_is_dead(vd) &&
	!dsl_scan_resilvering(spa->spa_dsl_pool) &&
	!dsl_scan_resilver_scheduled(spa->spa_dsl_pool))
	spa_async_request(spa, SPA_ASYNC_RESILVER_DONE);

	spa_event_notify(spa, vd, NULL, ESC_ZFS_VDEV_CLEAR);
	}

	/*
	* When clearing a FMA-diagnosed fault, we always want to
	* unspare the device, as we assume that the original spare was
	* done in response to the FMA fault.
	*/
	if (!vdev_is_dead(vd) && vd->vdev_parent != NULL &&
	vd->vdev_parent->vdev_ops == &vdev_spare_ops &&
	vd->vdev_parent->vdev_child[0] == vd)
	vd->vdev_unspare = B_TRUE;
	}

	boolean_t
	vdev_is_dead(vdev_t *vd)
	{
	/*
	* Holes and missing devices are always considered "dead".
	* This simplifies the code since we don't have to check for
	* these types of devices in the various code paths.
	* Instead we rely on the fact that we skip over dead devices
	* before issuing I/O to them.
	*/
	return (vd->vdev_state < VDEV_STATE_DEGRADED \|\|
	vd->vdev_ops == &vdev_hole_ops \|\|
	vd->vdev_ops == &vdev_missing_ops);
	}

	boolean_t
	vdev_readable(vdev_t *vd)
	{
	return (!vdev_is_dead(vd) && !vd->vdev_cant_read);
	}

	boolean_t
	vdev_writeable(vdev_t *vd)
	{
	return (!vdev_is_dead(vd) && !vd->vdev_cant_write &&
	vdev_is_concrete(vd));
	}

	boolean_t
	vdev_allocatable(vdev_t *vd)
	{
	uint64_t state = vd->vdev_state;

	/*
	* We currently allow allocations from vdevs which may be in the
	* process of reopening (i.e. VDEV_STATE_CLOSED). If the device
	* fails to reopen then we'll catch it later when we're holding
	* the proper locks. Note that we have to get the vdev state
	* in a local variable because although it changes atomically,
	* we're asking two separate questions about it.
	*/
	return (!(state < VDEV_STATE_DEGRADED && state != VDEV_STATE_CLOSED) &&
	!vd->vdev_cant_write && vdev_is_concrete(vd) &&
	vd->vdev_mg->mg_initialized);
	}

	boolean_t
	vdev_accessible(vdev_t vd, zio_t zio)
	{
	ASSERT(zio->io_vd == vd);

	if (vdev_is_dead(vd) \|\| vd->vdev_remove_wanted)
	return (B_FALSE);

	if (zio->io_type == ZIO_TYPE_READ)
	return (!vd->vdev_cant_read);

	if (zio->io_type == ZIO_TYPE_WRITE)
	return (!vd->vdev_cant_write);

	return (B_TRUE);
	}

	static void
	vdev_get_child_stat(vdev_t cvd, vdev_stat_t vs, vdev_stat_t *cvs)
	{
	for (int t = 0; t < VS_ZIO_TYPES; t++) {
	vs->vs_ops[t] += cvs->vs_ops[t];
	vs->vs_bytes[t] += cvs->vs_bytes[t];
	}

	cvs->vs_scan_removing = cvd->vdev_removing;
	}

	/*
	* Get extended stats
	*/
	static void
	vdev_get_child_stat_ex(vdev_t cvd, vdev_stat_ex_t vsx, vdev_stat_ex_t *cvsx)
	{
	int t, b;
	for (t = 0; t < ZIO_TYPES; t++) {
	for (b = 0; b < ARRAY_SIZE(vsx->vsx_disk_histo[0]); b++)
	vsx->vsx_disk_histo[t][b] += cvsx->vsx_disk_histo[t][b];

	for (b = 0; b < ARRAY_SIZE(vsx->vsx_total_histo[0]); b++) {
	vsx->vsx_total_histo[t][b] +=
	cvsx->vsx_total_histo[t][b];
	}
	}

	for (t = 0; t < ZIO_PRIORITY_NUM_QUEUEABLE; t++) {
	for (b = 0; b < ARRAY_SIZE(vsx->vsx_queue_histo[0]); b++) {
	vsx->vsx_queue_histo[t][b] +=
	cvsx->vsx_queue_histo[t][b];
	}
	vsx->vsx_active_queue[t] += cvsx->vsx_active_queue[t];
	vsx->vsx_pend_queue[t] += cvsx->vsx_pend_queue[t];

	for (b = 0; b < ARRAY_SIZE(vsx->vsx_ind_histo[0]); b++)
	vsx->vsx_ind_histo[t][b] += cvsx->vsx_ind_histo[t][b];

	for (b = 0; b < ARRAY_SIZE(vsx->vsx_agg_histo[0]); b++)
	vsx->vsx_agg_histo[t][b] += cvsx->vsx_agg_histo[t][b];
	}

	}

	boolean_t
	vdev_is_spacemap_addressable(vdev_t *vd)
	{
	if (spa_feature_is_active(vd->vdev_spa, SPA_FEATURE_SPACEMAP_V2))
	return (B_TRUE);

	/*
	* If double-word space map entries are not enabled we assume
	* 47 bits of the space map entry are dedicated to the entry's
	* offset (see SM_OFFSET_BITS in space_map.h). We then use that
	* to calculate the maximum address that can be described by a
	* space map entry for the given device.
	*/
	uint64_t shift = vd->vdev_ashift + SM_OFFSET_BITS;

	if (shift >= 63) /* detect potential overflow */
	return (B_TRUE);

	return (vd->vdev_asize < (1ULL << shift));
	}

	/*
	* Get statistics for the given vdev.
	*/
	static void
	vdev_get_stats_ex_impl(vdev_t vd, vdev_stat_t vs, vdev_stat_ex_t *vsx)
	{
	int t;
	/*
	* If we're getting stats on the root vdev, aggregate the I/O counts
	* over all top-level vdevs (i.e. the direct children of the root).
	*/
	if (!vd->vdev_ops->vdev_op_leaf) {
	if (vs) {
	memset(vs->vs_ops, 0, sizeof (vs->vs_ops));
	memset(vs->vs_bytes, 0, sizeof (vs->vs_bytes));
	}
	if (vsx)
	memset(vsx, 0, sizeof (*vsx));

	for (int c = 0; c < vd->vdev_children; c++) {
	vdev_t *cvd = vd->vdev_child[c];
	vdev_stat_t *cvs = &cvd->vdev_stat;
	vdev_stat_ex_t *cvsx = &cvd->vdev_stat_ex;

	vdev_get_stats_ex_impl(cvd, cvs, cvsx);
	if (vs)
	vdev_get_child_stat(cvd, vs, cvs);
	if (vsx)
	vdev_get_child_stat_ex(cvd, vsx, cvsx);

	}
	} else {
	/*
	* We're a leaf. Just copy our ZIO active queue stats in. The
	* other leaf stats are updated in vdev_stat_update().
	*/
	if (!vsx)
	return;

	memcpy(vsx, &vd->vdev_stat_ex, sizeof (vd->vdev_stat_ex));

	for (t = 0; t < ARRAY_SIZE(vd->vdev_queue.vq_class); t++) {
	vsx->vsx_active_queue[t] =
	vd->vdev_queue.vq_class[t].vqc_active;
	vsx->vsx_pend_queue[t] = avl_numnodes(
	&vd->vdev_queue.vq_class[t].vqc_queued_tree);
	}
	}
	}

	void
	vdev_get_stats_ex(vdev_t vd, vdev_stat_t vs, vdev_stat_ex_t *vsx)
	{
	vdev_t *tvd = vd->vdev_top;
	mutex_enter(&vd->vdev_stat_lock);
	if (vs) {
	bcopy(&vd->vdev_stat, vs, sizeof (*vs));
	vs->vs_timestamp = gethrtime() - vs->vs_timestamp;
	vs->vs_state = vd->vdev_state;
	vs->vs_rsize = vdev_get_min_asize(vd);

	if (vd->vdev_ops->vdev_op_leaf) {
	vs->vs_rsize += VDEV_LABEL_START_SIZE +
	VDEV_LABEL_END_SIZE;
	/*
	* Report initializing progress. Since we don't
	* have the initializing locks held, this is only
	* an estimate (although a fairly accurate one).
	*/
	vs->vs_initialize_bytes_done =
	vd->vdev_initialize_bytes_done;
	vs->vs_initialize_bytes_est =
	vd->vdev_initialize_bytes_est;
	vs->vs_initialize_state = vd->vdev_initialize_state;
	vs->vs_initialize_action_time =
	vd->vdev_initialize_action_time;

	/*
	* Report manual TRIM progress. Since we don't have
	* the manual TRIM locks held, this is only an
	* estimate (although fairly accurate one).
	*/
	vs->vs_trim_notsup = !vd->vdev_has_trim;
	vs->vs_trim_bytes_done = vd->vdev_trim_bytes_done;
	vs->vs_trim_bytes_est = vd->vdev_trim_bytes_est;
	vs->vs_trim_state = vd->vdev_trim_state;
	vs->vs_trim_action_time = vd->vdev_trim_action_time;

	/* Set when there is a deferred resilver. */
	vs->vs_resilver_deferred = vd->vdev_resilver_deferred;
	}

	/*
	* Report expandable space on top-level, non-auxiliary devices
	* only. The expandable space is reported in terms of metaslab
	* sized units since that determines how much space the pool
	* can expand.
	*/
	if (vd->vdev_aux == NULL && tvd != NULL) {
	vs->vs_esize = P2ALIGN(
	vd->vdev_max_asize - vd->vdev_asize,
	1ULL << tvd->vdev_ms_shift);
	}

	vs->vs_configured_ashift = vd->vdev_top != NULL
	? vd->vdev_top->vdev_ashift : vd->vdev_ashift;
	vs->vs_logical_ashift = vd->vdev_logical_ashift;
	vs->vs_physical_ashift = vd->vdev_physical_ashift;

	/*
	* Report fragmentation and rebuild progress for top-level,
	* non-auxiliary, concrete devices.
	*/
	if (vd->vdev_aux == NULL && vd == vd->vdev_top &&
	vdev_is_concrete(vd)) {
	vs->vs_fragmentation = (vd->vdev_mg != NULL) ?
	vd->vdev_mg->mg_fragmentation : 0;
	}
	}

	vdev_get_stats_ex_impl(vd, vs, vsx);
	mutex_exit(&vd->vdev_stat_lock);
	}

	void
	vdev_get_stats(vdev_t vd, vdev_stat_t vs)
	{
	return (vdev_get_stats_ex(vd, vs, NULL));
	}

	void
	vdev_clear_stats(vdev_t *vd)
	{
	mutex_enter(&vd->vdev_stat_lock);
	vd->vdev_stat.vs_space = 0;
	vd->vdev_stat.vs_dspace = 0;
	vd->vdev_stat.vs_alloc = 0;
	mutex_exit(&vd->vdev_stat_lock);
	}

	void
	vdev_scan_stat_init(vdev_t *vd)
	{
	vdev_stat_t *vs = &vd->vdev_stat;

	for (int c = 0; c < vd->vdev_children; c++)
	vdev_scan_stat_init(vd->vdev_child[c]);

	mutex_enter(&vd->vdev_stat_lock);
	vs->vs_scan_processed = 0;
	mutex_exit(&vd->vdev_stat_lock);
	}

	void
	vdev_stat_update(zio_t *zio, uint64_t psize)
	{
	spa_t *spa = zio->io_spa;
	vdev_t *rvd = spa->spa_root_vdev;
	vdev_t *vd = zio->io_vd ? zio->io_vd : rvd;
	vdev_t *pvd;
	uint64_t txg = zio->io_txg;
	vdev_stat_t *vs = &vd->vdev_stat;
	vdev_stat_ex_t *vsx = &vd->vdev_stat_ex;
	zio_type_t type = zio->io_type;
	int flags = zio->io_flags;

	/*
	* If this i/o is a gang leader, it didn't do any actual work.
	*/
	if (zio->io_gang_tree)
	return;

	if (zio->io_error == 0) {
	/*
	* If this is a root i/o, don't count it -- we've already
	* counted the top-level vdevs, and vdev_get_stats() will
	* aggregate them when asked. This reduces contention on
	* the root vdev_stat_lock and implicitly handles blocks
	* that compress away to holes, for which there is no i/o.
	* (Holes never create vdev children, so all the counters
	* remain zero, which is what we want.)
	*
	* Note: this only applies to successful i/o (io_error == 0)
	* because unlike i/o counts, errors are not additive.
	* When reading a ditto block, for example, failure of
	* one top-level vdev does not imply a root-level error.
	*/
	if (vd == rvd)
	return;

	ASSERT(vd == zio->io_vd);

	if (flags & ZIO_FLAG_IO_BYPASS)
	return;

	mutex_enter(&vd->vdev_stat_lock);

	if (flags & ZIO_FLAG_IO_REPAIR) {
	/*
	* Repair is the result of a resilver issued by the
	* scan thread (spa_sync).
	*/
	if (flags & ZIO_FLAG_SCAN_THREAD) {
	dsl_scan_t *scn = spa->spa_dsl_pool->dp_scan;
	dsl_scan_phys_t *scn_phys = &scn->scn_phys;
	uint64_t *processed = &scn_phys->scn_processed;

	if (vd->vdev_ops->vdev_op_leaf)
	atomic_add_64(processed, psize);
	vs->vs_scan_processed += psize;
	}

	/*
	* Repair is the result of a rebuild issued by the
	* rebuild thread (vdev_rebuild_thread).
	*/
	if (zio->io_priority == ZIO_PRIORITY_REBUILD) {
	vdev_t *tvd = vd->vdev_top;
	vdev_rebuild_t *vr = &tvd->vdev_rebuild_config;
	vdev_rebuild_phys_t *vrp = &vr->vr_rebuild_phys;
	uint64_t *rebuilt = &vrp->vrp_bytes_rebuilt;

	if (vd->vdev_ops->vdev_op_leaf)
	atomic_add_64(rebuilt, psize);
	vs->vs_rebuild_processed += psize;
	}

	if (flags & ZIO_FLAG_SELF_HEAL)
	vs->vs_self_healed += psize;
	}

	/*
	* The bytes/ops/histograms are recorded at the leaf level and
	* aggregated into the higher level vdevs in vdev_get_stats().
	*/
	if (vd->vdev_ops->vdev_op_leaf &&
	(zio->io_priority < ZIO_PRIORITY_NUM_QUEUEABLE)) {
	zio_type_t vs_type = type;
	zio_priority_t priority = zio->io_priority;

	/*
	* TRIM ops and bytes are reported to user space as
	* ZIO_TYPE_IOCTL. This is done to preserve the
	* vdev_stat_t structure layout for user space.
	*/
	if (type == ZIO_TYPE_TRIM)
	vs_type = ZIO_TYPE_IOCTL;

	/*
	* Solely for the purposes of 'zpool iostat -lqrw'
	* reporting use the priority to catagorize the IO.
	* Only the following are reported to user space:
	*
	* ZIO_PRIORITY_SYNC_READ,
	* ZIO_PRIORITY_SYNC_WRITE,
	* ZIO_PRIORITY_ASYNC_READ,
	* ZIO_PRIORITY_ASYNC_WRITE,
	* ZIO_PRIORITY_SCRUB,
	* ZIO_PRIORITY_TRIM.
	*/
	if (priority == ZIO_PRIORITY_REBUILD) {
	priority = ((type == ZIO_TYPE_WRITE) ?
	ZIO_PRIORITY_ASYNC_WRITE :
	ZIO_PRIORITY_SCRUB);
	} else if (priority == ZIO_PRIORITY_INITIALIZING) {
	ASSERT3U(type, ==, ZIO_TYPE_WRITE);
	priority = ZIO_PRIORITY_ASYNC_WRITE;
	} else if (priority == ZIO_PRIORITY_REMOVAL) {
	priority = ((type == ZIO_TYPE_WRITE) ?
	ZIO_PRIORITY_ASYNC_WRITE :
	ZIO_PRIORITY_ASYNC_READ);
	}

	vs->vs_ops[vs_type]++;
	vs->vs_bytes[vs_type] += psize;

	if (flags & ZIO_FLAG_DELEGATED) {
	vsx->vsx_agg_histo[priority]
	[RQ_HISTO(zio->io_size)]++;
	} else {
	vsx->vsx_ind_histo[priority]
	[RQ_HISTO(zio->io_size)]++;
	}

	if (zio->io_delta && zio->io_delay) {
	vsx->vsx_queue_histo[priority]
	[L_HISTO(zio->io_delta - zio->io_delay)]++;
	vsx->vsx_disk_histo[type]
	[L_HISTO(zio->io_delay)]++;
	vsx->vsx_total_histo[type]
	[L_HISTO(zio->io_delta)]++;
	}
	}

	mutex_exit(&vd->vdev_stat_lock);
	return;
	}

	if (flags & ZIO_FLAG_SPECULATIVE)
	return;

	/*
	* If this is an I/O error that is going to be retried, then ignore the
	* error. Otherwise, the user may interpret B_FAILFAST I/O errors as
	* hard errors, when in reality they can happen for any number of
	* innocuous reasons (bus resets, MPxIO link failure, etc).
	*/
	if (zio->io_error == EIO &&
	!(zio->io_flags & ZIO_FLAG_IO_RETRY))
	return;

	/*
	* Intent logs writes won't propagate their error to the root
	* I/O so don't mark these types of failures as pool-level
	* errors.
	*/
	if (zio->io_vd == NULL && (zio->io_flags & ZIO_FLAG_DONT_PROPAGATE))
	return;

	if (spa->spa_load_state == SPA_LOAD_NONE &&
	type == ZIO_TYPE_WRITE && txg != 0 &&
	(!(flags & ZIO_FLAG_IO_REPAIR) \|\|
	(flags & ZIO_FLAG_SCAN_THREAD) \|\|
	spa->spa_claiming)) {
	/*
	* This is either a normal write (not a repair), or it's
	* a repair induced by the scrub thread, or it's a repair
	* made by zil_claim() during spa_load() in the first txg.
	* In the normal case, we commit the DTL change in the same
	* txg as the block was born. In the scrub-induced repair
	* case, we know that scrubs run in first-pass syncing context,
	* so we commit the DTL change in spa_syncing_txg(spa).
	* In the zil_claim() case, we commit in spa_first_txg(spa).
	*
	* We currently do not make DTL entries for failed spontaneous
	* self-healing writes triggered by normal (non-scrubbing)
	* reads, because we have no transactional context in which to
	* do so -- and it's not clear that it'd be desirable anyway.
	*/
	if (vd->vdev_ops->vdev_op_leaf) {
	uint64_t commit_txg = txg;
	if (flags & ZIO_FLAG_SCAN_THREAD) {
	ASSERT(flags & ZIO_FLAG_IO_REPAIR);
	ASSERT(spa_sync_pass(spa) == 1);
	vdev_dtl_dirty(vd, DTL_SCRUB, txg, 1);
	commit_txg = spa_syncing_txg(spa);
	} else if (spa->spa_claiming) {
	ASSERT(flags & ZIO_FLAG_IO_REPAIR);
	commit_txg = spa_first_txg(spa);
	}
	ASSERT(commit_txg >= spa_syncing_txg(spa));
	if (vdev_dtl_contains(vd, DTL_MISSING, txg, 1))
	return;
	for (pvd = vd; pvd != rvd; pvd = pvd->vdev_parent)
	vdev_dtl_dirty(pvd, DTL_PARTIAL, txg, 1);
	vdev_dirty(vd->vdev_top, VDD_DTL, vd, commit_txg);
	}
	if (vd != rvd)
	vdev_dtl_dirty(vd, DTL_MISSING, txg, 1);
	}
	}

	int64_t
	vdev_deflated_space(vdev_t *vd, int64_t space)
	{
	ASSERT((space & (SPA_MINBLOCKSIZE-1)) == 0);
	ASSERT(vd->vdev_deflate_ratio != 0 \|\| vd->vdev_isl2cache);

	return ((space >> SPA_MINBLOCKSHIFT) * vd->vdev_deflate_ratio);
	}

	/*
	* Update the in-core space usage stats for this vdev, its metaslab class,
	* and the root vdev.
	*/
	void
	vdev_space_update(vdev_t *vd, int64_t alloc_delta, int64_t defer_delta,
	int64_t space_delta)
	{
	int64_t dspace_delta;
	spa_t *spa = vd->vdev_spa;
	vdev_t *rvd = spa->spa_root_vdev;

	ASSERT(vd == vd->vdev_top);

	/*
	* Apply the inverse of the psize-to-asize (ie. RAID-Z) space-expansion
	* factor. We must calculate this here and not at the root vdev
	* because the root vdev's psize-to-asize is simply the max of its
	* children's, thus not accurate enough for us.
	*/
	dspace_delta = vdev_deflated_space(vd, space_delta);

	mutex_enter(&vd->vdev_stat_lock);
	/* ensure we won't underflow */
	if (alloc_delta < 0) {
	ASSERT3U(vd->vdev_stat.vs_alloc, >=, -alloc_delta);
	}

	vd->vdev_stat.vs_alloc += alloc_delta;
	vd->vdev_stat.vs_space += space_delta;
	vd->vdev_stat.vs_dspace += dspace_delta;
	mutex_exit(&vd->vdev_stat_lock);

	/* every class but log contributes to root space stats */
	if (vd->vdev_mg != NULL && !vd->vdev_islog) {
	ASSERT(!vd->vdev_isl2cache);
	mutex_enter(&rvd->vdev_stat_lock);
	rvd->vdev_stat.vs_alloc += alloc_delta;
	rvd->vdev_stat.vs_space += space_delta;
	rvd->vdev_stat.vs_dspace += dspace_delta;
	mutex_exit(&rvd->vdev_stat_lock);
	}
	/* Note: metaslab_class_space_update moved to metaslab_space_update */
	}

	/*
	* Mark a top-level vdev's config as dirty, placing it on the dirty list
	* so that it will be written out next time the vdev configuration is synced.
	* If the root vdev is specified (vdev_top == NULL), dirty all top-level vdevs.
	*/
	void
	vdev_config_dirty(vdev_t *vd)
	{
	spa_t *spa = vd->vdev_spa;
	vdev_t *rvd = spa->spa_root_vdev;
	int c;

	ASSERT(spa_writeable(spa));

	/*
	* If this is an aux vdev (as with l2cache and spare devices), then we
	* update the vdev config manually and set the sync flag.
	*/
	if (vd->vdev_aux != NULL) {
	spa_aux_vdev_t *sav = vd->vdev_aux;
	nvlist_t **aux;
	uint_t naux;

	for (c = 0; c < sav->sav_count; c++) {
	if (sav->sav_vdevs[c] == vd)
	break;
	}

	if (c == sav->sav_count) {
	/*
	* We're being removed. There's nothing more to do.
	*/
	ASSERT(sav->sav_sync == B_TRUE);
	return;
	}

	sav->sav_sync = B_TRUE;

	if (nvlist_lookup_nvlist_array(sav->sav_config,
	ZPOOL_CONFIG_L2CACHE, &aux, &naux) != 0) {
	VERIFY(nvlist_lookup_nvlist_array(sav->sav_config,
	ZPOOL_CONFIG_SPARES, &aux, &naux) == 0);
	}

	ASSERT(c < naux);

	/*
	* Setting the nvlist in the middle if the array is a little
	* sketchy, but it will work.
	*/
	nvlist_free(aux[c]);
	aux[c] = vdev_config_generate(spa, vd, B_TRUE, 0);

	return;
	}

	/*
	* The dirty list is protected by the SCL_CONFIG lock. The caller
	* must either hold SCL_CONFIG as writer, or must be the sync thread
	* (which holds SCL_CONFIG as reader). There's only one sync thread,
	* so this is sufficient to ensure mutual exclusion.
	*/
	ASSERT(spa_config_held(spa, SCL_CONFIG, RW_WRITER) \|\|
	(dsl_pool_sync_context(spa_get_dsl(spa)) &&
	spa_config_held(spa, SCL_CONFIG, RW_READER)));

	if (vd == rvd) {
	for (c = 0; c < rvd->vdev_children; c++)
	vdev_config_dirty(rvd->vdev_child[c]);
	} else {
	ASSERT(vd == vd->vdev_top);

	if (!list_link_active(&vd->vdev_config_dirty_node) &&
	vdev_is_concrete(vd)) {
	list_insert_head(&spa->spa_config_dirty_list, vd);
	}
	}
	}

	void
	vdev_config_clean(vdev_t *vd)
	{
	spa_t *spa = vd->vdev_spa;

	ASSERT(spa_config_held(spa, SCL_CONFIG, RW_WRITER) \|\|
	(dsl_pool_sync_context(spa_get_dsl(spa)) &&
	spa_config_held(spa, SCL_CONFIG, RW_READER)));

	ASSERT(list_link_active(&vd->vdev_config_dirty_node));
	list_remove(&spa->spa_config_dirty_list, vd);
	}

	/*
	* Mark a top-level vdev's state as dirty, so that the next pass of
	* spa_sync() can convert this into vdev_config_dirty(). We distinguish
	* the state changes from larger config changes because they require
	* much less locking, and are often needed for administrative actions.
	*/
	void
	vdev_state_dirty(vdev_t *vd)
	{
	spa_t *spa = vd->vdev_spa;

	ASSERT(spa_writeable(spa));
	ASSERT(vd == vd->vdev_top);

	/*
	* The state list is protected by the SCL_STATE lock. The caller
	* must either hold SCL_STATE as writer, or must be the sync thread
	* (which holds SCL_STATE as reader). There's only one sync thread,
	* so this is sufficient to ensure mutual exclusion.
	*/
	ASSERT(spa_config_held(spa, SCL_STATE, RW_WRITER) \|\|
	(dsl_pool_sync_context(spa_get_dsl(spa)) &&
	spa_config_held(spa, SCL_STATE, RW_READER)));

	if (!list_link_active(&vd->vdev_state_dirty_node) &&
	vdev_is_concrete(vd))
	list_insert_head(&spa->spa_state_dirty_list, vd);
	}

	void
	vdev_state_clean(vdev_t *vd)
	{
	spa_t *spa = vd->vdev_spa;

	ASSERT(spa_config_held(spa, SCL_STATE, RW_WRITER) \|\|
	(dsl_pool_sync_context(spa_get_dsl(spa)) &&
	spa_config_held(spa, SCL_STATE, RW_READER)));

	ASSERT(list_link_active(&vd->vdev_state_dirty_node));
	list_remove(&spa->spa_state_dirty_list, vd);
	}

	/*
	* Propagate vdev state up from children to parent.
	*/
	void
	vdev_propagate_state(vdev_t *vd)
	{
	spa_t *spa = vd->vdev_spa;
	vdev_t *rvd = spa->spa_root_vdev;
	int degraded = 0, faulted = 0;
	int corrupted = 0;
	vdev_t *child;

	if (vd->vdev_children > 0) {
	for (int c = 0; c < vd->vdev_children; c++) {
	child = vd->vdev_child[c];

	/*
	* Don't factor holes or indirect vdevs into the
	* decision.
	*/
	if (!vdev_is_concrete(child))
	continue;

	if (!vdev_readable(child) \|\|
	(!vdev_writeable(child) && spa_writeable(spa))) {
	/*
	* Root special: if there is a top-level log
	* device, treat the root vdev as if it were
	* degraded.
	*/
	if (child->vdev_islog && vd == rvd)
	degraded++;
	else
	faulted++;
	} else if (child->vdev_state <= VDEV_STATE_DEGRADED) {
	degraded++;
	}

	if (child->vdev_stat.vs_aux == VDEV_AUX_CORRUPT_DATA)
	corrupted++;
	}

	vd->vdev_ops->vdev_op_state_change(vd, faulted, degraded);

	/*
	* Root special: if there is a top-level vdev that cannot be
	* opened due to corrupted metadata, then propagate the root
	* vdev's aux state as 'corrupt' rather than 'insufficient
	* replicas'.
	*/
	if (corrupted && vd == rvd &&
	rvd->vdev_state == VDEV_STATE_CANT_OPEN)
	vdev_set_state(rvd, B_FALSE, VDEV_STATE_CANT_OPEN,
	VDEV_AUX_CORRUPT_DATA);
	}

	if (vd->vdev_parent)
	vdev_propagate_state(vd->vdev_parent);
	}

	/*
	* Set a vdev's state. If this is during an open, we don't update the parent
	* state, because we're in the process of opening children depth-first.
	* Otherwise, we propagate the change to the parent.
	*
	* If this routine places a device in a faulted state, an appropriate ereport is
	* generated.
	*/
	void
	vdev_set_state(vdev_t *vd, boolean_t isopen, vdev_state_t state, vdev_aux_t aux)
	{
	uint64_t save_state;
	spa_t *spa = vd->vdev_spa;

	if (state == vd->vdev_state) {
	/*
	* Since vdev_offline() code path is already in an offline
	* state we can miss a statechange event to OFFLINE. Check
	* the previous state to catch this condition.
	*/
	if (vd->vdev_ops->vdev_op_leaf &&
	(state == VDEV_STATE_OFFLINE) &&
	(vd->vdev_prevstate >= VDEV_STATE_FAULTED)) {
	/* post an offline state change */
	zfs_post_state_change(spa, vd, vd->vdev_prevstate);
	}
	vd->vdev_stat.vs_aux = aux;
	return;
	}

	save_state = vd->vdev_state;

	vd->vdev_state = state;
	vd->vdev_stat.vs_aux = aux;

	/*
	* If we are setting the vdev state to anything but an open state, then
	* always close the underlying device unless the device has requested
	* a delayed close (i.e. we're about to remove or fault the device).
	* Otherwise, we keep accessible but invalid devices open forever.
	* We don't call vdev_close() itself, because that implies some extra
	* checks (offline, etc) that we don't want here. This is limited to
	* leaf devices, because otherwise closing the device will affect other
	* children.
	*/
	if (!vd->vdev_delayed_close && vdev_is_dead(vd) &&
	vd->vdev_ops->vdev_op_leaf)
	vd->vdev_ops->vdev_op_close(vd);

	if (vd->vdev_removed &&
	state == VDEV_STATE_CANT_OPEN &&
	(aux == VDEV_AUX_OPEN_FAILED \|\| vd->vdev_checkremove)) {
	/*
	* If the previous state is set to VDEV_STATE_REMOVED, then this
	* device was previously marked removed and someone attempted to
	* reopen it. If this failed due to a nonexistent device, then
	* keep the device in the REMOVED state. We also let this be if
	* it is one of our special test online cases, which is only
	* attempting to online the device and shouldn't generate an FMA
	* fault.
	*/
	vd->vdev_state = VDEV_STATE_REMOVED;
	vd->vdev_stat.vs_aux = VDEV_AUX_NONE;
	} else if (state == VDEV_STATE_REMOVED) {
	vd->vdev_removed = B_TRUE;
	} else if (state == VDEV_STATE_CANT_OPEN) {
	/*
	* If we fail to open a vdev during an import or recovery, we
	* mark it as "not available", which signifies that it was
	* never there to begin with. Failure to open such a device
	* is not considered an error.
	*/
	if ((spa_load_state(spa) == SPA_LOAD_IMPORT \|\|
	spa_load_state(spa) == SPA_LOAD_RECOVER) &&
	vd->vdev_ops->vdev_op_leaf)
	vd->vdev_not_present = 1;

	/*
	* Post the appropriate ereport. If the 'prevstate' field is
	* set to something other than VDEV_STATE_UNKNOWN, it indicates
	* that this is part of a vdev_reopen(). In this case, we don't
	* want to post the ereport if the device was already in the
	* CANT_OPEN state beforehand.
	*
	* If the 'checkremove' flag is set, then this is an attempt to
	* online the device in response to an insertion event. If we
	* hit this case, then we have detected an insertion event for a
	* faulted or offline device that wasn't in the removed state.
	* In this scenario, we don't post an ereport because we are
	* about to replace the device, or attempt an online with
	* vdev_forcefault, which will generate the fault for us.
	*/
	if ((vd->vdev_prevstate != state \|\| vd->vdev_forcefault) &&
	!vd->vdev_not_present && !vd->vdev_checkremove &&
	vd != spa->spa_root_vdev) {
	const char *class;

	switch (aux) {
	case VDEV_AUX_OPEN_FAILED:
	class = FM_EREPORT_ZFS_DEVICE_OPEN_FAILED;
	break;
	case VDEV_AUX_CORRUPT_DATA:
	class = FM_EREPORT_ZFS_DEVICE_CORRUPT_DATA;
	break;
	case VDEV_AUX_NO_REPLICAS:
	class = FM_EREPORT_ZFS_DEVICE_NO_REPLICAS;
	break;
	case VDEV_AUX_BAD_GUID_SUM:
	class = FM_EREPORT_ZFS_DEVICE_BAD_GUID_SUM;
	break;
	case VDEV_AUX_TOO_SMALL:
	class = FM_EREPORT_ZFS_DEVICE_TOO_SMALL;
	break;
	case VDEV_AUX_BAD_LABEL:
	class = FM_EREPORT_ZFS_DEVICE_BAD_LABEL;
	break;
	case VDEV_AUX_BAD_ASHIFT:
	class = FM_EREPORT_ZFS_DEVICE_BAD_ASHIFT;
	break;
	default:
	class = FM_EREPORT_ZFS_DEVICE_UNKNOWN;
	}

	(void) zfs_ereport_post(class, spa, vd, NULL, NULL,
	save_state);
	}

	/* Erase any notion of persistent removed state */
	vd->vdev_removed = B_FALSE;
	} else {
	vd->vdev_removed = B_FALSE;
	}

	/*
	* Notify ZED of any significant state-change on a leaf vdev.
	*
	*/
	if (vd->vdev_ops->vdev_op_leaf) {
	/* preserve original state from a vdev_reopen() */
	if ((vd->vdev_prevstate != VDEV_STATE_UNKNOWN) &&
	(vd->vdev_prevstate != vd->vdev_state) &&
	(save_state <= VDEV_STATE_CLOSED))
	save_state = vd->vdev_prevstate;

	/* filter out state change due to initial vdev_open */
	if (save_state > VDEV_STATE_CLOSED)
	zfs_post_state_change(spa, vd, save_state);
	}

	if (!isopen && vd->vdev_parent)
	vdev_propagate_state(vd->vdev_parent);
	}

	boolean_t
	vdev_children_are_offline(vdev_t *vd)
	{
	ASSERT(!vd->vdev_ops->vdev_op_leaf);

	for (uint64_t i = 0; i < vd->vdev_children; i++) {
	if (vd->vdev_child[i]->vdev_state != VDEV_STATE_OFFLINE)
	return (B_FALSE);
	}

	return (B_TRUE);
	}

	/*
	* Check the vdev configuration to ensure that it's capable of supporting
	* a root pool. We do not support partial configuration.
	*/
	boolean_t
	vdev_is_bootable(vdev_t *vd)
	{
	if (!vd->vdev_ops->vdev_op_leaf) {
	const char *vdev_type = vd->vdev_ops->vdev_op_type;

	if (strcmp(vdev_type, VDEV_TYPE_MISSING) == 0 \|\|
	strcmp(vdev_type, VDEV_TYPE_INDIRECT) == 0) {
	return (B_FALSE);
	}
	}

	for (int c = 0; c < vd->vdev_children; c++) {
	if (!vdev_is_bootable(vd->vdev_child[c]))
	return (B_FALSE);
	}
	return (B_TRUE);
	}

	boolean_t
	vdev_is_concrete(vdev_t *vd)
	{
	vdev_ops_t *ops = vd->vdev_ops;
	if (ops == &vdev_indirect_ops \|\| ops == &vdev_hole_ops \|\|
	ops == &vdev_missing_ops \|\| ops == &vdev_root_ops) {
	return (B_FALSE);
	} else {
	return (B_TRUE);
	}
	}

	/*
	* Determine if a log device has valid content. If the vdev was
	* removed or faulted in the MOS config then we know that
	* the content on the log device has already been written to the pool.
	*/
	boolean_t
	vdev_log_state_valid(vdev_t *vd)
	{
	if (vd->vdev_ops->vdev_op_leaf && !vd->vdev_faulted &&
	!vd->vdev_removed)
	return (B_TRUE);

	for (int c = 0; c < vd->vdev_children; c++)
	if (vdev_log_state_valid(vd->vdev_child[c]))
	return (B_TRUE);

	return (B_FALSE);
	}

	/*
	* Expand a vdev if possible.
	*/
	void
	vdev_expand(vdev_t *vd, uint64_t txg)
	{
	ASSERT(vd->vdev_top == vd);
	ASSERT(spa_config_held(vd->vdev_spa, SCL_ALL, RW_WRITER) == SCL_ALL);
	ASSERT(vdev_is_concrete(vd));

	vdev_set_deflate_ratio(vd);

	if ((vd->vdev_asize >> vd->vdev_ms_shift) > vd->vdev_ms_count &&
	vdev_is_concrete(vd)) {
	vdev_metaslab_group_create(vd);
	VERIFY(vdev_metaslab_init(vd, txg) == 0);
	vdev_config_dirty(vd);
	}
	}

	/*
	* Split a vdev.
	*/
	void
	vdev_split(vdev_t *vd)
	{
	vdev_t cvd, pvd = vd->vdev_parent;

	vdev_remove_child(pvd, vd);
	vdev_compact_children(pvd);

	cvd = pvd->vdev_child[0];
	if (pvd->vdev_children == 1) {
	vdev_remove_parent(cvd);
	cvd->vdev_splitting = B_TRUE;
	}
	vdev_propagate_state(cvd);
	}

	void
	vdev_deadman(vdev_t vd, char tag)
	{
	for (int c = 0; c < vd->vdev_children; c++) {
	vdev_t *cvd = vd->vdev_child[c];

	vdev_deadman(cvd, tag);
	}

	if (vd->vdev_ops->vdev_op_leaf) {
	vdev_queue_t *vq = &vd->vdev_queue;

	mutex_enter(&vq->vq_lock);
	if (avl_numnodes(&vq->vq_active_tree) > 0) {
	spa_t *spa = vd->vdev_spa;
	zio_t *fio;
	uint64_t delta;

	zfs_dbgmsg("slow vdev: %s has %d active IOs",
	vd->vdev_path, avl_numnodes(&vq->vq_active_tree));

	/*
	* Look at the head of all the pending queues,
	* if any I/O has been outstanding for longer than
	* the spa_deadman_synctime invoke the deadman logic.
	*/
	fio = avl_first(&vq->vq_active_tree);
	delta = gethrtime() - fio->io_timestamp;
	if (delta > spa_deadman_synctime(spa))
	zio_deadman(fio, tag);
	}
	mutex_exit(&vq->vq_lock);
	}
	}

	void
	vdev_defer_resilver(vdev_t *vd)
	{
	ASSERT(vd->vdev_ops->vdev_op_leaf);

	vd->vdev_resilver_deferred = B_TRUE;
	vd->vdev_spa->spa_resilver_deferred = B_TRUE;
	}

	/*
	* Clears the resilver deferred flag on all leaf devs under vd. Returns
	* B_TRUE if we have devices that need to be resilvered and are available to
	* accept resilver I/Os.
	*/
	boolean_t
	vdev_clear_resilver_deferred(vdev_t vd, dmu_tx_t tx)
	{
	boolean_t resilver_needed = B_FALSE;
	spa_t *spa = vd->vdev_spa;

	for (int c = 0; c < vd->vdev_children; c++) {
	vdev_t *cvd = vd->vdev_child[c];
	resilver_needed \|= vdev_clear_resilver_deferred(cvd, tx);
	}

	if (vd == spa->spa_root_vdev &&
	spa_feature_is_active(spa, SPA_FEATURE_RESILVER_DEFER)) {
	spa_feature_decr(spa, SPA_FEATURE_RESILVER_DEFER, tx);
	vdev_config_dirty(vd);
	spa->spa_resilver_deferred = B_FALSE;
	return (resilver_needed);
	}

	if (!vdev_is_concrete(vd) \|\| vd->vdev_aux \|\|
	!vd->vdev_ops->vdev_op_leaf)
	return (resilver_needed);

	vd->vdev_resilver_deferred = B_FALSE;

	return (!vdev_is_dead(vd) && !vd->vdev_offline &&
	vdev_resilver_needed(vd, NULL, NULL));
	}

	/*
	* Translate a logical range to the physical range for the specified vdev_t.
	* This function is initially called with a leaf vdev and will walk each
	* parent vdev until it reaches a top-level vdev. Once the top-level is
	* reached the physical range is initialized and the recursive function
	* begins to unwind. As it unwinds it calls the parent's vdev specific
	* translation function to do the real conversion.
	*/
	void
	vdev_xlate(vdev_t vd, const range_seg64_t logical_rs,
	range_seg64_t *physical_rs)
	{
	/*
	* Walk up the vdev tree
	*/
	if (vd != vd->vdev_top) {
	vdev_xlate(vd->vdev_parent, logical_rs, physical_rs);
	} else {
	/*
	* We've reached the top-level vdev, initialize the
	* physical range to the logical range and start to
	* unwind.
	*/
	physical_rs->rs_start = logical_rs->rs_start;
	physical_rs->rs_end = logical_rs->rs_end;
	return;
	}

	vdev_t *pvd = vd->vdev_parent;
	ASSERT3P(pvd, !=, NULL);
	ASSERT3P(pvd->vdev_ops->vdev_op_xlate, !=, NULL);

	/*
	* As this recursive function unwinds, translate the logical
	* range into its physical components by calling the
	* vdev specific translate function.
	*/
	range_seg64_t intermediate = { 0 };
	pvd->vdev_ops->vdev_op_xlate(vd, physical_rs, &intermediate);

	physical_rs->rs_start = intermediate.rs_start;
	physical_rs->rs_end = intermediate.rs_end;
	}

	/*
	* Look at the vdev tree and determine whether any devices are currently being
	* replaced.
	*/
	boolean_t
	vdev_replace_in_progress(vdev_t *vdev)
	{
	ASSERT(spa_config_held(vdev->vdev_spa, SCL_ALL, RW_READER) != 0);

	if (vdev->vdev_ops == &vdev_replacing_ops)
	return (B_TRUE);

	/*
	* A 'spare' vdev indicates that we have a replace in progress, unless
	* it has exactly two children, and the second, the hot spare, has
	* finished being resilvered.
	*/
	if (vdev->vdev_ops == &vdev_spare_ops && (vdev->vdev_children > 2 \|\|
	!vdev_dtl_empty(vdev->vdev_child[1], DTL_MISSING)))
	return (B_TRUE);

	for (int i = 0; i < vdev->vdev_children; i++) {
	if (vdev_replace_in_progress(vdev->vdev_child[i]))
	return (B_TRUE);
	}

	return (B_FALSE);
	}

	EXPORT_SYMBOL(vdev_fault);
	EXPORT_SYMBOL(vdev_degrade);
	EXPORT_SYMBOL(vdev_online);
	EXPORT_SYMBOL(vdev_offline);
	EXPORT_SYMBOL(vdev_clear);

	/* BEGIN CSTYLED */
	ZFS_MODULE_PARAM(zfs_vdev, zfs_vdev_, default_ms_count, INT, ZMOD_RW,
	"Target number of metaslabs per top-level vdev");

	ZFS_MODULE_PARAM(zfs_vdev, zfs_vdev_, default_ms_shift, INT, ZMOD_RW,
	"Default limit for metaslab size");

	ZFS_MODULE_PARAM(zfs_vdev, zfs_vdev_, min_ms_count, INT, ZMOD_RW,
	"Minimum number of metaslabs per top-level vdev");

	ZFS_MODULE_PARAM(zfs_vdev, zfs_vdev_, ms_count_limit, INT, ZMOD_RW,
	"Practical upper limit of total metaslabs per top-level vdev");

	ZFS_MODULE_PARAM(zfs, zfs_, slow_io_events_per_second, UINT, ZMOD_RW,
	"Rate limit slow IO (delay) events to this many per second");

	ZFS_MODULE_PARAM(zfs, zfs_, checksum_events_per_second, UINT, ZMOD_RW,
	"Rate limit checksum events to this many checksum errors per second "
	"(do not set below zed threshold).");

	ZFS_MODULE_PARAM(zfs, zfs_, scan_ignore_errors, INT, ZMOD_RW,
	"Ignore errors during resilver/scrub");

	ZFS_MODULE_PARAM(zfs_vdev, vdev_, validate_skip, INT, ZMOD_RW,
	"Bypass vdev_validate()");

	ZFS_MODULE_PARAM(zfs, zfs_, nocacheflush, INT, ZMOD_RW,
	"Disable cache flushes");

	ZFS_MODULE_PARAM_CALL(zfs_vdev, zfs_vdev_, min_auto_ashift,
	param_set_min_auto_ashift, param_get_ulong, ZMOD_RW,
	"Minimum ashift used when creating new top-level vdevs");

	ZFS_MODULE_PARAM_CALL(zfs_vdev, zfs_vdev_, max_auto_ashift,
	param_set_max_auto_ashift, param_get_ulong, ZMOD_RW,
	"Maximum ashift used when optimizing for logical -> physical sector "
	"size on new top-level vdevs");
	/* END CSTYLED */
	Index: vendor-sys/openzfs/dist/module/zfs/vdev_label.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/zfs/vdev_label.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/zfs/vdev_label.c (revision 366774)
	@@ -1,1967 +1,1968 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/

	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2012, 2020 by Delphix. All rights reserved.
	* Copyright (c) 2017, Intel Corporation.
	*/

	/*
	* Virtual Device Labels
	* ---------------------
	*
	* The vdev label serves several distinct purposes:
	*
	* 1. Uniquely identify this device as part of a ZFS pool and confirm its
	* identity within the pool.
	*
	* 2. Verify that all the devices given in a configuration are present
	* within the pool.
	*
	* 3. Determine the uberblock for the pool.
	*
	* 4. In case of an import operation, determine the configuration of the
	* toplevel vdev of which it is a part.
	*
	* 5. If an import operation cannot find all the devices in the pool,
	* provide enough information to the administrator to determine which
	* devices are missing.
	*
	* It is important to note that while the kernel is responsible for writing the
	* label, it only consumes the information in the first three cases. The
	* latter information is only consumed in userland when determining the
	* configuration to import a pool.
	*
	*
	* Label Organization
	* ------------------
	*
	* Before describing the contents of the label, it's important to understand how
	* the labels are written and updated with respect to the uberblock.
	*
	* When the pool configuration is altered, either because it was newly created
	* or a device was added, we want to update all the labels such that we can deal
	* with fatal failure at any point. To this end, each disk has two labels which
	* are updated before and after the uberblock is synced. Assuming we have
	* labels and an uberblock with the following transaction groups:
	*
	* L1 UB L2
	* +------+ +------+ +------+
	* \| \| \| \| \| \|
	* \| t10 \| \| t10 \| \| t10 \|
	* \| \| \| \| \| \|
	* +------+ +------+ +------+
	*
	* In this stable state, the labels and the uberblock were all updated within
	* the same transaction group (10). Each label is mirrored and checksummed, so
	* that we can detect when we fail partway through writing the label.
	*
	* In order to identify which labels are valid, the labels are written in the
	* following manner:
	*
	* 1. For each vdev, update 'L1' to the new label
	* 2. Update the uberblock
	* 3. For each vdev, update 'L2' to the new label
	*
	* Given arbitrary failure, we can determine the correct label to use based on
	* the transaction group. If we fail after updating L1 but before updating the
	* UB, we will notice that L1's transaction group is greater than the uberblock,
	* so L2 must be valid. If we fail after writing the uberblock but before
	* writing L2, we will notice that L2's transaction group is less than L1, and
	* therefore L1 is valid.
	*
	* Another added complexity is that not every label is updated when the config
	* is synced. If we add a single device, we do not want to have to re-write
	* every label for every device in the pool. This means that both L1 and L2 may
	* be older than the pool uberblock, because the necessary information is stored
	* on another vdev.
	*
	*
	* On-disk Format
	* --------------
	*
	* The vdev label consists of two distinct parts, and is wrapped within the
	* vdev_label_t structure. The label includes 8k of padding to permit legacy
	* VTOC disk labels, but is otherwise ignored.
	*
	* The first half of the label is a packed nvlist which contains pool wide
	* properties, per-vdev properties, and configuration information. It is
	* described in more detail below.
	*
	* The latter half of the label consists of a redundant array of uberblocks.
	* These uberblocks are updated whenever a transaction group is committed,
	* or when the configuration is updated. When a pool is loaded, we scan each
	* vdev for the 'best' uberblock.
	*
	*
	* Configuration Information
	* -------------------------
	*
	* The nvlist describing the pool and vdev contains the following elements:
	*
	* version ZFS on-disk version
	* name Pool name
	* state Pool state
	* txg Transaction group in which this label was written
	* pool_guid Unique identifier for this pool
	* vdev_tree An nvlist describing vdev tree.
	* features_for_read
	* An nvlist of the features necessary for reading the MOS.
	*
	* Each leaf device label also contains the following:
	*
	* top_guid Unique ID for top-level vdev in which this is contained
	* guid Unique ID for the leaf vdev
	*
	* The 'vs' configuration follows the format described in 'spa_config.c'.
	*/

	#include <sys/zfs_context.h>
	#include <sys/spa.h>
	#include <sys/spa_impl.h>
	#include <sys/dmu.h>
	#include <sys/zap.h>
	#include <sys/vdev.h>
	#include <sys/vdev_impl.h>
	#include <sys/uberblock_impl.h>
	#include <sys/metaslab.h>
	#include <sys/metaslab_impl.h>
	#include <sys/zio.h>
	#include <sys/dsl_scan.h>
	#include <sys/abd.h>
	#include <sys/fs/zfs.h>
	#include <sys/byteorder.h>
	#include <sys/zfs_bootenv.h>

	/*
	* Basic routines to read and write from a vdev label.
	* Used throughout the rest of this file.
	*/
	uint64_t
	vdev_label_offset(uint64_t psize, int l, uint64_t offset)
	{
	ASSERT(offset < sizeof (vdev_label_t));
	ASSERT(P2PHASE_TYPED(psize, sizeof (vdev_label_t), uint64_t) == 0);

	return (offset + l * sizeof (vdev_label_t) + (l < VDEV_LABELS / 2 ?
	0 : psize - VDEV_LABELS * sizeof (vdev_label_t)));
	}

	/*
	* Returns back the vdev label associated with the passed in offset.
	*/
	int
	vdev_label_number(uint64_t psize, uint64_t offset)
	{
	int l;

	if (offset >= psize - VDEV_LABEL_END_SIZE) {
	offset -= psize - VDEV_LABEL_END_SIZE;
	offset += (VDEV_LABELS / 2) * sizeof (vdev_label_t);
	}
	l = offset / sizeof (vdev_label_t);
	return (l < VDEV_LABELS ? l : -1);
	}

	static void
	vdev_label_read(zio_t zio, vdev_t vd, int l, abd_t *buf, uint64_t offset,
	uint64_t size, zio_done_func_t done, void private, int flags)
	{
	ASSERT(
	spa_config_held(zio->io_spa, SCL_STATE, RW_READER) == SCL_STATE \|\|
	spa_config_held(zio->io_spa, SCL_STATE, RW_WRITER) == SCL_STATE);
	ASSERT(flags & ZIO_FLAG_CONFIG_WRITER);

	zio_nowait(zio_read_phys(zio, vd,
	vdev_label_offset(vd->vdev_psize, l, offset),
	size, buf, ZIO_CHECKSUM_LABEL, done, private,
	ZIO_PRIORITY_SYNC_READ, flags, B_TRUE));
	}

	void
	vdev_label_write(zio_t zio, vdev_t vd, int l, abd_t *buf, uint64_t offset,
	uint64_t size, zio_done_func_t done, void private, int flags)
	{
	ASSERT(
	spa_config_held(zio->io_spa, SCL_STATE, RW_READER) == SCL_STATE \|\|
	spa_config_held(zio->io_spa, SCL_STATE, RW_WRITER) == SCL_STATE);
	ASSERT(flags & ZIO_FLAG_CONFIG_WRITER);

	zio_nowait(zio_write_phys(zio, vd,
	vdev_label_offset(vd->vdev_psize, l, offset),
	size, buf, ZIO_CHECKSUM_LABEL, done, private,
	ZIO_PRIORITY_SYNC_WRITE, flags, B_TRUE));
	}

	/*
	* Generate the nvlist representing this vdev's stats
	*/
	void
	vdev_config_generate_stats(vdev_t vd, nvlist_t nv)
	{
	nvlist_t *nvx;
	vdev_stat_t *vs;
	vdev_stat_ex_t *vsx;

	vs = kmem_alloc(sizeof (*vs), KM_SLEEP);
	vsx = kmem_alloc(sizeof (*vsx), KM_SLEEP);

	vdev_get_stats_ex(vd, vs, vsx);
	fnvlist_add_uint64_array(nv, ZPOOL_CONFIG_VDEV_STATS,
	(uint64_t )vs, sizeof (vs) / sizeof (uint64_t));

	/*
	* Add extended stats into a special extended stats nvlist. This keeps
	* all the extended stats nicely grouped together. The extended stats
	* nvlist is then added to the main nvlist.
	*/
	nvx = fnvlist_alloc();

	/* ZIOs in flight to disk */
	fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_SYNC_R_ACTIVE_QUEUE,
	vsx->vsx_active_queue[ZIO_PRIORITY_SYNC_READ]);

	fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_SYNC_W_ACTIVE_QUEUE,
	vsx->vsx_active_queue[ZIO_PRIORITY_SYNC_WRITE]);

	fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_ASYNC_R_ACTIVE_QUEUE,
	vsx->vsx_active_queue[ZIO_PRIORITY_ASYNC_READ]);

	fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_ASYNC_W_ACTIVE_QUEUE,
	vsx->vsx_active_queue[ZIO_PRIORITY_ASYNC_WRITE]);

	fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_SCRUB_ACTIVE_QUEUE,
	vsx->vsx_active_queue[ZIO_PRIORITY_SCRUB]);

	fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_TRIM_ACTIVE_QUEUE,
	vsx->vsx_active_queue[ZIO_PRIORITY_TRIM]);

	/* ZIOs pending */
	fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_SYNC_R_PEND_QUEUE,
	vsx->vsx_pend_queue[ZIO_PRIORITY_SYNC_READ]);

	fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_SYNC_W_PEND_QUEUE,
	vsx->vsx_pend_queue[ZIO_PRIORITY_SYNC_WRITE]);

	fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_ASYNC_R_PEND_QUEUE,
	vsx->vsx_pend_queue[ZIO_PRIORITY_ASYNC_READ]);

	fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_ASYNC_W_PEND_QUEUE,
	vsx->vsx_pend_queue[ZIO_PRIORITY_ASYNC_WRITE]);

	fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_SCRUB_PEND_QUEUE,
	vsx->vsx_pend_queue[ZIO_PRIORITY_SCRUB]);

	fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_TRIM_PEND_QUEUE,
	vsx->vsx_pend_queue[ZIO_PRIORITY_TRIM]);

	/* Histograms */
	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_TOT_R_LAT_HISTO,
	vsx->vsx_total_histo[ZIO_TYPE_READ],
	ARRAY_SIZE(vsx->vsx_total_histo[ZIO_TYPE_READ]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_TOT_W_LAT_HISTO,
	vsx->vsx_total_histo[ZIO_TYPE_WRITE],
	ARRAY_SIZE(vsx->vsx_total_histo[ZIO_TYPE_WRITE]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_DISK_R_LAT_HISTO,
	vsx->vsx_disk_histo[ZIO_TYPE_READ],
	ARRAY_SIZE(vsx->vsx_disk_histo[ZIO_TYPE_READ]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_DISK_W_LAT_HISTO,
	vsx->vsx_disk_histo[ZIO_TYPE_WRITE],
	ARRAY_SIZE(vsx->vsx_disk_histo[ZIO_TYPE_WRITE]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_SYNC_R_LAT_HISTO,
	vsx->vsx_queue_histo[ZIO_PRIORITY_SYNC_READ],
	ARRAY_SIZE(vsx->vsx_queue_histo[ZIO_PRIORITY_SYNC_READ]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_SYNC_W_LAT_HISTO,
	vsx->vsx_queue_histo[ZIO_PRIORITY_SYNC_WRITE],
	ARRAY_SIZE(vsx->vsx_queue_histo[ZIO_PRIORITY_SYNC_WRITE]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_ASYNC_R_LAT_HISTO,
	vsx->vsx_queue_histo[ZIO_PRIORITY_ASYNC_READ],
	ARRAY_SIZE(vsx->vsx_queue_histo[ZIO_PRIORITY_ASYNC_READ]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_ASYNC_W_LAT_HISTO,
	vsx->vsx_queue_histo[ZIO_PRIORITY_ASYNC_WRITE],
	ARRAY_SIZE(vsx->vsx_queue_histo[ZIO_PRIORITY_ASYNC_WRITE]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_SCRUB_LAT_HISTO,
	vsx->vsx_queue_histo[ZIO_PRIORITY_SCRUB],
	ARRAY_SIZE(vsx->vsx_queue_histo[ZIO_PRIORITY_SCRUB]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_TRIM_LAT_HISTO,
	vsx->vsx_queue_histo[ZIO_PRIORITY_TRIM],
	ARRAY_SIZE(vsx->vsx_queue_histo[ZIO_PRIORITY_TRIM]));

	/* Request sizes */
	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_SYNC_IND_R_HISTO,
	vsx->vsx_ind_histo[ZIO_PRIORITY_SYNC_READ],
	ARRAY_SIZE(vsx->vsx_ind_histo[ZIO_PRIORITY_SYNC_READ]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_SYNC_IND_W_HISTO,
	vsx->vsx_ind_histo[ZIO_PRIORITY_SYNC_WRITE],
	ARRAY_SIZE(vsx->vsx_ind_histo[ZIO_PRIORITY_SYNC_WRITE]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_ASYNC_IND_R_HISTO,
	vsx->vsx_ind_histo[ZIO_PRIORITY_ASYNC_READ],
	ARRAY_SIZE(vsx->vsx_ind_histo[ZIO_PRIORITY_ASYNC_READ]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_ASYNC_IND_W_HISTO,
	vsx->vsx_ind_histo[ZIO_PRIORITY_ASYNC_WRITE],
	ARRAY_SIZE(vsx->vsx_ind_histo[ZIO_PRIORITY_ASYNC_WRITE]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_IND_SCRUB_HISTO,
	vsx->vsx_ind_histo[ZIO_PRIORITY_SCRUB],
	ARRAY_SIZE(vsx->vsx_ind_histo[ZIO_PRIORITY_SCRUB]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_IND_TRIM_HISTO,
	vsx->vsx_ind_histo[ZIO_PRIORITY_TRIM],
	ARRAY_SIZE(vsx->vsx_ind_histo[ZIO_PRIORITY_TRIM]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_SYNC_AGG_R_HISTO,
	vsx->vsx_agg_histo[ZIO_PRIORITY_SYNC_READ],
	ARRAY_SIZE(vsx->vsx_agg_histo[ZIO_PRIORITY_SYNC_READ]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_SYNC_AGG_W_HISTO,
	vsx->vsx_agg_histo[ZIO_PRIORITY_SYNC_WRITE],
	ARRAY_SIZE(vsx->vsx_agg_histo[ZIO_PRIORITY_SYNC_WRITE]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_ASYNC_AGG_R_HISTO,
	vsx->vsx_agg_histo[ZIO_PRIORITY_ASYNC_READ],
	ARRAY_SIZE(vsx->vsx_agg_histo[ZIO_PRIORITY_ASYNC_READ]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_ASYNC_AGG_W_HISTO,
	vsx->vsx_agg_histo[ZIO_PRIORITY_ASYNC_WRITE],
	ARRAY_SIZE(vsx->vsx_agg_histo[ZIO_PRIORITY_ASYNC_WRITE]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_AGG_SCRUB_HISTO,
	vsx->vsx_agg_histo[ZIO_PRIORITY_SCRUB],
	ARRAY_SIZE(vsx->vsx_agg_histo[ZIO_PRIORITY_SCRUB]));

	fnvlist_add_uint64_array(nvx, ZPOOL_CONFIG_VDEV_AGG_TRIM_HISTO,
	vsx->vsx_agg_histo[ZIO_PRIORITY_TRIM],
	ARRAY_SIZE(vsx->vsx_agg_histo[ZIO_PRIORITY_TRIM]));

	/* IO delays */
	fnvlist_add_uint64(nvx, ZPOOL_CONFIG_VDEV_SLOW_IOS, vs->vs_slow_ios);

	/* Add extended stats nvlist to main nvlist */
	fnvlist_add_nvlist(nv, ZPOOL_CONFIG_VDEV_STATS_EX, nvx);

	fnvlist_free(nvx);
	kmem_free(vs, sizeof (*vs));
	kmem_free(vsx, sizeof (*vsx));
	}

	static void
	root_vdev_actions_getprogress(vdev_t vd, nvlist_t nvl)
	{
	spa_t *spa = vd->vdev_spa;

	if (vd != spa->spa_root_vdev)
	return;

	/* provide either current or previous scan information */
	pool_scan_stat_t ps;
	if (spa_scan_get_stats(spa, &ps) == 0) {
	fnvlist_add_uint64_array(nvl,
	ZPOOL_CONFIG_SCAN_STATS, (uint64_t *)&ps,
	sizeof (pool_scan_stat_t) / sizeof (uint64_t));
	}

	pool_removal_stat_t prs;
	if (spa_removal_get_stats(spa, &prs) == 0) {
	fnvlist_add_uint64_array(nvl,
	ZPOOL_CONFIG_REMOVAL_STATS, (uint64_t *)&prs,
	sizeof (prs) / sizeof (uint64_t));
	}

	pool_checkpoint_stat_t pcs;
	if (spa_checkpoint_get_stats(spa, &pcs) == 0) {
	fnvlist_add_uint64_array(nvl,
	ZPOOL_CONFIG_CHECKPOINT_STATS, (uint64_t *)&pcs,
	sizeof (pcs) / sizeof (uint64_t));
	}
	}

	static void
	top_vdev_actions_getprogress(vdev_t vd, nvlist_t nvl)
	{
	if (vd == vd->vdev_top) {
	vdev_rebuild_stat_t vrs;
	if (vdev_rebuild_get_stats(vd, &vrs) == 0) {
	fnvlist_add_uint64_array(nvl,
	ZPOOL_CONFIG_REBUILD_STATS, (uint64_t *)&vrs,
	sizeof (vrs) / sizeof (uint64_t));
	}
	}
	}

	/*
	* Generate the nvlist representing this vdev's config.
	*/
	nvlist_t *
	vdev_config_generate(spa_t spa, vdev_t vd, boolean_t getstats,
	vdev_config_flag_t flags)
	{
	nvlist_t *nv = NULL;
	vdev_indirect_config_t *vic = &vd->vdev_indirect_config;

	nv = fnvlist_alloc();

	fnvlist_add_string(nv, ZPOOL_CONFIG_TYPE, vd->vdev_ops->vdev_op_type);
	if (!(flags & (VDEV_CONFIG_SPARE \| VDEV_CONFIG_L2CACHE)))
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_ID, vd->vdev_id);
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_GUID, vd->vdev_guid);

	if (vd->vdev_path != NULL)
	fnvlist_add_string(nv, ZPOOL_CONFIG_PATH, vd->vdev_path);

	if (vd->vdev_devid != NULL)
	fnvlist_add_string(nv, ZPOOL_CONFIG_DEVID, vd->vdev_devid);

	if (vd->vdev_physpath != NULL)
	fnvlist_add_string(nv, ZPOOL_CONFIG_PHYS_PATH,
	vd->vdev_physpath);

	if (vd->vdev_enc_sysfs_path != NULL)
	fnvlist_add_string(nv, ZPOOL_CONFIG_VDEV_ENC_SYSFS_PATH,
	vd->vdev_enc_sysfs_path);

	if (vd->vdev_fru != NULL)
	fnvlist_add_string(nv, ZPOOL_CONFIG_FRU, vd->vdev_fru);

	if (vd->vdev_nparity != 0) {
	ASSERT(strcmp(vd->vdev_ops->vdev_op_type,
	VDEV_TYPE_RAIDZ) == 0);

	/*
	* Make sure someone hasn't managed to sneak a fancy new vdev
	* into a crufty old storage pool.
	*/
	ASSERT(vd->vdev_nparity == 1 \|\|
	(vd->vdev_nparity <= 2 &&
	spa_version(spa) >= SPA_VERSION_RAIDZ2) \|\|
	(vd->vdev_nparity <= 3 &&
	spa_version(spa) >= SPA_VERSION_RAIDZ3));

	/*
	* Note that we'll add the nparity tag even on storage pools
	* that only support a single parity device -- older software
	* will just ignore it.
	*/
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_NPARITY, vd->vdev_nparity);
	}

	if (vd->vdev_wholedisk != -1ULL)
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_WHOLE_DISK,
	vd->vdev_wholedisk);

	if (vd->vdev_not_present && !(flags & VDEV_CONFIG_MISSING))
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_NOT_PRESENT, 1);

	if (vd->vdev_isspare)
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_IS_SPARE, 1);

	if (!(flags & (VDEV_CONFIG_SPARE \| VDEV_CONFIG_L2CACHE)) &&
	vd == vd->vdev_top) {
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_METASLAB_ARRAY,
	vd->vdev_ms_array);
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_METASLAB_SHIFT,
	vd->vdev_ms_shift);
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_ASHIFT, vd->vdev_ashift);
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_ASIZE,
	vd->vdev_asize);
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_IS_LOG, vd->vdev_islog);
	if (vd->vdev_removing) {
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_REMOVING,
	vd->vdev_removing);
	}

	/* zpool command expects alloc class data */
	if (getstats && vd->vdev_alloc_bias != VDEV_BIAS_NONE) {
	const char *bias = NULL;

	switch (vd->vdev_alloc_bias) {
	case VDEV_BIAS_LOG:
	bias = VDEV_ALLOC_BIAS_LOG;
	break;
	case VDEV_BIAS_SPECIAL:
	bias = VDEV_ALLOC_BIAS_SPECIAL;
	break;
	case VDEV_BIAS_DEDUP:
	bias = VDEV_ALLOC_BIAS_DEDUP;
	break;
	default:
	ASSERT3U(vd->vdev_alloc_bias, ==,
	VDEV_BIAS_NONE);
	}
	fnvlist_add_string(nv, ZPOOL_CONFIG_ALLOCATION_BIAS,
	bias);
	}
	}

	if (vd->vdev_dtl_sm != NULL) {
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_DTL,
	space_map_object(vd->vdev_dtl_sm));
	}

	if (vic->vic_mapping_object != 0) {
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_INDIRECT_OBJECT,
	vic->vic_mapping_object);
	}

	if (vic->vic_births_object != 0) {
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_INDIRECT_BIRTHS,
	vic->vic_births_object);
	}

	if (vic->vic_prev_indirect_vdev != UINT64_MAX) {
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_PREV_INDIRECT_VDEV,
	vic->vic_prev_indirect_vdev);
	}

	if (vd->vdev_crtxg)
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_CREATE_TXG, vd->vdev_crtxg);

	if (vd->vdev_expansion_time)
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_EXPANSION_TIME,
	vd->vdev_expansion_time);

	if (flags & VDEV_CONFIG_MOS) {
	if (vd->vdev_leaf_zap != 0) {
	ASSERT(vd->vdev_ops->vdev_op_leaf);
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_VDEV_LEAF_ZAP,
	vd->vdev_leaf_zap);
	}

	if (vd->vdev_top_zap != 0) {
	ASSERT(vd == vd->vdev_top);
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_VDEV_TOP_ZAP,
	vd->vdev_top_zap);
	}

	if (vd->vdev_resilver_deferred) {
	ASSERT(vd->vdev_ops->vdev_op_leaf);
	ASSERT(spa->spa_resilver_deferred);
	fnvlist_add_boolean(nv, ZPOOL_CONFIG_RESILVER_DEFER);
	}
	}

	if (getstats) {
	vdev_config_generate_stats(vd, nv);

	root_vdev_actions_getprogress(vd, nv);
	top_vdev_actions_getprogress(vd, nv);

	/*
	* Note: this can be called from open context
	* (spa_get_stats()), so we need the rwlock to prevent
	* the mapping from being changed by condensing.
	*/
	rw_enter(&vd->vdev_indirect_rwlock, RW_READER);
	if (vd->vdev_indirect_mapping != NULL) {
	ASSERT(vd->vdev_indirect_births != NULL);
	vdev_indirect_mapping_t *vim =
	vd->vdev_indirect_mapping;
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_INDIRECT_SIZE,
	vdev_indirect_mapping_size(vim));
	}
	rw_exit(&vd->vdev_indirect_rwlock);
	if (vd->vdev_mg != NULL &&
	vd->vdev_mg->mg_fragmentation != ZFS_FRAG_INVALID) {
	/*
	* Compute approximately how much memory would be used
	* for the indirect mapping if this device were to
	* be removed.
	*
	* Note: If the frag metric is invalid, then not
	* enough metaslabs have been converted to have
	* histograms.
	*/
	uint64_t seg_count = 0;
	uint64_t to_alloc = vd->vdev_stat.vs_alloc;

	/*
	* There are the same number of allocated segments
	* as free segments, so we will have at least one
	* entry per free segment. However, small free
	* segments (smaller than vdev_removal_max_span)
	* will be combined with adjacent allocated segments
	* as a single mapping.
	*/
	for (int i = 0; i < RANGE_TREE_HISTOGRAM_SIZE; i++) {
	- if (1ULL << (i + 1) < vdev_removal_max_span) {
	+ if (i + 1 < highbit64(vdev_removal_max_span)
	+ - 1) {
	to_alloc +=
	vd->vdev_mg->mg_histogram[i] <<
	(i + 1);
	} else {
	seg_count +=
	vd->vdev_mg->mg_histogram[i];
	}
	}

	/*
	* The maximum length of a mapping is
	* zfs_remove_max_segment, so we need at least one entry
	* per zfs_remove_max_segment of allocated data.
	*/
	seg_count += to_alloc / spa_remove_max_segment(spa);

	fnvlist_add_uint64(nv, ZPOOL_CONFIG_INDIRECT_SIZE,
	seg_count *
	sizeof (vdev_indirect_mapping_entry_phys_t));
	}
	}

	if (!vd->vdev_ops->vdev_op_leaf) {
	nvlist_t **child;
	int c, idx;

	ASSERT(!vd->vdev_ishole);

	child = kmem_alloc(vd->vdev_children * sizeof (nvlist_t *),
	KM_SLEEP);

	for (c = 0, idx = 0; c < vd->vdev_children; c++) {
	vdev_t *cvd = vd->vdev_child[c];

	/*
	* If we're generating an nvlist of removing
	* vdevs then skip over any device which is
	* not being removed.
	*/
	if ((flags & VDEV_CONFIG_REMOVING) &&
	!cvd->vdev_removing)
	continue;

	child[idx++] = vdev_config_generate(spa, cvd,
	getstats, flags);
	}

	if (idx) {
	fnvlist_add_nvlist_array(nv, ZPOOL_CONFIG_CHILDREN,
	child, idx);
	}

	for (c = 0; c < idx; c++)
	nvlist_free(child[c]);

	kmem_free(child, vd->vdev_children * sizeof (nvlist_t *));

	} else {
	const char *aux = NULL;

	if (vd->vdev_offline && !vd->vdev_tmpoffline)
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_OFFLINE, B_TRUE);
	if (vd->vdev_resilver_txg != 0)
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_RESILVER_TXG,
	vd->vdev_resilver_txg);
	if (vd->vdev_rebuild_txg != 0)
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_REBUILD_TXG,
	vd->vdev_rebuild_txg);
	if (vd->vdev_faulted)
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_FAULTED, B_TRUE);
	if (vd->vdev_degraded)
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_DEGRADED, B_TRUE);
	if (vd->vdev_removed)
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_REMOVED, B_TRUE);
	if (vd->vdev_unspare)
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_UNSPARE, B_TRUE);
	if (vd->vdev_ishole)
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_IS_HOLE, B_TRUE);

	/* Set the reason why we're FAULTED/DEGRADED. */
	switch (vd->vdev_stat.vs_aux) {
	case VDEV_AUX_ERR_EXCEEDED:
	aux = "err_exceeded";
	break;

	case VDEV_AUX_EXTERNAL:
	aux = "external";
	break;
	}

	if (aux != NULL && !vd->vdev_tmpoffline) {
	fnvlist_add_string(nv, ZPOOL_CONFIG_AUX_STATE, aux);
	} else {
	/*
	* We're healthy - clear any previous AUX_STATE values.
	*/
	if (nvlist_exists(nv, ZPOOL_CONFIG_AUX_STATE))
	nvlist_remove_all(nv, ZPOOL_CONFIG_AUX_STATE);
	}

	if (vd->vdev_splitting && vd->vdev_orig_guid != 0LL) {
	fnvlist_add_uint64(nv, ZPOOL_CONFIG_ORIG_GUID,
	vd->vdev_orig_guid);
	}
	}

	return (nv);
	}

	/*
	* Generate a view of the top-level vdevs. If we currently have holes
	* in the namespace, then generate an array which contains a list of holey
	* vdevs. Additionally, add the number of top-level children that currently
	* exist.
	*/
	void
	vdev_top_config_generate(spa_t spa, nvlist_t config)
	{
	vdev_t *rvd = spa->spa_root_vdev;
	uint64_t *array;
	uint_t c, idx;

	array = kmem_alloc(rvd->vdev_children * sizeof (uint64_t), KM_SLEEP);

	for (c = 0, idx = 0; c < rvd->vdev_children; c++) {
	vdev_t *tvd = rvd->vdev_child[c];

	if (tvd->vdev_ishole) {
	array[idx++] = c;
	}
	}

	if (idx) {
	VERIFY(nvlist_add_uint64_array(config, ZPOOL_CONFIG_HOLE_ARRAY,
	array, idx) == 0);
	}

	VERIFY(nvlist_add_uint64(config, ZPOOL_CONFIG_VDEV_CHILDREN,
	rvd->vdev_children) == 0);

	kmem_free(array, rvd->vdev_children * sizeof (uint64_t));
	}

	/*
	* Returns the configuration from the label of the given vdev. For vdevs
	* which don't have a txg value stored on their label (i.e. spares/cache)
	* or have not been completely initialized (txg = 0) just return
	* the configuration from the first valid label we find. Otherwise,
	* find the most up-to-date label that does not exceed the specified
	* 'txg' value.
	*/
	nvlist_t *
	vdev_label_read_config(vdev_t *vd, uint64_t txg)
	{
	spa_t *spa = vd->vdev_spa;
	nvlist_t *config = NULL;
	vdev_phys_t *vp;
	abd_t *vp_abd;
	zio_t *zio;
	uint64_t best_txg = 0;
	uint64_t label_txg = 0;
	int error = 0;
	int flags = ZIO_FLAG_CONFIG_WRITER \| ZIO_FLAG_CANFAIL \|
	ZIO_FLAG_SPECULATIVE;

	ASSERT(spa_config_held(spa, SCL_STATE_ALL, RW_WRITER) == SCL_STATE_ALL);

	if (!vdev_readable(vd))
	return (NULL);

	vp_abd = abd_alloc_linear(sizeof (vdev_phys_t), B_TRUE);
	vp = abd_to_buf(vp_abd);

	retry:
	for (int l = 0; l < VDEV_LABELS; l++) {
	nvlist_t *label = NULL;

	zio = zio_root(spa, NULL, NULL, flags);

	vdev_label_read(zio, vd, l, vp_abd,
	offsetof(vdev_label_t, vl_vdev_phys),
	sizeof (vdev_phys_t), NULL, NULL, flags);

	if (zio_wait(zio) == 0 &&
	nvlist_unpack(vp->vp_nvlist, sizeof (vp->vp_nvlist),
	&label, 0) == 0) {
	/*
	* Auxiliary vdevs won't have txg values in their
	* labels and newly added vdevs may not have been
	* completely initialized so just return the
	* configuration from the first valid label we
	* encounter.
	*/
	error = nvlist_lookup_uint64(label,
	ZPOOL_CONFIG_POOL_TXG, &label_txg);
	if ((error \|\| label_txg == 0) && !config) {
	config = label;
	break;
	} else if (label_txg <= txg && label_txg > best_txg) {
	best_txg = label_txg;
	nvlist_free(config);
	config = fnvlist_dup(label);
	}
	}

	if (label != NULL) {
	nvlist_free(label);
	label = NULL;
	}
	}

	if (config == NULL && !(flags & ZIO_FLAG_TRYHARD)) {
	flags \|= ZIO_FLAG_TRYHARD;
	goto retry;
	}

	/*
	* We found a valid label but it didn't pass txg restrictions.
	*/
	if (config == NULL && label_txg != 0) {
	vdev_dbgmsg(vd, "label discarded as txg is too large "
	"(%llu > %llu)", (u_longlong_t)label_txg,
	(u_longlong_t)txg);
	}

	abd_free(vp_abd);

	return (config);
	}

	/*
	* Determine if a device is in use. The 'spare_guid' parameter will be filled
	* in with the device guid if this spare is active elsewhere on the system.
	*/
	static boolean_t
	vdev_inuse(vdev_t *vd, uint64_t crtxg, vdev_labeltype_t reason,
	uint64_t spare_guid, uint64_t l2cache_guid)
	{
	spa_t *spa = vd->vdev_spa;
	uint64_t state, pool_guid, device_guid, txg, spare_pool;
	uint64_t vdtxg = 0;
	nvlist_t *label;

	if (spare_guid)
	*spare_guid = 0ULL;
	if (l2cache_guid)
	*l2cache_guid = 0ULL;

	/*
	* Read the label, if any, and perform some basic sanity checks.
	*/
	if ((label = vdev_label_read_config(vd, -1ULL)) == NULL)
	return (B_FALSE);

	(void) nvlist_lookup_uint64(label, ZPOOL_CONFIG_CREATE_TXG,
	&vdtxg);

	if (nvlist_lookup_uint64(label, ZPOOL_CONFIG_POOL_STATE,
	&state) != 0 \|\|
	nvlist_lookup_uint64(label, ZPOOL_CONFIG_GUID,
	&device_guid) != 0) {
	nvlist_free(label);
	return (B_FALSE);
	}

	if (state != POOL_STATE_SPARE && state != POOL_STATE_L2CACHE &&
	(nvlist_lookup_uint64(label, ZPOOL_CONFIG_POOL_GUID,
	&pool_guid) != 0 \|\|
	nvlist_lookup_uint64(label, ZPOOL_CONFIG_POOL_TXG,
	&txg) != 0)) {
	nvlist_free(label);
	return (B_FALSE);
	}

	nvlist_free(label);

	/*
	* Check to see if this device indeed belongs to the pool it claims to
	* be a part of. The only way this is allowed is if the device is a hot
	* spare (which we check for later on).
	*/
	if (state != POOL_STATE_SPARE && state != POOL_STATE_L2CACHE &&
	!spa_guid_exists(pool_guid, device_guid) &&
	!spa_spare_exists(device_guid, NULL, NULL) &&
	!spa_l2cache_exists(device_guid, NULL))
	return (B_FALSE);

	/*
	* If the transaction group is zero, then this an initialized (but
	* unused) label. This is only an error if the create transaction
	* on-disk is the same as the one we're using now, in which case the
	* user has attempted to add the same vdev multiple times in the same
	* transaction.
	*/
	if (state != POOL_STATE_SPARE && state != POOL_STATE_L2CACHE &&
	txg == 0 && vdtxg == crtxg)
	return (B_TRUE);

	/*
	* Check to see if this is a spare device. We do an explicit check for
	* spa_has_spare() here because it may be on our pending list of spares
	* to add. We also check if it is an l2cache device.
	*/
	if (spa_spare_exists(device_guid, &spare_pool, NULL) \|\|
	spa_has_spare(spa, device_guid)) {
	if (spare_guid)
	*spare_guid = device_guid;

	switch (reason) {
	case VDEV_LABEL_CREATE:
	case VDEV_LABEL_L2CACHE:
	return (B_TRUE);

	case VDEV_LABEL_REPLACE:
	return (!spa_has_spare(spa, device_guid) \|\|
	spare_pool != 0ULL);

	case VDEV_LABEL_SPARE:
	return (spa_has_spare(spa, device_guid));
	default:
	break;
	}
	}

	/*
	* Check to see if this is an l2cache device.
	*/
	if (spa_l2cache_exists(device_guid, NULL))
	return (B_TRUE);

	/*
	* We can't rely on a pool's state if it's been imported
	* read-only. Instead we look to see if the pools is marked
	* read-only in the namespace and set the state to active.
	*/
	if (state != POOL_STATE_SPARE && state != POOL_STATE_L2CACHE &&
	(spa = spa_by_guid(pool_guid, device_guid)) != NULL &&
	spa_mode(spa) == SPA_MODE_READ)
	state = POOL_STATE_ACTIVE;

	/*
	* If the device is marked ACTIVE, then this device is in use by another
	* pool on the system.
	*/
	return (state == POOL_STATE_ACTIVE);
	}

	/*
	* Initialize a vdev label. We check to make sure each leaf device is not in
	* use, and writable. We put down an initial label which we will later
	* overwrite with a complete label. Note that it's important to do this
	* sequentially, not in parallel, so that we catch cases of multiple use of the
	* same leaf vdev in the vdev we're creating -- e.g. mirroring a disk with
	* itself.
	*/
	int
	vdev_label_init(vdev_t *vd, uint64_t crtxg, vdev_labeltype_t reason)
	{
	spa_t *spa = vd->vdev_spa;
	nvlist_t *label;
	vdev_phys_t *vp;
	abd_t *vp_abd;
	abd_t *bootenv;
	uberblock_t *ub;
	abd_t *ub_abd;
	zio_t *zio;
	char *buf;
	size_t buflen;
	int error;
	uint64_t spare_guid = 0, l2cache_guid = 0;
	int flags = ZIO_FLAG_CONFIG_WRITER \| ZIO_FLAG_CANFAIL;

	ASSERT(spa_config_held(spa, SCL_ALL, RW_WRITER) == SCL_ALL);

	for (int c = 0; c < vd->vdev_children; c++)
	if ((error = vdev_label_init(vd->vdev_child[c],
	crtxg, reason)) != 0)
	return (error);

	/* Track the creation time for this vdev */
	vd->vdev_crtxg = crtxg;

	if (!vd->vdev_ops->vdev_op_leaf \|\| !spa_writeable(spa))
	return (0);

	/*
	* Dead vdevs cannot be initialized.
	*/
	if (vdev_is_dead(vd))
	return (SET_ERROR(EIO));

	/*
	* Determine if the vdev is in use.
	*/
	if (reason != VDEV_LABEL_REMOVE && reason != VDEV_LABEL_SPLIT &&
	vdev_inuse(vd, crtxg, reason, &spare_guid, &l2cache_guid))
	return (SET_ERROR(EBUSY));

	/*
	* If this is a request to add or replace a spare or l2cache device
	* that is in use elsewhere on the system, then we must update the
	* guid (which was initialized to a random value) to reflect the
	* actual GUID (which is shared between multiple pools).
	*/
	if (reason != VDEV_LABEL_REMOVE && reason != VDEV_LABEL_L2CACHE &&
	spare_guid != 0ULL) {
	uint64_t guid_delta = spare_guid - vd->vdev_guid;

	vd->vdev_guid += guid_delta;

	for (vdev_t *pvd = vd; pvd != NULL; pvd = pvd->vdev_parent)
	pvd->vdev_guid_sum += guid_delta;

	/*
	* If this is a replacement, then we want to fallthrough to the
	* rest of the code. If we're adding a spare, then it's already
	* labeled appropriately and we can just return.
	*/
	if (reason == VDEV_LABEL_SPARE)
	return (0);
	ASSERT(reason == VDEV_LABEL_REPLACE \|\|
	reason == VDEV_LABEL_SPLIT);
	}

	if (reason != VDEV_LABEL_REMOVE && reason != VDEV_LABEL_SPARE &&
	l2cache_guid != 0ULL) {
	uint64_t guid_delta = l2cache_guid - vd->vdev_guid;

	vd->vdev_guid += guid_delta;

	for (vdev_t *pvd = vd; pvd != NULL; pvd = pvd->vdev_parent)
	pvd->vdev_guid_sum += guid_delta;

	/*
	* If this is a replacement, then we want to fallthrough to the
	* rest of the code. If we're adding an l2cache, then it's
	* already labeled appropriately and we can just return.
	*/
	if (reason == VDEV_LABEL_L2CACHE)
	return (0);
	ASSERT(reason == VDEV_LABEL_REPLACE);
	}

	/*
	* Initialize its label.
	*/
	vp_abd = abd_alloc_linear(sizeof (vdev_phys_t), B_TRUE);
	abd_zero(vp_abd, sizeof (vdev_phys_t));
	vp = abd_to_buf(vp_abd);

	/*
	* Generate a label describing the pool and our top-level vdev.
	* We mark it as being from txg 0 to indicate that it's not
	* really part of an active pool just yet. The labels will
	* be written again with a meaningful txg by spa_sync().
	*/
	if (reason == VDEV_LABEL_SPARE \|\|
	(reason == VDEV_LABEL_REMOVE && vd->vdev_isspare)) {
	/*
	* For inactive hot spares, we generate a special label that
	* identifies as a mutually shared hot spare. We write the
	* label if we are adding a hot spare, or if we are removing an
	* active hot spare (in which case we want to revert the
	* labels).
	*/
	VERIFY(nvlist_alloc(&label, NV_UNIQUE_NAME, KM_SLEEP) == 0);

	VERIFY(nvlist_add_uint64(label, ZPOOL_CONFIG_VERSION,
	spa_version(spa)) == 0);
	VERIFY(nvlist_add_uint64(label, ZPOOL_CONFIG_POOL_STATE,
	POOL_STATE_SPARE) == 0);
	VERIFY(nvlist_add_uint64(label, ZPOOL_CONFIG_GUID,
	vd->vdev_guid) == 0);
	} else if (reason == VDEV_LABEL_L2CACHE \|\|
	(reason == VDEV_LABEL_REMOVE && vd->vdev_isl2cache)) {
	/*
	* For level 2 ARC devices, add a special label.
	*/
	VERIFY(nvlist_alloc(&label, NV_UNIQUE_NAME, KM_SLEEP) == 0);

	VERIFY(nvlist_add_uint64(label, ZPOOL_CONFIG_VERSION,
	spa_version(spa)) == 0);
	VERIFY(nvlist_add_uint64(label, ZPOOL_CONFIG_POOL_STATE,
	POOL_STATE_L2CACHE) == 0);
	VERIFY(nvlist_add_uint64(label, ZPOOL_CONFIG_GUID,
	vd->vdev_guid) == 0);
	} else {
	uint64_t txg = 0ULL;

	if (reason == VDEV_LABEL_SPLIT)
	txg = spa->spa_uberblock.ub_txg;
	label = spa_config_generate(spa, vd, txg, B_FALSE);

	/*
	* Add our creation time. This allows us to detect multiple
	* vdev uses as described above, and automatically expires if we
	* fail.
	*/
	VERIFY(nvlist_add_uint64(label, ZPOOL_CONFIG_CREATE_TXG,
	crtxg) == 0);
	}

	buf = vp->vp_nvlist;
	buflen = sizeof (vp->vp_nvlist);

	error = nvlist_pack(label, &buf, &buflen, NV_ENCODE_XDR, KM_SLEEP);
	if (error != 0) {
	nvlist_free(label);
	abd_free(vp_abd);
	/* EFAULT means nvlist_pack ran out of room */
	return (SET_ERROR(error == EFAULT ? ENAMETOOLONG : EINVAL));
	}

	/*
	* Initialize uberblock template.
	*/
	ub_abd = abd_alloc_linear(VDEV_UBERBLOCK_RING, B_TRUE);
	abd_zero(ub_abd, VDEV_UBERBLOCK_RING);
	abd_copy_from_buf(ub_abd, &spa->spa_uberblock, sizeof (uberblock_t));
	ub = abd_to_buf(ub_abd);
	ub->ub_txg = 0;

	/* Initialize the 2nd padding area. */
	bootenv = abd_alloc_for_io(VDEV_PAD_SIZE, B_TRUE);
	abd_zero(bootenv, VDEV_PAD_SIZE);

	/*
	* Write everything in parallel.
	*/
	retry:
	zio = zio_root(spa, NULL, NULL, flags);

	for (int l = 0; l < VDEV_LABELS; l++) {

	vdev_label_write(zio, vd, l, vp_abd,
	offsetof(vdev_label_t, vl_vdev_phys),
	sizeof (vdev_phys_t), NULL, NULL, flags);

	/*
	* Skip the 1st padding area.
	* Zero out the 2nd padding area where it might have
	* left over data from previous filesystem format.
	*/
	vdev_label_write(zio, vd, l, bootenv,
	offsetof(vdev_label_t, vl_be),
	VDEV_PAD_SIZE, NULL, NULL, flags);

	vdev_label_write(zio, vd, l, ub_abd,
	offsetof(vdev_label_t, vl_uberblock),
	VDEV_UBERBLOCK_RING, NULL, NULL, flags);
	}

	error = zio_wait(zio);

	if (error != 0 && !(flags & ZIO_FLAG_TRYHARD)) {
	flags \|= ZIO_FLAG_TRYHARD;
	goto retry;
	}

	nvlist_free(label);
	abd_free(bootenv);
	abd_free(ub_abd);
	abd_free(vp_abd);

	/*
	* If this vdev hasn't been previously identified as a spare, then we
	* mark it as such only if a) we are labeling it as a spare, or b) it
	* exists as a spare elsewhere in the system. Do the same for
	* level 2 ARC devices.
	*/
	if (error == 0 && !vd->vdev_isspare &&
	(reason == VDEV_LABEL_SPARE \|\|
	spa_spare_exists(vd->vdev_guid, NULL, NULL)))
	spa_spare_add(vd);

	if (error == 0 && !vd->vdev_isl2cache &&
	(reason == VDEV_LABEL_L2CACHE \|\|
	spa_l2cache_exists(vd->vdev_guid, NULL)))
	spa_l2cache_add(vd);

	return (error);
	}

	/*
	* Done callback for vdev_label_read_bootenv_impl. If this is the first
	* callback to finish, store our abd in the callback pointer. Otherwise, we
	* just free our abd and return.
	*/
	static void
	vdev_label_read_bootenv_done(zio_t *zio)
	{
	zio_t *rio = zio->io_private;
	abd_t **cbp = rio->io_private;

	ASSERT3U(zio->io_size, ==, VDEV_PAD_SIZE);

	if (zio->io_error == 0) {
	mutex_enter(&rio->io_lock);
	if (*cbp == NULL) {
	/* Will free this buffer in vdev_label_read_bootenv. */
	*cbp = zio->io_abd;
	} else {
	abd_free(zio->io_abd);
	}
	mutex_exit(&rio->io_lock);
	} else {
	abd_free(zio->io_abd);
	}
	}

	static void
	vdev_label_read_bootenv_impl(zio_t zio, vdev_t vd, int flags)
	{
	for (int c = 0; c < vd->vdev_children; c++)
	vdev_label_read_bootenv_impl(zio, vd->vdev_child[c], flags);

	/*
	* We just use the first label that has a correct checksum; the
	* bootloader should have rewritten them all to be the same on boot,
	* and any changes we made since boot have been the same across all
	* labels.
	*/
	if (vd->vdev_ops->vdev_op_leaf && vdev_readable(vd)) {
	for (int l = 0; l < VDEV_LABELS; l++) {
	vdev_label_read(zio, vd, l,
	abd_alloc_linear(VDEV_PAD_SIZE, B_FALSE),
	offsetof(vdev_label_t, vl_be), VDEV_PAD_SIZE,
	vdev_label_read_bootenv_done, zio, flags);
	}
	}
	}

	int
	vdev_label_read_bootenv(vdev_t rvd, nvlist_t bootenv)
	{
	nvlist_t *config;
	spa_t *spa = rvd->vdev_spa;
	abd_t *abd = NULL;
	int flags = ZIO_FLAG_CONFIG_WRITER \| ZIO_FLAG_CANFAIL \|
	ZIO_FLAG_SPECULATIVE \| ZIO_FLAG_TRYHARD;

	ASSERT(bootenv);
	ASSERT(spa_config_held(spa, SCL_ALL, RW_WRITER) == SCL_ALL);

	zio_t *zio = zio_root(spa, NULL, &abd, flags);
	vdev_label_read_bootenv_impl(zio, rvd, flags);
	int err = zio_wait(zio);

	if (abd != NULL) {
	char *buf;
	vdev_boot_envblock_t *vbe = abd_to_buf(abd);

	vbe->vbe_version = ntohll(vbe->vbe_version);
	switch (vbe->vbe_version) {
	case VB_RAW:
	/*
	* if we have textual data in vbe_bootenv, create nvlist
	* with key "envmap".
	*/
	fnvlist_add_uint64(bootenv, BOOTENV_VERSION, VB_RAW);
	vbe->vbe_bootenv[sizeof (vbe->vbe_bootenv) - 1] = '\0';
	fnvlist_add_string(bootenv, GRUB_ENVMAP,
	vbe->vbe_bootenv);
	break;

	case VB_NVLIST:
	err = nvlist_unpack(vbe->vbe_bootenv,
	sizeof (vbe->vbe_bootenv), &config, 0);
	if (err == 0) {
	fnvlist_merge(bootenv, config);
	nvlist_free(config);
	break;
	}
	/* FALLTHROUGH */
	default:
	/* Check for FreeBSD zfs bootonce command string */
	buf = abd_to_buf(abd);
	if (*buf == '\0') {
	fnvlist_add_uint64(bootenv, BOOTENV_VERSION,
	VB_NVLIST);
	break;
	}
	fnvlist_add_string(bootenv, FREEBSD_BOOTONCE, buf);
	}

	/*
	* abd was allocated in vdev_label_read_bootenv_impl()
	*/
	abd_free(abd);
	/*
	* If we managed to read any successfully,
	* return success.
	*/
	return (0);
	}
	return (err);
	}

	int
	vdev_label_write_bootenv(vdev_t vd, nvlist_t env)
	{
	zio_t *zio;
	spa_t *spa = vd->vdev_spa;
	vdev_boot_envblock_t *bootenv;
	int flags = ZIO_FLAG_CONFIG_WRITER \| ZIO_FLAG_CANFAIL;
	int error;
	size_t nvsize;
	char *nvbuf;

	error = nvlist_size(env, &nvsize, NV_ENCODE_XDR);
	if (error != 0)
	return (SET_ERROR(error));

	if (nvsize >= sizeof (bootenv->vbe_bootenv)) {
	return (SET_ERROR(E2BIG));
	}

	ASSERT(spa_config_held(spa, SCL_ALL, RW_WRITER) == SCL_ALL);

	error = ENXIO;
	for (int c = 0; c < vd->vdev_children; c++) {
	int child_err;

	child_err = vdev_label_write_bootenv(vd->vdev_child[c], env);
	/*
	* As long as any of the disks managed to write all of their
	* labels successfully, return success.
	*/
	if (child_err == 0)
	error = child_err;
	}

	if (!vd->vdev_ops->vdev_op_leaf \|\| vdev_is_dead(vd) \|\|
	!vdev_writeable(vd)) {
	return (error);
	}
	ASSERT3U(sizeof (*bootenv), ==, VDEV_PAD_SIZE);
	abd_t *abd = abd_alloc_for_io(VDEV_PAD_SIZE, B_TRUE);
	abd_zero(abd, VDEV_PAD_SIZE);

	bootenv = abd_borrow_buf_copy(abd, VDEV_PAD_SIZE);
	nvbuf = bootenv->vbe_bootenv;
	nvsize = sizeof (bootenv->vbe_bootenv);

	bootenv->vbe_version = fnvlist_lookup_uint64(env, BOOTENV_VERSION);
	switch (bootenv->vbe_version) {
	case VB_RAW:
	if (nvlist_lookup_string(env, GRUB_ENVMAP, &nvbuf) == 0) {
	(void) strlcpy(bootenv->vbe_bootenv, nvbuf, nvsize);
	}
	error = 0;
	break;

	case VB_NVLIST:
	error = nvlist_pack(env, &nvbuf, &nvsize, NV_ENCODE_XDR,
	KM_SLEEP);
	break;

	default:
	error = EINVAL;
	break;
	}

	if (error == 0) {
	bootenv->vbe_version = htonll(bootenv->vbe_version);
	abd_return_buf_copy(abd, bootenv, VDEV_PAD_SIZE);
	} else {
	abd_free(abd);
	return (SET_ERROR(error));
	}

	retry:
	zio = zio_root(spa, NULL, NULL, flags);
	for (int l = 0; l < VDEV_LABELS; l++) {
	vdev_label_write(zio, vd, l, abd,
	offsetof(vdev_label_t, vl_be),
	VDEV_PAD_SIZE, NULL, NULL, flags);
	}

	error = zio_wait(zio);
	if (error != 0 && !(flags & ZIO_FLAG_TRYHARD)) {
	flags \|= ZIO_FLAG_TRYHARD;
	goto retry;
	}

	abd_free(abd);
	return (error);
	}

	/*
	* ==========================================================================
	* uberblock load/sync
	* ==========================================================================
	*/

	/*
	* Consider the following situation: txg is safely synced to disk. We've
	* written the first uberblock for txg + 1, and then we lose power. When we
	* come back up, we fail to see the uberblock for txg + 1 because, say,
	* it was on a mirrored device and the replica to which we wrote txg + 1
	* is now offline. If we then make some changes and sync txg + 1, and then
	* the missing replica comes back, then for a few seconds we'll have two
	* conflicting uberblocks on disk with the same txg. The solution is simple:
	* among uberblocks with equal txg, choose the one with the latest timestamp.
	*/
	static int
	vdev_uberblock_compare(const uberblock_t ub1, const uberblock_t ub2)
	{
	int cmp = TREE_CMP(ub1->ub_txg, ub2->ub_txg);

	if (likely(cmp))
	return (cmp);

	cmp = TREE_CMP(ub1->ub_timestamp, ub2->ub_timestamp);
	if (likely(cmp))
	return (cmp);

	/*
	* If MMP_VALID(ub) && MMP_SEQ_VALID(ub) then the host has an MMP-aware
	- * ZFS, e.g. zfsonlinux >= 0.7.
	+ * ZFS, e.g. OpenZFS >= 0.7.
	*
	* If one ub has MMP and the other does not, they were written by
	* different hosts, which matters for MMP. So we treat no MMP/no SEQ as
	* a 0 value.
	*
	* Since timestamp and txg are the same if we get this far, either is
	* acceptable for importing the pool.
	*/
	unsigned int seq1 = 0;
	unsigned int seq2 = 0;

	if (MMP_VALID(ub1) && MMP_SEQ_VALID(ub1))
	seq1 = MMP_SEQ(ub1);

	if (MMP_VALID(ub2) && MMP_SEQ_VALID(ub2))
	seq2 = MMP_SEQ(ub2);

	return (TREE_CMP(seq1, seq2));
	}

	struct ubl_cbdata {
	uberblock_t ubl_ubbest; / Best uberblock */
	vdev_t ubl_vd; / vdev associated with the above */
	};

	static void
	vdev_uberblock_load_done(zio_t *zio)
	{
	vdev_t *vd = zio->io_vd;
	spa_t *spa = zio->io_spa;
	zio_t *rio = zio->io_private;
	uberblock_t *ub = abd_to_buf(zio->io_abd);
	struct ubl_cbdata *cbp = rio->io_private;

	ASSERT3U(zio->io_size, ==, VDEV_UBERBLOCK_SIZE(vd));

	if (zio->io_error == 0 && uberblock_verify(ub) == 0) {
	mutex_enter(&rio->io_lock);
	if (ub->ub_txg <= spa->spa_load_max_txg &&
	vdev_uberblock_compare(ub, cbp->ubl_ubbest) > 0) {
	/*
	* Keep track of the vdev in which this uberblock
	* was found. We will use this information later
	* to obtain the config nvlist associated with
	* this uberblock.
	*/
	cbp->ubl_ubbest = ub;
	cbp->ubl_vd = vd;
	}
	mutex_exit(&rio->io_lock);
	}

	abd_free(zio->io_abd);
	}

	static void
	vdev_uberblock_load_impl(zio_t zio, vdev_t vd, int flags,
	struct ubl_cbdata *cbp)
	{
	for (int c = 0; c < vd->vdev_children; c++)
	vdev_uberblock_load_impl(zio, vd->vdev_child[c], flags, cbp);

	if (vd->vdev_ops->vdev_op_leaf && vdev_readable(vd)) {
	for (int l = 0; l < VDEV_LABELS; l++) {
	for (int n = 0; n < VDEV_UBERBLOCK_COUNT(vd); n++) {
	vdev_label_read(zio, vd, l,
	abd_alloc_linear(VDEV_UBERBLOCK_SIZE(vd),
	B_TRUE), VDEV_UBERBLOCK_OFFSET(vd, n),
	VDEV_UBERBLOCK_SIZE(vd),
	vdev_uberblock_load_done, zio, flags);
	}
	}
	}
	}

	/*
	* Reads the 'best' uberblock from disk along with its associated
	* configuration. First, we read the uberblock array of each label of each
	* vdev, keeping track of the uberblock with the highest txg in each array.
	* Then, we read the configuration from the same vdev as the best uberblock.
	*/
	void
	vdev_uberblock_load(vdev_t rvd, uberblock_t ub, nvlist_t **config)
	{
	zio_t *zio;
	spa_t *spa = rvd->vdev_spa;
	struct ubl_cbdata cb;
	int flags = ZIO_FLAG_CONFIG_WRITER \| ZIO_FLAG_CANFAIL \|
	ZIO_FLAG_SPECULATIVE \| ZIO_FLAG_TRYHARD;

	ASSERT(ub);
	ASSERT(config);

	bzero(ub, sizeof (uberblock_t));
	*config = NULL;

	cb.ubl_ubbest = ub;
	cb.ubl_vd = NULL;

	spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER);
	zio = zio_root(spa, NULL, &cb, flags);
	vdev_uberblock_load_impl(zio, rvd, flags, &cb);
	(void) zio_wait(zio);

	/*
	* It's possible that the best uberblock was discovered on a label
	* that has a configuration which was written in a future txg.
	* Search all labels on this vdev to find the configuration that
	* matches the txg for our uberblock.
	*/
	if (cb.ubl_vd != NULL) {
	vdev_dbgmsg(cb.ubl_vd, "best uberblock found for spa %s. "
	"txg %llu", spa->spa_name, (u_longlong_t)ub->ub_txg);

	*config = vdev_label_read_config(cb.ubl_vd, ub->ub_txg);
	if (*config == NULL && spa->spa_extreme_rewind) {
	vdev_dbgmsg(cb.ubl_vd, "failed to read label config. "
	"Trying again without txg restrictions.");
	*config = vdev_label_read_config(cb.ubl_vd, UINT64_MAX);
	}
	if (*config == NULL) {
	vdev_dbgmsg(cb.ubl_vd, "failed to read label config");
	}
	}
	spa_config_exit(spa, SCL_ALL, FTAG);
	}

	/*
	* For use when a leaf vdev is expanded.
	* The location of labels 2 and 3 changed, and at the new location the
	* uberblock rings are either empty or contain garbage. The sync will write
	* new configs there because the vdev is dirty, but expansion also needs the
	* uberblock rings copied. Read them from label 0 which did not move.
	*
	* Since the point is to populate labels {2,3} with valid uberblocks,
	* we zero uberblocks we fail to read or which are not valid.
	*/

	static void
	vdev_copy_uberblocks(vdev_t *vd)
	{
	abd_t *ub_abd;
	zio_t *write_zio;
	int locks = (SCL_L2ARC \| SCL_ZIO);
	int flags = ZIO_FLAG_CONFIG_WRITER \| ZIO_FLAG_CANFAIL \|
	ZIO_FLAG_SPECULATIVE;

	ASSERT(spa_config_held(vd->vdev_spa, SCL_STATE, RW_READER) ==
	SCL_STATE);
	ASSERT(vd->vdev_ops->vdev_op_leaf);

	spa_config_enter(vd->vdev_spa, locks, FTAG, RW_READER);

	ub_abd = abd_alloc_linear(VDEV_UBERBLOCK_SIZE(vd), B_TRUE);

	write_zio = zio_root(vd->vdev_spa, NULL, NULL, flags);
	for (int n = 0; n < VDEV_UBERBLOCK_COUNT(vd); n++) {
	const int src_label = 0;
	zio_t *zio;

	zio = zio_root(vd->vdev_spa, NULL, NULL, flags);
	vdev_label_read(zio, vd, src_label, ub_abd,
	VDEV_UBERBLOCK_OFFSET(vd, n), VDEV_UBERBLOCK_SIZE(vd),
	NULL, NULL, flags);

	if (zio_wait(zio) \|\| uberblock_verify(abd_to_buf(ub_abd)))
	abd_zero(ub_abd, VDEV_UBERBLOCK_SIZE(vd));

	for (int l = 2; l < VDEV_LABELS; l++)
	vdev_label_write(write_zio, vd, l, ub_abd,
	VDEV_UBERBLOCK_OFFSET(vd, n),
	VDEV_UBERBLOCK_SIZE(vd), NULL, NULL,
	flags \| ZIO_FLAG_DONT_PROPAGATE);
	}
	(void) zio_wait(write_zio);

	spa_config_exit(vd->vdev_spa, locks, FTAG);

	abd_free(ub_abd);
	}

	/*
	* On success, increment root zio's count of good writes.
	* We only get credit for writes to known-visible vdevs; see spa_vdev_add().
	*/
	static void
	vdev_uberblock_sync_done(zio_t *zio)
	{
	uint64_t *good_writes = zio->io_private;

	if (zio->io_error == 0 && zio->io_vd->vdev_top->vdev_ms_array != 0)
	atomic_inc_64(good_writes);
	}

	/*
	* Write the uberblock to all labels of all leaves of the specified vdev.
	*/
	static void
	vdev_uberblock_sync(zio_t zio, uint64_t good_writes,
	uberblock_t ub, vdev_t vd, int flags)
	{
	for (uint64_t c = 0; c < vd->vdev_children; c++) {
	vdev_uberblock_sync(zio, good_writes,
	ub, vd->vdev_child[c], flags);
	}

	if (!vd->vdev_ops->vdev_op_leaf)
	return;

	if (!vdev_writeable(vd))
	return;

	/* If the vdev was expanded, need to copy uberblock rings. */
	if (vd->vdev_state == VDEV_STATE_HEALTHY &&
	vd->vdev_copy_uberblocks == B_TRUE) {
	vdev_copy_uberblocks(vd);
	vd->vdev_copy_uberblocks = B_FALSE;
	}

	int m = spa_multihost(vd->vdev_spa) ? MMP_BLOCKS_PER_LABEL : 0;
	int n = ub->ub_txg % (VDEV_UBERBLOCK_COUNT(vd) - m);

	/* Copy the uberblock_t into the ABD */
	abd_t *ub_abd = abd_alloc_for_io(VDEV_UBERBLOCK_SIZE(vd), B_TRUE);
	abd_zero(ub_abd, VDEV_UBERBLOCK_SIZE(vd));
	abd_copy_from_buf(ub_abd, ub, sizeof (uberblock_t));

	for (int l = 0; l < VDEV_LABELS; l++)
	vdev_label_write(zio, vd, l, ub_abd,
	VDEV_UBERBLOCK_OFFSET(vd, n), VDEV_UBERBLOCK_SIZE(vd),
	vdev_uberblock_sync_done, good_writes,
	flags \| ZIO_FLAG_DONT_PROPAGATE);

	abd_free(ub_abd);
	}

	/* Sync the uberblocks to all vdevs in svd[] */
	static int
	vdev_uberblock_sync_list(vdev_t *svd, int svdcount, uberblock_t ub, int flags)
	{
	spa_t *spa = svd[0]->vdev_spa;
	zio_t *zio;
	uint64_t good_writes = 0;

	zio = zio_root(spa, NULL, NULL, flags);

	for (int v = 0; v < svdcount; v++)
	vdev_uberblock_sync(zio, &good_writes, ub, svd[v], flags);

	(void) zio_wait(zio);

	/*
	* Flush the uberblocks to disk. This ensures that the odd labels
	* are no longer needed (because the new uberblocks and the even
	* labels are safely on disk), so it is safe to overwrite them.
	*/
	zio = zio_root(spa, NULL, NULL, flags);

	for (int v = 0; v < svdcount; v++) {
	if (vdev_writeable(svd[v])) {
	zio_flush(zio, svd[v]);
	}
	}

	(void) zio_wait(zio);

	return (good_writes >= 1 ? 0 : EIO);
	}

	/*
	* On success, increment the count of good writes for our top-level vdev.
	*/
	static void
	vdev_label_sync_done(zio_t *zio)
	{
	uint64_t *good_writes = zio->io_private;

	if (zio->io_error == 0)
	atomic_inc_64(good_writes);
	}

	/*
	* If there weren't enough good writes, indicate failure to the parent.
	*/
	static void
	vdev_label_sync_top_done(zio_t *zio)
	{
	uint64_t *good_writes = zio->io_private;

	if (*good_writes == 0)
	zio->io_error = SET_ERROR(EIO);

	kmem_free(good_writes, sizeof (uint64_t));
	}

	/*
	* We ignore errors for log and cache devices, simply free the private data.
	*/
	static void
	vdev_label_sync_ignore_done(zio_t *zio)
	{
	kmem_free(zio->io_private, sizeof (uint64_t));
	}

	/*
	* Write all even or odd labels to all leaves of the specified vdev.
	*/
	static void
	vdev_label_sync(zio_t zio, uint64_t good_writes,
	vdev_t *vd, int l, uint64_t txg, int flags)
	{
	nvlist_t *label;
	vdev_phys_t *vp;
	abd_t *vp_abd;
	char *buf;
	size_t buflen;

	for (int c = 0; c < vd->vdev_children; c++) {
	vdev_label_sync(zio, good_writes,
	vd->vdev_child[c], l, txg, flags);
	}

	if (!vd->vdev_ops->vdev_op_leaf)
	return;

	if (!vdev_writeable(vd))
	return;

	/*
	* Generate a label describing the top-level config to which we belong.
	*/
	label = spa_config_generate(vd->vdev_spa, vd, txg, B_FALSE);

	vp_abd = abd_alloc_linear(sizeof (vdev_phys_t), B_TRUE);
	abd_zero(vp_abd, sizeof (vdev_phys_t));
	vp = abd_to_buf(vp_abd);

	buf = vp->vp_nvlist;
	buflen = sizeof (vp->vp_nvlist);

	if (!nvlist_pack(label, &buf, &buflen, NV_ENCODE_XDR, KM_SLEEP)) {
	for (; l < VDEV_LABELS; l += 2) {
	vdev_label_write(zio, vd, l, vp_abd,
	offsetof(vdev_label_t, vl_vdev_phys),
	sizeof (vdev_phys_t),
	vdev_label_sync_done, good_writes,
	flags \| ZIO_FLAG_DONT_PROPAGATE);
	}
	}

	abd_free(vp_abd);
	nvlist_free(label);
	}

	static int
	vdev_label_sync_list(spa_t *spa, int l, uint64_t txg, int flags)
	{
	list_t *dl = &spa->spa_config_dirty_list;
	vdev_t *vd;
	zio_t *zio;
	int error;

	/*
	* Write the new labels to disk.
	*/
	zio = zio_root(spa, NULL, NULL, flags);

	for (vd = list_head(dl); vd != NULL; vd = list_next(dl, vd)) {
	uint64_t *good_writes;

	ASSERT(!vd->vdev_ishole);

	good_writes = kmem_zalloc(sizeof (uint64_t), KM_SLEEP);
	zio_t *vio = zio_null(zio, spa, NULL,
	(vd->vdev_islog \|\| vd->vdev_aux != NULL) ?
	vdev_label_sync_ignore_done : vdev_label_sync_top_done,
	good_writes, flags);
	vdev_label_sync(vio, good_writes, vd, l, txg, flags);
	zio_nowait(vio);
	}

	error = zio_wait(zio);

	/*
	* Flush the new labels to disk.
	*/
	zio = zio_root(spa, NULL, NULL, flags);

	for (vd = list_head(dl); vd != NULL; vd = list_next(dl, vd))
	zio_flush(zio, vd);

	(void) zio_wait(zio);

	return (error);
	}

	/*
	* Sync the uberblock and any changes to the vdev configuration.
	*
	* The order of operations is carefully crafted to ensure that
	* if the system panics or loses power at any time, the state on disk
	* is still transactionally consistent. The in-line comments below
	* describe the failure semantics at each stage.
	*
	* Moreover, vdev_config_sync() is designed to be idempotent: if it fails
	* at any time, you can just call it again, and it will resume its work.
	*/
	int
	vdev_config_sync(vdev_t **svd, int svdcount, uint64_t txg)
	{
	spa_t *spa = svd[0]->vdev_spa;
	uberblock_t *ub = &spa->spa_uberblock;
	int error = 0;
	int flags = ZIO_FLAG_CONFIG_WRITER \| ZIO_FLAG_CANFAIL;

	ASSERT(svdcount != 0);
	retry:
	/*
	* Normally, we don't want to try too hard to write every label and
	* uberblock. If there is a flaky disk, we don't want the rest of the
	* sync process to block while we retry. But if we can't write a
	* single label out, we should retry with ZIO_FLAG_TRYHARD before
	* bailing out and declaring the pool faulted.
	*/
	if (error != 0) {
	if ((flags & ZIO_FLAG_TRYHARD) != 0)
	return (error);
	flags \|= ZIO_FLAG_TRYHARD;
	}

	ASSERT(ub->ub_txg <= txg);

	/*
	* If this isn't a resync due to I/O errors,
	* and nothing changed in this transaction group,
	* and the vdev configuration hasn't changed,
	* then there's nothing to do.
	*/
	if (ub->ub_txg < txg) {
	boolean_t changed = uberblock_update(ub, spa->spa_root_vdev,
	txg, spa->spa_mmp.mmp_delay);

	if (!changed && list_is_empty(&spa->spa_config_dirty_list))
	return (0);
	}

	if (txg > spa_freeze_txg(spa))
	return (0);

	ASSERT(txg <= spa->spa_final_txg);

	/*
	* Flush the write cache of every disk that's been written to
	* in this transaction group. This ensures that all blocks
	* written in this txg will be committed to stable storage
	* before any uberblock that references them.
	*/
	zio_t *zio = zio_root(spa, NULL, NULL, flags);

	for (vdev_t *vd =
	txg_list_head(&spa->spa_vdev_txg_list, TXG_CLEAN(txg)); vd != NULL;
	vd = txg_list_next(&spa->spa_vdev_txg_list, vd, TXG_CLEAN(txg)))
	zio_flush(zio, vd);

	(void) zio_wait(zio);

	/*
	* Sync out the even labels (L0, L2) for every dirty vdev. If the
	* system dies in the middle of this process, that's OK: all of the
	* even labels that made it to disk will be newer than any uberblock,
	* and will therefore be considered invalid. The odd labels (L1, L3),
	* which have not yet been touched, will still be valid. We flush
	* the new labels to disk to ensure that all even-label updates
	* are committed to stable storage before the uberblock update.
	*/
	if ((error = vdev_label_sync_list(spa, 0, txg, flags)) != 0) {
	if ((flags & ZIO_FLAG_TRYHARD) != 0) {
	zfs_dbgmsg("vdev_label_sync_list() returned error %d "
	"for pool '%s' when syncing out the even labels "
	"of dirty vdevs", error, spa_name(spa));
	}
	goto retry;
	}

	/*
	* Sync the uberblocks to all vdevs in svd[].
	* If the system dies in the middle of this step, there are two cases
	* to consider, and the on-disk state is consistent either way:
	*
	* (1) If none of the new uberblocks made it to disk, then the
	* previous uberblock will be the newest, and the odd labels
	* (which had not yet been touched) will be valid with respect
	* to that uberblock.
	*
	* (2) If one or more new uberblocks made it to disk, then they
	* will be the newest, and the even labels (which had all
	* been successfully committed) will be valid with respect
	* to the new uberblocks.
	*/
	if ((error = vdev_uberblock_sync_list(svd, svdcount, ub, flags)) != 0) {
	if ((flags & ZIO_FLAG_TRYHARD) != 0) {
	zfs_dbgmsg("vdev_uberblock_sync_list() returned error "
	"%d for pool '%s'", error, spa_name(spa));
	}
	goto retry;
	}

	if (spa_multihost(spa))
	mmp_update_uberblock(spa, ub);

	/*
	* Sync out odd labels for every dirty vdev. If the system dies
	* in the middle of this process, the even labels and the new
	* uberblocks will suffice to open the pool. The next time
	* the pool is opened, the first thing we'll do -- before any
	* user data is modified -- is mark every vdev dirty so that
	* all labels will be brought up to date. We flush the new labels
	* to disk to ensure that all odd-label updates are committed to
	* stable storage before the next transaction group begins.
	*/
	if ((error = vdev_label_sync_list(spa, 1, txg, flags)) != 0) {
	if ((flags & ZIO_FLAG_TRYHARD) != 0) {
	zfs_dbgmsg("vdev_label_sync_list() returned error %d "
	"for pool '%s' when syncing out the odd labels of "
	"dirty vdevs", error, spa_name(spa));
	}
	goto retry;
	}

	return (0);
	}
	Index: vendor-sys/openzfs/dist/module/zfs/vdev_removal.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/zfs/vdev_removal.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/zfs/vdev_removal.c (revision 366774)
	@@ -1,2339 +1,2359 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/

	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	- * Copyright (c) 2011, 2019 by Delphix. All rights reserved.
	+ * Copyright (c) 2011, 2020 by Delphix. All rights reserved.
	* Copyright (c) 2019, loli10K <ezomori.nozomu@gmail.com>. All rights reserved.
	*/

	#include <sys/zfs_context.h>
	#include <sys/spa_impl.h>
	#include <sys/dmu.h>
	#include <sys/dmu_tx.h>
	#include <sys/zap.h>
	#include <sys/vdev_impl.h>
	#include <sys/metaslab.h>
	#include <sys/metaslab_impl.h>
	#include <sys/uberblock_impl.h>
	#include <sys/txg.h>
	#include <sys/avl.h>
	#include <sys/bpobj.h>
	#include <sys/dsl_pool.h>
	#include <sys/dsl_synctask.h>
	#include <sys/dsl_dir.h>
	#include <sys/arc.h>
	#include <sys/zfeature.h>
	#include <sys/vdev_indirect_births.h>
	#include <sys/vdev_indirect_mapping.h>
	#include <sys/abd.h>
	#include <sys/vdev_initialize.h>
	#include <sys/vdev_trim.h>
	#include <sys/trace_zfs.h>

	/*
	* This file contains the necessary logic to remove vdevs from a
	* storage pool. Currently, the only devices that can be removed
	* are log, cache, and spare devices; and top level vdevs from a pool
	* w/o raidz or mirrors. (Note that members of a mirror can be removed
	* by the detach operation.)
	*
	* Log vdevs are removed by evacuating them and then turning the vdev
	* into a hole vdev while holding spa config locks.
	*
	* Top level vdevs are removed and converted into an indirect vdev via
	* a multi-step process:
	*
	* - Disable allocations from this device (spa_vdev_remove_top).
	*
	* - From a new thread (spa_vdev_remove_thread), copy data from
	* the removing vdev to a different vdev. The copy happens in open
	* context (spa_vdev_copy_impl) and issues a sync task
	* (vdev_mapping_sync) so the sync thread can update the partial
	* indirect mappings in core and on disk.
	*
	* - If a free happens during a removal, it is freed from the
	* removing vdev, and if it has already been copied, from the new
	* location as well (free_from_removing_vdev).
	*
	* - After the removal is completed, the copy thread converts the vdev
	* into an indirect vdev (vdev_remove_complete) before instructing
	* the sync thread to destroy the space maps and finish the removal
	* (spa_finish_removal).
	*/

	typedef struct vdev_copy_arg {
	metaslab_t *vca_msp;
	uint64_t vca_outstanding_bytes;
	uint64_t vca_read_error_bytes;
	uint64_t vca_write_error_bytes;
	kcondvar_t vca_cv;
	kmutex_t vca_lock;
	} vdev_copy_arg_t;

	/*
	* The maximum amount of memory we can use for outstanding i/o while
	* doing a device removal. This determines how much i/o we can have
	* in flight concurrently.
	*/
	int zfs_remove_max_copy_bytes = 64 * 1024 * 1024;

	/*
	* The largest contiguous segment that we will attempt to allocate when
	* removing a device. This can be no larger than SPA_MAXBLOCKSIZE. If
	* there is a performance problem with attempting to allocate large blocks,
	* consider decreasing this.
	*
	* See also the accessor function spa_remove_max_segment().
	*/
	int zfs_remove_max_segment = SPA_MAXBLOCKSIZE;

	/*
	* Ignore hard IO errors during device removal. When set if a device
	* encounters hard IO error during the removal process the removal will
	* not be cancelled. This can result in a normally recoverable block
	* becoming permanently damaged and is not recommended.
	*/
	int zfs_removal_ignore_errors = 0;

	/*
	* Allow a remap segment to span free chunks of at most this size. The main
	* impact of a larger span is that we will read and write larger, more
	* contiguous chunks, with more "unnecessary" data -- trading off bandwidth
	* for iops. The value here was chosen to align with
	* zfs_vdev_read_gap_limit, which is a similar concept when doing regular
	* reads (but there's no reason it has to be the same).
	*
	* Additionally, a higher span will have the following relatively minor
	* effects:
	* - the mapping will be smaller, since one entry can cover more allocated
	* segments
	* - more of the fragmentation in the removing device will be preserved
	* - we'll do larger allocations, which may fail and fall back on smaller
	* allocations
	*/
	int vdev_removal_max_span = 32 * 1024;

	/*
	* This is used by the test suite so that it can ensure that certain
	* actions happen while in the middle of a removal.
	*/
	int zfs_removal_suspend_progress = 0;

	#define VDEV_REMOVAL_ZAP_OBJS "lzap"

	static void spa_vdev_remove_thread(void *arg);
	static int spa_vdev_remove_cancel_impl(spa_t *spa);

	static void
	spa_sync_removing_state(spa_t spa, dmu_tx_t tx)
	{
	VERIFY0(zap_update(spa->spa_dsl_pool->dp_meta_objset,
	DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_REMOVING, sizeof (uint64_t),
	sizeof (spa->spa_removing_phys) / sizeof (uint64_t),
	&spa->spa_removing_phys, tx));
	}

	static nvlist_t *
	spa_nvlist_lookup_by_guid(nvlist_t **nvpp, int count, uint64_t target_guid)
	{
	for (int i = 0; i < count; i++) {
	uint64_t guid =
	fnvlist_lookup_uint64(nvpp[i], ZPOOL_CONFIG_GUID);

	if (guid == target_guid)
	return (nvpp[i]);
	}

	return (NULL);
	}

	static void
	spa_vdev_remove_aux(nvlist_t config, char name, nvlist_t **dev, int count,
	nvlist_t *dev_to_remove)
	{
	nvlist_t **newdev = NULL;

	if (count > 1)
	newdev = kmem_alloc((count - 1) * sizeof (void *), KM_SLEEP);

	for (int i = 0, j = 0; i < count; i++) {
	if (dev[i] == dev_to_remove)
	continue;
	VERIFY(nvlist_dup(dev[i], &newdev[j++], KM_SLEEP) == 0);
	}

	VERIFY(nvlist_remove(config, name, DATA_TYPE_NVLIST_ARRAY) == 0);
	VERIFY(nvlist_add_nvlist_array(config, name, newdev, count - 1) == 0);

	for (int i = 0; i < count - 1; i++)
	nvlist_free(newdev[i]);

	if (count > 1)
	kmem_free(newdev, (count - 1) * sizeof (void *));
	}

	static spa_vdev_removal_t *
	spa_vdev_removal_create(vdev_t *vd)
	{
	spa_vdev_removal_t svr = kmem_zalloc(sizeof (svr), KM_SLEEP);
	mutex_init(&svr->svr_lock, NULL, MUTEX_DEFAULT, NULL);
	cv_init(&svr->svr_cv, NULL, CV_DEFAULT, NULL);
	svr->svr_allocd_segs = range_tree_create(NULL, RANGE_SEG64, NULL, 0, 0);
	svr->svr_vdev_id = vd->vdev_id;

	for (int i = 0; i < TXG_SIZE; i++) {
	svr->svr_frees[i] = range_tree_create(NULL, RANGE_SEG64, NULL,
	0, 0);
	list_create(&svr->svr_new_segments[i],
	sizeof (vdev_indirect_mapping_entry_t),
	offsetof(vdev_indirect_mapping_entry_t, vime_node));
	}

	return (svr);
	}

	void
	spa_vdev_removal_destroy(spa_vdev_removal_t *svr)
	{
	for (int i = 0; i < TXG_SIZE; i++) {
	ASSERT0(svr->svr_bytes_done[i]);
	ASSERT0(svr->svr_max_offset_to_sync[i]);
	range_tree_destroy(svr->svr_frees[i]);
	list_destroy(&svr->svr_new_segments[i]);
	}

	range_tree_destroy(svr->svr_allocd_segs);
	mutex_destroy(&svr->svr_lock);
	cv_destroy(&svr->svr_cv);
	kmem_free(svr, sizeof (*svr));
	}

	/*
	* This is called as a synctask in the txg in which we will mark this vdev
	* as removing (in the config stored in the MOS).
	*
	* It begins the evacuation of a toplevel vdev by:
	* - initializing the spa_removing_phys which tracks this removal
	* - computing the amount of space to remove for accounting purposes
	* - dirtying all dbufs in the spa_config_object
	* - creating the spa_vdev_removal
	* - starting the spa_vdev_remove_thread
	*/
	static void
	vdev_remove_initiate_sync(void arg, dmu_tx_t tx)
	{
	int vdev_id = (uintptr_t)arg;
	spa_t *spa = dmu_tx_pool(tx)->dp_spa;
	vdev_t *vd = vdev_lookup_top(spa, vdev_id);
	vdev_indirect_config_t *vic = &vd->vdev_indirect_config;
	objset_t *mos = spa->spa_dsl_pool->dp_meta_objset;
	spa_vdev_removal_t *svr = NULL;
	uint64_t txg __maybe_unused = dmu_tx_get_txg(tx);

	ASSERT3P(vd->vdev_ops, !=, &vdev_raidz_ops);
	svr = spa_vdev_removal_create(vd);

	ASSERT(vd->vdev_removing);
	ASSERT3P(vd->vdev_indirect_mapping, ==, NULL);

	spa_feature_incr(spa, SPA_FEATURE_DEVICE_REMOVAL, tx);
	if (spa_feature_is_enabled(spa, SPA_FEATURE_OBSOLETE_COUNTS)) {
	/*
	* By activating the OBSOLETE_COUNTS feature, we prevent
	* the pool from being downgraded and ensure that the
	* refcounts are precise.
	*/
	spa_feature_incr(spa, SPA_FEATURE_OBSOLETE_COUNTS, tx);
	uint64_t one = 1;
	VERIFY0(zap_add(spa->spa_meta_objset, vd->vdev_top_zap,
	VDEV_TOP_ZAP_OBSOLETE_COUNTS_ARE_PRECISE, sizeof (one), 1,
	&one, tx));
	boolean_t are_precise __maybe_unused;
	ASSERT0(vdev_obsolete_counts_are_precise(vd, &are_precise));
	ASSERT3B(are_precise, ==, B_TRUE);
	}

	vic->vic_mapping_object = vdev_indirect_mapping_alloc(mos, tx);
	vd->vdev_indirect_mapping =
	vdev_indirect_mapping_open(mos, vic->vic_mapping_object);
	vic->vic_births_object = vdev_indirect_births_alloc(mos, tx);
	vd->vdev_indirect_births =
	vdev_indirect_births_open(mos, vic->vic_births_object);
	spa->spa_removing_phys.sr_removing_vdev = vd->vdev_id;
	spa->spa_removing_phys.sr_start_time = gethrestime_sec();
	spa->spa_removing_phys.sr_end_time = 0;
	spa->spa_removing_phys.sr_state = DSS_SCANNING;
	spa->spa_removing_phys.sr_to_copy = 0;
	spa->spa_removing_phys.sr_copied = 0;

	/*
	* Note: We can't use vdev_stat's vs_alloc for sr_to_copy, because
	* there may be space in the defer tree, which is free, but still
	* counted in vs_alloc.
	*/
	for (uint64_t i = 0; i < vd->vdev_ms_count; i++) {
	metaslab_t *ms = vd->vdev_ms[i];
	if (ms->ms_sm == NULL)
	continue;

	spa->spa_removing_phys.sr_to_copy +=
	metaslab_allocated_space(ms);

	/*
	* Space which we are freeing this txg does not need to
	* be copied.
	*/
	spa->spa_removing_phys.sr_to_copy -=
	range_tree_space(ms->ms_freeing);

	ASSERT0(range_tree_space(ms->ms_freed));
	for (int t = 0; t < TXG_SIZE; t++)
	ASSERT0(range_tree_space(ms->ms_allocating[t]));
	}

	/*
	* Sync tasks are called before metaslab_sync(), so there should
	* be no already-synced metaslabs in the TXG_CLEAN list.
	*/
	ASSERT3P(txg_list_head(&vd->vdev_ms_list, TXG_CLEAN(txg)), ==, NULL);

	spa_sync_removing_state(spa, tx);

	/*
	* All blocks that we need to read the most recent mapping must be
	* stored on concrete vdevs. Therefore, we must dirty anything that
	* is read before spa_remove_init(). Specifically, the
	* spa_config_object. (Note that although we already modified the
	* spa_config_object in spa_sync_removing_state, that may not have
	* modified all blocks of the object.)
	*/
	dmu_object_info_t doi;
	VERIFY0(dmu_object_info(mos, DMU_POOL_DIRECTORY_OBJECT, &doi));
	for (uint64_t offset = 0; offset < doi.doi_max_offset; ) {
	dmu_buf_t *dbuf;
	VERIFY0(dmu_buf_hold(mos, DMU_POOL_DIRECTORY_OBJECT,
	offset, FTAG, &dbuf, 0));
	dmu_buf_will_dirty(dbuf, tx);
	offset += dbuf->db_size;
	dmu_buf_rele(dbuf, FTAG);
	}

	/*
	* Now that we've allocated the im_object, dirty the vdev to ensure
	* that the object gets written to the config on disk.
	*/
	vdev_config_dirty(vd);

	zfs_dbgmsg("starting removal thread for vdev %llu (%px) in txg %llu "
	"im_obj=%llu", vd->vdev_id, vd, dmu_tx_get_txg(tx),
	vic->vic_mapping_object);

	spa_history_log_internal(spa, "vdev remove started", tx,
	"%s vdev %llu %s", spa_name(spa), (u_longlong_t)vd->vdev_id,
	(vd->vdev_path != NULL) ? vd->vdev_path : "-");
	/*
	* Setting spa_vdev_removal causes subsequent frees to call
	* free_from_removing_vdev(). Note that we don't need any locking
	* because we are the sync thread, and metaslab_free_impl() is only
	* called from syncing context (potentially from a zio taskq thread,
	* but in any case only when there are outstanding free i/os, which
	* there are not).
	*/
	ASSERT3P(spa->spa_vdev_removal, ==, NULL);
	spa->spa_vdev_removal = svr;
	svr->svr_thread = thread_create(NULL, 0,
	spa_vdev_remove_thread, spa, 0, &p0, TS_RUN, minclsyspri);
	}

	/*
	* When we are opening a pool, we must read the mapping for each
	* indirect vdev in order from most recently removed to least
	* recently removed. We do this because the blocks for the mapping
	* of older indirect vdevs may be stored on more recently removed vdevs.
	* In order to read each indirect mapping object, we must have
	* initialized all more recently removed vdevs.
	*/
	int
	spa_remove_init(spa_t *spa)
	{
	int error;

	error = zap_lookup(spa->spa_dsl_pool->dp_meta_objset,
	DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_REMOVING, sizeof (uint64_t),
	sizeof (spa->spa_removing_phys) / sizeof (uint64_t),
	&spa->spa_removing_phys);

	if (error == ENOENT) {
	spa->spa_removing_phys.sr_state = DSS_NONE;
	spa->spa_removing_phys.sr_removing_vdev = -1;
	spa->spa_removing_phys.sr_prev_indirect_vdev = -1;
	spa->spa_indirect_vdevs_loaded = B_TRUE;
	return (0);
	} else if (error != 0) {
	return (error);
	}

	if (spa->spa_removing_phys.sr_state == DSS_SCANNING) {
	/*
	* We are currently removing a vdev. Create and
	* initialize a spa_vdev_removal_t from the bonus
	* buffer of the removing vdevs vdev_im_object, and
	* initialize its partial mapping.
	*/
	spa_config_enter(spa, SCL_STATE, FTAG, RW_READER);
	vdev_t *vd = vdev_lookup_top(spa,
	spa->spa_removing_phys.sr_removing_vdev);

	if (vd == NULL) {
	spa_config_exit(spa, SCL_STATE, FTAG);
	return (EINVAL);
	}

	vdev_indirect_config_t *vic = &vd->vdev_indirect_config;

	ASSERT(vdev_is_concrete(vd));
	spa_vdev_removal_t *svr = spa_vdev_removal_create(vd);
	ASSERT3U(svr->svr_vdev_id, ==, vd->vdev_id);
	ASSERT(vd->vdev_removing);

	vd->vdev_indirect_mapping = vdev_indirect_mapping_open(
	spa->spa_meta_objset, vic->vic_mapping_object);
	vd->vdev_indirect_births = vdev_indirect_births_open(
	spa->spa_meta_objset, vic->vic_births_object);
	spa_config_exit(spa, SCL_STATE, FTAG);

	spa->spa_vdev_removal = svr;
	}

	spa_config_enter(spa, SCL_STATE, FTAG, RW_READER);
	uint64_t indirect_vdev_id =
	spa->spa_removing_phys.sr_prev_indirect_vdev;
	while (indirect_vdev_id != UINT64_MAX) {
	vdev_t *vd = vdev_lookup_top(spa, indirect_vdev_id);
	vdev_indirect_config_t *vic = &vd->vdev_indirect_config;

	ASSERT3P(vd->vdev_ops, ==, &vdev_indirect_ops);
	vd->vdev_indirect_mapping = vdev_indirect_mapping_open(
	spa->spa_meta_objset, vic->vic_mapping_object);
	vd->vdev_indirect_births = vdev_indirect_births_open(
	spa->spa_meta_objset, vic->vic_births_object);

	indirect_vdev_id = vic->vic_prev_indirect_vdev;
	}
	spa_config_exit(spa, SCL_STATE, FTAG);

	/*
	* Now that we've loaded all the indirect mappings, we can allow
	* reads from other blocks (e.g. via predictive prefetch).
	*/
	spa->spa_indirect_vdevs_loaded = B_TRUE;
	return (0);
	}

	void
	spa_restart_removal(spa_t *spa)
	{
	spa_vdev_removal_t *svr = spa->spa_vdev_removal;

	if (svr == NULL)
	return;

	/*
	* In general when this function is called there is no
	* removal thread running. The only scenario where this
	* is not true is during spa_import() where this function
	* is called twice [once from spa_import_impl() and
	* spa_async_resume()]. Thus, in the scenario where we
	* import a pool that has an ongoing removal we don't
	* want to spawn a second thread.
	*/
	if (svr->svr_thread != NULL)
	return;

	if (!spa_writeable(spa))
	return;

	zfs_dbgmsg("restarting removal of %llu", svr->svr_vdev_id);
	svr->svr_thread = thread_create(NULL, 0, spa_vdev_remove_thread, spa,
	0, &p0, TS_RUN, minclsyspri);
	}

	/*
	* Process freeing from a device which is in the middle of being removed.
	* We must handle this carefully so that we attempt to copy freed data,
	* and we correctly free already-copied data.
	*/
	void
	free_from_removing_vdev(vdev_t *vd, uint64_t offset, uint64_t size)
	{
	spa_t *spa = vd->vdev_spa;
	spa_vdev_removal_t *svr = spa->spa_vdev_removal;
	vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping;
	uint64_t txg = spa_syncing_txg(spa);
	uint64_t max_offset_yet = 0;

	ASSERT(vd->vdev_indirect_config.vic_mapping_object != 0);
	ASSERT3U(vd->vdev_indirect_config.vic_mapping_object, ==,
	vdev_indirect_mapping_object(vim));
	ASSERT3U(vd->vdev_id, ==, svr->svr_vdev_id);

	mutex_enter(&svr->svr_lock);

	/*
	* Remove the segment from the removing vdev's spacemap. This
	* ensures that we will not attempt to copy this space (if the
	* removal thread has not yet visited it), and also ensures
	* that we know what is actually allocated on the new vdevs
	* (needed if we cancel the removal).
	*
	* Note: we must do the metaslab_free_concrete() with the svr_lock
	* held, so that the remove_thread can not load this metaslab and then
	* visit this offset between the time that we metaslab_free_concrete()
	* and when we check to see if it has been visited.
	*
	* Note: The checkpoint flag is set to false as having/taking
	* a checkpoint and removing a device can't happen at the same
	* time.
	*/
	ASSERT(!spa_has_checkpoint(spa));
	metaslab_free_concrete(vd, offset, size, B_FALSE);

	uint64_t synced_size = 0;
	uint64_t synced_offset = 0;
	uint64_t max_offset_synced = vdev_indirect_mapping_max_offset(vim);
	if (offset < max_offset_synced) {
	/*
	* The mapping for this offset is already on disk.
	* Free from the new location.
	*
	* Note that we use svr_max_synced_offset because it is
	* updated atomically with respect to the in-core mapping.
	* By contrast, vim_max_offset is not.
	*
	* This block may be split between a synced entry and an
	* in-flight or unvisited entry. Only process the synced
	* portion of it here.
	*/
	synced_size = MIN(size, max_offset_synced - offset);
	synced_offset = offset;

	ASSERT3U(max_offset_yet, <=, max_offset_synced);
	max_offset_yet = max_offset_synced;

	DTRACE_PROBE3(remove__free__synced,
	spa_t *, spa,
	uint64_t, offset,
	uint64_t, synced_size);

	size -= synced_size;
	offset += synced_size;
	}

	/*
	* Look at all in-flight txgs starting from the currently syncing one
	* and see if a section of this free is being copied. By starting from
	* this txg and iterating forward, we might find that this region
	* was copied in two different txgs and handle it appropriately.
	*/
	for (int i = 0; i < TXG_CONCURRENT_STATES; i++) {
	int txgoff = (txg + i) & TXG_MASK;
	if (size > 0 && offset < svr->svr_max_offset_to_sync[txgoff]) {
	/*
	* The mapping for this offset is in flight, and
	* will be synced in txg+i.
	*/
	uint64_t inflight_size = MIN(size,
	svr->svr_max_offset_to_sync[txgoff] - offset);

	DTRACE_PROBE4(remove__free__inflight,
	spa_t *, spa,
	uint64_t, offset,
	uint64_t, inflight_size,
	uint64_t, txg + i);

	/*
	* We copy data in order of increasing offset.
	* Therefore the max_offset_to_sync[] must increase
	* (or be zero, indicating that nothing is being
	* copied in that txg).
	*/
	if (svr->svr_max_offset_to_sync[txgoff] != 0) {
	ASSERT3U(svr->svr_max_offset_to_sync[txgoff],
	>=, max_offset_yet);
	max_offset_yet =
	svr->svr_max_offset_to_sync[txgoff];
	}

	/*
	* We've already committed to copying this segment:
	* we have allocated space elsewhere in the pool for
	* it and have an IO outstanding to copy the data. We
	* cannot free the space before the copy has
	* completed, or else the copy IO might overwrite any
	* new data. To free that space, we record the
	* segment in the appropriate svr_frees tree and free
	* the mapped space later, in the txg where we have
	* completed the copy and synced the mapping (see
	* vdev_mapping_sync).
	*/
	range_tree_add(svr->svr_frees[txgoff],
	offset, inflight_size);
	size -= inflight_size;
	offset += inflight_size;

	/*
	* This space is already accounted for as being
	* done, because it is being copied in txg+i.
	* However, if i!=0, then it is being copied in
	* a future txg. If we crash after this txg
	* syncs but before txg+i syncs, then the space
	* will be free. Therefore we must account
	* for the space being done in this txg
	* (when it is freed) rather than the future txg
	* (when it will be copied).
	*/
	ASSERT3U(svr->svr_bytes_done[txgoff], >=,
	inflight_size);
	svr->svr_bytes_done[txgoff] -= inflight_size;
	svr->svr_bytes_done[txg & TXG_MASK] += inflight_size;
	}
	}
	ASSERT0(svr->svr_max_offset_to_sync[TXG_CLEAN(txg) & TXG_MASK]);

	if (size > 0) {
	/*
	* The copy thread has not yet visited this offset. Ensure
	* that it doesn't.
	*/

	DTRACE_PROBE3(remove__free__unvisited,
	spa_t *, spa,
	uint64_t, offset,
	uint64_t, size);

	if (svr->svr_allocd_segs != NULL)
	range_tree_clear(svr->svr_allocd_segs, offset, size);

	/*
	* Since we now do not need to copy this data, for
	* accounting purposes we have done our job and can count
	* it as completed.
	*/
	svr->svr_bytes_done[txg & TXG_MASK] += size;
	}
	mutex_exit(&svr->svr_lock);

	/*
	* Now that we have dropped svr_lock, process the synced portion
	* of this free.
	*/
	if (synced_size > 0) {
	vdev_indirect_mark_obsolete(vd, synced_offset, synced_size);

	/*
	* Note: this can only be called from syncing context,
	* and the vdev_indirect_mapping is only changed from the
	* sync thread, so we don't need svr_lock while doing
	* metaslab_free_impl_cb.
	*/
	boolean_t checkpoint = B_FALSE;
	vdev_indirect_ops.vdev_op_remap(vd, synced_offset, synced_size,
	metaslab_free_impl_cb, &checkpoint);
	}
	}

	/*
	* Stop an active removal and update the spa_removing phys.
	*/
	static void
	spa_finish_removal(spa_t spa, dsl_scan_state_t state, dmu_tx_t tx)
	{
	spa_vdev_removal_t *svr = spa->spa_vdev_removal;
	ASSERT3U(dmu_tx_get_txg(tx), ==, spa_syncing_txg(spa));

	/* Ensure the removal thread has completed before we free the svr. */
	spa_vdev_remove_suspend(spa);

	ASSERT(state == DSS_FINISHED \|\| state == DSS_CANCELED);

	if (state == DSS_FINISHED) {
	spa_removing_phys_t *srp = &spa->spa_removing_phys;
	vdev_t *vd = vdev_lookup_top(spa, svr->svr_vdev_id);
	vdev_indirect_config_t *vic = &vd->vdev_indirect_config;

	if (srp->sr_prev_indirect_vdev != -1) {
	vdev_t *pvd;
	pvd = vdev_lookup_top(spa,
	srp->sr_prev_indirect_vdev);
	ASSERT3P(pvd->vdev_ops, ==, &vdev_indirect_ops);
	}

	vic->vic_prev_indirect_vdev = srp->sr_prev_indirect_vdev;
	srp->sr_prev_indirect_vdev = vd->vdev_id;
	}
	spa->spa_removing_phys.sr_state = state;
	spa->spa_removing_phys.sr_end_time = gethrestime_sec();

	spa->spa_vdev_removal = NULL;
	spa_vdev_removal_destroy(svr);

	spa_sync_removing_state(spa, tx);
	spa_notify_waiters(spa);

	vdev_config_dirty(spa->spa_root_vdev);
	}

	static void
	free_mapped_segment_cb(void *arg, uint64_t offset, uint64_t size)
	{
	vdev_t *vd = arg;
	vdev_indirect_mark_obsolete(vd, offset, size);
	boolean_t checkpoint = B_FALSE;
	vdev_indirect_ops.vdev_op_remap(vd, offset, size,
	metaslab_free_impl_cb, &checkpoint);
	}

	/*
	* On behalf of the removal thread, syncs an incremental bit more of
	* the indirect mapping to disk and updates the in-memory mapping.
	* Called as a sync task in every txg that the removal thread makes progress.
	*/
	static void
	vdev_mapping_sync(void arg, dmu_tx_t tx)
	{
	spa_vdev_removal_t *svr = arg;
	spa_t *spa = dmu_tx_pool(tx)->dp_spa;
	vdev_t *vd = vdev_lookup_top(spa, svr->svr_vdev_id);
	vdev_indirect_config_t *vic __maybe_unused = &vd->vdev_indirect_config;
	uint64_t txg = dmu_tx_get_txg(tx);
	vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping;

	ASSERT(vic->vic_mapping_object != 0);
	ASSERT3U(txg, ==, spa_syncing_txg(spa));

	vdev_indirect_mapping_add_entries(vim,
	&svr->svr_new_segments[txg & TXG_MASK], tx);
	vdev_indirect_births_add_entry(vd->vdev_indirect_births,
	vdev_indirect_mapping_max_offset(vim), dmu_tx_get_txg(tx), tx);

	/*
	* Free the copied data for anything that was freed while the
	* mapping entries were in flight.
	*/
	mutex_enter(&svr->svr_lock);
	range_tree_vacate(svr->svr_frees[txg & TXG_MASK],
	free_mapped_segment_cb, vd);
	ASSERT3U(svr->svr_max_offset_to_sync[txg & TXG_MASK], >=,
	vdev_indirect_mapping_max_offset(vim));
	svr->svr_max_offset_to_sync[txg & TXG_MASK] = 0;
	mutex_exit(&svr->svr_lock);

	spa_sync_removing_state(spa, tx);
	}

	typedef struct vdev_copy_segment_arg {
	spa_t *vcsa_spa;
	dva_t *vcsa_dest_dva;
	uint64_t vcsa_txg;
	range_tree_t *vcsa_obsolete_segs;
	} vdev_copy_segment_arg_t;

	static void
	unalloc_seg(void *arg, uint64_t start, uint64_t size)
	{
	vdev_copy_segment_arg_t *vcsa = arg;
	spa_t *spa = vcsa->vcsa_spa;
	blkptr_t bp = { { { {0} } } };

	BP_SET_BIRTH(&bp, TXG_INITIAL, TXG_INITIAL);
	BP_SET_LSIZE(&bp, size);
	BP_SET_PSIZE(&bp, size);
	BP_SET_COMPRESS(&bp, ZIO_COMPRESS_OFF);
	BP_SET_CHECKSUM(&bp, ZIO_CHECKSUM_OFF);
	BP_SET_TYPE(&bp, DMU_OT_NONE);
	BP_SET_LEVEL(&bp, 0);
	BP_SET_DEDUP(&bp, 0);
	BP_SET_BYTEORDER(&bp, ZFS_HOST_BYTEORDER);

	DVA_SET_VDEV(&bp.blk_dva[0], DVA_GET_VDEV(vcsa->vcsa_dest_dva));
	DVA_SET_OFFSET(&bp.blk_dva[0],
	DVA_GET_OFFSET(vcsa->vcsa_dest_dva) + start);
	DVA_SET_ASIZE(&bp.blk_dva[0], size);

	zio_free(spa, vcsa->vcsa_txg, &bp);
	}

	/*
	* All reads and writes associated with a call to spa_vdev_copy_segment()
	* are done.
	*/
	static void
	spa_vdev_copy_segment_done(zio_t *zio)
	{
	vdev_copy_segment_arg_t *vcsa = zio->io_private;

	range_tree_vacate(vcsa->vcsa_obsolete_segs,
	unalloc_seg, vcsa);
	range_tree_destroy(vcsa->vcsa_obsolete_segs);
	kmem_free(vcsa, sizeof (*vcsa));

	spa_config_exit(zio->io_spa, SCL_STATE, zio->io_spa);
	}

	/*
	* The write of the new location is done.
	*/
	static void
	spa_vdev_copy_segment_write_done(zio_t *zio)
	{
	vdev_copy_arg_t *vca = zio->io_private;

	abd_free(zio->io_abd);

	mutex_enter(&vca->vca_lock);
	vca->vca_outstanding_bytes -= zio->io_size;

	if (zio->io_error != 0)
	vca->vca_write_error_bytes += zio->io_size;

	cv_signal(&vca->vca_cv);
	mutex_exit(&vca->vca_lock);
	}

	/*
	* The read of the old location is done. The parent zio is the write to
	* the new location. Allow it to start.
	*/
	static void
	spa_vdev_copy_segment_read_done(zio_t *zio)
	{
	vdev_copy_arg_t *vca = zio->io_private;

	if (zio->io_error != 0) {
	mutex_enter(&vca->vca_lock);
	vca->vca_read_error_bytes += zio->io_size;
	mutex_exit(&vca->vca_lock);
	}

	zio_nowait(zio_unique_parent(zio));
	}

	/*
	* If the old and new vdevs are mirrors, we will read both sides of the old
	* mirror, and write each copy to the corresponding side of the new mirror.
	* If the old and new vdevs have a different number of children, we will do
	* this as best as possible. Since we aren't verifying checksums, this
	* ensures that as long as there's a good copy of the data, we'll have a
	* good copy after the removal, even if there's silent damage to one side
	* of the mirror. If we're removing a mirror that has some silent damage,
	* we'll have exactly the same damage in the new location (assuming that
	* the new location is also a mirror).
	*
	* We accomplish this by creating a tree of zio_t's, with as many writes as
	* there are "children" of the new vdev (a non-redundant vdev counts as one
	* child, a 2-way mirror has 2 children, etc). Each write has an associated
	* read from a child of the old vdev. Typically there will be the same
	* number of children of the old and new vdevs. However, if there are more
	* children of the new vdev, some child(ren) of the old vdev will be issued
	* multiple reads. If there are more children of the old vdev, some copies
	* will be dropped.
	*
	* For example, the tree of zio_t's for a 2-way mirror is:
	*
	* null
	* / \
	* write(new vdev, child 0) write(new vdev, child 1)
	* \| \|
	* read(old vdev, child 0) read(old vdev, child 1)
	*
	* Child zio's complete before their parents complete. However, zio's
	* created with zio_vdev_child_io() may be issued before their children
	* complete. In this case we need to make sure that the children (reads)
	* complete before the parents (writes) are issued. We do this by not
	* calling zio_nowait() on each write until its corresponding read has
	* completed.
	*
	* The spa_config_lock must be held while zio's created by
	* zio_vdev_child_io() are in progress, to ensure that the vdev tree does
	* not change (e.g. due to a concurrent "zpool attach/detach"). The "null"
	* zio is needed to release the spa_config_lock after all the reads and
	* writes complete. (Note that we can't grab the config lock for each read,
	* because it is not reentrant - we could deadlock with a thread waiting
	* for a write lock.)
	*/
	static void
	spa_vdev_copy_one_child(vdev_copy_arg_t vca, zio_t nzio,
	vdev_t *source_vd, uint64_t source_offset,
	vdev_t *dest_child_vd, uint64_t dest_offset, int dest_id, uint64_t size)
	{
	ASSERT3U(spa_config_held(nzio->io_spa, SCL_ALL, RW_READER), !=, 0);

	/*
	* If the destination child in unwritable then there is no point
	* in issuing the source reads which cannot be written.
	*/
	if (!vdev_writeable(dest_child_vd))
	return;

	mutex_enter(&vca->vca_lock);
	vca->vca_outstanding_bytes += size;
	mutex_exit(&vca->vca_lock);

	abd_t *abd = abd_alloc_for_io(size, B_FALSE);

	vdev_t *source_child_vd = NULL;
	if (source_vd->vdev_ops == &vdev_mirror_ops && dest_id != -1) {
	/*
	* Source and dest are both mirrors. Copy from the same
	* child id as we are copying to (wrapping around if there
	* are more dest children than source children). If the
	* preferred source child is unreadable select another.
	*/
	for (int i = 0; i < source_vd->vdev_children; i++) {
	source_child_vd = source_vd->vdev_child[
	(dest_id + i) % source_vd->vdev_children];
	if (vdev_readable(source_child_vd))
	break;
	}
	} else {
	source_child_vd = source_vd;
	}

	/*
	* There should always be at least one readable source child or
	* the pool would be in a suspended state. Somehow selecting an
	* unreadable child would result in IO errors, the removal process
	* being cancelled, and the pool reverting to its pre-removal state.
	*/
	ASSERT3P(source_child_vd, !=, NULL);

	zio_t *write_zio = zio_vdev_child_io(nzio, NULL,
	dest_child_vd, dest_offset, abd, size,
	ZIO_TYPE_WRITE, ZIO_PRIORITY_REMOVAL,
	ZIO_FLAG_CANFAIL,
	spa_vdev_copy_segment_write_done, vca);

	zio_nowait(zio_vdev_child_io(write_zio, NULL,
	source_child_vd, source_offset, abd, size,
	ZIO_TYPE_READ, ZIO_PRIORITY_REMOVAL,
	ZIO_FLAG_CANFAIL,
	spa_vdev_copy_segment_read_done, vca));
	}

	/*
	* Allocate a new location for this segment, and create the zio_t's to
	* read from the old location and write to the new location.
	*/
	static int
	spa_vdev_copy_segment(vdev_t vd, range_tree_t segs,
	uint64_t maxalloc, uint64_t txg,
	vdev_copy_arg_t vca, zio_alloc_list_t zal)
	{
	metaslab_group_t *mg = vd->vdev_mg;
	spa_t *spa = vd->vdev_spa;
	spa_vdev_removal_t *svr = spa->spa_vdev_removal;
	vdev_indirect_mapping_entry_t *entry;
	dva_t dst = {{ 0 }};
	uint64_t start = range_tree_min(segs);
	ASSERT0(P2PHASE(start, 1 << spa->spa_min_ashift));

	ASSERT3U(maxalloc, <=, SPA_MAXBLOCKSIZE);
	ASSERT0(P2PHASE(maxalloc, 1 << spa->spa_min_ashift));

	uint64_t size = range_tree_span(segs);
	if (range_tree_span(segs) > maxalloc) {
	/*
	* We can't allocate all the segments. Prefer to end
	* the allocation at the end of a segment, thus avoiding
	* additional split blocks.
	*/
	range_seg_max_t search;
	zfs_btree_index_t where;
	rs_set_start(&search, segs, start + maxalloc);
	rs_set_end(&search, segs, start + maxalloc);
	(void) zfs_btree_find(&segs->rt_root, &search, &where);
	range_seg_t *rs = zfs_btree_prev(&segs->rt_root, &where,
	&where);
	if (rs != NULL) {
	size = rs_get_end(rs, segs) - start;
	} else {
	/*
	* There are no segments that end before maxalloc.
	* I.e. the first segment is larger than maxalloc,
	* so we must split it.
	*/
	size = maxalloc;
	}
	}
	ASSERT3U(size, <=, maxalloc);
	ASSERT0(P2PHASE(size, 1 << spa->spa_min_ashift));

	/*
	* An allocation class might not have any remaining vdevs or space
	*/
	metaslab_class_t *mc = mg->mg_class;
	if (mc != spa_normal_class(spa) && mc->mc_groups <= 1)
	mc = spa_normal_class(spa);
	int error = metaslab_alloc_dva(spa, mc, size, &dst, 0, NULL, txg, 0,
	zal, 0);
	if (error == ENOSPC && mc != spa_normal_class(spa)) {
	error = metaslab_alloc_dva(spa, spa_normal_class(spa), size,
	&dst, 0, NULL, txg, 0, zal, 0);
	}
	if (error != 0)
	return (error);

	/*
	* Determine the ranges that are not actually needed. Offsets are
	* relative to the start of the range to be copied (i.e. relative to the
	* local variable "start").
	*/
	range_tree_t *obsolete_segs = range_tree_create(NULL, RANGE_SEG64, NULL,
	0, 0);

	zfs_btree_index_t where;
	range_seg_t *rs = zfs_btree_first(&segs->rt_root, &where);
	ASSERT3U(rs_get_start(rs, segs), ==, start);
	uint64_t prev_seg_end = rs_get_end(rs, segs);
	while ((rs = zfs_btree_next(&segs->rt_root, &where, &where)) != NULL) {
	if (rs_get_start(rs, segs) >= start + size) {
	break;
	} else {
	range_tree_add(obsolete_segs,
	prev_seg_end - start,
	rs_get_start(rs, segs) - prev_seg_end);
	}
	prev_seg_end = rs_get_end(rs, segs);
	}
	/* We don't end in the middle of an obsolete range */
	ASSERT3U(start + size, <=, prev_seg_end);

	range_tree_clear(segs, start, size);

	/*
	* We can't have any padding of the allocated size, otherwise we will
	* misunderstand what's allocated, and the size of the mapping. We
	* prevent padding by ensuring that all devices in the pool have the
	* same ashift, and the allocation size is a multiple of the ashift.
	*/
	VERIFY3U(DVA_GET_ASIZE(&dst), ==, size);

	entry = kmem_zalloc(sizeof (vdev_indirect_mapping_entry_t), KM_SLEEP);
	DVA_MAPPING_SET_SRC_OFFSET(&entry->vime_mapping, start);
	entry->vime_mapping.vimep_dst = dst;
	if (spa_feature_is_enabled(spa, SPA_FEATURE_OBSOLETE_COUNTS)) {
	entry->vime_obsolete_count = range_tree_space(obsolete_segs);
	}

	vdev_copy_segment_arg_t vcsa = kmem_zalloc(sizeof (vcsa), KM_SLEEP);
	vcsa->vcsa_dest_dva = &entry->vime_mapping.vimep_dst;
	vcsa->vcsa_obsolete_segs = obsolete_segs;
	vcsa->vcsa_spa = spa;
	vcsa->vcsa_txg = txg;

	/*
	* See comment before spa_vdev_copy_one_child().
	*/
	spa_config_enter(spa, SCL_STATE, spa, RW_READER);
	zio_t *nzio = zio_null(spa->spa_txg_zio[txg & TXG_MASK], spa, NULL,
	spa_vdev_copy_segment_done, vcsa, 0);
	vdev_t *dest_vd = vdev_lookup_top(spa, DVA_GET_VDEV(&dst));
	if (dest_vd->vdev_ops == &vdev_mirror_ops) {
	for (int i = 0; i < dest_vd->vdev_children; i++) {
	vdev_t *child = dest_vd->vdev_child[i];
	spa_vdev_copy_one_child(vca, nzio, vd, start,
	child, DVA_GET_OFFSET(&dst), i, size);
	}
	} else {
	spa_vdev_copy_one_child(vca, nzio, vd, start,
	dest_vd, DVA_GET_OFFSET(&dst), -1, size);
	}
	zio_nowait(nzio);

	list_insert_tail(&svr->svr_new_segments[txg & TXG_MASK], entry);
	ASSERT3U(start + size, <=, vd->vdev_ms_count << vd->vdev_ms_shift);
	vdev_dirty(vd, 0, NULL, txg);

	return (0);
	}

	/*
	* Complete the removal of a toplevel vdev. This is called as a
	* synctask in the same txg that we will sync out the new config (to the
	* MOS object) which indicates that this vdev is indirect.
	*/
	static void
	vdev_remove_complete_sync(void arg, dmu_tx_t tx)
	{
	spa_vdev_removal_t *svr = arg;
	spa_t *spa = dmu_tx_pool(tx)->dp_spa;
	vdev_t *vd = vdev_lookup_top(spa, svr->svr_vdev_id);

	ASSERT3P(vd->vdev_ops, ==, &vdev_indirect_ops);

	for (int i = 0; i < TXG_SIZE; i++) {
	ASSERT0(svr->svr_bytes_done[i]);
	}

	ASSERT3U(spa->spa_removing_phys.sr_copied, ==,
	spa->spa_removing_phys.sr_to_copy);

	vdev_destroy_spacemaps(vd, tx);

	/* destroy leaf zaps, if any */
	ASSERT3P(svr->svr_zaplist, !=, NULL);
	for (nvpair_t *pair = nvlist_next_nvpair(svr->svr_zaplist, NULL);
	pair != NULL;
	pair = nvlist_next_nvpair(svr->svr_zaplist, pair)) {
	vdev_destroy_unlink_zap(vd, fnvpair_value_uint64(pair), tx);
	}
	fnvlist_free(svr->svr_zaplist);

	spa_finish_removal(dmu_tx_pool(tx)->dp_spa, DSS_FINISHED, tx);
	/* vd->vdev_path is not available here */
	spa_history_log_internal(spa, "vdev remove completed", tx,
	"%s vdev %llu", spa_name(spa), (u_longlong_t)vd->vdev_id);
	}

	static void
	vdev_remove_enlist_zaps(vdev_t vd, nvlist_t zlist)
	{
	ASSERT3P(zlist, !=, NULL);
	ASSERT3P(vd->vdev_ops, !=, &vdev_raidz_ops);

	if (vd->vdev_leaf_zap != 0) {
	char zkey[32];
	(void) snprintf(zkey, sizeof (zkey), "%s-%llu",
	VDEV_REMOVAL_ZAP_OBJS, (u_longlong_t)vd->vdev_leaf_zap);
	fnvlist_add_uint64(zlist, zkey, vd->vdev_leaf_zap);
	}

	for (uint64_t id = 0; id < vd->vdev_children; id++) {
	vdev_remove_enlist_zaps(vd->vdev_child[id], zlist);
	}
	}

	static void
	vdev_remove_replace_with_indirect(vdev_t *vd, uint64_t txg)
	{
	vdev_t *ivd;
	dmu_tx_t *tx;
	spa_t *spa = vd->vdev_spa;
	spa_vdev_removal_t *svr = spa->spa_vdev_removal;

	/*
	* First, build a list of leaf zaps to be destroyed.
	* This is passed to the sync context thread,
	* which does the actual unlinking.
	*/
	svr->svr_zaplist = fnvlist_alloc();
	vdev_remove_enlist_zaps(vd, svr->svr_zaplist);

	ivd = vdev_add_parent(vd, &vdev_indirect_ops);
	ivd->vdev_removing = 0;

	vd->vdev_leaf_zap = 0;

	vdev_remove_child(ivd, vd);
	vdev_compact_children(ivd);

	ASSERT(!list_link_active(&vd->vdev_state_dirty_node));

	mutex_enter(&svr->svr_lock);
	svr->svr_thread = NULL;
	cv_broadcast(&svr->svr_cv);
	mutex_exit(&svr->svr_lock);

	/* After this, we can not use svr. */
	tx = dmu_tx_create_assigned(spa->spa_dsl_pool, txg);
	dsl_sync_task_nowait(spa->spa_dsl_pool,
	vdev_remove_complete_sync, svr, tx);
	dmu_tx_commit(tx);
	}

	/*
	* Complete the removal of a toplevel vdev. This is called in open
	* context by the removal thread after we have copied all vdev's data.
	*/
	static void
	vdev_remove_complete(spa_t *spa)
	{
	uint64_t txg;

	/*
	* Wait for any deferred frees to be synced before we call
	* vdev_metaslab_fini()
	*/
	txg_wait_synced(spa->spa_dsl_pool, 0);
	txg = spa_vdev_enter(spa);
	vdev_t *vd = vdev_lookup_top(spa, spa->spa_vdev_removal->svr_vdev_id);
	ASSERT3P(vd->vdev_initialize_thread, ==, NULL);
	ASSERT3P(vd->vdev_trim_thread, ==, NULL);
	ASSERT3P(vd->vdev_autotrim_thread, ==, NULL);

	sysevent_t *ev = spa_event_create(spa, vd, NULL,
	ESC_ZFS_VDEV_REMOVE_DEV);

	zfs_dbgmsg("finishing device removal for vdev %llu in txg %llu",
	vd->vdev_id, txg);

	/*
	* Discard allocation state.
	*/
	if (vd->vdev_mg != NULL) {
	vdev_metaslab_fini(vd);
	metaslab_group_destroy(vd->vdev_mg);
	vd->vdev_mg = NULL;
	spa_log_sm_set_blocklimit(spa);
	}
	ASSERT0(vd->vdev_stat.vs_space);
	ASSERT0(vd->vdev_stat.vs_dspace);

	vdev_remove_replace_with_indirect(vd, txg);

	/*
	* We now release the locks, allowing spa_sync to run and finish the
	* removal via vdev_remove_complete_sync in syncing context.
	*
	* Note that we hold on to the vdev_t that has been replaced. Since
	* it isn't part of the vdev tree any longer, it can't be concurrently
	* manipulated, even while we don't have the config lock.
	*/
	(void) spa_vdev_exit(spa, NULL, txg, 0);

	/*
	* Top ZAP should have been transferred to the indirect vdev in
	* vdev_remove_replace_with_indirect.
	*/
	ASSERT0(vd->vdev_top_zap);

	/*
	* Leaf ZAP should have been moved in vdev_remove_replace_with_indirect.
	*/
	ASSERT0(vd->vdev_leaf_zap);

	txg = spa_vdev_enter(spa);
	(void) vdev_label_init(vd, 0, VDEV_LABEL_REMOVE);
	/*
	* Request to update the config and the config cachefile.
	*/
	vdev_config_dirty(spa->spa_root_vdev);
	(void) spa_vdev_exit(spa, vd, txg, 0);

	if (ev != NULL)
	spa_event_post(ev);
	}

	/*
	* Evacuates a segment of size at most max_alloc from the vdev
	* via repeated calls to spa_vdev_copy_segment. If an allocation
	* fails, the pool is probably too fragmented to handle such a
	* large size, so decrease max_alloc so that the caller will not try
	* this size again this txg.
	*/
	static void
	spa_vdev_copy_impl(vdev_t vd, spa_vdev_removal_t svr, vdev_copy_arg_t *vca,
	uint64_t max_alloc, dmu_tx_t tx)
	{
	uint64_t txg = dmu_tx_get_txg(tx);
	spa_t *spa = dmu_tx_pool(tx)->dp_spa;

	mutex_enter(&svr->svr_lock);

	/*
	* Determine how big of a chunk to copy. We can allocate up
	* to max_alloc bytes, and we can span up to vdev_removal_max_span
	* bytes of unallocated space at a time. "segs" will track the
	* allocated segments that we are copying. We may also be copying
	* free segments (of up to vdev_removal_max_span bytes).
	*/
	range_tree_t *segs = range_tree_create(NULL, RANGE_SEG64, NULL, 0, 0);
	for (;;) {
	range_tree_t *rt = svr->svr_allocd_segs;
	range_seg_t *rs = range_tree_first(rt);

	if (rs == NULL)
	break;

	uint64_t seg_length;

	if (range_tree_is_empty(segs)) {
	/* need to truncate the first seg based on max_alloc */
	seg_length = MIN(rs_get_end(rs, rt) - rs_get_start(rs,
	rt), *max_alloc);
	} else {
	if (rs_get_start(rs, rt) - range_tree_max(segs) >
	vdev_removal_max_span) {
	/*
	* Including this segment would cause us to
	* copy a larger unneeded chunk than is allowed.
	*/
	break;
	} else if (rs_get_end(rs, rt) - range_tree_min(segs) >
	*max_alloc) {
	/*
	* This additional segment would extend past
	* max_alloc. Rather than splitting this
	* segment, leave it for the next mapping.
	*/
	break;
	} else {
	seg_length = rs_get_end(rs, rt) -
	rs_get_start(rs, rt);
	}
	}

	range_tree_add(segs, rs_get_start(rs, rt), seg_length);
	range_tree_remove(svr->svr_allocd_segs,
	rs_get_start(rs, rt), seg_length);
	}

	if (range_tree_is_empty(segs)) {
	mutex_exit(&svr->svr_lock);
	range_tree_destroy(segs);
	return;
	}

	if (svr->svr_max_offset_to_sync[txg & TXG_MASK] == 0) {
	dsl_sync_task_nowait(dmu_tx_pool(tx), vdev_mapping_sync,
	svr, tx);
	}

	svr->svr_max_offset_to_sync[txg & TXG_MASK] = range_tree_max(segs);

	/*
	* Note: this is the amount of allocated space
	* that we are taking care of each txg.
	*/
	svr->svr_bytes_done[txg & TXG_MASK] += range_tree_space(segs);

	mutex_exit(&svr->svr_lock);

	zio_alloc_list_t zal;
	metaslab_trace_init(&zal);
	uint64_t thismax = SPA_MAXBLOCKSIZE;
	while (!range_tree_is_empty(segs)) {
	int error = spa_vdev_copy_segment(vd,
	segs, thismax, txg, vca, &zal);

	if (error == ENOSPC) {
	/*
	* Cut our segment in half, and don't try this
	* segment size again this txg. Note that the
	* allocation size must be aligned to the highest
	* ashift in the pool, so that the allocation will
	* not be padded out to a multiple of the ashift,
	* which could cause us to think that this mapping
	* is larger than we intended.
	*/
	ASSERT3U(spa->spa_max_ashift, >=, SPA_MINBLOCKSHIFT);
	ASSERT3U(spa->spa_max_ashift, ==, spa->spa_min_ashift);
	uint64_t attempted =
	MIN(range_tree_span(segs), thismax);
	thismax = P2ROUNDUP(attempted / 2,
	1 << spa->spa_max_ashift);
	/*
	* The minimum-size allocation can not fail.
	*/
	ASSERT3U(attempted, >, 1 << spa->spa_max_ashift);
	*max_alloc = attempted - (1 << spa->spa_max_ashift);
	} else {
	ASSERT0(error);

	/*
	* We've performed an allocation, so reset the
	* alloc trace list.
	*/
	metaslab_trace_fini(&zal);
	metaslab_trace_init(&zal);
	}
	}
	metaslab_trace_fini(&zal);
	range_tree_destroy(segs);
	}

	/*
	* The size of each removal mapping is limited by the tunable
	* zfs_remove_max_segment, but we must adjust this to be a multiple of the
	* pool's ashift, so that we don't try to split individual sectors regardless
	* of the tunable value. (Note that device removal requires that all devices
	* have the same ashift, so there's no difference between spa_min_ashift and
	* spa_max_ashift.) The raw tunable should not be used elsewhere.
	*/
	uint64_t
	spa_remove_max_segment(spa_t *spa)
	{
	return (P2ROUNDUP(zfs_remove_max_segment, 1 << spa->spa_max_ashift));
	}

	/*
	* The removal thread operates in open context. It iterates over all
	* allocated space in the vdev, by loading each metaslab's spacemap.
	* For each contiguous segment of allocated space (capping the segment
	* size at SPA_MAXBLOCKSIZE), we:
	* - Allocate space for it on another vdev.
	* - Create a new mapping from the old location to the new location
	* (as a record in svr_new_segments).
	* - Initiate a physical read zio to get the data off the removing disk.
	* - In the read zio's done callback, initiate a physical write zio to
	* write it to the new vdev.
	* Note that all of this will take effect when a particular TXG syncs.
	* The sync thread ensures that all the phys reads and writes for the syncing
	* TXG have completed (see spa_txg_zio) and writes the new mappings to disk
	* (see vdev_mapping_sync()).
	*/
	static void
	spa_vdev_remove_thread(void *arg)
	{
	spa_t *spa = arg;
	spa_vdev_removal_t *svr = spa->spa_vdev_removal;
	vdev_copy_arg_t vca;
	uint64_t max_alloc = spa_remove_max_segment(spa);
	uint64_t last_txg = 0;

	spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER);
	vdev_t *vd = vdev_lookup_top(spa, svr->svr_vdev_id);
	vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping;
	uint64_t start_offset = vdev_indirect_mapping_max_offset(vim);

	ASSERT3P(vd->vdev_ops, !=, &vdev_indirect_ops);
	ASSERT(vdev_is_concrete(vd));
	ASSERT(vd->vdev_removing);
	ASSERT(vd->vdev_indirect_config.vic_mapping_object != 0);
	ASSERT(vim != NULL);

	mutex_init(&vca.vca_lock, NULL, MUTEX_DEFAULT, NULL);
	cv_init(&vca.vca_cv, NULL, CV_DEFAULT, NULL);
	vca.vca_outstanding_bytes = 0;
	vca.vca_read_error_bytes = 0;
	vca.vca_write_error_bytes = 0;

	mutex_enter(&svr->svr_lock);

	/*
	* Start from vim_max_offset so we pick up where we left off
	* if we are restarting the removal after opening the pool.
	*/
	uint64_t msi;
	for (msi = start_offset >> vd->vdev_ms_shift;
	msi < vd->vdev_ms_count && !svr->svr_thread_exit; msi++) {
	metaslab_t *msp = vd->vdev_ms[msi];
	ASSERT3U(msi, <=, vd->vdev_ms_count);

	ASSERT0(range_tree_space(svr->svr_allocd_segs));

	mutex_enter(&msp->ms_sync_lock);
	mutex_enter(&msp->ms_lock);

	/*
	* Assert nothing in flight -- ms_*tree is empty.
	*/
	for (int i = 0; i < TXG_SIZE; i++) {
	ASSERT0(range_tree_space(msp->ms_allocating[i]));
	}

	/*
	* If the metaslab has ever been allocated from (ms_sm!=NULL),
	* read the allocated segments from the space map object
	* into svr_allocd_segs. Since we do this while holding
	* svr_lock and ms_sync_lock, concurrent frees (which
	* would have modified the space map) will wait for us
	* to finish loading the spacemap, and then take the
	* appropriate action (see free_from_removing_vdev()).
	*/
	if (msp->ms_sm != NULL) {
	VERIFY0(space_map_load(msp->ms_sm,
	svr->svr_allocd_segs, SM_ALLOC));

	range_tree_walk(msp->ms_unflushed_allocs,
	range_tree_add, svr->svr_allocd_segs);
	range_tree_walk(msp->ms_unflushed_frees,
	range_tree_remove, svr->svr_allocd_segs);
	range_tree_walk(msp->ms_freeing,
	range_tree_remove, svr->svr_allocd_segs);

	/*
	* When we are resuming from a paused removal (i.e.
	* when importing a pool with a removal in progress),
	* discard any state that we have already processed.
	*/
	range_tree_clear(svr->svr_allocd_segs, 0, start_offset);
	}
	mutex_exit(&msp->ms_lock);
	mutex_exit(&msp->ms_sync_lock);

	vca.vca_msp = msp;
	zfs_dbgmsg("copying %llu segments for metaslab %llu",
	zfs_btree_numnodes(&svr->svr_allocd_segs->rt_root),
	msp->ms_id);

	while (!svr->svr_thread_exit &&
	!range_tree_is_empty(svr->svr_allocd_segs)) {

	mutex_exit(&svr->svr_lock);

	/*
	* We need to periodically drop the config lock so that
	* writers can get in. Additionally, we can't wait
	* for a txg to sync while holding a config lock
	* (since a waiting writer could cause a 3-way deadlock
	* with the sync thread, which also gets a config
	* lock for reader). So we can't hold the config lock
	* while calling dmu_tx_assign().
	*/
	spa_config_exit(spa, SCL_CONFIG, FTAG);

	/*
	* This delay will pause the removal around the point
	* specified by zfs_removal_suspend_progress. We do this
	* solely from the test suite or during debugging.
	*/
	uint64_t bytes_copied =
	spa->spa_removing_phys.sr_copied;
	for (int i = 0; i < TXG_SIZE; i++)
	bytes_copied += svr->svr_bytes_done[i];
	while (zfs_removal_suspend_progress &&
	!svr->svr_thread_exit)
	delay(hz);

	mutex_enter(&vca.vca_lock);
	while (vca.vca_outstanding_bytes >
	zfs_remove_max_copy_bytes) {
	cv_wait(&vca.vca_cv, &vca.vca_lock);
	}
	mutex_exit(&vca.vca_lock);

	dmu_tx_t *tx =
	dmu_tx_create_dd(spa_get_dsl(spa)->dp_mos_dir);

	VERIFY0(dmu_tx_assign(tx, TXG_WAIT));
	uint64_t txg = dmu_tx_get_txg(tx);

	/*
	* Reacquire the vdev_config lock. The vdev_t
	* that we're removing may have changed, e.g. due
	* to a vdev_attach or vdev_detach.
	*/
	spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER);
	vd = vdev_lookup_top(spa, svr->svr_vdev_id);

	if (txg != last_txg)
	max_alloc = spa_remove_max_segment(spa);
	last_txg = txg;

	spa_vdev_copy_impl(vd, svr, &vca, &max_alloc, tx);

	dmu_tx_commit(tx);
	mutex_enter(&svr->svr_lock);
	}

	mutex_enter(&vca.vca_lock);
	if (zfs_removal_ignore_errors == 0 &&
	(vca.vca_read_error_bytes > 0 \|\|
	vca.vca_write_error_bytes > 0)) {
	svr->svr_thread_exit = B_TRUE;
	}
	mutex_exit(&vca.vca_lock);
	}

	mutex_exit(&svr->svr_lock);

	spa_config_exit(spa, SCL_CONFIG, FTAG);

	/*
	* Wait for all copies to finish before cleaning up the vca.
	*/
	txg_wait_synced(spa->spa_dsl_pool, 0);
	ASSERT0(vca.vca_outstanding_bytes);

	mutex_destroy(&vca.vca_lock);
	cv_destroy(&vca.vca_cv);

	if (svr->svr_thread_exit) {
	mutex_enter(&svr->svr_lock);
	range_tree_vacate(svr->svr_allocd_segs, NULL, NULL);
	svr->svr_thread = NULL;
	cv_broadcast(&svr->svr_cv);
	mutex_exit(&svr->svr_lock);

	/*
	* During the removal process an unrecoverable read or write
	* error was encountered. The removal process must be
	* cancelled or this damage may become permanent.
	*/
	if (zfs_removal_ignore_errors == 0 &&
	(vca.vca_read_error_bytes > 0 \|\|
	vca.vca_write_error_bytes > 0)) {
	zfs_dbgmsg("canceling removal due to IO errors: "
	"[read_error_bytes=%llu] [write_error_bytes=%llu]",
	vca.vca_read_error_bytes,
	vca.vca_write_error_bytes);
	spa_vdev_remove_cancel_impl(spa);
	}
	} else {
	ASSERT0(range_tree_space(svr->svr_allocd_segs));
	vdev_remove_complete(spa);
	}

	thread_exit();
	}

	void
	spa_vdev_remove_suspend(spa_t *spa)
	{
	spa_vdev_removal_t *svr = spa->spa_vdev_removal;

	if (svr == NULL)
	return;

	mutex_enter(&svr->svr_lock);
	svr->svr_thread_exit = B_TRUE;
	while (svr->svr_thread != NULL)
	cv_wait(&svr->svr_cv, &svr->svr_lock);
	svr->svr_thread_exit = B_FALSE;
	mutex_exit(&svr->svr_lock);
	}

	/* ARGSUSED */
	static int
	spa_vdev_remove_cancel_check(void arg, dmu_tx_t tx)
	{
	spa_t *spa = dmu_tx_pool(tx)->dp_spa;

	if (spa->spa_vdev_removal == NULL)
	return (ENOTACTIVE);
	return (0);
	}

	/*
	* Cancel a removal by freeing all entries from the partial mapping
	* and marking the vdev as no longer being removing.
	*/
	/* ARGSUSED */
	static void
	spa_vdev_remove_cancel_sync(void arg, dmu_tx_t tx)
	{
	spa_t *spa = dmu_tx_pool(tx)->dp_spa;
	spa_vdev_removal_t *svr = spa->spa_vdev_removal;
	vdev_t *vd = vdev_lookup_top(spa, svr->svr_vdev_id);
	vdev_indirect_config_t *vic = &vd->vdev_indirect_config;
	vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping;
	objset_t *mos = spa->spa_meta_objset;

	ASSERT3P(svr->svr_thread, ==, NULL);

	spa_feature_decr(spa, SPA_FEATURE_DEVICE_REMOVAL, tx);

	boolean_t are_precise;
	VERIFY0(vdev_obsolete_counts_are_precise(vd, &are_precise));
	if (are_precise) {
	spa_feature_decr(spa, SPA_FEATURE_OBSOLETE_COUNTS, tx);
	VERIFY0(zap_remove(spa->spa_meta_objset, vd->vdev_top_zap,
	VDEV_TOP_ZAP_OBSOLETE_COUNTS_ARE_PRECISE, tx));
	}

	uint64_t obsolete_sm_object;
	VERIFY0(vdev_obsolete_sm_object(vd, &obsolete_sm_object));
	if (obsolete_sm_object != 0) {
	ASSERT(vd->vdev_obsolete_sm != NULL);
	ASSERT3U(obsolete_sm_object, ==,
	space_map_object(vd->vdev_obsolete_sm));

	space_map_free(vd->vdev_obsolete_sm, tx);
	VERIFY0(zap_remove(spa->spa_meta_objset, vd->vdev_top_zap,
	VDEV_TOP_ZAP_INDIRECT_OBSOLETE_SM, tx));
	space_map_close(vd->vdev_obsolete_sm);
	vd->vdev_obsolete_sm = NULL;
	spa_feature_decr(spa, SPA_FEATURE_OBSOLETE_COUNTS, tx);
	}
	for (int i = 0; i < TXG_SIZE; i++) {
	ASSERT(list_is_empty(&svr->svr_new_segments[i]));
	ASSERT3U(svr->svr_max_offset_to_sync[i], <=,
	vdev_indirect_mapping_max_offset(vim));
	}

	for (uint64_t msi = 0; msi < vd->vdev_ms_count; msi++) {
	metaslab_t *msp = vd->vdev_ms[msi];

	if (msp->ms_start >= vdev_indirect_mapping_max_offset(vim))
	break;

	ASSERT0(range_tree_space(svr->svr_allocd_segs));

	mutex_enter(&msp->ms_lock);

	/*
	* Assert nothing in flight -- ms_*tree is empty.
	*/
	for (int i = 0; i < TXG_SIZE; i++)
	ASSERT0(range_tree_space(msp->ms_allocating[i]));
	for (int i = 0; i < TXG_DEFER_SIZE; i++)
	ASSERT0(range_tree_space(msp->ms_defer[i]));
	ASSERT0(range_tree_space(msp->ms_freed));

	if (msp->ms_sm != NULL) {
	mutex_enter(&svr->svr_lock);
	VERIFY0(space_map_load(msp->ms_sm,
	svr->svr_allocd_segs, SM_ALLOC));

	range_tree_walk(msp->ms_unflushed_allocs,
	range_tree_add, svr->svr_allocd_segs);
	range_tree_walk(msp->ms_unflushed_frees,
	range_tree_remove, svr->svr_allocd_segs);
	range_tree_walk(msp->ms_freeing,
	range_tree_remove, svr->svr_allocd_segs);

	/*
	* Clear everything past what has been synced,
	* because we have not allocated mappings for it yet.
	*/
	uint64_t syncd = vdev_indirect_mapping_max_offset(vim);
	uint64_t sm_end = msp->ms_sm->sm_start +
	msp->ms_sm->sm_size;
	if (sm_end > syncd)
	range_tree_clear(svr->svr_allocd_segs,
	syncd, sm_end - syncd);

	mutex_exit(&svr->svr_lock);
	}
	mutex_exit(&msp->ms_lock);

	mutex_enter(&svr->svr_lock);
	range_tree_vacate(svr->svr_allocd_segs,
	free_mapped_segment_cb, vd);
	mutex_exit(&svr->svr_lock);
	}

	/*
	* Note: this must happen after we invoke free_mapped_segment_cb,
	* because it adds to the obsolete_segments.
	*/
	range_tree_vacate(vd->vdev_obsolete_segments, NULL, NULL);

	ASSERT3U(vic->vic_mapping_object, ==,
	vdev_indirect_mapping_object(vd->vdev_indirect_mapping));
	vdev_indirect_mapping_close(vd->vdev_indirect_mapping);
	vd->vdev_indirect_mapping = NULL;
	vdev_indirect_mapping_free(mos, vic->vic_mapping_object, tx);
	vic->vic_mapping_object = 0;

	ASSERT3U(vic->vic_births_object, ==,
	vdev_indirect_births_object(vd->vdev_indirect_births));
	vdev_indirect_births_close(vd->vdev_indirect_births);
	vd->vdev_indirect_births = NULL;
	vdev_indirect_births_free(mos, vic->vic_births_object, tx);
	vic->vic_births_object = 0;

	/*
	* We may have processed some frees from the removing vdev in this
	* txg, thus increasing svr_bytes_done; discard that here to
	* satisfy the assertions in spa_vdev_removal_destroy().
	* Note that future txg's can not have any bytes_done, because
	* future TXG's are only modified from open context, and we have
	* already shut down the copying thread.
	*/
	svr->svr_bytes_done[dmu_tx_get_txg(tx) & TXG_MASK] = 0;
	spa_finish_removal(spa, DSS_CANCELED, tx);

	vd->vdev_removing = B_FALSE;
	vdev_config_dirty(vd);

	zfs_dbgmsg("canceled device removal for vdev %llu in %llu",
	vd->vdev_id, dmu_tx_get_txg(tx));
	spa_history_log_internal(spa, "vdev remove canceled", tx,
	"%s vdev %llu %s", spa_name(spa),
	(u_longlong_t)vd->vdev_id,
	(vd->vdev_path != NULL) ? vd->vdev_path : "-");
	}

	static int
	spa_vdev_remove_cancel_impl(spa_t *spa)
	{
	uint64_t vdid = spa->spa_vdev_removal->svr_vdev_id;

	int error = dsl_sync_task(spa->spa_name, spa_vdev_remove_cancel_check,
	spa_vdev_remove_cancel_sync, NULL, 0,
	ZFS_SPACE_CHECK_EXTRA_RESERVED);

	if (error == 0) {
	spa_config_enter(spa, SCL_ALLOC \| SCL_VDEV, FTAG, RW_WRITER);
	vdev_t *vd = vdev_lookup_top(spa, vdid);
	metaslab_group_activate(vd->vdev_mg);
	spa_config_exit(spa, SCL_ALLOC \| SCL_VDEV, FTAG);
	}

	return (error);
	}

	int
	spa_vdev_remove_cancel(spa_t *spa)
	{
	spa_vdev_remove_suspend(spa);

	if (spa->spa_vdev_removal == NULL)
	return (ENOTACTIVE);

	return (spa_vdev_remove_cancel_impl(spa));
	}

	void
	svr_sync(spa_t spa, dmu_tx_t tx)
	{
	spa_vdev_removal_t *svr = spa->spa_vdev_removal;
	int txgoff = dmu_tx_get_txg(tx) & TXG_MASK;

	if (svr == NULL)
	return;

	/*
	* This check is necessary so that we do not dirty the
	* DIRECTORY_OBJECT via spa_sync_removing_state() when there
	* is nothing to do. Dirtying it every time would prevent us
	* from syncing-to-convergence.
	*/
	if (svr->svr_bytes_done[txgoff] == 0)
	return;

	/*
	* Update progress accounting.
	*/
	spa->spa_removing_phys.sr_copied += svr->svr_bytes_done[txgoff];
	svr->svr_bytes_done[txgoff] = 0;

	spa_sync_removing_state(spa, tx);
	}

	static void
	vdev_remove_make_hole_and_free(vdev_t *vd)
	{
	uint64_t id = vd->vdev_id;
	spa_t *spa = vd->vdev_spa;
	vdev_t *rvd = spa->spa_root_vdev;

	ASSERT(MUTEX_HELD(&spa_namespace_lock));
	ASSERT(spa_config_held(spa, SCL_ALL, RW_WRITER) == SCL_ALL);

	vdev_free(vd);

	vd = vdev_alloc_common(spa, id, 0, &vdev_hole_ops);
	vdev_add_child(rvd, vd);
	vdev_config_dirty(rvd);

	/*
	* Reassess the health of our root vdev.
	*/
	vdev_reopen(rvd);
	}

	/*
	* Remove a log device. The config lock is held for the specified TXG.
	*/
	static int
	spa_vdev_remove_log(vdev_t vd, uint64_t txg)
	{
	metaslab_group_t *mg = vd->vdev_mg;
	spa_t *spa = vd->vdev_spa;
	int error = 0;

	ASSERT(vd->vdev_islog);
	ASSERT(vd == vd->vdev_top);
	ASSERT(MUTEX_HELD(&spa_namespace_lock));

	/*
	* Stop allocating from this vdev.
	*/
	metaslab_group_passivate(mg);

	/*
	* Wait for the youngest allocations and frees to sync,
	* and then wait for the deferral of those frees to finish.
	*/
	spa_vdev_config_exit(spa, NULL,
	*txg + TXG_CONCURRENT_STATES + TXG_DEFER_SIZE, 0, FTAG);

	/*
	* Cancel any initialize or TRIM which was in progress.
	*/
	vdev_initialize_stop_all(vd, VDEV_INITIALIZE_CANCELED);
	vdev_trim_stop_all(vd, VDEV_TRIM_CANCELED);
	vdev_autotrim_stop_wait(vd);

	/*
	* Evacuate the device. We don't hold the config lock as
	* writer since we need to do I/O but we do keep the
	* spa_namespace_lock held. Once this completes the device
	* should no longer have any blocks allocated on it.
	*/
	ASSERT(MUTEX_HELD(&spa_namespace_lock));
	if (vd->vdev_stat.vs_alloc != 0)
	error = spa_reset_logs(spa);

	*txg = spa_vdev_config_enter(spa);

	if (error != 0) {
	metaslab_group_activate(mg);
	return (error);
	}
	ASSERT0(vd->vdev_stat.vs_alloc);

	/*
	* The evacuation succeeded. Remove any remaining MOS metadata
	* associated with this vdev, and wait for these changes to sync.
	*/
	vd->vdev_removing = B_TRUE;

	vdev_dirty_leaves(vd, VDD_DTL, *txg);
	vdev_config_dirty(vd);

	/*
	* When the log space map feature is enabled we look at
	* the vdev's top_zap to find the on-disk flush data of
	* the metaslab we just flushed. Thus, while removing a
	* log vdev we make sure to call vdev_metaslab_fini()
	* first, which removes all metaslabs of this vdev from
	* spa_metaslabs_by_flushed before vdev_remove_empty()
	* destroys the top_zap of this log vdev.
	*
	* This avoids the scenario where we flush a metaslab
	* from the log vdev being removed that doesn't have a
	* top_zap and end up failing to lookup its on-disk flush
	* data.
	*
	* We don't call metaslab_group_destroy() right away
	* though (it will be called in vdev_free() later) as
	* during metaslab_sync() of metaslabs from other vdevs
	* we may touch the metaslab group of this vdev through
	* metaslab_class_histogram_verify()
	*/
	vdev_metaslab_fini(vd);
	spa_log_sm_set_blocklimit(spa);

	spa_vdev_config_exit(spa, NULL, *txg, 0, FTAG);
	*txg = spa_vdev_config_enter(spa);

	sysevent_t *ev = spa_event_create(spa, vd, NULL,
	ESC_ZFS_VDEV_REMOVE_DEV);
	ASSERT(MUTEX_HELD(&spa_namespace_lock));
	ASSERT(spa_config_held(spa, SCL_ALL, RW_WRITER) == SCL_ALL);

	/* The top ZAP should have been destroyed by vdev_remove_empty. */
	ASSERT0(vd->vdev_top_zap);
	/* The leaf ZAP should have been destroyed by vdev_dtl_sync. */
	ASSERT0(vd->vdev_leaf_zap);

	(void) vdev_label_init(vd, 0, VDEV_LABEL_REMOVE);

	if (list_link_active(&vd->vdev_state_dirty_node))
	vdev_state_clean(vd);
	if (list_link_active(&vd->vdev_config_dirty_node))
	vdev_config_clean(vd);

	ASSERT0(vd->vdev_stat.vs_alloc);

	/*
	* Clean up the vdev namespace.
	*/
	vdev_remove_make_hole_and_free(vd);

	if (ev != NULL)
	spa_event_post(ev);

	return (0);
	}

	static int
	spa_vdev_remove_top_check(vdev_t *vd)
	{
	spa_t *spa = vd->vdev_spa;

	if (vd != vd->vdev_top)
	return (SET_ERROR(ENOTSUP));

	if (!vdev_is_concrete(vd))
	return (SET_ERROR(ENOTSUP));

	if (!spa_feature_is_enabled(spa, SPA_FEATURE_DEVICE_REMOVAL))
	return (SET_ERROR(ENOTSUP));

	/* available space in the pool's normal class */
	uint64_t available = dsl_dir_space_available(
	spa->spa_dsl_pool->dp_root_dir, NULL, 0, B_TRUE);

	metaslab_class_t *mc = vd->vdev_mg->mg_class;

	/*
	* When removing a vdev from an allocation class that has
	* remaining vdevs, include available space from the class.
	*/
	if (mc != spa_normal_class(spa) && mc->mc_groups > 1) {
	uint64_t class_avail = metaslab_class_get_space(mc) -
	metaslab_class_get_alloc(mc);

	/* add class space, adjusted for overhead */
	available += (class_avail * 94) / 100;
	}

	/*
	* There has to be enough free space to remove the
	* device and leave double the "slop" space (i.e. we
	* must leave at least 3% of the pool free, in addition to
	* the normal slop space).
	*/
	if (available < vd->vdev_stat.vs_dspace + spa_get_slop_space(spa)) {
	return (SET_ERROR(ENOSPC));
	}

	/*
	* There can not be a removal in progress.
	*/
	if (spa->spa_removing_phys.sr_state == DSS_SCANNING)
	return (SET_ERROR(EBUSY));

	/*
	* The device must have all its data.
	*/
	if (!vdev_dtl_empty(vd, DTL_MISSING) \|\|
	!vdev_dtl_empty(vd, DTL_OUTAGE))
	return (SET_ERROR(EBUSY));

	/*
	* The device must be healthy.
	*/
	if (!vdev_readable(vd))
	return (SET_ERROR(EIO));

	/*
	* All vdevs in normal class must have the same ashift.
	*/
	if (spa->spa_max_ashift != spa->spa_min_ashift) {
	return (SET_ERROR(EINVAL));
	}

	/*
	+ * A removed special/dedup vdev must have same ashift as normal class.
	+ */
	+ ASSERT(!vd->vdev_islog);
	+ if (vd->vdev_alloc_bias != VDEV_BIAS_NONE &&
	+ vd->vdev_ashift != spa->spa_max_ashift) {
	+ return (SET_ERROR(EINVAL));
	+ }
	+
	+ /*
	* All vdevs in normal class must have the same ashift
	* and not be raidz.
	*/
	vdev_t *rvd = spa->spa_root_vdev;
	int num_indirect = 0;
	for (uint64_t id = 0; id < rvd->vdev_children; id++) {
	vdev_t *cvd = rvd->vdev_child[id];
	- if (cvd->vdev_ashift != 0 && !cvd->vdev_islog)
	+
	+ /*
	+ * A removed special/dedup vdev must have the same ashift
	+ * across all vdevs in its class.
	+ */
	+ if (vd->vdev_alloc_bias != VDEV_BIAS_NONE &&
	+ cvd->vdev_alloc_bias == vd->vdev_alloc_bias &&
	+ cvd->vdev_ashift != vd->vdev_ashift) {
	+ return (SET_ERROR(EINVAL));
	+ }
	+ if (cvd->vdev_ashift != 0 &&
	+ cvd->vdev_alloc_bias == VDEV_BIAS_NONE)
	ASSERT3U(cvd->vdev_ashift, ==, spa->spa_max_ashift);
	if (cvd->vdev_ops == &vdev_indirect_ops)
	num_indirect++;
	if (!vdev_is_concrete(cvd))
	continue;
	if (cvd->vdev_ops == &vdev_raidz_ops)
	return (SET_ERROR(EINVAL));
	/*
	* Need the mirror to be mirror of leaf vdevs only
	*/
	if (cvd->vdev_ops == &vdev_mirror_ops) {
	for (uint64_t cid = 0;
	cid < cvd->vdev_children; cid++) {
	if (!cvd->vdev_child[cid]->vdev_ops->
	vdev_op_leaf)
	return (SET_ERROR(EINVAL));
	}
	}
	}

	return (0);
	}

	/*
	* Initiate removal of a top-level vdev, reducing the total space in the pool.
	* The config lock is held for the specified TXG. Once initiated,
	* evacuation of all allocated space (copying it to other vdevs) happens
	* in the background (see spa_vdev_remove_thread()), and can be canceled
	* (see spa_vdev_remove_cancel()). If successful, the vdev will
	* be transformed to an indirect vdev (see spa_vdev_remove_complete()).
	*/
	static int
	spa_vdev_remove_top(vdev_t vd, uint64_t txg)
	{
	spa_t *spa = vd->vdev_spa;
	int error;

	/*
	* Check for errors up-front, so that we don't waste time
	* passivating the metaslab group and clearing the ZIL if there
	* are errors.
	*/
	error = spa_vdev_remove_top_check(vd);
	if (error != 0)
	return (error);

	/*
	* Stop allocating from this vdev. Note that we must check
	* that this is not the only device in the pool before
	* passivating, otherwise we will not be able to make
	* progress because we can't allocate from any vdevs.
	* The above check for sufficient free space serves this
	* purpose.
	*/
	metaslab_group_t *mg = vd->vdev_mg;
	metaslab_group_passivate(mg);

	/*
	* Wait for the youngest allocations and frees to sync,
	* and then wait for the deferral of those frees to finish.
	*/
	spa_vdev_config_exit(spa, NULL,
	*txg + TXG_CONCURRENT_STATES + TXG_DEFER_SIZE, 0, FTAG);

	/*
	* We must ensure that no "stubby" log blocks are allocated
	* on the device to be removed. These blocks could be
	* written at any time, including while we are in the middle
	* of copying them.
	*/
	error = spa_reset_logs(spa);

	/*
	* We stop any initializing and TRIM that is currently in progress
	* but leave the state as "active". This will allow the process to
	* resume if the removal is canceled sometime later.
	*/
	vdev_initialize_stop_all(vd, VDEV_INITIALIZE_ACTIVE);
	vdev_trim_stop_all(vd, VDEV_TRIM_ACTIVE);
	vdev_autotrim_stop_wait(vd);

	*txg = spa_vdev_config_enter(spa);

	/*
	* Things might have changed while the config lock was dropped
	* (e.g. space usage). Check for errors again.
	*/
	if (error == 0)
	error = spa_vdev_remove_top_check(vd);

	if (error != 0) {
	metaslab_group_activate(mg);
	spa_async_request(spa, SPA_ASYNC_INITIALIZE_RESTART);
	spa_async_request(spa, SPA_ASYNC_TRIM_RESTART);
	spa_async_request(spa, SPA_ASYNC_AUTOTRIM_RESTART);
	return (error);
	}

	vd->vdev_removing = B_TRUE;

	vdev_dirty_leaves(vd, VDD_DTL, *txg);
	vdev_config_dirty(vd);
	dmu_tx_t tx = dmu_tx_create_assigned(spa->spa_dsl_pool, txg);
	dsl_sync_task_nowait(spa->spa_dsl_pool,
	vdev_remove_initiate_sync, (void *)(uintptr_t)vd->vdev_id, tx);
	dmu_tx_commit(tx);

	return (0);
	}

	/*
	* Remove a device from the pool.
	*
	* Removing a device from the vdev namespace requires several steps
	* and can take a significant amount of time. As a result we use
	* the spa_vdev_config_[enter/exit] functions which allow us to
	* grab and release the spa_config_lock while still holding the namespace
	* lock. During each step the configuration is synced out.
	*/
	int
	spa_vdev_remove(spa_t *spa, uint64_t guid, boolean_t unspare)
	{
	vdev_t *vd;
	nvlist_t spares, l2cache, *nv;
	uint64_t txg = 0;
	uint_t nspares, nl2cache;
	int error = 0, error_log;
	boolean_t locked = MUTEX_HELD(&spa_namespace_lock);
	sysevent_t *ev = NULL;
	char vd_type = NULL, vd_path = NULL;

	ASSERT(spa_writeable(spa));

	if (!locked)
	txg = spa_vdev_enter(spa);

	ASSERT(MUTEX_HELD(&spa_namespace_lock));
	if (spa_feature_is_active(spa, SPA_FEATURE_POOL_CHECKPOINT)) {
	error = (spa_has_checkpoint(spa)) ?
	ZFS_ERR_CHECKPOINT_EXISTS : ZFS_ERR_DISCARDING_CHECKPOINT;

	if (!locked)
	return (spa_vdev_exit(spa, NULL, txg, error));

	return (error);
	}

	vd = spa_lookup_by_guid(spa, guid, B_FALSE);

	if (spa->spa_spares.sav_vdevs != NULL &&
	nvlist_lookup_nvlist_array(spa->spa_spares.sav_config,
	ZPOOL_CONFIG_SPARES, &spares, &nspares) == 0 &&
	(nv = spa_nvlist_lookup_by_guid(spares, nspares, guid)) != NULL) {
	/*
	* Only remove the hot spare if it's not currently in use
	* in this pool.
	*/
	if (vd == NULL \|\| unspare) {
	if (vd == NULL)
	vd = spa_lookup_by_guid(spa, guid, B_TRUE);
	ev = spa_event_create(spa, vd, NULL,
	ESC_ZFS_VDEV_REMOVE_AUX);

	vd_type = VDEV_TYPE_SPARE;
	vd_path = spa_strdup(fnvlist_lookup_string(
	nv, ZPOOL_CONFIG_PATH));
	spa_vdev_remove_aux(spa->spa_spares.sav_config,
	ZPOOL_CONFIG_SPARES, spares, nspares, nv);
	spa_load_spares(spa);
	spa->spa_spares.sav_sync = B_TRUE;
	} else {
	error = SET_ERROR(EBUSY);
	}
	} else if (spa->spa_l2cache.sav_vdevs != NULL &&
	nvlist_lookup_nvlist_array(spa->spa_l2cache.sav_config,
	ZPOOL_CONFIG_L2CACHE, &l2cache, &nl2cache) == 0 &&
	(nv = spa_nvlist_lookup_by_guid(l2cache, nl2cache, guid)) != NULL) {
	vd_type = VDEV_TYPE_L2CACHE;
	vd_path = spa_strdup(fnvlist_lookup_string(
	nv, ZPOOL_CONFIG_PATH));
	/*
	* Cache devices can always be removed.
	*/
	vd = spa_lookup_by_guid(spa, guid, B_TRUE);

	/*
	* Stop trimming the cache device. We need to release the
	* config lock to allow the syncing of TRIM transactions
	* without releasing the spa_namespace_lock. The same
	* strategy is employed in spa_vdev_remove_top().
	*/
	spa_vdev_config_exit(spa, NULL,
	txg + TXG_CONCURRENT_STATES + TXG_DEFER_SIZE, 0, FTAG);
	mutex_enter(&vd->vdev_trim_lock);
	vdev_trim_stop(vd, VDEV_TRIM_CANCELED, NULL);
	mutex_exit(&vd->vdev_trim_lock);
	txg = spa_vdev_config_enter(spa);

	ev = spa_event_create(spa, vd, NULL, ESC_ZFS_VDEV_REMOVE_AUX);
	spa_vdev_remove_aux(spa->spa_l2cache.sav_config,
	ZPOOL_CONFIG_L2CACHE, l2cache, nl2cache, nv);
	spa_load_l2cache(spa);
	spa->spa_l2cache.sav_sync = B_TRUE;
	} else if (vd != NULL && vd->vdev_islog) {
	ASSERT(!locked);
	vd_type = VDEV_TYPE_LOG;
	vd_path = spa_strdup((vd->vdev_path != NULL) ?
	vd->vdev_path : "-");
	error = spa_vdev_remove_log(vd, &txg);
	} else if (vd != NULL) {
	ASSERT(!locked);
	error = spa_vdev_remove_top(vd, &txg);
	} else {
	/*
	* There is no vdev of any kind with the specified guid.
	*/
	error = SET_ERROR(ENOENT);
	}

	error_log = error;

	if (!locked)
	error = spa_vdev_exit(spa, NULL, txg, error);

	/*
	* Logging must be done outside the spa config lock. Otherwise,
	* this code path could end up holding the spa config lock while
	* waiting for a txg_sync so it can write to the internal log.
	* Doing that would prevent the txg sync from actually happening,
	* causing a deadlock.
	*/
	if (error_log == 0 && vd_type != NULL && vd_path != NULL) {
	spa_history_log_internal(spa, "vdev remove", NULL,
	"%s vdev (%s) %s", spa_name(spa), vd_type, vd_path);
	}
	if (vd_path != NULL)
	spa_strfree(vd_path);

	if (ev != NULL)
	spa_event_post(ev);

	return (error);
	}

	int
	spa_removal_get_stats(spa_t spa, pool_removal_stat_t prs)
	{
	prs->prs_state = spa->spa_removing_phys.sr_state;

	if (prs->prs_state == DSS_NONE)
	return (SET_ERROR(ENOENT));

	prs->prs_removing_vdev = spa->spa_removing_phys.sr_removing_vdev;
	prs->prs_start_time = spa->spa_removing_phys.sr_start_time;
	prs->prs_end_time = spa->spa_removing_phys.sr_end_time;
	prs->prs_to_copy = spa->spa_removing_phys.sr_to_copy;
	prs->prs_copied = spa->spa_removing_phys.sr_copied;

	prs->prs_mapping_memory = 0;
	uint64_t indirect_vdev_id =
	spa->spa_removing_phys.sr_prev_indirect_vdev;
	while (indirect_vdev_id != -1) {
	vdev_t *vd = spa->spa_root_vdev->vdev_child[indirect_vdev_id];
	vdev_indirect_config_t *vic = &vd->vdev_indirect_config;
	vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping;

	ASSERT3P(vd->vdev_ops, ==, &vdev_indirect_ops);
	prs->prs_mapping_memory += vdev_indirect_mapping_size(vim);
	indirect_vdev_id = vic->vic_prev_indirect_vdev;
	}

	return (0);
	}

	/* BEGIN CSTYLED */
	ZFS_MODULE_PARAM(zfs_vdev, zfs_, removal_ignore_errors, INT, ZMOD_RW,
	"Ignore hard IO errors when removing device");

	ZFS_MODULE_PARAM(zfs_vdev, zfs_, remove_max_segment, INT, ZMOD_RW,
	"Largest contiguous segment to allocate when removing device");

	ZFS_MODULE_PARAM(zfs_vdev, vdev_, removal_max_span, INT, ZMOD_RW,
	"Largest span of free chunks a remap segment can span");

	ZFS_MODULE_PARAM(zfs_vdev, zfs_, removal_suspend_progress, INT, ZMOD_RW,
	"Pause device removal after this many bytes are copied "
	"(debug use only - causes removal to hang)");
	/* END CSTYLED */

	EXPORT_SYMBOL(free_from_removing_vdev);
	EXPORT_SYMBOL(spa_removal_get_stats);
	EXPORT_SYMBOL(spa_remove_init);
	EXPORT_SYMBOL(spa_restart_removal);
	EXPORT_SYMBOL(spa_vdev_removal_destroy);
	EXPORT_SYMBOL(spa_vdev_remove);
	EXPORT_SYMBOL(spa_vdev_remove_cancel);
	EXPORT_SYMBOL(spa_vdev_remove_suspend);
	EXPORT_SYMBOL(svr_sync);
	Index: vendor-sys/openzfs/dist/module/zfs/zfeature.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/zfs/zfeature.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/zfs/zfeature.c (revision 366774)
	@@ -1,526 +1,526 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/

	/*
	* Copyright (c) 2011, 2015 by Delphix. All rights reserved.
	*/

	#include <sys/zfs_context.h>
	#include <sys/zfeature.h>
	#include <sys/dmu.h>
	#include <sys/nvpair.h>
	#include <sys/zap.h>
	#include <sys/dmu_tx.h>
	#include "zfeature_common.h"
	#include <sys/spa_impl.h>

	/*
	* ZFS Feature Flags
	* -----------------
	*
	* ZFS feature flags are used to provide fine-grained versioning to the ZFS
	* on-disk format. Once enabled on a pool feature flags replace the old
	* spa_version() number.
	*
	* Each new on-disk format change will be given a uniquely identifying string
	* GUID rather than a version number. This avoids the problem of different
	* organizations creating new on-disk formats with the same version number. To
	* keep feature GUIDs unique they should consist of the reverse dns name of the
	* organization which implemented the feature and a short name for the feature,
	* separated by a colon (e.g. com.delphix:async_destroy).
	*
	* Reference Counts
	* ----------------
	*
	* Within each pool features can be in one of three states: disabled, enabled,
	* or active. These states are differentiated by a reference count stored on
	* disk for each feature:
	*
	* 1) If there is no reference count stored on disk the feature is disabled.
	* 2) If the reference count is 0 a system administrator has enabled the
	* feature, but the feature has not been used yet, so no on-disk
	* format changes have been made.
	* 3) If the reference count is greater than 0 the feature is active.
	* The format changes required by the feature are currently on disk.
	* Note that if the feature's format changes are reversed the feature
	* may choose to set its reference count back to 0.
	*
	* Feature flags makes no differentiation between non-zero reference counts
	* for an active feature (e.g. a reference count of 1 means the same thing as a
	* reference count of 27834721), but feature implementations may choose to use
	* the reference count to store meaningful information. For example, a new RAID
	* implementation might set the reference count to the number of vdevs using
	* it. If all those disks are removed from the pool the feature goes back to
	* having a reference count of 0.
	*
	* It is the responsibility of the individual features to maintain a non-zero
	* reference count as long as the feature's format changes are present on disk.
	*
	* Dependencies
	* ------------
	*
	* Each feature may depend on other features. The only effect of this
	* relationship is that when a feature is enabled all of its dependencies are
	* automatically enabled as well. Any future work to support disabling of
	* features would need to ensure that features cannot be disabled if other
	* enabled features depend on them.
	*
	* On-disk Format
	* --------------
	*
	* When feature flags are enabled spa_version() is set to SPA_VERSION_FEATURES
	* (5000). In order for this to work the pool is automatically upgraded to
	* SPA_VERSION_BEFORE_FEATURES (28) first, so all pre-feature flags on disk
	* format changes will be in use.
	*
	* Information about features is stored in 3 ZAP objects in the pool's MOS.
	* These objects are linked to by the following names in the pool directory
	* object:
	*
	* 1) features_for_read: feature GUID -> reference count
	* Features needed to open the pool for reading.
	* 2) features_for_write: feature GUID -> reference count
	* Features needed to open the pool for writing.
	* 3) feature_descriptions: feature GUID -> descriptive string
	* A human readable string.
	*
	* All enabled features appear in either features_for_read or
	* features_for_write, but not both.
	*
	* To open a pool in read-only mode only the features listed in
	* features_for_read need to be supported.
	*
	* To open the pool in read-write mode features in both features_for_read and
	* features_for_write need to be supported.
	*
	* Some features may be required to read the ZAP objects containing feature
	* information. To allow software to check for compatibility with these features
	* before the pool is opened their names must be stored in the label in a
	* new "features_for_read" entry (note that features that are only required
	* to write to a pool never need to be stored in the label since the
	* features_for_write ZAP object can be read before the pool is written to).
	* To save space in the label features must be explicitly marked as needing to
	* be written to the label. Also, reference counts are not stored in the label,
	* instead any feature whose reference count drops to 0 is removed from the
	* label.
	*
	* Adding New Features
	* -------------------
	*
	* Features must be registered in zpool_feature_init() function in
	* zfeature_common.c using the zfeature_register() function. This function
	* has arguments to specify if the feature should be stored in the
	* features_for_read or features_for_write ZAP object and if it needs to be
	* written to the label when active.
	*
	* Once a feature is registered it will appear as a "feature@<feature name>"
	* property which can be set by an administrator. Feature implementors should
	* use the spa_feature_is_enabled() and spa_feature_is_active() functions to
	* query the state of a feature and the spa_feature_incr() and
	* spa_feature_decr() functions to change an enabled feature's reference count.
	* Reference counts may only be updated in the syncing context.
	*
	* Features may not perform enable-time initialization. Instead, any such
	* initialization should occur when the feature is first used. This design
	* enforces that on-disk changes be made only when features are used. Code
	* should only check if a feature is enabled using spa_feature_is_enabled(),
	* not by relying on any feature specific metadata existing. If a feature is
	* enabled, but the feature's metadata is not on disk yet then it should be
	* created as needed.
	*
	* As an example, consider the com.delphix:async_destroy feature. This feature
	* relies on the existence of a bptree in the MOS that store blocks for
	* asynchronous freeing. This bptree is not created when async_destroy is
	* enabled. Instead, when a dataset is destroyed spa_feature_is_enabled() is
	* called to check if async_destroy is enabled. If it is and the bptree object
	* does not exist yet, the bptree object is created as part of the dataset
	* destroy and async_destroy's reference count is incremented to indicate it
	* has made an on-disk format change. Later, after the destroyed dataset's
	* blocks have all been asynchronously freed there is no longer any use for the
	* bptree object, so it is destroyed and async_destroy's reference count is
	* decremented back to 0 to indicate that it has undone its on-disk format
	* changes.
	*/

	typedef enum {
	FEATURE_ACTION_INCR,
	FEATURE_ACTION_DECR,
	} feature_action_t;

	/*
	* Checks that the active features in the pool are supported by
	* this software. Adds each unsupported feature (name -> description) to
	* the supplied nvlist.
	*/
	boolean_t
	spa_features_check(spa_t *spa, boolean_t for_write,
	nvlist_t unsup_feat, nvlist_t enabled_feat)
	{
	objset_t *os = spa->spa_meta_objset;
	boolean_t supported;
	zap_cursor_t *zc;
	zap_attribute_t *za;
	uint64_t obj = for_write ?
	spa->spa_feat_for_write_obj : spa->spa_feat_for_read_obj;
	char *buf;

	zc = kmem_alloc(sizeof (zap_cursor_t), KM_SLEEP);
	za = kmem_alloc(sizeof (zap_attribute_t), KM_SLEEP);
	buf = kmem_alloc(MAXPATHLEN, KM_SLEEP);

	supported = B_TRUE;
	for (zap_cursor_init(zc, os, obj);
	zap_cursor_retrieve(zc, za) == 0;
	zap_cursor_advance(zc)) {
	ASSERT(za->za_integer_length == sizeof (uint64_t) &&
	za->za_num_integers == 1);

	if (NULL != enabled_feat) {
	fnvlist_add_uint64(enabled_feat, za->za_name,
	za->za_first_integer);
	}

	if (za->za_first_integer != 0 &&
	!zfeature_is_supported(za->za_name)) {
	supported = B_FALSE;

	if (NULL != unsup_feat) {
	- char *desc = "";
	+ const char *desc = "";

	if (zap_lookup(os, spa->spa_feat_desc_obj,
	za->za_name, 1, MAXPATHLEN, buf) == 0)
	desc = buf;

	VERIFY(nvlist_add_string(unsup_feat,
	za->za_name, desc) == 0);
	}
	}
	}
	zap_cursor_fini(zc);

	kmem_free(buf, MAXPATHLEN);
	kmem_free(za, sizeof (zap_attribute_t));
	kmem_free(zc, sizeof (zap_cursor_t));

	return (supported);
	}

	/*
	* Use an in-memory cache of feature refcounts for quick retrieval.
	*
	* Note: well-designed features will not need to use this; they should
	* use spa_feature_is_enabled() and spa_feature_is_active() instead.
	* However, this is non-static for zdb, zhack, and spa_add_feature_stats().
	*/
	int
	feature_get_refcount(spa_t spa, zfeature_info_t feature, uint64_t *res)
	{
	ASSERT(VALID_FEATURE_FID(feature->fi_feature));
	if (spa->spa_feat_refcount_cache[feature->fi_feature] ==
	SPA_FEATURE_DISABLED) {
	return (SET_ERROR(ENOTSUP));
	}
	*res = spa->spa_feat_refcount_cache[feature->fi_feature];
	return (0);
	}

	/*
	* Note: well-designed features will not need to use this; they should
	* use spa_feature_is_enabled() and spa_feature_is_active() instead.
	* However, this is non-static for zdb and zhack.
	*/
	int
	feature_get_refcount_from_disk(spa_t spa, zfeature_info_t feature,
	uint64_t *res)
	{
	int err;
	uint64_t refcount;
	uint64_t zapobj = (feature->fi_flags & ZFEATURE_FLAG_READONLY_COMPAT) ?
	spa->spa_feat_for_write_obj : spa->spa_feat_for_read_obj;

	/*
	* If the pool is currently being created, the feature objects may not
	* have been allocated yet. Act as though all features are disabled.
	*/
	if (zapobj == 0)
	return (SET_ERROR(ENOTSUP));

	err = zap_lookup(spa->spa_meta_objset, zapobj,
	feature->fi_guid, sizeof (uint64_t), 1, &refcount);
	if (err != 0) {
	if (err == ENOENT)
	return (SET_ERROR(ENOTSUP));
	else
	return (err);
	}
	*res = refcount;
	return (0);
	}


	static int
	feature_get_enabled_txg(spa_t spa, zfeature_info_t feature, uint64_t *res)
	{
	uint64_t enabled_txg_obj __maybe_unused = spa->spa_feat_enabled_txg_obj;

	ASSERT(zfeature_depends_on(feature->fi_feature,
	SPA_FEATURE_ENABLED_TXG));

	if (!spa_feature_is_enabled(spa, feature->fi_feature)) {
	return (SET_ERROR(ENOTSUP));
	}

	ASSERT(enabled_txg_obj != 0);

	VERIFY0(zap_lookup(spa->spa_meta_objset, spa->spa_feat_enabled_txg_obj,
	feature->fi_guid, sizeof (uint64_t), 1, res));

	return (0);
	}

	/*
	* This function is non-static for zhack; it should otherwise not be used
	* outside this file.
	*/
	void
	feature_sync(spa_t spa, zfeature_info_t feature, uint64_t refcount,
	dmu_tx_t *tx)
	{
	ASSERT(VALID_FEATURE_OR_NONE(feature->fi_feature));
	uint64_t zapobj = (feature->fi_flags & ZFEATURE_FLAG_READONLY_COMPAT) ?
	spa->spa_feat_for_write_obj : spa->spa_feat_for_read_obj;
	VERIFY0(zap_update(spa->spa_meta_objset, zapobj, feature->fi_guid,
	sizeof (uint64_t), 1, &refcount, tx));

	/*
	* feature_sync is called directly from zhack, allowing the
	* creation of arbitrary features whose fi_feature field may
	* be greater than SPA_FEATURES. When called from zhack, the
	* zfeature_info_t object's fi_feature field will be set to
	* SPA_FEATURE_NONE.
	*/
	if (feature->fi_feature != SPA_FEATURE_NONE) {
	uint64_t *refcount_cache =
	&spa->spa_feat_refcount_cache[feature->fi_feature];
	VERIFY3U(*refcount_cache, ==,
	atomic_swap_64(refcount_cache, refcount));
	}

	if (refcount == 0)
	spa_deactivate_mos_feature(spa, feature->fi_guid);
	else if (feature->fi_flags & ZFEATURE_FLAG_MOS)
	spa_activate_mos_feature(spa, feature->fi_guid, tx);
	}

	/*
	* This function is non-static for zhack; it should otherwise not be used
	* outside this file.
	*/
	void
	feature_enable_sync(spa_t spa, zfeature_info_t feature, dmu_tx_t *tx)
	{
	uint64_t initial_refcount =
	(feature->fi_flags & ZFEATURE_FLAG_ACTIVATE_ON_ENABLE) ? 1 : 0;
	uint64_t zapobj = (feature->fi_flags & ZFEATURE_FLAG_READONLY_COMPAT) ?
	spa->spa_feat_for_write_obj : spa->spa_feat_for_read_obj;

	ASSERT(0 != zapobj);
	ASSERT(zfeature_is_valid_guid(feature->fi_guid));
	ASSERT3U(spa_version(spa), >=, SPA_VERSION_FEATURES);

	/*
	* If the feature is already enabled, ignore the request.
	*/
	if (zap_contains(spa->spa_meta_objset, zapobj, feature->fi_guid) == 0)
	return;

	for (int i = 0; feature->fi_depends[i] != SPA_FEATURE_NONE; i++)
	spa_feature_enable(spa, feature->fi_depends[i], tx);

	VERIFY0(zap_update(spa->spa_meta_objset, spa->spa_feat_desc_obj,
	feature->fi_guid, 1, strlen(feature->fi_desc) + 1,
	feature->fi_desc, tx));

	feature_sync(spa, feature, initial_refcount, tx);

	if (spa_feature_is_enabled(spa, SPA_FEATURE_ENABLED_TXG)) {
	uint64_t enabling_txg = dmu_tx_get_txg(tx);

	if (spa->spa_feat_enabled_txg_obj == 0ULL) {
	spa->spa_feat_enabled_txg_obj =
	zap_create_link(spa->spa_meta_objset,
	DMU_OTN_ZAP_METADATA, DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_FEATURE_ENABLED_TXG, tx);
	}
	spa_feature_incr(spa, SPA_FEATURE_ENABLED_TXG, tx);

	VERIFY0(zap_add(spa->spa_meta_objset,
	spa->spa_feat_enabled_txg_obj, feature->fi_guid,
	sizeof (uint64_t), 1, &enabling_txg, tx));
	}

	/*
	* Errata #4 is mostly a problem with encrypted datasets, but it
	* is also a problem where the old encryption feature did not
	* depend on the bookmark_v2 feature. If the pool does not have
	* any encrypted datasets we can resolve this issue simply by
	* enabling this dependency.
	*/
	if (spa->spa_errata == ZPOOL_ERRATA_ZOL_8308_ENCRYPTION &&
	spa_feature_is_enabled(spa, SPA_FEATURE_ENCRYPTION) &&
	!spa_feature_is_active(spa, SPA_FEATURE_ENCRYPTION) &&
	feature->fi_feature == SPA_FEATURE_BOOKMARK_V2)
	spa->spa_errata = 0;
	}

	static void
	feature_do_action(spa_t *spa, spa_feature_t fid, feature_action_t action,
	dmu_tx_t *tx)
	{
	uint64_t refcount = 0;
	zfeature_info_t *feature = &spa_feature_table[fid];
	uint64_t zapobj __maybe_unused =
	(feature->fi_flags & ZFEATURE_FLAG_READONLY_COMPAT) ?
	spa->spa_feat_for_write_obj : spa->spa_feat_for_read_obj;

	ASSERT(VALID_FEATURE_FID(fid));
	ASSERT(0 != zapobj);
	ASSERT(zfeature_is_valid_guid(feature->fi_guid));

	ASSERT(dmu_tx_is_syncing(tx));
	ASSERT3U(spa_version(spa), >=, SPA_VERSION_FEATURES);

	VERIFY3U(feature_get_refcount(spa, feature, &refcount), !=, ENOTSUP);

	switch (action) {
	case FEATURE_ACTION_INCR:
	VERIFY3U(refcount, !=, UINT64_MAX);
	refcount++;
	break;
	case FEATURE_ACTION_DECR:
	VERIFY3U(refcount, !=, 0);
	refcount--;
	break;
	default:
	ASSERT(0);
	break;
	}

	feature_sync(spa, feature, refcount, tx);
	}

	void
	spa_feature_create_zap_objects(spa_t spa, dmu_tx_t tx)
	{
	/*
	* We create feature flags ZAP objects in two instances: during pool
	* creation and during pool upgrade.
	*/
	ASSERT((!spa->spa_sync_on && tx->tx_txg == TXG_INITIAL) \|\|
	dsl_pool_sync_context(spa_get_dsl(spa)));

	spa->spa_feat_for_read_obj = zap_create_link(spa->spa_meta_objset,
	DMU_OTN_ZAP_METADATA, DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_FEATURES_FOR_READ, tx);
	spa->spa_feat_for_write_obj = zap_create_link(spa->spa_meta_objset,
	DMU_OTN_ZAP_METADATA, DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_FEATURES_FOR_WRITE, tx);
	spa->spa_feat_desc_obj = zap_create_link(spa->spa_meta_objset,
	DMU_OTN_ZAP_METADATA, DMU_POOL_DIRECTORY_OBJECT,
	DMU_POOL_FEATURE_DESCRIPTIONS, tx);
	}

	/*
	* Enable any required dependencies, then enable the requested feature.
	*/
	void
	spa_feature_enable(spa_t spa, spa_feature_t fid, dmu_tx_t tx)
	{
	ASSERT3U(spa_version(spa), >=, SPA_VERSION_FEATURES);
	ASSERT(VALID_FEATURE_FID(fid));
	feature_enable_sync(spa, &spa_feature_table[fid], tx);
	}

	void
	spa_feature_incr(spa_t spa, spa_feature_t fid, dmu_tx_t tx)
	{
	feature_do_action(spa, fid, FEATURE_ACTION_INCR, tx);
	}

	void
	spa_feature_decr(spa_t spa, spa_feature_t fid, dmu_tx_t tx)
	{
	feature_do_action(spa, fid, FEATURE_ACTION_DECR, tx);
	}

	boolean_t
	spa_feature_is_enabled(spa_t *spa, spa_feature_t fid)
	{
	int err;
	uint64_t refcount = 0;

	ASSERT(VALID_FEATURE_FID(fid));
	if (spa_version(spa) < SPA_VERSION_FEATURES)
	return (B_FALSE);

	err = feature_get_refcount(spa, &spa_feature_table[fid], &refcount);
	ASSERT(err == 0 \|\| err == ENOTSUP);
	return (err == 0);
	}

	boolean_t
	spa_feature_is_active(spa_t *spa, spa_feature_t fid)
	{
	int err;
	uint64_t refcount = 0;

	ASSERT(VALID_FEATURE_FID(fid));
	if (spa_version(spa) < SPA_VERSION_FEATURES)
	return (B_FALSE);

	err = feature_get_refcount(spa, &spa_feature_table[fid], &refcount);
	ASSERT(err == 0 \|\| err == ENOTSUP);
	return (err == 0 && refcount > 0);
	}

	/*
	* For the feature specified by fid (which must depend on
	* SPA_FEATURE_ENABLED_TXG), return the TXG at which it was enabled in the
	* OUT txg argument.
	*
	* Returns B_TRUE if the feature is enabled, in which case txg will be filled
	* with the transaction group in which the specified feature was enabled.
	* Returns B_FALSE otherwise (i.e. if the feature is not enabled).
	*/
	boolean_t
	spa_feature_enabled_txg(spa_t spa, spa_feature_t fid, uint64_t txg)
	{
	int err;

	ASSERT(VALID_FEATURE_FID(fid));
	if (spa_version(spa) < SPA_VERSION_FEATURES)
	return (B_FALSE);

	err = feature_get_enabled_txg(spa, &spa_feature_table[fid], txg);
	ASSERT(err == 0 \|\| err == ENOTSUP);

	return (err == 0);
	}
	Index: vendor-sys/openzfs/dist/module/zfs/zfs_ioctl.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/zfs/zfs_ioctl.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/zfs/zfs_ioctl.c (revision 366774)
	@@ -1,7629 +1,7630 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/

	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Portions Copyright 2011 Martin Matuska
	* Copyright 2015, OmniTI Computer Consulting, Inc. All rights reserved.
	* Portions Copyright 2012 Pawel Jakub Dawidek <pawel@dawidek.net>
	* Copyright (c) 2014, 2016 Joyent, Inc. All rights reserved.
	* Copyright 2016 Nexenta Systems, Inc. All rights reserved.
	* Copyright (c) 2014, Joyent, Inc. All rights reserved.
	* Copyright (c) 2011, 2020 by Delphix. All rights reserved.
	* Copyright (c) 2013 by Saso Kiselkov. All rights reserved.
	* Copyright (c) 2013 Steven Hartland. All rights reserved.
	* Copyright (c) 2014 Integros [integros.com]
	* Copyright 2016 Toomas Soome <tsoome@me.com>
	* Copyright (c) 2016 Actifio, Inc. All rights reserved.
	* Copyright (c) 2018, loli10K <ezomori.nozomu@gmail.com>. All rights reserved.
	* Copyright 2017 RackTop Systems.
	* Copyright (c) 2017 Open-E, Inc. All Rights Reserved.
	* Copyright (c) 2019 Datto Inc.
	* Copyright (c) 2019, 2020 by Christian Schwarz. All rights reserved.
	* Copyright (c) 2019, Klara Inc.
	* Copyright (c) 2019, Allan Jude
	*/

	/*
	* ZFS ioctls.
	*
	* This file handles the ioctls to /dev/zfs, used for configuring ZFS storage
	* pools and filesystems, e.g. with /sbin/zfs and /sbin/zpool.
	*
	* There are two ways that we handle ioctls: the legacy way where almost
	* all of the logic is in the ioctl callback, and the new way where most
	* of the marshalling is handled in the common entry point, zfsdev_ioctl().
	*
	* Non-legacy ioctls should be registered by calling
	* zfs_ioctl_register() from zfs_ioctl_init(). The ioctl is invoked
	* from userland by lzc_ioctl().
	*
	* The registration arguments are as follows:
	*
	* const char *name
	* The name of the ioctl. This is used for history logging. If the
	* ioctl returns successfully (the callback returns 0), and allow_log
	* is true, then a history log entry will be recorded with the input &
	* output nvlists. The log entry can be printed with "zpool history -i".
	*
	* zfs_ioc_t ioc
	* The ioctl request number, which userland will pass to ioctl(2).
	* We want newer versions of libzfs and libzfs_core to run against
	* existing zfs kernel modules (i.e. a deferred reboot after an update).
	* Therefore the ioctl numbers cannot change from release to release.
	*
	* zfs_secpolicy_func_t *secpolicy
	* This function will be called before the zfs_ioc_func_t, to
	* determine if this operation is permitted. It should return EPERM
	* on failure, and 0 on success. Checks include determining if the
	* dataset is visible in this zone, and if the user has either all
	* zfs privileges in the zone (SYS_MOUNT), or has been granted permission
	* to do this operation on this dataset with "zfs allow".
	*
	* zfs_ioc_namecheck_t namecheck
	* This specifies what to expect in the zfs_cmd_t:zc_name -- a pool
	* name, a dataset name, or nothing. If the name is not well-formed,
	* the ioctl will fail and the callback will not be called.
	* Therefore, the callback can assume that the name is well-formed
	* (e.g. is null-terminated, doesn't have more than one '@' character,
	* doesn't have invalid characters).
	*
	* zfs_ioc_poolcheck_t pool_check
	* This specifies requirements on the pool state. If the pool does
	* not meet them (is suspended or is readonly), the ioctl will fail
	* and the callback will not be called. If any checks are specified
	* (i.e. it is not POOL_CHECK_NONE), namecheck must not be NO_NAME.
	* Multiple checks can be or-ed together (e.g. POOL_CHECK_SUSPENDED \|
	* POOL_CHECK_READONLY).
	*
	* zfs_ioc_key_t *nvl_keys
	* The list of expected/allowable innvl input keys. This list is used
	* to validate the nvlist input to the ioctl.
	*
	* boolean_t smush_outnvlist
	* If smush_outnvlist is true, then the output is presumed to be a
	* list of errors, and it will be "smushed" down to fit into the
	* caller's buffer, by removing some entries and replacing them with a
	* single "N_MORE_ERRORS" entry indicating how many were removed. See
	* nvlist_smush() for details. If smush_outnvlist is false, and the
	* outnvlist does not fit into the userland-provided buffer, then the
	* ioctl will fail with ENOMEM.
	*
	* zfs_ioc_func_t *func
	* The callback function that will perform the operation.
	*
	* The callback should return 0 on success, or an error number on
	* failure. If the function fails, the userland ioctl will return -1,
	* and errno will be set to the callback's return value. The callback
	* will be called with the following arguments:
	*
	* const char *name
	* The name of the pool or dataset to operate on, from
	* zfs_cmd_t:zc_name. The 'namecheck' argument specifies the
	* expected type (pool, dataset, or none).
	*
	* nvlist_t *innvl
	* The input nvlist, deserialized from zfs_cmd_t:zc_nvlist_src. Or
	* NULL if no input nvlist was provided. Changes to this nvlist are
	* ignored. If the input nvlist could not be deserialized, the
	* ioctl will fail and the callback will not be called.
	*
	* nvlist_t *outnvl
	* The output nvlist, initially empty. The callback can fill it in,
	* and it will be returned to userland by serializing it into
	* zfs_cmd_t:zc_nvlist_dst. If it is non-empty, and serialization
	* fails (e.g. because the caller didn't supply a large enough
	* buffer), then the overall ioctl will fail. See the
	* 'smush_nvlist' argument above for additional behaviors.
	*
	* There are two typical uses of the output nvlist:
	* - To return state, e.g. property values. In this case,
	* smush_outnvlist should be false. If the buffer was not large
	* enough, the caller will reallocate a larger buffer and try
	* the ioctl again.
	*
	* - To return multiple errors from an ioctl which makes on-disk
	* changes. In this case, smush_outnvlist should be true.
	* Ioctls which make on-disk modifications should generally not
	* use the outnvl if they succeed, because the caller can not
	* distinguish between the operation failing, and
	* deserialization failing.
	*
	* IOCTL Interface Errors
	*
	* The following ioctl input errors can be returned:
	* ZFS_ERR_IOC_CMD_UNAVAIL the ioctl number is not supported by kernel
	* ZFS_ERR_IOC_ARG_UNAVAIL an input argument is not supported by kernel
	* ZFS_ERR_IOC_ARG_REQUIRED a required input argument is missing
	* ZFS_ERR_IOC_ARG_BADTYPE an input argument has an invalid type
	*/

	#include <sys/types.h>
	#include <sys/param.h>
	#include <sys/errno.h>
	#include <sys/uio.h>
	#include <sys/file.h>
	#include <sys/kmem.h>
	#include <sys/cmn_err.h>
	#include <sys/stat.h>
	#include <sys/zfs_ioctl.h>
	#include <sys/zfs_quota.h>
	#include <sys/zfs_vfsops.h>
	#include <sys/zfs_znode.h>
	#include <sys/zap.h>
	#include <sys/spa.h>
	#include <sys/spa_impl.h>
	#include <sys/vdev.h>
	#include <sys/vdev_impl.h>
	#include <sys/dmu.h>
	#include <sys/dsl_dir.h>
	#include <sys/dsl_dataset.h>
	#include <sys/dsl_prop.h>
	#include <sys/dsl_deleg.h>
	#include <sys/dmu_objset.h>
	#include <sys/dmu_impl.h>
	#include <sys/dmu_redact.h>
	#include <sys/dmu_tx.h>
	#include <sys/sunddi.h>
	#include <sys/policy.h>
	#include <sys/zone.h>
	#include <sys/nvpair.h>
	#include <sys/pathname.h>
	#include <sys/fs/zfs.h>
	#include <sys/zfs_ctldir.h>
	#include <sys/zfs_dir.h>
	#include <sys/zfs_onexit.h>
	#include <sys/zvol.h>
	#include <sys/dsl_scan.h>
	#include <sys/fm/util.h>
	#include <sys/dsl_crypt.h>
	#include <sys/rrwlock.h>
	#include <sys/zfs_file.h>

	#include <sys/dmu_recv.h>
	#include <sys/dmu_send.h>
	#include <sys/dmu_recv.h>
	#include <sys/dsl_destroy.h>
	#include <sys/dsl_bookmark.h>
	#include <sys/dsl_userhold.h>
	#include <sys/zfeature.h>
	#include <sys/zcp.h>
	#include <sys/zio_checksum.h>
	#include <sys/vdev_removal.h>
	#include <sys/vdev_impl.h>
	#include <sys/vdev_initialize.h>
	#include <sys/vdev_trim.h>

	#include "zfs_namecheck.h"
	#include "zfs_prop.h"
	#include "zfs_deleg.h"
	#include "zfs_comutil.h"

	#include <sys/lua/lua.h>
	#include <sys/lua/lauxlib.h>
	#include <sys/zfs_ioctl_impl.h>

	kmutex_t zfsdev_state_lock;
	zfsdev_state_t *zfsdev_state_list;

	/*
	* Limit maximum nvlist size. We don't want users passing in insane values
	* for zc->zc_nvlist_src_size, since we will need to allocate that much memory.
	* Defaults to 0=auto which is handled by platform code.
	*/
	unsigned long zfs_max_nvlist_src_size = 0;

	uint_t zfs_fsyncer_key;
	uint_t zfs_allow_log_key;

	/* DATA_TYPE_ANY is used when zkey_type can vary. */
	#define DATA_TYPE_ANY DATA_TYPE_UNKNOWN

	typedef struct zfs_ioc_vec {
	zfs_ioc_legacy_func_t *zvec_legacy_func;
	zfs_ioc_func_t *zvec_func;
	zfs_secpolicy_func_t *zvec_secpolicy;
	zfs_ioc_namecheck_t zvec_namecheck;
	boolean_t zvec_allow_log;
	zfs_ioc_poolcheck_t zvec_pool_check;
	boolean_t zvec_smush_outnvlist;
	const char *zvec_name;
	const zfs_ioc_key_t *zvec_nvl_keys;
	size_t zvec_nvl_key_count;
	} zfs_ioc_vec_t;

	/* This array is indexed by zfs_userquota_prop_t */
	static const char *userquota_perms[] = {
	ZFS_DELEG_PERM_USERUSED,
	ZFS_DELEG_PERM_USERQUOTA,
	ZFS_DELEG_PERM_GROUPUSED,
	ZFS_DELEG_PERM_GROUPQUOTA,
	ZFS_DELEG_PERM_USEROBJUSED,
	ZFS_DELEG_PERM_USEROBJQUOTA,
	ZFS_DELEG_PERM_GROUPOBJUSED,
	ZFS_DELEG_PERM_GROUPOBJQUOTA,
	ZFS_DELEG_PERM_PROJECTUSED,
	ZFS_DELEG_PERM_PROJECTQUOTA,
	ZFS_DELEG_PERM_PROJECTOBJUSED,
	ZFS_DELEG_PERM_PROJECTOBJQUOTA,
	};

	static int zfs_ioc_userspace_upgrade(zfs_cmd_t *zc);
	static int zfs_ioc_id_quota_upgrade(zfs_cmd_t *zc);
	static int zfs_check_settable(const char name, nvpair_t property,
	cred_t *cr);
	-static int zfs_check_clearable(char dataset, nvlist_t props,
	+static int zfs_check_clearable(const char dataset, nvlist_t props,
	nvlist_t **errors);
	static int zfs_fill_zplprops_root(uint64_t, nvlist_t , nvlist_t ,
	boolean_t *);
	int zfs_set_prop_nvlist(const char , zprop_source_t, nvlist_t , nvlist_t *);
	static int get_nvlist(uint64_t nvl, uint64_t size, int iflag, nvlist_t **nvp);

	static void
	history_str_free(char *buf)
	{
	kmem_free(buf, HIS_MAX_RECORD_LEN);
	}

	static char *
	history_str_get(zfs_cmd_t *zc)
	{
	char *buf;

	if (zc->zc_history == 0)
	return (NULL);

	buf = kmem_alloc(HIS_MAX_RECORD_LEN, KM_SLEEP);
	if (copyinstr((void *)(uintptr_t)zc->zc_history,
	buf, HIS_MAX_RECORD_LEN, NULL) != 0) {
	history_str_free(buf);
	return (NULL);
	}

	buf[HIS_MAX_RECORD_LEN -1] = '\0';

	return (buf);
	}

	/*
	* Return non-zero if the spa version is less than requested version.
	*/
	static int
	zfs_earlier_version(const char *name, int version)
	{
	spa_t *spa;

	if (spa_open(name, &spa, FTAG) == 0) {
	if (spa_version(spa) < version) {
	spa_close(spa, FTAG);
	return (1);
	}
	spa_close(spa, FTAG);
	}
	return (0);
	}

	/*
	* Return TRUE if the ZPL version is less than requested version.
	*/
	static boolean_t
	zpl_earlier_version(const char *name, int version)
	{
	objset_t *os;
	boolean_t rc = B_TRUE;

	if (dmu_objset_hold(name, FTAG, &os) == 0) {
	uint64_t zplversion;

	if (dmu_objset_type(os) != DMU_OST_ZFS) {
	dmu_objset_rele(os, FTAG);
	return (B_TRUE);
	}
	/* XXX reading from non-owned objset */
	if (zfs_get_zplprop(os, ZFS_PROP_VERSION, &zplversion) == 0)
	rc = zplversion < version;
	dmu_objset_rele(os, FTAG);
	}
	return (rc);
	}

	static void
	zfs_log_history(zfs_cmd_t *zc)
	{
	spa_t *spa;
	char *buf;

	if ((buf = history_str_get(zc)) == NULL)
	return;

	if (spa_open(zc->zc_name, &spa, FTAG) == 0) {
	if (spa_version(spa) >= SPA_VERSION_ZPOOL_HISTORY)
	(void) spa_history_log(spa, buf);
	spa_close(spa, FTAG);
	}
	history_str_free(buf);
	}

	/*
	* Policy for top-level read operations (list pools). Requires no privileges,
	* and can be used in the local zone, as there is no associated dataset.
	*/
	/* ARGSUSED */
	static int
	zfs_secpolicy_none(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	return (0);
	}

	/*
	* Policy for dataset read operations (list children, get statistics). Requires
	* no privileges, but must be visible in the local zone.
	*/
	/* ARGSUSED */
	static int
	zfs_secpolicy_read(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	if (INGLOBALZONE(curproc) \|\|
	zone_dataset_visible(zc->zc_name, NULL))
	return (0);

	return (SET_ERROR(ENOENT));
	}

	static int
	zfs_dozonecheck_impl(const char dataset, uint64_t zoned, cred_t cr)
	{
	int writable = 1;

	/*
	* The dataset must be visible by this zone -- check this first
	* so they don't see EPERM on something they shouldn't know about.
	*/
	if (!INGLOBALZONE(curproc) &&
	!zone_dataset_visible(dataset, &writable))
	return (SET_ERROR(ENOENT));

	if (INGLOBALZONE(curproc)) {
	/*
	* If the fs is zoned, only root can access it from the
	* global zone.
	*/
	if (secpolicy_zfs(cr) && zoned)
	return (SET_ERROR(EPERM));
	} else {
	/*
	* If we are in a local zone, the 'zoned' property must be set.
	*/
	if (!zoned)
	return (SET_ERROR(EPERM));

	/* must be writable by this zone */
	if (!writable)
	return (SET_ERROR(EPERM));
	}
	return (0);
	}

	static int
	zfs_dozonecheck(const char dataset, cred_t cr)
	{
	uint64_t zoned;

	if (dsl_prop_get_integer(dataset, zfs_prop_to_name(ZFS_PROP_ZONED),
	&zoned, NULL))
	return (SET_ERROR(ENOENT));

	return (zfs_dozonecheck_impl(dataset, zoned, cr));
	}

	static int
	zfs_dozonecheck_ds(const char dataset, dsl_dataset_t ds, cred_t *cr)
	{
	uint64_t zoned;

	if (dsl_prop_get_int_ds(ds, zfs_prop_to_name(ZFS_PROP_ZONED), &zoned))
	return (SET_ERROR(ENOENT));

	return (zfs_dozonecheck_impl(dataset, zoned, cr));
	}

	static int
	zfs_secpolicy_write_perms_ds(const char name, dsl_dataset_t ds,
	const char perm, cred_t cr)
	{
	int error;

	error = zfs_dozonecheck_ds(name, ds, cr);
	if (error == 0) {
	error = secpolicy_zfs(cr);
	if (error != 0)
	error = dsl_deleg_access_impl(ds, perm, cr);
	}
	return (error);
	}

	static int
	zfs_secpolicy_write_perms(const char name, const char perm, cred_t *cr)
	{
	int error;
	dsl_dataset_t *ds;
	dsl_pool_t *dp;

	/*
	* First do a quick check for root in the global zone, which
	* is allowed to do all write_perms. This ensures that zfs_ioc_*
	* will get to handle nonexistent datasets.
	*/
	if (INGLOBALZONE(curproc) && secpolicy_zfs(cr) == 0)
	return (0);

	error = dsl_pool_hold(name, FTAG, &dp);
	if (error != 0)
	return (error);

	error = dsl_dataset_hold(dp, name, FTAG, &ds);
	if (error != 0) {
	dsl_pool_rele(dp, FTAG);
	return (error);
	}

	error = zfs_secpolicy_write_perms_ds(name, ds, perm, cr);

	dsl_dataset_rele(ds, FTAG);
	dsl_pool_rele(dp, FTAG);
	return (error);
	}

	/*
	* Policy for setting the security label property.
	*
	* Returns 0 for success, non-zero for access and other errors.
	*/
	static int
	-zfs_set_slabel_policy(const char name, char strval, cred_t *cr)
	+zfs_set_slabel_policy(const char name, const char strval, cred_t *cr)
	{
	#ifdef HAVE_MLSLABEL
	char ds_hexsl[MAXNAMELEN];
	bslabel_t ds_sl, new_sl;
	boolean_t new_default = FALSE;
	uint64_t zoned;
	int needed_priv = -1;
	int error;

	/* First get the existing dataset label. */
	error = dsl_prop_get(name, zfs_prop_to_name(ZFS_PROP_MLSLABEL),
	1, sizeof (ds_hexsl), &ds_hexsl, NULL);
	if (error != 0)
	return (SET_ERROR(EPERM));

	if (strcasecmp(strval, ZFS_MLSLABEL_DEFAULT) == 0)
	new_default = TRUE;

	/* The label must be translatable */
	if (!new_default && (hexstr_to_label(strval, &new_sl) != 0))
	return (SET_ERROR(EINVAL));

	/*
	* In a non-global zone, disallow attempts to set a label that
	* doesn't match that of the zone; otherwise no other checks
	* are needed.
	*/
	if (!INGLOBALZONE(curproc)) {
	if (new_default \|\| !blequal(&new_sl, CR_SL(CRED())))
	return (SET_ERROR(EPERM));
	return (0);
	}

	/*
	* For global-zone datasets (i.e., those whose zoned property is
	* "off", verify that the specified new label is valid for the
	* global zone.
	*/
	if (dsl_prop_get_integer(name,
	zfs_prop_to_name(ZFS_PROP_ZONED), &zoned, NULL))
	return (SET_ERROR(EPERM));
	if (!zoned) {
	if (zfs_check_global_label(name, strval) != 0)
	return (SET_ERROR(EPERM));
	}

	/*
	* If the existing dataset label is nondefault, check if the
	* dataset is mounted (label cannot be changed while mounted).
	* Get the zfsvfs_t; if there isn't one, then the dataset isn't
	* mounted (or isn't a dataset, doesn't exist, ...).
	*/
	if (strcasecmp(ds_hexsl, ZFS_MLSLABEL_DEFAULT) != 0) {
	objset_t *os;
	- static char *setsl_tag = "setsl_tag";
	+ static const char *setsl_tag = "setsl_tag";

	/*
	* Try to own the dataset; abort if there is any error,
	* (e.g., already mounted, in use, or other error).
	*/
	error = dmu_objset_own(name, DMU_OST_ZFS, B_TRUE, B_TRUE,
	setsl_tag, &os);
	if (error != 0)
	return (SET_ERROR(EPERM));

	dmu_objset_disown(os, B_TRUE, setsl_tag);

	if (new_default) {
	needed_priv = PRIV_FILE_DOWNGRADE_SL;
	goto out_check;
	}

	if (hexstr_to_label(strval, &new_sl) != 0)
	return (SET_ERROR(EPERM));

	if (blstrictdom(&ds_sl, &new_sl))
	needed_priv = PRIV_FILE_DOWNGRADE_SL;
	else if (blstrictdom(&new_sl, &ds_sl))
	needed_priv = PRIV_FILE_UPGRADE_SL;
	} else {
	/* dataset currently has a default label */
	if (!new_default)
	needed_priv = PRIV_FILE_UPGRADE_SL;
	}

	out_check:
	if (needed_priv != -1)
	return (PRIV_POLICY(cr, needed_priv, B_FALSE, EPERM, NULL));
	return (0);
	#else
	return (SET_ERROR(ENOTSUP));
	#endif /* HAVE_MLSLABEL */
	}

	static int
	zfs_secpolicy_setprop(const char dsname, zfs_prop_t prop, nvpair_t propval,
	cred_t *cr)
	{
	char *strval;

	/*
	* Check permissions for special properties.
	*/
	switch (prop) {
	default:
	break;
	case ZFS_PROP_ZONED:
	/*
	* Disallow setting of 'zoned' from within a local zone.
	*/
	if (!INGLOBALZONE(curproc))
	return (SET_ERROR(EPERM));
	break;

	case ZFS_PROP_QUOTA:
	case ZFS_PROP_FILESYSTEM_LIMIT:
	case ZFS_PROP_SNAPSHOT_LIMIT:
	if (!INGLOBALZONE(curproc)) {
	uint64_t zoned;
	char setpoint[ZFS_MAX_DATASET_NAME_LEN];
	/*
	* Unprivileged users are allowed to modify the
	* limit on things under (ie. contained by)
	* the thing they own.
	*/
	if (dsl_prop_get_integer(dsname,
	zfs_prop_to_name(ZFS_PROP_ZONED), &zoned, setpoint))
	return (SET_ERROR(EPERM));
	if (!zoned \|\| strlen(dsname) <= strlen(setpoint))
	return (SET_ERROR(EPERM));
	}
	break;

	case ZFS_PROP_MLSLABEL:
	if (!is_system_labeled())
	return (SET_ERROR(EPERM));

	if (nvpair_value_string(propval, &strval) == 0) {
	int err;

	err = zfs_set_slabel_policy(dsname, strval, CRED());
	if (err != 0)
	return (err);
	}
	break;
	}

	return (zfs_secpolicy_write_perms(dsname, zfs_prop_to_name(prop), cr));
	}

	/* ARGSUSED */
	static int
	zfs_secpolicy_set_fsacl(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	int error;

	error = zfs_dozonecheck(zc->zc_name, cr);
	if (error != 0)
	return (error);

	/*
	* permission to set permissions will be evaluated later in
	* dsl_deleg_can_allow()
	*/
	return (0);
	}

	/* ARGSUSED */
	static int
	zfs_secpolicy_rollback(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	return (zfs_secpolicy_write_perms(zc->zc_name,
	ZFS_DELEG_PERM_ROLLBACK, cr));
	}

	/* ARGSUSED */
	static int
	zfs_secpolicy_send(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	dsl_pool_t *dp;
	dsl_dataset_t *ds;
	- char *cp;
	+ const char *cp;
	int error;

	/*
	* Generate the current snapshot name from the given objsetid, then
	* use that name for the secpolicy/zone checks.
	*/
	cp = strchr(zc->zc_name, '@');
	if (cp == NULL)
	return (SET_ERROR(EINVAL));
	error = dsl_pool_hold(zc->zc_name, FTAG, &dp);
	if (error != 0)
	return (error);

	error = dsl_dataset_hold_obj(dp, zc->zc_sendobj, FTAG, &ds);
	if (error != 0) {
	dsl_pool_rele(dp, FTAG);
	return (error);
	}

	dsl_dataset_name(ds, zc->zc_name);

	error = zfs_secpolicy_write_perms_ds(zc->zc_name, ds,
	ZFS_DELEG_PERM_SEND, cr);
	dsl_dataset_rele(ds, FTAG);
	dsl_pool_rele(dp, FTAG);

	return (error);
	}

	/* ARGSUSED */
	static int
	zfs_secpolicy_send_new(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	return (zfs_secpolicy_write_perms(zc->zc_name,
	ZFS_DELEG_PERM_SEND, cr));
	}

	static int
	zfs_secpolicy_share(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	return (SET_ERROR(ENOTSUP));
	}

	static int
	zfs_secpolicy_smb_acl(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	return (SET_ERROR(ENOTSUP));
	}

	static int
	zfs_get_parent(const char datasetname, char parent, int parentsize)
	{
	char *cp;

	/*
	* Remove the @bla or /bla from the end of the name to get the parent.
	*/
	(void) strncpy(parent, datasetname, parentsize);
	cp = strrchr(parent, '@');
	if (cp != NULL) {
	cp[0] = '\0';
	} else {
	cp = strrchr(parent, '/');
	if (cp == NULL)
	return (SET_ERROR(ENOENT));
	cp[0] = '\0';
	}

	return (0);
	}

	int
	zfs_secpolicy_destroy_perms(const char name, cred_t cr)
	{
	int error;

	if ((error = zfs_secpolicy_write_perms(name,
	ZFS_DELEG_PERM_MOUNT, cr)) != 0)
	return (error);

	return (zfs_secpolicy_write_perms(name, ZFS_DELEG_PERM_DESTROY, cr));
	}

	/* ARGSUSED */
	static int
	zfs_secpolicy_destroy(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	return (zfs_secpolicy_destroy_perms(zc->zc_name, cr));
	}

	/*
	* Destroying snapshots with delegated permissions requires
	* descendant mount and destroy permissions.
	*/
	/* ARGSUSED */
	static int
	zfs_secpolicy_destroy_snaps(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	nvlist_t *snaps;
	nvpair_t pair, nextpair;
	int error = 0;

	snaps = fnvlist_lookup_nvlist(innvl, "snaps");

	for (pair = nvlist_next_nvpair(snaps, NULL); pair != NULL;
	pair = nextpair) {
	nextpair = nvlist_next_nvpair(snaps, pair);
	error = zfs_secpolicy_destroy_perms(nvpair_name(pair), cr);
	if (error == ENOENT) {
	/*
	* Ignore any snapshots that don't exist (we consider
	* them "already destroyed"). Remove the name from the
	* nvl here in case the snapshot is created between
	* now and when we try to destroy it (in which case
	* we don't want to destroy it since we haven't
	* checked for permission).
	*/
	fnvlist_remove_nvpair(snaps, pair);
	error = 0;
	}
	if (error != 0)
	break;
	}

	return (error);
	}

	int
	zfs_secpolicy_rename_perms(const char from, const char to, cred_t *cr)
	{
	char parentname[ZFS_MAX_DATASET_NAME_LEN];
	int error;

	if ((error = zfs_secpolicy_write_perms(from,
	ZFS_DELEG_PERM_RENAME, cr)) != 0)
	return (error);

	if ((error = zfs_secpolicy_write_perms(from,
	ZFS_DELEG_PERM_MOUNT, cr)) != 0)
	return (error);

	if ((error = zfs_get_parent(to, parentname,
	sizeof (parentname))) != 0)
	return (error);

	if ((error = zfs_secpolicy_write_perms(parentname,
	ZFS_DELEG_PERM_CREATE, cr)) != 0)
	return (error);

	if ((error = zfs_secpolicy_write_perms(parentname,
	ZFS_DELEG_PERM_MOUNT, cr)) != 0)
	return (error);

	return (error);
	}

	/* ARGSUSED */
	static int
	zfs_secpolicy_rename(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	return (zfs_secpolicy_rename_perms(zc->zc_name, zc->zc_value, cr));
	}

	/* ARGSUSED */
	static int
	zfs_secpolicy_promote(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	dsl_pool_t *dp;
	dsl_dataset_t *clone;
	int error;

	error = zfs_secpolicy_write_perms(zc->zc_name,
	ZFS_DELEG_PERM_PROMOTE, cr);
	if (error != 0)
	return (error);

	error = dsl_pool_hold(zc->zc_name, FTAG, &dp);
	if (error != 0)
	return (error);

	error = dsl_dataset_hold(dp, zc->zc_name, FTAG, &clone);

	if (error == 0) {
	char parentname[ZFS_MAX_DATASET_NAME_LEN];
	dsl_dataset_t *origin = NULL;
	dsl_dir_t *dd;
	dd = clone->ds_dir;

	error = dsl_dataset_hold_obj(dd->dd_pool,
	dsl_dir_phys(dd)->dd_origin_obj, FTAG, &origin);
	if (error != 0) {
	dsl_dataset_rele(clone, FTAG);
	dsl_pool_rele(dp, FTAG);
	return (error);
	}

	error = zfs_secpolicy_write_perms_ds(zc->zc_name, clone,
	ZFS_DELEG_PERM_MOUNT, cr);

	dsl_dataset_name(origin, parentname);
	if (error == 0) {
	error = zfs_secpolicy_write_perms_ds(parentname, origin,
	ZFS_DELEG_PERM_PROMOTE, cr);
	}
	dsl_dataset_rele(clone, FTAG);
	dsl_dataset_rele(origin, FTAG);
	}
	dsl_pool_rele(dp, FTAG);
	return (error);
	}

	/* ARGSUSED */
	static int
	zfs_secpolicy_recv(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	int error;

	if ((error = zfs_secpolicy_write_perms(zc->zc_name,
	ZFS_DELEG_PERM_RECEIVE, cr)) != 0)
	return (error);

	if ((error = zfs_secpolicy_write_perms(zc->zc_name,
	ZFS_DELEG_PERM_MOUNT, cr)) != 0)
	return (error);

	return (zfs_secpolicy_write_perms(zc->zc_name,
	ZFS_DELEG_PERM_CREATE, cr));
	}

	/* ARGSUSED */
	static int
	zfs_secpolicy_recv_new(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	return (zfs_secpolicy_recv(zc, innvl, cr));
	}

	int
	zfs_secpolicy_snapshot_perms(const char name, cred_t cr)
	{
	return (zfs_secpolicy_write_perms(name,
	ZFS_DELEG_PERM_SNAPSHOT, cr));
	}

	/*
	* Check for permission to create each snapshot in the nvlist.
	*/
	/* ARGSUSED */
	static int
	zfs_secpolicy_snapshot(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	nvlist_t *snaps;
	int error = 0;
	nvpair_t *pair;

	snaps = fnvlist_lookup_nvlist(innvl, "snaps");

	for (pair = nvlist_next_nvpair(snaps, NULL); pair != NULL;
	pair = nvlist_next_nvpair(snaps, pair)) {
	char *name = nvpair_name(pair);
	char *atp = strchr(name, '@');

	if (atp == NULL) {
	error = SET_ERROR(EINVAL);
	break;
	}
	*atp = '\0';
	error = zfs_secpolicy_snapshot_perms(name, cr);
	*atp = '@';
	if (error != 0)
	break;
	}
	return (error);
	}

	/*
	* Check for permission to create each bookmark in the nvlist.
	*/
	/* ARGSUSED */
	static int
	zfs_secpolicy_bookmark(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	int error = 0;

	for (nvpair_t *pair = nvlist_next_nvpair(innvl, NULL);
	pair != NULL; pair = nvlist_next_nvpair(innvl, pair)) {
	char *name = nvpair_name(pair);
	char *hashp = strchr(name, '#');

	if (hashp == NULL) {
	error = SET_ERROR(EINVAL);
	break;
	}
	*hashp = '\0';
	error = zfs_secpolicy_write_perms(name,
	ZFS_DELEG_PERM_BOOKMARK, cr);
	*hashp = '#';
	if (error != 0)
	break;
	}
	return (error);
	}

	/* ARGSUSED */
	static int
	zfs_secpolicy_destroy_bookmarks(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	nvpair_t pair, nextpair;
	int error = 0;

	for (pair = nvlist_next_nvpair(innvl, NULL); pair != NULL;
	pair = nextpair) {
	char *name = nvpair_name(pair);
	char *hashp = strchr(name, '#');
	nextpair = nvlist_next_nvpair(innvl, pair);

	if (hashp == NULL) {
	error = SET_ERROR(EINVAL);
	break;
	}

	*hashp = '\0';
	error = zfs_secpolicy_write_perms(name,
	ZFS_DELEG_PERM_DESTROY, cr);
	*hashp = '#';
	if (error == ENOENT) {
	/*
	* Ignore any filesystems that don't exist (we consider
	* their bookmarks "already destroyed"). Remove
	* the name from the nvl here in case the filesystem
	* is created between now and when we try to destroy
	* the bookmark (in which case we don't want to
	* destroy it since we haven't checked for permission).
	*/
	fnvlist_remove_nvpair(innvl, pair);
	error = 0;
	}
	if (error != 0)
	break;
	}

	return (error);
	}

	/* ARGSUSED */
	static int
	zfs_secpolicy_log_history(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	/*
	* Even root must have a proper TSD so that we know what pool
	* to log to.
	*/
	if (tsd_get(zfs_allow_log_key) == NULL)
	return (SET_ERROR(EPERM));
	return (0);
	}

	static int
	zfs_secpolicy_create_clone(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	char parentname[ZFS_MAX_DATASET_NAME_LEN];
	int error;
	char *origin;

	if ((error = zfs_get_parent(zc->zc_name, parentname,
	sizeof (parentname))) != 0)
	return (error);

	if (nvlist_lookup_string(innvl, "origin", &origin) == 0 &&
	(error = zfs_secpolicy_write_perms(origin,
	ZFS_DELEG_PERM_CLONE, cr)) != 0)
	return (error);

	if ((error = zfs_secpolicy_write_perms(parentname,
	ZFS_DELEG_PERM_CREATE, cr)) != 0)
	return (error);

	return (zfs_secpolicy_write_perms(parentname,
	ZFS_DELEG_PERM_MOUNT, cr));
	}

	/*
	* Policy for pool operations - create/destroy pools, add vdevs, etc. Requires
	* SYS_CONFIG privilege, which is not available in a local zone.
	*/
	/* ARGSUSED */
	int
	zfs_secpolicy_config(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	if (secpolicy_sys_config(cr, B_FALSE) != 0)
	return (SET_ERROR(EPERM));

	return (0);
	}

	/*
	* Policy for object to name lookups.
	*/
	/* ARGSUSED */
	static int
	zfs_secpolicy_diff(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	int error;

	if ((error = secpolicy_sys_config(cr, B_FALSE)) == 0)
	return (0);

	error = zfs_secpolicy_write_perms(zc->zc_name, ZFS_DELEG_PERM_DIFF, cr);
	return (error);
	}

	/*
	* Policy for fault injection. Requires all privileges.
	*/
	/* ARGSUSED */
	static int
	zfs_secpolicy_inject(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	return (secpolicy_zinject(cr));
	}

	/* ARGSUSED */
	static int
	zfs_secpolicy_inherit_prop(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	zfs_prop_t prop = zfs_name_to_prop(zc->zc_value);

	if (prop == ZPROP_INVAL) {
	if (!zfs_prop_user(zc->zc_value))
	return (SET_ERROR(EINVAL));
	return (zfs_secpolicy_write_perms(zc->zc_name,
	ZFS_DELEG_PERM_USERPROP, cr));
	} else {
	return (zfs_secpolicy_setprop(zc->zc_name, prop,
	NULL, cr));
	}
	}

	static int
	zfs_secpolicy_userspace_one(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	int err = zfs_secpolicy_read(zc, innvl, cr);
	if (err)
	return (err);

	if (zc->zc_objset_type >= ZFS_NUM_USERQUOTA_PROPS)
	return (SET_ERROR(EINVAL));

	if (zc->zc_value[0] == 0) {
	/*
	* They are asking about a posix uid/gid. If it's
	* themself, allow it.
	*/
	if (zc->zc_objset_type == ZFS_PROP_USERUSED \|\|
	zc->zc_objset_type == ZFS_PROP_USERQUOTA \|\|
	zc->zc_objset_type == ZFS_PROP_USEROBJUSED \|\|
	zc->zc_objset_type == ZFS_PROP_USEROBJQUOTA) {
	if (zc->zc_guid == crgetuid(cr))
	return (0);
	} else if (zc->zc_objset_type == ZFS_PROP_GROUPUSED \|\|
	zc->zc_objset_type == ZFS_PROP_GROUPQUOTA \|\|
	zc->zc_objset_type == ZFS_PROP_GROUPOBJUSED \|\|
	zc->zc_objset_type == ZFS_PROP_GROUPOBJQUOTA) {
	if (groupmember(zc->zc_guid, cr))
	return (0);
	}
	/* else is for project quota/used */
	}

	return (zfs_secpolicy_write_perms(zc->zc_name,
	userquota_perms[zc->zc_objset_type], cr));
	}

	static int
	zfs_secpolicy_userspace_many(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	int err = zfs_secpolicy_read(zc, innvl, cr);
	if (err)
	return (err);

	if (zc->zc_objset_type >= ZFS_NUM_USERQUOTA_PROPS)
	return (SET_ERROR(EINVAL));

	return (zfs_secpolicy_write_perms(zc->zc_name,
	userquota_perms[zc->zc_objset_type], cr));
	}

	/* ARGSUSED */
	static int
	zfs_secpolicy_userspace_upgrade(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	return (zfs_secpolicy_setprop(zc->zc_name, ZFS_PROP_VERSION,
	NULL, cr));
	}

	/* ARGSUSED */
	static int
	zfs_secpolicy_hold(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	nvpair_t *pair;
	nvlist_t *holds;
	int error;

	holds = fnvlist_lookup_nvlist(innvl, "holds");

	for (pair = nvlist_next_nvpair(holds, NULL); pair != NULL;
	pair = nvlist_next_nvpair(holds, pair)) {
	char fsname[ZFS_MAX_DATASET_NAME_LEN];
	error = dmu_fsname(nvpair_name(pair), fsname);
	if (error != 0)
	return (error);
	error = zfs_secpolicy_write_perms(fsname,
	ZFS_DELEG_PERM_HOLD, cr);
	if (error != 0)
	return (error);
	}
	return (0);
	}

	/* ARGSUSED */
	static int
	zfs_secpolicy_release(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	nvpair_t *pair;
	int error;

	for (pair = nvlist_next_nvpair(innvl, NULL); pair != NULL;
	pair = nvlist_next_nvpair(innvl, pair)) {
	char fsname[ZFS_MAX_DATASET_NAME_LEN];
	error = dmu_fsname(nvpair_name(pair), fsname);
	if (error != 0)
	return (error);
	error = zfs_secpolicy_write_perms(fsname,
	ZFS_DELEG_PERM_RELEASE, cr);
	if (error != 0)
	return (error);
	}
	return (0);
	}

	/*
	* Policy for allowing temporary snapshots to be taken or released
	*/
	static int
	zfs_secpolicy_tmp_snapshot(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	/*
	* A temporary snapshot is the same as a snapshot,
	* hold, destroy and release all rolled into one.
	* Delegated diff alone is sufficient that we allow this.
	*/
	int error;

	if ((error = zfs_secpolicy_write_perms(zc->zc_name,
	ZFS_DELEG_PERM_DIFF, cr)) == 0)
	return (0);

	error = zfs_secpolicy_snapshot_perms(zc->zc_name, cr);

	if (innvl != NULL) {
	if (error == 0)
	error = zfs_secpolicy_hold(zc, innvl, cr);
	if (error == 0)
	error = zfs_secpolicy_release(zc, innvl, cr);
	if (error == 0)
	error = zfs_secpolicy_destroy(zc, innvl, cr);
	}
	return (error);
	}

	static int
	zfs_secpolicy_load_key(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	return (zfs_secpolicy_write_perms(zc->zc_name,
	ZFS_DELEG_PERM_LOAD_KEY, cr));
	}

	static int
	zfs_secpolicy_change_key(zfs_cmd_t zc, nvlist_t innvl, cred_t *cr)
	{
	return (zfs_secpolicy_write_perms(zc->zc_name,
	ZFS_DELEG_PERM_CHANGE_KEY, cr));
	}

	/*
	* Returns the nvlist as specified by the user in the zfs_cmd_t.
	*/
	static int
	get_nvlist(uint64_t nvl, uint64_t size, int iflag, nvlist_t **nvp)
	{
	char *packed;
	int error;
	nvlist_t *list = NULL;

	/*
	* Read in and unpack the user-supplied nvlist.
	*/
	if (size == 0)
	return (SET_ERROR(EINVAL));

	packed = vmem_alloc(size, KM_SLEEP);

	if ((error = ddi_copyin((void *)(uintptr_t)nvl, packed, size,
	iflag)) != 0) {
	vmem_free(packed, size);
	return (SET_ERROR(EFAULT));
	}

	if ((error = nvlist_unpack(packed, size, &list, 0)) != 0) {
	vmem_free(packed, size);
	return (error);
	}

	vmem_free(packed, size);

	*nvp = list;
	return (0);
	}

	/*
	* Reduce the size of this nvlist until it can be serialized in 'max' bytes.
	* Entries will be removed from the end of the nvlist, and one int32 entry
	* named "N_MORE_ERRORS" will be added indicating how many entries were
	* removed.
	*/
	static int
	nvlist_smush(nvlist_t *errors, size_t max)
	{
	size_t size;

	size = fnvlist_size(errors);

	if (size > max) {
	nvpair_t *more_errors;
	int n = 0;

	if (max < 1024)
	return (SET_ERROR(ENOMEM));

	fnvlist_add_int32(errors, ZPROP_N_MORE_ERRORS, 0);
	more_errors = nvlist_prev_nvpair(errors, NULL);

	do {
	nvpair_t *pair = nvlist_prev_nvpair(errors,
	more_errors);
	fnvlist_remove_nvpair(errors, pair);
	n++;
	size = fnvlist_size(errors);
	} while (size > max);

	fnvlist_remove_nvpair(errors, more_errors);
	fnvlist_add_int32(errors, ZPROP_N_MORE_ERRORS, n);
	ASSERT3U(fnvlist_size(errors), <=, max);
	}

	return (0);
	}

	static int
	put_nvlist(zfs_cmd_t zc, nvlist_t nvl)
	{
	char *packed = NULL;
	int error = 0;
	size_t size;

	size = fnvlist_size(nvl);

	if (size > zc->zc_nvlist_dst_size) {
	error = SET_ERROR(ENOMEM);
	} else {
	packed = fnvlist_pack(nvl, &size);
	if (ddi_copyout(packed, (void *)(uintptr_t)zc->zc_nvlist_dst,
	size, zc->zc_iflags) != 0)
	error = SET_ERROR(EFAULT);
	fnvlist_pack_free(packed, size);
	}

	zc->zc_nvlist_dst_size = size;
	zc->zc_nvlist_dst_filled = B_TRUE;
	return (error);
	}

	int
	getzfsvfs_impl(objset_t os, zfsvfs_t *zfvp)
	{
	int error = 0;
	if (dmu_objset_type(os) != DMU_OST_ZFS) {
	return (SET_ERROR(EINVAL));
	}

	mutex_enter(&os->os_user_ptr_lock);
	*zfvp = dmu_objset_get_user(os);
	/* bump s_active only when non-zero to prevent umount race */
	error = zfs_vfs_ref(zfvp);
	mutex_exit(&os->os_user_ptr_lock);
	return (error);
	}

	int
	getzfsvfs(const char dsname, zfsvfs_t *zfvp)
	{
	objset_t *os;
	int error;

	error = dmu_objset_hold(dsname, FTAG, &os);
	if (error != 0)
	return (error);

	error = getzfsvfs_impl(os, zfvp);
	dmu_objset_rele(os, FTAG);
	return (error);
	}

	/*
	* Find a zfsvfs_t for a mounted filesystem, or create our own, in which
	* case its z_sb will be NULL, and it will be opened as the owner.
	* If 'writer' is set, the z_teardown_lock will be held for RW_WRITER,
	* which prevents all inode ops from running.
	*/
	static int
	zfsvfs_hold(const char name, void tag, zfsvfs_t **zfvp, boolean_t writer)
	{
	int error = 0;

	if (getzfsvfs(name, zfvp) != 0)
	error = zfsvfs_create(name, B_FALSE, zfvp);
	if (error == 0) {
	rrm_enter(&(*zfvp)->z_teardown_lock, (writer) ? RW_WRITER :
	RW_READER, tag);
	if ((*zfvp)->z_unmounted) {
	/*
	* XXX we could probably try again, since the unmounting
	* thread should be just about to disassociate the
	* objset from the zfsvfs.
	*/
	rrm_exit(&(*zfvp)->z_teardown_lock, tag);
	return (SET_ERROR(EBUSY));
	}
	}
	return (error);
	}

	static void
	zfsvfs_rele(zfsvfs_t zfsvfs, void tag)
	{
	rrm_exit(&zfsvfs->z_teardown_lock, tag);

	if (zfs_vfs_held(zfsvfs)) {
	zfs_vfs_rele(zfsvfs);
	} else {
	dmu_objset_disown(zfsvfs->z_os, B_TRUE, zfsvfs);
	zfsvfs_free(zfsvfs);
	}
	}

	static int
	zfs_ioc_pool_create(zfs_cmd_t *zc)
	{
	int error;
	nvlist_t config, props = NULL;
	nvlist_t *rootprops = NULL;
	nvlist_t *zplprops = NULL;
	dsl_crypto_params_t *dcp = NULL;
	- char *spa_name = zc->zc_name;
	+ const char *spa_name = zc->zc_name;
	boolean_t unload_wkey = B_TRUE;

	if ((error = get_nvlist(zc->zc_nvlist_conf, zc->zc_nvlist_conf_size,
	zc->zc_iflags, &config)))
	return (error);

	if (zc->zc_nvlist_src_size != 0 && (error =
	get_nvlist(zc->zc_nvlist_src, zc->zc_nvlist_src_size,
	zc->zc_iflags, &props))) {
	nvlist_free(config);
	return (error);
	}

	if (props) {
	nvlist_t *nvl = NULL;
	nvlist_t *hidden_args = NULL;
	uint64_t version = SPA_VERSION;
	char *tname;

	(void) nvlist_lookup_uint64(props,
	zpool_prop_to_name(ZPOOL_PROP_VERSION), &version);
	if (!SPA_VERSION_IS_SUPPORTED(version)) {
	error = SET_ERROR(EINVAL);
	goto pool_props_bad;
	}
	(void) nvlist_lookup_nvlist(props, ZPOOL_ROOTFS_PROPS, &nvl);
	if (nvl) {
	error = nvlist_dup(nvl, &rootprops, KM_SLEEP);
	if (error != 0)
	goto pool_props_bad;
	(void) nvlist_remove_all(props, ZPOOL_ROOTFS_PROPS);
	}

	(void) nvlist_lookup_nvlist(props, ZPOOL_HIDDEN_ARGS,
	&hidden_args);
	error = dsl_crypto_params_create_nvlist(DCP_CMD_NONE,
	rootprops, hidden_args, &dcp);
	if (error != 0)
	goto pool_props_bad;
	(void) nvlist_remove_all(props, ZPOOL_HIDDEN_ARGS);

	VERIFY(nvlist_alloc(&zplprops, NV_UNIQUE_NAME, KM_SLEEP) == 0);
	error = zfs_fill_zplprops_root(version, rootprops,
	zplprops, NULL);
	if (error != 0)
	goto pool_props_bad;

	if (nvlist_lookup_string(props,
	zpool_prop_to_name(ZPOOL_PROP_TNAME), &tname) == 0)
	spa_name = tname;
	}

	error = spa_create(zc->zc_name, config, props, zplprops, dcp);

	/*
	* Set the remaining root properties
	*/
	if (!error && (error = zfs_set_prop_nvlist(spa_name,
	ZPROP_SRC_LOCAL, rootprops, NULL)) != 0) {
	(void) spa_destroy(spa_name);
	unload_wkey = B_FALSE; /* spa_destroy() unloads wrapping keys */
	}

	pool_props_bad:
	nvlist_free(rootprops);
	nvlist_free(zplprops);
	nvlist_free(config);
	nvlist_free(props);
	dsl_crypto_params_free(dcp, unload_wkey && !!error);

	return (error);
	}

	static int
	zfs_ioc_pool_destroy(zfs_cmd_t *zc)
	{
	int error;
	zfs_log_history(zc);
	error = spa_destroy(zc->zc_name);

	return (error);
	}

	static int
	zfs_ioc_pool_import(zfs_cmd_t *zc)
	{
	nvlist_t config, props = NULL;
	uint64_t guid;
	int error;

	if ((error = get_nvlist(zc->zc_nvlist_conf, zc->zc_nvlist_conf_size,
	zc->zc_iflags, &config)) != 0)
	return (error);

	if (zc->zc_nvlist_src_size != 0 && (error =
	get_nvlist(zc->zc_nvlist_src, zc->zc_nvlist_src_size,
	zc->zc_iflags, &props))) {
	nvlist_free(config);
	return (error);
	}

	if (nvlist_lookup_uint64(config, ZPOOL_CONFIG_POOL_GUID, &guid) != 0 \|\|
	guid != zc->zc_guid)
	error = SET_ERROR(EINVAL);
	else
	error = spa_import(zc->zc_name, config, props, zc->zc_cookie);

	if (zc->zc_nvlist_dst != 0) {
	int err;

	if ((err = put_nvlist(zc, config)) != 0)
	error = err;
	}

	nvlist_free(config);
	nvlist_free(props);

	return (error);
	}

	static int
	zfs_ioc_pool_export(zfs_cmd_t *zc)
	{
	int error;
	boolean_t force = (boolean_t)zc->zc_cookie;
	boolean_t hardforce = (boolean_t)zc->zc_guid;

	zfs_log_history(zc);
	error = spa_export(zc->zc_name, NULL, force, hardforce);

	return (error);
	}

	static int
	zfs_ioc_pool_configs(zfs_cmd_t *zc)
	{
	nvlist_t *configs;
	int error;

	if ((configs = spa_all_configs(&zc->zc_cookie)) == NULL)
	return (SET_ERROR(EEXIST));

	error = put_nvlist(zc, configs);

	nvlist_free(configs);

	return (error);
	}

	/*
	* inputs:
	* zc_name name of the pool
	*
	* outputs:
	* zc_cookie real errno
	* zc_nvlist_dst config nvlist
	* zc_nvlist_dst_size size of config nvlist
	*/
	static int
	zfs_ioc_pool_stats(zfs_cmd_t *zc)
	{
	nvlist_t *config;
	int error;
	int ret = 0;

	error = spa_get_stats(zc->zc_name, &config, zc->zc_value,
	sizeof (zc->zc_value));

	if (config != NULL) {
	ret = put_nvlist(zc, config);
	nvlist_free(config);

	/*
	* The config may be present even if 'error' is non-zero.
	* In this case we return success, and preserve the real errno
	* in 'zc_cookie'.
	*/
	zc->zc_cookie = error;
	} else {
	ret = error;
	}

	return (ret);
	}

	/*
	* Try to import the given pool, returning pool stats as appropriate so that
	* user land knows which devices are available and overall pool health.
	*/
	static int
	zfs_ioc_pool_tryimport(zfs_cmd_t *zc)
	{
	nvlist_t tryconfig, config = NULL;
	int error;

	if ((error = get_nvlist(zc->zc_nvlist_conf, zc->zc_nvlist_conf_size,
	zc->zc_iflags, &tryconfig)) != 0)
	return (error);

	config = spa_tryimport(tryconfig);

	nvlist_free(tryconfig);

	if (config == NULL)
	return (SET_ERROR(EINVAL));

	error = put_nvlist(zc, config);
	nvlist_free(config);

	return (error);
	}

	/*
	* inputs:
	* zc_name name of the pool
	* zc_cookie scan func (pool_scan_func_t)
	* zc_flags scrub pause/resume flag (pool_scrub_cmd_t)
	*/
	static int
	zfs_ioc_pool_scan(zfs_cmd_t *zc)
	{
	spa_t *spa;
	int error;

	if (zc->zc_flags >= POOL_SCRUB_FLAGS_END)
	return (SET_ERROR(EINVAL));

	if ((error = spa_open(zc->zc_name, &spa, FTAG)) != 0)
	return (error);

	if (zc->zc_flags == POOL_SCRUB_PAUSE)
	error = spa_scrub_pause_resume(spa, POOL_SCRUB_PAUSE);
	else if (zc->zc_cookie == POOL_SCAN_NONE)
	error = spa_scan_stop(spa);
	else
	error = spa_scan(spa, zc->zc_cookie);

	spa_close(spa, FTAG);

	return (error);
	}

	static int
	zfs_ioc_pool_freeze(zfs_cmd_t *zc)
	{
	spa_t *spa;
	int error;

	error = spa_open(zc->zc_name, &spa, FTAG);
	if (error == 0) {
	spa_freeze(spa);
	spa_close(spa, FTAG);
	}
	return (error);
	}

	static int
	zfs_ioc_pool_upgrade(zfs_cmd_t *zc)
	{
	spa_t *spa;
	int error;

	if ((error = spa_open(zc->zc_name, &spa, FTAG)) != 0)
	return (error);

	if (zc->zc_cookie < spa_version(spa) \|\|
	!SPA_VERSION_IS_SUPPORTED(zc->zc_cookie)) {
	spa_close(spa, FTAG);
	return (SET_ERROR(EINVAL));
	}

	spa_upgrade(spa, zc->zc_cookie);
	spa_close(spa, FTAG);

	return (error);
	}

	static int
	zfs_ioc_pool_get_history(zfs_cmd_t *zc)
	{
	spa_t *spa;
	char *hist_buf;
	uint64_t size;
	int error;

	if ((size = zc->zc_history_len) == 0)
	return (SET_ERROR(EINVAL));

	if ((error = spa_open(zc->zc_name, &spa, FTAG)) != 0)
	return (error);

	if (spa_version(spa) < SPA_VERSION_ZPOOL_HISTORY) {
	spa_close(spa, FTAG);
	return (SET_ERROR(ENOTSUP));
	}

	hist_buf = vmem_alloc(size, KM_SLEEP);
	if ((error = spa_history_get(spa, &zc->zc_history_offset,
	&zc->zc_history_len, hist_buf)) == 0) {
	error = ddi_copyout(hist_buf,
	(void *)(uintptr_t)zc->zc_history,
	zc->zc_history_len, zc->zc_iflags);
	}

	spa_close(spa, FTAG);
	vmem_free(hist_buf, size);
	return (error);
	}

	static int
	zfs_ioc_pool_reguid(zfs_cmd_t *zc)
	{
	spa_t *spa;
	int error;

	error = spa_open(zc->zc_name, &spa, FTAG);
	if (error == 0) {
	error = spa_change_guid(spa);
	spa_close(spa, FTAG);
	}
	return (error);
	}

	static int
	zfs_ioc_dsobj_to_dsname(zfs_cmd_t *zc)
	{
	return (dsl_dsobj_to_dsname(zc->zc_name, zc->zc_obj, zc->zc_value));
	}

	/*
	* inputs:
	* zc_name name of filesystem
	* zc_obj object to find
	*
	* outputs:
	* zc_value name of object
	*/
	static int
	zfs_ioc_obj_to_path(zfs_cmd_t *zc)
	{
	objset_t *os;
	int error;

	/* XXX reading from objset not owned */
	if ((error = dmu_objset_hold_flags(zc->zc_name, B_TRUE,
	FTAG, &os)) != 0)
	return (error);
	if (dmu_objset_type(os) != DMU_OST_ZFS) {
	dmu_objset_rele_flags(os, B_TRUE, FTAG);
	return (SET_ERROR(EINVAL));
	}
	error = zfs_obj_to_path(os, zc->zc_obj, zc->zc_value,
	sizeof (zc->zc_value));
	dmu_objset_rele_flags(os, B_TRUE, FTAG);

	return (error);
	}

	/*
	* inputs:
	* zc_name name of filesystem
	* zc_obj object to find
	*
	* outputs:
	* zc_stat stats on object
	* zc_value path to object
	*/
	static int
	zfs_ioc_obj_to_stats(zfs_cmd_t *zc)
	{
	objset_t *os;
	int error;

	/* XXX reading from objset not owned */
	if ((error = dmu_objset_hold_flags(zc->zc_name, B_TRUE,
	FTAG, &os)) != 0)
	return (error);
	if (dmu_objset_type(os) != DMU_OST_ZFS) {
	dmu_objset_rele_flags(os, B_TRUE, FTAG);
	return (SET_ERROR(EINVAL));
	}
	error = zfs_obj_to_stats(os, zc->zc_obj, &zc->zc_stat, zc->zc_value,
	sizeof (zc->zc_value));
	dmu_objset_rele_flags(os, B_TRUE, FTAG);

	return (error);
	}

	static int
	zfs_ioc_vdev_add(zfs_cmd_t *zc)
	{
	spa_t *spa;
	int error;
	nvlist_t *config;

	error = spa_open(zc->zc_name, &spa, FTAG);
	if (error != 0)
	return (error);

	error = get_nvlist(zc->zc_nvlist_conf, zc->zc_nvlist_conf_size,
	zc->zc_iflags, &config);
	if (error == 0) {
	error = spa_vdev_add(spa, config);
	nvlist_free(config);
	}
	spa_close(spa, FTAG);
	return (error);
	}

	/*
	* inputs:
	* zc_name name of the pool
	* zc_guid guid of vdev to remove
	* zc_cookie cancel removal
	*/
	static int
	zfs_ioc_vdev_remove(zfs_cmd_t *zc)
	{
	spa_t *spa;
	int error;

	error = spa_open(zc->zc_name, &spa, FTAG);
	if (error != 0)
	return (error);
	if (zc->zc_cookie != 0) {
	error = spa_vdev_remove_cancel(spa);
	} else {
	error = spa_vdev_remove(spa, zc->zc_guid, B_FALSE);
	}
	spa_close(spa, FTAG);
	return (error);
	}

	static int
	zfs_ioc_vdev_set_state(zfs_cmd_t *zc)
	{
	spa_t *spa;
	int error;
	vdev_state_t newstate = VDEV_STATE_UNKNOWN;

	if ((error = spa_open(zc->zc_name, &spa, FTAG)) != 0)
	return (error);
	switch (zc->zc_cookie) {
	case VDEV_STATE_ONLINE:
	error = vdev_online(spa, zc->zc_guid, zc->zc_obj, &newstate);
	break;

	case VDEV_STATE_OFFLINE:
	error = vdev_offline(spa, zc->zc_guid, zc->zc_obj);
	break;

	case VDEV_STATE_FAULTED:
	if (zc->zc_obj != VDEV_AUX_ERR_EXCEEDED &&
	zc->zc_obj != VDEV_AUX_EXTERNAL &&
	zc->zc_obj != VDEV_AUX_EXTERNAL_PERSIST)
	zc->zc_obj = VDEV_AUX_ERR_EXCEEDED;

	error = vdev_fault(spa, zc->zc_guid, zc->zc_obj);
	break;

	case VDEV_STATE_DEGRADED:
	if (zc->zc_obj != VDEV_AUX_ERR_EXCEEDED &&
	zc->zc_obj != VDEV_AUX_EXTERNAL)
	zc->zc_obj = VDEV_AUX_ERR_EXCEEDED;

	error = vdev_degrade(spa, zc->zc_guid, zc->zc_obj);
	break;

	default:
	error = SET_ERROR(EINVAL);
	}
	zc->zc_cookie = newstate;
	spa_close(spa, FTAG);
	return (error);
	}

	static int
	zfs_ioc_vdev_attach(zfs_cmd_t *zc)
	{
	spa_t *spa;
	nvlist_t *config;
	int replacing = zc->zc_cookie;
	int rebuild = zc->zc_simple;
	int error;

	if ((error = spa_open(zc->zc_name, &spa, FTAG)) != 0)
	return (error);

	if ((error = get_nvlist(zc->zc_nvlist_conf, zc->zc_nvlist_conf_size,
	zc->zc_iflags, &config)) == 0) {
	error = spa_vdev_attach(spa, zc->zc_guid, config, replacing,
	rebuild);
	nvlist_free(config);
	}

	spa_close(spa, FTAG);
	return (error);
	}

	static int
	zfs_ioc_vdev_detach(zfs_cmd_t *zc)
	{
	spa_t *spa;
	int error;

	if ((error = spa_open(zc->zc_name, &spa, FTAG)) != 0)
	return (error);

	error = spa_vdev_detach(spa, zc->zc_guid, 0, B_FALSE);

	spa_close(spa, FTAG);
	return (error);
	}

	static int
	zfs_ioc_vdev_split(zfs_cmd_t *zc)
	{
	spa_t *spa;
	nvlist_t config, props = NULL;
	int error;
	boolean_t exp = !!(zc->zc_cookie & ZPOOL_EXPORT_AFTER_SPLIT);

	if ((error = spa_open(zc->zc_name, &spa, FTAG)) != 0)
	return (error);

	if ((error = get_nvlist(zc->zc_nvlist_conf, zc->zc_nvlist_conf_size,
	zc->zc_iflags, &config))) {
	spa_close(spa, FTAG);
	return (error);
	}

	if (zc->zc_nvlist_src_size != 0 && (error =
	get_nvlist(zc->zc_nvlist_src, zc->zc_nvlist_src_size,
	zc->zc_iflags, &props))) {
	spa_close(spa, FTAG);
	nvlist_free(config);
	return (error);
	}

	error = spa_vdev_split_mirror(spa, zc->zc_string, config, props, exp);

	spa_close(spa, FTAG);

	nvlist_free(config);
	nvlist_free(props);

	return (error);
	}

	static int
	zfs_ioc_vdev_setpath(zfs_cmd_t *zc)
	{
	spa_t *spa;
	- char *path = zc->zc_value;
	+ const char *path = zc->zc_value;
	uint64_t guid = zc->zc_guid;
	int error;

	error = spa_open(zc->zc_name, &spa, FTAG);
	if (error != 0)
	return (error);

	error = spa_vdev_setpath(spa, guid, path);
	spa_close(spa, FTAG);
	return (error);
	}

	static int
	zfs_ioc_vdev_setfru(zfs_cmd_t *zc)
	{
	spa_t *spa;
	- char *fru = zc->zc_value;
	+ const char *fru = zc->zc_value;
	uint64_t guid = zc->zc_guid;
	int error;

	error = spa_open(zc->zc_name, &spa, FTAG);
	if (error != 0)
	return (error);

	error = spa_vdev_setfru(spa, guid, fru);
	spa_close(spa, FTAG);
	return (error);
	}

	static int
	zfs_ioc_objset_stats_impl(zfs_cmd_t zc, objset_t os)
	{
	int error = 0;
	nvlist_t *nv;

	dmu_objset_fast_stat(os, &zc->zc_objset_stats);

	if (zc->zc_nvlist_dst != 0 &&
	(error = dsl_prop_get_all(os, &nv)) == 0) {
	dmu_objset_stats(os, nv);
	/*
	* NB: zvol_get_stats() will read the objset contents,
	* which we aren't supposed to do with a
	* DS_MODE_USER hold, because it could be
	* inconsistent. So this is a bit of a workaround...
	* XXX reading without owning
	*/
	if (!zc->zc_objset_stats.dds_inconsistent &&
	dmu_objset_type(os) == DMU_OST_ZVOL) {
	error = zvol_get_stats(os, nv);
	if (error == EIO) {
	nvlist_free(nv);
	return (error);
	}
	VERIFY0(error);
	}
	if (error == 0)
	error = put_nvlist(zc, nv);
	nvlist_free(nv);
	}

	return (error);
	}

	/*
	* inputs:
	* zc_name name of filesystem
	* zc_nvlist_dst_size size of buffer for property nvlist
	*
	* outputs:
	* zc_objset_stats stats
	* zc_nvlist_dst property nvlist
	* zc_nvlist_dst_size size of property nvlist
	*/
	static int
	zfs_ioc_objset_stats(zfs_cmd_t *zc)
	{
	objset_t *os;
	int error;

	error = dmu_objset_hold(zc->zc_name, FTAG, &os);
	if (error == 0) {
	error = zfs_ioc_objset_stats_impl(zc, os);
	dmu_objset_rele(os, FTAG);
	}

	return (error);
	}

	/*
	* inputs:
	* zc_name name of filesystem
	* zc_nvlist_dst_size size of buffer for property nvlist
	*
	* outputs:
	* zc_nvlist_dst received property nvlist
	* zc_nvlist_dst_size size of received property nvlist
	*
	* Gets received properties (distinct from local properties on or after
	* SPA_VERSION_RECVD_PROPS) for callers who want to differentiate received from
	* local property values.
	*/
	static int
	zfs_ioc_objset_recvd_props(zfs_cmd_t *zc)
	{
	int error = 0;
	nvlist_t *nv;

	/*
	* Without this check, we would return local property values if the
	* caller has not already received properties on or after
	* SPA_VERSION_RECVD_PROPS.
	*/
	if (!dsl_prop_get_hasrecvd(zc->zc_name))
	return (SET_ERROR(ENOTSUP));

	if (zc->zc_nvlist_dst != 0 &&
	(error = dsl_prop_get_received(zc->zc_name, &nv)) == 0) {
	error = put_nvlist(zc, nv);
	nvlist_free(nv);
	}

	return (error);
	}

	static int
	nvl_add_zplprop(objset_t os, nvlist_t props, zfs_prop_t prop)
	{
	uint64_t value;
	int error;

	/*
	* zfs_get_zplprop() will either find a value or give us
	* the default value (if there is one).
	*/
	if ((error = zfs_get_zplprop(os, prop, &value)) != 0)
	return (error);
	VERIFY(nvlist_add_uint64(props, zfs_prop_to_name(prop), value) == 0);
	return (0);
	}

	/*
	* inputs:
	* zc_name name of filesystem
	* zc_nvlist_dst_size size of buffer for zpl property nvlist
	*
	* outputs:
	* zc_nvlist_dst zpl property nvlist
	* zc_nvlist_dst_size size of zpl property nvlist
	*/
	static int
	zfs_ioc_objset_zplprops(zfs_cmd_t *zc)
	{
	objset_t *os;
	int err;

	/* XXX reading without owning */
	if ((err = dmu_objset_hold(zc->zc_name, FTAG, &os)))
	return (err);

	dmu_objset_fast_stat(os, &zc->zc_objset_stats);

	/*
	* NB: nvl_add_zplprop() will read the objset contents,
	* which we aren't supposed to do with a DS_MODE_USER
	* hold, because it could be inconsistent.
	*/
	if (zc->zc_nvlist_dst != 0 &&
	!zc->zc_objset_stats.dds_inconsistent &&
	dmu_objset_type(os) == DMU_OST_ZFS) {
	nvlist_t *nv;

	VERIFY(nvlist_alloc(&nv, NV_UNIQUE_NAME, KM_SLEEP) == 0);
	if ((err = nvl_add_zplprop(os, nv, ZFS_PROP_VERSION)) == 0 &&
	(err = nvl_add_zplprop(os, nv, ZFS_PROP_NORMALIZE)) == 0 &&
	(err = nvl_add_zplprop(os, nv, ZFS_PROP_UTF8ONLY)) == 0 &&
	(err = nvl_add_zplprop(os, nv, ZFS_PROP_CASE)) == 0)
	err = put_nvlist(zc, nv);
	nvlist_free(nv);
	} else {
	err = SET_ERROR(ENOENT);
	}
	dmu_objset_rele(os, FTAG);
	return (err);
	}

	/*
	* inputs:
	* zc_name name of filesystem
	* zc_cookie zap cursor
	* zc_nvlist_dst_size size of buffer for property nvlist
	*
	* outputs:
	* zc_name name of next filesystem
	* zc_cookie zap cursor
	* zc_objset_stats stats
	* zc_nvlist_dst property nvlist
	* zc_nvlist_dst_size size of property nvlist
	*/
	static int
	zfs_ioc_dataset_list_next(zfs_cmd_t *zc)
	{
	objset_t *os;
	int error;
	char *p;
	size_t orig_len = strlen(zc->zc_name);

	top:
	if ((error = dmu_objset_hold(zc->zc_name, FTAG, &os))) {
	if (error == ENOENT)
	error = SET_ERROR(ESRCH);
	return (error);
	}

	p = strrchr(zc->zc_name, '/');
	if (p == NULL \|\| p[1] != '\0')
	(void) strlcat(zc->zc_name, "/", sizeof (zc->zc_name));
	p = zc->zc_name + strlen(zc->zc_name);

	do {
	error = dmu_dir_list_next(os,
	sizeof (zc->zc_name) - (p - zc->zc_name), p,
	NULL, &zc->zc_cookie);
	if (error == ENOENT)
	error = SET_ERROR(ESRCH);
	} while (error == 0 && zfs_dataset_name_hidden(zc->zc_name));
	dmu_objset_rele(os, FTAG);

	/*
	* If it's an internal dataset (ie. with a '$' in its name),
	* don't try to get stats for it, otherwise we'll return ENOENT.
	*/
	if (error == 0 && strchr(zc->zc_name, '$') == NULL) {
	error = zfs_ioc_objset_stats(zc); /* fill in the stats */
	if (error == ENOENT) {
	/* We lost a race with destroy, get the next one. */
	zc->zc_name[orig_len] = '\0';
	goto top;
	}
	}
	return (error);
	}

	/*
	* inputs:
	* zc_name name of filesystem
	* zc_cookie zap cursor
	* zc_nvlist_src iteration range nvlist
	* zc_nvlist_src_size size of iteration range nvlist
	*
	* outputs:
	* zc_name name of next snapshot
	* zc_objset_stats stats
	* zc_nvlist_dst property nvlist
	* zc_nvlist_dst_size size of property nvlist
	*/
	static int
	zfs_ioc_snapshot_list_next(zfs_cmd_t *zc)
	{
	int error;
	objset_t os, ossnap;
	dsl_dataset_t *ds;
	uint64_t min_txg = 0, max_txg = 0;

	if (zc->zc_nvlist_src_size != 0) {
	nvlist_t *props = NULL;
	error = get_nvlist(zc->zc_nvlist_src, zc->zc_nvlist_src_size,
	zc->zc_iflags, &props);
	if (error != 0)
	return (error);
	(void) nvlist_lookup_uint64(props, SNAP_ITER_MIN_TXG,
	&min_txg);
	(void) nvlist_lookup_uint64(props, SNAP_ITER_MAX_TXG,
	&max_txg);
	nvlist_free(props);
	}

	error = dmu_objset_hold(zc->zc_name, FTAG, &os);
	if (error != 0) {
	return (error == ENOENT ? SET_ERROR(ESRCH) : error);
	}

	/*
	* A dataset name of maximum length cannot have any snapshots,
	* so exit immediately.
	*/
	if (strlcat(zc->zc_name, "@", sizeof (zc->zc_name)) >=
	ZFS_MAX_DATASET_NAME_LEN) {
	dmu_objset_rele(os, FTAG);
	return (SET_ERROR(ESRCH));
	}

	while (error == 0) {
	if (issig(JUSTLOOKING) && issig(FORREAL)) {
	error = SET_ERROR(EINTR);
	break;
	}

	error = dmu_snapshot_list_next(os,
	sizeof (zc->zc_name) - strlen(zc->zc_name),
	zc->zc_name + strlen(zc->zc_name), &zc->zc_obj,
	&zc->zc_cookie, NULL);
	if (error == ENOENT) {
	error = SET_ERROR(ESRCH);
	break;
	} else if (error != 0) {
	break;
	}

	error = dsl_dataset_hold_obj(dmu_objset_pool(os), zc->zc_obj,
	FTAG, &ds);
	if (error != 0)
	break;

	if ((min_txg != 0 && dsl_get_creationtxg(ds) < min_txg) \|\|
	(max_txg != 0 && dsl_get_creationtxg(ds) > max_txg)) {
	dsl_dataset_rele(ds, FTAG);
	/* undo snapshot name append */
	*(strchr(zc->zc_name, '@') + 1) = '\0';
	/* skip snapshot */
	continue;
	}

	if (zc->zc_simple) {
	dsl_dataset_rele(ds, FTAG);
	break;
	}

	if ((error = dmu_objset_from_ds(ds, &ossnap)) != 0) {
	dsl_dataset_rele(ds, FTAG);
	break;
	}
	if ((error = zfs_ioc_objset_stats_impl(zc, ossnap)) != 0) {
	dsl_dataset_rele(ds, FTAG);
	break;
	}
	dsl_dataset_rele(ds, FTAG);
	break;
	}

	dmu_objset_rele(os, FTAG);
	/* if we failed, undo the @ that we tacked on to zc_name */
	if (error != 0)
	*strchr(zc->zc_name, '@') = '\0';
	return (error);
	}

	static int
	zfs_prop_set_userquota(const char dsname, nvpair_t pair)
	{
	const char *propname = nvpair_name(pair);
	uint64_t *valary;
	unsigned int vallen;
	- const char *domain;
	- char *dash;
	+ const char dash, domain;
	zfs_userquota_prop_t type;
	uint64_t rid;
	uint64_t quota;
	zfsvfs_t *zfsvfs;
	int err;

	if (nvpair_type(pair) == DATA_TYPE_NVLIST) {
	nvlist_t *attrs;
	VERIFY(nvpair_value_nvlist(pair, &attrs) == 0);
	if (nvlist_lookup_nvpair(attrs, ZPROP_VALUE,
	&pair) != 0)
	return (SET_ERROR(EINVAL));
	}

	/*
	* A correctly constructed propname is encoded as
	* userquota@<rid>-<domain>.
	*/
	if ((dash = strchr(propname, '-')) == NULL \|\|
	nvpair_value_uint64_array(pair, &valary, &vallen) != 0 \|\|
	vallen != 3)
	return (SET_ERROR(EINVAL));

	domain = dash + 1;
	type = valary[0];
	rid = valary[1];
	quota = valary[2];

	err = zfsvfs_hold(dsname, FTAG, &zfsvfs, B_FALSE);
	if (err == 0) {
	err = zfs_set_userquota(zfsvfs, type, domain, rid, quota);
	zfsvfs_rele(zfsvfs, FTAG);
	}

	return (err);
	}

	/*
	* If the named property is one that has a special function to set its value,
	* return 0 on success and a positive error code on failure; otherwise if it is
	* not one of the special properties handled by this function, return -1.
	*
	* XXX: It would be better for callers of the property interface if we handled
	* these special cases in dsl_prop.c (in the dsl layer).
	*/
	static int
	zfs_prop_set_special(const char *dsname, zprop_source_t source,
	nvpair_t *pair)
	{
	const char *propname = nvpair_name(pair);
	zfs_prop_t prop = zfs_name_to_prop(propname);
	uint64_t intval = 0;
	- char *strval = NULL;
	+ const char *strval = NULL;
	int err = -1;

	if (prop == ZPROP_INVAL) {
	if (zfs_prop_userquota(propname))
	return (zfs_prop_set_userquota(dsname, pair));
	return (-1);
	}

	if (nvpair_type(pair) == DATA_TYPE_NVLIST) {
	nvlist_t *attrs;
	VERIFY(nvpair_value_nvlist(pair, &attrs) == 0);
	VERIFY(nvlist_lookup_nvpair(attrs, ZPROP_VALUE,
	&pair) == 0);
	}

	/* all special properties are numeric except for keylocation */
	if (zfs_prop_get_type(prop) == PROP_TYPE_STRING) {
	strval = fnvpair_value_string(pair);
	} else {
	intval = fnvpair_value_uint64(pair);
	}

	switch (prop) {
	case ZFS_PROP_QUOTA:
	err = dsl_dir_set_quota(dsname, source, intval);
	break;
	case ZFS_PROP_REFQUOTA:
	err = dsl_dataset_set_refquota(dsname, source, intval);
	break;
	case ZFS_PROP_FILESYSTEM_LIMIT:
	case ZFS_PROP_SNAPSHOT_LIMIT:
	if (intval == UINT64_MAX) {
	/* clearing the limit, just do it */
	err = 0;
	} else {
	err = dsl_dir_activate_fs_ss_limit(dsname);
	}
	/*
	* Set err to -1 to force the zfs_set_prop_nvlist code down the
	* default path to set the value in the nvlist.
	*/
	if (err == 0)
	err = -1;
	break;
	case ZFS_PROP_KEYLOCATION:
	err = dsl_crypto_can_set_keylocation(dsname, strval);

	/*
	* Set err to -1 to force the zfs_set_prop_nvlist code down the
	* default path to set the value in the nvlist.
	*/
	if (err == 0)
	err = -1;
	break;
	case ZFS_PROP_RESERVATION:
	err = dsl_dir_set_reservation(dsname, source, intval);
	break;
	case ZFS_PROP_REFRESERVATION:
	err = dsl_dataset_set_refreservation(dsname, source, intval);
	break;
	case ZFS_PROP_COMPRESSION:
	err = dsl_dataset_set_compression(dsname, source, intval);
	/*
	* Set err to -1 to force the zfs_set_prop_nvlist code down the
	* default path to set the value in the nvlist.
	*/
	if (err == 0)
	err = -1;
	break;
	case ZFS_PROP_VOLSIZE:
	err = zvol_set_volsize(dsname, intval);
	break;
	case ZFS_PROP_SNAPDEV:
	err = zvol_set_snapdev(dsname, source, intval);
	break;
	case ZFS_PROP_VOLMODE:
	err = zvol_set_volmode(dsname, source, intval);
	break;
	case ZFS_PROP_VERSION:
	{
	zfsvfs_t *zfsvfs;

	if ((err = zfsvfs_hold(dsname, FTAG, &zfsvfs, B_TRUE)) != 0)
	break;

	err = zfs_set_version(zfsvfs, intval);
	zfsvfs_rele(zfsvfs, FTAG);

	if (err == 0 && intval >= ZPL_VERSION_USERSPACE) {
	zfs_cmd_t *zc;

	zc = kmem_zalloc(sizeof (zfs_cmd_t), KM_SLEEP);
	(void) strlcpy(zc->zc_name, dsname,
	sizeof (zc->zc_name));
	(void) zfs_ioc_userspace_upgrade(zc);
	(void) zfs_ioc_id_quota_upgrade(zc);
	kmem_free(zc, sizeof (zfs_cmd_t));
	}
	break;
	}
	default:
	err = -1;
	}

	return (err);
	}

	/*
	* This function is best effort. If it fails to set any of the given properties,
	* it continues to set as many as it can and returns the last error
	* encountered. If the caller provides a non-NULL errlist, it will be filled in
	* with the list of names of all the properties that failed along with the
	* corresponding error numbers.
	*
	* If every property is set successfully, zero is returned and errlist is not
	* modified.
	*/
	int
	zfs_set_prop_nvlist(const char dsname, zprop_source_t source, nvlist_t nvl,
	nvlist_t *errlist)
	{
	nvpair_t *pair;
	nvpair_t *propval;
	int rv = 0;
	uint64_t intval;
	- char *strval;
	+ const char *strval;

	nvlist_t *genericnvl = fnvlist_alloc();
	nvlist_t *retrynvl = fnvlist_alloc();
	retry:
	pair = NULL;
	while ((pair = nvlist_next_nvpair(nvl, pair)) != NULL) {
	const char *propname = nvpair_name(pair);
	zfs_prop_t prop = zfs_name_to_prop(propname);
	int err = 0;

	/* decode the property value */
	propval = pair;
	if (nvpair_type(pair) == DATA_TYPE_NVLIST) {
	nvlist_t *attrs;
	attrs = fnvpair_value_nvlist(pair);
	if (nvlist_lookup_nvpair(attrs, ZPROP_VALUE,
	&propval) != 0)
	err = SET_ERROR(EINVAL);
	}

	/* Validate value type */
	if (err == 0 && source == ZPROP_SRC_INHERITED) {
	/* inherited properties are expected to be booleans */
	if (nvpair_type(propval) != DATA_TYPE_BOOLEAN)
	err = SET_ERROR(EINVAL);
	} else if (err == 0 && prop == ZPROP_INVAL) {
	if (zfs_prop_user(propname)) {
	if (nvpair_type(propval) != DATA_TYPE_STRING)
	err = SET_ERROR(EINVAL);
	} else if (zfs_prop_userquota(propname)) {
	if (nvpair_type(propval) !=
	DATA_TYPE_UINT64_ARRAY)
	err = SET_ERROR(EINVAL);
	} else {
	err = SET_ERROR(EINVAL);
	}
	} else if (err == 0) {
	if (nvpair_type(propval) == DATA_TYPE_STRING) {
	if (zfs_prop_get_type(prop) != PROP_TYPE_STRING)
	err = SET_ERROR(EINVAL);
	} else if (nvpair_type(propval) == DATA_TYPE_UINT64) {
	const char *unused;

	intval = fnvpair_value_uint64(propval);

	switch (zfs_prop_get_type(prop)) {
	case PROP_TYPE_NUMBER:
	break;
	case PROP_TYPE_STRING:
	err = SET_ERROR(EINVAL);
	break;
	case PROP_TYPE_INDEX:
	if (zfs_prop_index_to_string(prop,
	intval, &unused) != 0)
	err =
	SET_ERROR(ZFS_ERR_BADPROP);
	break;
	default:
	cmn_err(CE_PANIC,
	"unknown property type");
	}
	} else {
	err = SET_ERROR(EINVAL);
	}
	}

	/* Validate permissions */
	if (err == 0)
	err = zfs_check_settable(dsname, pair, CRED());

	if (err == 0) {
	if (source == ZPROP_SRC_INHERITED)
	err = -1; /* does not need special handling */
	else
	err = zfs_prop_set_special(dsname, source,
	pair);
	if (err == -1) {
	/*
	* For better performance we build up a list of
	* properties to set in a single transaction.
	*/
	err = nvlist_add_nvpair(genericnvl, pair);
	} else if (err != 0 && nvl != retrynvl) {
	/*
	* This may be a spurious error caused by
	* receiving quota and reservation out of order.
	* Try again in a second pass.
	*/
	err = nvlist_add_nvpair(retrynvl, pair);
	}
	}

	if (err != 0) {
	if (errlist != NULL)
	fnvlist_add_int32(errlist, propname, err);
	rv = err;
	}
	}

	if (nvl != retrynvl && !nvlist_empty(retrynvl)) {
	nvl = retrynvl;
	goto retry;
	}

	if (!nvlist_empty(genericnvl) &&
	dsl_props_set(dsname, source, genericnvl) != 0) {
	/*
	* If this fails, we still want to set as many properties as we
	* can, so try setting them individually.
	*/
	pair = NULL;
	while ((pair = nvlist_next_nvpair(genericnvl, pair)) != NULL) {
	const char *propname = nvpair_name(pair);
	int err = 0;

	propval = pair;
	if (nvpair_type(pair) == DATA_TYPE_NVLIST) {
	nvlist_t *attrs;
	attrs = fnvpair_value_nvlist(pair);
	propval = fnvlist_lookup_nvpair(attrs,
	ZPROP_VALUE);
	}

	if (nvpair_type(propval) == DATA_TYPE_STRING) {
	strval = fnvpair_value_string(propval);
	err = dsl_prop_set_string(dsname, propname,
	source, strval);
	} else if (nvpair_type(propval) == DATA_TYPE_BOOLEAN) {
	err = dsl_prop_inherit(dsname, propname,
	source);
	} else {
	intval = fnvpair_value_uint64(propval);
	err = dsl_prop_set_int(dsname, propname, source,
	intval);
	}

	if (err != 0) {
	if (errlist != NULL) {
	fnvlist_add_int32(errlist, propname,
	err);
	}
	rv = err;
	}
	}
	}
	nvlist_free(genericnvl);
	nvlist_free(retrynvl);

	return (rv);
	}

	/*
	* Check that all the properties are valid user properties.
	*/
	static int
	zfs_check_userprops(nvlist_t *nvl)
	{
	nvpair_t *pair = NULL;

	while ((pair = nvlist_next_nvpair(nvl, pair)) != NULL) {
	const char *propname = nvpair_name(pair);

	if (!zfs_prop_user(propname) \|\|
	nvpair_type(pair) != DATA_TYPE_STRING)
	return (SET_ERROR(EINVAL));

	if (strlen(propname) >= ZAP_MAXNAMELEN)
	return (SET_ERROR(ENAMETOOLONG));

	if (strlen(fnvpair_value_string(pair)) >= ZAP_MAXVALUELEN)
	return (SET_ERROR(E2BIG));
	}
	return (0);
	}

	static void
	props_skip(nvlist_t props, nvlist_t skipped, nvlist_t **newprops)
	{
	nvpair_t *pair;

	VERIFY(nvlist_alloc(newprops, NV_UNIQUE_NAME, KM_SLEEP) == 0);

	pair = NULL;
	while ((pair = nvlist_next_nvpair(props, pair)) != NULL) {
	if (nvlist_exists(skipped, nvpair_name(pair)))
	continue;

	VERIFY(nvlist_add_nvpair(*newprops, pair) == 0);
	}
	}

	static int
	clear_received_props(const char dsname, nvlist_t props,
	nvlist_t *skipped)
	{
	int err = 0;
	nvlist_t *cleared_props = NULL;
	props_skip(props, skipped, &cleared_props);
	if (!nvlist_empty(cleared_props)) {
	/*
	* Acts on local properties until the dataset has received
	* properties at least once on or after SPA_VERSION_RECVD_PROPS.
	*/
	zprop_source_t flags = (ZPROP_SRC_NONE \|
	(dsl_prop_get_hasrecvd(dsname) ? ZPROP_SRC_RECEIVED : 0));
	err = zfs_set_prop_nvlist(dsname, flags, cleared_props, NULL);
	}
	nvlist_free(cleared_props);
	return (err);
	}

	/*
	* inputs:
	* zc_name name of filesystem
	* zc_value name of property to set
	* zc_nvlist_src{_size} nvlist of properties to apply
	* zc_cookie received properties flag
	*
	* outputs:
	* zc_nvlist_dst{_size} error for each unapplied received property
	*/
	static int
	zfs_ioc_set_prop(zfs_cmd_t *zc)
	{
	nvlist_t *nvl;
	boolean_t received = zc->zc_cookie;
	zprop_source_t source = (received ? ZPROP_SRC_RECEIVED :
	ZPROP_SRC_LOCAL);
	nvlist_t *errors;
	int error;

	if ((error = get_nvlist(zc->zc_nvlist_src, zc->zc_nvlist_src_size,
	zc->zc_iflags, &nvl)) != 0)
	return (error);

	if (received) {
	nvlist_t *origprops;

	if (dsl_prop_get_received(zc->zc_name, &origprops) == 0) {
	(void) clear_received_props(zc->zc_name,
	origprops, nvl);
	nvlist_free(origprops);
	}

	error = dsl_prop_set_hasrecvd(zc->zc_name);
	}

	errors = fnvlist_alloc();
	if (error == 0)
	error = zfs_set_prop_nvlist(zc->zc_name, source, nvl, errors);

	if (zc->zc_nvlist_dst != 0 && errors != NULL) {
	(void) put_nvlist(zc, errors);
	}

	nvlist_free(errors);
	nvlist_free(nvl);
	return (error);
	}

	/*
	* inputs:
	* zc_name name of filesystem
	* zc_value name of property to inherit
	* zc_cookie revert to received value if TRUE
	*
	* outputs: none
	*/
	static int
	zfs_ioc_inherit_prop(zfs_cmd_t *zc)
	{
	const char *propname = zc->zc_value;
	zfs_prop_t prop = zfs_name_to_prop(propname);
	boolean_t received = zc->zc_cookie;
	zprop_source_t source = (received
	? ZPROP_SRC_NONE /* revert to received value, if any */
	: ZPROP_SRC_INHERITED); /* explicitly inherit */
	nvlist_t *dummy;
	nvpair_t *pair;
	zprop_type_t type;
	int err;

	if (!received) {
	/*
	* Only check this in the non-received case. We want to allow
	* 'inherit -S' to revert non-inheritable properties like quota
	* and reservation to the received or default values even though
	* they are not considered inheritable.
	*/
	if (prop != ZPROP_INVAL && !zfs_prop_inheritable(prop))
	return (SET_ERROR(EINVAL));
	}

	if (prop == ZPROP_INVAL) {
	if (!zfs_prop_user(propname))
	return (SET_ERROR(EINVAL));

	type = PROP_TYPE_STRING;
	} else if (prop == ZFS_PROP_VOLSIZE \|\| prop == ZFS_PROP_VERSION) {
	return (SET_ERROR(EINVAL));
	} else {
	type = zfs_prop_get_type(prop);
	}

	/*
	* zfs_prop_set_special() expects properties in the form of an
	* nvpair with type info.
	*/
	dummy = fnvlist_alloc();

	switch (type) {
	case PROP_TYPE_STRING:
	VERIFY(0 == nvlist_add_string(dummy, propname, ""));
	break;
	case PROP_TYPE_NUMBER:
	case PROP_TYPE_INDEX:
	VERIFY(0 == nvlist_add_uint64(dummy, propname, 0));
	break;
	default:
	err = SET_ERROR(EINVAL);
	goto errout;
	}

	pair = nvlist_next_nvpair(dummy, NULL);
	if (pair == NULL) {
	err = SET_ERROR(EINVAL);
	} else {
	err = zfs_prop_set_special(zc->zc_name, source, pair);
	if (err == -1) /* property is not "special", needs handling */
	err = dsl_prop_inherit(zc->zc_name, zc->zc_value,
	source);
	}

	errout:
	nvlist_free(dummy);
	return (err);
	}

	static int
	zfs_ioc_pool_set_props(zfs_cmd_t *zc)
	{
	nvlist_t *props;
	spa_t *spa;
	int error;
	nvpair_t *pair;

	if ((error = get_nvlist(zc->zc_nvlist_src, zc->zc_nvlist_src_size,
	zc->zc_iflags, &props)))
	return (error);

	/*
	* If the only property is the configfile, then just do a spa_lookup()
	* to handle the faulted case.
	*/
	pair = nvlist_next_nvpair(props, NULL);
	if (pair != NULL && strcmp(nvpair_name(pair),
	zpool_prop_to_name(ZPOOL_PROP_CACHEFILE)) == 0 &&
	nvlist_next_nvpair(props, pair) == NULL) {
	mutex_enter(&spa_namespace_lock);
	if ((spa = spa_lookup(zc->zc_name)) != NULL) {
	spa_configfile_set(spa, props, B_FALSE);
	spa_write_cachefile(spa, B_FALSE, B_TRUE);
	}
	mutex_exit(&spa_namespace_lock);
	if (spa != NULL) {
	nvlist_free(props);
	return (0);
	}
	}

	if ((error = spa_open(zc->zc_name, &spa, FTAG)) != 0) {
	nvlist_free(props);
	return (error);
	}

	error = spa_prop_set(spa, props);

	nvlist_free(props);
	spa_close(spa, FTAG);

	return (error);
	}

	static int
	zfs_ioc_pool_get_props(zfs_cmd_t *zc)
	{
	spa_t *spa;
	int error;
	nvlist_t *nvp = NULL;

	if ((error = spa_open(zc->zc_name, &spa, FTAG)) != 0) {
	/*
	* If the pool is faulted, there may be properties we can still
	* get (such as altroot and cachefile), so attempt to get them
	* anyway.
	*/
	mutex_enter(&spa_namespace_lock);
	if ((spa = spa_lookup(zc->zc_name)) != NULL)
	error = spa_prop_get(spa, &nvp);
	mutex_exit(&spa_namespace_lock);
	} else {
	error = spa_prop_get(spa, &nvp);
	spa_close(spa, FTAG);
	}

	if (error == 0 && zc->zc_nvlist_dst != 0)
	error = put_nvlist(zc, nvp);
	else
	error = SET_ERROR(EFAULT);

	nvlist_free(nvp);
	return (error);
	}

	/*
	* inputs:
	* zc_name name of filesystem
	* zc_nvlist_src{_size} nvlist of delegated permissions
	* zc_perm_action allow/unallow flag
	*
	* outputs: none
	*/
	static int
	zfs_ioc_set_fsacl(zfs_cmd_t *zc)
	{
	int error;
	nvlist_t *fsaclnv = NULL;

	if ((error = get_nvlist(zc->zc_nvlist_src, zc->zc_nvlist_src_size,
	zc->zc_iflags, &fsaclnv)) != 0)
	return (error);

	/*
	* Verify nvlist is constructed correctly
	*/
	if ((error = zfs_deleg_verify_nvlist(fsaclnv)) != 0) {
	nvlist_free(fsaclnv);
	return (SET_ERROR(EINVAL));
	}

	/*
	* If we don't have PRIV_SYS_MOUNT, then validate
	* that user is allowed to hand out each permission in
	* the nvlist(s)
	*/

	error = secpolicy_zfs(CRED());
	if (error != 0) {
	if (zc->zc_perm_action == B_FALSE) {
	error = dsl_deleg_can_allow(zc->zc_name,
	fsaclnv, CRED());
	} else {
	error = dsl_deleg_can_unallow(zc->zc_name,
	fsaclnv, CRED());
	}
	}

	if (error == 0)
	error = dsl_deleg_set(zc->zc_name, fsaclnv, zc->zc_perm_action);

	nvlist_free(fsaclnv);
	return (error);
	}

	/*
	* inputs:
	* zc_name name of filesystem
	*
	* outputs:
	* zc_nvlist_src{_size} nvlist of delegated permissions
	*/
	static int
	zfs_ioc_get_fsacl(zfs_cmd_t *zc)
	{
	nvlist_t *nvp;
	int error;

	if ((error = dsl_deleg_get(zc->zc_name, &nvp)) == 0) {
	error = put_nvlist(zc, nvp);
	nvlist_free(nvp);
	}

	return (error);
	}

	/* ARGSUSED */
	static void
	zfs_create_cb(objset_t os, void arg, cred_t cr, dmu_tx_t tx)
	{
	zfs_creat_t *zct = arg;

	zfs_create_fs(os, cr, zct->zct_zplprops, tx);
	}

	#define ZFS_PROP_UNDEFINED ((uint64_t)-1)

	/*
	* inputs:
	* os parent objset pointer (NULL if root fs)
	* fuids_ok fuids allowed in this version of the spa?
	* sa_ok SAs allowed in this version of the spa?
	* createprops list of properties requested by creator
	*
	* outputs:
	* zplprops values for the zplprops we attach to the master node object
	* is_ci true if requested file system will be purely case-insensitive
	*
	* Determine the settings for utf8only, normalization and
	* casesensitivity. Specific values may have been requested by the
	* creator and/or we can inherit values from the parent dataset. If
	* the file system is of too early a vintage, a creator can not
	* request settings for these properties, even if the requested
	* setting is the default value. We don't actually want to create dsl
	* properties for these, so remove them from the source nvlist after
	* processing.
	*/
	static int
	zfs_fill_zplprops_impl(objset_t *os, uint64_t zplver,
	boolean_t fuids_ok, boolean_t sa_ok, nvlist_t *createprops,
	nvlist_t zplprops, boolean_t is_ci)
	{
	uint64_t sense = ZFS_PROP_UNDEFINED;
	uint64_t norm = ZFS_PROP_UNDEFINED;
	uint64_t u8 = ZFS_PROP_UNDEFINED;
	int error;

	ASSERT(zplprops != NULL);

	/* parent dataset must be a filesystem */
	if (os != NULL && os->os_phys->os_type != DMU_OST_ZFS)
	return (SET_ERROR(ZFS_ERR_WRONG_PARENT));

	/*
	* Pull out creator prop choices, if any.
	*/
	if (createprops) {
	(void) nvlist_lookup_uint64(createprops,
	zfs_prop_to_name(ZFS_PROP_VERSION), &zplver);
	(void) nvlist_lookup_uint64(createprops,
	zfs_prop_to_name(ZFS_PROP_NORMALIZE), &norm);
	(void) nvlist_remove_all(createprops,
	zfs_prop_to_name(ZFS_PROP_NORMALIZE));
	(void) nvlist_lookup_uint64(createprops,
	zfs_prop_to_name(ZFS_PROP_UTF8ONLY), &u8);
	(void) nvlist_remove_all(createprops,
	zfs_prop_to_name(ZFS_PROP_UTF8ONLY));
	(void) nvlist_lookup_uint64(createprops,
	zfs_prop_to_name(ZFS_PROP_CASE), &sense);
	(void) nvlist_remove_all(createprops,
	zfs_prop_to_name(ZFS_PROP_CASE));
	}

	/*
	* If the zpl version requested is whacky or the file system
	* or pool is version is too "young" to support normalization
	* and the creator tried to set a value for one of the props,
	* error out.
	*/
	if ((zplver < ZPL_VERSION_INITIAL \|\| zplver > ZPL_VERSION) \|\|
	(zplver >= ZPL_VERSION_FUID && !fuids_ok) \|\|
	(zplver >= ZPL_VERSION_SA && !sa_ok) \|\|
	(zplver < ZPL_VERSION_NORMALIZATION &&
	(norm != ZFS_PROP_UNDEFINED \|\| u8 != ZFS_PROP_UNDEFINED \|\|
	sense != ZFS_PROP_UNDEFINED)))
	return (SET_ERROR(ENOTSUP));

	/*
	* Put the version in the zplprops
	*/
	VERIFY(nvlist_add_uint64(zplprops,
	zfs_prop_to_name(ZFS_PROP_VERSION), zplver) == 0);

	if (norm == ZFS_PROP_UNDEFINED &&
	(error = zfs_get_zplprop(os, ZFS_PROP_NORMALIZE, &norm)) != 0)
	return (error);
	VERIFY(nvlist_add_uint64(zplprops,
	zfs_prop_to_name(ZFS_PROP_NORMALIZE), norm) == 0);

	/*
	* If we're normalizing, names must always be valid UTF-8 strings.
	*/
	if (norm)
	u8 = 1;
	if (u8 == ZFS_PROP_UNDEFINED &&
	(error = zfs_get_zplprop(os, ZFS_PROP_UTF8ONLY, &u8)) != 0)
	return (error);
	VERIFY(nvlist_add_uint64(zplprops,
	zfs_prop_to_name(ZFS_PROP_UTF8ONLY), u8) == 0);

	if (sense == ZFS_PROP_UNDEFINED &&
	(error = zfs_get_zplprop(os, ZFS_PROP_CASE, &sense)) != 0)
	return (error);
	VERIFY(nvlist_add_uint64(zplprops,
	zfs_prop_to_name(ZFS_PROP_CASE), sense) == 0);

	if (is_ci)
	*is_ci = (sense == ZFS_CASE_INSENSITIVE);

	return (0);
	}

	static int
	zfs_fill_zplprops(const char dataset, nvlist_t createprops,
	nvlist_t zplprops, boolean_t is_ci)
	{
	boolean_t fuids_ok, sa_ok;
	uint64_t zplver = ZPL_VERSION;
	objset_t *os = NULL;
	char parentname[ZFS_MAX_DATASET_NAME_LEN];
	spa_t *spa;
	uint64_t spa_vers;
	int error;

	zfs_get_parent(dataset, parentname, sizeof (parentname));

	if ((error = spa_open(dataset, &spa, FTAG)) != 0)
	return (error);

	spa_vers = spa_version(spa);
	spa_close(spa, FTAG);

	zplver = zfs_zpl_version_map(spa_vers);
	fuids_ok = (zplver >= ZPL_VERSION_FUID);
	sa_ok = (zplver >= ZPL_VERSION_SA);

	/*
	* Open parent object set so we can inherit zplprop values.
	*/
	if ((error = dmu_objset_hold(parentname, FTAG, &os)) != 0)
	return (error);

	error = zfs_fill_zplprops_impl(os, zplver, fuids_ok, sa_ok, createprops,
	zplprops, is_ci);
	dmu_objset_rele(os, FTAG);
	return (error);
	}

	static int
	zfs_fill_zplprops_root(uint64_t spa_vers, nvlist_t *createprops,
	nvlist_t zplprops, boolean_t is_ci)
	{
	boolean_t fuids_ok;
	boolean_t sa_ok;
	uint64_t zplver = ZPL_VERSION;
	int error;

	zplver = zfs_zpl_version_map(spa_vers);
	fuids_ok = (zplver >= ZPL_VERSION_FUID);
	sa_ok = (zplver >= ZPL_VERSION_SA);

	error = zfs_fill_zplprops_impl(NULL, zplver, fuids_ok, sa_ok,
	createprops, zplprops, is_ci);
	return (error);
	}

	/*
	* innvl: {
	* "type" -> dmu_objset_type_t (int32)
	* (optional) "props" -> { prop -> value }
	* (optional) "hidden_args" -> { "wkeydata" -> value }
	* raw uint8_t array of encryption wrapping key data (32 bytes)
	* }
	*
	* outnvl: propname -> error code (int32)
	*/

	static const zfs_ioc_key_t zfs_keys_create[] = {
	{"type", DATA_TYPE_INT32, 0},
	{"props", DATA_TYPE_NVLIST, ZK_OPTIONAL},
	{"hidden_args", DATA_TYPE_NVLIST, ZK_OPTIONAL},
	};

	static int
	zfs_ioc_create(const char fsname, nvlist_t innvl, nvlist_t *outnvl)
	{
	int error = 0;
	zfs_creat_t zct = { 0 };
	nvlist_t *nvprops = NULL;
	nvlist_t *hidden_args = NULL;
	void (cbfunc)(objset_t os, void arg, cred_t cr, dmu_tx_t *tx);
	dmu_objset_type_t type;
	boolean_t is_insensitive = B_FALSE;
	dsl_crypto_params_t *dcp = NULL;

	type = (dmu_objset_type_t)fnvlist_lookup_int32(innvl, "type");
	(void) nvlist_lookup_nvlist(innvl, "props", &nvprops);
	(void) nvlist_lookup_nvlist(innvl, ZPOOL_HIDDEN_ARGS, &hidden_args);

	switch (type) {
	case DMU_OST_ZFS:
	cbfunc = zfs_create_cb;
	break;

	case DMU_OST_ZVOL:
	cbfunc = zvol_create_cb;
	break;

	default:
	cbfunc = NULL;
	break;
	}
	if (strchr(fsname, '@') \|\|
	strchr(fsname, '%'))
	return (SET_ERROR(EINVAL));

	zct.zct_props = nvprops;

	if (cbfunc == NULL)
	return (SET_ERROR(EINVAL));

	if (type == DMU_OST_ZVOL) {
	uint64_t volsize, volblocksize;

	if (nvprops == NULL)
	return (SET_ERROR(EINVAL));
	if (nvlist_lookup_uint64(nvprops,
	zfs_prop_to_name(ZFS_PROP_VOLSIZE), &volsize) != 0)
	return (SET_ERROR(EINVAL));

	if ((error = nvlist_lookup_uint64(nvprops,
	zfs_prop_to_name(ZFS_PROP_VOLBLOCKSIZE),
	&volblocksize)) != 0 && error != ENOENT)
	return (SET_ERROR(EINVAL));

	if (error != 0)
	volblocksize = zfs_prop_default_numeric(
	ZFS_PROP_VOLBLOCKSIZE);

	if ((error = zvol_check_volblocksize(fsname,
	volblocksize)) != 0 \|\|
	(error = zvol_check_volsize(volsize,
	volblocksize)) != 0)
	return (error);
	} else if (type == DMU_OST_ZFS) {
	int error;

	/*
	* We have to have normalization and
	* case-folding flags correct when we do the
	* file system creation, so go figure them out
	* now.
	*/
	VERIFY(nvlist_alloc(&zct.zct_zplprops,
	NV_UNIQUE_NAME, KM_SLEEP) == 0);
	error = zfs_fill_zplprops(fsname, nvprops,
	zct.zct_zplprops, &is_insensitive);
	if (error != 0) {
	nvlist_free(zct.zct_zplprops);
	return (error);
	}
	}

	error = dsl_crypto_params_create_nvlist(DCP_CMD_NONE, nvprops,
	hidden_args, &dcp);
	if (error != 0) {
	nvlist_free(zct.zct_zplprops);
	return (error);
	}

	error = dmu_objset_create(fsname, type,
	is_insensitive ? DS_FLAG_CI_DATASET : 0, dcp, cbfunc, &zct);

	nvlist_free(zct.zct_zplprops);
	dsl_crypto_params_free(dcp, !!error);

	/*
	* It would be nice to do this atomically.
	*/
	if (error == 0) {
	error = zfs_set_prop_nvlist(fsname, ZPROP_SRC_LOCAL,
	nvprops, outnvl);
	if (error != 0) {
	spa_t *spa;
	int error2;

	/*
	* Volumes will return EBUSY and cannot be destroyed
	* until all asynchronous minor handling (e.g. from
	* setting the volmode property) has completed. Wait for
	* the spa_zvol_taskq to drain then retry.
	*/
	error2 = dsl_destroy_head(fsname);
	while ((error2 == EBUSY) && (type == DMU_OST_ZVOL)) {
	error2 = spa_open(fsname, &spa, FTAG);
	if (error2 == 0) {
	taskq_wait(spa->spa_zvol_taskq);
	spa_close(spa, FTAG);
	}
	error2 = dsl_destroy_head(fsname);
	}
	}
	}
	return (error);
	}

	/*
	* innvl: {
	* "origin" -> name of origin snapshot
	* (optional) "props" -> { prop -> value }
	* (optional) "hidden_args" -> { "wkeydata" -> value }
	* raw uint8_t array of encryption wrapping key data (32 bytes)
	* }
	*
	* outputs:
	* outnvl: propname -> error code (int32)
	*/
	static const zfs_ioc_key_t zfs_keys_clone[] = {
	{"origin", DATA_TYPE_STRING, 0},
	{"props", DATA_TYPE_NVLIST, ZK_OPTIONAL},
	{"hidden_args", DATA_TYPE_NVLIST, ZK_OPTIONAL},
	};

	static int
	zfs_ioc_clone(const char fsname, nvlist_t innvl, nvlist_t *outnvl)
	{
	int error = 0;
	nvlist_t *nvprops = NULL;
	- char *origin_name;
	+ const char *origin_name;

	origin_name = fnvlist_lookup_string(innvl, "origin");
	(void) nvlist_lookup_nvlist(innvl, "props", &nvprops);

	if (strchr(fsname, '@') \|\|
	strchr(fsname, '%'))
	return (SET_ERROR(EINVAL));

	if (dataset_namecheck(origin_name, NULL, NULL) != 0)
	return (SET_ERROR(EINVAL));

	error = dmu_objset_clone(fsname, origin_name);

	/*
	* It would be nice to do this atomically.
	*/
	if (error == 0) {
	error = zfs_set_prop_nvlist(fsname, ZPROP_SRC_LOCAL,
	nvprops, outnvl);
	if (error != 0)
	(void) dsl_destroy_head(fsname);
	}
	return (error);
	}

	static const zfs_ioc_key_t zfs_keys_remap[] = {
	/* no nvl keys */
	};

	/* ARGSUSED */
	static int
	zfs_ioc_remap(const char fsname, nvlist_t innvl, nvlist_t *outnvl)
	{
	/* This IOCTL is no longer supported. */
	return (0);
	}

	/*
	* innvl: {
	* "snaps" -> { snapshot1, snapshot2 }
	* (optional) "props" -> { prop -> value (string) }
	* }
	*
	* outnvl: snapshot -> error code (int32)
	*/
	static const zfs_ioc_key_t zfs_keys_snapshot[] = {
	{"snaps", DATA_TYPE_NVLIST, 0},
	{"props", DATA_TYPE_NVLIST, ZK_OPTIONAL},
	};

	static int
	zfs_ioc_snapshot(const char poolname, nvlist_t innvl, nvlist_t *outnvl)
	{
	nvlist_t *snaps;
	nvlist_t *props = NULL;
	int error, poollen;
	nvpair_t *pair;

	(void) nvlist_lookup_nvlist(innvl, "props", &props);
	if (!nvlist_empty(props) &&
	zfs_earlier_version(poolname, SPA_VERSION_SNAP_PROPS))
	return (SET_ERROR(ENOTSUP));
	if ((error = zfs_check_userprops(props)) != 0)
	return (error);

	snaps = fnvlist_lookup_nvlist(innvl, "snaps");
	poollen = strlen(poolname);
	for (pair = nvlist_next_nvpair(snaps, NULL); pair != NULL;
	pair = nvlist_next_nvpair(snaps, pair)) {
	const char *name = nvpair_name(pair);
	char *cp = strchr(name, '@');

	/*
	* The snap name must contain an @, and the part after it must
	* contain only valid characters.
	*/
	if (cp == NULL \|\|
	zfs_component_namecheck(cp + 1, NULL, NULL) != 0)
	return (SET_ERROR(EINVAL));

	/*
	* The snap must be in the specified pool.
	*/
	if (strncmp(name, poolname, poollen) != 0 \|\|
	(name[poollen] != '/' && name[poollen] != '@'))
	return (SET_ERROR(EXDEV));

	/*
	* Check for permission to set the properties on the fs.
	*/
	if (!nvlist_empty(props)) {
	*cp = '\0';
	error = zfs_secpolicy_write_perms(name,
	ZFS_DELEG_PERM_USERPROP, CRED());
	*cp = '@';
	if (error != 0)
	return (error);
	}

	/* This must be the only snap of this fs. */
	for (nvpair_t *pair2 = nvlist_next_nvpair(snaps, pair);
	pair2 != NULL; pair2 = nvlist_next_nvpair(snaps, pair2)) {
	if (strncmp(name, nvpair_name(pair2), cp - name + 1)
	== 0) {
	return (SET_ERROR(EXDEV));
	}
	}
	}

	error = dsl_dataset_snapshot(snaps, props, outnvl);

	return (error);
	}

	/*
	* innvl: "message" -> string
	*/
	static const zfs_ioc_key_t zfs_keys_log_history[] = {
	{"message", DATA_TYPE_STRING, 0},
	};

	/* ARGSUSED */
	static int
	zfs_ioc_log_history(const char unused, nvlist_t innvl, nvlist_t *outnvl)
	{
	- char *message;
	+ const char *message;
	+ char *poolname;
	spa_t *spa;
	int error;
	- char *poolname;

	/*
	* The poolname in the ioctl is not set, we get it from the TSD,
	* which was set at the end of the last successful ioctl that allows
	* logging. The secpolicy func already checked that it is set.
	* Only one log ioctl is allowed after each successful ioctl, so
	* we clear the TSD here.
	*/
	poolname = tsd_get(zfs_allow_log_key);
	if (poolname == NULL)
	return (SET_ERROR(EINVAL));
	(void) tsd_set(zfs_allow_log_key, NULL);
	error = spa_open(poolname, &spa, FTAG);
	kmem_strfree(poolname);
	if (error != 0)
	return (error);

	message = fnvlist_lookup_string(innvl, "message");

	if (spa_version(spa) < SPA_VERSION_ZPOOL_HISTORY) {
	spa_close(spa, FTAG);
	return (SET_ERROR(ENOTSUP));
	}

	error = spa_history_log(spa, message);
	spa_close(spa, FTAG);
	return (error);
	}

	/*
	* This ioctl is used to set the bootenv configuration on the current
	* pool. This configuration is stored in the second padding area of the label,
	* and it is used by the bootloader(s) to store the bootloader and/or system
	* specific data.
	* The data is stored as nvlist data stream, and is protected by
	* an embedded checksum.
	* The version can have two possible values:
	* VB_RAW: nvlist should have key GRUB_ENVMAP, value DATA_TYPE_STRING.
	* VB_NVLIST: nvlist with arbitrary <key, value> pairs.
	*/
	static const zfs_ioc_key_t zfs_keys_set_bootenv[] = {
	{"version", DATA_TYPE_UINT64, 0},
	{"<keys>", DATA_TYPE_ANY, ZK_OPTIONAL \| ZK_WILDCARDLIST},
	};

	static int
	zfs_ioc_set_bootenv(const char name, nvlist_t innvl, nvlist_t *outnvl)
	{
	int error;
	spa_t *spa;

	if ((error = spa_open(name, &spa, FTAG)) != 0)
	return (error);
	spa_vdev_state_enter(spa, SCL_ALL);
	error = vdev_label_write_bootenv(spa->spa_root_vdev, innvl);
	(void) spa_vdev_state_exit(spa, NULL, 0);
	spa_close(spa, FTAG);
	return (error);
	}

	static const zfs_ioc_key_t zfs_keys_get_bootenv[] = {
	/* no nvl keys */
	};

	static int
	zfs_ioc_get_bootenv(const char name, nvlist_t innvl, nvlist_t *outnvl)
	{
	spa_t *spa;
	int error;

	if ((error = spa_open(name, &spa, FTAG)) != 0)
	return (error);
	spa_vdev_state_enter(spa, SCL_ALL);
	error = vdev_label_read_bootenv(spa->spa_root_vdev, outnvl);
	(void) spa_vdev_state_exit(spa, NULL, 0);
	spa_close(spa, FTAG);
	return (error);
	}

	/*
	* The dp_config_rwlock must not be held when calling this, because the
	* unmount may need to write out data.
	*
	* This function is best-effort. Callers must deal gracefully if it
	* remains mounted (or is remounted after this call).
	*
	* Returns 0 if the argument is not a snapshot, or it is not currently a
	* filesystem, or we were able to unmount it. Returns error code otherwise.
	*/
	void
	zfs_unmount_snap(const char *snapname)
	{
	if (strchr(snapname, '@') == NULL)
	return;

	- (void) zfsctl_snapshot_unmount((char *)snapname, MNT_FORCE);
	+ (void) zfsctl_snapshot_unmount(snapname, MNT_FORCE);
	}

	/* ARGSUSED */
	static int
	zfs_unmount_snap_cb(const char snapname, void arg)
	{
	zfs_unmount_snap(snapname);
	return (0);
	}

	/*
	* When a clone is destroyed, its origin may also need to be destroyed,
	* in which case it must be unmounted. This routine will do that unmount
	* if necessary.
	*/
	void
	zfs_destroy_unmount_origin(const char *fsname)
	{
	int error;
	objset_t *os;
	dsl_dataset_t *ds;

	error = dmu_objset_hold(fsname, FTAG, &os);
	if (error != 0)
	return;
	ds = dmu_objset_ds(os);
	if (dsl_dir_is_clone(ds->ds_dir) && DS_IS_DEFER_DESTROY(ds->ds_prev)) {
	char originname[ZFS_MAX_DATASET_NAME_LEN];
	dsl_dataset_name(ds->ds_prev, originname);
	dmu_objset_rele(os, FTAG);
	zfs_unmount_snap(originname);
	} else {
	dmu_objset_rele(os, FTAG);
	}
	}

	/*
	* innvl: {
	* "snaps" -> { snapshot1, snapshot2 }
	* (optional boolean) "defer"
	* }
	*
	* outnvl: snapshot -> error code (int32)
	*/
	static const zfs_ioc_key_t zfs_keys_destroy_snaps[] = {
	{"snaps", DATA_TYPE_NVLIST, 0},
	{"defer", DATA_TYPE_BOOLEAN, ZK_OPTIONAL},
	};

	/* ARGSUSED */
	static int
	zfs_ioc_destroy_snaps(const char poolname, nvlist_t innvl, nvlist_t *outnvl)
	{
	int poollen;
	nvlist_t *snaps;
	nvpair_t *pair;
	boolean_t defer;
	spa_t *spa;

	snaps = fnvlist_lookup_nvlist(innvl, "snaps");
	defer = nvlist_exists(innvl, "defer");

	poollen = strlen(poolname);
	for (pair = nvlist_next_nvpair(snaps, NULL); pair != NULL;
	pair = nvlist_next_nvpair(snaps, pair)) {
	const char *name = nvpair_name(pair);

	/*
	* The snap must be in the specified pool to prevent the
	* invalid removal of zvol minors below.
	*/
	if (strncmp(name, poolname, poollen) != 0 \|\|
	(name[poollen] != '/' && name[poollen] != '@'))
	return (SET_ERROR(EXDEV));

	zfs_unmount_snap(nvpair_name(pair));
	if (spa_open(name, &spa, FTAG) == 0) {
	zvol_remove_minors(spa, name, B_TRUE);
	spa_close(spa, FTAG);
	}
	}

	return (dsl_destroy_snapshots_nvl(snaps, defer, outnvl));
	}

	/*
	* Create bookmarks. The bookmark names are of the form <fs>#<bmark>.
	* All bookmarks and snapshots must be in the same pool.
	* dsl_bookmark_create_nvl_validate describes the nvlist schema in more detail.
	*
	* innvl: {
	* new_bookmark1 -> existing_snapshot,
	* new_bookmark2 -> existing_bookmark,
	* }
	*
	* outnvl: bookmark -> error code (int32)
	*
	*/
	static const zfs_ioc_key_t zfs_keys_bookmark[] = {
	{"<bookmark>...", DATA_TYPE_STRING, ZK_WILDCARDLIST},
	};

	/* ARGSUSED */
	static int
	zfs_ioc_bookmark(const char poolname, nvlist_t innvl, nvlist_t *outnvl)
	{
	return (dsl_bookmark_create(innvl, outnvl));
	}

	/*
	* innvl: {
	* property 1, property 2, ...
	* }
	*
	* outnvl: {
	* bookmark name 1 -> { property 1, property 2, ... },
	* bookmark name 2 -> { property 1, property 2, ... }
	* }
	*
	*/
	static const zfs_ioc_key_t zfs_keys_get_bookmarks[] = {
	{"<property>...", DATA_TYPE_BOOLEAN, ZK_WILDCARDLIST \| ZK_OPTIONAL},
	};

	static int
	zfs_ioc_get_bookmarks(const char fsname, nvlist_t innvl, nvlist_t *outnvl)
	{
	return (dsl_get_bookmarks(fsname, innvl, outnvl));
	}

	/*
	* innvl is not used.
	*
	* outnvl: {
	* property 1, property 2, ...
	* }
	*
	*/
	static const zfs_ioc_key_t zfs_keys_get_bookmark_props[] = {
	/* no nvl keys */
	};

	/* ARGSUSED */
	static int
	zfs_ioc_get_bookmark_props(const char bookmark, nvlist_t innvl,
	nvlist_t *outnvl)
	{
	char fsname[ZFS_MAX_DATASET_NAME_LEN];
	char *bmname;

	bmname = strchr(bookmark, '#');
	if (bmname == NULL)
	return (SET_ERROR(EINVAL));
	bmname++;

	(void) strlcpy(fsname, bookmark, sizeof (fsname));
	*(strchr(fsname, '#')) = '\0';

	return (dsl_get_bookmark_props(fsname, bmname, outnvl));
	}

	/*
	* innvl: {
	* bookmark name 1, bookmark name 2
	* }
	*
	* outnvl: bookmark -> error code (int32)
	*
	*/
	static const zfs_ioc_key_t zfs_keys_destroy_bookmarks[] = {
	{"<bookmark>...", DATA_TYPE_BOOLEAN, ZK_WILDCARDLIST},
	};

	static int
	zfs_ioc_destroy_bookmarks(const char poolname, nvlist_t innvl,
	nvlist_t *outnvl)
	{
	int error, poollen;

	poollen = strlen(poolname);
	for (nvpair_t *pair = nvlist_next_nvpair(innvl, NULL);
	pair != NULL; pair = nvlist_next_nvpair(innvl, pair)) {
	const char *name = nvpair_name(pair);
	const char *cp = strchr(name, '#');

	/*
	* The bookmark name must contain an #, and the part after it
	* must contain only valid characters.
	*/
	if (cp == NULL \|\|
	zfs_component_namecheck(cp + 1, NULL, NULL) != 0)
	return (SET_ERROR(EINVAL));

	/*
	* The bookmark must be in the specified pool.
	*/
	if (strncmp(name, poolname, poollen) != 0 \|\|
	(name[poollen] != '/' && name[poollen] != '#'))
	return (SET_ERROR(EXDEV));
	}

	error = dsl_bookmark_destroy(innvl, outnvl);
	return (error);
	}

	static const zfs_ioc_key_t zfs_keys_channel_program[] = {
	{"program", DATA_TYPE_STRING, 0},
	{"arg", DATA_TYPE_ANY, 0},
	{"sync", DATA_TYPE_BOOLEAN_VALUE, ZK_OPTIONAL},
	{"instrlimit", DATA_TYPE_UINT64, ZK_OPTIONAL},
	{"memlimit", DATA_TYPE_UINT64, ZK_OPTIONAL},
	};

	static int
	zfs_ioc_channel_program(const char poolname, nvlist_t innvl,
	nvlist_t *outnvl)
	{
	char *program;
	uint64_t instrlimit, memlimit;
	boolean_t sync_flag;
	nvpair_t *nvarg = NULL;

	program = fnvlist_lookup_string(innvl, ZCP_ARG_PROGRAM);
	if (0 != nvlist_lookup_boolean_value(innvl, ZCP_ARG_SYNC, &sync_flag)) {
	sync_flag = B_TRUE;
	}
	if (0 != nvlist_lookup_uint64(innvl, ZCP_ARG_INSTRLIMIT, &instrlimit)) {
	instrlimit = ZCP_DEFAULT_INSTRLIMIT;
	}
	if (0 != nvlist_lookup_uint64(innvl, ZCP_ARG_MEMLIMIT, &memlimit)) {
	memlimit = ZCP_DEFAULT_MEMLIMIT;
	}
	nvarg = fnvlist_lookup_nvpair(innvl, ZCP_ARG_ARGLIST);

	if (instrlimit == 0 \|\| instrlimit > zfs_lua_max_instrlimit)
	return (SET_ERROR(EINVAL));
	if (memlimit == 0 \|\| memlimit > zfs_lua_max_memlimit)
	return (SET_ERROR(EINVAL));

	return (zcp_eval(poolname, program, sync_flag, instrlimit, memlimit,
	nvarg, outnvl));
	}

	/*
	* innvl: unused
	* outnvl: empty
	*/
	static const zfs_ioc_key_t zfs_keys_pool_checkpoint[] = {
	/* no nvl keys */
	};

	/* ARGSUSED */
	static int
	zfs_ioc_pool_checkpoint(const char poolname, nvlist_t innvl, nvlist_t *outnvl)
	{
	return (spa_checkpoint(poolname));
	}

	/*
	* innvl: unused
	* outnvl: empty
	*/
	static const zfs_ioc_key_t zfs_keys_pool_discard_checkpoint[] = {
	/* no nvl keys */
	};

	/* ARGSUSED */
	static int
	zfs_ioc_pool_discard_checkpoint(const char poolname, nvlist_t innvl,
	nvlist_t *outnvl)
	{
	return (spa_checkpoint_discard(poolname));
	}

	/*
	* inputs:
	* zc_name name of dataset to destroy
	* zc_defer_destroy mark for deferred destroy
	*
	* outputs: none
	*/
	static int
	zfs_ioc_destroy(zfs_cmd_t *zc)
	{
	objset_t *os;
	dmu_objset_type_t ost;
	int err;

	err = dmu_objset_hold(zc->zc_name, FTAG, &os);
	if (err != 0)
	return (err);
	ost = dmu_objset_type(os);
	dmu_objset_rele(os, FTAG);

	if (ost == DMU_OST_ZFS)
	zfs_unmount_snap(zc->zc_name);

	if (strchr(zc->zc_name, '@')) {
	err = dsl_destroy_snapshot(zc->zc_name, zc->zc_defer_destroy);
	} else {
	err = dsl_destroy_head(zc->zc_name);
	if (err == EEXIST) {
	/*
	* It is possible that the given DS may have
	* hidden child (%recv) datasets - "leftovers"
	* resulting from the previously interrupted
	* 'zfs receive'.
	*
	* 6 extra bytes for /%recv
	*/
	char namebuf[ZFS_MAX_DATASET_NAME_LEN + 6];

	if (snprintf(namebuf, sizeof (namebuf), "%s/%s",
	zc->zc_name, recv_clone_name) >=
	sizeof (namebuf))
	return (SET_ERROR(EINVAL));

	/*
	* Try to remove the hidden child (%recv) and after
	* that try to remove the target dataset.
	* If the hidden child (%recv) does not exist
	* the original error (EEXIST) will be returned
	*/
	err = dsl_destroy_head(namebuf);
	if (err == 0)
	err = dsl_destroy_head(zc->zc_name);
	else if (err == ENOENT)
	err = SET_ERROR(EEXIST);
	}
	}

	return (err);
	}

	/*
	* innvl: {
	* "initialize_command" -> POOL_INITIALIZE_{CANCEL\|START\|SUSPEND} (uint64)
	* "initialize_vdevs": { -> guids to initialize (nvlist)
	* "vdev_path_1": vdev_guid_1, (uint64),
	* "vdev_path_2": vdev_guid_2, (uint64),
	* ...
	* },
	* }
	*
	* outnvl: {
	* "initialize_vdevs": { -> initialization errors (nvlist)
	* "vdev_path_1": errno, see function body for possible errnos (uint64)
	* "vdev_path_2": errno, ... (uint64)
	* ...
	* }
	* }
	*
	* EINVAL is returned for an unknown commands or if any of the provided vdev
	* guids have be specified with a type other than uint64.
	*/
	static const zfs_ioc_key_t zfs_keys_pool_initialize[] = {
	{ZPOOL_INITIALIZE_COMMAND, DATA_TYPE_UINT64, 0},
	{ZPOOL_INITIALIZE_VDEVS, DATA_TYPE_NVLIST, 0}
	};

	static int
	zfs_ioc_pool_initialize(const char poolname, nvlist_t innvl, nvlist_t *outnvl)
	{
	uint64_t cmd_type;
	if (nvlist_lookup_uint64(innvl, ZPOOL_INITIALIZE_COMMAND,
	&cmd_type) != 0) {
	return (SET_ERROR(EINVAL));
	}

	if (!(cmd_type == POOL_INITIALIZE_CANCEL \|\|
	cmd_type == POOL_INITIALIZE_START \|\|
	cmd_type == POOL_INITIALIZE_SUSPEND)) {
	return (SET_ERROR(EINVAL));
	}

	nvlist_t *vdev_guids;
	if (nvlist_lookup_nvlist(innvl, ZPOOL_INITIALIZE_VDEVS,
	&vdev_guids) != 0) {
	return (SET_ERROR(EINVAL));
	}

	for (nvpair_t *pair = nvlist_next_nvpair(vdev_guids, NULL);
	pair != NULL; pair = nvlist_next_nvpair(vdev_guids, pair)) {
	uint64_t vdev_guid;
	if (nvpair_value_uint64(pair, &vdev_guid) != 0) {
	return (SET_ERROR(EINVAL));
	}
	}

	spa_t *spa;
	int error = spa_open(poolname, &spa, FTAG);
	if (error != 0)
	return (error);

	nvlist_t *vdev_errlist = fnvlist_alloc();
	int total_errors = spa_vdev_initialize(spa, vdev_guids, cmd_type,
	vdev_errlist);

	if (fnvlist_size(vdev_errlist) > 0) {
	fnvlist_add_nvlist(outnvl, ZPOOL_INITIALIZE_VDEVS,
	vdev_errlist);
	}
	fnvlist_free(vdev_errlist);

	spa_close(spa, FTAG);
	return (total_errors > 0 ? EINVAL : 0);
	}

	/*
	* innvl: {
	* "trim_command" -> POOL_TRIM_{CANCEL\|START\|SUSPEND} (uint64)
	* "trim_vdevs": { -> guids to TRIM (nvlist)
	* "vdev_path_1": vdev_guid_1, (uint64),
	* "vdev_path_2": vdev_guid_2, (uint64),
	* ...
	* },
	* "trim_rate" -> Target TRIM rate in bytes/sec.
	* "trim_secure" -> Set to request a secure TRIM.
	* }
	*
	* outnvl: {
	* "trim_vdevs": { -> TRIM errors (nvlist)
	* "vdev_path_1": errno, see function body for possible errnos (uint64)
	* "vdev_path_2": errno, ... (uint64)
	* ...
	* }
	* }
	*
	* EINVAL is returned for an unknown commands or if any of the provided vdev
	* guids have be specified with a type other than uint64.
	*/
	static const zfs_ioc_key_t zfs_keys_pool_trim[] = {
	{ZPOOL_TRIM_COMMAND, DATA_TYPE_UINT64, 0},
	{ZPOOL_TRIM_VDEVS, DATA_TYPE_NVLIST, 0},
	{ZPOOL_TRIM_RATE, DATA_TYPE_UINT64, ZK_OPTIONAL},
	{ZPOOL_TRIM_SECURE, DATA_TYPE_BOOLEAN_VALUE, ZK_OPTIONAL},
	};

	static int
	zfs_ioc_pool_trim(const char poolname, nvlist_t innvl, nvlist_t *outnvl)
	{
	uint64_t cmd_type;
	if (nvlist_lookup_uint64(innvl, ZPOOL_TRIM_COMMAND, &cmd_type) != 0)
	return (SET_ERROR(EINVAL));

	if (!(cmd_type == POOL_TRIM_CANCEL \|\|
	cmd_type == POOL_TRIM_START \|\|
	cmd_type == POOL_TRIM_SUSPEND)) {
	return (SET_ERROR(EINVAL));
	}

	nvlist_t *vdev_guids;
	if (nvlist_lookup_nvlist(innvl, ZPOOL_TRIM_VDEVS, &vdev_guids) != 0)
	return (SET_ERROR(EINVAL));

	for (nvpair_t *pair = nvlist_next_nvpair(vdev_guids, NULL);
	pair != NULL; pair = nvlist_next_nvpair(vdev_guids, pair)) {
	uint64_t vdev_guid;
	if (nvpair_value_uint64(pair, &vdev_guid) != 0) {
	return (SET_ERROR(EINVAL));
	}
	}

	/* Optional, defaults to maximum rate when not provided */
	uint64_t rate;
	if (nvlist_lookup_uint64(innvl, ZPOOL_TRIM_RATE, &rate) != 0)
	rate = 0;

	/* Optional, defaults to standard TRIM when not provided */
	boolean_t secure;
	if (nvlist_lookup_boolean_value(innvl, ZPOOL_TRIM_SECURE,
	&secure) != 0) {
	secure = B_FALSE;
	}

	spa_t *spa;
	int error = spa_open(poolname, &spa, FTAG);
	if (error != 0)
	return (error);

	nvlist_t *vdev_errlist = fnvlist_alloc();
	int total_errors = spa_vdev_trim(spa, vdev_guids, cmd_type,
	rate, !!zfs_trim_metaslab_skip, secure, vdev_errlist);

	if (fnvlist_size(vdev_errlist) > 0)
	fnvlist_add_nvlist(outnvl, ZPOOL_TRIM_VDEVS, vdev_errlist);

	fnvlist_free(vdev_errlist);

	spa_close(spa, FTAG);
	return (total_errors > 0 ? EINVAL : 0);
	}

	/*
	* This ioctl waits for activity of a particular type to complete. If there is
	* no activity of that type in progress, it returns immediately, and the
	* returned value "waited" is false. If there is activity in progress, and no
	* tag is passed in, the ioctl blocks until all activity of that type is
	* complete, and then returns with "waited" set to true.
	*
	* If a tag is provided, it identifies a particular instance of an activity to
	* wait for. Currently, this is only valid for use with 'initialize', because
	* that is the only activity for which there can be multiple instances running
	* concurrently. In the case of 'initialize', the tag corresponds to the guid of
	* the vdev on which to wait.
	*
	* If a thread waiting in the ioctl receives a signal, the call will return
	* immediately, and the return value will be EINTR.
	*
	* innvl: {
	* "wait_activity" -> int32_t
	* (optional) "wait_tag" -> uint64_t
	* }
	*
	* outnvl: "waited" -> boolean_t
	*/
	static const zfs_ioc_key_t zfs_keys_pool_wait[] = {
	{ZPOOL_WAIT_ACTIVITY, DATA_TYPE_INT32, 0},
	{ZPOOL_WAIT_TAG, DATA_TYPE_UINT64, ZK_OPTIONAL},
	};

	static int
	zfs_ioc_wait(const char name, nvlist_t innvl, nvlist_t *outnvl)
	{
	int32_t activity;
	uint64_t tag;
	boolean_t waited;
	int error;

	if (nvlist_lookup_int32(innvl, ZPOOL_WAIT_ACTIVITY, &activity) != 0)
	return (EINVAL);

	if (nvlist_lookup_uint64(innvl, ZPOOL_WAIT_TAG, &tag) == 0)
	error = spa_wait_tag(name, activity, tag, &waited);
	else
	error = spa_wait(name, activity, &waited);

	if (error == 0)
	fnvlist_add_boolean_value(outnvl, ZPOOL_WAIT_WAITED, waited);

	return (error);
	}

	/*
	* This ioctl waits for activity of a particular type to complete. If there is
	* no activity of that type in progress, it returns immediately, and the
	* returned value "waited" is false. If there is activity in progress, and no
	* tag is passed in, the ioctl blocks until all activity of that type is
	* complete, and then returns with "waited" set to true.
	*
	* If a thread waiting in the ioctl receives a signal, the call will return
	* immediately, and the return value will be EINTR.
	*
	* innvl: {
	* "wait_activity" -> int32_t
	* }
	*
	* outnvl: "waited" -> boolean_t
	*/
	static const zfs_ioc_key_t zfs_keys_fs_wait[] = {
	{ZFS_WAIT_ACTIVITY, DATA_TYPE_INT32, 0},
	};

	static int
	zfs_ioc_wait_fs(const char name, nvlist_t innvl, nvlist_t *outnvl)
	{
	int32_t activity;
	boolean_t waited = B_FALSE;
	int error;
	dsl_pool_t *dp;
	dsl_dir_t *dd;
	dsl_dataset_t *ds;

	if (nvlist_lookup_int32(innvl, ZFS_WAIT_ACTIVITY, &activity) != 0)
	return (SET_ERROR(EINVAL));

	if (activity >= ZFS_WAIT_NUM_ACTIVITIES \|\| activity < 0)
	return (SET_ERROR(EINVAL));

	if ((error = dsl_pool_hold(name, FTAG, &dp)) != 0)
	return (error);

	if ((error = dsl_dataset_hold(dp, name, FTAG, &ds)) != 0) {
	dsl_pool_rele(dp, FTAG);
	return (error);
	}

	dd = ds->ds_dir;
	mutex_enter(&dd->dd_activity_lock);
	dd->dd_activity_waiters++;

	/*
	* We get a long-hold here so that the dsl_dataset_t and dsl_dir_t
	* aren't evicted while we're waiting. Normally this is prevented by
	* holding the pool, but we can't do that while we're waiting since
	* that would prevent TXGs from syncing out. Some of the functionality
	* of long-holds (e.g. preventing deletion) is unnecessary for this
	* case, since we would cancel the waiters before proceeding with a
	* deletion. An alternative mechanism for keeping the dataset around
	* could be developed but this is simpler.
	*/
	dsl_dataset_long_hold(ds, FTAG);
	dsl_pool_rele(dp, FTAG);

	error = dsl_dir_wait(dd, ds, activity, &waited);

	dsl_dataset_long_rele(ds, FTAG);
	dd->dd_activity_waiters--;
	if (dd->dd_activity_waiters == 0)
	cv_signal(&dd->dd_activity_cv);
	mutex_exit(&dd->dd_activity_lock);

	dsl_dataset_rele(ds, FTAG);

	if (error == 0)
	fnvlist_add_boolean_value(outnvl, ZFS_WAIT_WAITED, waited);

	return (error);
	}

	/*
	* fsname is name of dataset to rollback (to most recent snapshot)
	*
	* innvl may contain name of expected target snapshot
	*
	* outnvl: "target" -> name of most recent snapshot
	* }
	*/
	static const zfs_ioc_key_t zfs_keys_rollback[] = {
	{"target", DATA_TYPE_STRING, ZK_OPTIONAL},
	};

	/* ARGSUSED */
	static int
	zfs_ioc_rollback(const char fsname, nvlist_t innvl, nvlist_t *outnvl)
	{
	zfsvfs_t *zfsvfs;
	zvol_state_handle_t *zv;
	char *target = NULL;
	int error;

	(void) nvlist_lookup_string(innvl, "target", &target);
	if (target != NULL) {
	const char *cp = strchr(target, '@');

	/*
	* The snap name must contain an @, and the part after it must
	* contain only valid characters.
	*/
	if (cp == NULL \|\|
	zfs_component_namecheck(cp + 1, NULL, NULL) != 0)
	return (SET_ERROR(EINVAL));
	}

	if (getzfsvfs(fsname, &zfsvfs) == 0) {
	dsl_dataset_t *ds;

	ds = dmu_objset_ds(zfsvfs->z_os);
	error = zfs_suspend_fs(zfsvfs);
	if (error == 0) {
	int resume_err;

	error = dsl_dataset_rollback(fsname, target, zfsvfs,
	outnvl);
	resume_err = zfs_resume_fs(zfsvfs, ds);
	error = error ? error : resume_err;
	}
	zfs_vfs_rele(zfsvfs);
	} else if ((zv = zvol_suspend(fsname)) != NULL) {
	error = dsl_dataset_rollback(fsname, target, zvol_tag(zv),
	outnvl);
	zvol_resume(zv);
	} else {
	error = dsl_dataset_rollback(fsname, target, NULL, outnvl);
	}
	return (error);
	}

	static int
	recursive_unmount(const char fsname, void arg)
	{
	const char *snapname = arg;
	char *fullname;

	fullname = kmem_asprintf("%s@%s", fsname, snapname);
	zfs_unmount_snap(fullname);
	kmem_strfree(fullname);

	return (0);
	}

	/*
	*
	* snapname is the snapshot to redact.
	* innvl: {
	* "bookname" -> (string)
	* shortname of the redaction bookmark to generate
	* "snapnv" -> (nvlist, values ignored)
	* snapshots to redact snapname with respect to
	* }
	*
	* outnvl is unused
	*/

	/* ARGSUSED */
	static const zfs_ioc_key_t zfs_keys_redact[] = {
	{"bookname", DATA_TYPE_STRING, 0},
	{"snapnv", DATA_TYPE_NVLIST, 0},
	};
	static int
	zfs_ioc_redact(const char snapname, nvlist_t innvl, nvlist_t *outnvl)
	{
	nvlist_t *redactnvl = NULL;
	char *redactbook = NULL;

	if (nvlist_lookup_nvlist(innvl, "snapnv", &redactnvl) != 0)
	return (SET_ERROR(EINVAL));
	if (fnvlist_num_pairs(redactnvl) == 0)
	return (SET_ERROR(ENXIO));
	if (nvlist_lookup_string(innvl, "bookname", &redactbook) != 0)
	return (SET_ERROR(EINVAL));

	return (dmu_redact_snap(snapname, redactnvl, redactbook));
	}

	/*
	* inputs:
	* zc_name old name of dataset
	* zc_value new name of dataset
	* zc_cookie recursive flag (only valid for snapshots)
	*
	* outputs: none
	*/
	static int
	zfs_ioc_rename(zfs_cmd_t *zc)
	{
	objset_t *os;
	dmu_objset_type_t ost;
	boolean_t recursive = zc->zc_cookie & 1;
	boolean_t nounmount = !!(zc->zc_cookie & 2);
	char *at;
	int err;

	/* "zfs rename" from and to ...%recv datasets should both fail */
	zc->zc_name[sizeof (zc->zc_name) - 1] = '\0';
	zc->zc_value[sizeof (zc->zc_value) - 1] = '\0';
	if (dataset_namecheck(zc->zc_name, NULL, NULL) != 0 \|\|
	dataset_namecheck(zc->zc_value, NULL, NULL) != 0 \|\|
	strchr(zc->zc_name, '%') \|\| strchr(zc->zc_value, '%'))
	return (SET_ERROR(EINVAL));

	err = dmu_objset_hold(zc->zc_name, FTAG, &os);
	if (err != 0)
	return (err);
	ost = dmu_objset_type(os);
	dmu_objset_rele(os, FTAG);

	at = strchr(zc->zc_name, '@');
	if (at != NULL) {
	/* snaps must be in same fs */
	int error;

	if (strncmp(zc->zc_name, zc->zc_value, at - zc->zc_name + 1))
	return (SET_ERROR(EXDEV));
	*at = '\0';
	if (ost == DMU_OST_ZFS && !nounmount) {
	error = dmu_objset_find(zc->zc_name,
	recursive_unmount, at + 1,
	recursive ? DS_FIND_CHILDREN : 0);
	if (error != 0) {
	*at = '@';
	return (error);
	}
	}
	error = dsl_dataset_rename_snapshot(zc->zc_name,
	at + 1, strchr(zc->zc_value, '@') + 1, recursive);
	*at = '@';

	return (error);
	} else {
	return (dsl_dir_rename(zc->zc_name, zc->zc_value));
	}
	}

	static int
	zfs_check_settable(const char dsname, nvpair_t pair, cred_t *cr)
	{
	const char *propname = nvpair_name(pair);
	boolean_t issnap = (strchr(dsname, '@') != NULL);
	zfs_prop_t prop = zfs_name_to_prop(propname);
	uint64_t intval, compval;
	int err;

	if (prop == ZPROP_INVAL) {
	if (zfs_prop_user(propname)) {
	if ((err = zfs_secpolicy_write_perms(dsname,
	ZFS_DELEG_PERM_USERPROP, cr)))
	return (err);
	return (0);
	}

	if (!issnap && zfs_prop_userquota(propname)) {
	const char *perm = NULL;
	const char *uq_prefix =
	zfs_userquota_prop_prefixes[ZFS_PROP_USERQUOTA];
	const char *gq_prefix =
	zfs_userquota_prop_prefixes[ZFS_PROP_GROUPQUOTA];
	const char *uiq_prefix =
	zfs_userquota_prop_prefixes[ZFS_PROP_USEROBJQUOTA];
	const char *giq_prefix =
	zfs_userquota_prop_prefixes[ZFS_PROP_GROUPOBJQUOTA];
	const char *pq_prefix =
	zfs_userquota_prop_prefixes[ZFS_PROP_PROJECTQUOTA];
	const char *piq_prefix = zfs_userquota_prop_prefixes[\
	ZFS_PROP_PROJECTOBJQUOTA];

	if (strncmp(propname, uq_prefix,
	strlen(uq_prefix)) == 0) {
	perm = ZFS_DELEG_PERM_USERQUOTA;
	} else if (strncmp(propname, uiq_prefix,
	strlen(uiq_prefix)) == 0) {
	perm = ZFS_DELEG_PERM_USEROBJQUOTA;
	} else if (strncmp(propname, gq_prefix,
	strlen(gq_prefix)) == 0) {
	perm = ZFS_DELEG_PERM_GROUPQUOTA;
	} else if (strncmp(propname, giq_prefix,
	strlen(giq_prefix)) == 0) {
	perm = ZFS_DELEG_PERM_GROUPOBJQUOTA;
	} else if (strncmp(propname, pq_prefix,
	strlen(pq_prefix)) == 0) {
	perm = ZFS_DELEG_PERM_PROJECTQUOTA;
	} else if (strncmp(propname, piq_prefix,
	strlen(piq_prefix)) == 0) {
	perm = ZFS_DELEG_PERM_PROJECTOBJQUOTA;
	} else {
	/* {USER\|GROUP\|PROJECT}USED are read-only */
	return (SET_ERROR(EINVAL));
	}

	if ((err = zfs_secpolicy_write_perms(dsname, perm, cr)))
	return (err);
	return (0);
	}

	return (SET_ERROR(EINVAL));
	}

	if (issnap)
	return (SET_ERROR(EINVAL));

	if (nvpair_type(pair) == DATA_TYPE_NVLIST) {
	/*
	* dsl_prop_get_all_impl() returns properties in this
	* format.
	*/
	nvlist_t *attrs;
	VERIFY(nvpair_value_nvlist(pair, &attrs) == 0);
	VERIFY(nvlist_lookup_nvpair(attrs, ZPROP_VALUE,
	&pair) == 0);
	}

	/*
	* Check that this value is valid for this pool version
	*/
	switch (prop) {
	case ZFS_PROP_COMPRESSION:
	/*
	* If the user specified gzip compression, make sure
	* the SPA supports it. We ignore any errors here since
	* we'll catch them later.
	*/
	if (nvpair_value_uint64(pair, &intval) == 0) {
	compval = ZIO_COMPRESS_ALGO(intval);
	if (compval >= ZIO_COMPRESS_GZIP_1 &&
	compval <= ZIO_COMPRESS_GZIP_9 &&
	zfs_earlier_version(dsname,
	SPA_VERSION_GZIP_COMPRESSION)) {
	return (SET_ERROR(ENOTSUP));
	}

	if (compval == ZIO_COMPRESS_ZLE &&
	zfs_earlier_version(dsname,
	SPA_VERSION_ZLE_COMPRESSION))
	return (SET_ERROR(ENOTSUP));

	if (compval == ZIO_COMPRESS_LZ4) {
	spa_t *spa;

	if ((err = spa_open(dsname, &spa, FTAG)) != 0)
	return (err);

	if (!spa_feature_is_enabled(spa,
	SPA_FEATURE_LZ4_COMPRESS)) {
	spa_close(spa, FTAG);
	return (SET_ERROR(ENOTSUP));
	}
	spa_close(spa, FTAG);
	}

	if (compval == ZIO_COMPRESS_ZSTD) {
	spa_t *spa;

	if ((err = spa_open(dsname, &spa, FTAG)) != 0)
	return (err);

	if (!spa_feature_is_enabled(spa,
	SPA_FEATURE_ZSTD_COMPRESS)) {
	spa_close(spa, FTAG);
	return (SET_ERROR(ENOTSUP));
	}
	spa_close(spa, FTAG);
	}
	}
	break;

	case ZFS_PROP_COPIES:
	if (zfs_earlier_version(dsname, SPA_VERSION_DITTO_BLOCKS))
	return (SET_ERROR(ENOTSUP));
	break;

	case ZFS_PROP_VOLBLOCKSIZE:
	case ZFS_PROP_RECORDSIZE:
	/* Record sizes above 128k need the feature to be enabled */
	if (nvpair_value_uint64(pair, &intval) == 0 &&
	intval > SPA_OLD_MAXBLOCKSIZE) {
	spa_t *spa;

	/*
	* We don't allow setting the property above 1MB,
	* unless the tunable has been changed.
	*/
	if (intval > zfs_max_recordsize \|\|
	intval > SPA_MAXBLOCKSIZE)
	return (SET_ERROR(ERANGE));

	if ((err = spa_open(dsname, &spa, FTAG)) != 0)
	return (err);

	if (!spa_feature_is_enabled(spa,
	SPA_FEATURE_LARGE_BLOCKS)) {
	spa_close(spa, FTAG);
	return (SET_ERROR(ENOTSUP));
	}
	spa_close(spa, FTAG);
	}
	break;

	case ZFS_PROP_DNODESIZE:
	/* Dnode sizes above 512 need the feature to be enabled */
	if (nvpair_value_uint64(pair, &intval) == 0 &&
	intval != ZFS_DNSIZE_LEGACY) {
	spa_t *spa;

	if ((err = spa_open(dsname, &spa, FTAG)) != 0)
	return (err);

	if (!spa_feature_is_enabled(spa,
	SPA_FEATURE_LARGE_DNODE)) {
	spa_close(spa, FTAG);
	return (SET_ERROR(ENOTSUP));
	}
	spa_close(spa, FTAG);
	}
	break;

	case ZFS_PROP_SPECIAL_SMALL_BLOCKS:
	/*
	* This property could require the allocation classes
	* feature to be active for setting, however we allow
	* it so that tests of settable properties succeed.
	* The CLI will issue a warning in this case.
	*/
	break;

	case ZFS_PROP_SHARESMB:
	if (zpl_earlier_version(dsname, ZPL_VERSION_FUID))
	return (SET_ERROR(ENOTSUP));
	break;

	case ZFS_PROP_ACLINHERIT:
	if (nvpair_type(pair) == DATA_TYPE_UINT64 &&
	nvpair_value_uint64(pair, &intval) == 0) {
	if (intval == ZFS_ACL_PASSTHROUGH_X &&
	zfs_earlier_version(dsname,
	SPA_VERSION_PASSTHROUGH_X))
	return (SET_ERROR(ENOTSUP));
	}
	break;
	case ZFS_PROP_CHECKSUM:
	case ZFS_PROP_DEDUP:
	{
	spa_feature_t feature;
	spa_t *spa;
	int err;

	/* dedup feature version checks */
	if (prop == ZFS_PROP_DEDUP &&
	zfs_earlier_version(dsname, SPA_VERSION_DEDUP))
	return (SET_ERROR(ENOTSUP));

	if (nvpair_type(pair) == DATA_TYPE_UINT64 &&
	nvpair_value_uint64(pair, &intval) == 0) {
	/* check prop value is enabled in features */
	feature = zio_checksum_to_feature(
	intval & ZIO_CHECKSUM_MASK);
	if (feature == SPA_FEATURE_NONE)
	break;

	if ((err = spa_open(dsname, &spa, FTAG)) != 0)
	return (err);

	if (!spa_feature_is_enabled(spa, feature)) {
	spa_close(spa, FTAG);
	return (SET_ERROR(ENOTSUP));
	}
	spa_close(spa, FTAG);
	}
	break;
	}

	default:
	break;
	}

	return (zfs_secpolicy_setprop(dsname, prop, pair, CRED()));
	}

	/*
	* Removes properties from the given props list that fail permission checks
	* needed to clear them and to restore them in case of a receive error. For each
	* property, make sure we have both set and inherit permissions.
	*
	* Returns the first error encountered if any permission checks fail. If the
	* caller provides a non-NULL errlist, it also gives the complete list of names
	* of all the properties that failed a permission check along with the
	* corresponding error numbers. The caller is responsible for freeing the
	* returned errlist.
	*
	* If every property checks out successfully, zero is returned and the list
	* pointed at by errlist is NULL.
	*/
	static int
	-zfs_check_clearable(char dataset, nvlist_t props, nvlist_t **errlist)
	+zfs_check_clearable(const char dataset, nvlist_t props, nvlist_t **errlist)
	{
	zfs_cmd_t *zc;
	nvpair_t pair, next_pair;
	nvlist_t *errors;
	int err, rv = 0;

	if (props == NULL)
	return (0);

	VERIFY(nvlist_alloc(&errors, NV_UNIQUE_NAME, KM_SLEEP) == 0);

	zc = kmem_alloc(sizeof (zfs_cmd_t), KM_SLEEP);
	(void) strlcpy(zc->zc_name, dataset, sizeof (zc->zc_name));
	pair = nvlist_next_nvpair(props, NULL);
	while (pair != NULL) {
	next_pair = nvlist_next_nvpair(props, pair);

	(void) strlcpy(zc->zc_value, nvpair_name(pair),
	sizeof (zc->zc_value));
	if ((err = zfs_check_settable(dataset, pair, CRED())) != 0 \|\|
	(err = zfs_secpolicy_inherit_prop(zc, NULL, CRED())) != 0) {
	VERIFY(nvlist_remove_nvpair(props, pair) == 0);
	VERIFY(nvlist_add_int32(errors,
	zc->zc_value, err) == 0);
	}
	pair = next_pair;
	}
	kmem_free(zc, sizeof (zfs_cmd_t));

	if ((pair = nvlist_next_nvpair(errors, NULL)) == NULL) {
	nvlist_free(errors);
	errors = NULL;
	} else {
	VERIFY(nvpair_value_int32(pair, &rv) == 0);
	}

	if (errlist == NULL)
	nvlist_free(errors);
	else
	*errlist = errors;

	return (rv);
	}

	static boolean_t
	propval_equals(nvpair_t p1, nvpair_t p2)
	{
	if (nvpair_type(p1) == DATA_TYPE_NVLIST) {
	/* dsl_prop_get_all_impl() format */
	nvlist_t *attrs;
	VERIFY(nvpair_value_nvlist(p1, &attrs) == 0);
	VERIFY(nvlist_lookup_nvpair(attrs, ZPROP_VALUE,
	&p1) == 0);
	}

	if (nvpair_type(p2) == DATA_TYPE_NVLIST) {
	nvlist_t *attrs;
	VERIFY(nvpair_value_nvlist(p2, &attrs) == 0);
	VERIFY(nvlist_lookup_nvpair(attrs, ZPROP_VALUE,
	&p2) == 0);
	}

	if (nvpair_type(p1) != nvpair_type(p2))
	return (B_FALSE);

	if (nvpair_type(p1) == DATA_TYPE_STRING) {
	char valstr1, valstr2;

	VERIFY(nvpair_value_string(p1, (char **)&valstr1) == 0);
	VERIFY(nvpair_value_string(p2, (char **)&valstr2) == 0);
	return (strcmp(valstr1, valstr2) == 0);
	} else {
	uint64_t intval1, intval2;

	VERIFY(nvpair_value_uint64(p1, &intval1) == 0);
	VERIFY(nvpair_value_uint64(p2, &intval2) == 0);
	return (intval1 == intval2);
	}
	}

	/*
	* Remove properties from props if they are not going to change (as determined
	* by comparison with origprops). Remove them from origprops as well, since we
	* do not need to clear or restore properties that won't change.
	*/
	static void
	props_reduce(nvlist_t props, nvlist_t origprops)
	{
	nvpair_t pair, next_pair;

	if (origprops == NULL)
	return; /* all props need to be received */

	pair = nvlist_next_nvpair(props, NULL);
	while (pair != NULL) {
	const char *propname = nvpair_name(pair);
	nvpair_t *match;

	next_pair = nvlist_next_nvpair(props, pair);

	if ((nvlist_lookup_nvpair(origprops, propname,
	&match) != 0) \|\| !propval_equals(pair, match))
	goto next; /* need to set received value */

	/* don't clear the existing received value */
	(void) nvlist_remove_nvpair(origprops, match);
	/* don't bother receiving the property */
	(void) nvlist_remove_nvpair(props, pair);
	next:
	pair = next_pair;
	}
	}

	/*
	* Extract properties that cannot be set PRIOR to the receipt of a dataset.
	* For example, refquota cannot be set until after the receipt of a dataset,
	* because in replication streams, an older/earlier snapshot may exceed the
	* refquota. We want to receive the older/earlier snapshot, but setting
	* refquota pre-receipt will set the dsl's ACTUAL quota, which will prevent
	* the older/earlier snapshot from being received (with EDQUOT).
	*
	* The ZFS test "zfs_receive_011_pos" demonstrates such a scenario.
	*
	* libzfs will need to be judicious handling errors encountered by props
	* extracted by this function.
	*/
	static nvlist_t *
	extract_delay_props(nvlist_t *props)
	{
	nvlist_t *delayprops;
	nvpair_t nvp, tmp;
	static const zfs_prop_t delayable[] = {
	ZFS_PROP_REFQUOTA,
	ZFS_PROP_KEYLOCATION,
	0
	};
	int i;

	VERIFY(nvlist_alloc(&delayprops, NV_UNIQUE_NAME, KM_SLEEP) == 0);

	for (nvp = nvlist_next_nvpair(props, NULL); nvp != NULL;
	nvp = nvlist_next_nvpair(props, nvp)) {
	/*
	* strcmp() is safe because zfs_prop_to_name() always returns
	* a bounded string.
	*/
	for (i = 0; delayable[i] != 0; i++) {
	if (strcmp(zfs_prop_to_name(delayable[i]),
	nvpair_name(nvp)) == 0) {
	break;
	}
	}
	if (delayable[i] != 0) {
	tmp = nvlist_prev_nvpair(props, nvp);
	VERIFY(nvlist_add_nvpair(delayprops, nvp) == 0);
	VERIFY(nvlist_remove_nvpair(props, nvp) == 0);
	nvp = tmp;
	}
	}

	if (nvlist_empty(delayprops)) {
	nvlist_free(delayprops);
	delayprops = NULL;
	}
	return (delayprops);
	}

	static void
	zfs_allow_log_destroy(void *arg)
	{
	char *poolname = arg;

	if (poolname != NULL)
	kmem_strfree(poolname);
	}

	#ifdef ZFS_DEBUG
	static boolean_t zfs_ioc_recv_inject_err;
	#endif

	/*
	* nvlist 'errors' is always allocated. It will contain descriptions of
	* encountered errors, if any. It's the callers responsibility to free.
	*/
	static int
	zfs_ioc_recv_impl(char tofs, char tosnap, char origin, nvlist_t recvprops,
	nvlist_t localprops, nvlist_t hidden_args, boolean_t force,
	boolean_t resumable, int input_fd,
	dmu_replay_record_t begin_record, uint64_t read_bytes,
	uint64_t errflags, nvlist_t *errors)
	{
	dmu_recv_cookie_t drc;
	int error = 0;
	int props_error = 0;
	offset_t off, noff;
	nvlist_t *local_delayprops = NULL;
	nvlist_t *recv_delayprops = NULL;
	nvlist_t origprops = NULL; / existing properties */
	nvlist_t origrecvd = NULL; / existing received properties */
	boolean_t first_recvd_props = B_FALSE;
	boolean_t tofs_was_redacted;
	zfs_file_t *input_fp;

	*read_bytes = 0;
	*errflags = 0;
	*errors = fnvlist_alloc();
	off = 0;

	if ((error = zfs_file_get(input_fd, &input_fp)))
	return (error);

	noff = off = zfs_file_off(input_fp);
	error = dmu_recv_begin(tofs, tosnap, begin_record, force,
	resumable, localprops, hidden_args, origin, &drc, input_fp,
	&off);
	if (error != 0)
	goto out;
	tofs_was_redacted = dsl_get_redacted(drc.drc_ds);

	/*
	* Set properties before we receive the stream so that they are applied
	* to the new data. Note that we must call dmu_recv_stream() if
	* dmu_recv_begin() succeeds.
	*/
	if (recvprops != NULL && !drc.drc_newfs) {
	if (spa_version(dsl_dataset_get_spa(drc.drc_ds)) >=
	SPA_VERSION_RECVD_PROPS &&
	!dsl_prop_get_hasrecvd(tofs))
	first_recvd_props = B_TRUE;

	/*
	* If new received properties are supplied, they are to
	* completely replace the existing received properties,
	* so stash away the existing ones.
	*/
	if (dsl_prop_get_received(tofs, &origrecvd) == 0) {
	nvlist_t *errlist = NULL;
	/*
	* Don't bother writing a property if its value won't
	* change (and avoid the unnecessary security checks).
	*
	* The first receive after SPA_VERSION_RECVD_PROPS is a
	* special case where we blow away all local properties
	* regardless.
	*/
	if (!first_recvd_props)
	props_reduce(recvprops, origrecvd);
	if (zfs_check_clearable(tofs, origrecvd, &errlist) != 0)
	(void) nvlist_merge(*errors, errlist, 0);
	nvlist_free(errlist);

	if (clear_received_props(tofs, origrecvd,
	first_recvd_props ? NULL : recvprops) != 0)
	*errflags \|= ZPROP_ERR_NOCLEAR;
	} else {
	*errflags \|= ZPROP_ERR_NOCLEAR;
	}
	}

	/*
	* Stash away existing properties so we can restore them on error unless
	* we're doing the first receive after SPA_VERSION_RECVD_PROPS, in which
	* case "origrecvd" will take care of that.
	*/
	if (localprops != NULL && !drc.drc_newfs && !first_recvd_props) {
	objset_t *os;
	if (dmu_objset_hold(tofs, FTAG, &os) == 0) {
	if (dsl_prop_get_all(os, &origprops) != 0) {
	*errflags \|= ZPROP_ERR_NOCLEAR;
	}
	dmu_objset_rele(os, FTAG);
	} else {
	*errflags \|= ZPROP_ERR_NOCLEAR;
	}
	}

	if (recvprops != NULL) {
	props_error = dsl_prop_set_hasrecvd(tofs);

	if (props_error == 0) {
	recv_delayprops = extract_delay_props(recvprops);
	(void) zfs_set_prop_nvlist(tofs, ZPROP_SRC_RECEIVED,
	recvprops, *errors);
	}
	}

	if (localprops != NULL) {
	nvlist_t *oprops = fnvlist_alloc();
	nvlist_t *xprops = fnvlist_alloc();
	nvpair_t *nvp = NULL;

	while ((nvp = nvlist_next_nvpair(localprops, nvp)) != NULL) {
	if (nvpair_type(nvp) == DATA_TYPE_BOOLEAN) {
	/* -x property */
	const char *name = nvpair_name(nvp);
	zfs_prop_t prop = zfs_name_to_prop(name);
	if (prop != ZPROP_INVAL) {
	if (!zfs_prop_inheritable(prop))
	continue;
	} else if (!zfs_prop_user(name))
	continue;
	fnvlist_add_boolean(xprops, name);
	} else {
	/* -o property=value */
	fnvlist_add_nvpair(oprops, nvp);
	}
	}

	local_delayprops = extract_delay_props(oprops);
	(void) zfs_set_prop_nvlist(tofs, ZPROP_SRC_LOCAL,
	oprops, *errors);
	(void) zfs_set_prop_nvlist(tofs, ZPROP_SRC_INHERITED,
	xprops, *errors);

	nvlist_free(oprops);
	nvlist_free(xprops);
	}

	error = dmu_recv_stream(&drc, &off);

	if (error == 0) {
	zfsvfs_t *zfsvfs = NULL;
	zvol_state_handle_t *zv = NULL;

	if (getzfsvfs(tofs, &zfsvfs) == 0) {
	/* online recv */
	dsl_dataset_t *ds;
	int end_err;
	boolean_t stream_is_redacted = DMU_GET_FEATUREFLAGS(
	begin_record->drr_u.drr_begin.
	drr_versioninfo) & DMU_BACKUP_FEATURE_REDACTED;

	ds = dmu_objset_ds(zfsvfs->z_os);
	error = zfs_suspend_fs(zfsvfs);
	/*
	* If the suspend fails, then the recv_end will
	* likely also fail, and clean up after itself.
	*/
	end_err = dmu_recv_end(&drc, zfsvfs);
	/*
	* If the dataset was not redacted, but we received a
	* redacted stream onto it, we need to unmount the
	* dataset. Otherwise, resume the filesystem.
	*/
	if (error == 0 && !drc.drc_newfs &&
	stream_is_redacted && !tofs_was_redacted) {
	error = zfs_end_fs(zfsvfs, ds);
	} else if (error == 0) {
	error = zfs_resume_fs(zfsvfs, ds);
	}
	error = error ? error : end_err;
	zfs_vfs_rele(zfsvfs);
	} else if ((zv = zvol_suspend(tofs)) != NULL) {
	error = dmu_recv_end(&drc, zvol_tag(zv));
	zvol_resume(zv);
	} else {
	error = dmu_recv_end(&drc, NULL);
	}

	/* Set delayed properties now, after we're done receiving. */
	if (recv_delayprops != NULL && error == 0) {
	(void) zfs_set_prop_nvlist(tofs, ZPROP_SRC_RECEIVED,
	recv_delayprops, *errors);
	}
	if (local_delayprops != NULL && error == 0) {
	(void) zfs_set_prop_nvlist(tofs, ZPROP_SRC_LOCAL,
	local_delayprops, *errors);
	}
	}

	/*
	* Merge delayed props back in with initial props, in case
	* we're DEBUG and zfs_ioc_recv_inject_err is set (which means
	* we have to make sure clear_received_props() includes
	* the delayed properties).
	*
	* Since zfs_ioc_recv_inject_err is only in DEBUG kernels,
	* using ASSERT() will be just like a VERIFY.
	*/
	if (recv_delayprops != NULL) {
	ASSERT(nvlist_merge(recvprops, recv_delayprops, 0) == 0);
	nvlist_free(recv_delayprops);
	}
	if (local_delayprops != NULL) {
	ASSERT(nvlist_merge(localprops, local_delayprops, 0) == 0);
	nvlist_free(local_delayprops);
	}
	*read_bytes = off - noff;

	#ifdef ZFS_DEBUG
	if (zfs_ioc_recv_inject_err) {
	zfs_ioc_recv_inject_err = B_FALSE;
	error = 1;
	}
	#endif

	/*
	* On error, restore the original props.
	*/
	if (error != 0 && recvprops != NULL && !drc.drc_newfs) {
	if (clear_received_props(tofs, recvprops, NULL) != 0) {
	/*
	* We failed to clear the received properties.
	* Since we may have left a $recvd value on the
	* system, we can't clear the $hasrecvd flag.
	*/
	*errflags \|= ZPROP_ERR_NORESTORE;
	} else if (first_recvd_props) {
	dsl_prop_unset_hasrecvd(tofs);
	}

	if (origrecvd == NULL && !drc.drc_newfs) {
	/* We failed to stash the original properties. */
	*errflags \|= ZPROP_ERR_NORESTORE;
	}

	/*
	* dsl_props_set() will not convert RECEIVED to LOCAL on or
	* after SPA_VERSION_RECVD_PROPS, so we need to specify LOCAL
	* explicitly if we're restoring local properties cleared in the
	* first new-style receive.
	*/
	if (origrecvd != NULL &&
	zfs_set_prop_nvlist(tofs, (first_recvd_props ?
	ZPROP_SRC_LOCAL : ZPROP_SRC_RECEIVED),
	origrecvd, NULL) != 0) {
	/*
	* We stashed the original properties but failed to
	* restore them.
	*/
	*errflags \|= ZPROP_ERR_NORESTORE;
	}
	}
	if (error != 0 && localprops != NULL && !drc.drc_newfs &&
	!first_recvd_props) {
	nvlist_t *setprops;
	nvlist_t *inheritprops;
	nvpair_t *nvp;

	if (origprops == NULL) {
	/* We failed to stash the original properties. */
	*errflags \|= ZPROP_ERR_NORESTORE;
	goto out;
	}

	/* Restore original props */
	setprops = fnvlist_alloc();
	inheritprops = fnvlist_alloc();
	nvp = NULL;
	while ((nvp = nvlist_next_nvpair(localprops, nvp)) != NULL) {
	const char *name = nvpair_name(nvp);
	const char *source;
	nvlist_t *attrs;

	if (!nvlist_exists(origprops, name)) {
	/*
	* Property was not present or was explicitly
	* inherited before the receive, restore this.
	*/
	fnvlist_add_boolean(inheritprops, name);
	continue;
	}
	attrs = fnvlist_lookup_nvlist(origprops, name);
	source = fnvlist_lookup_string(attrs, ZPROP_SOURCE);

	/* Skip received properties */
	if (strcmp(source, ZPROP_SOURCE_VAL_RECVD) == 0)
	continue;

	if (strcmp(source, tofs) == 0) {
	/* Property was locally set */
	fnvlist_add_nvlist(setprops, name, attrs);
	} else {
	/* Property was implicitly inherited */
	fnvlist_add_boolean(inheritprops, name);
	}
	}

	if (zfs_set_prop_nvlist(tofs, ZPROP_SRC_LOCAL, setprops,
	NULL) != 0)
	*errflags \|= ZPROP_ERR_NORESTORE;
	if (zfs_set_prop_nvlist(tofs, ZPROP_SRC_INHERITED, inheritprops,
	NULL) != 0)
	*errflags \|= ZPROP_ERR_NORESTORE;

	nvlist_free(setprops);
	nvlist_free(inheritprops);
	}
	out:
	zfs_file_put(input_fd);
	nvlist_free(origrecvd);
	nvlist_free(origprops);

	if (error == 0)
	error = props_error;

	return (error);
	}

	/*
	* inputs:
	* zc_name name of containing filesystem (unused)
	* zc_nvlist_src{_size} nvlist of properties to apply
	* zc_nvlist_conf{_size} nvlist of properties to exclude
	* (DATA_TYPE_BOOLEAN) and override (everything else)
	* zc_value name of snapshot to create
	* zc_string name of clone origin (if DRR_FLAG_CLONE)
	* zc_cookie file descriptor to recv from
	* zc_begin_record the BEGIN record of the stream (not byteswapped)
	* zc_guid force flag
	*
	* outputs:
	* zc_cookie number of bytes read
	* zc_obj zprop_errflags_t
	* zc_nvlist_dst{_size} error for each unapplied received property
	*/
	static int
	zfs_ioc_recv(zfs_cmd_t *zc)
	{
	dmu_replay_record_t begin_record;
	nvlist_t *errors = NULL;
	nvlist_t *recvdprops = NULL;
	nvlist_t *localprops = NULL;
	char *origin = NULL;
	char *tosnap;
	char tofs[ZFS_MAX_DATASET_NAME_LEN];
	int error = 0;

	if (dataset_namecheck(zc->zc_value, NULL, NULL) != 0 \|\|
	strchr(zc->zc_value, '@') == NULL \|\|
	strchr(zc->zc_value, '%'))
	return (SET_ERROR(EINVAL));

	(void) strlcpy(tofs, zc->zc_value, sizeof (tofs));
	tosnap = strchr(tofs, '@');
	*tosnap++ = '\0';

	if (zc->zc_nvlist_src != 0 &&
	(error = get_nvlist(zc->zc_nvlist_src, zc->zc_nvlist_src_size,
	zc->zc_iflags, &recvdprops)) != 0)
	return (error);

	if (zc->zc_nvlist_conf != 0 &&
	(error = get_nvlist(zc->zc_nvlist_conf, zc->zc_nvlist_conf_size,
	zc->zc_iflags, &localprops)) != 0)
	return (error);

	if (zc->zc_string[0])
	origin = zc->zc_string;

	begin_record.drr_type = DRR_BEGIN;
	begin_record.drr_payloadlen = 0;
	begin_record.drr_u.drr_begin = zc->zc_begin_record;

	error = zfs_ioc_recv_impl(tofs, tosnap, origin, recvdprops, localprops,
	NULL, zc->zc_guid, B_FALSE, zc->zc_cookie, &begin_record,
	&zc->zc_cookie, &zc->zc_obj, &errors);
	nvlist_free(recvdprops);
	nvlist_free(localprops);

	/*
	* Now that all props, initial and delayed, are set, report the prop
	* errors to the caller.
	*/
	if (zc->zc_nvlist_dst_size != 0 && errors != NULL &&
	(nvlist_smush(errors, zc->zc_nvlist_dst_size) != 0 \|\|
	put_nvlist(zc, errors) != 0)) {
	/*
	* Caller made zc->zc_nvlist_dst less than the minimum expected
	* size or supplied an invalid address.
	*/
	error = SET_ERROR(EINVAL);
	}

	nvlist_free(errors);

	return (error);
	}

	/*
	* innvl: {
	* "snapname" -> full name of the snapshot to create
	* (optional) "props" -> received properties to set (nvlist)
	* (optional) "localprops" -> override and exclude properties (nvlist)
	* (optional) "origin" -> name of clone origin (DRR_FLAG_CLONE)
	* "begin_record" -> non-byteswapped dmu_replay_record_t
	* "input_fd" -> file descriptor to read stream from (int32)
	* (optional) "force" -> force flag (value ignored)
	* (optional) "resumable" -> resumable flag (value ignored)
	* (optional) "cleanup_fd" -> unused
	* (optional) "action_handle" -> unused
	* (optional) "hidden_args" -> { "wkeydata" -> value }
	* }
	*
	* outnvl: {
	* "read_bytes" -> number of bytes read
	* "error_flags" -> zprop_errflags_t
	* "errors" -> error for each unapplied received property (nvlist)
	* }
	*/
	static const zfs_ioc_key_t zfs_keys_recv_new[] = {
	{"snapname", DATA_TYPE_STRING, 0},
	{"props", DATA_TYPE_NVLIST, ZK_OPTIONAL},
	{"localprops", DATA_TYPE_NVLIST, ZK_OPTIONAL},
	{"origin", DATA_TYPE_STRING, ZK_OPTIONAL},
	{"begin_record", DATA_TYPE_BYTE_ARRAY, 0},
	{"input_fd", DATA_TYPE_INT32, 0},
	{"force", DATA_TYPE_BOOLEAN, ZK_OPTIONAL},
	{"resumable", DATA_TYPE_BOOLEAN, ZK_OPTIONAL},
	{"cleanup_fd", DATA_TYPE_INT32, ZK_OPTIONAL},
	{"action_handle", DATA_TYPE_UINT64, ZK_OPTIONAL},
	{"hidden_args", DATA_TYPE_NVLIST, ZK_OPTIONAL},
	};

	static int
	zfs_ioc_recv_new(const char fsname, nvlist_t innvl, nvlist_t *outnvl)
	{
	dmu_replay_record_t *begin_record;
	uint_t begin_record_size;
	nvlist_t *errors = NULL;
	nvlist_t *recvprops = NULL;
	nvlist_t *localprops = NULL;
	nvlist_t *hidden_args = NULL;
	char *snapname;
	char *origin = NULL;
	char *tosnap;
	char tofs[ZFS_MAX_DATASET_NAME_LEN];
	boolean_t force;
	boolean_t resumable;
	uint64_t read_bytes = 0;
	uint64_t errflags = 0;
	int input_fd = -1;
	int error;

	snapname = fnvlist_lookup_string(innvl, "snapname");

	if (dataset_namecheck(snapname, NULL, NULL) != 0 \|\|
	strchr(snapname, '@') == NULL \|\|
	strchr(snapname, '%'))
	return (SET_ERROR(EINVAL));

	(void) strlcpy(tofs, snapname, sizeof (tofs));
	tosnap = strchr(tofs, '@');
	*tosnap++ = '\0';

	error = nvlist_lookup_string(innvl, "origin", &origin);
	if (error && error != ENOENT)
	return (error);

	error = nvlist_lookup_byte_array(innvl, "begin_record",
	(uchar_t **)&begin_record, &begin_record_size);
	if (error != 0 \|\| begin_record_size != sizeof (*begin_record))
	return (SET_ERROR(EINVAL));

	input_fd = fnvlist_lookup_int32(innvl, "input_fd");

	force = nvlist_exists(innvl, "force");
	resumable = nvlist_exists(innvl, "resumable");

	/* we still use "props" here for backwards compatibility */
	error = nvlist_lookup_nvlist(innvl, "props", &recvprops);
	if (error && error != ENOENT)
	return (error);

	error = nvlist_lookup_nvlist(innvl, "localprops", &localprops);
	if (error && error != ENOENT)
	return (error);

	error = nvlist_lookup_nvlist(innvl, ZPOOL_HIDDEN_ARGS, &hidden_args);
	if (error && error != ENOENT)
	return (error);

	error = zfs_ioc_recv_impl(tofs, tosnap, origin, recvprops, localprops,
	hidden_args, force, resumable, input_fd, begin_record,
	&read_bytes, &errflags, &errors);

	fnvlist_add_uint64(outnvl, "read_bytes", read_bytes);
	fnvlist_add_uint64(outnvl, "error_flags", errflags);
	fnvlist_add_nvlist(outnvl, "errors", errors);

	nvlist_free(errors);
	nvlist_free(recvprops);
	nvlist_free(localprops);

	return (error);
	}

	typedef struct dump_bytes_io {
	zfs_file_t *dbi_fp;
	caddr_t dbi_buf;
	int dbi_len;
	int dbi_err;
	} dump_bytes_io_t;

	static void
	dump_bytes_cb(void *arg)
	{
	dump_bytes_io_t dbi = (dump_bytes_io_t )arg;
	zfs_file_t *fp;
	caddr_t buf;

	fp = dbi->dbi_fp;
	buf = dbi->dbi_buf;

	dbi->dbi_err = zfs_file_write(fp, buf, dbi->dbi_len, NULL);
	}

	static int
	dump_bytes(objset_t os, void buf, int len, void *arg)
	{
	dump_bytes_io_t dbi;

	dbi.dbi_fp = arg;
	dbi.dbi_buf = buf;
	dbi.dbi_len = len;

	#if defined(HAVE_LARGE_STACKS)
	dump_bytes_cb(&dbi);
	#else
	/*
	* The vn_rdwr() call is performed in a taskq to ensure that there is
	* always enough stack space to write safely to the target filesystem.
	* The ZIO_TYPE_FREE threads are used because there can be a lot of
	* them and they are used in vdev_file.c for a similar purpose.
	*/
	spa_taskq_dispatch_sync(dmu_objset_spa(os), ZIO_TYPE_FREE,
	ZIO_TASKQ_ISSUE, dump_bytes_cb, &dbi, TQ_SLEEP);
	#endif /* HAVE_LARGE_STACKS */

	return (dbi.dbi_err);
	}

	/*
	* inputs:
	* zc_name name of snapshot to send
	* zc_cookie file descriptor to send stream to
	* zc_obj fromorigin flag (mutually exclusive with zc_fromobj)
	* zc_sendobj objsetid of snapshot to send
	* zc_fromobj objsetid of incremental fromsnap (may be zero)
	* zc_guid if set, estimate size of stream only. zc_cookie is ignored.
	* output size in zc_objset_type.
	* zc_flags lzc_send_flags
	*
	* outputs:
	* zc_objset_type estimated size, if zc_guid is set
	*
	* NOTE: This is no longer the preferred interface, any new functionality
	* should be added to zfs_ioc_send_new() instead.
	*/
	static int
	zfs_ioc_send(zfs_cmd_t *zc)
	{
	int error;
	offset_t off;
	boolean_t estimate = (zc->zc_guid != 0);
	boolean_t embedok = (zc->zc_flags & 0x1);
	boolean_t large_block_ok = (zc->zc_flags & 0x2);
	boolean_t compressok = (zc->zc_flags & 0x4);
	boolean_t rawok = (zc->zc_flags & 0x8);
	boolean_t savedok = (zc->zc_flags & 0x10);

	if (zc->zc_obj != 0) {
	dsl_pool_t *dp;
	dsl_dataset_t *tosnap;

	error = dsl_pool_hold(zc->zc_name, FTAG, &dp);
	if (error != 0)
	return (error);

	error = dsl_dataset_hold_obj(dp, zc->zc_sendobj, FTAG, &tosnap);
	if (error != 0) {
	dsl_pool_rele(dp, FTAG);
	return (error);
	}

	if (dsl_dir_is_clone(tosnap->ds_dir))
	zc->zc_fromobj =
	dsl_dir_phys(tosnap->ds_dir)->dd_origin_obj;
	dsl_dataset_rele(tosnap, FTAG);
	dsl_pool_rele(dp, FTAG);
	}

	if (estimate) {
	dsl_pool_t *dp;
	dsl_dataset_t *tosnap;
	dsl_dataset_t *fromsnap = NULL;

	error = dsl_pool_hold(zc->zc_name, FTAG, &dp);
	if (error != 0)
	return (error);

	error = dsl_dataset_hold_obj(dp, zc->zc_sendobj,
	FTAG, &tosnap);
	if (error != 0) {
	dsl_pool_rele(dp, FTAG);
	return (error);
	}

	if (zc->zc_fromobj != 0) {
	error = dsl_dataset_hold_obj(dp, zc->zc_fromobj,
	FTAG, &fromsnap);
	if (error != 0) {
	dsl_dataset_rele(tosnap, FTAG);
	dsl_pool_rele(dp, FTAG);
	return (error);
	}
	}

	error = dmu_send_estimate_fast(tosnap, fromsnap, NULL,
	compressok \|\| rawok, savedok, &zc->zc_objset_type);

	if (fromsnap != NULL)
	dsl_dataset_rele(fromsnap, FTAG);
	dsl_dataset_rele(tosnap, FTAG);
	dsl_pool_rele(dp, FTAG);
	} else {
	zfs_file_t *fp;
	dmu_send_outparams_t out = {0};

	if ((error = zfs_file_get(zc->zc_cookie, &fp)))
	return (error);

	off = zfs_file_off(fp);
	out.dso_outfunc = dump_bytes;
	out.dso_arg = fp;
	out.dso_dryrun = B_FALSE;
	error = dmu_send_obj(zc->zc_name, zc->zc_sendobj,
	zc->zc_fromobj, embedok, large_block_ok, compressok,
	rawok, savedok, zc->zc_cookie, &off, &out);

	zfs_file_put(zc->zc_cookie);
	}
	return (error);
	}

	/*
	* inputs:
	* zc_name name of snapshot on which to report progress
	* zc_cookie file descriptor of send stream
	*
	* outputs:
	* zc_cookie number of bytes written in send stream thus far
	* zc_objset_type logical size of data traversed by send thus far
	*/
	static int
	zfs_ioc_send_progress(zfs_cmd_t *zc)
	{
	dsl_pool_t *dp;
	dsl_dataset_t *ds;
	dmu_sendstatus_t *dsp = NULL;
	int error;

	error = dsl_pool_hold(zc->zc_name, FTAG, &dp);
	if (error != 0)
	return (error);

	error = dsl_dataset_hold(dp, zc->zc_name, FTAG, &ds);
	if (error != 0) {
	dsl_pool_rele(dp, FTAG);
	return (error);
	}

	mutex_enter(&ds->ds_sendstream_lock);

	/*
	* Iterate over all the send streams currently active on this dataset.
	* If there's one which matches the specified file descriptor _and_ the
	* stream was started by the current process, return the progress of
	* that stream.
	*/

	for (dsp = list_head(&ds->ds_sendstreams); dsp != NULL;
	dsp = list_next(&ds->ds_sendstreams, dsp)) {
	if (dsp->dss_outfd == zc->zc_cookie &&
	zfs_proc_is_caller(dsp->dss_proc))
	break;
	}

	if (dsp != NULL) {
	zc->zc_cookie = atomic_cas_64((volatile uint64_t *)dsp->dss_off,
	0, 0);
	/* This is the closest thing we have to atomic_read_64. */
	zc->zc_objset_type = atomic_cas_64(&dsp->dss_blocks, 0, 0);
	} else {
	error = SET_ERROR(ENOENT);
	}

	mutex_exit(&ds->ds_sendstream_lock);
	dsl_dataset_rele(ds, FTAG);
	dsl_pool_rele(dp, FTAG);
	return (error);
	}

	static int
	zfs_ioc_inject_fault(zfs_cmd_t *zc)
	{
	int id, error;

	error = zio_inject_fault(zc->zc_name, (int)zc->zc_guid, &id,
	&zc->zc_inject_record);

	if (error == 0)
	zc->zc_guid = (uint64_t)id;

	return (error);
	}

	static int
	zfs_ioc_clear_fault(zfs_cmd_t *zc)
	{
	return (zio_clear_fault((int)zc->zc_guid));
	}

	static int
	zfs_ioc_inject_list_next(zfs_cmd_t *zc)
	{
	int id = (int)zc->zc_guid;
	int error;

	error = zio_inject_list_next(&id, zc->zc_name, sizeof (zc->zc_name),
	&zc->zc_inject_record);

	zc->zc_guid = id;

	return (error);
	}

	static int
	zfs_ioc_error_log(zfs_cmd_t *zc)
	{
	spa_t *spa;
	int error;
	size_t count = (size_t)zc->zc_nvlist_dst_size;

	if ((error = spa_open(zc->zc_name, &spa, FTAG)) != 0)
	return (error);

	error = spa_get_errlog(spa, (void *)(uintptr_t)zc->zc_nvlist_dst,
	&count);
	if (error == 0)
	zc->zc_nvlist_dst_size = count;
	else
	zc->zc_nvlist_dst_size = spa_get_errlog_size(spa);

	spa_close(spa, FTAG);

	return (error);
	}

	static int
	zfs_ioc_clear(zfs_cmd_t *zc)
	{
	spa_t *spa;
	vdev_t *vd;
	int error;

	/*
	* On zpool clear we also fix up missing slogs
	*/
	mutex_enter(&spa_namespace_lock);
	spa = spa_lookup(zc->zc_name);
	if (spa == NULL) {
	mutex_exit(&spa_namespace_lock);
	return (SET_ERROR(EIO));
	}
	if (spa_get_log_state(spa) == SPA_LOG_MISSING) {
	/* we need to let spa_open/spa_load clear the chains */
	spa_set_log_state(spa, SPA_LOG_CLEAR);
	}
	spa->spa_last_open_failed = 0;
	mutex_exit(&spa_namespace_lock);

	if (zc->zc_cookie & ZPOOL_NO_REWIND) {
	error = spa_open(zc->zc_name, &spa, FTAG);
	} else {
	nvlist_t *policy;
	nvlist_t *config = NULL;

	if (zc->zc_nvlist_src == 0)
	return (SET_ERROR(EINVAL));

	if ((error = get_nvlist(zc->zc_nvlist_src,
	zc->zc_nvlist_src_size, zc->zc_iflags, &policy)) == 0) {
	error = spa_open_rewind(zc->zc_name, &spa, FTAG,
	policy, &config);
	if (config != NULL) {
	int err;

	if ((err = put_nvlist(zc, config)) != 0)
	error = err;
	nvlist_free(config);
	}
	nvlist_free(policy);
	}
	}

	if (error != 0)
	return (error);

	/*
	* If multihost is enabled, resuming I/O is unsafe as another
	* host may have imported the pool.
	*/
	if (spa_multihost(spa) && spa_suspended(spa))
	return (SET_ERROR(EINVAL));

	spa_vdev_state_enter(spa, SCL_NONE);

	if (zc->zc_guid == 0) {
	vd = NULL;
	} else {
	vd = spa_lookup_by_guid(spa, zc->zc_guid, B_TRUE);
	if (vd == NULL) {
	error = SET_ERROR(ENODEV);
	(void) spa_vdev_state_exit(spa, NULL, error);
	spa_close(spa, FTAG);
	return (error);
	}
	}

	vdev_clear(spa, vd);

	(void) spa_vdev_state_exit(spa, spa_suspended(spa) ?
	NULL : spa->spa_root_vdev, 0);

	/*
	* Resume any suspended I/Os.
	*/
	if (zio_resume(spa) != 0)
	error = SET_ERROR(EIO);

	spa_close(spa, FTAG);

	return (error);
	}

	/*
	* Reopen all the vdevs associated with the pool.
	*
	* innvl: {
	* "scrub_restart" -> when true and scrub is running, allow to restart
	* scrub as the side effect of the reopen (boolean).
	* }
	*
	* outnvl is unused
	*/
	static const zfs_ioc_key_t zfs_keys_pool_reopen[] = {
	{"scrub_restart", DATA_TYPE_BOOLEAN_VALUE, ZK_OPTIONAL},
	};

	/* ARGSUSED */
	static int
	zfs_ioc_pool_reopen(const char pool, nvlist_t innvl, nvlist_t *outnvl)
	{
	spa_t *spa;
	int error;
	boolean_t rc, scrub_restart = B_TRUE;

	if (innvl) {
	error = nvlist_lookup_boolean_value(innvl,
	"scrub_restart", &rc);
	if (error == 0)
	scrub_restart = rc;
	}

	error = spa_open(pool, &spa, FTAG);
	if (error != 0)
	return (error);

	spa_vdev_state_enter(spa, SCL_NONE);

	/*
	* If the scrub_restart flag is B_FALSE and a scrub is already
	* in progress then set spa_scrub_reopen flag to B_TRUE so that
	* we don't restart the scrub as a side effect of the reopen.
	* Otherwise, let vdev_open() decided if a resilver is required.
	*/

	spa->spa_scrub_reopen = (!scrub_restart &&
	dsl_scan_scrubbing(spa->spa_dsl_pool));
	vdev_reopen(spa->spa_root_vdev);
	spa->spa_scrub_reopen = B_FALSE;

	(void) spa_vdev_state_exit(spa, NULL, 0);
	spa_close(spa, FTAG);
	return (0);
	}

	/*
	* inputs:
	* zc_name name of filesystem
	*
	* outputs:
	* zc_string name of conflicting snapshot, if there is one
	*/
	static int
	zfs_ioc_promote(zfs_cmd_t *zc)
	{
	dsl_pool_t *dp;
	dsl_dataset_t ds, ods;
	char origin[ZFS_MAX_DATASET_NAME_LEN];
	char *cp;
	int error;

	zc->zc_name[sizeof (zc->zc_name) - 1] = '\0';
	if (dataset_namecheck(zc->zc_name, NULL, NULL) != 0 \|\|
	strchr(zc->zc_name, '%'))
	return (SET_ERROR(EINVAL));

	error = dsl_pool_hold(zc->zc_name, FTAG, &dp);
	if (error != 0)
	return (error);

	error = dsl_dataset_hold(dp, zc->zc_name, FTAG, &ds);
	if (error != 0) {
	dsl_pool_rele(dp, FTAG);
	return (error);
	}

	if (!dsl_dir_is_clone(ds->ds_dir)) {
	dsl_dataset_rele(ds, FTAG);
	dsl_pool_rele(dp, FTAG);
	return (SET_ERROR(EINVAL));
	}

	error = dsl_dataset_hold_obj(dp,
	dsl_dir_phys(ds->ds_dir)->dd_origin_obj, FTAG, &ods);
	if (error != 0) {
	dsl_dataset_rele(ds, FTAG);
	dsl_pool_rele(dp, FTAG);
	return (error);
	}

	dsl_dataset_name(ods, origin);
	dsl_dataset_rele(ods, FTAG);
	dsl_dataset_rele(ds, FTAG);
	dsl_pool_rele(dp, FTAG);

	/*
	* We don't need to unmount all the origin fs's snapshots, but
	* it's easier.
	*/
	cp = strchr(origin, '@');
	if (cp)
	*cp = '\0';
	(void) dmu_objset_find(origin,
	zfs_unmount_snap_cb, NULL, DS_FIND_SNAPSHOTS);
	return (dsl_dataset_promote(zc->zc_name, zc->zc_string));
	}

	/*
	* Retrieve a single {user\|group\|project}{used\|quota}@... property.
	*
	* inputs:
	* zc_name name of filesystem
	* zc_objset_type zfs_userquota_prop_t
	* zc_value domain name (eg. "S-1-234-567-89")
	* zc_guid RID/UID/GID
	*
	* outputs:
	* zc_cookie property value
	*/
	static int
	zfs_ioc_userspace_one(zfs_cmd_t *zc)
	{
	zfsvfs_t *zfsvfs;
	int error;

	if (zc->zc_objset_type >= ZFS_NUM_USERQUOTA_PROPS)
	return (SET_ERROR(EINVAL));

	error = zfsvfs_hold(zc->zc_name, FTAG, &zfsvfs, B_FALSE);
	if (error != 0)
	return (error);

	error = zfs_userspace_one(zfsvfs,
	zc->zc_objset_type, zc->zc_value, zc->zc_guid, &zc->zc_cookie);
	zfsvfs_rele(zfsvfs, FTAG);

	return (error);
	}

	/*
	* inputs:
	* zc_name name of filesystem
	* zc_cookie zap cursor
	* zc_objset_type zfs_userquota_prop_t
	* zc_nvlist_dst[_size] buffer to fill (not really an nvlist)
	*
	* outputs:
	* zc_nvlist_dst[_size] data buffer (array of zfs_useracct_t)
	* zc_cookie zap cursor
	*/
	static int
	zfs_ioc_userspace_many(zfs_cmd_t *zc)
	{
	zfsvfs_t *zfsvfs;
	int bufsize = zc->zc_nvlist_dst_size;

	if (bufsize <= 0)
	return (SET_ERROR(ENOMEM));

	int error = zfsvfs_hold(zc->zc_name, FTAG, &zfsvfs, B_FALSE);
	if (error != 0)
	return (error);

	void *buf = vmem_alloc(bufsize, KM_SLEEP);

	error = zfs_userspace_many(zfsvfs, zc->zc_objset_type, &zc->zc_cookie,
	buf, &zc->zc_nvlist_dst_size);

	if (error == 0) {
	error = xcopyout(buf,
	(void *)(uintptr_t)zc->zc_nvlist_dst,
	zc->zc_nvlist_dst_size);
	}
	vmem_free(buf, bufsize);
	zfsvfs_rele(zfsvfs, FTAG);

	return (error);
	}

	/*
	* inputs:
	* zc_name name of filesystem
	*
	* outputs:
	* none
	*/
	static int
	zfs_ioc_userspace_upgrade(zfs_cmd_t *zc)
	{
	objset_t *os;
	int error = 0;
	zfsvfs_t *zfsvfs;

	if (getzfsvfs(zc->zc_name, &zfsvfs) == 0) {
	if (!dmu_objset_userused_enabled(zfsvfs->z_os)) {
	/*
	* If userused is not enabled, it may be because the
	* objset needs to be closed & reopened (to grow the
	* objset_phys_t). Suspend/resume the fs will do that.
	*/
	dsl_dataset_t ds, newds;

	ds = dmu_objset_ds(zfsvfs->z_os);
	error = zfs_suspend_fs(zfsvfs);
	if (error == 0) {
	dmu_objset_refresh_ownership(ds, &newds,
	B_TRUE, zfsvfs);
	error = zfs_resume_fs(zfsvfs, newds);
	}
	}
	if (error == 0)
	error = dmu_objset_userspace_upgrade(zfsvfs->z_os);
	zfs_vfs_rele(zfsvfs);
	} else {
	/* XXX kind of reading contents without owning */
	error = dmu_objset_hold_flags(zc->zc_name, B_TRUE, FTAG, &os);
	if (error != 0)
	return (error);

	error = dmu_objset_userspace_upgrade(os);
	dmu_objset_rele_flags(os, B_TRUE, FTAG);
	}

	return (error);
	}

	/*
	* inputs:
	* zc_name name of filesystem
	*
	* outputs:
	* none
	*/
	static int
	zfs_ioc_id_quota_upgrade(zfs_cmd_t *zc)
	{
	objset_t *os;
	int error;

	error = dmu_objset_hold_flags(zc->zc_name, B_TRUE, FTAG, &os);
	if (error != 0)
	return (error);

	if (dmu_objset_userobjspace_upgradable(os) \|\|
	dmu_objset_projectquota_upgradable(os)) {
	mutex_enter(&os->os_upgrade_lock);
	if (os->os_upgrade_id == 0) {
	/* clear potential error code and retry */
	os->os_upgrade_status = 0;
	mutex_exit(&os->os_upgrade_lock);

	dmu_objset_id_quota_upgrade(os);
	} else {
	mutex_exit(&os->os_upgrade_lock);
	}

	dsl_pool_rele(dmu_objset_pool(os), FTAG);

	taskq_wait_id(os->os_spa->spa_upgrade_taskq, os->os_upgrade_id);
	error = os->os_upgrade_status;
	} else {
	dsl_pool_rele(dmu_objset_pool(os), FTAG);
	}

	dsl_dataset_rele_flags(dmu_objset_ds(os), DS_HOLD_FLAG_DECRYPT, FTAG);

	return (error);
	}

	static int
	zfs_ioc_share(zfs_cmd_t *zc)
	{
	return (SET_ERROR(ENOSYS));
	}

	ace_t full_access[] = {
	{(uid_t)-1, ACE_ALL_PERMS, ACE_EVERYONE, 0}
	};

	/*
	* inputs:
	* zc_name name of containing filesystem
	* zc_obj object # beyond which we want next in-use object #
	*
	* outputs:
	* zc_obj next in-use object #
	*/
	static int
	zfs_ioc_next_obj(zfs_cmd_t *zc)
	{
	objset_t *os = NULL;
	int error;

	error = dmu_objset_hold(zc->zc_name, FTAG, &os);
	if (error != 0)
	return (error);

	error = dmu_object_next(os, &zc->zc_obj, B_FALSE, 0);

	dmu_objset_rele(os, FTAG);
	return (error);
	}

	/*
	* inputs:
	* zc_name name of filesystem
	* zc_value prefix name for snapshot
	* zc_cleanup_fd cleanup-on-exit file descriptor for calling process
	*
	* outputs:
	* zc_value short name of new snapshot
	*/
	static int
	zfs_ioc_tmp_snapshot(zfs_cmd_t *zc)
	{
	char *snap_name;
	char *hold_name;
	int error;
	minor_t minor;

	error = zfs_onexit_fd_hold(zc->zc_cleanup_fd, &minor);
	if (error != 0)
	return (error);

	snap_name = kmem_asprintf("%s-%016llx", zc->zc_value,
	(u_longlong_t)ddi_get_lbolt64());
	hold_name = kmem_asprintf("%%%s", zc->zc_value);

	error = dsl_dataset_snapshot_tmp(zc->zc_name, snap_name, minor,
	hold_name);
	if (error == 0)
	(void) strlcpy(zc->zc_value, snap_name,
	sizeof (zc->zc_value));
	kmem_strfree(snap_name);
	kmem_strfree(hold_name);
	zfs_onexit_fd_rele(zc->zc_cleanup_fd);
	return (error);
	}

	/*
	* inputs:
	* zc_name name of "to" snapshot
	* zc_value name of "from" snapshot
	* zc_cookie file descriptor to write diff data on
	*
	* outputs:
	* dmu_diff_record_t's to the file descriptor
	*/
	static int
	zfs_ioc_diff(zfs_cmd_t *zc)
	{
	zfs_file_t *fp;
	offset_t off;
	int error;

	if ((error = zfs_file_get(zc->zc_cookie, &fp)))
	return (error);

	off = zfs_file_off(fp);
	error = dmu_diff(zc->zc_name, zc->zc_value, fp, &off);

	zfs_file_put(zc->zc_cookie);

	return (error);
	}

	static int
	zfs_ioc_smb_acl(zfs_cmd_t *zc)
	{
	return (SET_ERROR(ENOTSUP));
	}

	/*
	* innvl: {
	* "holds" -> { snapname -> holdname (string), ... }
	* (optional) "cleanup_fd" -> fd (int32)
	* }
	*
	* outnvl: {
	* snapname -> error value (int32)
	* ...
	* }
	*/
	static const zfs_ioc_key_t zfs_keys_hold[] = {
	{"holds", DATA_TYPE_NVLIST, 0},
	{"cleanup_fd", DATA_TYPE_INT32, ZK_OPTIONAL},
	};

	/* ARGSUSED */
	static int
	zfs_ioc_hold(const char pool, nvlist_t args, nvlist_t *errlist)
	{
	nvpair_t *pair;
	nvlist_t *holds;
	int cleanup_fd = -1;
	int error;
	minor_t minor = 0;

	holds = fnvlist_lookup_nvlist(args, "holds");

	/* make sure the user didn't pass us any invalid (empty) tags */
	for (pair = nvlist_next_nvpair(holds, NULL); pair != NULL;
	pair = nvlist_next_nvpair(holds, pair)) {
	char *htag;

	error = nvpair_value_string(pair, &htag);
	if (error != 0)
	return (SET_ERROR(error));

	if (strlen(htag) == 0)
	return (SET_ERROR(EINVAL));
	}

	if (nvlist_lookup_int32(args, "cleanup_fd", &cleanup_fd) == 0) {
	error = zfs_onexit_fd_hold(cleanup_fd, &minor);
	if (error != 0)
	return (SET_ERROR(error));
	}

	error = dsl_dataset_user_hold(holds, minor, errlist);
	if (minor != 0)
	zfs_onexit_fd_rele(cleanup_fd);
	return (SET_ERROR(error));
	}

	/*
	* innvl is not used.
	*
	* outnvl: {
	* holdname -> time added (uint64 seconds since epoch)
	* ...
	* }
	*/
	static const zfs_ioc_key_t zfs_keys_get_holds[] = {
	/* no nvl keys */
	};

	/* ARGSUSED */
	static int
	zfs_ioc_get_holds(const char snapname, nvlist_t args, nvlist_t *outnvl)
	{
	return (dsl_dataset_get_holds(snapname, outnvl));
	}

	/*
	* innvl: {
	* snapname -> { holdname, ... }
	* ...
	* }
	*
	* outnvl: {
	* snapname -> error value (int32)
	* ...
	* }
	*/
	static const zfs_ioc_key_t zfs_keys_release[] = {
	{"<snapname>...", DATA_TYPE_NVLIST, ZK_WILDCARDLIST},
	};

	/* ARGSUSED */
	static int
	zfs_ioc_release(const char pool, nvlist_t holds, nvlist_t *errlist)
	{
	return (dsl_dataset_user_release(holds, errlist));
	}

	/*
	* inputs:
	* zc_guid flags (ZEVENT_NONBLOCK)
	* zc_cleanup_fd zevent file descriptor
	*
	* outputs:
	* zc_nvlist_dst next nvlist event
	* zc_cookie dropped events since last get
	*/
	static int
	zfs_ioc_events_next(zfs_cmd_t *zc)
	{
	zfs_zevent_t *ze;
	nvlist_t *event = NULL;
	minor_t minor;
	uint64_t dropped = 0;
	int error;

	error = zfs_zevent_fd_hold(zc->zc_cleanup_fd, &minor, &ze);
	if (error != 0)
	return (error);

	do {
	error = zfs_zevent_next(ze, &event,
	&zc->zc_nvlist_dst_size, &dropped);
	if (event != NULL) {
	zc->zc_cookie = dropped;
	error = put_nvlist(zc, event);
	nvlist_free(event);
	}

	if (zc->zc_guid & ZEVENT_NONBLOCK)
	break;

	if ((error == 0) \|\| (error != ENOENT))
	break;

	error = zfs_zevent_wait(ze);
	if (error != 0)
	break;
	} while (1);

	zfs_zevent_fd_rele(zc->zc_cleanup_fd);

	return (error);
	}

	/*
	* outputs:
	* zc_cookie cleared events count
	*/
	static int
	zfs_ioc_events_clear(zfs_cmd_t *zc)
	{
	int count;

	zfs_zevent_drain_all(&count);
	zc->zc_cookie = count;

	return (0);
	}

	/*
	* inputs:
	* zc_guid eid \| ZEVENT_SEEK_START \| ZEVENT_SEEK_END
	* zc_cleanup zevent file descriptor
	*/
	static int
	zfs_ioc_events_seek(zfs_cmd_t *zc)
	{
	zfs_zevent_t *ze;
	minor_t minor;
	int error;

	error = zfs_zevent_fd_hold(zc->zc_cleanup_fd, &minor, &ze);
	if (error != 0)
	return (error);

	error = zfs_zevent_seek(ze, zc->zc_guid);
	zfs_zevent_fd_rele(zc->zc_cleanup_fd);

	return (error);
	}

	/*
	* inputs:
	* zc_name name of later filesystem or snapshot
	* zc_value full name of old snapshot or bookmark
	*
	* outputs:
	* zc_cookie space in bytes
	* zc_objset_type compressed space in bytes
	* zc_perm_action uncompressed space in bytes
	*/
	static int
	zfs_ioc_space_written(zfs_cmd_t *zc)
	{
	int error;
	dsl_pool_t *dp;
	dsl_dataset_t *new;

	error = dsl_pool_hold(zc->zc_name, FTAG, &dp);
	if (error != 0)
	return (error);
	error = dsl_dataset_hold(dp, zc->zc_name, FTAG, &new);
	if (error != 0) {
	dsl_pool_rele(dp, FTAG);
	return (error);
	}
	if (strchr(zc->zc_value, '#') != NULL) {
	zfs_bookmark_phys_t bmp;
	error = dsl_bookmark_lookup(dp, zc->zc_value,
	new, &bmp);
	if (error == 0) {
	error = dsl_dataset_space_written_bookmark(&bmp, new,
	&zc->zc_cookie,
	&zc->zc_objset_type, &zc->zc_perm_action);
	}
	} else {
	dsl_dataset_t *old;
	error = dsl_dataset_hold(dp, zc->zc_value, FTAG, &old);

	if (error == 0) {
	error = dsl_dataset_space_written(old, new,
	&zc->zc_cookie,
	&zc->zc_objset_type, &zc->zc_perm_action);
	dsl_dataset_rele(old, FTAG);
	}
	}
	dsl_dataset_rele(new, FTAG);
	dsl_pool_rele(dp, FTAG);
	return (error);
	}

	/*
	* innvl: {
	* "firstsnap" -> snapshot name
	* }
	*
	* outnvl: {
	* "used" -> space in bytes
	* "compressed" -> compressed space in bytes
	* "uncompressed" -> uncompressed space in bytes
	* }
	*/
	static const zfs_ioc_key_t zfs_keys_space_snaps[] = {
	{"firstsnap", DATA_TYPE_STRING, 0},
	};

	static int
	zfs_ioc_space_snaps(const char lastsnap, nvlist_t innvl, nvlist_t *outnvl)
	{
	int error;
	dsl_pool_t *dp;
	dsl_dataset_t new, old;
	char *firstsnap;
	uint64_t used, comp, uncomp;

	firstsnap = fnvlist_lookup_string(innvl, "firstsnap");

	error = dsl_pool_hold(lastsnap, FTAG, &dp);
	if (error != 0)
	return (error);

	error = dsl_dataset_hold(dp, lastsnap, FTAG, &new);
	if (error == 0 && !new->ds_is_snapshot) {
	dsl_dataset_rele(new, FTAG);
	error = SET_ERROR(EINVAL);
	}
	if (error != 0) {
	dsl_pool_rele(dp, FTAG);
	return (error);
	}
	error = dsl_dataset_hold(dp, firstsnap, FTAG, &old);
	if (error == 0 && !old->ds_is_snapshot) {
	dsl_dataset_rele(old, FTAG);
	error = SET_ERROR(EINVAL);
	}
	if (error != 0) {
	dsl_dataset_rele(new, FTAG);
	dsl_pool_rele(dp, FTAG);
	return (error);
	}

	error = dsl_dataset_space_wouldfree(old, new, &used, &comp, &uncomp);
	dsl_dataset_rele(old, FTAG);
	dsl_dataset_rele(new, FTAG);
	dsl_pool_rele(dp, FTAG);
	fnvlist_add_uint64(outnvl, "used", used);
	fnvlist_add_uint64(outnvl, "compressed", comp);
	fnvlist_add_uint64(outnvl, "uncompressed", uncomp);
	return (error);
	}

	/*
	* innvl: {
	* "fd" -> file descriptor to write stream to (int32)
	* (optional) "fromsnap" -> full snap name to send an incremental from
	* (optional) "largeblockok" -> (value ignored)
	* indicates that blocks > 128KB are permitted
	* (optional) "embedok" -> (value ignored)
	* presence indicates DRR_WRITE_EMBEDDED records are permitted
	* (optional) "compressok" -> (value ignored)
	* presence indicates compressed DRR_WRITE records are permitted
	* (optional) "rawok" -> (value ignored)
	* presence indicates raw encrypted records should be used.
	* (optional) "savedok" -> (value ignored)
	* presence indicates we should send a partially received snapshot
	* (optional) "resume_object" and "resume_offset" -> (uint64)
	* if present, resume send stream from specified object and offset.
	* (optional) "redactbook" -> (string)
	* if present, use this bookmark's redaction list to generate a redacted
	* send stream
	* }
	*
	* outnvl is unused
	*/
	static const zfs_ioc_key_t zfs_keys_send_new[] = {
	{"fd", DATA_TYPE_INT32, 0},
	{"fromsnap", DATA_TYPE_STRING, ZK_OPTIONAL},
	{"largeblockok", DATA_TYPE_BOOLEAN, ZK_OPTIONAL},
	{"embedok", DATA_TYPE_BOOLEAN, ZK_OPTIONAL},
	{"compressok", DATA_TYPE_BOOLEAN, ZK_OPTIONAL},
	{"rawok", DATA_TYPE_BOOLEAN, ZK_OPTIONAL},
	{"savedok", DATA_TYPE_BOOLEAN, ZK_OPTIONAL},
	{"resume_object", DATA_TYPE_UINT64, ZK_OPTIONAL},
	{"resume_offset", DATA_TYPE_UINT64, ZK_OPTIONAL},
	{"redactbook", DATA_TYPE_STRING, ZK_OPTIONAL},
	};

	/* ARGSUSED */
	static int
	zfs_ioc_send_new(const char snapname, nvlist_t innvl, nvlist_t *outnvl)
	{
	int error;
	offset_t off;
	char *fromname = NULL;
	int fd;
	zfs_file_t *fp;
	boolean_t largeblockok;
	boolean_t embedok;
	boolean_t compressok;
	boolean_t rawok;
	boolean_t savedok;
	uint64_t resumeobj = 0;
	uint64_t resumeoff = 0;
	char *redactbook = NULL;

	fd = fnvlist_lookup_int32(innvl, "fd");

	(void) nvlist_lookup_string(innvl, "fromsnap", &fromname);

	largeblockok = nvlist_exists(innvl, "largeblockok");
	embedok = nvlist_exists(innvl, "embedok");
	compressok = nvlist_exists(innvl, "compressok");
	rawok = nvlist_exists(innvl, "rawok");
	savedok = nvlist_exists(innvl, "savedok");

	(void) nvlist_lookup_uint64(innvl, "resume_object", &resumeobj);
	(void) nvlist_lookup_uint64(innvl, "resume_offset", &resumeoff);

	(void) nvlist_lookup_string(innvl, "redactbook", &redactbook);

	if ((error = zfs_file_get(fd, &fp)))
	return (error);

	off = zfs_file_off(fp);

	dmu_send_outparams_t out = {0};
	out.dso_outfunc = dump_bytes;
	out.dso_arg = fp;
	out.dso_dryrun = B_FALSE;
	error = dmu_send(snapname, fromname, embedok, largeblockok,
	compressok, rawok, savedok, resumeobj, resumeoff,
	redactbook, fd, &off, &out);

	zfs_file_put(fd);
	return (error);
	}

	/* ARGSUSED */
	static int
	send_space_sum(objset_t os, void buf, int len, void *arg)
	{
	uint64_t *size = arg;
	*size += len;
	return (0);
	}

	/*
	* Determine approximately how large a zfs send stream will be -- the number
	* of bytes that will be written to the fd supplied to zfs_ioc_send_new().
	*
	* innvl: {
	* (optional) "from" -> full snap or bookmark name to send an incremental
	* from
	* (optional) "largeblockok" -> (value ignored)
	* indicates that blocks > 128KB are permitted
	* (optional) "embedok" -> (value ignored)
	* presence indicates DRR_WRITE_EMBEDDED records are permitted
	* (optional) "compressok" -> (value ignored)
	* presence indicates compressed DRR_WRITE records are permitted
	- * (optional) "rawok" -> (value ignored)
	+ * (optional) "rawok" -> (value ignored)
	* presence indicates raw encrypted records should be used.
	+ * (optional) "resume_object" and "resume_offset" -> (uint64)
	+ * if present, resume send stream from specified object and offset.
	* (optional) "fd" -> file descriptor to use as a cookie for progress
	* tracking (int32)
	* }
	*
	* outnvl: {
	* "space" -> bytes of space (uint64)
	* }
	*/
	static const zfs_ioc_key_t zfs_keys_send_space[] = {
	{"from", DATA_TYPE_STRING, ZK_OPTIONAL},
	{"fromsnap", DATA_TYPE_STRING, ZK_OPTIONAL},
	{"largeblockok", DATA_TYPE_BOOLEAN, ZK_OPTIONAL},
	{"embedok", DATA_TYPE_BOOLEAN, ZK_OPTIONAL},
	{"compressok", DATA_TYPE_BOOLEAN, ZK_OPTIONAL},
	{"rawok", DATA_TYPE_BOOLEAN, ZK_OPTIONAL},
	{"fd", DATA_TYPE_INT32, ZK_OPTIONAL},
	{"redactbook", DATA_TYPE_STRING, ZK_OPTIONAL},
	- {"resumeobj", DATA_TYPE_UINT64, ZK_OPTIONAL},
	- {"resumeoff", DATA_TYPE_UINT64, ZK_OPTIONAL},
	- {"bytes", DATA_TYPE_UINT64, ZK_OPTIONAL},
	+ {"resume_object", DATA_TYPE_UINT64, ZK_OPTIONAL},
	+ {"resume_offset", DATA_TYPE_UINT64, ZK_OPTIONAL},
	+ {"bytes", DATA_TYPE_UINT64, ZK_OPTIONAL},
	};

	static int
	zfs_ioc_send_space(const char snapname, nvlist_t innvl, nvlist_t *outnvl)
	{
	dsl_pool_t *dp;
	dsl_dataset_t *tosnap;
	dsl_dataset_t *fromsnap = NULL;
	int error;
	char *fromname = NULL;
	char *redactlist_book = NULL;
	boolean_t largeblockok;
	boolean_t embedok;
	boolean_t compressok;
	boolean_t rawok;
	boolean_t savedok;
	uint64_t space = 0;
	boolean_t full_estimate = B_FALSE;
	uint64_t resumeobj = 0;
	uint64_t resumeoff = 0;
	uint64_t resume_bytes = 0;
	int32_t fd = -1;
	zfs_bookmark_phys_t zbm = {0};

	error = dsl_pool_hold(snapname, FTAG, &dp);
	if (error != 0)
	return (error);

	error = dsl_dataset_hold(dp, snapname, FTAG, &tosnap);
	if (error != 0) {
	dsl_pool_rele(dp, FTAG);
	return (error);
	}
	(void) nvlist_lookup_int32(innvl, "fd", &fd);

	largeblockok = nvlist_exists(innvl, "largeblockok");
	embedok = nvlist_exists(innvl, "embedok");
	compressok = nvlist_exists(innvl, "compressok");
	rawok = nvlist_exists(innvl, "rawok");
	savedok = nvlist_exists(innvl, "savedok");
	boolean_t from = (nvlist_lookup_string(innvl, "from", &fromname) == 0);
	boolean_t altbook = (nvlist_lookup_string(innvl, "redactbook",
	&redactlist_book) == 0);

	(void) nvlist_lookup_uint64(innvl, "resume_object", &resumeobj);
	(void) nvlist_lookup_uint64(innvl, "resume_offset", &resumeoff);
	(void) nvlist_lookup_uint64(innvl, "bytes", &resume_bytes);

	if (altbook) {
	full_estimate = B_TRUE;
	} else if (from) {
	if (strchr(fromname, '#')) {
	error = dsl_bookmark_lookup(dp, fromname, tosnap, &zbm);

	/*
	* dsl_bookmark_lookup() will fail with EXDEV if
	* the from-bookmark and tosnap are at the same txg.
	* However, it's valid to do a send (and therefore,
	* a send estimate) from and to the same time point,
	* if the bookmark is redacted (the incremental send
	* can change what's redacted on the target). In
	* this case, dsl_bookmark_lookup() fills in zbm
	* but returns EXDEV. Ignore this error.
	*/
	if (error == EXDEV && zbm.zbm_redaction_obj != 0 &&
	zbm.zbm_guid ==
	dsl_dataset_phys(tosnap)->ds_guid)
	error = 0;

	if (error != 0) {
	dsl_dataset_rele(tosnap, FTAG);
	dsl_pool_rele(dp, FTAG);
	return (error);
	}
	if (zbm.zbm_redaction_obj != 0 \|\| !(zbm.zbm_flags &
	ZBM_FLAG_HAS_FBN)) {
	full_estimate = B_TRUE;
	}
	} else if (strchr(fromname, '@')) {
	error = dsl_dataset_hold(dp, fromname, FTAG, &fromsnap);
	if (error != 0) {
	dsl_dataset_rele(tosnap, FTAG);
	dsl_pool_rele(dp, FTAG);
	return (error);
	}

	if (!dsl_dataset_is_before(tosnap, fromsnap, 0)) {
	full_estimate = B_TRUE;
	dsl_dataset_rele(fromsnap, FTAG);
	}
	} else {
	/*
	* from is not properly formatted as a snapshot or
	* bookmark
	*/
	dsl_dataset_rele(tosnap, FTAG);
	dsl_pool_rele(dp, FTAG);
	return (SET_ERROR(EINVAL));
	}
	}

	if (full_estimate) {
	dmu_send_outparams_t out = {0};
	offset_t off = 0;
	out.dso_outfunc = send_space_sum;
	out.dso_arg = &space;
	out.dso_dryrun = B_TRUE;
	/*
	* We have to release these holds so dmu_send can take them. It
	* will do all the error checking we need.
	*/
	dsl_dataset_rele(tosnap, FTAG);
	dsl_pool_rele(dp, FTAG);
	error = dmu_send(snapname, fromname, embedok, largeblockok,
	compressok, rawok, savedok, resumeobj, resumeoff,
	redactlist_book, fd, &off, &out);
	} else {
	error = dmu_send_estimate_fast(tosnap, fromsnap,
	(from && strchr(fromname, '#') != NULL ? &zbm : NULL),
	compressok \|\| rawok, savedok, &space);
	space -= resume_bytes;
	if (fromsnap != NULL)
	dsl_dataset_rele(fromsnap, FTAG);
	dsl_dataset_rele(tosnap, FTAG);
	dsl_pool_rele(dp, FTAG);
	}

	fnvlist_add_uint64(outnvl, "space", space);

	return (error);
	}

	/*
	* Sync the currently open TXG to disk for the specified pool.
	* This is somewhat similar to 'zfs_sync()'.
	* For cases that do not result in error this ioctl will wait for
	* the currently open TXG to commit before returning back to the caller.
	*
	* innvl: {
	* "force" -> when true, force uberblock update even if there is no dirty data.
	* In addition this will cause the vdev configuration to be written
	* out including updating the zpool cache file. (boolean_t)
	* }
	*
	* onvl is unused
	*/
	static const zfs_ioc_key_t zfs_keys_pool_sync[] = {
	{"force", DATA_TYPE_BOOLEAN_VALUE, 0},
	};

	/* ARGSUSED */
	static int
	zfs_ioc_pool_sync(const char pool, nvlist_t innvl, nvlist_t *onvl)
	{
	int err;
	boolean_t force = B_FALSE;
	spa_t *spa;

	if ((err = spa_open(pool, &spa, FTAG)) != 0)
	return (err);

	if (innvl)
	force = fnvlist_lookup_boolean_value(innvl, "force");

	if (force) {
	spa_config_enter(spa, SCL_CONFIG, FTAG, RW_WRITER);
	vdev_config_dirty(spa->spa_root_vdev);
	spa_config_exit(spa, SCL_CONFIG, FTAG);
	}
	txg_wait_synced(spa_get_dsl(spa), 0);

	spa_close(spa, FTAG);

	return (err);
	}

	/*
	* Load a user's wrapping key into the kernel.
	* innvl: {
	* "hidden_args" -> { "wkeydata" -> value }
	* raw uint8_t array of encryption wrapping key data (32 bytes)
	* (optional) "noop" -> (value ignored)
	* presence indicated key should only be verified, not loaded
	* }
	*/
	static const zfs_ioc_key_t zfs_keys_load_key[] = {
	{"hidden_args", DATA_TYPE_NVLIST, 0},
	{"noop", DATA_TYPE_BOOLEAN, ZK_OPTIONAL},
	};

	/* ARGSUSED */
	static int
	zfs_ioc_load_key(const char dsname, nvlist_t innvl, nvlist_t *outnvl)
	{
	int ret;
	dsl_crypto_params_t *dcp = NULL;
	nvlist_t *hidden_args;
	boolean_t noop = nvlist_exists(innvl, "noop");

	if (strchr(dsname, '@') != NULL \|\| strchr(dsname, '%') != NULL) {
	ret = SET_ERROR(EINVAL);
	goto error;
	}

	hidden_args = fnvlist_lookup_nvlist(innvl, ZPOOL_HIDDEN_ARGS);

	ret = dsl_crypto_params_create_nvlist(DCP_CMD_NONE, NULL,
	hidden_args, &dcp);
	if (ret != 0)
	goto error;

	ret = spa_keystore_load_wkey(dsname, dcp, noop);
	if (ret != 0)
	goto error;

	dsl_crypto_params_free(dcp, noop);

	return (0);

	error:
	dsl_crypto_params_free(dcp, B_TRUE);
	return (ret);
	}

	/*
	* Unload a user's wrapping key from the kernel.
	* Both innvl and outnvl are unused.
	*/
	static const zfs_ioc_key_t zfs_keys_unload_key[] = {
	/* no nvl keys */
	};

	/* ARGSUSED */
	static int
	zfs_ioc_unload_key(const char dsname, nvlist_t innvl, nvlist_t *outnvl)
	{
	int ret = 0;

	if (strchr(dsname, '@') != NULL \|\| strchr(dsname, '%') != NULL) {
	ret = (SET_ERROR(EINVAL));
	goto out;
	}

	ret = spa_keystore_unload_wkey(dsname);
	if (ret != 0)
	goto out;

	out:
	return (ret);
	}

	/*
	* Changes a user's wrapping key used to decrypt a dataset. The keyformat,
	* keylocation, pbkdf2salt, and pbkdf2iters properties can also be specified
	* here to change how the key is derived in userspace.
	*
	* innvl: {
	* "hidden_args" (optional) -> { "wkeydata" -> value }
	* raw uint8_t array of new encryption wrapping key data (32 bytes)
	* "props" (optional) -> { prop -> value }
	* }
	*
	* outnvl is unused
	*/
	static const zfs_ioc_key_t zfs_keys_change_key[] = {
	{"crypt_cmd", DATA_TYPE_UINT64, ZK_OPTIONAL},
	{"hidden_args", DATA_TYPE_NVLIST, ZK_OPTIONAL},
	{"props", DATA_TYPE_NVLIST, ZK_OPTIONAL},
	};

	/* ARGSUSED */
	static int
	zfs_ioc_change_key(const char dsname, nvlist_t innvl, nvlist_t *outnvl)
	{
	int ret;
	uint64_t cmd = DCP_CMD_NONE;
	dsl_crypto_params_t *dcp = NULL;
	nvlist_t args = NULL, hidden_args = NULL;

	if (strchr(dsname, '@') != NULL \|\| strchr(dsname, '%') != NULL) {
	ret = (SET_ERROR(EINVAL));
	goto error;
	}

	(void) nvlist_lookup_uint64(innvl, "crypt_cmd", &cmd);
	(void) nvlist_lookup_nvlist(innvl, "props", &args);
	(void) nvlist_lookup_nvlist(innvl, ZPOOL_HIDDEN_ARGS, &hidden_args);

	ret = dsl_crypto_params_create_nvlist(cmd, args, hidden_args, &dcp);
	if (ret != 0)
	goto error;

	ret = spa_keystore_change_key(dsname, dcp);
	if (ret != 0)
	goto error;

	dsl_crypto_params_free(dcp, B_FALSE);

	return (0);

	error:
	dsl_crypto_params_free(dcp, B_TRUE);
	return (ret);
	}

	static zfs_ioc_vec_t zfs_ioc_vec[ZFS_IOC_LAST - ZFS_IOC_FIRST];

	static void
	zfs_ioctl_register_legacy(zfs_ioc_t ioc, zfs_ioc_legacy_func_t *func,
	zfs_secpolicy_func_t *secpolicy, zfs_ioc_namecheck_t namecheck,
	boolean_t log_history, zfs_ioc_poolcheck_t pool_check)
	{
	zfs_ioc_vec_t *vec = &zfs_ioc_vec[ioc - ZFS_IOC_FIRST];

	ASSERT3U(ioc, >=, ZFS_IOC_FIRST);
	ASSERT3U(ioc, <, ZFS_IOC_LAST);
	ASSERT3P(vec->zvec_legacy_func, ==, NULL);
	ASSERT3P(vec->zvec_func, ==, NULL);

	vec->zvec_legacy_func = func;
	vec->zvec_secpolicy = secpolicy;
	vec->zvec_namecheck = namecheck;
	vec->zvec_allow_log = log_history;
	vec->zvec_pool_check = pool_check;
	}

	/*
	* See the block comment at the beginning of this file for details on
	* each argument to this function.
	*/
	void
	zfs_ioctl_register(const char name, zfs_ioc_t ioc, zfs_ioc_func_t func,
	zfs_secpolicy_func_t *secpolicy, zfs_ioc_namecheck_t namecheck,
	zfs_ioc_poolcheck_t pool_check, boolean_t smush_outnvlist,
	boolean_t allow_log, const zfs_ioc_key_t *nvl_keys, size_t num_keys)
	{
	zfs_ioc_vec_t *vec = &zfs_ioc_vec[ioc - ZFS_IOC_FIRST];

	ASSERT3U(ioc, >=, ZFS_IOC_FIRST);
	ASSERT3U(ioc, <, ZFS_IOC_LAST);
	ASSERT3P(vec->zvec_legacy_func, ==, NULL);
	ASSERT3P(vec->zvec_func, ==, NULL);

	/* if we are logging, the name must be valid */
	ASSERT(!allow_log \|\| namecheck != NO_NAME);

	vec->zvec_name = name;
	vec->zvec_func = func;
	vec->zvec_secpolicy = secpolicy;
	vec->zvec_namecheck = namecheck;
	vec->zvec_pool_check = pool_check;
	vec->zvec_smush_outnvlist = smush_outnvlist;
	vec->zvec_allow_log = allow_log;
	vec->zvec_nvl_keys = nvl_keys;
	vec->zvec_nvl_key_count = num_keys;
	}

	static void
	zfs_ioctl_register_pool(zfs_ioc_t ioc, zfs_ioc_legacy_func_t *func,
	zfs_secpolicy_func_t *secpolicy, boolean_t log_history,
	zfs_ioc_poolcheck_t pool_check)
	{
	zfs_ioctl_register_legacy(ioc, func, secpolicy,
	POOL_NAME, log_history, pool_check);
	}

	void
	zfs_ioctl_register_dataset_nolog(zfs_ioc_t ioc, zfs_ioc_legacy_func_t *func,
	zfs_secpolicy_func_t *secpolicy, zfs_ioc_poolcheck_t pool_check)
	{
	zfs_ioctl_register_legacy(ioc, func, secpolicy,
	DATASET_NAME, B_FALSE, pool_check);
	}

	static void
	zfs_ioctl_register_pool_modify(zfs_ioc_t ioc, zfs_ioc_legacy_func_t *func)
	{
	zfs_ioctl_register_legacy(ioc, func, zfs_secpolicy_config,
	POOL_NAME, B_TRUE, POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY);
	}

	static void
	zfs_ioctl_register_pool_meta(zfs_ioc_t ioc, zfs_ioc_legacy_func_t *func,
	zfs_secpolicy_func_t *secpolicy)
	{
	zfs_ioctl_register_legacy(ioc, func, secpolicy,
	NO_NAME, B_FALSE, POOL_CHECK_NONE);
	}

	static void
	zfs_ioctl_register_dataset_read_secpolicy(zfs_ioc_t ioc,
	zfs_ioc_legacy_func_t func, zfs_secpolicy_func_t secpolicy)
	{
	zfs_ioctl_register_legacy(ioc, func, secpolicy,
	DATASET_NAME, B_FALSE, POOL_CHECK_SUSPENDED);
	}

	static void
	zfs_ioctl_register_dataset_read(zfs_ioc_t ioc, zfs_ioc_legacy_func_t *func)
	{
	zfs_ioctl_register_dataset_read_secpolicy(ioc, func,
	zfs_secpolicy_read);
	}

	static void
	zfs_ioctl_register_dataset_modify(zfs_ioc_t ioc, zfs_ioc_legacy_func_t *func,
	zfs_secpolicy_func_t *secpolicy)
	{
	zfs_ioctl_register_legacy(ioc, func, secpolicy,
	DATASET_NAME, B_TRUE, POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY);
	}

	static void
	zfs_ioctl_init(void)
	{
	zfs_ioctl_register("snapshot", ZFS_IOC_SNAPSHOT,
	zfs_ioc_snapshot, zfs_secpolicy_snapshot, POOL_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_TRUE, B_TRUE,
	zfs_keys_snapshot, ARRAY_SIZE(zfs_keys_snapshot));

	zfs_ioctl_register("log_history", ZFS_IOC_LOG_HISTORY,
	zfs_ioc_log_history, zfs_secpolicy_log_history, NO_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_FALSE, B_FALSE,
	zfs_keys_log_history, ARRAY_SIZE(zfs_keys_log_history));

	zfs_ioctl_register("space_snaps", ZFS_IOC_SPACE_SNAPS,
	zfs_ioc_space_snaps, zfs_secpolicy_read, DATASET_NAME,
	POOL_CHECK_SUSPENDED, B_FALSE, B_FALSE,
	zfs_keys_space_snaps, ARRAY_SIZE(zfs_keys_space_snaps));

	zfs_ioctl_register("send", ZFS_IOC_SEND_NEW,
	zfs_ioc_send_new, zfs_secpolicy_send_new, DATASET_NAME,
	POOL_CHECK_SUSPENDED, B_FALSE, B_FALSE,
	zfs_keys_send_new, ARRAY_SIZE(zfs_keys_send_new));

	zfs_ioctl_register("send_space", ZFS_IOC_SEND_SPACE,
	zfs_ioc_send_space, zfs_secpolicy_read, DATASET_NAME,
	POOL_CHECK_SUSPENDED, B_FALSE, B_FALSE,
	zfs_keys_send_space, ARRAY_SIZE(zfs_keys_send_space));

	zfs_ioctl_register("create", ZFS_IOC_CREATE,
	zfs_ioc_create, zfs_secpolicy_create_clone, DATASET_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_TRUE, B_TRUE,
	zfs_keys_create, ARRAY_SIZE(zfs_keys_create));

	zfs_ioctl_register("clone", ZFS_IOC_CLONE,
	zfs_ioc_clone, zfs_secpolicy_create_clone, DATASET_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_TRUE, B_TRUE,
	zfs_keys_clone, ARRAY_SIZE(zfs_keys_clone));

	zfs_ioctl_register("remap", ZFS_IOC_REMAP,
	zfs_ioc_remap, zfs_secpolicy_none, DATASET_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_FALSE, B_TRUE,
	zfs_keys_remap, ARRAY_SIZE(zfs_keys_remap));

	zfs_ioctl_register("destroy_snaps", ZFS_IOC_DESTROY_SNAPS,
	zfs_ioc_destroy_snaps, zfs_secpolicy_destroy_snaps, POOL_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_TRUE, B_TRUE,
	zfs_keys_destroy_snaps, ARRAY_SIZE(zfs_keys_destroy_snaps));

	zfs_ioctl_register("hold", ZFS_IOC_HOLD,
	zfs_ioc_hold, zfs_secpolicy_hold, POOL_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_TRUE, B_TRUE,
	zfs_keys_hold, ARRAY_SIZE(zfs_keys_hold));
	zfs_ioctl_register("release", ZFS_IOC_RELEASE,
	zfs_ioc_release, zfs_secpolicy_release, POOL_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_TRUE, B_TRUE,
	zfs_keys_release, ARRAY_SIZE(zfs_keys_release));

	zfs_ioctl_register("get_holds", ZFS_IOC_GET_HOLDS,
	zfs_ioc_get_holds, zfs_secpolicy_read, DATASET_NAME,
	POOL_CHECK_SUSPENDED, B_FALSE, B_FALSE,
	zfs_keys_get_holds, ARRAY_SIZE(zfs_keys_get_holds));

	zfs_ioctl_register("rollback", ZFS_IOC_ROLLBACK,
	zfs_ioc_rollback, zfs_secpolicy_rollback, DATASET_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_FALSE, B_TRUE,
	zfs_keys_rollback, ARRAY_SIZE(zfs_keys_rollback));

	zfs_ioctl_register("bookmark", ZFS_IOC_BOOKMARK,
	zfs_ioc_bookmark, zfs_secpolicy_bookmark, POOL_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_TRUE, B_TRUE,
	zfs_keys_bookmark, ARRAY_SIZE(zfs_keys_bookmark));

	zfs_ioctl_register("get_bookmarks", ZFS_IOC_GET_BOOKMARKS,
	zfs_ioc_get_bookmarks, zfs_secpolicy_read, DATASET_NAME,
	POOL_CHECK_SUSPENDED, B_FALSE, B_FALSE,
	zfs_keys_get_bookmarks, ARRAY_SIZE(zfs_keys_get_bookmarks));

	zfs_ioctl_register("get_bookmark_props", ZFS_IOC_GET_BOOKMARK_PROPS,
	zfs_ioc_get_bookmark_props, zfs_secpolicy_read, ENTITY_NAME,
	POOL_CHECK_SUSPENDED, B_FALSE, B_FALSE, zfs_keys_get_bookmark_props,
	ARRAY_SIZE(zfs_keys_get_bookmark_props));

	zfs_ioctl_register("destroy_bookmarks", ZFS_IOC_DESTROY_BOOKMARKS,
	zfs_ioc_destroy_bookmarks, zfs_secpolicy_destroy_bookmarks,
	POOL_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_TRUE, B_TRUE,
	zfs_keys_destroy_bookmarks,
	ARRAY_SIZE(zfs_keys_destroy_bookmarks));

	zfs_ioctl_register("receive", ZFS_IOC_RECV_NEW,
	zfs_ioc_recv_new, zfs_secpolicy_recv_new, DATASET_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_TRUE, B_TRUE,
	zfs_keys_recv_new, ARRAY_SIZE(zfs_keys_recv_new));
	zfs_ioctl_register("load-key", ZFS_IOC_LOAD_KEY,
	zfs_ioc_load_key, zfs_secpolicy_load_key,
	DATASET_NAME, POOL_CHECK_SUSPENDED, B_TRUE, B_TRUE,
	zfs_keys_load_key, ARRAY_SIZE(zfs_keys_load_key));
	zfs_ioctl_register("unload-key", ZFS_IOC_UNLOAD_KEY,
	zfs_ioc_unload_key, zfs_secpolicy_load_key,
	DATASET_NAME, POOL_CHECK_SUSPENDED, B_TRUE, B_TRUE,
	zfs_keys_unload_key, ARRAY_SIZE(zfs_keys_unload_key));
	zfs_ioctl_register("change-key", ZFS_IOC_CHANGE_KEY,
	zfs_ioc_change_key, zfs_secpolicy_change_key,
	DATASET_NAME, POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY,
	B_TRUE, B_TRUE, zfs_keys_change_key,
	ARRAY_SIZE(zfs_keys_change_key));

	zfs_ioctl_register("sync", ZFS_IOC_POOL_SYNC,
	zfs_ioc_pool_sync, zfs_secpolicy_none, POOL_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_FALSE, B_FALSE,
	zfs_keys_pool_sync, ARRAY_SIZE(zfs_keys_pool_sync));
	zfs_ioctl_register("reopen", ZFS_IOC_POOL_REOPEN, zfs_ioc_pool_reopen,
	zfs_secpolicy_config, POOL_NAME, POOL_CHECK_SUSPENDED, B_TRUE,
	B_TRUE, zfs_keys_pool_reopen, ARRAY_SIZE(zfs_keys_pool_reopen));

	zfs_ioctl_register("channel_program", ZFS_IOC_CHANNEL_PROGRAM,
	zfs_ioc_channel_program, zfs_secpolicy_config,
	POOL_NAME, POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_TRUE,
	B_TRUE, zfs_keys_channel_program,
	ARRAY_SIZE(zfs_keys_channel_program));

	zfs_ioctl_register("redact", ZFS_IOC_REDACT,
	zfs_ioc_redact, zfs_secpolicy_config, DATASET_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_TRUE, B_TRUE,
	zfs_keys_redact, ARRAY_SIZE(zfs_keys_redact));

	zfs_ioctl_register("zpool_checkpoint", ZFS_IOC_POOL_CHECKPOINT,
	zfs_ioc_pool_checkpoint, zfs_secpolicy_config, POOL_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_TRUE, B_TRUE,
	zfs_keys_pool_checkpoint, ARRAY_SIZE(zfs_keys_pool_checkpoint));

	zfs_ioctl_register("zpool_discard_checkpoint",
	ZFS_IOC_POOL_DISCARD_CHECKPOINT, zfs_ioc_pool_discard_checkpoint,
	zfs_secpolicy_config, POOL_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_TRUE, B_TRUE,
	zfs_keys_pool_discard_checkpoint,
	ARRAY_SIZE(zfs_keys_pool_discard_checkpoint));

	zfs_ioctl_register("initialize", ZFS_IOC_POOL_INITIALIZE,
	zfs_ioc_pool_initialize, zfs_secpolicy_config, POOL_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_TRUE, B_TRUE,
	zfs_keys_pool_initialize, ARRAY_SIZE(zfs_keys_pool_initialize));

	zfs_ioctl_register("trim", ZFS_IOC_POOL_TRIM,
	zfs_ioc_pool_trim, zfs_secpolicy_config, POOL_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_TRUE, B_TRUE,
	zfs_keys_pool_trim, ARRAY_SIZE(zfs_keys_pool_trim));

	zfs_ioctl_register("wait", ZFS_IOC_WAIT,
	zfs_ioc_wait, zfs_secpolicy_none, POOL_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_FALSE, B_FALSE,
	zfs_keys_pool_wait, ARRAY_SIZE(zfs_keys_pool_wait));

	zfs_ioctl_register("wait_fs", ZFS_IOC_WAIT_FS,
	zfs_ioc_wait_fs, zfs_secpolicy_none, DATASET_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_FALSE, B_FALSE,
	zfs_keys_fs_wait, ARRAY_SIZE(zfs_keys_fs_wait));

	zfs_ioctl_register("set_bootenv", ZFS_IOC_SET_BOOTENV,
	zfs_ioc_set_bootenv, zfs_secpolicy_config, POOL_NAME,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY, B_FALSE, B_TRUE,
	zfs_keys_set_bootenv, ARRAY_SIZE(zfs_keys_set_bootenv));

	zfs_ioctl_register("get_bootenv", ZFS_IOC_GET_BOOTENV,
	zfs_ioc_get_bootenv, zfs_secpolicy_none, POOL_NAME,
	POOL_CHECK_SUSPENDED, B_FALSE, B_TRUE,
	zfs_keys_get_bootenv, ARRAY_SIZE(zfs_keys_get_bootenv));

	/* IOCTLS that use the legacy function signature */

	zfs_ioctl_register_legacy(ZFS_IOC_POOL_FREEZE, zfs_ioc_pool_freeze,
	zfs_secpolicy_config, NO_NAME, B_FALSE, POOL_CHECK_READONLY);

	zfs_ioctl_register_pool(ZFS_IOC_POOL_CREATE, zfs_ioc_pool_create,
	zfs_secpolicy_config, B_TRUE, POOL_CHECK_NONE);
	zfs_ioctl_register_pool_modify(ZFS_IOC_POOL_SCAN,
	zfs_ioc_pool_scan);
	zfs_ioctl_register_pool_modify(ZFS_IOC_POOL_UPGRADE,
	zfs_ioc_pool_upgrade);
	zfs_ioctl_register_pool_modify(ZFS_IOC_VDEV_ADD,
	zfs_ioc_vdev_add);
	zfs_ioctl_register_pool_modify(ZFS_IOC_VDEV_REMOVE,
	zfs_ioc_vdev_remove);
	zfs_ioctl_register_pool_modify(ZFS_IOC_VDEV_SET_STATE,
	zfs_ioc_vdev_set_state);
	zfs_ioctl_register_pool_modify(ZFS_IOC_VDEV_ATTACH,
	zfs_ioc_vdev_attach);
	zfs_ioctl_register_pool_modify(ZFS_IOC_VDEV_DETACH,
	zfs_ioc_vdev_detach);
	zfs_ioctl_register_pool_modify(ZFS_IOC_VDEV_SETPATH,
	zfs_ioc_vdev_setpath);
	zfs_ioctl_register_pool_modify(ZFS_IOC_VDEV_SETFRU,
	zfs_ioc_vdev_setfru);
	zfs_ioctl_register_pool_modify(ZFS_IOC_POOL_SET_PROPS,
	zfs_ioc_pool_set_props);
	zfs_ioctl_register_pool_modify(ZFS_IOC_VDEV_SPLIT,
	zfs_ioc_vdev_split);
	zfs_ioctl_register_pool_modify(ZFS_IOC_POOL_REGUID,
	zfs_ioc_pool_reguid);

	zfs_ioctl_register_pool_meta(ZFS_IOC_POOL_CONFIGS,
	zfs_ioc_pool_configs, zfs_secpolicy_none);
	zfs_ioctl_register_pool_meta(ZFS_IOC_POOL_TRYIMPORT,
	zfs_ioc_pool_tryimport, zfs_secpolicy_config);
	zfs_ioctl_register_pool_meta(ZFS_IOC_INJECT_FAULT,
	zfs_ioc_inject_fault, zfs_secpolicy_inject);
	zfs_ioctl_register_pool_meta(ZFS_IOC_CLEAR_FAULT,
	zfs_ioc_clear_fault, zfs_secpolicy_inject);
	zfs_ioctl_register_pool_meta(ZFS_IOC_INJECT_LIST_NEXT,
	zfs_ioc_inject_list_next, zfs_secpolicy_inject);

	/*
	* pool destroy, and export don't log the history as part of
	* zfsdev_ioctl, but rather zfs_ioc_pool_export
	* does the logging of those commands.
	*/
	zfs_ioctl_register_pool(ZFS_IOC_POOL_DESTROY, zfs_ioc_pool_destroy,
	zfs_secpolicy_config, B_FALSE, POOL_CHECK_SUSPENDED);
	zfs_ioctl_register_pool(ZFS_IOC_POOL_EXPORT, zfs_ioc_pool_export,
	zfs_secpolicy_config, B_FALSE, POOL_CHECK_SUSPENDED);

	zfs_ioctl_register_pool(ZFS_IOC_POOL_STATS, zfs_ioc_pool_stats,
	zfs_secpolicy_read, B_FALSE, POOL_CHECK_NONE);
	zfs_ioctl_register_pool(ZFS_IOC_POOL_GET_PROPS, zfs_ioc_pool_get_props,
	zfs_secpolicy_read, B_FALSE, POOL_CHECK_NONE);

	zfs_ioctl_register_pool(ZFS_IOC_ERROR_LOG, zfs_ioc_error_log,
	zfs_secpolicy_inject, B_FALSE, POOL_CHECK_SUSPENDED);
	zfs_ioctl_register_pool(ZFS_IOC_DSOBJ_TO_DSNAME,
	zfs_ioc_dsobj_to_dsname,
	zfs_secpolicy_diff, B_FALSE, POOL_CHECK_SUSPENDED);
	zfs_ioctl_register_pool(ZFS_IOC_POOL_GET_HISTORY,
	zfs_ioc_pool_get_history,
	zfs_secpolicy_config, B_FALSE, POOL_CHECK_SUSPENDED);

	zfs_ioctl_register_pool(ZFS_IOC_POOL_IMPORT, zfs_ioc_pool_import,
	zfs_secpolicy_config, B_TRUE, POOL_CHECK_NONE);

	zfs_ioctl_register_pool(ZFS_IOC_CLEAR, zfs_ioc_clear,
	zfs_secpolicy_config, B_TRUE, POOL_CHECK_READONLY);

	zfs_ioctl_register_dataset_read(ZFS_IOC_SPACE_WRITTEN,
	zfs_ioc_space_written);
	zfs_ioctl_register_dataset_read(ZFS_IOC_OBJSET_RECVD_PROPS,
	zfs_ioc_objset_recvd_props);
	zfs_ioctl_register_dataset_read(ZFS_IOC_NEXT_OBJ,
	zfs_ioc_next_obj);
	zfs_ioctl_register_dataset_read(ZFS_IOC_GET_FSACL,
	zfs_ioc_get_fsacl);
	zfs_ioctl_register_dataset_read(ZFS_IOC_OBJSET_STATS,
	zfs_ioc_objset_stats);
	zfs_ioctl_register_dataset_read(ZFS_IOC_OBJSET_ZPLPROPS,
	zfs_ioc_objset_zplprops);
	zfs_ioctl_register_dataset_read(ZFS_IOC_DATASET_LIST_NEXT,
	zfs_ioc_dataset_list_next);
	zfs_ioctl_register_dataset_read(ZFS_IOC_SNAPSHOT_LIST_NEXT,
	zfs_ioc_snapshot_list_next);
	zfs_ioctl_register_dataset_read(ZFS_IOC_SEND_PROGRESS,
	zfs_ioc_send_progress);

	zfs_ioctl_register_dataset_read_secpolicy(ZFS_IOC_DIFF,
	zfs_ioc_diff, zfs_secpolicy_diff);
	zfs_ioctl_register_dataset_read_secpolicy(ZFS_IOC_OBJ_TO_STATS,
	zfs_ioc_obj_to_stats, zfs_secpolicy_diff);
	zfs_ioctl_register_dataset_read_secpolicy(ZFS_IOC_OBJ_TO_PATH,
	zfs_ioc_obj_to_path, zfs_secpolicy_diff);
	zfs_ioctl_register_dataset_read_secpolicy(ZFS_IOC_USERSPACE_ONE,
	zfs_ioc_userspace_one, zfs_secpolicy_userspace_one);
	zfs_ioctl_register_dataset_read_secpolicy(ZFS_IOC_USERSPACE_MANY,
	zfs_ioc_userspace_many, zfs_secpolicy_userspace_many);
	zfs_ioctl_register_dataset_read_secpolicy(ZFS_IOC_SEND,
	zfs_ioc_send, zfs_secpolicy_send);

	zfs_ioctl_register_dataset_modify(ZFS_IOC_SET_PROP, zfs_ioc_set_prop,
	zfs_secpolicy_none);
	zfs_ioctl_register_dataset_modify(ZFS_IOC_DESTROY, zfs_ioc_destroy,
	zfs_secpolicy_destroy);
	zfs_ioctl_register_dataset_modify(ZFS_IOC_RENAME, zfs_ioc_rename,
	zfs_secpolicy_rename);
	zfs_ioctl_register_dataset_modify(ZFS_IOC_RECV, zfs_ioc_recv,
	zfs_secpolicy_recv);
	zfs_ioctl_register_dataset_modify(ZFS_IOC_PROMOTE, zfs_ioc_promote,
	zfs_secpolicy_promote);
	zfs_ioctl_register_dataset_modify(ZFS_IOC_INHERIT_PROP,
	zfs_ioc_inherit_prop, zfs_secpolicy_inherit_prop);
	zfs_ioctl_register_dataset_modify(ZFS_IOC_SET_FSACL, zfs_ioc_set_fsacl,
	zfs_secpolicy_set_fsacl);

	zfs_ioctl_register_dataset_nolog(ZFS_IOC_SHARE, zfs_ioc_share,
	zfs_secpolicy_share, POOL_CHECK_NONE);
	zfs_ioctl_register_dataset_nolog(ZFS_IOC_SMB_ACL, zfs_ioc_smb_acl,
	zfs_secpolicy_smb_acl, POOL_CHECK_NONE);
	zfs_ioctl_register_dataset_nolog(ZFS_IOC_USERSPACE_UPGRADE,
	zfs_ioc_userspace_upgrade, zfs_secpolicy_userspace_upgrade,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY);
	zfs_ioctl_register_dataset_nolog(ZFS_IOC_TMP_SNAPSHOT,
	zfs_ioc_tmp_snapshot, zfs_secpolicy_tmp_snapshot,
	POOL_CHECK_SUSPENDED \| POOL_CHECK_READONLY);

	zfs_ioctl_register_legacy(ZFS_IOC_EVENTS_NEXT, zfs_ioc_events_next,
	zfs_secpolicy_config, NO_NAME, B_FALSE, POOL_CHECK_NONE);
	zfs_ioctl_register_legacy(ZFS_IOC_EVENTS_CLEAR, zfs_ioc_events_clear,
	zfs_secpolicy_config, NO_NAME, B_FALSE, POOL_CHECK_NONE);
	zfs_ioctl_register_legacy(ZFS_IOC_EVENTS_SEEK, zfs_ioc_events_seek,
	zfs_secpolicy_config, NO_NAME, B_FALSE, POOL_CHECK_NONE);

	zfs_ioctl_init_os();
	}

	/*
	* Verify that for non-legacy ioctls the input nvlist
	* pairs match against the expected input.
	*
	* Possible errors are:
	* ZFS_ERR_IOC_ARG_UNAVAIL An unrecognized nvpair was encountered
	* ZFS_ERR_IOC_ARG_REQUIRED A required nvpair is missing
	* ZFS_ERR_IOC_ARG_BADTYPE Invalid type for nvpair
	*/
	static int
	zfs_check_input_nvpairs(nvlist_t innvl, const zfs_ioc_vec_t vec)
	{
	const zfs_ioc_key_t *nvl_keys = vec->zvec_nvl_keys;
	boolean_t required_keys_found = B_FALSE;

	/*
	* examine each input pair
	*/
	for (nvpair_t *pair = nvlist_next_nvpair(innvl, NULL);
	pair != NULL; pair = nvlist_next_nvpair(innvl, pair)) {
	char *name = nvpair_name(pair);
	data_type_t type = nvpair_type(pair);
	boolean_t identified = B_FALSE;

	/*
	* check pair against the documented names and type
	*/
	for (int k = 0; k < vec->zvec_nvl_key_count; k++) {
	/* if not a wild card name, check for an exact match */
	if ((nvl_keys[k].zkey_flags & ZK_WILDCARDLIST) == 0 &&
	strcmp(nvl_keys[k].zkey_name, name) != 0)
	continue;

	identified = B_TRUE;

	if (nvl_keys[k].zkey_type != DATA_TYPE_ANY &&
	nvl_keys[k].zkey_type != type) {
	return (SET_ERROR(ZFS_ERR_IOC_ARG_BADTYPE));
	}

	if (nvl_keys[k].zkey_flags & ZK_OPTIONAL)
	continue;

	required_keys_found = B_TRUE;
	break;
	}

	/* allow an 'optional' key, everything else is invalid */
	if (!identified &&
	(strcmp(name, "optional") != 0 \|\|
	type != DATA_TYPE_NVLIST)) {
	return (SET_ERROR(ZFS_ERR_IOC_ARG_UNAVAIL));
	}
	}

	/* verify that all required keys were found */
	for (int k = 0; k < vec->zvec_nvl_key_count; k++) {
	if (nvl_keys[k].zkey_flags & ZK_OPTIONAL)
	continue;

	if (nvl_keys[k].zkey_flags & ZK_WILDCARDLIST) {
	/* at least one non-optional key is expected here */
	if (!required_keys_found)
	return (SET_ERROR(ZFS_ERR_IOC_ARG_REQUIRED));
	continue;
	}

	if (!nvlist_exists(innvl, nvl_keys[k].zkey_name))
	return (SET_ERROR(ZFS_ERR_IOC_ARG_REQUIRED));
	}

	return (0);
	}

	static int
	pool_status_check(const char *name, zfs_ioc_namecheck_t type,
	zfs_ioc_poolcheck_t check)
	{
	spa_t *spa;
	int error;

	ASSERT(type == POOL_NAME \|\| type == DATASET_NAME \|\|
	type == ENTITY_NAME);

	if (check & POOL_CHECK_NONE)
	return (0);

	error = spa_open(name, &spa, FTAG);
	if (error == 0) {
	if ((check & POOL_CHECK_SUSPENDED) && spa_suspended(spa))
	error = SET_ERROR(EAGAIN);
	else if ((check & POOL_CHECK_READONLY) && !spa_writeable(spa))
	error = SET_ERROR(EROFS);
	spa_close(spa, FTAG);
	}
	return (error);
	}

	int
	zfsdev_getminor(int fd, minor_t *minorp)
	{
	zfsdev_state_t zs, fpd;
	zfs_file_t *fp;
	int rc;

	ASSERT(!MUTEX_HELD(&zfsdev_state_lock));

	if ((rc = zfs_file_get(fd, &fp)))
	return (rc);

	fpd = zfs_file_private(fp);
	if (fpd == NULL)
	return (SET_ERROR(EBADF));

	mutex_enter(&zfsdev_state_lock);

	for (zs = zfsdev_state_list; zs != NULL; zs = zs->zs_next) {

	if (zs->zs_minor == -1)
	continue;

	if (fpd == zs) {
	*minorp = fpd->zs_minor;
	mutex_exit(&zfsdev_state_lock);
	return (0);
	}
	}

	mutex_exit(&zfsdev_state_lock);

	return (SET_ERROR(EBADF));
	}

	static void *
	zfsdev_get_state_impl(minor_t minor, enum zfsdev_state_type which)
	{
	zfsdev_state_t *zs;

	for (zs = zfsdev_state_list; zs != NULL; zs = zs->zs_next) {
	if (zs->zs_minor == minor) {
	smp_rmb();
	switch (which) {
	case ZST_ONEXIT:
	return (zs->zs_onexit);
	case ZST_ZEVENT:
	return (zs->zs_zevent);
	case ZST_ALL:
	return (zs);
	}
	}
	}

	return (NULL);
	}

	void *
	zfsdev_get_state(minor_t minor, enum zfsdev_state_type which)
	{
	void *ptr;

	ptr = zfsdev_get_state_impl(minor, which);

	return (ptr);
	}

	/*
	* Find a free minor number. The zfsdev_state_list is expected to
	* be short since it is only a list of currently open file handles.
	*/
	minor_t
	zfsdev_minor_alloc(void)
	{
	static minor_t last_minor = 0;
	minor_t m;

	ASSERT(MUTEX_HELD(&zfsdev_state_lock));

	for (m = last_minor + 1; m != last_minor; m++) {
	if (m > ZFSDEV_MAX_MINOR)
	m = 1;
	if (zfsdev_get_state_impl(m, ZST_ALL) == NULL) {
	last_minor = m;
	return (m);
	}
	}

	return (0);
	}

	long
	zfsdev_ioctl_common(uint_t vecnum, zfs_cmd_t *zc, int flag)
	{
	int error, cmd;
	const zfs_ioc_vec_t *vec;
	char *saved_poolname = NULL;
	uint64_t max_nvlist_src_size;
	size_t saved_poolname_len = 0;
	nvlist_t *innvl = NULL;
	fstrans_cookie_t cookie;

	cmd = vecnum;
	error = 0;
	if (vecnum >= sizeof (zfs_ioc_vec) / sizeof (zfs_ioc_vec[0]))
	return (SET_ERROR(ZFS_ERR_IOC_CMD_UNAVAIL));

	vec = &zfs_ioc_vec[vecnum];

	/*
	* The registered ioctl list may be sparse, verify that either
	* a normal or legacy handler are registered.
	*/
	if (vec->zvec_func == NULL && vec->zvec_legacy_func == NULL)
	return (SET_ERROR(ZFS_ERR_IOC_CMD_UNAVAIL));

	zc->zc_iflags = flag & FKIOCTL;
	max_nvlist_src_size = zfs_max_nvlist_src_size_os();
	if (zc->zc_nvlist_src_size > max_nvlist_src_size) {
	/*
	* Make sure the user doesn't pass in an insane value for
	* zc_nvlist_src_size. We have to check, since we will end
	* up allocating that much memory inside of get_nvlist(). This
	* prevents a nefarious user from allocating tons of kernel
	* memory.
	*
	* Also, we return EINVAL instead of ENOMEM here. The reason
	* being that returning ENOMEM from an ioctl() has a special
	* connotation; that the user's size value is too small and
	* needs to be expanded to hold the nvlist. See
	* zcmd_expand_dst_nvlist() for details.
	*/
	error = SET_ERROR(EINVAL); /* User's size too big */

	} else if (zc->zc_nvlist_src_size != 0) {
	error = get_nvlist(zc->zc_nvlist_src, zc->zc_nvlist_src_size,
	zc->zc_iflags, &innvl);
	if (error != 0)
	goto out;
	}

	/*
	* Ensure that all pool/dataset names are valid before we pass down to
	* the lower layers.
	*/
	zc->zc_name[sizeof (zc->zc_name) - 1] = '\0';
	switch (vec->zvec_namecheck) {
	case POOL_NAME:
	if (pool_namecheck(zc->zc_name, NULL, NULL) != 0)
	error = SET_ERROR(EINVAL);
	else
	error = pool_status_check(zc->zc_name,
	vec->zvec_namecheck, vec->zvec_pool_check);
	break;

	case DATASET_NAME:
	if (dataset_namecheck(zc->zc_name, NULL, NULL) != 0)
	error = SET_ERROR(EINVAL);
	else
	error = pool_status_check(zc->zc_name,
	vec->zvec_namecheck, vec->zvec_pool_check);
	break;

	case ENTITY_NAME:
	if (entity_namecheck(zc->zc_name, NULL, NULL) != 0) {
	error = SET_ERROR(EINVAL);
	} else {
	error = pool_status_check(zc->zc_name,
	vec->zvec_namecheck, vec->zvec_pool_check);
	}
	break;

	case NO_NAME:
	break;
	}
	/*
	* Ensure that all input pairs are valid before we pass them down
	* to the lower layers.
	*
	* The vectored functions can use fnvlist_lookup_{type} for any
	* required pairs since zfs_check_input_nvpairs() confirmed that
	* they exist and are of the correct type.
	*/
	if (error == 0 && vec->zvec_func != NULL) {
	error = zfs_check_input_nvpairs(innvl, vec);
	if (error != 0)
	goto out;
	}

	if (error == 0) {
	cookie = spl_fstrans_mark();
	error = vec->zvec_secpolicy(zc, innvl, CRED());
	spl_fstrans_unmark(cookie);
	}

	if (error != 0)
	goto out;

	/* legacy ioctls can modify zc_name */
	/*
	* Can't use kmem_strdup() as we might truncate the string and
	* kmem_strfree() would then free with incorrect size.
	*/
	saved_poolname_len = strlen(zc->zc_name) + 1;
	saved_poolname = kmem_alloc(saved_poolname_len, KM_SLEEP);

	strlcpy(saved_poolname, zc->zc_name, saved_poolname_len);
	saved_poolname[strcspn(saved_poolname, "/@#")] = '\0';

	if (vec->zvec_func != NULL) {
	nvlist_t *outnvl;
	int puterror = 0;
	spa_t *spa;
	nvlist_t *lognv = NULL;

	ASSERT(vec->zvec_legacy_func == NULL);

	/*
	* Add the innvl to the lognv before calling the func,
	* in case the func changes the innvl.
	*/
	if (vec->zvec_allow_log) {
	lognv = fnvlist_alloc();
	fnvlist_add_string(lognv, ZPOOL_HIST_IOCTL,
	vec->zvec_name);
	if (!nvlist_empty(innvl)) {
	fnvlist_add_nvlist(lognv, ZPOOL_HIST_INPUT_NVL,
	innvl);
	}
	}

	outnvl = fnvlist_alloc();
	cookie = spl_fstrans_mark();
	error = vec->zvec_func(zc->zc_name, innvl, outnvl);
	spl_fstrans_unmark(cookie);

	/*
	* Some commands can partially execute, modify state, and still
	* return an error. In these cases, attempt to record what
	* was modified.
	*/
	if ((error == 0 \|\|
	(cmd == ZFS_IOC_CHANNEL_PROGRAM && error != EINVAL)) &&
	vec->zvec_allow_log &&
	spa_open(zc->zc_name, &spa, FTAG) == 0) {
	if (!nvlist_empty(outnvl)) {
	fnvlist_add_nvlist(lognv, ZPOOL_HIST_OUTPUT_NVL,
	outnvl);
	}
	if (error != 0) {
	fnvlist_add_int64(lognv, ZPOOL_HIST_ERRNO,
	error);
	}
	(void) spa_history_log_nvl(spa, lognv);
	spa_close(spa, FTAG);
	}
	fnvlist_free(lognv);

	if (!nvlist_empty(outnvl) \|\| zc->zc_nvlist_dst_size != 0) {
	int smusherror = 0;
	if (vec->zvec_smush_outnvlist) {
	smusherror = nvlist_smush(outnvl,
	zc->zc_nvlist_dst_size);
	}
	if (smusherror == 0)
	puterror = put_nvlist(zc, outnvl);
	}

	if (puterror != 0)
	error = puterror;

	nvlist_free(outnvl);
	} else {
	cookie = spl_fstrans_mark();
	error = vec->zvec_legacy_func(zc);
	spl_fstrans_unmark(cookie);
	}

	out:
	nvlist_free(innvl);
	if (error == 0 && vec->zvec_allow_log) {
	char *s = tsd_get(zfs_allow_log_key);
	if (s != NULL)
	kmem_strfree(s);
	(void) tsd_set(zfs_allow_log_key, kmem_strdup(saved_poolname));
	}
	if (saved_poolname != NULL)
	kmem_free(saved_poolname, saved_poolname_len);

	return (error);
	}

	int
	zfs_kmod_init(void)
	{
	int error;

	if ((error = zvol_init()) != 0)
	return (error);

	spa_init(SPA_MODE_READ \| SPA_MODE_WRITE);
	zfs_init();

	zfs_ioctl_init();

	mutex_init(&zfsdev_state_lock, NULL, MUTEX_DEFAULT, NULL);
	zfsdev_state_list = kmem_zalloc(sizeof (zfsdev_state_t), KM_SLEEP);
	zfsdev_state_list->zs_minor = -1;

	if ((error = zfsdev_attach()) != 0)
	goto out;

	tsd_create(&zfs_fsyncer_key, NULL);
	tsd_create(&rrw_tsd_key, rrw_tsd_destroy);
	tsd_create(&zfs_allow_log_key, zfs_allow_log_destroy);

	return (0);
	out:
	zfs_fini();
	spa_fini();
	zvol_fini();

	return (error);
	}

	void
	zfs_kmod_fini(void)
	{
	zfsdev_state_t zs, zsnext = NULL;

	zfsdev_detach();

	mutex_destroy(&zfsdev_state_lock);

	for (zs = zfsdev_state_list; zs != NULL; zs = zsnext) {
	zsnext = zs->zs_next;
	if (zs->zs_onexit)
	zfs_onexit_destroy(zs->zs_onexit);
	if (zs->zs_zevent)
	zfs_zevent_destroy(zs->zs_zevent);
	kmem_free(zs, sizeof (zfsdev_state_t));
	}

	zfs_ereport_taskq_fini(); /* run before zfs_fini() on Linux */
	zfs_fini();
	spa_fini();
	zvol_fini();

	tsd_destroy(&zfs_fsyncer_key);
	tsd_destroy(&rrw_tsd_key);
	tsd_destroy(&zfs_allow_log_key);
	}

	/* BEGIN CSTYLED */
	ZFS_MODULE_PARAM(zfs, zfs_, max_nvlist_src_size, ULONG, ZMOD_RW,
	"Maximum size in bytes allowed for src nvlist passed with ZFS ioctls");
	/* END CSTYLED */
	Index: vendor-sys/openzfs/dist/module/zfs/zfs_log.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/zfs/zfs_log.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/zfs/zfs_log.c (revision 366774)
	@@ -1,781 +1,781 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2015, 2018 by Delphix. All rights reserved.
	*/


	#include <sys/types.h>
	#include <sys/param.h>
	#include <sys/sysmacros.h>
	#include <sys/cmn_err.h>
	#include <sys/kmem.h>
	#include <sys/thread.h>
	#include <sys/file.h>
	#include <sys/vfs.h>
	#include <sys/zfs_znode.h>
	#include <sys/zfs_dir.h>
	#include <sys/zil.h>
	#include <sys/zil_impl.h>
	#include <sys/byteorder.h>
	#include <sys/policy.h>
	#include <sys/stat.h>
	#include <sys/acl.h>
	#include <sys/dmu.h>
	#include <sys/dbuf.h>
	#include <sys/spa.h>
	#include <sys/zfs_fuid.h>
	#include <sys/dsl_dataset.h>

	/*
	* These zfs_log_* functions must be called within a dmu tx, in one
	* of 2 contexts depending on zilog->z_replay:
	*
	* Non replay mode
	* ---------------
	* We need to record the transaction so that if it is committed to
	* the Intent Log then it can be replayed. An intent log transaction
	* structure (itx_t) is allocated and all the information necessary to
	* possibly replay the transaction is saved in it. The itx is then assigned
	* a sequence number and inserted in the in-memory list anchored in the zilog.
	*
	* Replay mode
	* -----------
	* We need to mark the intent log record as replayed in the log header.
	* This is done in the same transaction as the replay so that they
	* commit atomically.
	*/

	int
	zfs_log_create_txtype(zil_create_t type, vsecattr_t vsecp, vattr_t vap)
	{
	int isxvattr = (vap->va_mask & ATTR_XVATTR);
	switch (type) {
	case Z_FILE:
	if (vsecp == NULL && !isxvattr)
	return (TX_CREATE);
	if (vsecp && isxvattr)
	return (TX_CREATE_ACL_ATTR);
	if (vsecp)
	return (TX_CREATE_ACL);
	else
	return (TX_CREATE_ATTR);
	/NOTREACHED/
	case Z_DIR:
	if (vsecp == NULL && !isxvattr)
	return (TX_MKDIR);
	if (vsecp && isxvattr)
	return (TX_MKDIR_ACL_ATTR);
	if (vsecp)
	return (TX_MKDIR_ACL);
	else
	return (TX_MKDIR_ATTR);
	case Z_XATTRDIR:
	return (TX_MKXATTR);
	}
	ASSERT(0);
	return (TX_MAX_TYPE);
	}

	/*
	* build up the log data necessary for logging xvattr_t
	* First lr_attr_t is initialized. following the lr_attr_t
	* is the mapsize and attribute bitmap copied from the xvattr_t.
	* Following the bitmap and bitmapsize two 64 bit words are reserved
	* for the create time which may be set. Following the create time
	* records a single 64 bit integer which has the bits to set on
	* replay for the xvattr.
	*/
	static void
	zfs_log_xvattr(lr_attr_t lrattr, xvattr_t xvap)
	{
	uint32_t *bitmap;
	uint64_t *attrs;
	uint64_t *crtime;
	xoptattr_t *xoap;
	void *scanstamp;
	int i;

	xoap = xva_getxoptattr(xvap);
	ASSERT(xoap);

	lrattr->lr_attr_masksize = xvap->xva_mapsize;
	bitmap = &lrattr->lr_attr_bitmap;
	for (i = 0; i != xvap->xva_mapsize; i++, bitmap++) {
	*bitmap = xvap->xva_reqattrmap[i];
	}

	/* Now pack the attributes up in a single uint64_t */
	attrs = (uint64_t *)bitmap;
	crtime = attrs + 1;
	scanstamp = (caddr_t)(crtime + 2);
	*attrs = 0;
	if (XVA_ISSET_REQ(xvap, XAT_READONLY))
	*attrs \|= (xoap->xoa_readonly == 0) ? 0 :
	XAT0_READONLY;
	if (XVA_ISSET_REQ(xvap, XAT_HIDDEN))
	*attrs \|= (xoap->xoa_hidden == 0) ? 0 :
	XAT0_HIDDEN;
	if (XVA_ISSET_REQ(xvap, XAT_SYSTEM))
	*attrs \|= (xoap->xoa_system == 0) ? 0 :
	XAT0_SYSTEM;
	if (XVA_ISSET_REQ(xvap, XAT_ARCHIVE))
	*attrs \|= (xoap->xoa_archive == 0) ? 0 :
	XAT0_ARCHIVE;
	if (XVA_ISSET_REQ(xvap, XAT_IMMUTABLE))
	*attrs \|= (xoap->xoa_immutable == 0) ? 0 :
	XAT0_IMMUTABLE;
	if (XVA_ISSET_REQ(xvap, XAT_NOUNLINK))
	*attrs \|= (xoap->xoa_nounlink == 0) ? 0 :
	XAT0_NOUNLINK;
	if (XVA_ISSET_REQ(xvap, XAT_APPENDONLY))
	*attrs \|= (xoap->xoa_appendonly == 0) ? 0 :
	XAT0_APPENDONLY;
	if (XVA_ISSET_REQ(xvap, XAT_OPAQUE))
	*attrs \|= (xoap->xoa_opaque == 0) ? 0 :
	XAT0_APPENDONLY;
	if (XVA_ISSET_REQ(xvap, XAT_NODUMP))
	*attrs \|= (xoap->xoa_nodump == 0) ? 0 :
	XAT0_NODUMP;
	if (XVA_ISSET_REQ(xvap, XAT_AV_QUARANTINED))
	*attrs \|= (xoap->xoa_av_quarantined == 0) ? 0 :
	XAT0_AV_QUARANTINED;
	if (XVA_ISSET_REQ(xvap, XAT_AV_MODIFIED))
	*attrs \|= (xoap->xoa_av_modified == 0) ? 0 :
	XAT0_AV_MODIFIED;
	if (XVA_ISSET_REQ(xvap, XAT_CREATETIME))
	ZFS_TIME_ENCODE(&xoap->xoa_createtime, crtime);
	if (XVA_ISSET_REQ(xvap, XAT_AV_SCANSTAMP)) {
	ASSERT(!XVA_ISSET_REQ(xvap, XAT_PROJID));

	bcopy(xoap->xoa_av_scanstamp, scanstamp, AV_SCANSTAMP_SZ);
	} else if (XVA_ISSET_REQ(xvap, XAT_PROJID)) {
	/*
	* XAT_PROJID and XAT_AV_SCANSTAMP will never be valid
	* at the same time, so we can share the same space.
	*/
	bcopy(&xoap->xoa_projid, scanstamp, sizeof (uint64_t));
	}
	if (XVA_ISSET_REQ(xvap, XAT_REPARSE))
	*attrs \|= (xoap->xoa_reparse == 0) ? 0 :
	XAT0_REPARSE;
	if (XVA_ISSET_REQ(xvap, XAT_OFFLINE))
	*attrs \|= (xoap->xoa_offline == 0) ? 0 :
	XAT0_OFFLINE;
	if (XVA_ISSET_REQ(xvap, XAT_SPARSE))
	*attrs \|= (xoap->xoa_sparse == 0) ? 0 :
	XAT0_SPARSE;
	if (XVA_ISSET_REQ(xvap, XAT_PROJINHERIT))
	*attrs \|= (xoap->xoa_projinherit == 0) ? 0 :
	XAT0_PROJINHERIT;
	}

	static void *
	zfs_log_fuid_ids(zfs_fuid_info_t fuidp, void start)
	{
	zfs_fuid_t *zfuid;
	uint64_t *fuidloc = start;

	/* First copy in the ACE FUIDs */
	for (zfuid = list_head(&fuidp->z_fuids); zfuid;
	zfuid = list_next(&fuidp->z_fuids, zfuid)) {
	*fuidloc++ = zfuid->z_logfuid;
	}
	return (fuidloc);
	}


	static void *
	zfs_log_fuid_domains(zfs_fuid_info_t fuidp, void start)
	{
	zfs_fuid_domain_t *zdomain;

	/* now copy in the domain info, if any */
	if (fuidp->z_domain_str_sz != 0) {
	for (zdomain = list_head(&fuidp->z_domains); zdomain;
	zdomain = list_next(&fuidp->z_domains, zdomain)) {
	bcopy((void *)zdomain->z_domain, start,
	strlen(zdomain->z_domain) + 1);
	start = (caddr_t)start +
	strlen(zdomain->z_domain) + 1;
	}
	}
	return (start);
	}

	/*
	* If zp is an xattr node, check whether the xattr owner is unlinked.
	* We don't want to log anything if the owner is unlinked.
	*/
	static int
	zfs_xattr_owner_unlinked(znode_t *zp)
	{
	int unlinked = 0;
	znode_t *dzp;
	#ifdef __FreeBSD__
	znode_t *tzp = zp;

	/*
	* zrele drops the vnode lock which violates the VOP locking contract
	* on FreeBSD. See comment at the top of zfs_replay.c for more detail.
	*/
	/*
	* if zp is XATTR node, keep walking up via z_xattr_parent until we
	* get the owner
	*/
	while (tzp->z_pflags & ZFS_XATTR) {
	ASSERT3U(zp->z_xattr_parent, !=, 0);
	if (zfs_zget(ZTOZSB(tzp), tzp->z_xattr_parent, &dzp) != 0) {
	unlinked = 1;
	break;
	}

	if (tzp != zp)
	zrele(tzp);
	tzp = dzp;
	unlinked = tzp->z_unlinked;
	}
	if (tzp != zp)
	zrele(tzp);
	#else
	zhold(zp);
	/*
	* if zp is XATTR node, keep walking up via z_xattr_parent until we
	* get the owner
	*/
	while (zp->z_pflags & ZFS_XATTR) {
	ASSERT3U(zp->z_xattr_parent, !=, 0);
	if (zfs_zget(ZTOZSB(zp), zp->z_xattr_parent, &dzp) != 0) {
	unlinked = 1;
	break;
	}

	zrele(zp);
	zp = dzp;
	unlinked = zp->z_unlinked;
	}
	zrele(zp);
	#endif
	return (unlinked);
	}

	/*
	* Handles TX_CREATE, TX_CREATE_ATTR, TX_MKDIR, TX_MKDIR_ATTR and
	* TK_MKXATTR transactions.
	*
	* TX_CREATE and TX_MKDIR are standard creates, but they may have FUID
	* domain information appended prior to the name. In this case the
	* uid/gid in the log record will be a log centric FUID.
	*
	* TX_CREATE_ACL_ATTR and TX_MKDIR_ACL_ATTR handle special creates that
	* may contain attributes, ACL and optional fuid information.
	*
	* TX_CREATE_ACL and TX_MKDIR_ACL handle special creates that specify
	* and ACL and normal users/groups in the ACEs.
	*
	* There may be an optional xvattr attribute information similar
	* to zfs_log_setattr.
	*
	* Also, after the file name "domain" strings may be appended.
	*/
	void
	zfs_log_create(zilog_t zilog, dmu_tx_t tx, uint64_t txtype,
	- znode_t dzp, znode_t zp, char name, vsecattr_t vsecp,
	+ znode_t dzp, znode_t zp, const char name, vsecattr_t vsecp,
	zfs_fuid_info_t fuidp, vattr_t vap)
	{
	itx_t *itx;
	lr_create_t *lr;
	lr_acl_create_t *lracl;
	size_t aclsize = 0;
	size_t xvatsize = 0;
	size_t txsize;
	xvattr_t xvap = (xvattr_t )vap;
	void *end;
	size_t lrsize;
	size_t namesize = strlen(name) + 1;
	size_t fuidsz = 0;

	if (zil_replaying(zilog, tx) \|\| zfs_xattr_owner_unlinked(dzp))
	return;

	/*
	* If we have FUIDs present then add in space for
	* domains and ACE fuid's if any.
	*/
	if (fuidp) {
	fuidsz += fuidp->z_domain_str_sz;
	fuidsz += fuidp->z_fuid_cnt * sizeof (uint64_t);
	}

	if (vap->va_mask & ATTR_XVATTR)
	xvatsize = ZIL_XVAT_SIZE(xvap->xva_mapsize);

	if ((int)txtype == TX_CREATE_ATTR \|\| (int)txtype == TX_MKDIR_ATTR \|\|
	(int)txtype == TX_CREATE \|\| (int)txtype == TX_MKDIR \|\|
	(int)txtype == TX_MKXATTR) {
	txsize = sizeof (*lr) + namesize + fuidsz + xvatsize;
	lrsize = sizeof (*lr);
	} else {
	txsize =
	sizeof (lr_acl_create_t) + namesize + fuidsz +
	ZIL_ACE_LENGTH(aclsize) + xvatsize;
	lrsize = sizeof (lr_acl_create_t);
	}

	itx = zil_itx_create(txtype, txsize);

	lr = (lr_create_t *)&itx->itx_lr;
	lr->lr_doid = dzp->z_id;
	lr->lr_foid = zp->z_id;
	/* Store dnode slot count in 8 bits above object id. */
	LR_FOID_SET_SLOTS(lr->lr_foid, zp->z_dnodesize >> DNODE_SHIFT);
	lr->lr_mode = zp->z_mode;
	if (!IS_EPHEMERAL(KUID_TO_SUID(ZTOUID(zp)))) {
	lr->lr_uid = (uint64_t)KUID_TO_SUID(ZTOUID(zp));
	} else {
	lr->lr_uid = fuidp->z_fuid_owner;
	}
	if (!IS_EPHEMERAL(KGID_TO_SGID(ZTOGID(zp)))) {
	lr->lr_gid = (uint64_t)KGID_TO_SGID(ZTOGID(zp));
	} else {
	lr->lr_gid = fuidp->z_fuid_group;
	}
	(void) sa_lookup(zp->z_sa_hdl, SA_ZPL_GEN(ZTOZSB(zp)), &lr->lr_gen,
	sizeof (uint64_t));
	(void) sa_lookup(zp->z_sa_hdl, SA_ZPL_CRTIME(ZTOZSB(zp)),
	lr->lr_crtime, sizeof (uint64_t) * 2);

	if (sa_lookup(zp->z_sa_hdl, SA_ZPL_RDEV(ZTOZSB(zp)), &lr->lr_rdev,
	sizeof (lr->lr_rdev)) != 0)
	lr->lr_rdev = 0;

	/*
	* Fill in xvattr info if any
	*/
	if (vap->va_mask & ATTR_XVATTR) {
	zfs_log_xvattr((lr_attr_t *)((caddr_t)lr + lrsize), xvap);
	end = (caddr_t)lr + lrsize + xvatsize;
	} else {
	end = (caddr_t)lr + lrsize;
	}

	/* Now fill in any ACL info */

	if (vsecp) {
	lracl = (lr_acl_create_t *)&itx->itx_lr;
	lracl->lr_aclcnt = vsecp->vsa_aclcnt;
	lracl->lr_acl_bytes = aclsize;
	lracl->lr_domcnt = fuidp ? fuidp->z_domain_cnt : 0;
	lracl->lr_fuidcnt = fuidp ? fuidp->z_fuid_cnt : 0;
	if (vsecp->vsa_aclflags & VSA_ACE_ACLFLAGS)
	lracl->lr_acl_flags = (uint64_t)vsecp->vsa_aclflags;
	else
	lracl->lr_acl_flags = 0;

	bcopy(vsecp->vsa_aclentp, end, aclsize);
	end = (caddr_t)end + ZIL_ACE_LENGTH(aclsize);
	}

	/* drop in FUID info */
	if (fuidp) {
	end = zfs_log_fuid_ids(fuidp, end);
	end = zfs_log_fuid_domains(fuidp, end);
	}
	/*
	* Now place file name in log record
	*/
	bcopy(name, end, namesize);

	zil_itx_assign(zilog, itx, tx);
	}

	/*
	* Handles both TX_REMOVE and TX_RMDIR transactions.
	*/
	void
	zfs_log_remove(zilog_t zilog, dmu_tx_t tx, uint64_t txtype,
	- znode_t dzp, char name, uint64_t foid, boolean_t unlinked)
	+ znode_t dzp, const char name, uint64_t foid, boolean_t unlinked)
	{
	itx_t *itx;
	lr_remove_t *lr;
	size_t namesize = strlen(name) + 1;

	if (zil_replaying(zilog, tx) \|\| zfs_xattr_owner_unlinked(dzp))
	return;

	itx = zil_itx_create(txtype, sizeof (*lr) + namesize);
	lr = (lr_remove_t *)&itx->itx_lr;
	lr->lr_doid = dzp->z_id;
	bcopy(name, (char *)(lr + 1), namesize);

	itx->itx_oid = foid;

	/*
	* Object ids can be re-instantiated in the next txg so
	* remove any async transactions to avoid future leaks.
	* This can happen if a fsync occurs on the re-instantiated
	* object for a WR_INDIRECT or WR_NEED_COPY write, which gets
	* the new file data and flushes a write record for the old object.
	*/
	if (unlinked) {
	ASSERT((txtype & ~TX_CI) == TX_REMOVE);
	zil_remove_async(zilog, foid);
	}
	zil_itx_assign(zilog, itx, tx);
	}

	/*
	* Handles TX_LINK transactions.
	*/
	void
	zfs_log_link(zilog_t zilog, dmu_tx_t tx, uint64_t txtype,
	- znode_t dzp, znode_t zp, char *name)
	+ znode_t dzp, znode_t zp, const char *name)
	{
	itx_t *itx;
	lr_link_t *lr;
	size_t namesize = strlen(name) + 1;

	if (zil_replaying(zilog, tx))
	return;

	itx = zil_itx_create(txtype, sizeof (*lr) + namesize);
	lr = (lr_link_t *)&itx->itx_lr;
	lr->lr_doid = dzp->z_id;
	lr->lr_link_obj = zp->z_id;
	bcopy(name, (char *)(lr + 1), namesize);

	zil_itx_assign(zilog, itx, tx);
	}

	/*
	* Handles TX_SYMLINK transactions.
	*/
	void
	zfs_log_symlink(zilog_t zilog, dmu_tx_t tx, uint64_t txtype,
	- znode_t dzp, znode_t zp, char name, char link)
	+ znode_t dzp, znode_t zp, const char name, const char link)
	{
	itx_t *itx;
	lr_create_t *lr;
	size_t namesize = strlen(name) + 1;
	size_t linksize = strlen(link) + 1;

	if (zil_replaying(zilog, tx))
	return;

	itx = zil_itx_create(txtype, sizeof (*lr) + namesize + linksize);
	lr = (lr_create_t *)&itx->itx_lr;
	lr->lr_doid = dzp->z_id;
	lr->lr_foid = zp->z_id;
	lr->lr_uid = KUID_TO_SUID(ZTOUID(zp));
	lr->lr_gid = KGID_TO_SGID(ZTOGID(zp));
	lr->lr_mode = zp->z_mode;
	(void) sa_lookup(zp->z_sa_hdl, SA_ZPL_GEN(ZTOZSB(zp)), &lr->lr_gen,
	sizeof (uint64_t));
	(void) sa_lookup(zp->z_sa_hdl, SA_ZPL_CRTIME(ZTOZSB(zp)),
	lr->lr_crtime, sizeof (uint64_t) * 2);
	bcopy(name, (char *)(lr + 1), namesize);
	bcopy(link, (char *)(lr + 1) + namesize, linksize);

	zil_itx_assign(zilog, itx, tx);
	}

	/*
	* Handles TX_RENAME transactions.
	*/
	void
	-zfs_log_rename(zilog_t zilog, dmu_tx_t tx, uint64_t txtype,
	- znode_t sdzp, char sname, znode_t tdzp, char dname, znode_t *szp)
	+zfs_log_rename(zilog_t zilog, dmu_tx_t tx, uint64_t txtype, znode_t *sdzp,
	+ const char sname, znode_t tdzp, const char dname, znode_t szp)
	{
	itx_t *itx;
	lr_rename_t *lr;
	size_t snamesize = strlen(sname) + 1;
	size_t dnamesize = strlen(dname) + 1;

	if (zil_replaying(zilog, tx))
	return;

	itx = zil_itx_create(txtype, sizeof (*lr) + snamesize + dnamesize);
	lr = (lr_rename_t *)&itx->itx_lr;
	lr->lr_sdoid = sdzp->z_id;
	lr->lr_tdoid = tdzp->z_id;
	bcopy(sname, (char *)(lr + 1), snamesize);
	bcopy(dname, (char *)(lr + 1) + snamesize, dnamesize);
	itx->itx_oid = szp->z_id;

	zil_itx_assign(zilog, itx, tx);
	}

	/*
	* zfs_log_write() handles TX_WRITE transactions. The specified callback is
	* called as soon as the write is on stable storage (be it via a DMU sync or a
	* ZIL commit).
	*/
	long zfs_immediate_write_sz = 32768;

	void
	zfs_log_write(zilog_t zilog, dmu_tx_t tx, int txtype,
	znode_t *zp, offset_t off, ssize_t resid, int ioflag,
	zil_callback_t callback, void *callback_data)
	{
	dmu_buf_impl_t db = (dmu_buf_impl_t )sa_get_db(zp->z_sa_hdl);
	uint32_t blocksize = zp->z_blksz;
	itx_wr_state_t write_state;
	uintptr_t fsync_cnt;

	if (zil_replaying(zilog, tx) \|\| zp->z_unlinked \|\|
	zfs_xattr_owner_unlinked(zp)) {
	if (callback != NULL)
	callback(callback_data);
	return;
	}

	if (zilog->zl_logbias == ZFS_LOGBIAS_THROUGHPUT)
	write_state = WR_INDIRECT;
	else if (!spa_has_slogs(zilog->zl_spa) &&
	resid >= zfs_immediate_write_sz)
	write_state = WR_INDIRECT;
	else if (ioflag & (O_SYNC \| O_DSYNC))
	write_state = WR_COPIED;
	else
	write_state = WR_NEED_COPY;

	if ((fsync_cnt = (uintptr_t)tsd_get(zfs_fsyncer_key)) != 0) {
	(void) tsd_set(zfs_fsyncer_key, (void *)(fsync_cnt - 1));
	}

	while (resid) {
	itx_t *itx;
	lr_write_t *lr;
	itx_wr_state_t wr_state = write_state;
	ssize_t len = resid;

	/*
	* A WR_COPIED record must fit entirely in one log block.
	* Large writes can use WR_NEED_COPY, which the ZIL will
	* split into multiple records across several log blocks
	* if necessary.
	*/
	if (wr_state == WR_COPIED &&
	resid > zil_max_copied_data(zilog))
	wr_state = WR_NEED_COPY;
	else if (wr_state == WR_INDIRECT)
	len = MIN(blocksize - P2PHASE(off, blocksize), resid);

	itx = zil_itx_create(txtype, sizeof (*lr) +
	(wr_state == WR_COPIED ? len : 0));
	lr = (lr_write_t *)&itx->itx_lr;

	/*
	* For WR_COPIED records, copy the data into the lr_write_t.
	*/
	if (wr_state == WR_COPIED) {
	int err;
	DB_DNODE_ENTER(db);
	err = dmu_read_by_dnode(DB_DNODE(db), off, len, lr + 1,
	DMU_READ_NO_PREFETCH);
	if (err != 0) {
	zil_itx_destroy(itx);
	itx = zil_itx_create(txtype, sizeof (*lr));
	lr = (lr_write_t *)&itx->itx_lr;
	wr_state = WR_NEED_COPY;
	}
	DB_DNODE_EXIT(db);
	}

	itx->itx_wr_state = wr_state;
	lr->lr_foid = zp->z_id;
	lr->lr_offset = off;
	lr->lr_length = len;
	lr->lr_blkoff = 0;
	BP_ZERO(&lr->lr_blkptr);

	itx->itx_private = ZTOZSB(zp);

	if (!(ioflag & (O_SYNC \| O_DSYNC)) && (zp->z_sync_cnt == 0) &&
	(fsync_cnt == 0))
	itx->itx_sync = B_FALSE;

	itx->itx_callback = callback;
	itx->itx_callback_data = callback_data;
	zil_itx_assign(zilog, itx, tx);

	off += len;
	resid -= len;
	}
	}

	/*
	* Handles TX_TRUNCATE transactions.
	*/
	void
	zfs_log_truncate(zilog_t zilog, dmu_tx_t tx, int txtype,
	znode_t *zp, uint64_t off, uint64_t len)
	{
	itx_t *itx;
	lr_truncate_t *lr;

	if (zil_replaying(zilog, tx) \|\| zp->z_unlinked \|\|
	zfs_xattr_owner_unlinked(zp))
	return;

	itx = zil_itx_create(txtype, sizeof (*lr));
	lr = (lr_truncate_t *)&itx->itx_lr;
	lr->lr_foid = zp->z_id;
	lr->lr_offset = off;
	lr->lr_length = len;

	itx->itx_sync = (zp->z_sync_cnt != 0);
	zil_itx_assign(zilog, itx, tx);
	}

	/*
	* Handles TX_SETATTR transactions.
	*/
	void
	zfs_log_setattr(zilog_t zilog, dmu_tx_t tx, int txtype,
	znode_t zp, vattr_t vap, uint_t mask_applied, zfs_fuid_info_t *fuidp)
	{
	itx_t *itx;
	lr_setattr_t *lr;
	xvattr_t xvap = (xvattr_t )vap;
	size_t recsize = sizeof (lr_setattr_t);
	void *start;

	if (zil_replaying(zilog, tx) \|\| zp->z_unlinked)
	return;

	/*
	* If XVATTR set, then log record size needs to allow
	* for lr_attr_t + xvattr mask, mapsize and create time
	* plus actual attribute values
	*/
	if (vap->va_mask & ATTR_XVATTR)
	recsize = sizeof (*lr) + ZIL_XVAT_SIZE(xvap->xva_mapsize);

	if (fuidp)
	recsize += fuidp->z_domain_str_sz;

	itx = zil_itx_create(txtype, recsize);
	lr = (lr_setattr_t *)&itx->itx_lr;
	lr->lr_foid = zp->z_id;
	lr->lr_mask = (uint64_t)mask_applied;
	lr->lr_mode = (uint64_t)vap->va_mode;
	if ((mask_applied & ATTR_UID) && IS_EPHEMERAL(vap->va_uid))
	lr->lr_uid = fuidp->z_fuid_owner;
	else
	lr->lr_uid = (uint64_t)vap->va_uid;

	if ((mask_applied & ATTR_GID) && IS_EPHEMERAL(vap->va_gid))
	lr->lr_gid = fuidp->z_fuid_group;
	else
	lr->lr_gid = (uint64_t)vap->va_gid;

	lr->lr_size = (uint64_t)vap->va_size;
	ZFS_TIME_ENCODE(&vap->va_atime, lr->lr_atime);
	ZFS_TIME_ENCODE(&vap->va_mtime, lr->lr_mtime);
	start = (lr_setattr_t *)(lr + 1);
	if (vap->va_mask & ATTR_XVATTR) {
	zfs_log_xvattr((lr_attr_t *)start, xvap);
	start = (caddr_t)start + ZIL_XVAT_SIZE(xvap->xva_mapsize);
	}

	/*
	* Now stick on domain information if any on end
	*/

	if (fuidp)
	(void) zfs_log_fuid_domains(fuidp, start);

	itx->itx_sync = (zp->z_sync_cnt != 0);
	zil_itx_assign(zilog, itx, tx);
	}

	/*
	* Handles TX_ACL transactions.
	*/
	void
	zfs_log_acl(zilog_t zilog, dmu_tx_t tx, znode_t *zp,
	vsecattr_t vsecp, zfs_fuid_info_t fuidp)
	{
	itx_t *itx;
	lr_acl_v0_t *lrv0;
	lr_acl_t *lr;
	int txtype;
	int lrsize;
	size_t txsize;
	size_t aclbytes = vsecp->vsa_aclentsz;

	if (zil_replaying(zilog, tx) \|\| zp->z_unlinked)
	return;

	txtype = (ZTOZSB(zp)->z_version < ZPL_VERSION_FUID) ?
	TX_ACL_V0 : TX_ACL;

	if (txtype == TX_ACL)
	lrsize = sizeof (*lr);
	else
	lrsize = sizeof (*lrv0);

	txsize = lrsize +
	((txtype == TX_ACL) ? ZIL_ACE_LENGTH(aclbytes) : aclbytes) +
	(fuidp ? fuidp->z_domain_str_sz : 0) +
	sizeof (uint64_t) * (fuidp ? fuidp->z_fuid_cnt : 0);

	itx = zil_itx_create(txtype, txsize);

	lr = (lr_acl_t *)&itx->itx_lr;
	lr->lr_foid = zp->z_id;
	if (txtype == TX_ACL) {
	lr->lr_acl_bytes = aclbytes;
	lr->lr_domcnt = fuidp ? fuidp->z_domain_cnt : 0;
	lr->lr_fuidcnt = fuidp ? fuidp->z_fuid_cnt : 0;
	if (vsecp->vsa_mask & VSA_ACE_ACLFLAGS)
	lr->lr_acl_flags = (uint64_t)vsecp->vsa_aclflags;
	else
	lr->lr_acl_flags = 0;
	}
	lr->lr_aclcnt = (uint64_t)vsecp->vsa_aclcnt;

	if (txtype == TX_ACL_V0) {
	lrv0 = (lr_acl_v0_t *)lr;
	bcopy(vsecp->vsa_aclentp, (ace_t *)(lrv0 + 1), aclbytes);
	} else {
	void start = (ace_t )(lr + 1);

	bcopy(vsecp->vsa_aclentp, start, aclbytes);

	start = (caddr_t)start + ZIL_ACE_LENGTH(aclbytes);

	if (fuidp) {
	start = zfs_log_fuid_ids(fuidp, start);
	(void) zfs_log_fuid_domains(fuidp, start);
	}
	}

	itx->itx_sync = (zp->z_sync_cnt != 0);
	zil_itx_assign(zilog, itx, tx);
	}

	/* BEGIN CSTYLED */
	ZFS_MODULE_PARAM(zfs, zfs_, immediate_write_sz, LONG, ZMOD_RW,
	"Largest data block to write to zil");
	/* END CSTYLED */
	Index: vendor-sys/openzfs/dist/module/zfs/zil.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/zfs/zil.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/zfs/zil.c (revision 366774)
	@@ -1,3688 +1,3695 @@
	/*
	* CDDL HEADER START
	*
	* The contents of this file are subject to the terms of the
	* Common Development and Distribution License (the "License").
	* You may not use this file except in compliance with the License.
	*
	* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	* or http://www.opensolaris.org/os/licensing.
	* See the License for the specific language governing permissions
	* and limitations under the License.
	*
	* When distributing Covered Code, include this CDDL HEADER in each
	* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	* If applicable, add the following below this CDDL HEADER, with the
	* fields enclosed by brackets "[]" replaced with your own identifying
	* information: Portions Copyright [yyyy] [name of copyright owner]
	*
	* CDDL HEADER END
	*/
	/*
	* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
	* Copyright (c) 2011, 2018 by Delphix. All rights reserved.
	* Copyright (c) 2014 Integros [integros.com]
	* Copyright (c) 2018 Datto Inc.
	*/

	/* Portions Copyright 2010 Robert Milkowski */

	#include <sys/zfs_context.h>
	#include <sys/spa.h>
	#include <sys/spa_impl.h>
	#include <sys/dmu.h>
	#include <sys/zap.h>
	#include <sys/arc.h>
	#include <sys/stat.h>
	#include <sys/zil.h>
	#include <sys/zil_impl.h>
	#include <sys/dsl_dataset.h>
	#include <sys/vdev_impl.h>
	#include <sys/dmu_tx.h>
	#include <sys/dsl_pool.h>
	#include <sys/metaslab.h>
	#include <sys/trace_zfs.h>
	#include <sys/abd.h>

	/*
	* The ZFS Intent Log (ZIL) saves "transaction records" (itxs) of system
	* calls that change the file system. Each itx has enough information to
	* be able to replay them after a system crash, power loss, or
	* equivalent failure mode. These are stored in memory until either:
	*
	* 1. they are committed to the pool by the DMU transaction group
	* (txg), at which point they can be discarded; or
	* 2. they are committed to the on-disk ZIL for the dataset being
	* modified (e.g. due to an fsync, O_DSYNC, or other synchronous
	* requirement).
	*
	* In the event of a crash or power loss, the itxs contained by each
	* dataset's on-disk ZIL will be replayed when that dataset is first
	* instantiated (e.g. if the dataset is a normal filesystem, when it is
	* first mounted).
	*
	* As hinted at above, there is one ZIL per dataset (both the in-memory
	* representation, and the on-disk representation). The on-disk format
	* consists of 3 parts:
	*
	* - a single, per-dataset, ZIL header; which points to a chain of
	* - zero or more ZIL blocks; each of which contains
	* - zero or more ZIL records
	*
	* A ZIL record holds the information necessary to replay a single
	* system call transaction. A ZIL block can hold many ZIL records, and
	* the blocks are chained together, similarly to a singly linked list.
	*
	* Each ZIL block contains a block pointer (blkptr_t) to the next ZIL
	* block in the chain, and the ZIL header points to the first block in
	* the chain.
	*
	* Note, there is not a fixed place in the pool to hold these ZIL
	* blocks; they are dynamically allocated and freed as needed from the
	* blocks available on the pool, though they can be preferentially
	* allocated from a dedicated "log" vdev.
	*/

	/*
	* This controls the amount of time that a ZIL block (lwb) will remain
	* "open" when it isn't "full", and it has a thread waiting for it to be
	* committed to stable storage. Please refer to the zil_commit_waiter()
	* function (and the comments within it) for more details.
	*/
	int zfs_commit_timeout_pct = 5;

	/*
	* See zil.h for more information about these fields.
	*/
	zil_stats_t zil_stats = {
	{ "zil_commit_count", KSTAT_DATA_UINT64 },
	{ "zil_commit_writer_count", KSTAT_DATA_UINT64 },
	{ "zil_itx_count", KSTAT_DATA_UINT64 },
	{ "zil_itx_indirect_count", KSTAT_DATA_UINT64 },
	{ "zil_itx_indirect_bytes", KSTAT_DATA_UINT64 },
	{ "zil_itx_copied_count", KSTAT_DATA_UINT64 },
	{ "zil_itx_copied_bytes", KSTAT_DATA_UINT64 },
	{ "zil_itx_needcopy_count", KSTAT_DATA_UINT64 },
	{ "zil_itx_needcopy_bytes", KSTAT_DATA_UINT64 },
	{ "zil_itx_metaslab_normal_count", KSTAT_DATA_UINT64 },
	{ "zil_itx_metaslab_normal_bytes", KSTAT_DATA_UINT64 },
	{ "zil_itx_metaslab_slog_count", KSTAT_DATA_UINT64 },
	{ "zil_itx_metaslab_slog_bytes", KSTAT_DATA_UINT64 },
	};

	static kstat_t *zil_ksp;

	/*
	* Disable intent logging replay. This global ZIL switch affects all pools.
	*/
	int zil_replay_disable = 0;

	/*
	* Disable the DKIOCFLUSHWRITECACHE commands that are normally sent to
	* the disk(s) by the ZIL after an LWB write has completed. Setting this
	* will cause ZIL corruption on power loss if a volatile out-of-order
	* write cache is enabled.
	*/
	int zil_nocacheflush = 0;

	/*
	* Limit SLOG write size per commit executed with synchronous priority.
	* Any writes above that will be executed with lower (asynchronous) priority
	* to limit potential SLOG device abuse by single active ZIL writer.
	*/
	unsigned long zil_slog_bulk = 768 * 1024;

	static kmem_cache_t *zil_lwb_cache;
	static kmem_cache_t *zil_zcw_cache;

	#define LWB_EMPTY(lwb) ((BP_GET_LSIZE(&lwb->lwb_blk) - \
	sizeof (zil_chain_t)) == (lwb->lwb_sz - lwb->lwb_nused))

	static int
	zil_bp_compare(const void x1, const void x2)
	{
	const dva_t dva1 = &((zil_bp_node_t )x1)->zn_dva;
	const dva_t dva2 = &((zil_bp_node_t )x2)->zn_dva;

	int cmp = TREE_CMP(DVA_GET_VDEV(dva1), DVA_GET_VDEV(dva2));
	if (likely(cmp))
	return (cmp);

	return (TREE_CMP(DVA_GET_OFFSET(dva1), DVA_GET_OFFSET(dva2)));
	}

	static void
	zil_bp_tree_init(zilog_t *zilog)
	{
	avl_create(&zilog->zl_bp_tree, zil_bp_compare,
	sizeof (zil_bp_node_t), offsetof(zil_bp_node_t, zn_node));
	}

	static void
	zil_bp_tree_fini(zilog_t *zilog)
	{
	avl_tree_t *t = &zilog->zl_bp_tree;
	zil_bp_node_t *zn;
	void *cookie = NULL;

	while ((zn = avl_destroy_nodes(t, &cookie)) != NULL)
	kmem_free(zn, sizeof (zil_bp_node_t));

	avl_destroy(t);
	}

	int
	zil_bp_tree_add(zilog_t zilog, const blkptr_t bp)
	{
	avl_tree_t *t = &zilog->zl_bp_tree;
	const dva_t *dva;
	zil_bp_node_t *zn;
	avl_index_t where;

	if (BP_IS_EMBEDDED(bp))
	return (0);

	dva = BP_IDENTITY(bp);

	if (avl_find(t, dva, &where) != NULL)
	return (SET_ERROR(EEXIST));

	zn = kmem_alloc(sizeof (zil_bp_node_t), KM_SLEEP);
	zn->zn_dva = *dva;
	avl_insert(t, zn, where);

	return (0);
	}

	static zil_header_t *
	zil_header_in_syncing_context(zilog_t *zilog)
	{
	return ((zil_header_t *)zilog->zl_header);
	}

	static void
	zil_init_log_chain(zilog_t zilog, blkptr_t bp)
	{
	zio_cksum_t *zc = &bp->blk_cksum;

	zc->zc_word[ZIL_ZC_GUID_0] = spa_get_random(-1ULL);
	zc->zc_word[ZIL_ZC_GUID_1] = spa_get_random(-1ULL);
	zc->zc_word[ZIL_ZC_OBJSET] = dmu_objset_id(zilog->zl_os);
	zc->zc_word[ZIL_ZC_SEQ] = 1ULL;
	}

	/*
	* Read a log block and make sure it's valid.
	*/
	static int
	zil_read_log_block(zilog_t zilog, boolean_t decrypt, const blkptr_t bp,
	blkptr_t nbp, void dst, char **end)
	{
	enum zio_flag zio_flags = ZIO_FLAG_CANFAIL;
	arc_flags_t aflags = ARC_FLAG_WAIT;
	arc_buf_t *abuf = NULL;
	zbookmark_phys_t zb;
	int error;

	if (zilog->zl_header->zh_claim_txg == 0)
	zio_flags \|= ZIO_FLAG_SPECULATIVE \| ZIO_FLAG_SCRUB;

	if (!(zilog->zl_header->zh_flags & ZIL_CLAIM_LR_SEQ_VALID))
	zio_flags \|= ZIO_FLAG_SPECULATIVE;

	if (!decrypt)
	zio_flags \|= ZIO_FLAG_RAW;

	SET_BOOKMARK(&zb, bp->blk_cksum.zc_word[ZIL_ZC_OBJSET],
	ZB_ZIL_OBJECT, ZB_ZIL_LEVEL, bp->blk_cksum.zc_word[ZIL_ZC_SEQ]);

	error = arc_read(NULL, zilog->zl_spa, bp, arc_getbuf_func,
	&abuf, ZIO_PRIORITY_SYNC_READ, zio_flags, &aflags, &zb);

	if (error == 0) {
	zio_cksum_t cksum = bp->blk_cksum;

	/*
	* Validate the checksummed log block.
	*
	* Sequence numbers should be... sequential. The checksum
	* verifier for the next block should be bp's checksum plus 1.
	*
	* Also check the log chain linkage and size used.
	*/
	cksum.zc_word[ZIL_ZC_SEQ]++;

	if (BP_GET_CHECKSUM(bp) == ZIO_CHECKSUM_ZILOG2) {
	zil_chain_t *zilc = abuf->b_data;
	char lr = (char )(zilc + 1);
	uint64_t len = zilc->zc_nused - sizeof (zil_chain_t);

	if (bcmp(&cksum, &zilc->zc_next_blk.blk_cksum,
	sizeof (cksum)) \|\| BP_IS_HOLE(&zilc->zc_next_blk)) {
	error = SET_ERROR(ECKSUM);
	} else {
	ASSERT3U(len, <=, SPA_OLD_MAXBLOCKSIZE);
	bcopy(lr, dst, len);
	end = (char )dst + len;
	*nbp = zilc->zc_next_blk;
	}
	} else {
	char *lr = abuf->b_data;
	uint64_t size = BP_GET_LSIZE(bp);
	zil_chain_t zilc = (zil_chain_t )(lr + size) - 1;

	if (bcmp(&cksum, &zilc->zc_next_blk.blk_cksum,
	sizeof (cksum)) \|\| BP_IS_HOLE(&zilc->zc_next_blk) \|\|
	(zilc->zc_nused > (size - sizeof (*zilc)))) {
	error = SET_ERROR(ECKSUM);
	} else {
	ASSERT3U(zilc->zc_nused, <=,
	SPA_OLD_MAXBLOCKSIZE);
	bcopy(lr, dst, zilc->zc_nused);
	end = (char )dst + zilc->zc_nused;
	*nbp = zilc->zc_next_blk;
	}
	}

	arc_buf_destroy(abuf, &abuf);
	}

	return (error);
	}

	/*
	* Read a TX_WRITE log data block.
	*/
	static int
	zil_read_log_data(zilog_t zilog, const lr_write_t lr, void *wbuf)
	{
	enum zio_flag zio_flags = ZIO_FLAG_CANFAIL;
	const blkptr_t *bp = &lr->lr_blkptr;
	arc_flags_t aflags = ARC_FLAG_WAIT;
	arc_buf_t *abuf = NULL;
	zbookmark_phys_t zb;
	int error;

	if (BP_IS_HOLE(bp)) {
	if (wbuf != NULL)
	bzero(wbuf, MAX(BP_GET_LSIZE(bp), lr->lr_length));
	return (0);
	}

	if (zilog->zl_header->zh_claim_txg == 0)
	zio_flags \|= ZIO_FLAG_SPECULATIVE \| ZIO_FLAG_SCRUB;

	/*
	* If we are not using the resulting data, we are just checking that
	* it hasn't been corrupted so we don't need to waste CPU time
	* decompressing and decrypting it.
	*/
	if (wbuf == NULL)
	zio_flags \|= ZIO_FLAG_RAW;

	SET_BOOKMARK(&zb, dmu_objset_id(zilog->zl_os), lr->lr_foid,
	ZB_ZIL_LEVEL, lr->lr_offset / BP_GET_LSIZE(bp));

	error = arc_read(NULL, zilog->zl_spa, bp, arc_getbuf_func, &abuf,
	ZIO_PRIORITY_SYNC_READ, zio_flags, &aflags, &zb);

	if (error == 0) {
	if (wbuf != NULL)
	bcopy(abuf->b_data, wbuf, arc_buf_size(abuf));
	arc_buf_destroy(abuf, &abuf);
	}

	return (error);
	}

	/*
	* Parse the intent log, and call parse_func for each valid record within.
	*/
	int
	zil_parse(zilog_t zilog, zil_parse_blk_func_t parse_blk_func,
	zil_parse_lr_func_t parse_lr_func, void arg, uint64_t txg,
	boolean_t decrypt)
	{
	const zil_header_t *zh = zilog->zl_header;
	boolean_t claimed = !!zh->zh_claim_txg;
	uint64_t claim_blk_seq = claimed ? zh->zh_claim_blk_seq : UINT64_MAX;
	uint64_t claim_lr_seq = claimed ? zh->zh_claim_lr_seq : UINT64_MAX;
	uint64_t max_blk_seq = 0;
	uint64_t max_lr_seq = 0;
	uint64_t blk_count = 0;
	uint64_t lr_count = 0;
	blkptr_t blk, next_blk;
	char lrbuf, lrp;
	int error = 0;

	bzero(&next_blk, sizeof (blkptr_t));

	/*
	* Old logs didn't record the maximum zh_claim_lr_seq.
	*/
	if (!(zh->zh_flags & ZIL_CLAIM_LR_SEQ_VALID))
	claim_lr_seq = UINT64_MAX;

	/*
	* Starting at the block pointed to by zh_log we read the log chain.
	* For each block in the chain we strongly check that block to
	* ensure its validity. We stop when an invalid block is found.
	* For each block pointer in the chain we call parse_blk_func().
	* For each record in each valid block we call parse_lr_func().
	* If the log has been claimed, stop if we encounter a sequence
	* number greater than the highest claimed sequence number.
	*/
	lrbuf = zio_buf_alloc(SPA_OLD_MAXBLOCKSIZE);
	zil_bp_tree_init(zilog);

	for (blk = zh->zh_log; !BP_IS_HOLE(&blk); blk = next_blk) {
	uint64_t blk_seq = blk.blk_cksum.zc_word[ZIL_ZC_SEQ];
	int reclen;
	char *end = NULL;

	if (blk_seq > claim_blk_seq)
	break;

	error = parse_blk_func(zilog, &blk, arg, txg);
	if (error != 0)
	break;
	ASSERT3U(max_blk_seq, <, blk_seq);
	max_blk_seq = blk_seq;
	blk_count++;

	if (max_lr_seq == claim_lr_seq && max_blk_seq == claim_blk_seq)
	break;

	error = zil_read_log_block(zilog, decrypt, &blk, &next_blk,
	lrbuf, &end);
	if (error != 0)
	break;

	for (lrp = lrbuf; lrp < end; lrp += reclen) {
	lr_t lr = (lr_t )lrp;
	reclen = lr->lrc_reclen;
	ASSERT3U(reclen, >=, sizeof (lr_t));
	if (lr->lrc_seq > claim_lr_seq)
	goto done;

	error = parse_lr_func(zilog, lr, arg, txg);
	if (error != 0)
	goto done;
	ASSERT3U(max_lr_seq, <, lr->lrc_seq);
	max_lr_seq = lr->lrc_seq;
	lr_count++;
	}
	}
	done:
	zilog->zl_parse_error = error;
	zilog->zl_parse_blk_seq = max_blk_seq;
	zilog->zl_parse_lr_seq = max_lr_seq;
	zilog->zl_parse_blk_count = blk_count;
	zilog->zl_parse_lr_count = lr_count;

	ASSERT(!claimed \|\| !(zh->zh_flags & ZIL_CLAIM_LR_SEQ_VALID) \|\|
	(max_blk_seq == claim_blk_seq && max_lr_seq == claim_lr_seq) \|\|
	(decrypt && error == EIO));

	zil_bp_tree_fini(zilog);
	zio_buf_free(lrbuf, SPA_OLD_MAXBLOCKSIZE);

	return (error);
	}

	/* ARGSUSED */
	static int
	-zil_clear_log_block(zilog_t zilog, blkptr_t bp, void *tx, uint64_t first_txg)
	+zil_clear_log_block(zilog_t zilog, const blkptr_t bp, void *tx,
	+ uint64_t first_txg)
	{
	ASSERT(!BP_IS_HOLE(bp));

	/*
	* As we call this function from the context of a rewind to a
	* checkpoint, each ZIL block whose txg is later than the txg
	* that we rewind to is invalid. Thus, we return -1 so
	* zil_parse() doesn't attempt to read it.
	*/
	if (bp->blk_birth >= first_txg)
	return (-1);

	if (zil_bp_tree_add(zilog, bp) != 0)
	return (0);

	zio_free(zilog->zl_spa, first_txg, bp);
	return (0);
	}

	/* ARGSUSED */
	static int
	-zil_noop_log_record(zilog_t zilog, lr_t lrc, void *tx, uint64_t first_txg)
	+zil_noop_log_record(zilog_t zilog, const lr_t lrc, void *tx,
	+ uint64_t first_txg)
	{
	return (0);
	}

	static int
	-zil_claim_log_block(zilog_t zilog, blkptr_t bp, void *tx, uint64_t first_txg)
	+zil_claim_log_block(zilog_t zilog, const blkptr_t bp, void *tx,
	+ uint64_t first_txg)
	{
	/*
	* Claim log block if not already committed and not already claimed.
	* If tx == NULL, just verify that the block is claimable.
	*/
	if (BP_IS_HOLE(bp) \|\| bp->blk_birth < first_txg \|\|
	zil_bp_tree_add(zilog, bp) != 0)
	return (0);

	return (zio_wait(zio_claim(NULL, zilog->zl_spa,
	tx == NULL ? 0 : first_txg, bp, spa_claim_notify, NULL,
	ZIO_FLAG_CANFAIL \| ZIO_FLAG_SPECULATIVE \| ZIO_FLAG_SCRUB)));
	}

	static int
	-zil_claim_log_record(zilog_t zilog, lr_t lrc, void *tx, uint64_t first_txg)
	+zil_claim_log_record(zilog_t zilog, const lr_t lrc, void *tx,
	+ uint64_t first_txg)
	{
	lr_write_t lr = (lr_write_t )lrc;
	int error;

	if (lrc->lrc_txtype != TX_WRITE)
	return (0);

	/*
	* If the block is not readable, don't claim it. This can happen
	* in normal operation when a log block is written to disk before
	* some of the dmu_sync() blocks it points to. In this case, the
	* transaction cannot have been committed to anyone (we would have
	* waited for all writes to be stable first), so it is semantically
	* correct to declare this the end of the log.
	*/
	if (lr->lr_blkptr.blk_birth >= first_txg) {
	error = zil_read_log_data(zilog, lr, NULL);
	if (error != 0)
	return (error);
	}

	return (zil_claim_log_block(zilog, &lr->lr_blkptr, tx, first_txg));
	}

	/* ARGSUSED */
	static int
	-zil_free_log_block(zilog_t zilog, blkptr_t bp, void *tx, uint64_t claim_txg)
	+zil_free_log_block(zilog_t zilog, const blkptr_t bp, void *tx,
	+ uint64_t claim_txg)
	{
	zio_free(zilog->zl_spa, dmu_tx_get_txg(tx), bp);

	return (0);
	}

	static int
	-zil_free_log_record(zilog_t zilog, lr_t lrc, void *tx, uint64_t claim_txg)
	+zil_free_log_record(zilog_t zilog, const lr_t lrc, void *tx,
	+ uint64_t claim_txg)
	{
	lr_write_t lr = (lr_write_t )lrc;
	blkptr_t *bp = &lr->lr_blkptr;

	/*
	* If we previously claimed it, we need to free it.
	*/
	if (claim_txg != 0 && lrc->lrc_txtype == TX_WRITE &&
	bp->blk_birth >= claim_txg && zil_bp_tree_add(zilog, bp) == 0 &&
	!BP_IS_HOLE(bp))
	zio_free(zilog->zl_spa, dmu_tx_get_txg(tx), bp);

	return (0);
	}

	static int
	zil_lwb_vdev_compare(const void x1, const void x2)
	{
	const uint64_t v1 = ((zil_vdev_node_t *)x1)->zv_vdev;
	const uint64_t v2 = ((zil_vdev_node_t *)x2)->zv_vdev;

	return (TREE_CMP(v1, v2));
	}

	static lwb_t *
	zil_alloc_lwb(zilog_t zilog, blkptr_t bp, boolean_t slog, uint64_t txg,
	boolean_t fastwrite)
	{
	lwb_t *lwb;

	lwb = kmem_cache_alloc(zil_lwb_cache, KM_SLEEP);
	lwb->lwb_zilog = zilog;
	lwb->lwb_blk = *bp;
	lwb->lwb_fastwrite = fastwrite;
	lwb->lwb_slog = slog;
	lwb->lwb_state = LWB_STATE_CLOSED;
	lwb->lwb_buf = zio_buf_alloc(BP_GET_LSIZE(bp));
	lwb->lwb_max_txg = txg;
	lwb->lwb_write_zio = NULL;
	lwb->lwb_root_zio = NULL;
	lwb->lwb_tx = NULL;
	lwb->lwb_issued_timestamp = 0;
	if (BP_GET_CHECKSUM(bp) == ZIO_CHECKSUM_ZILOG2) {
	lwb->lwb_nused = sizeof (zil_chain_t);
	lwb->lwb_sz = BP_GET_LSIZE(bp);
	} else {
	lwb->lwb_nused = 0;
	lwb->lwb_sz = BP_GET_LSIZE(bp) - sizeof (zil_chain_t);
	}

	mutex_enter(&zilog->zl_lock);
	list_insert_tail(&zilog->zl_lwb_list, lwb);
	mutex_exit(&zilog->zl_lock);

	ASSERT(!MUTEX_HELD(&lwb->lwb_vdev_lock));
	ASSERT(avl_is_empty(&lwb->lwb_vdev_tree));
	VERIFY(list_is_empty(&lwb->lwb_waiters));
	VERIFY(list_is_empty(&lwb->lwb_itxs));

	return (lwb);
	}

	static void
	zil_free_lwb(zilog_t zilog, lwb_t lwb)
	{
	ASSERT(MUTEX_HELD(&zilog->zl_lock));
	ASSERT(!MUTEX_HELD(&lwb->lwb_vdev_lock));
	VERIFY(list_is_empty(&lwb->lwb_waiters));
	VERIFY(list_is_empty(&lwb->lwb_itxs));
	ASSERT(avl_is_empty(&lwb->lwb_vdev_tree));
	ASSERT3P(lwb->lwb_write_zio, ==, NULL);
	ASSERT3P(lwb->lwb_root_zio, ==, NULL);
	ASSERT3U(lwb->lwb_max_txg, <=, spa_syncing_txg(zilog->zl_spa));
	ASSERT(lwb->lwb_state == LWB_STATE_CLOSED \|\|
	lwb->lwb_state == LWB_STATE_FLUSH_DONE);

	/*
	* Clear the zilog's field to indicate this lwb is no longer
	* valid, and prevent use-after-free errors.
	*/
	if (zilog->zl_last_lwb_opened == lwb)
	zilog->zl_last_lwb_opened = NULL;

	kmem_cache_free(zil_lwb_cache, lwb);
	}

	/*
	* Called when we create in-memory log transactions so that we know
	* to cleanup the itxs at the end of spa_sync().
	*/
	static void
	zilog_dirty(zilog_t *zilog, uint64_t txg)
	{
	dsl_pool_t *dp = zilog->zl_dmu_pool;
	dsl_dataset_t *ds = dmu_objset_ds(zilog->zl_os);

	ASSERT(spa_writeable(zilog->zl_spa));

	if (ds->ds_is_snapshot)
	panic("dirtying snapshot!");

	if (txg_list_add(&dp->dp_dirty_zilogs, zilog, txg)) {
	/* up the hold count until we can be written out */
	dmu_buf_add_ref(ds->ds_dbuf, zilog);

	zilog->zl_dirty_max_txg = MAX(txg, zilog->zl_dirty_max_txg);
	}
	}

	/*
	* Determine if the zil is dirty in the specified txg. Callers wanting to
	* ensure that the dirty state does not change must hold the itxg_lock for
	* the specified txg. Holding the lock will ensure that the zil cannot be
	* dirtied (zil_itx_assign) or cleaned (zil_clean) while we check its current
	* state.
	*/
	static boolean_t __maybe_unused
	zilog_is_dirty_in_txg(zilog_t *zilog, uint64_t txg)
	{
	dsl_pool_t *dp = zilog->zl_dmu_pool;

	if (txg_list_member(&dp->dp_dirty_zilogs, zilog, txg & TXG_MASK))
	return (B_TRUE);
	return (B_FALSE);
	}

	/*
	* Determine if the zil is dirty. The zil is considered dirty if it has
	* any pending itx records that have not been cleaned by zil_clean().
	*/
	static boolean_t
	zilog_is_dirty(zilog_t *zilog)
	{
	dsl_pool_t *dp = zilog->zl_dmu_pool;

	for (int t = 0; t < TXG_SIZE; t++) {
	if (txg_list_member(&dp->dp_dirty_zilogs, zilog, t))
	return (B_TRUE);
	}
	return (B_FALSE);
	}

	/*
	* Create an on-disk intent log.
	*/
	static lwb_t *
	zil_create(zilog_t *zilog)
	{
	const zil_header_t *zh = zilog->zl_header;
	lwb_t *lwb = NULL;
	uint64_t txg = 0;
	dmu_tx_t *tx = NULL;
	blkptr_t blk;
	int error = 0;
	boolean_t fastwrite = FALSE;
	boolean_t slog = FALSE;

	/*
	* Wait for any previous destroy to complete.
	*/
	txg_wait_synced(zilog->zl_dmu_pool, zilog->zl_destroy_txg);

	ASSERT(zh->zh_claim_txg == 0);
	ASSERT(zh->zh_replay_seq == 0);

	blk = zh->zh_log;

	/*
	* Allocate an initial log block if:
	* - there isn't one already
	* - the existing block is the wrong endianness
	*/
	if (BP_IS_HOLE(&blk) \|\| BP_SHOULD_BYTESWAP(&blk)) {
	tx = dmu_tx_create(zilog->zl_os);
	VERIFY0(dmu_tx_assign(tx, TXG_WAIT));
	dsl_dataset_dirty(dmu_objset_ds(zilog->zl_os), tx);
	txg = dmu_tx_get_txg(tx);

	if (!BP_IS_HOLE(&blk)) {
	zio_free(zilog->zl_spa, txg, &blk);
	BP_ZERO(&blk);
	}

	error = zio_alloc_zil(zilog->zl_spa, zilog->zl_os, txg, &blk,
	ZIL_MIN_BLKSZ, &slog);
	fastwrite = TRUE;

	if (error == 0)
	zil_init_log_chain(zilog, &blk);
	}

	/*
	* Allocate a log write block (lwb) for the first log block.
	*/
	if (error == 0)
	lwb = zil_alloc_lwb(zilog, &blk, slog, txg, fastwrite);

	/*
	* If we just allocated the first log block, commit our transaction
	* and wait for zil_sync() to stuff the block pointer into zh_log.
	* (zh is part of the MOS, so we cannot modify it in open context.)
	*/
	if (tx != NULL) {
	dmu_tx_commit(tx);
	txg_wait_synced(zilog->zl_dmu_pool, txg);
	}

	ASSERT(error != 0 \|\| bcmp(&blk, &zh->zh_log, sizeof (blk)) == 0);
	IMPLY(error == 0, lwb != NULL);

	return (lwb);
	}

	/*
	* In one tx, free all log blocks and clear the log header. If keep_first
	* is set, then we're replaying a log with no content. We want to keep the
	* first block, however, so that the first synchronous transaction doesn't
	* require a txg_wait_synced() in zil_create(). We don't need to
	* txg_wait_synced() here either when keep_first is set, because both
	* zil_create() and zil_destroy() will wait for any in-progress destroys
	* to complete.
	*/
	void
	zil_destroy(zilog_t *zilog, boolean_t keep_first)
	{
	const zil_header_t *zh = zilog->zl_header;
	lwb_t *lwb;
	dmu_tx_t *tx;
	uint64_t txg;

	/*
	* Wait for any previous destroy to complete.
	*/
	txg_wait_synced(zilog->zl_dmu_pool, zilog->zl_destroy_txg);

	zilog->zl_old_header = zh; / debugging aid */

	if (BP_IS_HOLE(&zh->zh_log))
	return;

	tx = dmu_tx_create(zilog->zl_os);
	VERIFY0(dmu_tx_assign(tx, TXG_WAIT));
	dsl_dataset_dirty(dmu_objset_ds(zilog->zl_os), tx);
	txg = dmu_tx_get_txg(tx);

	mutex_enter(&zilog->zl_lock);

	ASSERT3U(zilog->zl_destroy_txg, <, txg);
	zilog->zl_destroy_txg = txg;
	zilog->zl_keep_first = keep_first;

	if (!list_is_empty(&zilog->zl_lwb_list)) {
	ASSERT(zh->zh_claim_txg == 0);
	VERIFY(!keep_first);
	while ((lwb = list_head(&zilog->zl_lwb_list)) != NULL) {
	if (lwb->lwb_fastwrite)
	metaslab_fastwrite_unmark(zilog->zl_spa,
	&lwb->lwb_blk);

	list_remove(&zilog->zl_lwb_list, lwb);
	if (lwb->lwb_buf != NULL)
	zio_buf_free(lwb->lwb_buf, lwb->lwb_sz);
	zio_free(zilog->zl_spa, txg, &lwb->lwb_blk);
	zil_free_lwb(zilog, lwb);
	}
	} else if (!keep_first) {
	zil_destroy_sync(zilog, tx);
	}
	mutex_exit(&zilog->zl_lock);

	dmu_tx_commit(tx);
	}

	void
	zil_destroy_sync(zilog_t zilog, dmu_tx_t tx)
	{
	ASSERT(list_is_empty(&zilog->zl_lwb_list));
	(void) zil_parse(zilog, zil_free_log_block,
	zil_free_log_record, tx, zilog->zl_header->zh_claim_txg, B_FALSE);
	}

	int
	zil_claim(dsl_pool_t dp, dsl_dataset_t ds, void *txarg)
	{
	dmu_tx_t *tx = txarg;
	zilog_t *zilog;
	uint64_t first_txg;
	zil_header_t *zh;
	objset_t *os;
	int error;

	error = dmu_objset_own_obj(dp, ds->ds_object,
	DMU_OST_ANY, B_FALSE, B_FALSE, FTAG, &os);
	if (error != 0) {
	/*
	* EBUSY indicates that the objset is inconsistent, in which
	* case it can not have a ZIL.
	*/
	if (error != EBUSY) {
	cmn_err(CE_WARN, "can't open objset for %llu, error %u",
	(unsigned long long)ds->ds_object, error);
	}

	return (0);
	}

	zilog = dmu_objset_zil(os);
	zh = zil_header_in_syncing_context(zilog);
	ASSERT3U(tx->tx_txg, ==, spa_first_txg(zilog->zl_spa));
	first_txg = spa_min_claim_txg(zilog->zl_spa);

	/*
	* If the spa_log_state is not set to be cleared, check whether
	* the current uberblock is a checkpoint one and if the current
	* header has been claimed before moving on.
	*
	* If the current uberblock is a checkpointed uberblock then
	* one of the following scenarios took place:
	*
	* 1] We are currently rewinding to the checkpoint of the pool.
	* 2] We crashed in the middle of a checkpoint rewind but we
	* did manage to write the checkpointed uberblock to the
	* vdev labels, so when we tried to import the pool again
	* the checkpointed uberblock was selected from the import
	* procedure.
	*
	* In both cases we want to zero out all the ZIL blocks, except
	* the ones that have been claimed at the time of the checkpoint
	* (their zh_claim_txg != 0). The reason is that these blocks
	* may be corrupted since we may have reused their locations on
	* disk after we took the checkpoint.
	*
	* We could try to set spa_log_state to SPA_LOG_CLEAR earlier
	* when we first figure out whether the current uberblock is
	* checkpointed or not. Unfortunately, that would discard all
	* the logs, including the ones that are claimed, and we would
	* leak space.
	*/
	if (spa_get_log_state(zilog->zl_spa) == SPA_LOG_CLEAR \|\|
	(zilog->zl_spa->spa_uberblock.ub_checkpoint_txg != 0 &&
	zh->zh_claim_txg == 0)) {
	if (!BP_IS_HOLE(&zh->zh_log)) {
	(void) zil_parse(zilog, zil_clear_log_block,
	zil_noop_log_record, tx, first_txg, B_FALSE);
	}
	BP_ZERO(&zh->zh_log);
	if (os->os_encrypted)
	os->os_next_write_raw[tx->tx_txg & TXG_MASK] = B_TRUE;
	dsl_dataset_dirty(dmu_objset_ds(os), tx);
	dmu_objset_disown(os, B_FALSE, FTAG);
	return (0);
	}

	/*
	* If we are not rewinding and opening the pool normally, then
	* the min_claim_txg should be equal to the first txg of the pool.
	*/
	ASSERT3U(first_txg, ==, spa_first_txg(zilog->zl_spa));

	/*
	* Claim all log blocks if we haven't already done so, and remember
	* the highest claimed sequence number. This ensures that if we can
	* read only part of the log now (e.g. due to a missing device),
	* but we can read the entire log later, we will not try to replay
	* or destroy beyond the last block we successfully claimed.
	*/
	ASSERT3U(zh->zh_claim_txg, <=, first_txg);
	if (zh->zh_claim_txg == 0 && !BP_IS_HOLE(&zh->zh_log)) {
	(void) zil_parse(zilog, zil_claim_log_block,
	zil_claim_log_record, tx, first_txg, B_FALSE);
	zh->zh_claim_txg = first_txg;
	zh->zh_claim_blk_seq = zilog->zl_parse_blk_seq;
	zh->zh_claim_lr_seq = zilog->zl_parse_lr_seq;
	if (zilog->zl_parse_lr_count \|\| zilog->zl_parse_blk_count > 1)
	zh->zh_flags \|= ZIL_REPLAY_NEEDED;
	zh->zh_flags \|= ZIL_CLAIM_LR_SEQ_VALID;
	if (os->os_encrypted)
	os->os_next_write_raw[tx->tx_txg & TXG_MASK] = B_TRUE;
	dsl_dataset_dirty(dmu_objset_ds(os), tx);
	}

	ASSERT3U(first_txg, ==, (spa_last_synced_txg(zilog->zl_spa) + 1));
	dmu_objset_disown(os, B_FALSE, FTAG);
	return (0);
	}

	/*
	* Check the log by walking the log chain.
	* Checksum errors are ok as they indicate the end of the chain.
	* Any other error (no device or read failure) returns an error.
	*/
	/* ARGSUSED */
	int
	zil_check_log_chain(dsl_pool_t dp, dsl_dataset_t ds, void *tx)
	{
	zilog_t *zilog;
	objset_t *os;
	blkptr_t *bp;
	int error;

	ASSERT(tx == NULL);

	error = dmu_objset_from_ds(ds, &os);
	if (error != 0) {
	cmn_err(CE_WARN, "can't open objset %llu, error %d",
	(unsigned long long)ds->ds_object, error);
	return (0);
	}

	zilog = dmu_objset_zil(os);
	bp = (blkptr_t *)&zilog->zl_header->zh_log;

	if (!BP_IS_HOLE(bp)) {
	vdev_t *vd;
	boolean_t valid = B_TRUE;

	/*
	* Check the first block and determine if it's on a log device
	* which may have been removed or faulted prior to loading this
	* pool. If so, there's no point in checking the rest of the
	* log as its content should have already been synced to the
	* pool.
	*/
	spa_config_enter(os->os_spa, SCL_STATE, FTAG, RW_READER);
	vd = vdev_lookup_top(os->os_spa, DVA_GET_VDEV(&bp->blk_dva[0]));
	if (vd->vdev_islog && vdev_is_dead(vd))
	valid = vdev_log_state_valid(vd);
	spa_config_exit(os->os_spa, SCL_STATE, FTAG);

	if (!valid)
	return (0);

	/*
	* Check whether the current uberblock is checkpointed (e.g.
	* we are rewinding) and whether the current header has been
	* claimed or not. If it hasn't then skip verifying it. We
	* do this because its ZIL blocks may be part of the pool's
	* state before the rewind, which is no longer valid.
	*/
	zil_header_t *zh = zil_header_in_syncing_context(zilog);
	if (zilog->zl_spa->spa_uberblock.ub_checkpoint_txg != 0 &&
	zh->zh_claim_txg == 0)
	return (0);
	}

	/*
	* Because tx == NULL, zil_claim_log_block() will not actually claim
	* any blocks, but just determine whether it is possible to do so.
	* In addition to checking the log chain, zil_claim_log_block()
	* will invoke zio_claim() with a done func of spa_claim_notify(),
	* which will update spa_max_claim_txg. See spa_load() for details.
	*/
	error = zil_parse(zilog, zil_claim_log_block, zil_claim_log_record, tx,
	zilog->zl_header->zh_claim_txg ? -1ULL :
	spa_min_claim_txg(os->os_spa), B_FALSE);

	return ((error == ECKSUM \|\| error == ENOENT) ? 0 : error);
	}

	/*
	* When an itx is "skipped", this function is used to properly mark the
	* waiter as "done, and signal any thread(s) waiting on it. An itx can
	* be skipped (and not committed to an lwb) for a variety of reasons,
	* one of them being that the itx was committed via spa_sync(), prior to
	* it being committed to an lwb; this can happen if a thread calling
	* zil_commit() is racing with spa_sync().
	*/
	static void
	zil_commit_waiter_skip(zil_commit_waiter_t *zcw)
	{
	mutex_enter(&zcw->zcw_lock);
	ASSERT3B(zcw->zcw_done, ==, B_FALSE);
	zcw->zcw_done = B_TRUE;
	cv_broadcast(&zcw->zcw_cv);
	mutex_exit(&zcw->zcw_lock);
	}

	/*
	* This function is used when the given waiter is to be linked into an
	* lwb's "lwb_waiter" list; i.e. when the itx is committed to the lwb.
	* At this point, the waiter will no longer be referenced by the itx,
	* and instead, will be referenced by the lwb.
	*/
	static void
	zil_commit_waiter_link_lwb(zil_commit_waiter_t zcw, lwb_t lwb)
	{
	/*
	* The lwb_waiters field of the lwb is protected by the zilog's
	* zl_lock, thus it must be held when calling this function.
	*/
	ASSERT(MUTEX_HELD(&lwb->lwb_zilog->zl_lock));

	mutex_enter(&zcw->zcw_lock);
	ASSERT(!list_link_active(&zcw->zcw_node));
	ASSERT3P(zcw->zcw_lwb, ==, NULL);
	ASSERT3P(lwb, !=, NULL);
	ASSERT(lwb->lwb_state == LWB_STATE_OPENED \|\|
	lwb->lwb_state == LWB_STATE_ISSUED \|\|
	lwb->lwb_state == LWB_STATE_WRITE_DONE);

	list_insert_tail(&lwb->lwb_waiters, zcw);
	zcw->zcw_lwb = lwb;
	mutex_exit(&zcw->zcw_lock);
	}

	/*
	* This function is used when zio_alloc_zil() fails to allocate a ZIL
	* block, and the given waiter must be linked to the "nolwb waiters"
	* list inside of zil_process_commit_list().
	*/
	static void
	zil_commit_waiter_link_nolwb(zil_commit_waiter_t zcw, list_t nolwb)
	{
	mutex_enter(&zcw->zcw_lock);
	ASSERT(!list_link_active(&zcw->zcw_node));
	ASSERT3P(zcw->zcw_lwb, ==, NULL);
	list_insert_tail(nolwb, zcw);
	mutex_exit(&zcw->zcw_lock);
	}

	void
	zil_lwb_add_block(lwb_t lwb, const blkptr_t bp)
	{
	avl_tree_t *t = &lwb->lwb_vdev_tree;
	avl_index_t where;
	zil_vdev_node_t *zv, zvsearch;
	int ndvas = BP_GET_NDVAS(bp);
	int i;

	if (zil_nocacheflush)
	return;

	mutex_enter(&lwb->lwb_vdev_lock);
	for (i = 0; i < ndvas; i++) {
	zvsearch.zv_vdev = DVA_GET_VDEV(&bp->blk_dva[i]);
	if (avl_find(t, &zvsearch, &where) == NULL) {
	zv = kmem_alloc(sizeof (*zv), KM_SLEEP);
	zv->zv_vdev = zvsearch.zv_vdev;
	avl_insert(t, zv, where);
	}
	}
	mutex_exit(&lwb->lwb_vdev_lock);
	}

	static void
	zil_lwb_flush_defer(lwb_t lwb, lwb_t nlwb)
	{
	avl_tree_t *src = &lwb->lwb_vdev_tree;
	avl_tree_t *dst = &nlwb->lwb_vdev_tree;
	void *cookie = NULL;
	zil_vdev_node_t *zv;

	ASSERT3S(lwb->lwb_state, ==, LWB_STATE_WRITE_DONE);
	ASSERT3S(nlwb->lwb_state, !=, LWB_STATE_WRITE_DONE);
	ASSERT3S(nlwb->lwb_state, !=, LWB_STATE_FLUSH_DONE);

	/*
	* While 'lwb' is at a point in its lifetime where lwb_vdev_tree does
	* not need the protection of lwb_vdev_lock (it will only be modified
	* while holding zilog->zl_lock) as its writes and those of its
	* children have all completed. The younger 'nlwb' may be waiting on
	* future writes to additional vdevs.
	*/
	mutex_enter(&nlwb->lwb_vdev_lock);
	/*
	* Tear down the 'lwb' vdev tree, ensuring that entries which do not
	* exist in 'nlwb' are moved to it, freeing any would-be duplicates.
	*/
	while ((zv = avl_destroy_nodes(src, &cookie)) != NULL) {
	avl_index_t where;

	if (avl_find(dst, zv, &where) == NULL) {
	avl_insert(dst, zv, where);
	} else {
	kmem_free(zv, sizeof (*zv));
	}
	}
	mutex_exit(&nlwb->lwb_vdev_lock);
	}

	void
	zil_lwb_add_txg(lwb_t *lwb, uint64_t txg)
	{
	lwb->lwb_max_txg = MAX(lwb->lwb_max_txg, txg);
	}

	/*
	* This function is a called after all vdevs associated with a given lwb
	* write have completed their DKIOCFLUSHWRITECACHE command; or as soon
	* as the lwb write completes, if "zil_nocacheflush" is set. Further,
	* all "previous" lwb's will have completed before this function is
	* called; i.e. this function is called for all previous lwbs before
	* it's called for "this" lwb (enforced via zio the dependencies
	* configured in zil_lwb_set_zio_dependency()).
	*
	* The intention is for this function to be called as soon as the
	* contents of an lwb are considered "stable" on disk, and will survive
	* any sudden loss of power. At this point, any threads waiting for the
	* lwb to reach this state are signalled, and the "waiter" structures
	* are marked "done".
	*/
	static void
	zil_lwb_flush_vdevs_done(zio_t *zio)
	{
	lwb_t *lwb = zio->io_private;
	zilog_t *zilog = lwb->lwb_zilog;
	dmu_tx_t *tx = lwb->lwb_tx;
	zil_commit_waiter_t *zcw;
	itx_t *itx;

	spa_config_exit(zilog->zl_spa, SCL_STATE, lwb);

	zio_buf_free(lwb->lwb_buf, lwb->lwb_sz);

	mutex_enter(&zilog->zl_lock);

	/*
	* Ensure the lwb buffer pointer is cleared before releasing the
	* txg. If we have had an allocation failure and the txg is
	* waiting to sync then we want zil_sync() to remove the lwb so
	* that it's not picked up as the next new one in
	* zil_process_commit_list(). zil_sync() will only remove the
	* lwb if lwb_buf is null.
	*/
	lwb->lwb_buf = NULL;
	lwb->lwb_tx = NULL;

	ASSERT3U(lwb->lwb_issued_timestamp, >, 0);
	zilog->zl_last_lwb_latency = gethrtime() - lwb->lwb_issued_timestamp;

	lwb->lwb_root_zio = NULL;

	ASSERT3S(lwb->lwb_state, ==, LWB_STATE_WRITE_DONE);
	lwb->lwb_state = LWB_STATE_FLUSH_DONE;

	if (zilog->zl_last_lwb_opened == lwb) {
	/*
	* Remember the highest committed log sequence number
	* for ztest. We only update this value when all the log
	* writes succeeded, because ztest wants to ASSERT that
	* it got the whole log chain.
	*/
	zilog->zl_commit_lr_seq = zilog->zl_lr_seq;
	}

	while ((itx = list_head(&lwb->lwb_itxs)) != NULL) {
	list_remove(&lwb->lwb_itxs, itx);
	zil_itx_destroy(itx);
	}

	while ((zcw = list_head(&lwb->lwb_waiters)) != NULL) {
	mutex_enter(&zcw->zcw_lock);

	ASSERT(list_link_active(&zcw->zcw_node));
	list_remove(&lwb->lwb_waiters, zcw);

	ASSERT3P(zcw->zcw_lwb, ==, lwb);
	zcw->zcw_lwb = NULL;

	zcw->zcw_zio_error = zio->io_error;

	ASSERT3B(zcw->zcw_done, ==, B_FALSE);
	zcw->zcw_done = B_TRUE;
	cv_broadcast(&zcw->zcw_cv);

	mutex_exit(&zcw->zcw_lock);
	}

	mutex_exit(&zilog->zl_lock);

	/*
	* Now that we've written this log block, we have a stable pointer
	* to the next block in the chain, so it's OK to let the txg in
	* which we allocated the next block sync.
	*/
	dmu_tx_commit(tx);
	}

	/*
	* This is called when an lwb's write zio completes. The callback's
	* purpose is to issue the DKIOCFLUSHWRITECACHE commands for the vdevs
	* in the lwb's lwb_vdev_tree. The tree will contain the vdevs involved
	* in writing out this specific lwb's data, and in the case that cache
	* flushes have been deferred, vdevs involved in writing the data for
	* previous lwbs. The writes corresponding to all the vdevs in the
	* lwb_vdev_tree will have completed by the time this is called, due to
	* the zio dependencies configured in zil_lwb_set_zio_dependency(),
	* which takes deferred flushes into account. The lwb will be "done"
	* once zil_lwb_flush_vdevs_done() is called, which occurs in the zio
	* completion callback for the lwb's root zio.
	*/
	static void
	zil_lwb_write_done(zio_t *zio)
	{
	lwb_t *lwb = zio->io_private;
	spa_t *spa = zio->io_spa;
	zilog_t *zilog = lwb->lwb_zilog;
	avl_tree_t *t = &lwb->lwb_vdev_tree;
	void *cookie = NULL;
	zil_vdev_node_t *zv;
	lwb_t *nlwb;

	ASSERT3S(spa_config_held(spa, SCL_STATE, RW_READER), !=, 0);

	ASSERT(BP_GET_COMPRESS(zio->io_bp) == ZIO_COMPRESS_OFF);
	ASSERT(BP_GET_TYPE(zio->io_bp) == DMU_OT_INTENT_LOG);
	ASSERT(BP_GET_LEVEL(zio->io_bp) == 0);
	ASSERT(BP_GET_BYTEORDER(zio->io_bp) == ZFS_HOST_BYTEORDER);
	ASSERT(!BP_IS_GANG(zio->io_bp));
	ASSERT(!BP_IS_HOLE(zio->io_bp));
	ASSERT(BP_GET_FILL(zio->io_bp) == 0);

	abd_put(zio->io_abd);

	mutex_enter(&zilog->zl_lock);
	ASSERT3S(lwb->lwb_state, ==, LWB_STATE_ISSUED);
	lwb->lwb_state = LWB_STATE_WRITE_DONE;
	lwb->lwb_write_zio = NULL;
	lwb->lwb_fastwrite = FALSE;
	nlwb = list_next(&zilog->zl_lwb_list, lwb);
	mutex_exit(&zilog->zl_lock);

	if (avl_numnodes(t) == 0)
	return;

	/*
	* If there was an IO error, we're not going to call zio_flush()
	* on these vdevs, so we simply empty the tree and free the
	* nodes. We avoid calling zio_flush() since there isn't any
	* good reason for doing so, after the lwb block failed to be
	* written out.
	*/
	if (zio->io_error != 0) {
	while ((zv = avl_destroy_nodes(t, &cookie)) != NULL)
	kmem_free(zv, sizeof (*zv));
	return;
	}

	/*
	* If this lwb does not have any threads waiting for it to
	* complete, we want to defer issuing the DKIOCFLUSHWRITECACHE
	* command to the vdevs written to by "this" lwb, and instead
	* rely on the "next" lwb to handle the DKIOCFLUSHWRITECACHE
	* command for those vdevs. Thus, we merge the vdev tree of
	* "this" lwb with the vdev tree of the "next" lwb in the list,
	* and assume the "next" lwb will handle flushing the vdevs (or
	* deferring the flush(s) again).
	*
	* This is a useful performance optimization, especially for
	* workloads with lots of async write activity and few sync
	* write and/or fsync activity, as it has the potential to
	* coalesce multiple flush commands to a vdev into one.
	*/
	if (list_head(&lwb->lwb_waiters) == NULL && nlwb != NULL) {
	zil_lwb_flush_defer(lwb, nlwb);
	ASSERT(avl_is_empty(&lwb->lwb_vdev_tree));
	return;
	}

	while ((zv = avl_destroy_nodes(t, &cookie)) != NULL) {
	vdev_t *vd = vdev_lookup_top(spa, zv->zv_vdev);
	if (vd != NULL)
	zio_flush(lwb->lwb_root_zio, vd);
	kmem_free(zv, sizeof (*zv));
	}
	}

	static void
	zil_lwb_set_zio_dependency(zilog_t zilog, lwb_t lwb)
	{
	lwb_t *last_lwb_opened = zilog->zl_last_lwb_opened;

	ASSERT(MUTEX_HELD(&zilog->zl_issuer_lock));
	ASSERT(MUTEX_HELD(&zilog->zl_lock));

	/*
	* The zilog's "zl_last_lwb_opened" field is used to build the
	* lwb/zio dependency chain, which is used to preserve the
	* ordering of lwb completions that is required by the semantics
	* of the ZIL. Each new lwb zio becomes a parent of the
	* "previous" lwb zio, such that the new lwb's zio cannot
	* complete until the "previous" lwb's zio completes.
	*
	* This is required by the semantics of zil_commit(); the commit
	* waiters attached to the lwbs will be woken in the lwb zio's
	* completion callback, so this zio dependency graph ensures the
	* waiters are woken in the correct order (the same order the
	* lwbs were created).
	*/
	if (last_lwb_opened != NULL &&
	last_lwb_opened->lwb_state != LWB_STATE_FLUSH_DONE) {
	ASSERT(last_lwb_opened->lwb_state == LWB_STATE_OPENED \|\|
	last_lwb_opened->lwb_state == LWB_STATE_ISSUED \|\|
	last_lwb_opened->lwb_state == LWB_STATE_WRITE_DONE);

	ASSERT3P(last_lwb_opened->lwb_root_zio, !=, NULL);
	zio_add_child(lwb->lwb_root_zio,
	last_lwb_opened->lwb_root_zio);

	/*
	* If the previous lwb's write hasn't already completed,
	* we also want to order the completion of the lwb write
	* zios (above, we only order the completion of the lwb
	* root zios). This is required because of how we can
	* defer the DKIOCFLUSHWRITECACHE commands for each lwb.
	*
	* When the DKIOCFLUSHWRITECACHE commands are deferred,
	* the previous lwb will rely on this lwb to flush the
	* vdevs written to by that previous lwb. Thus, we need
	* to ensure this lwb doesn't issue the flush until
	* after the previous lwb's write completes. We ensure
	* this ordering by setting the zio parent/child
	* relationship here.
	*
	* Without this relationship on the lwb's write zio,
	* it's possible for this lwb's write to complete prior
	* to the previous lwb's write completing; and thus, the
	* vdevs for the previous lwb would be flushed prior to
	* that lwb's data being written to those vdevs (the
	* vdevs are flushed in the lwb write zio's completion
	* handler, zil_lwb_write_done()).
	*/
	if (last_lwb_opened->lwb_state != LWB_STATE_WRITE_DONE) {
	ASSERT(last_lwb_opened->lwb_state == LWB_STATE_OPENED \|\|
	last_lwb_opened->lwb_state == LWB_STATE_ISSUED);

	ASSERT3P(last_lwb_opened->lwb_write_zio, !=, NULL);
	zio_add_child(lwb->lwb_write_zio,
	last_lwb_opened->lwb_write_zio);
	}
	}
	}


	/*
	* This function's purpose is to "open" an lwb such that it is ready to
	* accept new itxs being committed to it. To do this, the lwb's zio
	* structures are created, and linked to the lwb. This function is
	* idempotent; if the passed in lwb has already been opened, this
	* function is essentially a no-op.
	*/
	static void
	zil_lwb_write_open(zilog_t zilog, lwb_t lwb)
	{
	zbookmark_phys_t zb;
	zio_priority_t prio;

	ASSERT(MUTEX_HELD(&zilog->zl_issuer_lock));
	ASSERT3P(lwb, !=, NULL);
	EQUIV(lwb->lwb_root_zio == NULL, lwb->lwb_state == LWB_STATE_CLOSED);
	EQUIV(lwb->lwb_root_zio != NULL, lwb->lwb_state == LWB_STATE_OPENED);

	SET_BOOKMARK(&zb, lwb->lwb_blk.blk_cksum.zc_word[ZIL_ZC_OBJSET],
	ZB_ZIL_OBJECT, ZB_ZIL_LEVEL,
	lwb->lwb_blk.blk_cksum.zc_word[ZIL_ZC_SEQ]);

	/* Lock so zil_sync() doesn't fastwrite_unmark after zio is created */
	mutex_enter(&zilog->zl_lock);
	if (lwb->lwb_root_zio == NULL) {
	abd_t *lwb_abd = abd_get_from_buf(lwb->lwb_buf,
	BP_GET_LSIZE(&lwb->lwb_blk));

	if (!lwb->lwb_fastwrite) {
	metaslab_fastwrite_mark(zilog->zl_spa, &lwb->lwb_blk);
	lwb->lwb_fastwrite = 1;
	}

	if (!lwb->lwb_slog \|\| zilog->zl_cur_used <= zil_slog_bulk)
	prio = ZIO_PRIORITY_SYNC_WRITE;
	else
	prio = ZIO_PRIORITY_ASYNC_WRITE;

	lwb->lwb_root_zio = zio_root(zilog->zl_spa,
	zil_lwb_flush_vdevs_done, lwb, ZIO_FLAG_CANFAIL);
	ASSERT3P(lwb->lwb_root_zio, !=, NULL);

	lwb->lwb_write_zio = zio_rewrite(lwb->lwb_root_zio,
	zilog->zl_spa, 0, &lwb->lwb_blk, lwb_abd,
	BP_GET_LSIZE(&lwb->lwb_blk), zil_lwb_write_done, lwb,
	prio, ZIO_FLAG_CANFAIL \| ZIO_FLAG_DONT_PROPAGATE \|
	ZIO_FLAG_FASTWRITE, &zb);
	ASSERT3P(lwb->lwb_write_zio, !=, NULL);

	lwb->lwb_state = LWB_STATE_OPENED;

	zil_lwb_set_zio_dependency(zilog, lwb);
	zilog->zl_last_lwb_opened = lwb;
	}
	mutex_exit(&zilog->zl_lock);

	ASSERT3P(lwb->lwb_root_zio, !=, NULL);
	ASSERT3P(lwb->lwb_write_zio, !=, NULL);
	ASSERT3S(lwb->lwb_state, ==, LWB_STATE_OPENED);
	}

	/*
	* Define a limited set of intent log block sizes.
	*
	* These must be a multiple of 4KB. Note only the amount used (again
	* aligned to 4KB) actually gets written. However, we can't always just
	* allocate SPA_OLD_MAXBLOCKSIZE as the slog space could be exhausted.
	*/
	struct {
	uint64_t limit;
	uint64_t blksz;
	} zil_block_buckets[] = {
	{ 4096, 4096 }, /* non TX_WRITE */
	{ 8192 + 4096, 8192 + 4096 }, /* database */
	{ 32768 + 4096, 32768 + 4096 }, /* NFS writes */
	{ 65536 + 4096, 65536 + 4096 }, /* 64KB writes */
	{ 131072, 131072 }, /* < 128KB writes */
	{ 131072 +4096, 65536 + 4096 }, /* 128KB writes */
	{ UINT64_MAX, SPA_OLD_MAXBLOCKSIZE}, /* > 128KB writes */
	};

	/*
	* Maximum block size used by the ZIL. This is picked up when the ZIL is
	* initialized. Otherwise this should not be used directly; see
	* zl_max_block_size instead.
	*/
	int zil_maxblocksize = SPA_OLD_MAXBLOCKSIZE;

	/*
	* Start a log block write and advance to the next log block.
	* Calls are serialized.
	*/
	static lwb_t *
	zil_lwb_write_issue(zilog_t zilog, lwb_t lwb)
	{
	lwb_t *nlwb = NULL;
	zil_chain_t *zilc;
	spa_t *spa = zilog->zl_spa;
	blkptr_t *bp;
	dmu_tx_t *tx;
	uint64_t txg;
	uint64_t zil_blksz, wsz;
	int i, error;
	boolean_t slog;

	ASSERT(MUTEX_HELD(&zilog->zl_issuer_lock));
	ASSERT3P(lwb->lwb_root_zio, !=, NULL);
	ASSERT3P(lwb->lwb_write_zio, !=, NULL);
	ASSERT3S(lwb->lwb_state, ==, LWB_STATE_OPENED);

	if (BP_GET_CHECKSUM(&lwb->lwb_blk) == ZIO_CHECKSUM_ZILOG2) {
	zilc = (zil_chain_t *)lwb->lwb_buf;
	bp = &zilc->zc_next_blk;
	} else {
	zilc = (zil_chain_t *)(lwb->lwb_buf + lwb->lwb_sz);
	bp = &zilc->zc_next_blk;
	}

	ASSERT(lwb->lwb_nused <= lwb->lwb_sz);

	/*
	* Allocate the next block and save its address in this block
	* before writing it in order to establish the log chain.
	* Note that if the allocation of nlwb synced before we wrote
	* the block that points at it (lwb), we'd leak it if we crashed.
	* Therefore, we don't do dmu_tx_commit() until zil_lwb_write_done().
	* We dirty the dataset to ensure that zil_sync() will be called
	* to clean up in the event of allocation failure or I/O failure.
	*/

	tx = dmu_tx_create(zilog->zl_os);

	/*
	* Since we are not going to create any new dirty data, and we
	* can even help with clearing the existing dirty data, we
	* should not be subject to the dirty data based delays. We
	* use TXG_NOTHROTTLE to bypass the delay mechanism.
	*/
	VERIFY0(dmu_tx_assign(tx, TXG_WAIT \| TXG_NOTHROTTLE));

	dsl_dataset_dirty(dmu_objset_ds(zilog->zl_os), tx);
	txg = dmu_tx_get_txg(tx);

	lwb->lwb_tx = tx;

	/*
	* Log blocks are pre-allocated. Here we select the size of the next
	* block, based on size used in the last block.
	* - first find the smallest bucket that will fit the block from a
	* limited set of block sizes. This is because it's faster to write
	* blocks allocated from the same metaslab as they are adjacent or
	* close.
	* - next find the maximum from the new suggested size and an array of
	* previous sizes. This lessens a picket fence effect of wrongly
	* guessing the size if we have a stream of say 2k, 64k, 2k, 64k
	* requests.
	*
	* Note we only write what is used, but we can't just allocate
	* the maximum block size because we can exhaust the available
	* pool log space.
	*/
	zil_blksz = zilog->zl_cur_used + sizeof (zil_chain_t);
	for (i = 0; zil_blksz > zil_block_buckets[i].limit; i++)
	continue;
	zil_blksz = MIN(zil_block_buckets[i].blksz, zilog->zl_max_block_size);
	zilog->zl_prev_blks[zilog->zl_prev_rotor] = zil_blksz;
	for (i = 0; i < ZIL_PREV_BLKS; i++)
	zil_blksz = MAX(zil_blksz, zilog->zl_prev_blks[i]);
	zilog->zl_prev_rotor = (zilog->zl_prev_rotor + 1) & (ZIL_PREV_BLKS - 1);

	BP_ZERO(bp);
	error = zio_alloc_zil(spa, zilog->zl_os, txg, bp, zil_blksz, &slog);
	if (slog) {
	ZIL_STAT_BUMP(zil_itx_metaslab_slog_count);
	ZIL_STAT_INCR(zil_itx_metaslab_slog_bytes, lwb->lwb_nused);
	} else {
	ZIL_STAT_BUMP(zil_itx_metaslab_normal_count);
	ZIL_STAT_INCR(zil_itx_metaslab_normal_bytes, lwb->lwb_nused);
	}
	if (error == 0) {
	ASSERT3U(bp->blk_birth, ==, txg);
	bp->blk_cksum = lwb->lwb_blk.blk_cksum;
	bp->blk_cksum.zc_word[ZIL_ZC_SEQ]++;

	/*
	* Allocate a new log write block (lwb).
	*/
	nlwb = zil_alloc_lwb(zilog, bp, slog, txg, TRUE);
	}

	if (BP_GET_CHECKSUM(&lwb->lwb_blk) == ZIO_CHECKSUM_ZILOG2) {
	/* For Slim ZIL only write what is used. */
	wsz = P2ROUNDUP_TYPED(lwb->lwb_nused, ZIL_MIN_BLKSZ, uint64_t);
	ASSERT3U(wsz, <=, lwb->lwb_sz);
	zio_shrink(lwb->lwb_write_zio, wsz);

	} else {
	wsz = lwb->lwb_sz;
	}

	zilc->zc_pad = 0;
	zilc->zc_nused = lwb->lwb_nused;
	zilc->zc_eck.zec_cksum = lwb->lwb_blk.blk_cksum;

	/*
	* clear unused data for security
	*/
	bzero(lwb->lwb_buf + lwb->lwb_nused, wsz - lwb->lwb_nused);

	spa_config_enter(zilog->zl_spa, SCL_STATE, lwb, RW_READER);

	zil_lwb_add_block(lwb, &lwb->lwb_blk);
	lwb->lwb_issued_timestamp = gethrtime();
	lwb->lwb_state = LWB_STATE_ISSUED;

	zio_nowait(lwb->lwb_root_zio);
	zio_nowait(lwb->lwb_write_zio);

	/*
	* If there was an allocation failure then nlwb will be null which
	* forces a txg_wait_synced().
	*/
	return (nlwb);
	}

	/*
	* Maximum amount of write data that can be put into single log block.
	*/
	uint64_t
	zil_max_log_data(zilog_t *zilog)
	{
	return (zilog->zl_max_block_size -
	sizeof (zil_chain_t) - sizeof (lr_write_t));
	}

	/*
	* Maximum amount of log space we agree to waste to reduce number of
	* WR_NEED_COPY chunks to reduce zl_get_data() overhead (~12%).
	*/
	static inline uint64_t
	zil_max_waste_space(zilog_t *zilog)
	{
	return (zil_max_log_data(zilog) / 8);
	}

	/*
	* Maximum amount of write data for WR_COPIED. For correctness, consumers
	* must fall back to WR_NEED_COPY if we can't fit the entire record into one
	* maximum sized log block, because each WR_COPIED record must fit in a
	* single log block. For space efficiency, we want to fit two records into a
	* max-sized log block.
	*/
	uint64_t
	zil_max_copied_data(zilog_t *zilog)
	{
	return ((zilog->zl_max_block_size - sizeof (zil_chain_t)) / 2 -
	sizeof (lr_write_t));
	}

	static lwb_t *
	zil_lwb_commit(zilog_t zilog, itx_t itx, lwb_t *lwb)
	{
	lr_t lrcb, lrc;
	lr_write_t lrwb, lrw;
	char *lr_buf;
	uint64_t dlen, dnow, lwb_sp, reclen, txg, max_log_data;

	ASSERT(MUTEX_HELD(&zilog->zl_issuer_lock));
	ASSERT3P(lwb, !=, NULL);
	ASSERT3P(lwb->lwb_buf, !=, NULL);

	zil_lwb_write_open(zilog, lwb);

	lrc = &itx->itx_lr;
	lrw = (lr_write_t *)lrc;

	/*
	* A commit itx doesn't represent any on-disk state; instead
	* it's simply used as a place holder on the commit list, and
	* provides a mechanism for attaching a "commit waiter" onto the
	* correct lwb (such that the waiter can be signalled upon
	* completion of that lwb). Thus, we don't process this itx's
	* log record if it's a commit itx (these itx's don't have log
	* records), and instead link the itx's waiter onto the lwb's
	* list of waiters.
	*
	* For more details, see the comment above zil_commit().
	*/
	if (lrc->lrc_txtype == TX_COMMIT) {
	mutex_enter(&zilog->zl_lock);
	zil_commit_waiter_link_lwb(itx->itx_private, lwb);
	itx->itx_private = NULL;
	mutex_exit(&zilog->zl_lock);
	return (lwb);
	}

	if (lrc->lrc_txtype == TX_WRITE && itx->itx_wr_state == WR_NEED_COPY) {
	dlen = P2ROUNDUP_TYPED(
	lrw->lr_length, sizeof (uint64_t), uint64_t);
	} else {
	dlen = 0;
	}
	reclen = lrc->lrc_reclen;
	zilog->zl_cur_used += (reclen + dlen);
	txg = lrc->lrc_txg;

	ASSERT3U(zilog->zl_cur_used, <, UINT64_MAX - (reclen + dlen));

	cont:
	/*
	* If this record won't fit in the current log block, start a new one.
	* For WR_NEED_COPY optimize layout for minimal number of chunks.
	*/
	lwb_sp = lwb->lwb_sz - lwb->lwb_nused;
	max_log_data = zil_max_log_data(zilog);
	if (reclen > lwb_sp \|\| (reclen + dlen > lwb_sp &&
	lwb_sp < zil_max_waste_space(zilog) &&
	(dlen % max_log_data == 0 \|\|
	lwb_sp < reclen + dlen % max_log_data))) {
	lwb = zil_lwb_write_issue(zilog, lwb);
	if (lwb == NULL)
	return (NULL);
	zil_lwb_write_open(zilog, lwb);
	ASSERT(LWB_EMPTY(lwb));
	lwb_sp = lwb->lwb_sz - lwb->lwb_nused;

	/*
	* There must be enough space in the new, empty log block to
	* hold reclen. For WR_COPIED, we need to fit the whole
	* record in one block, and reclen is the header size + the
	* data size. For WR_NEED_COPY, we can create multiple
	* records, splitting the data into multiple blocks, so we
	* only need to fit one word of data per block; in this case
	* reclen is just the header size (no data).
	*/
	ASSERT3U(reclen + MIN(dlen, sizeof (uint64_t)), <=, lwb_sp);
	}

	dnow = MIN(dlen, lwb_sp - reclen);
	lr_buf = lwb->lwb_buf + lwb->lwb_nused;
	bcopy(lrc, lr_buf, reclen);
	lrcb = (lr_t )lr_buf; / Like lrc, but inside lwb. */
	lrwb = (lr_write_t )lrcb; / Like lrw, but inside lwb. */

	ZIL_STAT_BUMP(zil_itx_count);

	/*
	* If it's a write, fetch the data or get its blkptr as appropriate.
	*/
	if (lrc->lrc_txtype == TX_WRITE) {
	if (txg > spa_freeze_txg(zilog->zl_spa))
	txg_wait_synced(zilog->zl_dmu_pool, txg);
	if (itx->itx_wr_state == WR_COPIED) {
	ZIL_STAT_BUMP(zil_itx_copied_count);
	ZIL_STAT_INCR(zil_itx_copied_bytes, lrw->lr_length);
	} else {
	char *dbuf;
	int error;

	if (itx->itx_wr_state == WR_NEED_COPY) {
	dbuf = lr_buf + reclen;
	lrcb->lrc_reclen += dnow;
	if (lrwb->lr_length > dnow)
	lrwb->lr_length = dnow;
	lrw->lr_offset += dnow;
	lrw->lr_length -= dnow;
	ZIL_STAT_BUMP(zil_itx_needcopy_count);
	ZIL_STAT_INCR(zil_itx_needcopy_bytes, dnow);
	} else {
	ASSERT3S(itx->itx_wr_state, ==, WR_INDIRECT);
	dbuf = NULL;
	ZIL_STAT_BUMP(zil_itx_indirect_count);
	ZIL_STAT_INCR(zil_itx_indirect_bytes,
	lrw->lr_length);
	}

	/*
	* We pass in the "lwb_write_zio" rather than
	* "lwb_root_zio" so that the "lwb_write_zio"
	* becomes the parent of any zio's created by
	* the "zl_get_data" callback. The vdevs are
	* flushed after the "lwb_write_zio" completes,
	* so we want to make sure that completion
	* callback waits for these additional zio's,
	* such that the vdevs used by those zio's will
	* be included in the lwb's vdev tree, and those
	* vdevs will be properly flushed. If we passed
	* in "lwb_root_zio" here, then these additional
	* vdevs may not be flushed; e.g. if these zio's
	* completed after "lwb_write_zio" completed.
	*/
	error = zilog->zl_get_data(itx->itx_private,
	lrwb, dbuf, lwb, lwb->lwb_write_zio);

	if (error == EIO) {
	txg_wait_synced(zilog->zl_dmu_pool, txg);
	return (lwb);
	}
	if (error != 0) {
	ASSERT(error == ENOENT \|\| error == EEXIST \|\|
	error == EALREADY);
	return (lwb);
	}
	}
	}

	/*
	* We're actually making an entry, so update lrc_seq to be the
	* log record sequence number. Note that this is generally not
	* equal to the itx sequence number because not all transactions
	* are synchronous, and sometimes spa_sync() gets there first.
	*/
	lrcb->lrc_seq = ++zilog->zl_lr_seq;
	lwb->lwb_nused += reclen + dnow;

	zil_lwb_add_txg(lwb, txg);

	ASSERT3U(lwb->lwb_nused, <=, lwb->lwb_sz);
	ASSERT0(P2PHASE(lwb->lwb_nused, sizeof (uint64_t)));

	dlen -= dnow;
	if (dlen > 0) {
	zilog->zl_cur_used += reclen;
	goto cont;
	}

	return (lwb);
	}

	itx_t *
	zil_itx_create(uint64_t txtype, size_t lrsize)
	{
	size_t itxsize;
	itx_t *itx;

	lrsize = P2ROUNDUP_TYPED(lrsize, sizeof (uint64_t), size_t);
	itxsize = offsetof(itx_t, itx_lr) + lrsize;

	itx = zio_data_buf_alloc(itxsize);
	itx->itx_lr.lrc_txtype = txtype;
	itx->itx_lr.lrc_reclen = lrsize;
	itx->itx_lr.lrc_seq = 0; /* defensive */
	itx->itx_sync = B_TRUE; /* default is synchronous */
	itx->itx_callback = NULL;
	itx->itx_callback_data = NULL;
	itx->itx_size = itxsize;

	return (itx);
	}

	void
	zil_itx_destroy(itx_t *itx)
	{
	IMPLY(itx->itx_lr.lrc_txtype == TX_COMMIT, itx->itx_callback == NULL);
	IMPLY(itx->itx_callback != NULL, itx->itx_lr.lrc_txtype != TX_COMMIT);

	if (itx->itx_callback != NULL)
	itx->itx_callback(itx->itx_callback_data);

	zio_data_buf_free(itx, itx->itx_size);
	}

	/*
	* Free up the sync and async itxs. The itxs_t has already been detached
	* so no locks are needed.
	*/
	static void
	zil_itxg_clean(itxs_t *itxs)
	{
	itx_t *itx;
	list_t *list;
	avl_tree_t *t;
	void *cookie;
	itx_async_node_t *ian;

	list = &itxs->i_sync_list;
	while ((itx = list_head(list)) != NULL) {
	/*
	* In the general case, commit itxs will not be found
	* here, as they'll be committed to an lwb via
	* zil_lwb_commit(), and free'd in that function. Having
	* said that, it is still possible for commit itxs to be
	* found here, due to the following race:
	*
	* - a thread calls zil_commit() which assigns the
	* commit itx to a per-txg i_sync_list
	* - zil_itxg_clean() is called (e.g. via spa_sync())
	* while the waiter is still on the i_sync_list
	*
	* There's nothing to prevent syncing the txg while the
	* waiter is on the i_sync_list. This normally doesn't
	* happen because spa_sync() is slower than zil_commit(),
	* but if zil_commit() calls txg_wait_synced() (e.g.
	* because zil_create() or zil_commit_writer_stall() is
	* called) we will hit this case.
	*/
	if (itx->itx_lr.lrc_txtype == TX_COMMIT)
	zil_commit_waiter_skip(itx->itx_private);

	list_remove(list, itx);
	zil_itx_destroy(itx);
	}

	cookie = NULL;
	t = &itxs->i_async_tree;
	while ((ian = avl_destroy_nodes(t, &cookie)) != NULL) {
	list = &ian->ia_list;
	while ((itx = list_head(list)) != NULL) {
	list_remove(list, itx);
	/* commit itxs should never be on the async lists. */
	ASSERT3U(itx->itx_lr.lrc_txtype, !=, TX_COMMIT);
	zil_itx_destroy(itx);
	}
	list_destroy(list);
	kmem_free(ian, sizeof (itx_async_node_t));
	}
	avl_destroy(t);

	kmem_free(itxs, sizeof (itxs_t));
	}

	static int
	zil_aitx_compare(const void x1, const void x2)
	{
	const uint64_t o1 = ((itx_async_node_t *)x1)->ia_foid;
	const uint64_t o2 = ((itx_async_node_t *)x2)->ia_foid;

	return (TREE_CMP(o1, o2));
	}

	/*
	* Remove all async itx with the given oid.
	*/
	void
	zil_remove_async(zilog_t *zilog, uint64_t oid)
	{
	uint64_t otxg, txg;
	itx_async_node_t *ian;
	avl_tree_t *t;
	avl_index_t where;
	list_t clean_list;
	itx_t *itx;

	ASSERT(oid != 0);
	list_create(&clean_list, sizeof (itx_t), offsetof(itx_t, itx_node));

	if (spa_freeze_txg(zilog->zl_spa) != UINT64_MAX) /* ziltest support */
	otxg = ZILTEST_TXG;
	else
	otxg = spa_last_synced_txg(zilog->zl_spa) + 1;

	for (txg = otxg; txg < (otxg + TXG_CONCURRENT_STATES); txg++) {
	itxg_t *itxg = &zilog->zl_itxg[txg & TXG_MASK];

	mutex_enter(&itxg->itxg_lock);
	if (itxg->itxg_txg != txg) {
	mutex_exit(&itxg->itxg_lock);
	continue;
	}

	/*
	* Locate the object node and append its list.
	*/
	t = &itxg->itxg_itxs->i_async_tree;
	ian = avl_find(t, &oid, &where);
	if (ian != NULL)
	list_move_tail(&clean_list, &ian->ia_list);
	mutex_exit(&itxg->itxg_lock);
	}
	while ((itx = list_head(&clean_list)) != NULL) {
	list_remove(&clean_list, itx);
	/* commit itxs should never be on the async lists. */
	ASSERT3U(itx->itx_lr.lrc_txtype, !=, TX_COMMIT);
	zil_itx_destroy(itx);
	}
	list_destroy(&clean_list);
	}

	void
	zil_itx_assign(zilog_t zilog, itx_t itx, dmu_tx_t *tx)
	{
	uint64_t txg;
	itxg_t *itxg;
	itxs_t itxs, clean = NULL;

	/*
	* Ensure the data of a renamed file is committed before the rename.
	*/
	if ((itx->itx_lr.lrc_txtype & ~TX_CI) == TX_RENAME)
	zil_async_to_sync(zilog, itx->itx_oid);

	if (spa_freeze_txg(zilog->zl_spa) != UINT64_MAX)
	txg = ZILTEST_TXG;
	else
	txg = dmu_tx_get_txg(tx);

	itxg = &zilog->zl_itxg[txg & TXG_MASK];
	mutex_enter(&itxg->itxg_lock);
	itxs = itxg->itxg_itxs;
	if (itxg->itxg_txg != txg) {
	if (itxs != NULL) {
	/*
	* The zil_clean callback hasn't got around to cleaning
	* this itxg. Save the itxs for release below.
	* This should be rare.
	*/
	zfs_dbgmsg("zil_itx_assign: missed itx cleanup for "
	"txg %llu", itxg->itxg_txg);
	clean = itxg->itxg_itxs;
	}
	itxg->itxg_txg = txg;
	itxs = itxg->itxg_itxs = kmem_zalloc(sizeof (itxs_t),
	KM_SLEEP);

	list_create(&itxs->i_sync_list, sizeof (itx_t),
	offsetof(itx_t, itx_node));
	avl_create(&itxs->i_async_tree, zil_aitx_compare,
	sizeof (itx_async_node_t),
	offsetof(itx_async_node_t, ia_node));
	}
	if (itx->itx_sync) {
	list_insert_tail(&itxs->i_sync_list, itx);
	} else {
	avl_tree_t *t = &itxs->i_async_tree;
	uint64_t foid =
	LR_FOID_GET_OBJ(((lr_ooo_t *)&itx->itx_lr)->lr_foid);
	itx_async_node_t *ian;
	avl_index_t where;

	ian = avl_find(t, &foid, &where);
	if (ian == NULL) {
	ian = kmem_alloc(sizeof (itx_async_node_t),
	KM_SLEEP);
	list_create(&ian->ia_list, sizeof (itx_t),
	offsetof(itx_t, itx_node));
	ian->ia_foid = foid;
	avl_insert(t, ian, where);
	}
	list_insert_tail(&ian->ia_list, itx);
	}

	itx->itx_lr.lrc_txg = dmu_tx_get_txg(tx);

	/*
	* We don't want to dirty the ZIL using ZILTEST_TXG, because
	* zil_clean() will never be called using ZILTEST_TXG. Thus, we
	* need to be careful to always dirty the ZIL using the "real"
	* TXG (not itxg_txg) even when the SPA is frozen.
	*/
	zilog_dirty(zilog, dmu_tx_get_txg(tx));
	mutex_exit(&itxg->itxg_lock);

	/* Release the old itxs now we've dropped the lock */
	if (clean != NULL)
	zil_itxg_clean(clean);
	}

	/*
	* If there are any in-memory intent log transactions which have now been
	* synced then start up a taskq to free them. We should only do this after we
	* have written out the uberblocks (i.e. txg has been committed) so that
	* don't inadvertently clean out in-memory log records that would be required
	* by zil_commit().
	*/
	void
	zil_clean(zilog_t *zilog, uint64_t synced_txg)
	{
	itxg_t *itxg = &zilog->zl_itxg[synced_txg & TXG_MASK];
	itxs_t *clean_me;

	ASSERT3U(synced_txg, <, ZILTEST_TXG);

	mutex_enter(&itxg->itxg_lock);
	if (itxg->itxg_itxs == NULL \|\| itxg->itxg_txg == ZILTEST_TXG) {
	mutex_exit(&itxg->itxg_lock);
	return;
	}
	ASSERT3U(itxg->itxg_txg, <=, synced_txg);
	ASSERT3U(itxg->itxg_txg, !=, 0);
	clean_me = itxg->itxg_itxs;
	itxg->itxg_itxs = NULL;
	itxg->itxg_txg = 0;
	mutex_exit(&itxg->itxg_lock);
	/*
	* Preferably start a task queue to free up the old itxs but
	* if taskq_dispatch can't allocate resources to do that then
	* free it in-line. This should be rare. Note, using TQ_SLEEP
	* created a bad performance problem.
	*/
	ASSERT3P(zilog->zl_dmu_pool, !=, NULL);
	ASSERT3P(zilog->zl_dmu_pool->dp_zil_clean_taskq, !=, NULL);
	taskqid_t id = taskq_dispatch(zilog->zl_dmu_pool->dp_zil_clean_taskq,
	(void ()(void ))zil_itxg_clean, clean_me, TQ_NOSLEEP);
	if (id == TASKQID_INVALID)
	zil_itxg_clean(clean_me);
	}

	/*
	* This function will traverse the queue of itxs that need to be
	* committed, and move them onto the ZIL's zl_itx_commit_list.
	*/
	static void
	zil_get_commit_list(zilog_t *zilog)
	{
	uint64_t otxg, txg;
	list_t *commit_list = &zilog->zl_itx_commit_list;

	ASSERT(MUTEX_HELD(&zilog->zl_issuer_lock));

	if (spa_freeze_txg(zilog->zl_spa) != UINT64_MAX) /* ziltest support */
	otxg = ZILTEST_TXG;
	else
	otxg = spa_last_synced_txg(zilog->zl_spa) + 1;

	/*
	* This is inherently racy, since there is nothing to prevent
	* the last synced txg from changing. That's okay since we'll
	* only commit things in the future.
	*/
	for (txg = otxg; txg < (otxg + TXG_CONCURRENT_STATES); txg++) {
	itxg_t *itxg = &zilog->zl_itxg[txg & TXG_MASK];

	mutex_enter(&itxg->itxg_lock);
	if (itxg->itxg_txg != txg) {
	mutex_exit(&itxg->itxg_lock);
	continue;
	}

	/*
	* If we're adding itx records to the zl_itx_commit_list,
	* then the zil better be dirty in this "txg". We can assert
	* that here since we're holding the itxg_lock which will
	* prevent spa_sync from cleaning it. Once we add the itxs
	* to the zl_itx_commit_list we must commit it to disk even
	* if it's unnecessary (i.e. the txg was synced).
	*/
	ASSERT(zilog_is_dirty_in_txg(zilog, txg) \|\|
	spa_freeze_txg(zilog->zl_spa) != UINT64_MAX);
	list_move_tail(commit_list, &itxg->itxg_itxs->i_sync_list);

	mutex_exit(&itxg->itxg_lock);
	}
	}

	/*
	* Move the async itxs for a specified object to commit into sync lists.
	*/
	void
	zil_async_to_sync(zilog_t *zilog, uint64_t foid)
	{
	uint64_t otxg, txg;
	itx_async_node_t *ian;
	avl_tree_t *t;
	avl_index_t where;

	if (spa_freeze_txg(zilog->zl_spa) != UINT64_MAX) /* ziltest support */
	otxg = ZILTEST_TXG;
	else
	otxg = spa_last_synced_txg(zilog->zl_spa) + 1;

	/*
	* This is inherently racy, since there is nothing to prevent
	* the last synced txg from changing.
	*/
	for (txg = otxg; txg < (otxg + TXG_CONCURRENT_STATES); txg++) {
	itxg_t *itxg = &zilog->zl_itxg[txg & TXG_MASK];

	mutex_enter(&itxg->itxg_lock);
	if (itxg->itxg_txg != txg) {
	mutex_exit(&itxg->itxg_lock);
	continue;
	}

	/*
	* If a foid is specified then find that node and append its
	* list. Otherwise walk the tree appending all the lists
	* to the sync list. We add to the end rather than the
	* beginning to ensure the create has happened.
	*/
	t = &itxg->itxg_itxs->i_async_tree;
	if (foid != 0) {
	ian = avl_find(t, &foid, &where);
	if (ian != NULL) {
	list_move_tail(&itxg->itxg_itxs->i_sync_list,
	&ian->ia_list);
	}
	} else {
	void *cookie = NULL;

	while ((ian = avl_destroy_nodes(t, &cookie)) != NULL) {
	list_move_tail(&itxg->itxg_itxs->i_sync_list,
	&ian->ia_list);
	list_destroy(&ian->ia_list);
	kmem_free(ian, sizeof (itx_async_node_t));
	}
	}
	mutex_exit(&itxg->itxg_lock);
	}
	}

	/*
	* This function will prune commit itxs that are at the head of the
	* commit list (it won't prune past the first non-commit itx), and
	* either: a) attach them to the last lwb that's still pending
	* completion, or b) skip them altogether.
	*
	* This is used as a performance optimization to prevent commit itxs
	* from generating new lwbs when it's unnecessary to do so.
	*/
	static void
	zil_prune_commit_list(zilog_t *zilog)
	{
	itx_t *itx;

	ASSERT(MUTEX_HELD(&zilog->zl_issuer_lock));

	while ((itx = list_head(&zilog->zl_itx_commit_list)) != NULL) {
	lr_t *lrc = &itx->itx_lr;
	if (lrc->lrc_txtype != TX_COMMIT)
	break;

	mutex_enter(&zilog->zl_lock);

	lwb_t *last_lwb = zilog->zl_last_lwb_opened;
	if (last_lwb == NULL \|\|
	last_lwb->lwb_state == LWB_STATE_FLUSH_DONE) {
	/*
	* All of the itxs this waiter was waiting on
	* must have already completed (or there were
	* never any itx's for it to wait on), so it's
	* safe to skip this waiter and mark it done.
	*/
	zil_commit_waiter_skip(itx->itx_private);
	} else {
	zil_commit_waiter_link_lwb(itx->itx_private, last_lwb);
	itx->itx_private = NULL;
	}

	mutex_exit(&zilog->zl_lock);

	list_remove(&zilog->zl_itx_commit_list, itx);
	zil_itx_destroy(itx);
	}

	IMPLY(itx != NULL, itx->itx_lr.lrc_txtype != TX_COMMIT);
	}

	static void
	zil_commit_writer_stall(zilog_t *zilog)
	{
	/*
	* When zio_alloc_zil() fails to allocate the next lwb block on
	* disk, we must call txg_wait_synced() to ensure all of the
	* lwbs in the zilog's zl_lwb_list are synced and then freed (in
	* zil_sync()), such that any subsequent ZIL writer (i.e. a call
	* to zil_process_commit_list()) will have to call zil_create(),
	* and start a new ZIL chain.
	*
	* Since zil_alloc_zil() failed, the lwb that was previously
	* issued does not have a pointer to the "next" lwb on disk.
	* Thus, if another ZIL writer thread was to allocate the "next"
	* on-disk lwb, that block could be leaked in the event of a
	* crash (because the previous lwb on-disk would not point to
	* it).
	*
	* We must hold the zilog's zl_issuer_lock while we do this, to
	* ensure no new threads enter zil_process_commit_list() until
	* all lwb's in the zl_lwb_list have been synced and freed
	* (which is achieved via the txg_wait_synced() call).
	*/
	ASSERT(MUTEX_HELD(&zilog->zl_issuer_lock));
	txg_wait_synced(zilog->zl_dmu_pool, 0);
	ASSERT3P(list_tail(&zilog->zl_lwb_list), ==, NULL);
	}

	/*
	* This function will traverse the commit list, creating new lwbs as
	* needed, and committing the itxs from the commit list to these newly
	* created lwbs. Additionally, as a new lwb is created, the previous
	* lwb will be issued to the zio layer to be written to disk.
	*/
	static void
	zil_process_commit_list(zilog_t *zilog)
	{
	spa_t *spa = zilog->zl_spa;
	list_t nolwb_itxs;
	list_t nolwb_waiters;
	lwb_t *lwb;
	itx_t *itx;

	ASSERT(MUTEX_HELD(&zilog->zl_issuer_lock));

	/*
	* Return if there's nothing to commit before we dirty the fs by
	* calling zil_create().
	*/
	if (list_head(&zilog->zl_itx_commit_list) == NULL)
	return;

	list_create(&nolwb_itxs, sizeof (itx_t), offsetof(itx_t, itx_node));
	list_create(&nolwb_waiters, sizeof (zil_commit_waiter_t),
	offsetof(zil_commit_waiter_t, zcw_node));

	lwb = list_tail(&zilog->zl_lwb_list);
	if (lwb == NULL) {
	lwb = zil_create(zilog);
	} else {
	ASSERT3S(lwb->lwb_state, !=, LWB_STATE_ISSUED);
	ASSERT3S(lwb->lwb_state, !=, LWB_STATE_WRITE_DONE);
	ASSERT3S(lwb->lwb_state, !=, LWB_STATE_FLUSH_DONE);
	}

	while ((itx = list_head(&zilog->zl_itx_commit_list)) != NULL) {
	lr_t *lrc = &itx->itx_lr;
	uint64_t txg = lrc->lrc_txg;

	ASSERT3U(txg, !=, 0);

	if (lrc->lrc_txtype == TX_COMMIT) {
	DTRACE_PROBE2(zil__process__commit__itx,
	zilog_t , zilog, itx_t , itx);
	} else {
	DTRACE_PROBE2(zil__process__normal__itx,
	zilog_t , zilog, itx_t , itx);
	}

	list_remove(&zilog->zl_itx_commit_list, itx);

	boolean_t synced = txg <= spa_last_synced_txg(spa);
	boolean_t frozen = txg > spa_freeze_txg(spa);

	/*
	* If the txg of this itx has already been synced out, then
	* we don't need to commit this itx to an lwb. This is
	* because the data of this itx will have already been
	* written to the main pool. This is inherently racy, and
	* it's still ok to commit an itx whose txg has already
	* been synced; this will result in a write that's
	* unnecessary, but will do no harm.
	*
	* With that said, we always want to commit TX_COMMIT itxs
	* to an lwb, regardless of whether or not that itx's txg
	* has been synced out. We do this to ensure any OPENED lwb
	* will always have at least one zil_commit_waiter_t linked
	* to the lwb.
	*
	* As a counter-example, if we skipped TX_COMMIT itx's
	* whose txg had already been synced, the following
	* situation could occur if we happened to be racing with
	* spa_sync:
	*
	* 1. We commit a non-TX_COMMIT itx to an lwb, where the
	* itx's txg is 10 and the last synced txg is 9.
	* 2. spa_sync finishes syncing out txg 10.
	* 3. We move to the next itx in the list, it's a TX_COMMIT
	* whose txg is 10, so we skip it rather than committing
	* it to the lwb used in (1).
	*
	* If the itx that is skipped in (3) is the last TX_COMMIT
	* itx in the commit list, than it's possible for the lwb
	* used in (1) to remain in the OPENED state indefinitely.
	*
	* To prevent the above scenario from occurring, ensuring
	* that once an lwb is OPENED it will transition to ISSUED
	* and eventually DONE, we always commit TX_COMMIT itx's to
	* an lwb here, even if that itx's txg has already been
	* synced.
	*
	* Finally, if the pool is frozen, we _always_ commit the
	* itx. The point of freezing the pool is to prevent data
	* from being written to the main pool via spa_sync, and
	* instead rely solely on the ZIL to persistently store the
	* data; i.e. when the pool is frozen, the last synced txg
	* value can't be trusted.
	*/
	if (frozen \|\| !synced \|\| lrc->lrc_txtype == TX_COMMIT) {
	if (lwb != NULL) {
	lwb = zil_lwb_commit(zilog, itx, lwb);

	if (lwb == NULL)
	list_insert_tail(&nolwb_itxs, itx);
	else
	list_insert_tail(&lwb->lwb_itxs, itx);
	} else {
	if (lrc->lrc_txtype == TX_COMMIT) {
	zil_commit_waiter_link_nolwb(
	itx->itx_private, &nolwb_waiters);
	}

	list_insert_tail(&nolwb_itxs, itx);
	}
	} else {
	ASSERT3S(lrc->lrc_txtype, !=, TX_COMMIT);
	zil_itx_destroy(itx);
	}
	}

	if (lwb == NULL) {
	/*
	* This indicates zio_alloc_zil() failed to allocate the
	* "next" lwb on-disk. When this happens, we must stall
	* the ZIL write pipeline; see the comment within
	* zil_commit_writer_stall() for more details.
	*/
	zil_commit_writer_stall(zilog);

	/*
	* Additionally, we have to signal and mark the "nolwb"
	* waiters as "done" here, since without an lwb, we
	* can't do this via zil_lwb_flush_vdevs_done() like
	* normal.
	*/
	zil_commit_waiter_t *zcw;
	while ((zcw = list_head(&nolwb_waiters)) != NULL) {
	zil_commit_waiter_skip(zcw);
	list_remove(&nolwb_waiters, zcw);
	}

	/*
	* And finally, we have to destroy the itx's that
	* couldn't be committed to an lwb; this will also call
	* the itx's callback if one exists for the itx.
	*/
	while ((itx = list_head(&nolwb_itxs)) != NULL) {
	list_remove(&nolwb_itxs, itx);
	zil_itx_destroy(itx);
	}
	} else {
	ASSERT(list_is_empty(&nolwb_waiters));
	ASSERT3P(lwb, !=, NULL);
	ASSERT3S(lwb->lwb_state, !=, LWB_STATE_ISSUED);
	ASSERT3S(lwb->lwb_state, !=, LWB_STATE_WRITE_DONE);
	ASSERT3S(lwb->lwb_state, !=, LWB_STATE_FLUSH_DONE);

	/*
	* At this point, the ZIL block pointed at by the "lwb"
	* variable is in one of the following states: "closed"
	* or "open".
	*
	* If it's "closed", then no itxs have been committed to
	* it, so there's no point in issuing its zio (i.e. it's
	* "empty").
	*
	* If it's "open", then it contains one or more itxs that
	* eventually need to be committed to stable storage. In
	* this case we intentionally do not issue the lwb's zio
	* to disk yet, and instead rely on one of the following
	* two mechanisms for issuing the zio:
	*
	* 1. Ideally, there will be more ZIL activity occurring
	* on the system, such that this function will be
	* immediately called again (not necessarily by the same
	* thread) and this lwb's zio will be issued via
	* zil_lwb_commit(). This way, the lwb is guaranteed to
	* be "full" when it is issued to disk, and we'll make
	* use of the lwb's size the best we can.
	*
	* 2. If there isn't sufficient ZIL activity occurring on
	* the system, such that this lwb's zio isn't issued via
	* zil_lwb_commit(), zil_commit_waiter() will issue the
	* lwb's zio. If this occurs, the lwb is not guaranteed
	* to be "full" by the time its zio is issued, and means
	* the size of the lwb was "too large" given the amount
	* of ZIL activity occurring on the system at that time.
	*
	* We do this for a couple of reasons:
	*
	* 1. To try and reduce the number of IOPs needed to
	* write the same number of itxs. If an lwb has space
	* available in its buffer for more itxs, and more itxs
	* will be committed relatively soon (relative to the
	* latency of performing a write), then it's beneficial
	* to wait for these "next" itxs. This way, more itxs
	* can be committed to stable storage with fewer writes.
	*
	* 2. To try and use the largest lwb block size that the
	* incoming rate of itxs can support. Again, this is to
	* try and pack as many itxs into as few lwbs as
	* possible, without significantly impacting the latency
	* of each individual itx.
	*/
	}
	}

	/*
	* This function is responsible for ensuring the passed in commit waiter
	* (and associated commit itx) is committed to an lwb. If the waiter is
	* not already committed to an lwb, all itxs in the zilog's queue of
	* itxs will be processed. The assumption is the passed in waiter's
	* commit itx will found in the queue just like the other non-commit
	* itxs, such that when the entire queue is processed, the waiter will
	* have been committed to an lwb.
	*
	* The lwb associated with the passed in waiter is not guaranteed to
	* have been issued by the time this function completes. If the lwb is
	* not issued, we rely on future calls to zil_commit_writer() to issue
	* the lwb, or the timeout mechanism found in zil_commit_waiter().
	*/
	static void
	zil_commit_writer(zilog_t zilog, zil_commit_waiter_t zcw)
	{
	ASSERT(!MUTEX_HELD(&zilog->zl_lock));
	ASSERT(spa_writeable(zilog->zl_spa));

	mutex_enter(&zilog->zl_issuer_lock);

	if (zcw->zcw_lwb != NULL \|\| zcw->zcw_done) {
	/*
	* It's possible that, while we were waiting to acquire
	* the "zl_issuer_lock", another thread committed this
	* waiter to an lwb. If that occurs, we bail out early,
	* without processing any of the zilog's queue of itxs.
	*
	* On certain workloads and system configurations, the
	* "zl_issuer_lock" can become highly contended. In an
	* attempt to reduce this contention, we immediately drop
	* the lock if the waiter has already been processed.
	*
	* We've measured this optimization to reduce CPU spent
	* contending on this lock by up to 5%, using a system
	* with 32 CPUs, low latency storage (~50 usec writes),
	* and 1024 threads performing sync writes.
	*/
	goto out;
	}

	ZIL_STAT_BUMP(zil_commit_writer_count);

	zil_get_commit_list(zilog);
	zil_prune_commit_list(zilog);
	zil_process_commit_list(zilog);

	out:
	mutex_exit(&zilog->zl_issuer_lock);
	}

	static void
	zil_commit_waiter_timeout(zilog_t zilog, zil_commit_waiter_t zcw)
	{
	ASSERT(!MUTEX_HELD(&zilog->zl_issuer_lock));
	ASSERT(MUTEX_HELD(&zcw->zcw_lock));
	ASSERT3B(zcw->zcw_done, ==, B_FALSE);

	lwb_t *lwb = zcw->zcw_lwb;
	ASSERT3P(lwb, !=, NULL);
	ASSERT3S(lwb->lwb_state, !=, LWB_STATE_CLOSED);

	/*
	* If the lwb has already been issued by another thread, we can
	* immediately return since there's no work to be done (the
	* point of this function is to issue the lwb). Additionally, we
	* do this prior to acquiring the zl_issuer_lock, to avoid
	* acquiring it when it's not necessary to do so.
	*/
	if (lwb->lwb_state == LWB_STATE_ISSUED \|\|
	lwb->lwb_state == LWB_STATE_WRITE_DONE \|\|
	lwb->lwb_state == LWB_STATE_FLUSH_DONE)
	return;

	/*
	* In order to call zil_lwb_write_issue() we must hold the
	* zilog's "zl_issuer_lock". We can't simply acquire that lock,
	* since we're already holding the commit waiter's "zcw_lock",
	* and those two locks are acquired in the opposite order
	* elsewhere.
	*/
	mutex_exit(&zcw->zcw_lock);
	mutex_enter(&zilog->zl_issuer_lock);
	mutex_enter(&zcw->zcw_lock);

	/*
	* Since we just dropped and re-acquired the commit waiter's
	* lock, we have to re-check to see if the waiter was marked
	* "done" during that process. If the waiter was marked "done",
	* the "lwb" pointer is no longer valid (it can be free'd after
	* the waiter is marked "done"), so without this check we could
	* wind up with a use-after-free error below.
	*/
	if (zcw->zcw_done)
	goto out;

	ASSERT3P(lwb, ==, zcw->zcw_lwb);

	/*
	* We've already checked this above, but since we hadn't acquired
	* the zilog's zl_issuer_lock, we have to perform this check a
	* second time while holding the lock.
	*
	* We don't need to hold the zl_lock since the lwb cannot transition
	* from OPENED to ISSUED while we hold the zl_issuer_lock. The lwb
	* _can_ transition from ISSUED to DONE, but it's OK to race with
	* that transition since we treat the lwb the same, whether it's in
	* the ISSUED or DONE states.
	*
	* The important thing, is we treat the lwb differently depending on
	* if it's ISSUED or OPENED, and block any other threads that might
	* attempt to issue this lwb. For that reason we hold the
	* zl_issuer_lock when checking the lwb_state; we must not call
	* zil_lwb_write_issue() if the lwb had already been issued.
	*
	* See the comment above the lwb_state_t structure definition for
	* more details on the lwb states, and locking requirements.
	*/
	if (lwb->lwb_state == LWB_STATE_ISSUED \|\|
	lwb->lwb_state == LWB_STATE_WRITE_DONE \|\|
	lwb->lwb_state == LWB_STATE_FLUSH_DONE)
	goto out;

	ASSERT3S(lwb->lwb_state, ==, LWB_STATE_OPENED);

	/*
	* As described in the comments above zil_commit_waiter() and
	* zil_process_commit_list(), we need to issue this lwb's zio
	* since we've reached the commit waiter's timeout and it still
	* hasn't been issued.
	*/
	lwb_t *nlwb = zil_lwb_write_issue(zilog, lwb);

	IMPLY(nlwb != NULL, lwb->lwb_state != LWB_STATE_OPENED);

	/*
	* Since the lwb's zio hadn't been issued by the time this thread
	* reached its timeout, we reset the zilog's "zl_cur_used" field
	* to influence the zil block size selection algorithm.
	*
	* By having to issue the lwb's zio here, it means the size of the
	* lwb was too large, given the incoming throughput of itxs. By
	* setting "zl_cur_used" to zero, we communicate this fact to the
	* block size selection algorithm, so it can take this information
	* into account, and potentially select a smaller size for the
	* next lwb block that is allocated.
	*/
	zilog->zl_cur_used = 0;

	if (nlwb == NULL) {
	/*
	* When zil_lwb_write_issue() returns NULL, this
	* indicates zio_alloc_zil() failed to allocate the
	* "next" lwb on-disk. When this occurs, the ZIL write
	* pipeline must be stalled; see the comment within the
	* zil_commit_writer_stall() function for more details.
	*
	* We must drop the commit waiter's lock prior to
	* calling zil_commit_writer_stall() or else we can wind
	* up with the following deadlock:
	*
	* - This thread is waiting for the txg to sync while
	* holding the waiter's lock; txg_wait_synced() is
	* used within txg_commit_writer_stall().
	*
	* - The txg can't sync because it is waiting for this
	* lwb's zio callback to call dmu_tx_commit().
	*
	* - The lwb's zio callback can't call dmu_tx_commit()
	* because it's blocked trying to acquire the waiter's
	* lock, which occurs prior to calling dmu_tx_commit()
	*/
	mutex_exit(&zcw->zcw_lock);
	zil_commit_writer_stall(zilog);
	mutex_enter(&zcw->zcw_lock);
	}

	out:
	mutex_exit(&zilog->zl_issuer_lock);
	ASSERT(MUTEX_HELD(&zcw->zcw_lock));
	}

	/*
	* This function is responsible for performing the following two tasks:
	*
	* 1. its primary responsibility is to block until the given "commit
	* waiter" is considered "done".
	*
	* 2. its secondary responsibility is to issue the zio for the lwb that
	* the given "commit waiter" is waiting on, if this function has
	* waited "long enough" and the lwb is still in the "open" state.
	*
	* Given a sufficient amount of itxs being generated and written using
	* the ZIL, the lwb's zio will be issued via the zil_lwb_commit()
	* function. If this does not occur, this secondary responsibility will
	* ensure the lwb is issued even if there is not other synchronous
	* activity on the system.
	*
	* For more details, see zil_process_commit_list(); more specifically,
	* the comment at the bottom of that function.
	*/
	static void
	zil_commit_waiter(zilog_t zilog, zil_commit_waiter_t zcw)
	{
	ASSERT(!MUTEX_HELD(&zilog->zl_lock));
	ASSERT(!MUTEX_HELD(&zilog->zl_issuer_lock));
	ASSERT(spa_writeable(zilog->zl_spa));

	mutex_enter(&zcw->zcw_lock);

	/*
	* The timeout is scaled based on the lwb latency to avoid
	* significantly impacting the latency of each individual itx.
	* For more details, see the comment at the bottom of the
	* zil_process_commit_list() function.
	*/
	int pct = MAX(zfs_commit_timeout_pct, 1);
	hrtime_t sleep = (zilog->zl_last_lwb_latency * pct) / 100;
	hrtime_t wakeup = gethrtime() + sleep;
	boolean_t timedout = B_FALSE;

	while (!zcw->zcw_done) {
	ASSERT(MUTEX_HELD(&zcw->zcw_lock));

	lwb_t *lwb = zcw->zcw_lwb;

	/*
	* Usually, the waiter will have a non-NULL lwb field here,
	* but it's possible for it to be NULL as a result of
	* zil_commit() racing with spa_sync().
	*
	* When zil_clean() is called, it's possible for the itxg
	* list (which may be cleaned via a taskq) to contain
	* commit itxs. When this occurs, the commit waiters linked
	* off of these commit itxs will not be committed to an
	* lwb. Additionally, these commit waiters will not be
	* marked done until zil_commit_waiter_skip() is called via
	* zil_itxg_clean().
	*
	* Thus, it's possible for this commit waiter (i.e. the
	* "zcw" variable) to be found in this "in between" state;
	* where it's "zcw_lwb" field is NULL, and it hasn't yet
	* been skipped, so it's "zcw_done" field is still B_FALSE.
	*/
	IMPLY(lwb != NULL, lwb->lwb_state != LWB_STATE_CLOSED);

	if (lwb != NULL && lwb->lwb_state == LWB_STATE_OPENED) {
	ASSERT3B(timedout, ==, B_FALSE);

	/*
	* If the lwb hasn't been issued yet, then we
	* need to wait with a timeout, in case this
	* function needs to issue the lwb after the
	* timeout is reached; responsibility (2) from
	* the comment above this function.
	*/
	int rc = cv_timedwait_hires(&zcw->zcw_cv,
	&zcw->zcw_lock, wakeup, USEC2NSEC(1),
	CALLOUT_FLAG_ABSOLUTE);

	if (rc != -1 \|\| zcw->zcw_done)
	continue;

	timedout = B_TRUE;
	zil_commit_waiter_timeout(zilog, zcw);

	if (!zcw->zcw_done) {
	/*
	* If the commit waiter has already been
	* marked "done", it's possible for the
	* waiter's lwb structure to have already
	* been freed. Thus, we can only reliably
	* make these assertions if the waiter
	* isn't done.
	*/
	ASSERT3P(lwb, ==, zcw->zcw_lwb);
	ASSERT3S(lwb->lwb_state, !=, LWB_STATE_OPENED);
	}
	} else {
	/*
	* If the lwb isn't open, then it must have already
	* been issued. In that case, there's no need to
	* use a timeout when waiting for the lwb to
	* complete.
	*
	* Additionally, if the lwb is NULL, the waiter
	* will soon be signaled and marked done via
	* zil_clean() and zil_itxg_clean(), so no timeout
	* is required.
	*/

	IMPLY(lwb != NULL,
	lwb->lwb_state == LWB_STATE_ISSUED \|\|
	lwb->lwb_state == LWB_STATE_WRITE_DONE \|\|
	lwb->lwb_state == LWB_STATE_FLUSH_DONE);
	cv_wait(&zcw->zcw_cv, &zcw->zcw_lock);
	}
	}

	mutex_exit(&zcw->zcw_lock);
	}

	static zil_commit_waiter_t *
	zil_alloc_commit_waiter(void)
	{
	zil_commit_waiter_t *zcw = kmem_cache_alloc(zil_zcw_cache, KM_SLEEP);

	cv_init(&zcw->zcw_cv, NULL, CV_DEFAULT, NULL);
	mutex_init(&zcw->zcw_lock, NULL, MUTEX_DEFAULT, NULL);
	list_link_init(&zcw->zcw_node);
	zcw->zcw_lwb = NULL;
	zcw->zcw_done = B_FALSE;
	zcw->zcw_zio_error = 0;

	return (zcw);
	}

	static void
	zil_free_commit_waiter(zil_commit_waiter_t *zcw)
	{
	ASSERT(!list_link_active(&zcw->zcw_node));
	ASSERT3P(zcw->zcw_lwb, ==, NULL);
	ASSERT3B(zcw->zcw_done, ==, B_TRUE);
	mutex_destroy(&zcw->zcw_lock);
	cv_destroy(&zcw->zcw_cv);
	kmem_cache_free(zil_zcw_cache, zcw);
	}

	/*
	* This function is used to create a TX_COMMIT itx and assign it. This
	* way, it will be linked into the ZIL's list of synchronous itxs, and
	* then later committed to an lwb (or skipped) when
	* zil_process_commit_list() is called.
	*/
	static void
	zil_commit_itx_assign(zilog_t zilog, zil_commit_waiter_t zcw)
	{
	dmu_tx_t *tx = dmu_tx_create(zilog->zl_os);
	VERIFY0(dmu_tx_assign(tx, TXG_WAIT));

	itx_t *itx = zil_itx_create(TX_COMMIT, sizeof (lr_t));
	itx->itx_sync = B_TRUE;
	itx->itx_private = zcw;

	zil_itx_assign(zilog, itx, tx);

	dmu_tx_commit(tx);
	}

	/*
	* Commit ZFS Intent Log transactions (itxs) to stable storage.
	*
	* When writing ZIL transactions to the on-disk representation of the
	* ZIL, the itxs are committed to a Log Write Block (lwb). Multiple
	* itxs can be committed to a single lwb. Once a lwb is written and
	* committed to stable storage (i.e. the lwb is written, and vdevs have
	* been flushed), each itx that was committed to that lwb is also
	* considered to be committed to stable storage.
	*
	* When an itx is committed to an lwb, the log record (lr_t) contained
	* by the itx is copied into the lwb's zio buffer, and once this buffer
	* is written to disk, it becomes an on-disk ZIL block.
	*
	* As itxs are generated, they're inserted into the ZIL's queue of
	* uncommitted itxs. The semantics of zil_commit() are such that it will
	* block until all itxs that were in the queue when it was called, are
	* committed to stable storage.
	*
	* If "foid" is zero, this means all "synchronous" and "asynchronous"
	* itxs, for all objects in the dataset, will be committed to stable
	* storage prior to zil_commit() returning. If "foid" is non-zero, all
	* "synchronous" itxs for all objects, but only "asynchronous" itxs
	* that correspond to the foid passed in, will be committed to stable
	* storage prior to zil_commit() returning.
	*
	* Generally speaking, when zil_commit() is called, the consumer doesn't
	* actually care about _all_ of the uncommitted itxs. Instead, they're
	* simply trying to waiting for a specific itx to be committed to disk,
	* but the interface(s) for interacting with the ZIL don't allow such
	* fine-grained communication. A better interface would allow a consumer
	* to create and assign an itx, and then pass a reference to this itx to
	* zil_commit(); such that zil_commit() would return as soon as that
	* specific itx was committed to disk (instead of waiting for _all_
	* itxs to be committed).
	*
	* When a thread calls zil_commit() a special "commit itx" will be
	* generated, along with a corresponding "waiter" for this commit itx.
	* zil_commit() will wait on this waiter's CV, such that when the waiter
	* is marked done, and signaled, zil_commit() will return.
	*
	* This commit itx is inserted into the queue of uncommitted itxs. This
	* provides an easy mechanism for determining which itxs were in the
	* queue prior to zil_commit() having been called, and which itxs were
	* added after zil_commit() was called.
	*
	* The commit it is special; it doesn't have any on-disk representation.
	* When a commit itx is "committed" to an lwb, the waiter associated
	* with it is linked onto the lwb's list of waiters. Then, when that lwb
	* completes, each waiter on the lwb's list is marked done and signaled
	* -- allowing the thread waiting on the waiter to return from zil_commit().
	*
	* It's important to point out a few critical factors that allow us
	* to make use of the commit itxs, commit waiters, per-lwb lists of
	* commit waiters, and zio completion callbacks like we're doing:
	*
	* 1. The list of waiters for each lwb is traversed, and each commit
	* waiter is marked "done" and signaled, in the zio completion
	* callback of the lwb's zio[*].
	*
	* * Actually, the waiters are signaled in the zio completion
	* callback of the root zio for the DKIOCFLUSHWRITECACHE commands
	* that are sent to the vdevs upon completion of the lwb zio.
	*
	* 2. When the itxs are inserted into the ZIL's queue of uncommitted
	* itxs, the order in which they are inserted is preserved[*]; as
	* itxs are added to the queue, they are added to the tail of
	* in-memory linked lists.
	*
	* When committing the itxs to lwbs (to be written to disk), they
	* are committed in the same order in which the itxs were added to
	* the uncommitted queue's linked list(s); i.e. the linked list of
	* itxs to commit is traversed from head to tail, and each itx is
	* committed to an lwb in that order.
	*
	* * To clarify:
	*
	* - the order of "sync" itxs is preserved w.r.t. other
	* "sync" itxs, regardless of the corresponding objects.
	* - the order of "async" itxs is preserved w.r.t. other
	* "async" itxs corresponding to the same object.
	* - the order of "async" itxs is not preserved w.r.t. other
	* "async" itxs corresponding to different objects.
	* - the order of "sync" itxs w.r.t. "async" itxs (or vice
	* versa) is not preserved, even for itxs that correspond
	* to the same object.
	*
	* For more details, see: zil_itx_assign(), zil_async_to_sync(),
	* zil_get_commit_list(), and zil_process_commit_list().
	*
	* 3. The lwbs represent a linked list of blocks on disk. Thus, any
	* lwb cannot be considered committed to stable storage, until its
	* "previous" lwb is also committed to stable storage. This fact,
	* coupled with the fact described above, means that itxs are
	* committed in (roughly) the order in which they were generated.
	* This is essential because itxs are dependent on prior itxs.
	* Thus, we must not deem an itx as being committed to stable
	* storage, until all prior itxs have also been committed to
	* stable storage.
	*
	* To enforce this ordering of lwb zio's, while still leveraging as
	* much of the underlying storage performance as possible, we rely
	* on two fundamental concepts:
	*
	* 1. The creation and issuance of lwb zio's is protected by
	* the zilog's "zl_issuer_lock", which ensures only a single
	* thread is creating and/or issuing lwb's at a time
	* 2. The "previous" lwb is a child of the "current" lwb
	* (leveraging the zio parent-child dependency graph)
	*
	* By relying on this parent-child zio relationship, we can have
	* many lwb zio's concurrently issued to the underlying storage,
	* but the order in which they complete will be the same order in
	* which they were created.
	*/
	void
	zil_commit(zilog_t *zilog, uint64_t foid)
	{
	/*
	* We should never attempt to call zil_commit on a snapshot for
	* a couple of reasons:
	*
	* 1. A snapshot may never be modified, thus it cannot have any
	* in-flight itxs that would have modified the dataset.
	*
	* 2. By design, when zil_commit() is called, a commit itx will
	* be assigned to this zilog; as a result, the zilog will be
	* dirtied. We must not dirty the zilog of a snapshot; there's
	* checks in the code that enforce this invariant, and will
	* cause a panic if it's not upheld.
	*/
	ASSERT3B(dmu_objset_is_snapshot(zilog->zl_os), ==, B_FALSE);

	if (zilog->zl_sync == ZFS_SYNC_DISABLED)
	return;

	if (!spa_writeable(zilog->zl_spa)) {
	/*
	* If the SPA is not writable, there should never be any
	* pending itxs waiting to be committed to disk. If that
	* weren't true, we'd skip writing those itxs out, and
	* would break the semantics of zil_commit(); thus, we're
	* verifying that truth before we return to the caller.
	*/
	ASSERT(list_is_empty(&zilog->zl_lwb_list));
	ASSERT3P(zilog->zl_last_lwb_opened, ==, NULL);
	for (int i = 0; i < TXG_SIZE; i++)
	ASSERT3P(zilog->zl_itxg[i].itxg_itxs, ==, NULL);
	return;
	}

	/*
	* If the ZIL is suspended, we don't want to dirty it by calling
	* zil_commit_itx_assign() below, nor can we write out
	* lwbs like would be done in zil_commit_write(). Thus, we
	* simply rely on txg_wait_synced() to maintain the necessary
	* semantics, and avoid calling those functions altogether.
	*/
	if (zilog->zl_suspend > 0) {
	txg_wait_synced(zilog->zl_dmu_pool, 0);
	return;
	}

	zil_commit_impl(zilog, foid);
	}

	void
	zil_commit_impl(zilog_t *zilog, uint64_t foid)
	{
	ZIL_STAT_BUMP(zil_commit_count);

	/*
	* Move the "async" itxs for the specified foid to the "sync"
	* queues, such that they will be later committed (or skipped)
	* to an lwb when zil_process_commit_list() is called.
	*
	* Since these "async" itxs must be committed prior to this
	* call to zil_commit returning, we must perform this operation
	* before we call zil_commit_itx_assign().
	*/
	zil_async_to_sync(zilog, foid);

	/*
	* We allocate a new "waiter" structure which will initially be
	* linked to the commit itx using the itx's "itx_private" field.
	* Since the commit itx doesn't represent any on-disk state,
	* when it's committed to an lwb, rather than copying the its
	* lr_t into the lwb's buffer, the commit itx's "waiter" will be
	* added to the lwb's list of waiters. Then, when the lwb is
	* committed to stable storage, each waiter in the lwb's list of
	* waiters will be marked "done", and signalled.
	*
	* We must create the waiter and assign the commit itx prior to
	* calling zil_commit_writer(), or else our specific commit itx
	* is not guaranteed to be committed to an lwb prior to calling
	* zil_commit_waiter().
	*/
	zil_commit_waiter_t *zcw = zil_alloc_commit_waiter();
	zil_commit_itx_assign(zilog, zcw);

	zil_commit_writer(zilog, zcw);
	zil_commit_waiter(zilog, zcw);

	if (zcw->zcw_zio_error != 0) {
	/*
	* If there was an error writing out the ZIL blocks that
	* this thread is waiting on, then we fallback to
	* relying on spa_sync() to write out the data this
	* thread is waiting on. Obviously this has performance
	* implications, but the expectation is for this to be
	* an exceptional case, and shouldn't occur often.
	*/
	DTRACE_PROBE2(zil__commit__io__error,
	zilog_t , zilog, zil_commit_waiter_t , zcw);
	txg_wait_synced(zilog->zl_dmu_pool, 0);
	}

	zil_free_commit_waiter(zcw);
	}

	/*
	* Called in syncing context to free committed log blocks and update log header.
	*/
	void
	zil_sync(zilog_t zilog, dmu_tx_t tx)
	{
	zil_header_t *zh = zil_header_in_syncing_context(zilog);
	uint64_t txg = dmu_tx_get_txg(tx);
	spa_t *spa = zilog->zl_spa;
	uint64_t *replayed_seq = &zilog->zl_replayed_seq[txg & TXG_MASK];
	lwb_t *lwb;

	/*
	* We don't zero out zl_destroy_txg, so make sure we don't try
	* to destroy it twice.
	*/
	if (spa_sync_pass(spa) != 1)
	return;

	mutex_enter(&zilog->zl_lock);

	ASSERT(zilog->zl_stop_sync == 0);

	if (*replayed_seq != 0) {
	ASSERT(zh->zh_replay_seq < *replayed_seq);
	zh->zh_replay_seq = *replayed_seq;
	*replayed_seq = 0;
	}

	if (zilog->zl_destroy_txg == txg) {
	blkptr_t blk = zh->zh_log;

	ASSERT(list_head(&zilog->zl_lwb_list) == NULL);

	bzero(zh, sizeof (zil_header_t));
	bzero(zilog->zl_replayed_seq, sizeof (zilog->zl_replayed_seq));

	if (zilog->zl_keep_first) {
	/*
	* If this block was part of log chain that couldn't
	* be claimed because a device was missing during
	* zil_claim(), but that device later returns,
	* then this block could erroneously appear valid.
	* To guard against this, assign a new GUID to the new
	* log chain so it doesn't matter what blk points to.
	*/
	zil_init_log_chain(zilog, &blk);
	zh->zh_log = blk;
	}
	}

	while ((lwb = list_head(&zilog->zl_lwb_list)) != NULL) {
	zh->zh_log = lwb->lwb_blk;
	if (lwb->lwb_buf != NULL \|\| lwb->lwb_max_txg > txg)
	break;
	list_remove(&zilog->zl_lwb_list, lwb);
	zio_free(spa, txg, &lwb->lwb_blk);
	zil_free_lwb(zilog, lwb);

	/*
	* If we don't have anything left in the lwb list then
	* we've had an allocation failure and we need to zero
	* out the zil_header blkptr so that we don't end
	* up freeing the same block twice.
	*/
	if (list_head(&zilog->zl_lwb_list) == NULL)
	BP_ZERO(&zh->zh_log);
	}

	/*
	* Remove fastwrite on any blocks that have been pre-allocated for
	* the next commit. This prevents fastwrite counter pollution by
	* unused, long-lived LWBs.
	*/
	for (; lwb != NULL; lwb = list_next(&zilog->zl_lwb_list, lwb)) {
	if (lwb->lwb_fastwrite && !lwb->lwb_write_zio) {
	metaslab_fastwrite_unmark(zilog->zl_spa, &lwb->lwb_blk);
	lwb->lwb_fastwrite = 0;
	}
	}

	mutex_exit(&zilog->zl_lock);
	}

	/* ARGSUSED */
	static int
	zil_lwb_cons(void vbuf, void unused, int kmflag)
	{
	lwb_t *lwb = vbuf;
	list_create(&lwb->lwb_itxs, sizeof (itx_t), offsetof(itx_t, itx_node));
	list_create(&lwb->lwb_waiters, sizeof (zil_commit_waiter_t),
	offsetof(zil_commit_waiter_t, zcw_node));
	avl_create(&lwb->lwb_vdev_tree, zil_lwb_vdev_compare,
	sizeof (zil_vdev_node_t), offsetof(zil_vdev_node_t, zv_node));
	mutex_init(&lwb->lwb_vdev_lock, NULL, MUTEX_DEFAULT, NULL);
	return (0);
	}

	/* ARGSUSED */
	static void
	zil_lwb_dest(void vbuf, void unused)
	{
	lwb_t *lwb = vbuf;
	mutex_destroy(&lwb->lwb_vdev_lock);
	avl_destroy(&lwb->lwb_vdev_tree);
	list_destroy(&lwb->lwb_waiters);
	list_destroy(&lwb->lwb_itxs);
	}

	void
	zil_init(void)
	{
	zil_lwb_cache = kmem_cache_create("zil_lwb_cache",
	sizeof (lwb_t), 0, zil_lwb_cons, zil_lwb_dest, NULL, NULL, NULL, 0);

	zil_zcw_cache = kmem_cache_create("zil_zcw_cache",
	sizeof (zil_commit_waiter_t), 0, NULL, NULL, NULL, NULL, NULL, 0);

	zil_ksp = kstat_create("zfs", 0, "zil", "misc",
	KSTAT_TYPE_NAMED, sizeof (zil_stats) / sizeof (kstat_named_t),
	KSTAT_FLAG_VIRTUAL);

	if (zil_ksp != NULL) {
	zil_ksp->ks_data = &zil_stats;
	kstat_install(zil_ksp);
	}
	}

	void
	zil_fini(void)
	{
	kmem_cache_destroy(zil_zcw_cache);
	kmem_cache_destroy(zil_lwb_cache);

	if (zil_ksp != NULL) {
	kstat_delete(zil_ksp);
	zil_ksp = NULL;
	}
	}

	void
	zil_set_sync(zilog_t *zilog, uint64_t sync)
	{
	zilog->zl_sync = sync;
	}

	void
	zil_set_logbias(zilog_t *zilog, uint64_t logbias)
	{
	zilog->zl_logbias = logbias;
	}

	zilog_t *
	zil_alloc(objset_t os, zil_header_t zh_phys)
	{
	zilog_t *zilog;

	zilog = kmem_zalloc(sizeof (zilog_t), KM_SLEEP);

	zilog->zl_header = zh_phys;
	zilog->zl_os = os;
	zilog->zl_spa = dmu_objset_spa(os);
	zilog->zl_dmu_pool = dmu_objset_pool(os);
	zilog->zl_destroy_txg = TXG_INITIAL - 1;
	zilog->zl_logbias = dmu_objset_logbias(os);
	zilog->zl_sync = dmu_objset_syncprop(os);
	zilog->zl_dirty_max_txg = 0;
	zilog->zl_last_lwb_opened = NULL;
	zilog->zl_last_lwb_latency = 0;
	zilog->zl_max_block_size = zil_maxblocksize;

	mutex_init(&zilog->zl_lock, NULL, MUTEX_DEFAULT, NULL);
	mutex_init(&zilog->zl_issuer_lock, NULL, MUTEX_DEFAULT, NULL);

	for (int i = 0; i < TXG_SIZE; i++) {
	mutex_init(&zilog->zl_itxg[i].itxg_lock, NULL,
	MUTEX_DEFAULT, NULL);
	}

	list_create(&zilog->zl_lwb_list, sizeof (lwb_t),
	offsetof(lwb_t, lwb_node));

	list_create(&zilog->zl_itx_commit_list, sizeof (itx_t),
	offsetof(itx_t, itx_node));

	cv_init(&zilog->zl_cv_suspend, NULL, CV_DEFAULT, NULL);

	return (zilog);
	}

	void
	zil_free(zilog_t *zilog)
	{
	int i;

	zilog->zl_stop_sync = 1;

	ASSERT0(zilog->zl_suspend);
	ASSERT0(zilog->zl_suspending);

	ASSERT(list_is_empty(&zilog->zl_lwb_list));
	list_destroy(&zilog->zl_lwb_list);

	ASSERT(list_is_empty(&zilog->zl_itx_commit_list));
	list_destroy(&zilog->zl_itx_commit_list);

	for (i = 0; i < TXG_SIZE; i++) {
	/*
	* It's possible for an itx to be generated that doesn't dirty
	* a txg (e.g. ztest TX_TRUNCATE). So there's no zil_clean()
	* callback to remove the entry. We remove those here.
	*
	* Also free up the ziltest itxs.
	*/
	if (zilog->zl_itxg[i].itxg_itxs)
	zil_itxg_clean(zilog->zl_itxg[i].itxg_itxs);
	mutex_destroy(&zilog->zl_itxg[i].itxg_lock);
	}

	mutex_destroy(&zilog->zl_issuer_lock);
	mutex_destroy(&zilog->zl_lock);

	cv_destroy(&zilog->zl_cv_suspend);

	kmem_free(zilog, sizeof (zilog_t));
	}

	/*
	* Open an intent log.
	*/
	zilog_t *
	zil_open(objset_t os, zil_get_data_t get_data)
	{
	zilog_t *zilog = dmu_objset_zil(os);

	ASSERT3P(zilog->zl_get_data, ==, NULL);
	ASSERT3P(zilog->zl_last_lwb_opened, ==, NULL);
	ASSERT(list_is_empty(&zilog->zl_lwb_list));

	zilog->zl_get_data = get_data;

	return (zilog);
	}

	/*
	* Close an intent log.
	*/
	void
	zil_close(zilog_t *zilog)
	{
	lwb_t *lwb;
	uint64_t txg;

	if (!dmu_objset_is_snapshot(zilog->zl_os)) {
	zil_commit(zilog, 0);
	} else {
	ASSERT3P(list_tail(&zilog->zl_lwb_list), ==, NULL);
	ASSERT0(zilog->zl_dirty_max_txg);
	ASSERT3B(zilog_is_dirty(zilog), ==, B_FALSE);
	}

	mutex_enter(&zilog->zl_lock);
	lwb = list_tail(&zilog->zl_lwb_list);
	if (lwb == NULL)
	txg = zilog->zl_dirty_max_txg;
	else
	txg = MAX(zilog->zl_dirty_max_txg, lwb->lwb_max_txg);
	mutex_exit(&zilog->zl_lock);

	/*
	* We need to use txg_wait_synced() to wait long enough for the
	* ZIL to be clean, and to wait for all pending lwbs to be
	* written out.
	*/
	if (txg != 0)
	txg_wait_synced(zilog->zl_dmu_pool, txg);

	if (zilog_is_dirty(zilog))
	zfs_dbgmsg("zil (%px) is dirty, txg %llu", zilog, txg);
	if (txg < spa_freeze_txg(zilog->zl_spa))
	VERIFY(!zilog_is_dirty(zilog));

	zilog->zl_get_data = NULL;

	/*
	* We should have only one lwb left on the list; remove it now.
	*/
	mutex_enter(&zilog->zl_lock);
	lwb = list_head(&zilog->zl_lwb_list);
	if (lwb != NULL) {
	ASSERT3P(lwb, ==, list_tail(&zilog->zl_lwb_list));
	ASSERT3S(lwb->lwb_state, !=, LWB_STATE_ISSUED);

	if (lwb->lwb_fastwrite)
	metaslab_fastwrite_unmark(zilog->zl_spa, &lwb->lwb_blk);

	list_remove(&zilog->zl_lwb_list, lwb);
	zio_buf_free(lwb->lwb_buf, lwb->lwb_sz);
	zil_free_lwb(zilog, lwb);
	}
	mutex_exit(&zilog->zl_lock);
	}

	static char *suspend_tag = "zil suspending";

	/*
	* Suspend an intent log. While in suspended mode, we still honor
	* synchronous semantics, but we rely on txg_wait_synced() to do it.
	* On old version pools, we suspend the log briefly when taking a
	* snapshot so that it will have an empty intent log.
	*
	* Long holds are not really intended to be used the way we do here --
	* held for such a short time. A concurrent caller of dsl_dataset_long_held()
	* could fail. Therefore we take pains to only put a long hold if it is
	* actually necessary. Fortunately, it will only be necessary if the
	* objset is currently mounted (or the ZVOL equivalent). In that case it
	* will already have a long hold, so we are not really making things any worse.
	*
	* Ideally, we would locate the existing long-holder (i.e. the zfsvfs_t or
	* zvol_state_t), and use their mechanism to prevent their hold from being
	* dropped (e.g. VFS_HOLD()). However, that would be even more pain for
	* very little gain.
	*
	* if cookiep == NULL, this does both the suspend & resume.
	* Otherwise, it returns with the dataset "long held", and the cookie
	* should be passed into zil_resume().
	*/
	int
	zil_suspend(const char osname, void *cookiep)
	{
	objset_t *os;
	zilog_t *zilog;
	const zil_header_t *zh;
	int error;

	error = dmu_objset_hold(osname, suspend_tag, &os);
	if (error != 0)
	return (error);
	zilog = dmu_objset_zil(os);

	mutex_enter(&zilog->zl_lock);
	zh = zilog->zl_header;

	if (zh->zh_flags & ZIL_REPLAY_NEEDED) { /* unplayed log */
	mutex_exit(&zilog->zl_lock);
	dmu_objset_rele(os, suspend_tag);
	return (SET_ERROR(EBUSY));
	}

	/*
	* Don't put a long hold in the cases where we can avoid it. This
	* is when there is no cookie so we are doing a suspend & resume
	* (i.e. called from zil_vdev_offline()), and there's nothing to do
	* for the suspend because it's already suspended, or there's no ZIL.
	*/
	if (cookiep == NULL && !zilog->zl_suspending &&
	(zilog->zl_suspend > 0 \|\| BP_IS_HOLE(&zh->zh_log))) {
	mutex_exit(&zilog->zl_lock);
	dmu_objset_rele(os, suspend_tag);
	return (0);
	}

	dsl_dataset_long_hold(dmu_objset_ds(os), suspend_tag);
	dsl_pool_rele(dmu_objset_pool(os), suspend_tag);

	zilog->zl_suspend++;

	if (zilog->zl_suspend > 1) {
	/*
	* Someone else is already suspending it.
	* Just wait for them to finish.
	*/

	while (zilog->zl_suspending)
	cv_wait(&zilog->zl_cv_suspend, &zilog->zl_lock);
	mutex_exit(&zilog->zl_lock);

	if (cookiep == NULL)
	zil_resume(os);
	else
	*cookiep = os;
	return (0);
	}

	/*
	* If there is no pointer to an on-disk block, this ZIL must not
	* be active (e.g. filesystem not mounted), so there's nothing
	* to clean up.
	*/
	if (BP_IS_HOLE(&zh->zh_log)) {
	ASSERT(cookiep != NULL); /* fast path already handled */

	*cookiep = os;
	mutex_exit(&zilog->zl_lock);
	return (0);
	}

	/*
	* The ZIL has work to do. Ensure that the associated encryption
	* key will remain mapped while we are committing the log by
	* grabbing a reference to it. If the key isn't loaded we have no
	* choice but to return an error until the wrapping key is loaded.
	*/
	if (os->os_encrypted &&
	dsl_dataset_create_key_mapping(dmu_objset_ds(os)) != 0) {
	zilog->zl_suspend--;
	mutex_exit(&zilog->zl_lock);
	dsl_dataset_long_rele(dmu_objset_ds(os), suspend_tag);
	dsl_dataset_rele(dmu_objset_ds(os), suspend_tag);
	return (SET_ERROR(EACCES));
	}

	zilog->zl_suspending = B_TRUE;
	mutex_exit(&zilog->zl_lock);

	/*
	* We need to use zil_commit_impl to ensure we wait for all
	* LWB_STATE_OPENED and LWB_STATE_ISSUED lwbs to be committed
	* to disk before proceeding. If we used zil_commit instead, it
	* would just call txg_wait_synced(), because zl_suspend is set.
	* txg_wait_synced() doesn't wait for these lwb's to be
	* LWB_STATE_FLUSH_DONE before returning.
	*/
	zil_commit_impl(zilog, 0);

	/*
	* Now that we've ensured all lwb's are LWB_STATE_FLUSH_DONE, we
	* use txg_wait_synced() to ensure the data from the zilog has
	* migrated to the main pool before calling zil_destroy().
	*/
	txg_wait_synced(zilog->zl_dmu_pool, 0);

	zil_destroy(zilog, B_FALSE);

	mutex_enter(&zilog->zl_lock);
	zilog->zl_suspending = B_FALSE;
	cv_broadcast(&zilog->zl_cv_suspend);
	mutex_exit(&zilog->zl_lock);

	if (os->os_encrypted)
	dsl_dataset_remove_key_mapping(dmu_objset_ds(os));

	if (cookiep == NULL)
	zil_resume(os);
	else
	*cookiep = os;
	return (0);
	}

	void
	zil_resume(void *cookie)
	{
	objset_t *os = cookie;
	zilog_t *zilog = dmu_objset_zil(os);

	mutex_enter(&zilog->zl_lock);
	ASSERT(zilog->zl_suspend != 0);
	zilog->zl_suspend--;
	mutex_exit(&zilog->zl_lock);
	dsl_dataset_long_rele(dmu_objset_ds(os), suspend_tag);
	dsl_dataset_rele(dmu_objset_ds(os), suspend_tag);
	}

	typedef struct zil_replay_arg {
	zil_replay_func_t **zr_replay;
	void *zr_arg;
	boolean_t zr_byteswap;
	char *zr_lr;
	} zil_replay_arg_t;

	static int
	-zil_replay_error(zilog_t zilog, lr_t lr, int error)
	+zil_replay_error(zilog_t zilog, const lr_t lr, int error)
	{
	char name[ZFS_MAX_DATASET_NAME_LEN];

	zilog->zl_replaying_seq--; /* didn't actually replay this one */

	dmu_objset_name(zilog->zl_os, name);

	cmn_err(CE_WARN, "ZFS replay transaction error %d, "
	"dataset %s, seq 0x%llx, txtype %llu %s\n", error, name,
	(u_longlong_t)lr->lrc_seq,
	(u_longlong_t)(lr->lrc_txtype & ~TX_CI),
	(lr->lrc_txtype & TX_CI) ? "CI" : "");

	return (error);
	}

	static int
	-zil_replay_log_record(zilog_t zilog, lr_t lr, void *zra, uint64_t claim_txg)
	+zil_replay_log_record(zilog_t zilog, const lr_t lr, void *zra,
	+ uint64_t claim_txg)
	{
	zil_replay_arg_t *zr = zra;
	const zil_header_t *zh = zilog->zl_header;
	uint64_t reclen = lr->lrc_reclen;
	uint64_t txtype = lr->lrc_txtype;
	int error = 0;

	zilog->zl_replaying_seq = lr->lrc_seq;

	if (lr->lrc_seq <= zh->zh_replay_seq) /* already replayed */
	return (0);

	if (lr->lrc_txg < claim_txg) /* already committed */
	return (0);

	/* Strip case-insensitive bit, still present in log record */
	txtype &= ~TX_CI;

	if (txtype == 0 \|\| txtype >= TX_MAX_TYPE)
	return (zil_replay_error(zilog, lr, EINVAL));

	/*
	* If this record type can be logged out of order, the object
	* (lr_foid) may no longer exist. That's legitimate, not an error.
	*/
	if (TX_OOO(txtype)) {
	error = dmu_object_info(zilog->zl_os,
	LR_FOID_GET_OBJ(((lr_ooo_t *)lr)->lr_foid), NULL);
	if (error == ENOENT \|\| error == EEXIST)
	return (0);
	}

	/*
	* Make a copy of the data so we can revise and extend it.
	*/
	bcopy(lr, zr->zr_lr, reclen);

	/*
	* If this is a TX_WRITE with a blkptr, suck in the data.
	*/
	if (txtype == TX_WRITE && reclen == sizeof (lr_write_t)) {
	error = zil_read_log_data(zilog, (lr_write_t *)lr,
	zr->zr_lr + reclen);
	if (error != 0)
	return (zil_replay_error(zilog, lr, error));
	}

	/*
	* The log block containing this lr may have been byteswapped
	* so that we can easily examine common fields like lrc_txtype.
	* However, the log is a mix of different record types, and only the
	* replay vectors know how to byteswap their records. Therefore, if
	* the lr was byteswapped, undo it before invoking the replay vector.
	*/
	if (zr->zr_byteswap)
	byteswap_uint64_array(zr->zr_lr, reclen);

	/*
	* We must now do two things atomically: replay this log record,
	* and update the log header sequence number to reflect the fact that
	* we did so. At the end of each replay function the sequence number
	* is updated if we are in replay mode.
	*/
	error = zr->zr_replay[txtype](zr->zr_arg, zr->zr_lr, zr->zr_byteswap);
	if (error != 0) {
	/*
	* The DMU's dnode layer doesn't see removes until the txg
	* commits, so a subsequent claim can spuriously fail with
	* EEXIST. So if we receive any error we try syncing out
	* any removes then retry the transaction. Note that we
	* specify B_FALSE for byteswap now, so we don't do it twice.
	*/
	txg_wait_synced(spa_get_dsl(zilog->zl_spa), 0);
	error = zr->zr_replay[txtype](zr->zr_arg, zr->zr_lr, B_FALSE);
	if (error != 0)
	return (zil_replay_error(zilog, lr, error));
	}
	return (0);
	}

	/* ARGSUSED */
	static int
	-zil_incr_blks(zilog_t zilog, blkptr_t bp, void *arg, uint64_t claim_txg)
	+zil_incr_blks(zilog_t zilog, const blkptr_t bp, void *arg, uint64_t claim_txg)
	{
	zilog->zl_replay_blks++;

	return (0);
	}

	/*
	* If this dataset has a non-empty intent log, replay it and destroy it.
	*/
	void
	zil_replay(objset_t os, void arg, zil_replay_func_t *replay_func[TX_MAX_TYPE])
	{
	zilog_t *zilog = dmu_objset_zil(os);
	const zil_header_t *zh = zilog->zl_header;
	zil_replay_arg_t zr;

	if ((zh->zh_flags & ZIL_REPLAY_NEEDED) == 0) {
	zil_destroy(zilog, B_TRUE);
	return;
	}

	zr.zr_replay = replay_func;
	zr.zr_arg = arg;
	zr.zr_byteswap = BP_SHOULD_BYTESWAP(&zh->zh_log);
	zr.zr_lr = vmem_alloc(2 * SPA_MAXBLOCKSIZE, KM_SLEEP);

	/*
	* Wait for in-progress removes to sync before starting replay.
	*/
	txg_wait_synced(zilog->zl_dmu_pool, 0);

	zilog->zl_replay = B_TRUE;
	zilog->zl_replay_time = ddi_get_lbolt();
	ASSERT(zilog->zl_replay_blks == 0);
	(void) zil_parse(zilog, zil_incr_blks, zil_replay_log_record, &zr,
	zh->zh_claim_txg, B_TRUE);
	vmem_free(zr.zr_lr, 2 * SPA_MAXBLOCKSIZE);

	zil_destroy(zilog, B_FALSE);
	txg_wait_synced(zilog->zl_dmu_pool, zilog->zl_destroy_txg);
	zilog->zl_replay = B_FALSE;
	}

	boolean_t
	zil_replaying(zilog_t zilog, dmu_tx_t tx)
	{
	if (zilog->zl_sync == ZFS_SYNC_DISABLED)
	return (B_TRUE);

	if (zilog->zl_replay) {
	dsl_dataset_dirty(dmu_objset_ds(zilog->zl_os), tx);
	zilog->zl_replayed_seq[dmu_tx_get_txg(tx) & TXG_MASK] =
	zilog->zl_replaying_seq;
	return (B_TRUE);
	}

	return (B_FALSE);
	}

	/* ARGSUSED */
	int
	zil_reset(const char osname, void arg)
	{
	int error;

	error = zil_suspend(osname, NULL);
	/* EACCES means crypto key not loaded */
	if ((error == EACCES) \|\| (error == EBUSY))
	return (SET_ERROR(error));
	if (error != 0)
	return (SET_ERROR(EEXIST));
	return (0);
	}

	EXPORT_SYMBOL(zil_alloc);
	EXPORT_SYMBOL(zil_free);
	EXPORT_SYMBOL(zil_open);
	EXPORT_SYMBOL(zil_close);
	EXPORT_SYMBOL(zil_replay);
	EXPORT_SYMBOL(zil_replaying);
	EXPORT_SYMBOL(zil_destroy);
	EXPORT_SYMBOL(zil_destroy_sync);
	EXPORT_SYMBOL(zil_itx_create);
	EXPORT_SYMBOL(zil_itx_destroy);
	EXPORT_SYMBOL(zil_itx_assign);
	EXPORT_SYMBOL(zil_commit);
	EXPORT_SYMBOL(zil_claim);
	EXPORT_SYMBOL(zil_check_log_chain);
	EXPORT_SYMBOL(zil_sync);
	EXPORT_SYMBOL(zil_clean);
	EXPORT_SYMBOL(zil_suspend);
	EXPORT_SYMBOL(zil_resume);
	EXPORT_SYMBOL(zil_lwb_add_block);
	EXPORT_SYMBOL(zil_bp_tree_add);
	EXPORT_SYMBOL(zil_set_sync);
	EXPORT_SYMBOL(zil_set_logbias);

	/* BEGIN CSTYLED */
	ZFS_MODULE_PARAM(zfs, zfs_, commit_timeout_pct, INT, ZMOD_RW,
	"ZIL block open timeout percentage");

	ZFS_MODULE_PARAM(zfs_zil, zil_, replay_disable, INT, ZMOD_RW,
	"Disable intent logging replay");

	ZFS_MODULE_PARAM(zfs_zil, zil_, nocacheflush, INT, ZMOD_RW,
	"Disable ZIL cache flushes");

	ZFS_MODULE_PARAM(zfs_zil, zil_, slog_bulk, ULONG, ZMOD_RW,
	"Limit in bytes slog sync writes per commit");

	ZFS_MODULE_PARAM(zfs_zil, zil_, maxblocksize, INT, ZMOD_RW,
	"Limit in bytes of ZIL log block size");
	/* END CSTYLED */
	Index: vendor-sys/openzfs/dist/module/zstd/lib/zstd.c
	===================================================================
	--- vendor-sys/openzfs/dist/module/zstd/lib/zstd.c (revision 366773)
	+++ vendor-sys/openzfs/dist/module/zstd/lib/zstd.c (revision 366774)
	@@ -1,27826 +1,27826 @@
	/*
	* BSD 3-Clause Clear License
	*
	* Redistribution and use in source and binary forms, with or without
	* modification, are permitted provided that the following conditions are met:
	*
	* 1. Redistributions of source code must retain the above copyright notice,
	* this list of conditions and the following disclaimer.
	*
	* 2. Redistributions in binary form must reproduce the above copyright notice,
	* this list of conditions and the following disclaimer in the documentation
	* and/or other materials provided with the distribution.
	*
	* 3. Neither the name of the copyright holder nor the names of its
	* contributors may be used to endorse or promote products derived from this
	* software without specific prior written permission.
	*
	* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
	* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
	* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
	* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
	* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
	* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
	* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
	* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
	* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
	* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
	* POSSIBILITY OF SUCH DAMAGE.
	*/

	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc. All rights reserved.
	* Copyright (c) 2019-2020, Michael Niewöhner. All rights reserved.
	*/

	#define MEM_MODULE
	#define XXH_NAMESPACE ZSTD_
	#define XXH_PRIVATE_API
	#define XXH_INLINE_ALL
	#define ZSTD_LEGACY_SUPPORT 0
	#define ZSTD_LIB_DICTBUILDER 0
	#define ZSTD_LIB_DEPRECATED 0
	#define ZSTD_NOBENCH

	/** start inlining common/debug.c **/
	/* ******************************************************************
	* debug
	* Part of FSE library
	* Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
	*
	* You can contact the author at :
	* - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	****************************************************************** */


	/*
	* This module only hosts one global variable
	* which can be used to dynamically influence the verbosity of traces,
	* such as DEBUGLOG and RAWLOG
	*/

	/** start inlining debug.h **/
	/* ******************************************************************
	* debug
	* Part of FSE library
	* Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
	*
	* You can contact the author at :
	* - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	****************************************************************** */


	/*
	* The purpose of this header is to enable debug functions.
	* They regroup assert(), DEBUGLOG() and RAWLOG() for run-time,
	* and DEBUG_STATIC_ASSERT() for compile-time.
	*
	* By default, DEBUGLEVEL==0, which means run-time debug is disabled.
	*
	* Level 1 enables assert() only.
	* Starting level 2, traces can be generated and pushed to stderr.
	* The higher the level, the more verbose the traces.
	*
	* It's possible to dynamically adjust level using variable g_debug_level,
	* which is only declared if DEBUGLEVEL>=2,
	* and is a global variable, not multi-thread protected (use with care)
	*/

	#ifndef DEBUG_H_12987983217
	#define DEBUG_H_12987983217

	#if defined (__cplusplus)
	extern "C" {
	#endif


	/* static assert is triggered at compile time, leaving no runtime artefact.
	* static assert only works with compile-time constants.
	* Also, this variant can only be used inside a function. */
	#define DEBUG_STATIC_ASSERT(c) (void)sizeof(char[(c) ? 1 : -1])


	/* DEBUGLEVEL is expected to be defined externally,
	* typically through compiler command line.
	* Value must be a number. */
	#ifndef DEBUGLEVEL
	# define DEBUGLEVEL 0
	#endif


	/* DEBUGFILE can be defined externally,
	* typically through compiler command line.
	* note : currently useless.
	* Value must be stderr or stdout */
	#ifndef DEBUGFILE
	# define DEBUGFILE stderr
	#endif


	/* recommended values for DEBUGLEVEL :
	* 0 : release mode, no debug, all run-time checks disabled
	* 1 : enables assert() only, no display
	* 2 : reserved, for currently active debug path
	* 3 : events once per object lifetime (CCtx, CDict, etc.)
	* 4 : events once per frame
	* 5 : events once per block
	* 6 : events once per sequence (verbose)
	* 7+: events at every position (very verbose)
	*
	* It's generally inconvenient to output traces > 5.
	* In which case, it's possible to selectively trigger high verbosity levels
	* by modifying g_debug_level.
	*/

	#if (DEBUGLEVEL>=1)
	# include <assert.h>
	#else
	# ifndef assert /* assert may be already defined, due to prior #include <assert.h> */
	# define assert(condition) ((void)0) /* disable assert (default) */
	# endif
	#endif

	#if (DEBUGLEVEL>=2)
	# include <stdio.h>
	extern int g_debuglevel; /* the variable is only declared,
	it actually lives in debug.c,
	and is shared by the whole process.
	It's not thread-safe.
	It's useful when enabling very verbose levels
	on selective conditions (such as position in src) */

	# define RAWLOG(l, ...) { \
	if (l<=g_debuglevel) { \
	fprintf(stderr, __VA_ARGS__); \
	} }
	# define DEBUGLOG(l, ...) { \
	if (l<=g_debuglevel) { \
	fprintf(stderr, __FILE__ ": " __VA_ARGS__); \
	fprintf(stderr, " \n"); \
	} }
	#else
	# define RAWLOG(l, ...) {} /* disabled */
	# define DEBUGLOG(l, ...) {} /* disabled */
	#endif


	#if defined (__cplusplus)
	}
	#endif

	#endif /* DEBUG_H_12987983217 */
	/** ended inlining debug.h **/

	int g_debuglevel = DEBUGLEVEL;
	/** ended inlining common/debug.c **/
	/** start inlining common/entropy_common.c **/
	/* ******************************************************************
	* Common functions of New Generation Entropy library
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	*
	* You can contact the author at :
	* - FSE+HUF source repository : https://github.com/Cyan4973/FiniteStateEntropy
	* - Public forum : https://groups.google.com/forum/#!forum/lz4c
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	****************************************************************** */

	/* *************************************
	* Dependencies
	***************************************/
	/** start inlining mem.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef MEM_H_MODULE
	#define MEM_H_MODULE

	#if defined (__cplusplus)
	extern "C" {
	#endif

	/-***************************************
	* Dependencies
	******************************************/
	#include <stddef.h> /* size_t, ptrdiff_t */
	#include <string.h> /* memcpy */


	/-***************************************
	* Compiler specifics
	******************************************/
	#if defined(_MSC_VER) /* Visual Studio */
	# include <stdlib.h> /* _byteswap_ulong */
	# include <intrin.h> /* _byteswap_* */
	#endif
	#if defined(__GNUC__)
	# define MEM_STATIC static __inline __attribute__((unused))
	#elif defined (__cplusplus) \|\| (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */)
	# define MEM_STATIC static inline
	#elif defined(_MSC_VER)
	# define MEM_STATIC static __inline
	#else
	# define MEM_STATIC static /* this version may generate warnings for unused static functions; disable the relevant warning */
	#endif

	#ifndef __has_builtin
	# define __has_builtin(x) 0 /* compat. with non-clang compilers */
	#endif

	/* code only tested on 32 and 64 bits systems */
	#define MEM_STATIC_ASSERT(c) { enum { MEM_static_assert = 1/(int)(!!(c)) }; }
	MEM_STATIC void MEM_check(void) { MEM_STATIC_ASSERT((sizeof(size_t)==4) \|\| (sizeof(size_t)==8)); }

	/* detects whether we are being compiled under msan */
	#if defined (__has_feature)
	# if __has_feature(memory_sanitizer)
	# define MEMORY_SANITIZER 1
	# endif
	#endif

	#if defined (MEMORY_SANITIZER)
	/* Not all platforms that support msan provide sanitizers/msan_interface.h.
	* We therefore declare the functions we need ourselves, rather than trying to
	* include the header file... */

	#include <stdint.h> /* intptr_t */

	/* Make memory region fully initialized (without changing its contents). */
	void __msan_unpoison(const volatile void *a, size_t size);

	/* Make memory region fully uninitialized (without changing its contents).
	This is a legacy interface that does not update origin information. Use
	__msan_allocated_memory() instead. */
	void __msan_poison(const volatile void *a, size_t size);

	/* Returns the offset of the first (at least partially) poisoned byte in the
	memory range, or -1 if the whole range is good. */
	intptr_t __msan_test_shadow(const volatile void *x, size_t size);
	#endif

	/* detects whether we are being compiled under asan */
	#if defined (__has_feature)
	# if __has_feature(address_sanitizer)
	# define ADDRESS_SANITIZER 1
	# endif
	#elif defined(__SANITIZE_ADDRESS__)
	# define ADDRESS_SANITIZER 1
	#endif

	#if defined (ADDRESS_SANITIZER)
	/* Not all platforms that support asan provide sanitizers/asan_interface.h.
	* We therefore declare the functions we need ourselves, rather than trying to
	* include the header file... */

	/**
	* Marks a memory region (<c>[addr, addr+size)</c>) as unaddressable.
	*
	* This memory must be previously allocated by your program. Instrumented
	* code is forbidden from accessing addresses in this region until it is
	* unpoisoned. This function is not guaranteed to poison the entire region -
	* it could poison only a subregion of <c>[addr, addr+size)</c> due to ASan
	* alignment restrictions.
	*
	* \note This function is not thread-safe because no two threads can poison or
	* unpoison memory in the same memory region simultaneously.
	*
	* \param addr Start of memory region.
	* \param size Size of memory region. */
	void __asan_poison_memory_region(void const volatile *addr, size_t size);

	/**
	* Marks a memory region (<c>[addr, addr+size)</c>) as addressable.
	*
	* This memory must be previously allocated by your program. Accessing
	* addresses in this region is allowed until this region is poisoned again.
	* This function could unpoison a super-region of <c>[addr, addr+size)</c> due
	* to ASan alignment restrictions.
	*
	* \note This function is not thread-safe because no two threads can
	* poison or unpoison memory in the same memory region simultaneously.
	*
	* \param addr Start of memory region.
	* \param size Size of memory region. */
	void __asan_unpoison_memory_region(void const volatile *addr, size_t size);
	#endif


	/-*************************************************************
	* Basic Types
	*****************************************************************/
	#if !defined (__VMS) && (defined (__cplusplus) \|\| (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
	# include <stdint.h>
	typedef uint8_t BYTE;
	typedef uint16_t U16;
	typedef int16_t S16;
	typedef uint32_t U32;
	typedef int32_t S32;
	typedef uint64_t U64;
	typedef int64_t S64;
	#else
	# include <limits.h>
	#if CHAR_BIT != 8
	# error "this implementation requires char to be exactly 8-bit type"
	#endif
	typedef unsigned char BYTE;
	#if USHRT_MAX != 65535
	# error "this implementation requires short to be exactly 16-bit type"
	#endif
	typedef unsigned short U16;
	typedef signed short S16;
	#if UINT_MAX != 4294967295
	# error "this implementation requires int to be exactly 32-bit type"
	#endif
	typedef unsigned int U32;
	typedef signed int S32;
	/* note : there are no limits defined for long long type in C90.
	* limits exist in C99, however, in such case, <stdint.h> is preferred */
	typedef unsigned long long U64;
	typedef signed long long S64;
	#endif


	/-*************************************************************
	* Memory I/O
	*****************************************************************/
	/* MEM_FORCE_MEMORY_ACCESS :
	* By default, access to unaligned memory is controlled by `memcpy()`, which is safe and portable.
	* Unfortunately, on some target/compiler combinations, the generated assembly is sub-optimal.
	* The below switch allow to select different access method for improved performance.
	* Method 0 (default) : use `memcpy()`. Safe and portable.
	* Method 1 : `__packed` statement. It depends on compiler extension (i.e., not portable).
	* This method is safe if your compiler supports it, and generally as fast or faster than `memcpy`.
	* Method 2 : direct access. This method is portable but violate C standard.
	* It can generate buggy code on targets depending on alignment.
	* In some circumstances, it's the only known way to get the most performance (i.e. GCC + ARMv6)
	* See http://fastcompression.blogspot.fr/2015/08/accessing-unaligned-memory.html for details.
	* Prefer these methods in priority order (0 > 1 > 2)
	*/
	#ifndef MEM_FORCE_MEMORY_ACCESS /* can be defined externally, on command line for example */
	# if defined(__GNUC__) && ( defined(__ARM_ARCH_6__) \|\| defined(__ARM_ARCH_6J__) \|\| defined(__ARM_ARCH_6K__) \|\| defined(__ARM_ARCH_6Z__) \|\| defined(__ARM_ARCH_6ZK__) \|\| defined(__ARM_ARCH_6T2__) )
	# define MEM_FORCE_MEMORY_ACCESS 2
	# elif defined(__INTEL_COMPILER) \|\| defined(__GNUC__) \|\| defined(__ICCARM__)
	# define MEM_FORCE_MEMORY_ACCESS 1
	# endif
	#endif

	MEM_STATIC unsigned MEM_32bits(void) { return sizeof(size_t)==4; }
	MEM_STATIC unsigned MEM_64bits(void) { return sizeof(size_t)==8; }

	MEM_STATIC unsigned MEM_isLittleEndian(void)
	{
	const union { U32 u; BYTE c[4]; } one = { 1 }; /* don't use static : performance detrimental */
	return one.c[0];
	}

	#if defined(MEM_FORCE_MEMORY_ACCESS) && (MEM_FORCE_MEMORY_ACCESS==2)

	/* violates C standard, by lying on structure alignment.
	Only use if no other choice to achieve best performance on target platform */
	MEM_STATIC U16 MEM_read16(const void* memPtr) { return (const U16) memPtr; }
	MEM_STATIC U32 MEM_read32(const void* memPtr) { return (const U32) memPtr; }
	MEM_STATIC U64 MEM_read64(const void* memPtr) { return (const U64) memPtr; }
	MEM_STATIC size_t MEM_readST(const void* memPtr) { return (const size_t) memPtr; }

	MEM_STATIC void MEM_write16(void* memPtr, U16 value) { (U16)memPtr = value; }
	MEM_STATIC void MEM_write32(void* memPtr, U32 value) { (U32)memPtr = value; }
	MEM_STATIC void MEM_write64(void* memPtr, U64 value) { (U64)memPtr = value; }

	#elif defined(MEM_FORCE_MEMORY_ACCESS) && (MEM_FORCE_MEMORY_ACCESS==1)

	/* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */
	/* currently only defined for gcc and icc */
	#if defined(_MSC_VER) \|\| (defined(__INTEL_COMPILER) && defined(WIN32))
	__pragma( pack(push, 1) )
	typedef struct { U16 v; } unalign16;
	typedef struct { U32 v; } unalign32;
	typedef struct { U64 v; } unalign64;
	typedef struct { size_t v; } unalignArch;
	__pragma( pack(pop) )
	#else
	typedef struct { U16 v; } __attribute__((packed)) unalign16;
	typedef struct { U32 v; } __attribute__((packed)) unalign32;
	typedef struct { U64 v; } __attribute__((packed)) unalign64;
	typedef struct { size_t v; } __attribute__((packed)) unalignArch;
	#endif

	MEM_STATIC U16 MEM_read16(const void* ptr) { return ((const unalign16*)ptr)->v; }
	MEM_STATIC U32 MEM_read32(const void* ptr) { return ((const unalign32*)ptr)->v; }
	MEM_STATIC U64 MEM_read64(const void* ptr) { return ((const unalign64*)ptr)->v; }
	MEM_STATIC size_t MEM_readST(const void* ptr) { return ((const unalignArch*)ptr)->v; }

	MEM_STATIC void MEM_write16(void* memPtr, U16 value) { ((unalign16*)memPtr)->v = value; }
	MEM_STATIC void MEM_write32(void* memPtr, U32 value) { ((unalign32*)memPtr)->v = value; }
	MEM_STATIC void MEM_write64(void* memPtr, U64 value) { ((unalign64*)memPtr)->v = value; }

	#else

	/* default method, safe and standard.
	can sometimes prove slower */

	MEM_STATIC U16 MEM_read16(const void* memPtr)
	{
	U16 val; memcpy(&val, memPtr, sizeof(val)); return val;
	}

	MEM_STATIC U32 MEM_read32(const void* memPtr)
	{
	U32 val; memcpy(&val, memPtr, sizeof(val)); return val;
	}

	MEM_STATIC U64 MEM_read64(const void* memPtr)
	{
	U64 val; memcpy(&val, memPtr, sizeof(val)); return val;
	}

	MEM_STATIC size_t MEM_readST(const void* memPtr)
	{
	size_t val; memcpy(&val, memPtr, sizeof(val)); return val;
	}

	MEM_STATIC void MEM_write16(void* memPtr, U16 value)
	{
	memcpy(memPtr, &value, sizeof(value));
	}

	MEM_STATIC void MEM_write32(void* memPtr, U32 value)
	{
	memcpy(memPtr, &value, sizeof(value));
	}

	MEM_STATIC void MEM_write64(void* memPtr, U64 value)
	{
	memcpy(memPtr, &value, sizeof(value));
	}

	#endif /* MEM_FORCE_MEMORY_ACCESS */

	MEM_STATIC U32 MEM_swap32(U32 in)
	{
	#if defined(_MSC_VER) /* Visual Studio */
	return _byteswap_ulong(in);
	#elif (defined (__GNUC__) && (__GNUC__ * 100 + __GNUC_MINOR__ >= 403)) \
	\|\| (defined(__clang__) && __has_builtin(__builtin_bswap32))
	return __builtin_bswap32(in);
	#else
	return ((in << 24) & 0xff000000 ) \|
	((in << 8) & 0x00ff0000 ) \|
	((in >> 8) & 0x0000ff00 ) \|
	((in >> 24) & 0x000000ff );
	#endif
	}

	MEM_STATIC U64 MEM_swap64(U64 in)
	{
	#if defined(_MSC_VER) /* Visual Studio */
	return _byteswap_uint64(in);
	#elif (defined (__GNUC__) && (__GNUC__ * 100 + __GNUC_MINOR__ >= 403)) \
	\|\| (defined(__clang__) && __has_builtin(__builtin_bswap64))
	return __builtin_bswap64(in);
	#else
	return ((in << 56) & 0xff00000000000000ULL) \|
	((in << 40) & 0x00ff000000000000ULL) \|
	((in << 24) & 0x0000ff0000000000ULL) \|
	((in << 8) & 0x000000ff00000000ULL) \|
	((in >> 8) & 0x00000000ff000000ULL) \|
	((in >> 24) & 0x0000000000ff0000ULL) \|
	((in >> 40) & 0x000000000000ff00ULL) \|
	((in >> 56) & 0x00000000000000ffULL);
	#endif
	}

	MEM_STATIC size_t MEM_swapST(size_t in)
	{
	if (MEM_32bits())
	return (size_t)MEM_swap32((U32)in);
	else
	return (size_t)MEM_swap64((U64)in);
	}

	/=== Little endian r/w ===/

	MEM_STATIC U16 MEM_readLE16(const void* memPtr)
	{
	if (MEM_isLittleEndian())
	return MEM_read16(memPtr);
	else {
	const BYTE* p = (const BYTE*)memPtr;
	return (U16)(p[0] + (p[1]<<8));
	}
	}

	MEM_STATIC void MEM_writeLE16(void* memPtr, U16 val)
	{
	if (MEM_isLittleEndian()) {
	MEM_write16(memPtr, val);
	} else {
	BYTE* p = (BYTE*)memPtr;
	p[0] = (BYTE)val;
	p[1] = (BYTE)(val>>8);
	}
	}

	MEM_STATIC U32 MEM_readLE24(const void* memPtr)
	{
	return MEM_readLE16(memPtr) + (((const BYTE*)memPtr)[2] << 16);
	}

	MEM_STATIC void MEM_writeLE24(void* memPtr, U32 val)
	{
	MEM_writeLE16(memPtr, (U16)val);
	((BYTE*)memPtr)[2] = (BYTE)(val>>16);
	}

	MEM_STATIC U32 MEM_readLE32(const void* memPtr)
	{
	if (MEM_isLittleEndian())
	return MEM_read32(memPtr);
	else
	return MEM_swap32(MEM_read32(memPtr));
	}

	MEM_STATIC void MEM_writeLE32(void* memPtr, U32 val32)
	{
	if (MEM_isLittleEndian())
	MEM_write32(memPtr, val32);
	else
	MEM_write32(memPtr, MEM_swap32(val32));
	}

	MEM_STATIC U64 MEM_readLE64(const void* memPtr)
	{
	if (MEM_isLittleEndian())
	return MEM_read64(memPtr);
	else
	return MEM_swap64(MEM_read64(memPtr));
	}

	MEM_STATIC void MEM_writeLE64(void* memPtr, U64 val64)
	{
	if (MEM_isLittleEndian())
	MEM_write64(memPtr, val64);
	else
	MEM_write64(memPtr, MEM_swap64(val64));
	}

	MEM_STATIC size_t MEM_readLEST(const void* memPtr)
	{
	if (MEM_32bits())
	return (size_t)MEM_readLE32(memPtr);
	else
	return (size_t)MEM_readLE64(memPtr);
	}

	MEM_STATIC void MEM_writeLEST(void* memPtr, size_t val)
	{
	if (MEM_32bits())
	MEM_writeLE32(memPtr, (U32)val);
	else
	MEM_writeLE64(memPtr, (U64)val);
	}

	/=== Big endian r/w ===/

	MEM_STATIC U32 MEM_readBE32(const void* memPtr)
	{
	if (MEM_isLittleEndian())
	return MEM_swap32(MEM_read32(memPtr));
	else
	return MEM_read32(memPtr);
	}

	MEM_STATIC void MEM_writeBE32(void* memPtr, U32 val32)
	{
	if (MEM_isLittleEndian())
	MEM_write32(memPtr, MEM_swap32(val32));
	else
	MEM_write32(memPtr, val32);
	}

	MEM_STATIC U64 MEM_readBE64(const void* memPtr)
	{
	if (MEM_isLittleEndian())
	return MEM_swap64(MEM_read64(memPtr));
	else
	return MEM_read64(memPtr);
	}

	MEM_STATIC void MEM_writeBE64(void* memPtr, U64 val64)
	{
	if (MEM_isLittleEndian())
	MEM_write64(memPtr, MEM_swap64(val64));
	else
	MEM_write64(memPtr, val64);
	}

	MEM_STATIC size_t MEM_readBEST(const void* memPtr)
	{
	if (MEM_32bits())
	return (size_t)MEM_readBE32(memPtr);
	else
	return (size_t)MEM_readBE64(memPtr);
	}

	MEM_STATIC void MEM_writeBEST(void* memPtr, size_t val)
	{
	if (MEM_32bits())
	MEM_writeBE32(memPtr, (U32)val);
	else
	MEM_writeBE64(memPtr, (U64)val);
	}


	#if defined (__cplusplus)
	}
	#endif

	#endif /* MEM_H_MODULE */
	/** ended inlining mem.h **/
	/** start inlining error_private.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	/* Note : this module is expected to remain private, do not expose it */

	#ifndef ERROR_H_MODULE
	#define ERROR_H_MODULE

	#if defined (__cplusplus)
	extern "C" {
	#endif


	/* ****************************************
	* Dependencies
	******************************************/
	#include <stddef.h> /* size_t */
	/** start inlining zstd_errors.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTD_ERRORS_H_398273423
	#define ZSTD_ERRORS_H_398273423

	#if defined (__cplusplus)
	extern "C" {
	#endif

	/===== dependency =====/
	#include <stddef.h> /* size_t */


	/* ===== ZSTDERRORLIB_API : control library symbols visibility ===== */
	#ifndef ZSTDERRORLIB_VISIBILITY
	# if defined(__GNUC__) && (__GNUC__ >= 4)
	# define ZSTDERRORLIB_VISIBILITY __attribute__ ((visibility ("default")))
	# else
	# define ZSTDERRORLIB_VISIBILITY
	# endif
	#endif
	#if defined(ZSTD_DLL_EXPORT) && (ZSTD_DLL_EXPORT==1)
	# define ZSTDERRORLIB_API __declspec(dllexport) ZSTDERRORLIB_VISIBILITY
	#elif defined(ZSTD_DLL_IMPORT) && (ZSTD_DLL_IMPORT==1)
	# define ZSTDERRORLIB_API __declspec(dllimport) ZSTDERRORLIB_VISIBILITY /* It isn't required but allows to generate better code, saving a function pointer load from the IAT and an indirect jump.*/
	#else
	# define ZSTDERRORLIB_API ZSTDERRORLIB_VISIBILITY
	#endif

	/-********************************************
	* Error codes list
	-********************************************
	* Error codes _values_ are pinned down since v1.3.1 only.
	* Therefore, don't rely on values if you may link to any version < v1.3.1.
	*
	* Only values < 100 are considered stable.
	*
	* note 1 : this API shall be used with static linking only.
	* dynamic linking is not yet officially supported.
	* note 2 : Prefer relying on the enum than on its value whenever possible
	* This is the only supported way to use the error list < v1.3.1
	* note 3 : ZSTD_isError() is always correct, whatever the library version.
	**********************************************/
	typedef enum {
	ZSTD_error_no_error = 0,
	ZSTD_error_GENERIC = 1,
	ZSTD_error_prefix_unknown = 10,
	ZSTD_error_version_unsupported = 12,
	ZSTD_error_frameParameter_unsupported = 14,
	ZSTD_error_frameParameter_windowTooLarge = 16,
	ZSTD_error_corruption_detected = 20,
	ZSTD_error_checksum_wrong = 22,
	ZSTD_error_dictionary_corrupted = 30,
	ZSTD_error_dictionary_wrong = 32,
	ZSTD_error_dictionaryCreation_failed = 34,
	ZSTD_error_parameter_unsupported = 40,
	ZSTD_error_parameter_outOfBound = 42,
	ZSTD_error_tableLog_tooLarge = 44,
	ZSTD_error_maxSymbolValue_tooLarge = 46,
	ZSTD_error_maxSymbolValue_tooSmall = 48,
	ZSTD_error_stage_wrong = 60,
	ZSTD_error_init_missing = 62,
	ZSTD_error_memory_allocation = 64,
	ZSTD_error_workSpace_tooSmall= 66,
	ZSTD_error_dstSize_tooSmall = 70,
	ZSTD_error_srcSize_wrong = 72,
	ZSTD_error_dstBuffer_null = 74,
	/* following error codes are __NOT STABLE__, they can be removed or changed in future versions */
	ZSTD_error_frameIndex_tooLarge = 100,
	ZSTD_error_seekableIO = 102,
	ZSTD_error_dstBuffer_wrong = 104,
	ZSTD_error_maxCode = 120 /* never EVER use this value directly, it can change in future versions! Use ZSTD_isError() instead */
	} ZSTD_ErrorCode;

	/*! ZSTD_getErrorCode() :
	convert a `size_t` function result into a `ZSTD_ErrorCode` enum type,
	which can be used to compare with enum list published above */
	ZSTDERRORLIB_API ZSTD_ErrorCode ZSTD_getErrorCode(size_t functionResult);
	ZSTDERRORLIB_API const char* ZSTD_getErrorString(ZSTD_ErrorCode code); /*< Same as ZSTD_getErrorName, but using a `ZSTD_ErrorCode` enum argument /


	#if defined (__cplusplus)
	}
	#endif

	#endif /* ZSTD_ERRORS_H_398273423 */
	/** ended inlining zstd_errors.h **/


	/* ****************************************
	* Compiler-specific
	******************************************/
	#if defined(__GNUC__)
	# define ERR_STATIC static __attribute__((unused))
	#elif defined (__cplusplus) \|\| (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */)
	# define ERR_STATIC static inline
	#elif defined(_MSC_VER)
	# define ERR_STATIC static __inline
	#else
	# define ERR_STATIC static /* this version may generate warnings for unused static functions; disable the relevant warning */
	#endif


	/-***************************************
	* Customization (error_public.h)
	******************************************/
	typedef ZSTD_ErrorCode ERR_enum;
	#define PREFIX(name) ZSTD_error_##name


	/-***************************************
	* Error codes handling
	******************************************/
	#undef ERROR /* already defined on Visual Studio */
	#define ERROR(name) ZSTD_ERROR(name)
	#define ZSTD_ERROR(name) ((size_t)-PREFIX(name))

	ERR_STATIC unsigned ERR_isError(size_t code) { return (code > ERROR(maxCode)); }

	ERR_STATIC ERR_enum ERR_getErrorCode(size_t code) { if (!ERR_isError(code)) return (ERR_enum)0; return (ERR_enum) (0-code); }

	/* check and forward error code */
	#define CHECK_V_F(e, f) size_t const e = f; if (ERR_isError(e)) return e
	#define CHECK_F(f) { CHECK_V_F(_var_err__, f); }


	/-***************************************
	* Error Strings
	******************************************/

	const char* ERR_getErrorString(ERR_enum code); /* error_private.c */

	ERR_STATIC const char* ERR_getErrorName(size_t code)
	{
	return ERR_getErrorString(ERR_getErrorCode(code));
	}

	#if defined (__cplusplus)
	}
	#endif

	#endif /* ERROR_H_MODULE */
	/** ended inlining error_private.h **/
	#define FSE_STATIC_LINKING_ONLY /* FSE_MIN_TABLELOG */
	/** start inlining fse.h **/
	/* ******************************************************************
	* FSE : Finite State Entropy codec
	* Public Prototypes declaration
	* Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
	*
	* You can contact the author at :
	* - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	****************************************************************** */

	#if defined (__cplusplus)
	extern "C" {
	#endif

	#ifndef FSE_H
	#define FSE_H


	/-****************************************
	* Dependencies
	******************************************/
	#include <stddef.h> /* size_t, ptrdiff_t */


	/-****************************************
	* FSE_PUBLIC_API : control library symbols visibility
	******************************************/
	#if defined(FSE_DLL_EXPORT) && (FSE_DLL_EXPORT==1) && defined(__GNUC__) && (__GNUC__ >= 4)
	# define FSE_PUBLIC_API __attribute__ ((visibility ("default")))
	#elif defined(FSE_DLL_EXPORT) && (FSE_DLL_EXPORT==1) /* Visual expected */
	# define FSE_PUBLIC_API __declspec(dllexport)
	#elif defined(FSE_DLL_IMPORT) && (FSE_DLL_IMPORT==1)
	# define FSE_PUBLIC_API __declspec(dllimport) /* It isn't required but allows to generate better code, saving a function pointer load from the IAT and an indirect jump.*/
	#else
	# define FSE_PUBLIC_API
	#endif

	/------ Version ------/
	#define FSE_VERSION_MAJOR 0
	#define FSE_VERSION_MINOR 9
	#define FSE_VERSION_RELEASE 0

	#define FSE_LIB_VERSION FSE_VERSION_MAJOR.FSE_VERSION_MINOR.FSE_VERSION_RELEASE
	#define FSE_QUOTE(str) #str
	#define FSE_EXPAND_AND_QUOTE(str) FSE_QUOTE(str)
	#define FSE_VERSION_STRING FSE_EXPAND_AND_QUOTE(FSE_LIB_VERSION)

	#define FSE_VERSION_NUMBER (FSE_VERSION_MAJOR 100100 + FSE_VERSION_MINOR *100 + FSE_VERSION_RELEASE)
	FSE_PUBLIC_API unsigned FSE_versionNumber(void); /*< library version number; to be used when checking dll version /


	/-***************************************
	* FSE simple functions
	******************************************/
	/*! FSE_compress() :
	Compress content of buffer 'src', of size 'srcSize', into destination buffer 'dst'.
	'dst' buffer must be already allocated. Compression runs faster is dstCapacity >= FSE_compressBound(srcSize).
	@return : size of compressed data (<= dstCapacity).
	Special values : if return == 0, srcData is not compressible => Nothing is stored within dst !!!
	if return == 1, srcData is a single byte symbol * srcSize times. Use RLE compression instead.
	if FSE_isError(return), compression failed (more details using FSE_getErrorName())
	*/
	FSE_PUBLIC_API size_t FSE_compress(void* dst, size_t dstCapacity,
	const void* src, size_t srcSize);

	/*! FSE_decompress():
	Decompress FSE data from buffer 'cSrc', of size 'cSrcSize',
	into already allocated destination buffer 'dst', of size 'dstCapacity'.
	@return : size of regenerated data (<= maxDstSize),
	or an error code, which can be tested using FSE_isError() .

	Important : FSE_decompress() does not decompress non-compressible nor RLE data !!!
	Why ? : making this distinction requires a header.
	Header management is intentionally delegated to the user layer, which can better manage special cases.
	*/
	FSE_PUBLIC_API size_t FSE_decompress(void* dst, size_t dstCapacity,
	const void* cSrc, size_t cSrcSize);


	/-****************************************
	* Tool functions
	******************************************/
	FSE_PUBLIC_API size_t FSE_compressBound(size_t size); /* maximum compressed size */

	/* Error Management */
	FSE_PUBLIC_API unsigned FSE_isError(size_t code); /* tells if a return value is an error code */
	FSE_PUBLIC_API const char* FSE_getErrorName(size_t code); /* provides error code string (useful for debugging) */


	/-****************************************
	* FSE advanced functions
	******************************************/
	/*! FSE_compress2() :
	Same as FSE_compress(), but allows the selection of 'maxSymbolValue' and 'tableLog'
	Both parameters can be defined as '0' to mean : use default value
	@return : size of compressed data
	Special values : if return == 0, srcData is not compressible => Nothing is stored within cSrc !!!
	if return == 1, srcData is a single byte symbol * srcSize times. Use RLE compression.
	if FSE_isError(return), it's an error code.
	*/
	FSE_PUBLIC_API size_t FSE_compress2 (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog);


	/-****************************************
	* FSE detailed API
	******************************************/
	/*!
	FSE_compress() does the following:
	1. count symbol occurrence from source[] into table count[] (see hist.h)
	2. normalize counters so that sum(count[]) == Power_of_2 (2^tableLog)
	3. save normalized counters to memory buffer using writeNCount()
	4. build encoding table 'CTable' from normalized counters
	5. encode the data stream using encoding table 'CTable'

	FSE_decompress() does the following:
	1. read normalized counters with readNCount()
	2. build decoding table 'DTable' from normalized counters
	3. decode the data stream using decoding table 'DTable'

	The following API allows targeting specific sub-functions for advanced tasks.
	For example, it's possible to compress several blocks using the same 'CTable',
	or to save and provide normalized distribution using external method.
	*/

	/* * COMPRESSION * */

	/*! FSE_optimalTableLog():
	dynamically downsize 'tableLog' when conditions are met.
	It saves CPU time, by using smaller tables, while preserving or even improving compression ratio.
	@return : recommended tableLog (necessarily <= 'maxTableLog') */
	FSE_PUBLIC_API unsigned FSE_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue);

	/*! FSE_normalizeCount():
	normalize counts so that sum(count[]) == Power_of_2 (2^tableLog)
	'normalizedCounter' is a table of short, of minimum size (maxSymbolValue+1).
	@return : tableLog,
	or an errorCode, which can be tested using FSE_isError() */
	FSE_PUBLIC_API size_t FSE_normalizeCount(short* normalizedCounter, unsigned tableLog,
	const unsigned* count, size_t srcSize, unsigned maxSymbolValue);

	/*! FSE_NCountWriteBound():
	Provides the maximum possible size of an FSE normalized table, given 'maxSymbolValue' and 'tableLog'.
	Typically useful for allocation purpose. */
	FSE_PUBLIC_API size_t FSE_NCountWriteBound(unsigned maxSymbolValue, unsigned tableLog);

	/*! FSE_writeNCount():
	Compactly save 'normalizedCounter' into 'buffer'.
	@return : size of the compressed table,
	or an errorCode, which can be tested using FSE_isError(). */
	FSE_PUBLIC_API size_t FSE_writeNCount (void* buffer, size_t bufferSize,
	const short* normalizedCounter,
	unsigned maxSymbolValue, unsigned tableLog);

	/*! Constructor and Destructor of FSE_CTable.
	Note that FSE_CTable size depends on 'tableLog' and 'maxSymbolValue' */
	typedef unsigned FSE_CTable; /* don't allocate that. It's only meant to be more restrictive than void* */
	FSE_PUBLIC_API FSE_CTable* FSE_createCTable (unsigned maxSymbolValue, unsigned tableLog);
	FSE_PUBLIC_API void FSE_freeCTable (FSE_CTable* ct);

	/*! FSE_buildCTable():
	Builds `ct`, which must be already allocated, using FSE_createCTable().
	@return : 0, or an errorCode, which can be tested using FSE_isError() */
	FSE_PUBLIC_API size_t FSE_buildCTable(FSE_CTable* ct, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog);

	/*! FSE_compress_usingCTable():
	Compress `src` using `ct` into `dst` which must be already allocated.
	@return : size of compressed data (<= `dstCapacity`),
	or 0 if compressed data could not fit into `dst`,
	or an errorCode, which can be tested using FSE_isError() */
	FSE_PUBLIC_API size_t FSE_compress_usingCTable (void* dst, size_t dstCapacity, const void* src, size_t srcSize, const FSE_CTable* ct);

	/*!
	Tutorial :
	----------
	The first step is to count all symbols. FSE_count() does this job very fast.
	Result will be saved into 'count', a table of unsigned int, which must be already allocated, and have 'maxSymbolValuePtr[0]+1' cells.
	'src' is a table of bytes of size 'srcSize'. All values within 'src' MUST be <= maxSymbolValuePtr[0]
	maxSymbolValuePtr[0] will be updated, with its real value (necessarily <= original value)
	FSE_count() will return the number of occurrence of the most frequent symbol.
	This can be used to know if there is a single symbol within 'src', and to quickly evaluate its compressibility.
	If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError()).

	The next step is to normalize the frequencies.
	FSE_normalizeCount() will ensure that sum of frequencies is == 2 ^'tableLog'.
	It also guarantees a minimum of 1 to any Symbol with frequency >= 1.
	You can use 'tableLog'==0 to mean "use default tableLog value".
	If you are unsure of which tableLog value to use, you can ask FSE_optimalTableLog(),
	which will provide the optimal valid tableLog given sourceSize, maxSymbolValue, and a user-defined maximum (0 means "default").

	The result of FSE_normalizeCount() will be saved into a table,
	called 'normalizedCounter', which is a table of signed short.
	'normalizedCounter' must be already allocated, and have at least 'maxSymbolValue+1' cells.
	The return value is tableLog if everything proceeded as expected.
	It is 0 if there is a single symbol within distribution.
	If there is an error (ex: invalid tableLog value), the function will return an ErrorCode (which can be tested using FSE_isError()).

	'normalizedCounter' can be saved in a compact manner to a memory area using FSE_writeNCount().
	'buffer' must be already allocated.
	For guaranteed success, buffer size must be at least FSE_headerBound().
	The result of the function is the number of bytes written into 'buffer'.
	If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError(); ex : buffer size too small).

	'normalizedCounter' can then be used to create the compression table 'CTable'.
	The space required by 'CTable' must be already allocated, using FSE_createCTable().
	You can then use FSE_buildCTable() to fill 'CTable'.
	If there is an error, both functions will return an ErrorCode (which can be tested using FSE_isError()).

	'CTable' can then be used to compress 'src', with FSE_compress_usingCTable().
	Similar to FSE_count(), the convention is that 'src' is assumed to be a table of char of size 'srcSize'
	The function returns the size of compressed data (without header), necessarily <= `dstCapacity`.
	If it returns '0', compressed data could not fit into 'dst'.
	If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError()).
	*/


	/* * DECOMPRESSION * */

	/*! FSE_readNCount():
	Read compactly saved 'normalizedCounter' from 'rBuffer'.
	@return : size read from 'rBuffer',
	or an errorCode, which can be tested using FSE_isError().
	maxSymbolValuePtr[0] and tableLogPtr[0] will also be updated with their respective values */
	FSE_PUBLIC_API size_t FSE_readNCount (short* normalizedCounter,
	unsigned* maxSymbolValuePtr, unsigned* tableLogPtr,
	const void* rBuffer, size_t rBuffSize);

	/*! Constructor and Destructor of FSE_DTable.
	Note that its size depends on 'tableLog' */
	typedef unsigned FSE_DTable; /* don't allocate that. It's just a way to be more restrictive than void* */
	FSE_PUBLIC_API FSE_DTable* FSE_createDTable(unsigned tableLog);
	FSE_PUBLIC_API void FSE_freeDTable(FSE_DTable* dt);

	/*! FSE_buildDTable():
	Builds 'dt', which must be already allocated, using FSE_createDTable().
	return : 0, or an errorCode, which can be tested using FSE_isError() */
	FSE_PUBLIC_API size_t FSE_buildDTable (FSE_DTable* dt, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog);

	/*! FSE_decompress_usingDTable():
	Decompress compressed source `cSrc` of size `cSrcSize` using `dt`
	into `dst` which must be already allocated.
	@return : size of regenerated data (necessarily <= `dstCapacity`),
	or an errorCode, which can be tested using FSE_isError() */
	FSE_PUBLIC_API size_t FSE_decompress_usingDTable(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, const FSE_DTable* dt);

	/*!
	Tutorial :
	----------
	(Note : these functions only decompress FSE-compressed blocks.
	If block is uncompressed, use memcpy() instead
	If block is a single repeated byte, use memset() instead )

	The first step is to obtain the normalized frequencies of symbols.
	This can be performed by FSE_readNCount() if it was saved using FSE_writeNCount().
	'normalizedCounter' must be already allocated, and have at least 'maxSymbolValuePtr[0]+1' cells of signed short.
	In practice, that means it's necessary to know 'maxSymbolValue' beforehand,
	or size the table to handle worst case situations (typically 256).
	FSE_readNCount() will provide 'tableLog' and 'maxSymbolValue'.
	The result of FSE_readNCount() is the number of bytes read from 'rBuffer'.
	Note that 'rBufferSize' must be at least 4 bytes, even if useful information is less than that.
	If there is an error, the function will return an error code, which can be tested using FSE_isError().

	The next step is to build the decompression tables 'FSE_DTable' from 'normalizedCounter'.
	This is performed by the function FSE_buildDTable().
	The space required by 'FSE_DTable' must be already allocated using FSE_createDTable().
	If there is an error, the function will return an error code, which can be tested using FSE_isError().

	`FSE_DTable` can then be used to decompress `cSrc`, with FSE_decompress_usingDTable().
	`cSrcSize` must be strictly correct, otherwise decompression will fail.
	FSE_decompress_usingDTable() result will tell how many bytes were regenerated (<=`dstCapacity`).
	If there is an error, the function will return an error code, which can be tested using FSE_isError(). (ex: dst buffer too small)
	*/

	#endif /* FSE_H */

	#if defined(FSE_STATIC_LINKING_ONLY) && !defined(FSE_H_FSE_STATIC_LINKING_ONLY)
	#define FSE_H_FSE_STATIC_LINKING_ONLY

	/* * Dependency * */
	/** start inlining bitstream.h **/
	/* ******************************************************************
	* bitstream
	* Part of FSE library
	* Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
	*
	* You can contact the author at :
	* - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	****************************************************************** */
	#ifndef BITSTREAM_H_MODULE
	#define BITSTREAM_H_MODULE

	#if defined (__cplusplus)
	extern "C" {
	#endif

	/*
	* This API consists of small unitary functions, which must be inlined for best performance.
	* Since link-time-optimization is not available for all compilers,
	* these functions are defined into a .h to be included.
	*/

	/-***************************************
	* Dependencies
	******************************************/
	/** skipping file: mem.h **/
	/** start inlining compiler.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTD_COMPILER_H
	#define ZSTD_COMPILER_H

	/-******************************************************
	* Compiler specifics
	*********************************************************/
	/* force inlining */

	#if !defined(ZSTD_NO_INLINE)
	#if (defined(__GNUC__) && !defined(__STRICT_ANSI__)) \|\| defined(__cplusplus) \|\| defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */
	# define INLINE_KEYWORD inline
	#else
	# define INLINE_KEYWORD
	#endif

	#if defined(__GNUC__) \|\| defined(__ICCARM__)
	# define FORCE_INLINE_ATTR __attribute__((always_inline))
	#elif defined(_MSC_VER)
	# define FORCE_INLINE_ATTR __forceinline
	#else
	# define FORCE_INLINE_ATTR
	#endif

	#else

	#define INLINE_KEYWORD
	#define FORCE_INLINE_ATTR

	#endif

	/**
	* FORCE_INLINE_TEMPLATE is used to define C "templates", which take constant
	* parameters. They must be inlined for the compiler to eliminate the constant
	* branches.
	*/
	#define FORCE_INLINE_TEMPLATE static INLINE_KEYWORD FORCE_INLINE_ATTR
	/**
	* HINT_INLINE is used to help the compiler generate better code. It is not
	* used for "templates", so it can be tweaked based on the compilers
	* performance.
	*
	* gcc-4.8 and gcc-4.9 have been shown to benefit from leaving off the
	* always_inline attribute.
	*
	* clang up to 5.0.0 (trunk) benefit tremendously from the always_inline
	* attribute.
	*/
	#if !defined(__clang__) && defined(__GNUC__) && __GNUC__ >= 4 && __GNUC_MINOR__ >= 8 && __GNUC__ < 5
	# define HINT_INLINE static INLINE_KEYWORD
	#else
	# define HINT_INLINE static INLINE_KEYWORD FORCE_INLINE_ATTR
	#endif

	/* UNUSED_ATTR tells the compiler it is okay if the function is unused. */
	#if defined(__GNUC__)
	# define UNUSED_ATTR __attribute__((unused))
	#else
	# define UNUSED_ATTR
	#endif

	/* force no inlining */
	#ifdef _MSC_VER
	# define FORCE_NOINLINE static __declspec(noinline)
	#else
	# if defined(__GNUC__) \|\| defined(__ICCARM__)
	# define FORCE_NOINLINE static __attribute__((__noinline__))
	# else
	# define FORCE_NOINLINE static
	# endif
	#endif

	/* target attribute */
	#ifndef __has_attribute
	#define __has_attribute(x) 0 /* Compatibility with non-clang compilers. */
	#endif
	#if defined(__GNUC__) \|\| defined(__ICCARM__)
	# define TARGET_ATTRIBUTE(target) __attribute__((__target__(target)))
	#else
	# define TARGET_ATTRIBUTE(target)
	#endif

	/* Enable runtime BMI2 dispatch based on the CPU.
	* Enabled for clang & gcc >=4.8 on x86 when BMI2 isn't enabled by default.
	*/
	#ifndef DYNAMIC_BMI2
	#if ((defined(__clang__) && __has_attribute(__target__)) \
	\|\| (defined(__GNUC__) \
	&& (__GNUC__ >= 5 \|\| (__GNUC__ == 4 && __GNUC_MINOR__ >= 8)))) \
	&& (defined(__x86_64__) \|\| defined(_M_X86)) \
	&& !defined(__BMI2__)
	# define DYNAMIC_BMI2 1
	#else
	# define DYNAMIC_BMI2 0
	#endif
	#endif

	/* prefetch
	* can be disabled, by declaring NO_PREFETCH build macro */
	#if defined(NO_PREFETCH)
	# define PREFETCH_L1(ptr) (void)(ptr) /* disabled */
	# define PREFETCH_L2(ptr) (void)(ptr) /* disabled */
	#else
	# if defined(_MSC_VER) && (defined(_M_X64) \|\| defined(_M_I86)) /* _mm_prefetch() is not defined outside of x86/x64 */
	# include <mmintrin.h> /* https://msdn.microsoft.com/fr-fr/library/84szxsww(v=vs.90).aspx */
	# define PREFETCH_L1(ptr) _mm_prefetch((const char*)(ptr), _MM_HINT_T0)
	# define PREFETCH_L2(ptr) _mm_prefetch((const char*)(ptr), _MM_HINT_T1)
	# elif defined(__aarch64__)
	# define PREFETCH_L1(ptr) __asm__ __volatile__("prfm pldl1keep, %0" ::"Q"(*(ptr)))
	# define PREFETCH_L2(ptr) __asm__ __volatile__("prfm pldl2keep, %0" ::"Q"(*(ptr)))
	# elif defined(__GNUC__) && ( (__GNUC__ >= 4) \|\| ( (__GNUC__ == 3) && (__GNUC_MINOR__ >= 1) ) )
	# define PREFETCH_L1(ptr) __builtin_prefetch((ptr), 0 /* rw==read /, 3 / locality */)
	# define PREFETCH_L2(ptr) __builtin_prefetch((ptr), 0 /* rw==read /, 2 / locality */)
	# else
	# define PREFETCH_L1(ptr) (void)(ptr) /* disabled */
	# define PREFETCH_L2(ptr) (void)(ptr) /* disabled */
	# endif
	#endif /* NO_PREFETCH */

	#define CACHELINE_SIZE 64

	#define PREFETCH_AREA(p, s) { \
	const char* const _ptr = (const char*)(p); \
	size_t const _size = (size_t)(s); \
	size_t _pos; \
	for (_pos=0; _pos<_size; _pos+=CACHELINE_SIZE) { \
	PREFETCH_L2(_ptr + _pos); \
	} \
	}

	/* vectorization
	* older GCC (pre gcc-4.3 picked as the cutoff) uses a different syntax */
	#if !defined(__INTEL_COMPILER) && !defined(__clang__) && defined(__GNUC__)
	# if (__GNUC__ == 4 && __GNUC_MINOR__ > 3) \|\| (__GNUC__ >= 5)
	# define DONT_VECTORIZE __attribute__((optimize("no-tree-vectorize")))
	# else
	# define DONT_VECTORIZE _Pragma("GCC optimize(\"no-tree-vectorize\")")
	# endif
	#else
	# define DONT_VECTORIZE
	#endif

	/* Tell the compiler that a branch is likely or unlikely.
	* Only use these macros if it causes the compiler to generate better code.
	* If you can remove a LIKELY/UNLIKELY annotation without speed changes in gcc
	* and clang, please do.
	*/
	#if defined(__GNUC__)
	#define LIKELY(x) (__builtin_expect((x), 1))
	#define UNLIKELY(x) (__builtin_expect((x), 0))
	#else
	#define LIKELY(x) (x)
	#define UNLIKELY(x) (x)
	#endif

	/* disable warnings */
	#ifdef _MSC_VER /* Visual Studio */
	# include <intrin.h> /* For Visual 2005 */
	# pragma warning(disable : 4100) /* disable: C4100: unreferenced formal parameter */
	# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */
	# pragma warning(disable : 4204) /* disable: C4204: non-constant aggregate initializer */
	# pragma warning(disable : 4214) /* disable: C4214: non-int bitfields */
	# pragma warning(disable : 4324) /* disable: C4324: padded structure */
	#endif

	#endif /* ZSTD_COMPILER_H */
	/** ended inlining compiler.h **/
	/** skipping file: debug.h **/
	/** skipping file: error_private.h **/


	/*=========================================
	* Target specific
	=========================================*/
	#if defined(__BMI__) && defined(__GNUC__)
	# include <immintrin.h> /* support for bextr (experimental) */
	#elif defined(__ICCARM__)
	# include <intrinsics.h>
	#endif

	#define STREAM_ACCUMULATOR_MIN_32 25
	#define STREAM_ACCUMULATOR_MIN_64 57
	#define STREAM_ACCUMULATOR_MIN ((U32)(MEM_32bits() ? STREAM_ACCUMULATOR_MIN_32 : STREAM_ACCUMULATOR_MIN_64))


	/-*****************************************
	* bitStream encoding API (write forward)
	********************************************/
	/* bitStream can mix input from multiple sources.
	* A critical property of these streams is that they encode and decode in reverse direction.
	* So the first bit sequence you add will be the last to be read, like a LIFO stack.
	*/
	typedef struct {
	size_t bitContainer;
	unsigned bitPos;
	char* startPtr;
	char* ptr;
	char* endPtr;
	} BIT_CStream_t;

	MEM_STATIC size_t BIT_initCStream(BIT_CStream_t* bitC, void* dstBuffer, size_t dstCapacity);
	MEM_STATIC void BIT_addBits(BIT_CStream_t* bitC, size_t value, unsigned nbBits);
	MEM_STATIC void BIT_flushBits(BIT_CStream_t* bitC);
	MEM_STATIC size_t BIT_closeCStream(BIT_CStream_t* bitC);

	/* Start with initCStream, providing the size of buffer to write into.
	* bitStream will never write outside of this buffer.
	* `dstCapacity` must be >= sizeof(bitD->bitContainer), otherwise @return will be an error code.
	*
	* bits are first added to a local register.
	* Local register is size_t, hence 64-bits on 64-bits systems, or 32-bits on 32-bits systems.
	* Writing data into memory is an explicit operation, performed by the flushBits function.
	* Hence keep track how many bits are potentially stored into local register to avoid register overflow.
	* After a flushBits, a maximum of 7 bits might still be stored into local register.
	*
	* Avoid storing elements of more than 24 bits if you want compatibility with 32-bits bitstream readers.
	*
	* Last operation is to close the bitStream.
	* The function returns the final size of CStream in bytes.
	* If data couldn't fit into `dstBuffer`, it will return a 0 ( == not storable)
	*/


	/-*******************************************
	* bitStream decoding API (read backward)
	**********************************************/
	typedef struct {
	size_t bitContainer;
	unsigned bitsConsumed;
	const char* ptr;
	const char* start;
	const char* limitPtr;
	} BIT_DStream_t;

	typedef enum { BIT_DStream_unfinished = 0,
	BIT_DStream_endOfBuffer = 1,
	BIT_DStream_completed = 2,
	BIT_DStream_overflow = 3 } BIT_DStream_status; /* result of BIT_reloadDStream() */
	/* 1,2,4,8 would be better for bitmap combinations, but slows down performance a bit ... :( */

	MEM_STATIC size_t BIT_initDStream(BIT_DStream_t* bitD, const void* srcBuffer, size_t srcSize);
	MEM_STATIC size_t BIT_readBits(BIT_DStream_t* bitD, unsigned nbBits);
	MEM_STATIC BIT_DStream_status BIT_reloadDStream(BIT_DStream_t* bitD);
	MEM_STATIC unsigned BIT_endOfDStream(const BIT_DStream_t* bitD);


	/* Start by invoking BIT_initDStream().
	* A chunk of the bitStream is then stored into a local register.
	* Local register size is 64-bits on 64-bits systems, 32-bits on 32-bits systems (size_t).
	* You can then retrieve bitFields stored into the local register, in reverse order.
	* Local register is explicitly reloaded from memory by the BIT_reloadDStream() method.
	* A reload guarantee a minimum of ((8*sizeof(bitD->bitContainer))-7) bits when its result is BIT_DStream_unfinished.
	* Otherwise, it can be less than that, so proceed accordingly.
	* Checking if DStream has reached its end can be performed with BIT_endOfDStream().
	*/


	/-***************************************
	* unsafe API
	******************************************/
	MEM_STATIC void BIT_addBitsFast(BIT_CStream_t* bitC, size_t value, unsigned nbBits);
	/* faster, but works only if value is "clean", meaning all high bits above nbBits are 0 */

	MEM_STATIC void BIT_flushBitsFast(BIT_CStream_t* bitC);
	/* unsafe version; does not check buffer overflow */

	MEM_STATIC size_t BIT_readBitsFast(BIT_DStream_t* bitD, unsigned nbBits);
	/* faster, but works only if nbBits >= 1 */



	/-*************************************************************
	* Internal functions
	****************************************************************/
	MEM_STATIC unsigned BIT_highbit32 (U32 val)
	{
	assert(val != 0);
	{
	# if defined(_MSC_VER) /* Visual */
	unsigned long r=0;
	return _BitScanReverse ( &r, val ) ? (unsigned)r : 0;
	# elif defined(__GNUC__) && (__GNUC__ >= 3) /* Use GCC Intrinsic */
	return __builtin_clz (val) ^ 31;
	# elif defined(__ICCARM__) /* IAR Intrinsic */
	return 31 - __CLZ(val);
	# else /* Software version */
	static const unsigned DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29,
	11, 14, 16, 18, 22, 25, 3, 30,
	8, 12, 20, 28, 15, 17, 24, 7,
	19, 27, 23, 6, 26, 5, 4, 31 };
	U32 v = val;
	v \|= v >> 1;
	v \|= v >> 2;
	v \|= v >> 4;
	v \|= v >> 8;
	v \|= v >> 16;
	return DeBruijnClz[ (U32) (v * 0x07C4ACDDU) >> 27];
	# endif
	}
	}

	/===== Local Constants =====/
	static const unsigned BIT_mask[] = {
	0, 1, 3, 7, 0xF, 0x1F,
	0x3F, 0x7F, 0xFF, 0x1FF, 0x3FF, 0x7FF,
	0xFFF, 0x1FFF, 0x3FFF, 0x7FFF, 0xFFFF, 0x1FFFF,
	0x3FFFF, 0x7FFFF, 0xFFFFF, 0x1FFFFF, 0x3FFFFF, 0x7FFFFF,
	0xFFFFFF, 0x1FFFFFF, 0x3FFFFFF, 0x7FFFFFF, 0xFFFFFFF, 0x1FFFFFFF,
	0x3FFFFFFF, 0x7FFFFFFF}; /* up to 31 bits */
	#define BIT_MASK_SIZE (sizeof(BIT_mask) / sizeof(BIT_mask[0]))

	/-*************************************************************
	* bitStream encoding
	****************************************************************/
	/*! BIT_initCStream() :
	* `dstCapacity` must be > sizeof(size_t)
	* @return : 0 if success,
	* otherwise an error code (can be tested using ERR_isError()) */
	MEM_STATIC size_t BIT_initCStream(BIT_CStream_t* bitC,
	void* startPtr, size_t dstCapacity)
	{
	bitC->bitContainer = 0;
	bitC->bitPos = 0;
	bitC->startPtr = (char*)startPtr;
	bitC->ptr = bitC->startPtr;
	bitC->endPtr = bitC->startPtr + dstCapacity - sizeof(bitC->bitContainer);
	if (dstCapacity <= sizeof(bitC->bitContainer)) return ERROR(dstSize_tooSmall);
	return 0;
	}

	/*! BIT_addBits() :
	* can add up to 31 bits into `bitC`.
	* Note : does not check for register overflow ! */
	MEM_STATIC void BIT_addBits(BIT_CStream_t* bitC,
	size_t value, unsigned nbBits)
	{
	MEM_STATIC_ASSERT(BIT_MASK_SIZE == 32);
	assert(nbBits < BIT_MASK_SIZE);
	assert(nbBits + bitC->bitPos < sizeof(bitC->bitContainer) * 8);
	bitC->bitContainer \|= (value & BIT_mask[nbBits]) << bitC->bitPos;
	bitC->bitPos += nbBits;
	}

	/*! BIT_addBitsFast() :
	* works only if `value` is _clean_,
	* meaning all high bits above nbBits are 0 */
	MEM_STATIC void BIT_addBitsFast(BIT_CStream_t* bitC,
	size_t value, unsigned nbBits)
	{
	assert((value>>nbBits) == 0);
	assert(nbBits + bitC->bitPos < sizeof(bitC->bitContainer) * 8);
	bitC->bitContainer \|= value << bitC->bitPos;
	bitC->bitPos += nbBits;
	}

	/*! BIT_flushBitsFast() :
	* assumption : bitContainer has not overflowed
	* unsafe version; does not check buffer overflow */
	MEM_STATIC void BIT_flushBitsFast(BIT_CStream_t* bitC)
	{
	size_t const nbBytes = bitC->bitPos >> 3;
	assert(bitC->bitPos < sizeof(bitC->bitContainer) * 8);
	assert(bitC->ptr <= bitC->endPtr);
	MEM_writeLEST(bitC->ptr, bitC->bitContainer);
	bitC->ptr += nbBytes;
	bitC->bitPos &= 7;
	bitC->bitContainer >>= nbBytes*8;
	}

	/*! BIT_flushBits() :
	* assumption : bitContainer has not overflowed
	* safe version; check for buffer overflow, and prevents it.
	* note : does not signal buffer overflow.
	* overflow will be revealed later on using BIT_closeCStream() */
	MEM_STATIC void BIT_flushBits(BIT_CStream_t* bitC)
	{
	size_t const nbBytes = bitC->bitPos >> 3;
	assert(bitC->bitPos < sizeof(bitC->bitContainer) * 8);
	assert(bitC->ptr <= bitC->endPtr);
	MEM_writeLEST(bitC->ptr, bitC->bitContainer);
	bitC->ptr += nbBytes;
	if (bitC->ptr > bitC->endPtr) bitC->ptr = bitC->endPtr;
	bitC->bitPos &= 7;
	bitC->bitContainer >>= nbBytes*8;
	}

	/*! BIT_closeCStream() :
	* @return : size of CStream, in bytes,
	* or 0 if it could not fit into dstBuffer */
	MEM_STATIC size_t BIT_closeCStream(BIT_CStream_t* bitC)
	{
	BIT_addBitsFast(bitC, 1, 1); /* endMark */
	BIT_flushBits(bitC);
	if (bitC->ptr >= bitC->endPtr) return 0; /* overflow detected */
	return (bitC->ptr - bitC->startPtr) + (bitC->bitPos > 0);
	}


	/-*******************************************************
	* bitStream decoding
	**********************************************************/
	/*! BIT_initDStream() :
	* Initialize a BIT_DStream_t.
	* `bitD` : a pointer to an already allocated BIT_DStream_t structure.
	* `srcSize` must be the exact size of the bitStream, in bytes.
	* @return : size of stream (== srcSize), or an errorCode if a problem is detected
	*/
	MEM_STATIC size_t BIT_initDStream(BIT_DStream_t* bitD, const void* srcBuffer, size_t srcSize)
	{
	if (srcSize < 1) { memset(bitD, 0, sizeof(*bitD)); return ERROR(srcSize_wrong); }

	bitD->start = (const char*)srcBuffer;
	bitD->limitPtr = bitD->start + sizeof(bitD->bitContainer);

	if (srcSize >= sizeof(bitD->bitContainer)) { /* normal case */
	bitD->ptr = (const char*)srcBuffer + srcSize - sizeof(bitD->bitContainer);
	bitD->bitContainer = MEM_readLEST(bitD->ptr);
	{ BYTE const lastByte = ((const BYTE*)srcBuffer)[srcSize-1];
	bitD->bitsConsumed = lastByte ? 8 - BIT_highbit32(lastByte) : 0; /* ensures bitsConsumed is always set */
	if (lastByte == 0) return ERROR(GENERIC); /* endMark not present */ }
	} else {
	bitD->ptr = bitD->start;
	bitD->bitContainer = (const BYTE)(bitD->start);
	switch(srcSize)
	{
	case 7: bitD->bitContainer += (size_t)(((const BYTE)(srcBuffer))[6]) << (sizeof(bitD->bitContainer)8 - 16);
	/* fall-through */

	case 6: bitD->bitContainer += (size_t)(((const BYTE)(srcBuffer))[5]) << (sizeof(bitD->bitContainer)8 - 24);
	/* fall-through */

	case 5: bitD->bitContainer += (size_t)(((const BYTE)(srcBuffer))[4]) << (sizeof(bitD->bitContainer)8 - 32);
	/* fall-through */

	case 4: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[3]) << 24;
	/* fall-through */

	case 3: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[2]) << 16;
	/* fall-through */

	case 2: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[1]) << 8;
	/* fall-through */

	default: break;
	}
	{ BYTE const lastByte = ((const BYTE*)srcBuffer)[srcSize-1];
	bitD->bitsConsumed = lastByte ? 8 - BIT_highbit32(lastByte) : 0;
	if (lastByte == 0) return ERROR(corruption_detected); /* endMark not present */
	}
	bitD->bitsConsumed += (U32)(sizeof(bitD->bitContainer) - srcSize)*8;
	}

	return srcSize;
	}

	MEM_STATIC size_t BIT_getUpperBits(size_t bitContainer, U32 const start)
	{
	return bitContainer >> start;
	}

	MEM_STATIC size_t BIT_getMiddleBits(size_t bitContainer, U32 const start, U32 const nbBits)
	{
	U32 const regMask = sizeof(bitContainer)*8 - 1;
	/* if start > regMask, bitstream is corrupted, and result is undefined */
	assert(nbBits < BIT_MASK_SIZE);
	return (bitContainer >> (start & regMask)) & BIT_mask[nbBits];
	}

	MEM_STATIC size_t BIT_getLowerBits(size_t bitContainer, U32 const nbBits)
	{
	assert(nbBits < BIT_MASK_SIZE);
	return bitContainer & BIT_mask[nbBits];
	}

	/*! BIT_lookBits() :
	* Provides next n bits from local register.
	* local register is not modified.
	* On 32-bits, maxNbBits==24.
	* On 64-bits, maxNbBits==56.
	* @return : value extracted */
	MEM_STATIC size_t BIT_lookBits(const BIT_DStream_t* bitD, U32 nbBits)
	{
	/* arbitrate between double-shift and shift+mask */
	#if 1
	/* if bitD->bitsConsumed + nbBits > sizeof(bitD->bitContainer)*8,
	* bitstream is likely corrupted, and result is undefined */
	return BIT_getMiddleBits(bitD->bitContainer, (sizeof(bitD->bitContainer)*8) - bitD->bitsConsumed - nbBits, nbBits);
	#else
	/* this code path is slower on my os-x laptop */
	U32 const regMask = sizeof(bitD->bitContainer)*8 - 1;
	return ((bitD->bitContainer << (bitD->bitsConsumed & regMask)) >> 1) >> ((regMask-nbBits) & regMask);
	#endif
	}

	/*! BIT_lookBitsFast() :
	* unsafe version; only works if nbBits >= 1 */
	MEM_STATIC size_t BIT_lookBitsFast(const BIT_DStream_t* bitD, U32 nbBits)
	{
	U32 const regMask = sizeof(bitD->bitContainer)*8 - 1;
	assert(nbBits >= 1);
	return (bitD->bitContainer << (bitD->bitsConsumed & regMask)) >> (((regMask+1)-nbBits) & regMask);
	}

	MEM_STATIC void BIT_skipBits(BIT_DStream_t* bitD, U32 nbBits)
	{
	bitD->bitsConsumed += nbBits;
	}

	/*! BIT_readBits() :
	* Read (consume) next n bits from local register and update.
	* Pay attention to not read more than nbBits contained into local register.
	* @return : extracted value. */
	MEM_STATIC size_t BIT_readBits(BIT_DStream_t* bitD, unsigned nbBits)
	{
	size_t const value = BIT_lookBits(bitD, nbBits);
	BIT_skipBits(bitD, nbBits);
	return value;
	}

	/*! BIT_readBitsFast() :
	* unsafe version; only works only if nbBits >= 1 */
	MEM_STATIC size_t BIT_readBitsFast(BIT_DStream_t* bitD, unsigned nbBits)
	{
	size_t const value = BIT_lookBitsFast(bitD, nbBits);
	assert(nbBits >= 1);
	BIT_skipBits(bitD, nbBits);
	return value;
	}

	/*! BIT_reloadDStreamFast() :
	* Similar to BIT_reloadDStream(), but with two differences:
	* 1. bitsConsumed <= sizeof(bitD->bitContainer)*8 must hold!
	* 2. Returns BIT_DStream_overflow when bitD->ptr < bitD->limitPtr, at this
	* point you must use BIT_reloadDStream() to reload.
	*/
	MEM_STATIC BIT_DStream_status BIT_reloadDStreamFast(BIT_DStream_t* bitD)
	{
	if (UNLIKELY(bitD->ptr < bitD->limitPtr))
	return BIT_DStream_overflow;
	assert(bitD->bitsConsumed <= sizeof(bitD->bitContainer)*8);
	bitD->ptr -= bitD->bitsConsumed >> 3;
	bitD->bitsConsumed &= 7;
	bitD->bitContainer = MEM_readLEST(bitD->ptr);
	return BIT_DStream_unfinished;
	}

	/*! BIT_reloadDStream() :
	* Refill `bitD` from buffer previously set in BIT_initDStream() .
	* This function is safe, it guarantees it will not read beyond src buffer.
	* @return : status of `BIT_DStream_t` internal register.
	* when status == BIT_DStream_unfinished, internal register is filled with at least 25 or 57 bits */
	MEM_STATIC BIT_DStream_status BIT_reloadDStream(BIT_DStream_t* bitD)
	{
	if (bitD->bitsConsumed > (sizeof(bitD->bitContainer)8)) / overflow detected, like end of stream */
	return BIT_DStream_overflow;

	if (bitD->ptr >= bitD->limitPtr) {
	return BIT_reloadDStreamFast(bitD);
	}
	if (bitD->ptr == bitD->start) {
	if (bitD->bitsConsumed < sizeof(bitD->bitContainer)*8) return BIT_DStream_endOfBuffer;
	return BIT_DStream_completed;
	}
	/* start < ptr < limitPtr */
	{ U32 nbBytes = bitD->bitsConsumed >> 3;
	BIT_DStream_status result = BIT_DStream_unfinished;
	if (bitD->ptr - nbBytes < bitD->start) {
	nbBytes = (U32)(bitD->ptr - bitD->start); /* ptr > start */
	result = BIT_DStream_endOfBuffer;
	}
	bitD->ptr -= nbBytes;
	bitD->bitsConsumed -= nbBytes*8;
	bitD->bitContainer = MEM_readLEST(bitD->ptr); /* reminder : srcSize > sizeof(bitD->bitContainer), otherwise bitD->ptr == bitD->start */
	return result;
	}
	}

	/*! BIT_endOfDStream() :
	* @return : 1 if DStream has _exactly_ reached its end (all bits consumed).
	*/
	MEM_STATIC unsigned BIT_endOfDStream(const BIT_DStream_t* DStream)
	{
	return ((DStream->ptr == DStream->start) && (DStream->bitsConsumed == sizeof(DStream->bitContainer)*8));
	}

	#if defined (__cplusplus)
	}
	#endif

	#endif /* BITSTREAM_H_MODULE */
	/** ended inlining bitstream.h **/


	/* *****************************************
	* Static allocation
	*******************************************/
	/* FSE buffer bounds */
	#define FSE_NCOUNTBOUND 512
	#define FSE_BLOCKBOUND(size) (size + (size>>7) + 4 /* fse states / + sizeof(size_t) / bitContainer */)
	#define FSE_COMPRESSBOUND(size) (FSE_NCOUNTBOUND + FSE_BLOCKBOUND(size)) /* Macro version, useful for static allocation */

	/* It is possible to statically allocate FSE CTable/DTable as a table of FSE_CTable/FSE_DTable using below macros */
	#define FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue) (1 + (1<<(maxTableLog-1)) + ((maxSymbolValue+1)*2))
	#define FSE_DTABLE_SIZE_U32(maxTableLog) (1 + (1<<maxTableLog))

	/* or use the size to malloc() space directly. Pay attention to alignment restrictions though */
	#define FSE_CTABLE_SIZE(maxTableLog, maxSymbolValue) (FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue) * sizeof(FSE_CTable))
	#define FSE_DTABLE_SIZE(maxTableLog) (FSE_DTABLE_SIZE_U32(maxTableLog) * sizeof(FSE_DTable))


	/* *****************************************
	* FSE advanced API
	***************************************** */

	unsigned FSE_optimalTableLog_internal(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue, unsigned minus);
	/*< same as FSE_optimalTableLog(), which used `minus==2` /

	/* FSE_compress_wksp() :
	* Same as FSE_compress2(), but using an externally allocated scratch buffer (`workSpace`).
	* FSE_WKSP_SIZE_U32() provides the minimum size required for `workSpace` as a table of FSE_CTable.
	*/
	#define FSE_WKSP_SIZE_U32(maxTableLog, maxSymbolValue) ( FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue) + ((maxTableLog > 12) ? (1 << (maxTableLog - 2)) : 1024) )
	size_t FSE_compress_wksp (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize);

	size_t FSE_buildCTable_raw (FSE_CTable* ct, unsigned nbBits);
	/*< build a fake FSE_CTable, designed for a flat distribution, where each symbol uses nbBits /

	size_t FSE_buildCTable_rle (FSE_CTable* ct, unsigned char symbolValue);
	/*< build a fake FSE_CTable, designed to compress always the same symbolValue /

	/* FSE_buildCTable_wksp() :
	* Same as FSE_buildCTable(), but using an externally allocated scratch buffer (`workSpace`).
	* `wkspSize` must be >= `(1<<tableLog)`.
	*/
	size_t FSE_buildCTable_wksp(FSE_CTable* ct, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize);

	size_t FSE_buildDTable_raw (FSE_DTable* dt, unsigned nbBits);
	/*< build a fake FSE_DTable, designed to read a flat distribution where each symbol uses nbBits /

	size_t FSE_buildDTable_rle (FSE_DTable* dt, unsigned char symbolValue);
	/*< build a fake FSE_DTable, designed to always generate the same symbolValue /

	size_t FSE_decompress_wksp(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, FSE_DTable* workSpace, unsigned maxLog);
	/*< same as FSE_decompress(), using an externally allocated `workSpace` produced with `FSE_DTABLE_SIZE_U32(maxLog)` /

	typedef enum {
	FSE_repeat_none, /*< Cannot use the previous table /
	FSE_repeat_check, /*< Can use the previous table but it must be checked /
	FSE_repeat_valid /*< Can use the previous table and it is assumed to be valid /
	} FSE_repeat;

	/* *****************************************
	* FSE symbol compression API
	*******************************************/
	/*!
	This API consists of small unitary functions, which highly benefit from being inlined.
	Hence their body are included in next section.
	*/
	typedef struct {
	ptrdiff_t value;
	const void* stateTable;
	const void* symbolTT;
	unsigned stateLog;
	} FSE_CState_t;

	static void FSE_initCState(FSE_CState_t* CStatePtr, const FSE_CTable* ct);

	static void FSE_encodeSymbol(BIT_CStream_t* bitC, FSE_CState_t* CStatePtr, unsigned symbol);

	static void FSE_flushCState(BIT_CStream_t* bitC, const FSE_CState_t* CStatePtr);

	/**<
	These functions are inner components of FSE_compress_usingCTable().
	They allow the creation of custom streams, mixing multiple tables and bit sources.

	A key property to keep in mind is that encoding and decoding are done in reverse direction.
	So the first symbol you will encode is the last you will decode, like a LIFO stack.

	You will need a few variables to track your CStream. They are :

	FSE_CTable ct; // Provided by FSE_buildCTable()
	BIT_CStream_t bitStream; // bitStream tracking structure
	FSE_CState_t state; // State tracking structure (can have several)


	The first thing to do is to init bitStream and state.
	size_t errorCode = BIT_initCStream(&bitStream, dstBuffer, maxDstSize);
	FSE_initCState(&state, ct);

	Note that BIT_initCStream() can produce an error code, so its result should be tested, using FSE_isError();
	You can then encode your input data, byte after byte.
	FSE_encodeSymbol() outputs a maximum of 'tableLog' bits at a time.
	Remember decoding will be done in reverse direction.
	FSE_encodeByte(&bitStream, &state, symbol);

	At any time, you can also add any bit sequence.
	Note : maximum allowed nbBits is 25, for compatibility with 32-bits decoders
	BIT_addBits(&bitStream, bitField, nbBits);

	The above methods don't commit data to memory, they just store it into local register, for speed.
	Local register size is 64-bits on 64-bits systems, 32-bits on 32-bits systems (size_t).
	Writing data to memory is a manual operation, performed by the flushBits function.
	BIT_flushBits(&bitStream);

	Your last FSE encoding operation shall be to flush your last state value(s).
	FSE_flushState(&bitStream, &state);

	Finally, you must close the bitStream.
	The function returns the size of CStream in bytes.
	If data couldn't fit into dstBuffer, it will return a 0 ( == not compressible)
	If there is an error, it returns an errorCode (which can be tested using FSE_isError()).
	size_t size = BIT_closeCStream(&bitStream);
	*/


	/* *****************************************
	* FSE symbol decompression API
	*******************************************/
	typedef struct {
	size_t state;
	const void* table; /* precise table may vary, depending on U16 */
	} FSE_DState_t;


	static void FSE_initDState(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD, const FSE_DTable* dt);

	static unsigned char FSE_decodeSymbol(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD);

	static unsigned FSE_endOfDState(const FSE_DState_t* DStatePtr);

	/**<
	Let's now decompose FSE_decompress_usingDTable() into its unitary components.
	You will decode FSE-encoded symbols from the bitStream,
	and also any other bitFields you put in, in reverse order.

	You will need a few variables to track your bitStream. They are :

	BIT_DStream_t DStream; // Stream context
	FSE_DState_t DState; // State context. Multiple ones are possible
	FSE_DTable* DTablePtr; // Decoding table, provided by FSE_buildDTable()

	The first thing to do is to init the bitStream.
	errorCode = BIT_initDStream(&DStream, srcBuffer, srcSize);

	You should then retrieve your initial state(s)
	(in reverse flushing order if you have several ones) :
	errorCode = FSE_initDState(&DState, &DStream, DTablePtr);

	You can then decode your data, symbol after symbol.
	For information the maximum number of bits read by FSE_decodeSymbol() is 'tableLog'.
	Keep in mind that symbols are decoded in reverse order, like a LIFO stack (last in, first out).
	unsigned char symbol = FSE_decodeSymbol(&DState, &DStream);

	You can retrieve any bitfield you eventually stored into the bitStream (in reverse order)
	Note : maximum allowed nbBits is 25, for 32-bits compatibility
	size_t bitField = BIT_readBits(&DStream, nbBits);

	All above operations only read from local register (which size depends on size_t).
	Refueling the register from memory is manually performed by the reload method.
	endSignal = FSE_reloadDStream(&DStream);

	BIT_reloadDStream() result tells if there is still some more data to read from DStream.
	BIT_DStream_unfinished : there is still some data left into the DStream.
	BIT_DStream_endOfBuffer : Dstream reached end of buffer. Its container may no longer be completely filled.
	BIT_DStream_completed : Dstream reached its exact end, corresponding in general to decompression completed.
	BIT_DStream_tooFar : Dstream went too far. Decompression result is corrupted.

	When reaching end of buffer (BIT_DStream_endOfBuffer), progress slowly, notably if you decode multiple symbols per loop,
	to properly detect the exact end of stream.
	After each decoded symbol, check if DStream is fully consumed using this simple test :
	BIT_reloadDStream(&DStream) >= BIT_DStream_completed

	When it's done, verify decompression is fully completed, by checking both DStream and the relevant states.
	Checking if DStream has reached its end is performed by :
	BIT_endOfDStream(&DStream);
	Check also the states. There might be some symbols left there, if some high probability ones (>50%) are possible.
	FSE_endOfDState(&DState);
	*/


	/* *****************************************
	* FSE unsafe API
	*******************************************/
	static unsigned char FSE_decodeSymbolFast(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD);
	/* faster, but works only if nbBits is always >= 1 (otherwise, result will be corrupted) */


	/* *****************************************
	* Implementation of inlined functions
	*******************************************/
	typedef struct {
	int deltaFindState;
	U32 deltaNbBits;
	} FSE_symbolCompressionTransform; /* total 8 bytes */

	MEM_STATIC void FSE_initCState(FSE_CState_t* statePtr, const FSE_CTable* ct)
	{
	const void* ptr = ct;
	const U16* u16ptr = (const U16*) ptr;
	const U32 tableLog = MEM_read16(ptr);
	statePtr->value = (ptrdiff_t)1<<tableLog;
	statePtr->stateTable = u16ptr+2;
	statePtr->symbolTT = ct + 1 + (tableLog ? (1<<(tableLog-1)) : 1);
	statePtr->stateLog = tableLog;
	}


	/*! FSE_initCState2() :
	* Same as FSE_initCState(), but the first symbol to include (which will be the last to be read)
	* uses the smallest state value possible, saving the cost of this symbol */
	MEM_STATIC void FSE_initCState2(FSE_CState_t* statePtr, const FSE_CTable* ct, U32 symbol)
	{
	FSE_initCState(statePtr, ct);
	{ const FSE_symbolCompressionTransform symbolTT = ((const FSE_symbolCompressionTransform*)(statePtr->symbolTT))[symbol];
	const U16* stateTable = (const U16*)(statePtr->stateTable);
	U32 nbBitsOut = (U32)((symbolTT.deltaNbBits + (1<<15)) >> 16);
	statePtr->value = (nbBitsOut << 16) - symbolTT.deltaNbBits;
	statePtr->value = stateTable[(statePtr->value >> nbBitsOut) + symbolTT.deltaFindState];
	}
	}

	MEM_STATIC void FSE_encodeSymbol(BIT_CStream_t* bitC, FSE_CState_t* statePtr, unsigned symbol)
	{
	FSE_symbolCompressionTransform const symbolTT = ((const FSE_symbolCompressionTransform*)(statePtr->symbolTT))[symbol];
	const U16* const stateTable = (const U16*)(statePtr->stateTable);
	U32 const nbBitsOut = (U32)((statePtr->value + symbolTT.deltaNbBits) >> 16);
	BIT_addBits(bitC, statePtr->value, nbBitsOut);
	statePtr->value = stateTable[ (statePtr->value >> nbBitsOut) + symbolTT.deltaFindState];
	}

	MEM_STATIC void FSE_flushCState(BIT_CStream_t* bitC, const FSE_CState_t* statePtr)
	{
	BIT_addBits(bitC, statePtr->value, statePtr->stateLog);
	BIT_flushBits(bitC);
	}


	/* FSE_getMaxNbBits() :
	* Approximate maximum cost of a symbol, in bits.
	* Fractional get rounded up (i.e : a symbol with a normalized frequency of 3 gives the same result as a frequency of 2)
	* note 1 : assume symbolValue is valid (<= maxSymbolValue)
	* note 2 : if freq[symbolValue]==0, @return a fake cost of tableLog+1 bits */
	MEM_STATIC U32 FSE_getMaxNbBits(const void* symbolTTPtr, U32 symbolValue)
	{
	const FSE_symbolCompressionTransform* symbolTT = (const FSE_symbolCompressionTransform*) symbolTTPtr;
	return (symbolTT[symbolValue].deltaNbBits + ((1<<16)-1)) >> 16;
	}

	/* FSE_bitCost() :
	* Approximate symbol cost, as fractional value, using fixed-point format (accuracyLog fractional bits)
	* note 1 : assume symbolValue is valid (<= maxSymbolValue)
	* note 2 : if freq[symbolValue]==0, @return a fake cost of tableLog+1 bits */
	MEM_STATIC U32 FSE_bitCost(const void* symbolTTPtr, U32 tableLog, U32 symbolValue, U32 accuracyLog)
	{
	const FSE_symbolCompressionTransform* symbolTT = (const FSE_symbolCompressionTransform*) symbolTTPtr;
	U32 const minNbBits = symbolTT[symbolValue].deltaNbBits >> 16;
	U32 const threshold = (minNbBits+1) << 16;
	assert(tableLog < 16);
	assert(accuracyLog < 31-tableLog); /* ensure enough room for renormalization double shift */
	{ U32 const tableSize = 1 << tableLog;
	U32 const deltaFromThreshold = threshold - (symbolTT[symbolValue].deltaNbBits + tableSize);
	U32 const normalizedDeltaFromThreshold = (deltaFromThreshold << accuracyLog) >> tableLog; /* linear interpolation (very approximate) */
	U32 const bitMultiplier = 1 << accuracyLog;
	assert(symbolTT[symbolValue].deltaNbBits + tableSize <= threshold);
	assert(normalizedDeltaFromThreshold <= bitMultiplier);
	return (minNbBits+1)*bitMultiplier - normalizedDeltaFromThreshold;
	}
	}


	/* ====== Decompression ====== */

	typedef struct {
	U16 tableLog;
	U16 fastMode;
	} FSE_DTableHeader; /* sizeof U32 */

	typedef struct
	{
	unsigned short newState;
	unsigned char symbol;
	unsigned char nbBits;
	} FSE_decode_t; /* size == U32 */

	MEM_STATIC void FSE_initDState(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD, const FSE_DTable* dt)
	{
	const void* ptr = dt;
	const FSE_DTableHeader* const DTableH = (const FSE_DTableHeader*)ptr;
	DStatePtr->state = BIT_readBits(bitD, DTableH->tableLog);
	BIT_reloadDStream(bitD);
	DStatePtr->table = dt + 1;
	}

	MEM_STATIC BYTE FSE_peekSymbol(const FSE_DState_t* DStatePtr)
	{
	FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
	return DInfo.symbol;
	}

	MEM_STATIC void FSE_updateState(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD)
	{
	FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
	U32 const nbBits = DInfo.nbBits;
	size_t const lowBits = BIT_readBits(bitD, nbBits);
	DStatePtr->state = DInfo.newState + lowBits;
	}

	MEM_STATIC BYTE FSE_decodeSymbol(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD)
	{
	FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
	U32 const nbBits = DInfo.nbBits;
	BYTE const symbol = DInfo.symbol;
	size_t const lowBits = BIT_readBits(bitD, nbBits);

	DStatePtr->state = DInfo.newState + lowBits;
	return symbol;
	}

	/*! FSE_decodeSymbolFast() :
	unsafe, only works if no symbol has a probability > 50% */
	MEM_STATIC BYTE FSE_decodeSymbolFast(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD)
	{
	FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
	U32 const nbBits = DInfo.nbBits;
	BYTE const symbol = DInfo.symbol;
	size_t const lowBits = BIT_readBitsFast(bitD, nbBits);

	DStatePtr->state = DInfo.newState + lowBits;
	return symbol;
	}

	MEM_STATIC unsigned FSE_endOfDState(const FSE_DState_t* DStatePtr)
	{
	return DStatePtr->state == 0;
	}



	#ifndef FSE_COMMONDEFS_ONLY

	/* **************************************************************
	* Tuning parameters
	****************************************************************/
	/*!MEMORY_USAGE :
	* Memory usage formula : N->2^N Bytes (examples : 10 -> 1KB; 12 -> 4KB ; 16 -> 64KB; 20 -> 1MB; etc.)
	* Increasing memory usage improves compression ratio
	* Reduced memory usage can improve speed, due to cache effect
	* Recommended max value is 14, for 16KB, which nicely fits into Intel x86 L1 cache */
	#ifndef FSE_MAX_MEMORY_USAGE
	# define FSE_MAX_MEMORY_USAGE 14
	#endif
	#ifndef FSE_DEFAULT_MEMORY_USAGE
	# define FSE_DEFAULT_MEMORY_USAGE 13
	#endif

	/*!FSE_MAX_SYMBOL_VALUE :
	* Maximum symbol value authorized.
	* Required for proper stack allocation */
	#ifndef FSE_MAX_SYMBOL_VALUE
	# define FSE_MAX_SYMBOL_VALUE 255
	#endif

	/* **************************************************************
	* template functions type & suffix
	****************************************************************/
	#define FSE_FUNCTION_TYPE BYTE
	#define FSE_FUNCTION_EXTENSION
	#define FSE_DECODE_TYPE FSE_decode_t


	#endif /* !FSE_COMMONDEFS_ONLY */


	/* ***************************************************************
	* Constants
	*****************************************************************/
	#define FSE_MAX_TABLELOG (FSE_MAX_MEMORY_USAGE-2)
	#define FSE_MAX_TABLESIZE (1U<<FSE_MAX_TABLELOG)
	#define FSE_MAXTABLESIZE_MASK (FSE_MAX_TABLESIZE-1)
	#define FSE_DEFAULT_TABLELOG (FSE_DEFAULT_MEMORY_USAGE-2)
	#define FSE_MIN_TABLELOG 5

	#define FSE_TABLELOG_ABSOLUTE_MAX 15
	#if FSE_MAX_TABLELOG > FSE_TABLELOG_ABSOLUTE_MAX
	# error "FSE_MAX_TABLELOG > FSE_TABLELOG_ABSOLUTE_MAX is not supported"
	#endif

	#define FSE_TABLESTEP(tableSize) ((tableSize>>1) + (tableSize>>3) + 3)


	#endif /* FSE_STATIC_LINKING_ONLY */


	#if defined (__cplusplus)
	}
	#endif
	/** ended inlining fse.h **/
	#define HUF_STATIC_LINKING_ONLY /* HUF_TABLELOG_ABSOLUTEMAX */
	/** start inlining huf.h **/
	/* ******************************************************************
	* huff0 huffman codec,
	* part of Finite State Entropy library
	* Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
	*
	* You can contact the author at :
	* - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	****************************************************************** */

	#if defined (__cplusplus)
	extern "C" {
	#endif

	#ifndef HUF_H_298734234
	#define HUF_H_298734234

	/* * Dependencies * */
	#include <stddef.h> /* size_t */


	/* * library symbols visibility * */
	/* Note : when linking with -fvisibility=hidden on gcc, or by default on Visual,
	* HUF symbols remain "private" (internal symbols for library only).
	* Set macro FSE_DLL_EXPORT to 1 if you want HUF symbols visible on DLL interface */
	#if defined(FSE_DLL_EXPORT) && (FSE_DLL_EXPORT==1) && defined(__GNUC__) && (__GNUC__ >= 4)
	# define HUF_PUBLIC_API __attribute__ ((visibility ("default")))
	#elif defined(FSE_DLL_EXPORT) && (FSE_DLL_EXPORT==1) /* Visual expected */
	# define HUF_PUBLIC_API __declspec(dllexport)
	#elif defined(FSE_DLL_IMPORT) && (FSE_DLL_IMPORT==1)
	# define HUF_PUBLIC_API __declspec(dllimport) /* not required, just to generate faster code (saves a function pointer load from IAT and an indirect jump) */
	#else
	# define HUF_PUBLIC_API
	#endif


	/* ========================== */
	/* * simple functions * */
	/* ========================== */

	/** HUF_compress() :
	* Compress content from buffer 'src', of size 'srcSize', into buffer 'dst'.
	* 'dst' buffer must be already allocated.
	* Compression runs faster if `dstCapacity` >= HUF_compressBound(srcSize).
	* `srcSize` must be <= `HUF_BLOCKSIZE_MAX` == 128 KB.
	* @return : size of compressed data (<= `dstCapacity`).
	* Special values : if return == 0, srcData is not compressible => Nothing is stored within dst !!!
	* if HUF_isError(return), compression failed (more details using HUF_getErrorName())
	*/
	HUF_PUBLIC_API size_t HUF_compress(void* dst, size_t dstCapacity,
	const void* src, size_t srcSize);

	/** HUF_decompress() :
	* Decompress HUF data from buffer 'cSrc', of size 'cSrcSize',
	* into already allocated buffer 'dst', of minimum size 'dstSize'.
	* `originalSize` : must be the *exact* size of original (uncompressed) data.
	* Note : in contrast with FSE, HUF_decompress can regenerate
	* RLE (cSrcSize==1) and uncompressed (cSrcSize==dstSize) data,
	* because it knows size to regenerate (originalSize).
	* @return : size of regenerated data (== originalSize),
	* or an error code, which can be tested using HUF_isError()
	*/
	HUF_PUBLIC_API size_t HUF_decompress(void* dst, size_t originalSize,
	const void* cSrc, size_t cSrcSize);


	/* * Tool functions * */
	#define HUF_BLOCKSIZE_MAX (128 * 1024) /*< maximum input size for a single block compressed with HUF_compress /
	HUF_PUBLIC_API size_t HUF_compressBound(size_t size); /*< maximum compressed size (worst case) /

	/* Error Management */
	HUF_PUBLIC_API unsigned HUF_isError(size_t code); /*< tells if a return value is an error code /
	HUF_PUBLIC_API const char* HUF_getErrorName(size_t code); /*< provides error code string (useful for debugging) /


	/* * Advanced function * */

	/** HUF_compress2() :
	* Same as HUF_compress(), but offers control over `maxSymbolValue` and `tableLog`.
	* `maxSymbolValue` must be <= HUF_SYMBOLVALUE_MAX .
	* `tableLog` must be `<= HUF_TABLELOG_MAX` . */
	HUF_PUBLIC_API size_t HUF_compress2 (void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	unsigned maxSymbolValue, unsigned tableLog);

	/** HUF_compress4X_wksp() :
	* Same as HUF_compress2(), but uses externally allocated `workSpace`.
	* `workspace` must have minimum alignment of 4, and be at least as large as HUF_WORKSPACE_SIZE */
	#define HUF_WORKSPACE_SIZE ((6 << 10) + 256)
	#define HUF_WORKSPACE_SIZE_U32 (HUF_WORKSPACE_SIZE / sizeof(U32))
	HUF_PUBLIC_API size_t HUF_compress4X_wksp (void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	unsigned maxSymbolValue, unsigned tableLog,
	void* workSpace, size_t wkspSize);

	#endif /* HUF_H_298734234 */

	/* ******************************************************************
	* WARNING !!
	* The following section contains advanced and experimental definitions
	* which shall never be used in the context of a dynamic library,
	* because they are not guaranteed to remain stable in the future.
	* Only consider them in association with static linking.
	* *****************************************************************/
	#if defined(HUF_STATIC_LINKING_ONLY) && !defined(HUF_H_HUF_STATIC_LINKING_ONLY)
	#define HUF_H_HUF_STATIC_LINKING_ONLY

	/* * Dependencies * */
	/** skipping file: mem.h **/


	/* * Constants * */
	#define HUF_TABLELOG_MAX 12 /* max runtime value of tableLog (due to static allocation); can be modified up to HUF_ABSOLUTEMAX_TABLELOG */
	#define HUF_TABLELOG_DEFAULT 11 /* default tableLog value when none specified */
	#define HUF_SYMBOLVALUE_MAX 255

	#define HUF_TABLELOG_ABSOLUTEMAX 15 /* absolute limit of HUF_MAX_TABLELOG. Beyond that value, code does not work */
	#if (HUF_TABLELOG_MAX > HUF_TABLELOG_ABSOLUTEMAX)
	# error "HUF_TABLELOG_MAX is too large !"
	#endif


	/* ****************************************
	* Static allocation
	******************************************/
	/* HUF buffer bounds */
	#define HUF_CTABLEBOUND 129
	#define HUF_BLOCKBOUND(size) (size + (size>>8) + 8) /* only true when incompressible is pre-filtered with fast heuristic */
	#define HUF_COMPRESSBOUND(size) (HUF_CTABLEBOUND + HUF_BLOCKBOUND(size)) /* Macro version, useful for static allocation */

	/* static allocation of HUF's Compression Table */
	#define HUF_CTABLE_SIZE_U32(maxSymbolValue) ((maxSymbolValue)+1) /* Use tables of U32, for proper alignment */
	#define HUF_CTABLE_SIZE(maxSymbolValue) (HUF_CTABLE_SIZE_U32(maxSymbolValue) * sizeof(U32))
	#define HUF_CREATE_STATIC_CTABLE(name, maxSymbolValue) \
	U32 name##hb[HUF_CTABLE_SIZE_U32(maxSymbolValue)]; \
	void* name##hv = &(name##hb); \
	HUF_CElt* name = (HUF_CElt)(name##hv) / no final ; */

	/* static allocation of HUF's DTable */
	typedef U32 HUF_DTable;
	#define HUF_DTABLE_SIZE(maxTableLog) (1 + (1<<(maxTableLog)))
	#define HUF_CREATE_STATIC_DTABLEX1(DTable, maxTableLog) \
	HUF_DTable DTable[HUF_DTABLE_SIZE((maxTableLog)-1)] = { ((U32)((maxTableLog)-1) * 0x01000001) }
	#define HUF_CREATE_STATIC_DTABLEX2(DTable, maxTableLog) \
	HUF_DTable DTable[HUF_DTABLE_SIZE(maxTableLog)] = { ((U32)(maxTableLog) * 0x01000001) }


	/* ****************************************
	* Advanced decompression functions
	******************************************/
	size_t HUF_decompress4X1 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /*< single-symbol decoder /
	#ifndef HUF_FORCE_DECOMPRESS_X1
	size_t HUF_decompress4X2 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /*< double-symbols decoder /
	#endif

	size_t HUF_decompress4X_DCtx (HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /*< decodes RLE and uncompressed /
	size_t HUF_decompress4X_hufOnly(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /*< considers RLE and uncompressed as errors /
	size_t HUF_decompress4X_hufOnly_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize); /*< considers RLE and uncompressed as errors /
	size_t HUF_decompress4X1_DCtx(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /*< single-symbol decoder /
	size_t HUF_decompress4X1_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize); /*< single-symbol decoder /
	#ifndef HUF_FORCE_DECOMPRESS_X1
	size_t HUF_decompress4X2_DCtx(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /*< double-symbols decoder /
	size_t HUF_decompress4X2_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize); /*< double-symbols decoder /
	#endif


	/* ****************************************
	* HUF detailed API
	* ****************************************/

	/*! HUF_compress() does the following:
	* 1. count symbol occurrence from source[] into table count[] using FSE_count() (exposed within "fse.h")
	* 2. (optional) refine tableLog using HUF_optimalTableLog()
	* 3. build Huffman table from count using HUF_buildCTable()
	* 4. save Huffman table to memory buffer using HUF_writeCTable()
	* 5. encode the data stream using HUF_compress4X_usingCTable()
	*
	* The following API allows targeting specific sub-functions for advanced tasks.
	* For example, it's possible to compress several blocks using the same 'CTable',
	* or to save and regenerate 'CTable' using external methods.
	*/
	unsigned HUF_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue);
	typedef struct HUF_CElt_s HUF_CElt; /* incomplete type */
	size_t HUF_buildCTable (HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue, unsigned maxNbBits); /* @return : maxNbBits; CTable and count can overlap. In which case, CTable will overwrite count content */
	size_t HUF_writeCTable (void* dst, size_t maxDstSize, const HUF_CElt* CTable, unsigned maxSymbolValue, unsigned huffLog);
	size_t HUF_compress4X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable);
	size_t HUF_estimateCompressedSize(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue);
	int HUF_validateCTable(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue);

	typedef enum {
	HUF_repeat_none, /*< Cannot use the previous table /
	HUF_repeat_check, /*< Can use the previous table but it must be checked. Note : The previous table must have been constructed by HUF_compress{1, 4}X_repeat /
	HUF_repeat_valid /*< Can use the previous table and it is assumed to be valid /
	} HUF_repeat;
	/** HUF_compress4X_repeat() :
	* Same as HUF_compress4X_wksp(), but considers using hufTable if *repeat != HUF_repeat_none.
	* If it uses hufTable it does not modify hufTable or repeat.
	* If it doesn't, it sets *repeat = HUF_repeat_none, and it sets hufTable to the table used.
	* If preferRepeat then the old table will always be used if valid. */
	size_t HUF_compress4X_repeat(void* dst, size_t dstSize,
	const void* src, size_t srcSize,
	unsigned maxSymbolValue, unsigned tableLog,
	void* workSpace, size_t wkspSize, /*< `workSpace` must be aligned on 4-bytes boundaries, `wkspSize` must be >= HUF_WORKSPACE_SIZE /
	HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat, int bmi2);

	/** HUF_buildCTable_wksp() :
	* Same as HUF_buildCTable(), but using externally allocated scratch buffer.
	* `workSpace` must be aligned on 4-bytes boundaries, and its size must be >= HUF_CTABLE_WORKSPACE_SIZE.
	*/
	#define HUF_CTABLE_WORKSPACE_SIZE_U32 (2*HUF_SYMBOLVALUE_MAX +1 +1)
	#define HUF_CTABLE_WORKSPACE_SIZE (HUF_CTABLE_WORKSPACE_SIZE_U32 * sizeof(unsigned))
	size_t HUF_buildCTable_wksp (HUF_CElt* tree,
	const unsigned* count, U32 maxSymbolValue, U32 maxNbBits,
	void* workSpace, size_t wkspSize);

	/*! HUF_readStats() :
	* Read compact Huffman tree, saved by HUF_writeCTable().
	* `huffWeight` is destination buffer.
	* @return : size read from `src` , or an error Code .
	* Note : Needed by HUF_readCTable() and HUF_readDTableXn() . */
	size_t HUF_readStats(BYTE* huffWeight, size_t hwSize,
	U32* rankStats, U32* nbSymbolsPtr, U32* tableLogPtr,
	const void* src, size_t srcSize);

	/** HUF_readCTable() :
	* Loading a CTable saved with HUF_writeCTable() */
	size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize, unsigned *hasZeroWeights);

	/** HUF_getNbBits() :
	* Read nbBits from CTable symbolTable, for symbol `symbolValue` presumed <= HUF_SYMBOLVALUE_MAX
	* Note 1 : is not inlined, as HUF_CElt definition is private
	* Note 2 : const void* used, so that it can provide a statically allocated table as argument (which uses type U32) */
	U32 HUF_getNbBits(const void* symbolTable, U32 symbolValue);

	/*
	* HUF_decompress() does the following:
	* 1. select the decompression algorithm (X1, X2) based on pre-computed heuristics
	* 2. build Huffman table from save, using HUF_readDTableX?()
	* 3. decode 1 or 4 segments in parallel using HUF_decompress?X?_usingDTable()
	*/

	/** HUF_selectDecoder() :
	* Tells which decoder is likely to decode faster,
	* based on a set of pre-computed metrics.
	* @return : 0==HUF_decompress4X1, 1==HUF_decompress4X2 .
	* Assumption : 0 < dstSize <= 128 KB */
	U32 HUF_selectDecoder (size_t dstSize, size_t cSrcSize);

	/**
	* The minimum workspace size for the `workSpace` used in
	* HUF_readDTableX1_wksp() and HUF_readDTableX2_wksp().
	*
	* The space used depends on HUF_TABLELOG_MAX, ranging from ~1500 bytes when
	* HUF_TABLE_LOG_MAX=12 to ~1850 bytes when HUF_TABLE_LOG_MAX=15.
	* Buffer overflow errors may potentially occur if code modifications result in
	* a required workspace size greater than that specified in the following
	* macro.
	*/
	#define HUF_DECOMPRESS_WORKSPACE_SIZE (2 << 10)
	#define HUF_DECOMPRESS_WORKSPACE_SIZE_U32 (HUF_DECOMPRESS_WORKSPACE_SIZE / sizeof(U32))

	#ifndef HUF_FORCE_DECOMPRESS_X2
	size_t HUF_readDTableX1 (HUF_DTable* DTable, const void* src, size_t srcSize);
	size_t HUF_readDTableX1_wksp (HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize);
	#endif
	#ifndef HUF_FORCE_DECOMPRESS_X1
	size_t HUF_readDTableX2 (HUF_DTable* DTable, const void* src, size_t srcSize);
	size_t HUF_readDTableX2_wksp (HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize);
	#endif

	size_t HUF_decompress4X_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable);
	#ifndef HUF_FORCE_DECOMPRESS_X2
	size_t HUF_decompress4X1_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable);
	#endif
	#ifndef HUF_FORCE_DECOMPRESS_X1
	size_t HUF_decompress4X2_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable);
	#endif


	/* ====================== */
	/* single stream variants */
	/* ====================== */

	size_t HUF_compress1X (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog);
	size_t HUF_compress1X_wksp (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize); /*< `workSpace` must be a table of at least HUF_WORKSPACE_SIZE_U32 unsigned /
	size_t HUF_compress1X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable);
	/** HUF_compress1X_repeat() :
	* Same as HUF_compress1X_wksp(), but considers using hufTable if *repeat != HUF_repeat_none.
	* If it uses hufTable it does not modify hufTable or repeat.
	* If it doesn't, it sets *repeat = HUF_repeat_none, and it sets hufTable to the table used.
	* If preferRepeat then the old table will always be used if valid. */
	size_t HUF_compress1X_repeat(void* dst, size_t dstSize,
	const void* src, size_t srcSize,
	unsigned maxSymbolValue, unsigned tableLog,
	void* workSpace, size_t wkspSize, /*< `workSpace` must be aligned on 4-bytes boundaries, `wkspSize` must be >= HUF_WORKSPACE_SIZE /
	HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat, int bmi2);

	size_t HUF_decompress1X1 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /* single-symbol decoder */
	#ifndef HUF_FORCE_DECOMPRESS_X1
	size_t HUF_decompress1X2 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /* double-symbol decoder */
	#endif

	size_t HUF_decompress1X_DCtx (HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize);
	size_t HUF_decompress1X_DCtx_wksp (HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize);
	#ifndef HUF_FORCE_DECOMPRESS_X2
	size_t HUF_decompress1X1_DCtx(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /*< single-symbol decoder /
	size_t HUF_decompress1X1_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize); /*< single-symbol decoder /
	#endif
	#ifndef HUF_FORCE_DECOMPRESS_X1
	size_t HUF_decompress1X2_DCtx(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /*< double-symbols decoder /
	size_t HUF_decompress1X2_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize); /*< double-symbols decoder /
	#endif

	size_t HUF_decompress1X_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable); /*< automatic selection of sing or double symbol decoder, based on DTable /
	#ifndef HUF_FORCE_DECOMPRESS_X2
	size_t HUF_decompress1X1_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable);
	#endif
	#ifndef HUF_FORCE_DECOMPRESS_X1
	size_t HUF_decompress1X2_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable);
	#endif

	/* BMI2 variants.
	* If the CPU has BMI2 support, pass bmi2=1, otherwise pass bmi2=0.
	*/
	size_t HUF_decompress1X_usingDTable_bmi2(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, int bmi2);
	#ifndef HUF_FORCE_DECOMPRESS_X2
	size_t HUF_decompress1X1_DCtx_wksp_bmi2(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int bmi2);
	#endif
	size_t HUF_decompress4X_usingDTable_bmi2(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, int bmi2);
	size_t HUF_decompress4X_hufOnly_wksp_bmi2(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int bmi2);

	#endif /* HUF_STATIC_LINKING_ONLY */

	#if defined (__cplusplus)
	}
	#endif
	/** ended inlining huf.h **/


	/=== Version ===/
	unsigned FSE_versionNumber(void) { return FSE_VERSION_NUMBER; }


	/=== Error Management ===/
	unsigned FSE_isError(size_t code) { return ERR_isError(code); }
	const char* FSE_getErrorName(size_t code) { return ERR_getErrorName(code); }

	unsigned HUF_isError(size_t code) { return ERR_isError(code); }
	const char* HUF_getErrorName(size_t code) { return ERR_getErrorName(code); }


	/-*************************************************************
	* FSE NCount encoding-decoding
	****************************************************************/
	size_t FSE_readNCount (short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr,
	const void* headerBuffer, size_t hbSize)
	{
	const BYTE* const istart = (const BYTE*) headerBuffer;
	const BYTE* const iend = istart + hbSize;
	const BYTE* ip = istart;
	int nbBits;
	int remaining;
	int threshold;
	U32 bitStream;
	int bitCount;
	unsigned charnum = 0;
	int previous0 = 0;

	if (hbSize < 4) {
	/* This function only works when hbSize >= 4 */
	char buffer[4];
	memset(buffer, 0, sizeof(buffer));
	memcpy(buffer, headerBuffer, hbSize);
	{ size_t const countSize = FSE_readNCount(normalizedCounter, maxSVPtr, tableLogPtr,
	buffer, sizeof(buffer));
	if (FSE_isError(countSize)) return countSize;
	if (countSize > hbSize) return ERROR(corruption_detected);
	return countSize;
	} }
	assert(hbSize >= 4);

	/* init */
	memset(normalizedCounter, 0, (maxSVPtr+1) sizeof(normalizedCounter[0])); /* all symbols not present in NCount have a frequency of 0 */
	bitStream = MEM_readLE32(ip);
	nbBits = (bitStream & 0xF) + FSE_MIN_TABLELOG; /* extract tableLog */
	if (nbBits > FSE_TABLELOG_ABSOLUTE_MAX) return ERROR(tableLog_tooLarge);
	bitStream >>= 4;
	bitCount = 4;
	*tableLogPtr = nbBits;
	remaining = (1<<nbBits)+1;
	threshold = 1<<nbBits;
	nbBits++;

	while ((remaining>1) & (charnum<=*maxSVPtr)) {
	if (previous0) {
	unsigned n0 = charnum;
	while ((bitStream & 0xFFFF) == 0xFFFF) {
	n0 += 24;
	if (ip < iend-5) {
	ip += 2;
	bitStream = MEM_readLE32(ip) >> bitCount;
	} else {
	bitStream >>= 16;
	bitCount += 16;
	} }
	while ((bitStream & 3) == 3) {
	n0 += 3;
	bitStream >>= 2;
	bitCount += 2;
	}
	n0 += bitStream & 3;
	bitCount += 2;
	if (n0 > *maxSVPtr) return ERROR(maxSymbolValue_tooSmall);
	while (charnum < n0) normalizedCounter[charnum++] = 0;
	if ((ip <= iend-7) \|\| (ip + (bitCount>>3) <= iend-4)) {
	assert((bitCount >> 3) <= 3); /* For first condition to work */
	ip += bitCount>>3;
	bitCount &= 7;
	bitStream = MEM_readLE32(ip) >> bitCount;
	} else {
	bitStream >>= 2;
	} }
	{ int const max = (2*threshold-1) - remaining;
	int count;

	if ((bitStream & (threshold-1)) < (U32)max) {
	count = bitStream & (threshold-1);
	bitCount += nbBits-1;
	} else {
	count = bitStream & (2*threshold-1);
	if (count >= threshold) count -= max;
	bitCount += nbBits;
	}

	count--; /* extra accuracy */
	remaining -= count < 0 ? -count : count; /* -1 means +1 */
	normalizedCounter[charnum++] = (short)count;
	previous0 = !count;
	while (remaining < threshold) {
	nbBits--;
	threshold >>= 1;
	}

	if ((ip <= iend-7) \|\| (ip + (bitCount>>3) <= iend-4)) {
	ip += bitCount>>3;
	bitCount &= 7;
	} else {
	bitCount -= (int)(8 * (iend - 4 - ip));
	ip = iend - 4;
	}
	bitStream = MEM_readLE32(ip) >> (bitCount & 31);
	} } /* while ((remaining>1) & (charnum<=maxSVPtr)) /
	if (remaining != 1) return ERROR(corruption_detected);
	if (bitCount > 32) return ERROR(corruption_detected);
	*maxSVPtr = charnum-1;

	ip += (bitCount+7)>>3;
	return ip-istart;
	}


	/*! HUF_readStats() :
	Read compact Huffman tree, saved by HUF_writeCTable().
	`huffWeight` is destination buffer.
	`rankStats` is assumed to be a table of at least HUF_TABLELOG_MAX U32.
	@return : size read from `src` , or an error Code .
	Note : Needed by HUF_readCTable() and HUF_readDTableX?() .
	*/
	size_t HUF_readStats(BYTE* huffWeight, size_t hwSize, U32* rankStats,
	U32* nbSymbolsPtr, U32* tableLogPtr,
	const void* src, size_t srcSize)
	{
	U32 weightTotal;
	const BYTE* ip = (const BYTE*) src;
	size_t iSize;
	size_t oSize;

	if (!srcSize) return ERROR(srcSize_wrong);
	iSize = ip[0];
	/* memset(huffWeight, 0, hwSize); // is not necessary, even though some analyzer complain ... */

	if (iSize >= 128) { /* special header */
	oSize = iSize - 127;
	iSize = ((oSize+1)/2);
	if (iSize+1 > srcSize) return ERROR(srcSize_wrong);
	if (oSize >= hwSize) return ERROR(corruption_detected);
	ip += 1;
	{ U32 n;
	for (n=0; n<oSize; n+=2) {
	huffWeight[n] = ip[n/2] >> 4;
	huffWeight[n+1] = ip[n/2] & 15;
	} } }
	else { /* header compressed with FSE (normal case) */
	FSE_DTable fseWorkspace[FSE_DTABLE_SIZE_U32(6)]; /* 6 is max possible tableLog for HUF header (maybe even 5, to be tested) */
	if (iSize+1 > srcSize) return ERROR(srcSize_wrong);
	oSize = FSE_decompress_wksp(huffWeight, hwSize-1, ip+1, iSize, fseWorkspace, 6); /* max (hwSize-1) values decoded, as last one is implied */
	if (FSE_isError(oSize)) return oSize;
	}

	/* collect weight stats */
	memset(rankStats, 0, (HUF_TABLELOG_MAX + 1) * sizeof(U32));
	weightTotal = 0;
	{ U32 n; for (n=0; n<oSize; n++) {
	if (huffWeight[n] >= HUF_TABLELOG_MAX) return ERROR(corruption_detected);
	rankStats[huffWeight[n]]++;
	weightTotal += (1 << huffWeight[n]) >> 1;
	} }
	if (weightTotal == 0) return ERROR(corruption_detected);

	/* get last non-null symbol weight (implied, total must be 2^n) */
	{ U32 const tableLog = BIT_highbit32(weightTotal) + 1;
	if (tableLog > HUF_TABLELOG_MAX) return ERROR(corruption_detected);
	*tableLogPtr = tableLog;
	/* determine last weight */
	{ U32 const total = 1 << tableLog;
	U32 const rest = total - weightTotal;
	U32 const verif = 1 << BIT_highbit32(rest);
	U32 const lastWeight = BIT_highbit32(rest) + 1;
	if (verif != rest) return ERROR(corruption_detected); /* last value must be a clean power of 2 */
	huffWeight[oSize] = (BYTE)lastWeight;
	rankStats[lastWeight]++;
	} }

	/* check tree construction validity */
	if ((rankStats[1] < 2) \|\| (rankStats[1] & 1)) return ERROR(corruption_detected); /* by construction : at least 2 elts of rank 1, must be even */

	/* results */
	*nbSymbolsPtr = (U32)(oSize+1);
	return iSize+1;
	}
	/** ended inlining common/entropy_common.c **/
	/** start inlining common/error_private.c **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	/* The purpose of this file is to have a single list of error strings embedded in binary */

	/** skipping file: error_private.h **/

	const char* ERR_getErrorString(ERR_enum code)
	{
	#ifdef ZSTD_STRIP_ERROR_STRINGS
	(void)code;
	return "Error strings stripped";
	#else
	static const char* const notErrorCode = "Unspecified error code";
	switch( code )
	{
	case PREFIX(no_error): return "No error detected";
	case PREFIX(GENERIC): return "Error (generic)";
	case PREFIX(prefix_unknown): return "Unknown frame descriptor";
	case PREFIX(version_unsupported): return "Version not supported";
	case PREFIX(frameParameter_unsupported): return "Unsupported frame parameter";
	case PREFIX(frameParameter_windowTooLarge): return "Frame requires too much memory for decoding";
	case PREFIX(corruption_detected): return "Corrupted block detected";
	case PREFIX(checksum_wrong): return "Restored data doesn't match checksum";
	case PREFIX(parameter_unsupported): return "Unsupported parameter";
	case PREFIX(parameter_outOfBound): return "Parameter is out of bound";
	case PREFIX(init_missing): return "Context should be init first";
	case PREFIX(memory_allocation): return "Allocation error : not enough memory";
	case PREFIX(workSpace_tooSmall): return "workSpace buffer is not large enough";
	case PREFIX(stage_wrong): return "Operation not authorized at current processing stage";
	case PREFIX(tableLog_tooLarge): return "tableLog requires too much memory : unsupported";
	case PREFIX(maxSymbolValue_tooLarge): return "Unsupported max Symbol Value : too large";
	case PREFIX(maxSymbolValue_tooSmall): return "Specified maxSymbolValue is too small";
	case PREFIX(dictionary_corrupted): return "Dictionary is corrupted";
	case PREFIX(dictionary_wrong): return "Dictionary mismatch";
	case PREFIX(dictionaryCreation_failed): return "Cannot create Dictionary from provided samples";
	case PREFIX(dstSize_tooSmall): return "Destination buffer is too small";
	case PREFIX(srcSize_wrong): return "Src size is incorrect";
	case PREFIX(dstBuffer_null): return "Operation on NULL destination buffer";
	/* following error codes are not stable and may be removed or changed in a future version */
	case PREFIX(frameIndex_tooLarge): return "Frame index is too large";
	case PREFIX(seekableIO): return "An I/O error occurred when reading/seeking";
	case PREFIX(dstBuffer_wrong): return "Destination buffer is wrong";
	case PREFIX(maxCode):
	default: return notErrorCode;
	}
	#endif
	}
	/** ended inlining common/error_private.c **/
	/** start inlining common/fse_decompress.c **/
	/* ******************************************************************
	* FSE : Finite State Entropy decoder
	* Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
	*
	* You can contact the author at :
	* - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
	* - Public forum : https://groups.google.com/forum/#!forum/lz4c
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	****************************************************************** */


	/* **************************************************************
	* Includes
	****************************************************************/
	#include <stdlib.h> /* malloc, free, qsort */
	#include <string.h> /* memcpy, memset */
	/** skipping file: bitstream.h **/
	/** skipping file: compiler.h **/
	#define FSE_STATIC_LINKING_ONLY
	/** skipping file: fse.h **/
	/** skipping file: error_private.h **/


	/* **************************************************************
	* Error Management
	****************************************************************/
	#define FSE_isError ERR_isError
	#define FSE_STATIC_ASSERT(c) DEBUG_STATIC_ASSERT(c) /* use only after variable declarations */


	/* **************************************************************
	* Templates
	****************************************************************/
	/*
	designed to be included
	for type-specific functions (template emulation in C)
	Objective is to write these functions only once, for improved maintenance
	*/

	/* safety checks */
	#ifndef FSE_FUNCTION_EXTENSION
	# error "FSE_FUNCTION_EXTENSION must be defined"
	#endif
	#ifndef FSE_FUNCTION_TYPE
	# error "FSE_FUNCTION_TYPE must be defined"
	#endif

	/* Function names */
	#define FSE_CAT(X,Y) X##Y
	#define FSE_FUNCTION_NAME(X,Y) FSE_CAT(X,Y)
	#define FSE_TYPE_NAME(X,Y) FSE_CAT(X,Y)


	/* Function templates */
	FSE_DTable* FSE_createDTable (unsigned tableLog)
	{
	if (tableLog > FSE_TABLELOG_ABSOLUTE_MAX) tableLog = FSE_TABLELOG_ABSOLUTE_MAX;
	return (FSE_DTable)malloc( FSE_DTABLE_SIZE_U32(tableLog) sizeof (U32) );
	}

	void FSE_freeDTable (FSE_DTable* dt)
	{
	free(dt);
	}

	size_t FSE_buildDTable(FSE_DTable* dt, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog)
	{
	void* const tdPtr = dt+1; /* because dt is unsigned, 32-bits aligned on 32-bits /
	FSE_DECODE_TYPE* const tableDecode = (FSE_DECODE_TYPE*) (tdPtr);
	U16 symbolNext[FSE_MAX_SYMBOL_VALUE+1];

	U32 const maxSV1 = maxSymbolValue + 1;
	U32 const tableSize = 1 << tableLog;
	U32 highThreshold = tableSize-1;

	/* Sanity Checks */
	if (maxSymbolValue > FSE_MAX_SYMBOL_VALUE) return ERROR(maxSymbolValue_tooLarge);
	if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge);

	/* Init, lay down lowprob symbols */
	{ FSE_DTableHeader DTableH;
	DTableH.tableLog = (U16)tableLog;
	DTableH.fastMode = 1;
	{ S16 const largeLimit= (S16)(1 << (tableLog-1));
	U32 s;
	for (s=0; s<maxSV1; s++) {
	if (normalizedCounter[s]==-1) {
	tableDecode[highThreshold--].symbol = (FSE_FUNCTION_TYPE)s;
	symbolNext[s] = 1;
	} else {
	if (normalizedCounter[s] >= largeLimit) DTableH.fastMode=0;
	symbolNext[s] = normalizedCounter[s];
	} } }
	memcpy(dt, &DTableH, sizeof(DTableH));
	}

	/* Spread symbols */
	{ U32 const tableMask = tableSize-1;
	U32 const step = FSE_TABLESTEP(tableSize);
	U32 s, position = 0;
	for (s=0; s<maxSV1; s++) {
	int i;
	for (i=0; i<normalizedCounter[s]; i++) {
	tableDecode[position].symbol = (FSE_FUNCTION_TYPE)s;
	position = (position + step) & tableMask;
	while (position > highThreshold) position = (position + step) & tableMask; /* lowprob area */
	} }
	if (position!=0) return ERROR(GENERIC); /* position must reach all cells once, otherwise normalizedCounter is incorrect */
	}

	/* Build Decoding table */
	{ U32 u;
	for (u=0; u<tableSize; u++) {
	FSE_FUNCTION_TYPE const symbol = (FSE_FUNCTION_TYPE)(tableDecode[u].symbol);
	U32 const nextState = symbolNext[symbol]++;
	tableDecode[u].nbBits = (BYTE) (tableLog - BIT_highbit32(nextState) );
	tableDecode[u].newState = (U16) ( (nextState << tableDecode[u].nbBits) - tableSize);
	} }

	return 0;
	}


	#ifndef FSE_COMMONDEFS_ONLY

	/-******************************************************
	* Decompression (Byte symbols)
	*********************************************************/
	size_t FSE_buildDTable_rle (FSE_DTable* dt, BYTE symbolValue)
	{
	void* ptr = dt;
	FSE_DTableHeader* const DTableH = (FSE_DTableHeader*)ptr;
	void* dPtr = dt + 1;
	FSE_decode_t* const cell = (FSE_decode_t*)dPtr;

	DTableH->tableLog = 0;
	DTableH->fastMode = 0;

	cell->newState = 0;
	cell->symbol = symbolValue;
	cell->nbBits = 0;

	return 0;
	}


	size_t FSE_buildDTable_raw (FSE_DTable* dt, unsigned nbBits)
	{
	void* ptr = dt;
	FSE_DTableHeader* const DTableH = (FSE_DTableHeader*)ptr;
	void* dPtr = dt + 1;
	FSE_decode_t* const dinfo = (FSE_decode_t*)dPtr;
	const unsigned tableSize = 1 << nbBits;
	const unsigned tableMask = tableSize - 1;
	const unsigned maxSV1 = tableMask+1;
	unsigned s;

	/* Sanity checks */
	if (nbBits < 1) return ERROR(GENERIC); /* min size */

	/* Build Decoding Table */
	DTableH->tableLog = (U16)nbBits;
	DTableH->fastMode = 1;
	for (s=0; s<maxSV1; s++) {
	dinfo[s].newState = 0;
	dinfo[s].symbol = (BYTE)s;
	dinfo[s].nbBits = (BYTE)nbBits;
	}

	return 0;
	}

	FORCE_INLINE_TEMPLATE size_t FSE_decompress_usingDTable_generic(
	void* dst, size_t maxDstSize,
	const void* cSrc, size_t cSrcSize,
	const FSE_DTable* dt, const unsigned fast)
	{
	BYTE* const ostart = (BYTE*) dst;
	BYTE* op = ostart;
	BYTE* const omax = op + maxDstSize;
	BYTE* const olimit = omax-3;

	BIT_DStream_t bitD;
	FSE_DState_t state1;
	FSE_DState_t state2;

	/* Init */
	CHECK_F(BIT_initDStream(&bitD, cSrc, cSrcSize));

	FSE_initDState(&state1, &bitD, dt);
	FSE_initDState(&state2, &bitD, dt);

	#define FSE_GETSYMBOL(statePtr) fast ? FSE_decodeSymbolFast(statePtr, &bitD) : FSE_decodeSymbol(statePtr, &bitD)

	/* 4 symbols per loop */
	for ( ; (BIT_reloadDStream(&bitD)==BIT_DStream_unfinished) & (op<olimit) ; op+=4) {
	op[0] = FSE_GETSYMBOL(&state1);

	if (FSE_MAX_TABLELOG2+7 > sizeof(bitD.bitContainer)8) /* This test must be static */
	BIT_reloadDStream(&bitD);

	op[1] = FSE_GETSYMBOL(&state2);

	if (FSE_MAX_TABLELOG4+7 > sizeof(bitD.bitContainer)8) /* This test must be static */
	{ if (BIT_reloadDStream(&bitD) > BIT_DStream_unfinished) { op+=2; break; } }

	op[2] = FSE_GETSYMBOL(&state1);

	if (FSE_MAX_TABLELOG2+7 > sizeof(bitD.bitContainer)8) /* This test must be static */
	BIT_reloadDStream(&bitD);

	op[3] = FSE_GETSYMBOL(&state2);
	}

	/* tail */
	/* note : BIT_reloadDStream(&bitD) >= FSE_DStream_partiallyFilled; Ends at exactly BIT_DStream_completed */
	while (1) {
	if (op>(omax-2)) return ERROR(dstSize_tooSmall);
	*op++ = FSE_GETSYMBOL(&state1);
	if (BIT_reloadDStream(&bitD)==BIT_DStream_overflow) {
	*op++ = FSE_GETSYMBOL(&state2);
	break;
	}

	if (op>(omax-2)) return ERROR(dstSize_tooSmall);
	*op++ = FSE_GETSYMBOL(&state2);
	if (BIT_reloadDStream(&bitD)==BIT_DStream_overflow) {
	*op++ = FSE_GETSYMBOL(&state1);
	break;
	} }

	return op-ostart;
	}


	size_t FSE_decompress_usingDTable(void* dst, size_t originalSize,
	const void* cSrc, size_t cSrcSize,
	const FSE_DTable* dt)
	{
	const void* ptr = dt;
	const FSE_DTableHeader* DTableH = (const FSE_DTableHeader*)ptr;
	const U32 fastMode = DTableH->fastMode;

	/* select fast mode (static) */
	if (fastMode) return FSE_decompress_usingDTable_generic(dst, originalSize, cSrc, cSrcSize, dt, 1);
	return FSE_decompress_usingDTable_generic(dst, originalSize, cSrc, cSrcSize, dt, 0);
	}


	size_t FSE_decompress_wksp(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, FSE_DTable* workSpace, unsigned maxLog)
	{
	const BYTE* const istart = (const BYTE*)cSrc;
	const BYTE* ip = istart;
	short counting[FSE_MAX_SYMBOL_VALUE+1];
	unsigned tableLog;
	unsigned maxSymbolValue = FSE_MAX_SYMBOL_VALUE;

	/* normal FSE decoding mode */
	size_t const NCountLength = FSE_readNCount (counting, &maxSymbolValue, &tableLog, istart, cSrcSize);
	if (FSE_isError(NCountLength)) return NCountLength;
	/* if (NCountLength >= cSrcSize) return ERROR(srcSize_wrong); / / too small input size; supposed to be already checked in NCountLength, only remaining case : NCountLength==cSrcSize */
	if (tableLog > maxLog) return ERROR(tableLog_tooLarge);
	ip += NCountLength;
	cSrcSize -= NCountLength;

	CHECK_F( FSE_buildDTable (workSpace, counting, maxSymbolValue, tableLog) );

	return FSE_decompress_usingDTable (dst, dstCapacity, ip, cSrcSize, workSpace); /* always return, even if it is an error code */
	}


	typedef FSE_DTable DTable_max_t[FSE_DTABLE_SIZE_U32(FSE_MAX_TABLELOG)];

	size_t FSE_decompress(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize)
	{
	DTable_max_t dt; /* Static analyzer seems unable to understand this table will be properly initialized later */
	return FSE_decompress_wksp(dst, dstCapacity, cSrc, cSrcSize, dt, FSE_MAX_TABLELOG);
	}



	#endif /* FSE_COMMONDEFS_ONLY */
	/** ended inlining common/fse_decompress.c **/
	/** start inlining common/pool.c **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/


	/* ====== Dependencies ======= */
	#include <stddef.h> /* size_t */
	/** skipping file: debug.h **/
	/** start inlining zstd_internal.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTD_CCOMMON_H_MODULE
	#define ZSTD_CCOMMON_H_MODULE

	/* this module contains definitions which must be identical
	* across compression, decompression and dictBuilder.
	* It also contains a few functions useful to at least 2 of them
	* and which benefit from being inlined */

	/-************************************
	* Dependencies
	***************************************/
	-#ifdef __aarch64__
	+#if !defined(ZSTD_NO_INTRINSICS) && defined(__ARM_NEON)
	#include <arm_neon.h>
	#endif
	/** skipping file: compiler.h **/
	/** skipping file: mem.h **/
	/** skipping file: debug.h **/
	/** skipping file: error_private.h **/
	#define ZSTD_STATIC_LINKING_ONLY
	/** start inlining ../zstd.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/
	#if defined (__cplusplus)
	extern "C" {
	#endif

	#ifndef ZSTD_H_235446
	#define ZSTD_H_235446

	/* ====== Dependency ======*/
	#include <limits.h> /* INT_MAX */
	#include <stddef.h> /* size_t */


	/* ===== ZSTDLIB_API : control library symbols visibility ===== */
	#ifndef ZSTDLIB_VISIBILITY
	# if defined(__GNUC__) && (__GNUC__ >= 4)
	# define ZSTDLIB_VISIBILITY __attribute__ ((visibility ("default")))
	# else
	# define ZSTDLIB_VISIBILITY
	# endif
	#endif
	#if defined(ZSTD_DLL_EXPORT) && (ZSTD_DLL_EXPORT==1)
	# define ZSTDLIB_API __declspec(dllexport) ZSTDLIB_VISIBILITY
	#elif defined(ZSTD_DLL_IMPORT) && (ZSTD_DLL_IMPORT==1)
	# define ZSTDLIB_API __declspec(dllimport) ZSTDLIB_VISIBILITY /* It isn't required but allows to generate better code, saving a function pointer load from the IAT and an indirect jump.*/
	#else
	# define ZSTDLIB_API ZSTDLIB_VISIBILITY
	#endif


	/*******************************************************************************
	Introduction

	zstd, short for Zstandard, is a fast lossless compression algorithm, targeting
	real-time compression scenarios at zlib-level and better compression ratios.
	The zstd compression library provides in-memory compression and decompression
	functions.

	The library supports regular compression levels from 1 up to ZSTD_maxCLevel(),
	which is currently 22. Levels >= 20, labeled `--ultra`, should be used with
	caution, as they require more memory. The library also offers negative
	compression levels, which extend the range of speed vs. ratio preferences.
	The lower the level, the faster the speed (at the cost of compression).

	Compression can be done in:
	- a single step (described as Simple API)
	- a single step, reusing a context (described as Explicit context)
	- unbounded multiple steps (described as Streaming compression)

	The compression ratio achievable on small data can be highly improved using
	a dictionary. Dictionary compression can be performed in:
	- a single step (described as Simple dictionary API)
	- a single step, reusing a dictionary (described as Bulk-processing
	dictionary API)

	Advanced experimental functions can be accessed using
	`#define ZSTD_STATIC_LINKING_ONLY` before including zstd.h.

	Advanced experimental APIs should never be used with a dynamically-linked
	library. They are not "stable"; their definitions or signatures may change in
	the future. Only static linking is allowed.
	*******************************************************************************/

	/------ Version ------/
	#define ZSTD_VERSION_MAJOR 1
	#define ZSTD_VERSION_MINOR 4
	#define ZSTD_VERSION_RELEASE 5

	#define ZSTD_VERSION_NUMBER (ZSTD_VERSION_MAJOR 100100 + ZSTD_VERSION_MINOR *100 + ZSTD_VERSION_RELEASE)
	ZSTDLIB_API unsigned ZSTD_versionNumber(void); /*< to check runtime library version /

	#define ZSTD_LIB_VERSION ZSTD_VERSION_MAJOR.ZSTD_VERSION_MINOR.ZSTD_VERSION_RELEASE
	#define ZSTD_QUOTE(str) #str
	#define ZSTD_EXPAND_AND_QUOTE(str) ZSTD_QUOTE(str)
	#define ZSTD_VERSION_STRING ZSTD_EXPAND_AND_QUOTE(ZSTD_LIB_VERSION)
	ZSTDLIB_API const char* ZSTD_versionString(void); /* requires v1.3.0+ */

	/* *************************************
	* Default constant
	***************************************/
	#ifndef ZSTD_CLEVEL_DEFAULT
	# define ZSTD_CLEVEL_DEFAULT 3
	#endif

	/* *************************************
	* Constants
	***************************************/

	/* All magic numbers are supposed read/written to/from files/memory using little-endian convention */
	#define ZSTD_MAGICNUMBER 0xFD2FB528 /* valid since v0.8.0 */
	#define ZSTD_MAGIC_DICTIONARY 0xEC30A437 /* valid since v0.7.0 */
	#define ZSTD_MAGIC_SKIPPABLE_START 0x184D2A50 /* all 16 values, from 0x184D2A50 to 0x184D2A5F, signal the beginning of a skippable frame */
	#define ZSTD_MAGIC_SKIPPABLE_MASK 0xFFFFFFF0

	#define ZSTD_BLOCKSIZELOG_MAX 17
	#define ZSTD_BLOCKSIZE_MAX (1<<ZSTD_BLOCKSIZELOG_MAX)



	/***************************************
	* Simple API
	***************************************/
	/*! ZSTD_compress() :
	* Compresses `src` content as a single zstd compressed frame into already allocated `dst`.
	* Hint : compression runs faster if `dstCapacity` >= `ZSTD_compressBound(srcSize)`.
	* @return : compressed size written into `dst` (<= `dstCapacity),
	* or an error code if it fails (which can be tested using ZSTD_isError()). */
	ZSTDLIB_API size_t ZSTD_compress( void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	int compressionLevel);

	/*! ZSTD_decompress() :
	* `compressedSize` : must be the _exact_ size of some number of compressed and/or skippable frames.
	* `dstCapacity` is an upper bound of originalSize to regenerate.
	* If user cannot imply a maximum upper bound, it's better to use streaming mode to decompress data.
	* @return : the number of bytes decompressed into `dst` (<= `dstCapacity`),
	* or an errorCode if it fails (which can be tested using ZSTD_isError()). */
	ZSTDLIB_API size_t ZSTD_decompress( void* dst, size_t dstCapacity,
	const void* src, size_t compressedSize);

	/*! ZSTD_getFrameContentSize() : requires v1.3.0+
	* `src` should point to the start of a ZSTD encoded frame.
	* `srcSize` must be at least as large as the frame header.
	* hint : any size >= `ZSTD_frameHeaderSize_max` is large enough.
	* @return : - decompressed size of `src` frame content, if known
	* - ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined
	* - ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too small)
	* note 1 : a 0 return value means the frame is valid but "empty".
	* note 2 : decompressed size is an optional field, it may not be present, typically in streaming mode.
	* When `return==ZSTD_CONTENTSIZE_UNKNOWN`, data to decompress could be any size.
	* In which case, it's necessary to use streaming mode to decompress data.
	* Optionally, application can rely on some implicit limit,
	* as ZSTD_decompress() only needs an upper bound of decompressed size.
	* (For example, data could be necessarily cut into blocks <= 16 KB).
	* note 3 : decompressed size is always present when compression is completed using single-pass functions,
	* such as ZSTD_compress(), ZSTD_compressCCtx() ZSTD_compress_usingDict() or ZSTD_compress_usingCDict().
	* note 4 : decompressed size can be very large (64-bits value),
	* potentially larger than what local system can handle as a single memory segment.
	* In which case, it's necessary to use streaming mode to decompress data.
	* note 5 : If source is untrusted, decompressed size could be wrong or intentionally modified.
	* Always ensure return value fits within application's authorized limits.
	* Each application can set its own limits.
	* note 6 : This function replaces ZSTD_getDecompressedSize() */
	#define ZSTD_CONTENTSIZE_UNKNOWN (0ULL - 1)
	#define ZSTD_CONTENTSIZE_ERROR (0ULL - 2)
	ZSTDLIB_API unsigned long long ZSTD_getFrameContentSize(const void *src, size_t srcSize);

	/*! ZSTD_getDecompressedSize() :
	* NOTE: This function is now obsolete, in favor of ZSTD_getFrameContentSize().
	* Both functions work the same way, but ZSTD_getDecompressedSize() blends
	* "empty", "unknown" and "error" results to the same return value (0),
	* while ZSTD_getFrameContentSize() gives them separate return values.
	* @return : decompressed size of `src` frame content _if known and not empty_, 0 otherwise. */
	ZSTDLIB_API unsigned long long ZSTD_getDecompressedSize(const void* src, size_t srcSize);

	/*! ZSTD_findFrameCompressedSize() :
	* `src` should point to the start of a ZSTD frame or skippable frame.
	* `srcSize` must be >= first frame size
	* @return : the compressed size of the first frame starting at `src`,
	* suitable to pass as `srcSize` to `ZSTD_decompress` or similar,
	* or an error code if input is invalid */
	ZSTDLIB_API size_t ZSTD_findFrameCompressedSize(const void* src, size_t srcSize);


	/====== Helper functions ======/
	#define ZSTD_COMPRESSBOUND(srcSize) ((srcSize) + ((srcSize)>>8) + (((srcSize) < (128<<10)) ? (((128<<10) - (srcSize)) >> 11) /* margin, from 64 to 0 / : 0)) / this formula ensures that bound(A) + bound(B) <= bound(A+B) as long as A and B >= 128 KB */
	ZSTDLIB_API size_t ZSTD_compressBound(size_t srcSize); /!< maximum compressed size in worst case single-pass scenario /
	ZSTDLIB_API unsigned ZSTD_isError(size_t code); /!< tells if a `size_t` function result is an error code /
	ZSTDLIB_API const char* ZSTD_getErrorName(size_t code); /!< provides readable string from an error code /
	ZSTDLIB_API int ZSTD_minCLevel(void); /!< minimum negative compression level allowed /
	ZSTDLIB_API int ZSTD_maxCLevel(void); /!< maximum compression level available /


	/***************************************
	* Explicit context
	***************************************/
	/*= Compression context
	* When compressing many times,
	* it is recommended to allocate a context just once,
	* and re-use it for each successive compression operation.
	* This will make workload friendlier for system's memory.
	* Note : re-using context is just a speed / resource optimization.
	* It doesn't change the compression ratio, which remains identical.
	* Note 2 : In multi-threaded environments,
	* use one different context per thread for parallel execution.
	*/
	typedef struct ZSTD_CCtx_s ZSTD_CCtx;
	ZSTDLIB_API ZSTD_CCtx* ZSTD_createCCtx(void);
	ZSTDLIB_API size_t ZSTD_freeCCtx(ZSTD_CCtx* cctx);

	/*! ZSTD_compressCCtx() :
	* Same as ZSTD_compress(), using an explicit ZSTD_CCtx.
	* Important : in order to behave similarly to `ZSTD_compress()`,
	* this function compresses at requested compression level,
	* __ignoring any other parameter__ .
	* If any advanced parameter was set using the advanced API,
	* they will all be reset. Only `compressionLevel` remains.
	*/
	ZSTDLIB_API size_t ZSTD_compressCCtx(ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	int compressionLevel);

	/*= Decompression context
	* When decompressing many times,
	* it is recommended to allocate a context only once,
	* and re-use it for each successive compression operation.
	* This will make workload friendlier for system's memory.
	* Use one context per thread for parallel execution. */
	typedef struct ZSTD_DCtx_s ZSTD_DCtx;
	ZSTDLIB_API ZSTD_DCtx* ZSTD_createDCtx(void);
	ZSTDLIB_API size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx);

	/*! ZSTD_decompressDCtx() :
	* Same as ZSTD_decompress(),
	* requires an allocated ZSTD_DCtx.
	* Compatible with sticky parameters.
	*/
	ZSTDLIB_API size_t ZSTD_decompressDCtx(ZSTD_DCtx* dctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize);


	/***************************************
	* Advanced compression API
	***************************************/

	/* API design :
	* Parameters are pushed one by one into an existing context,
	* using ZSTD_CCtx_set*() functions.
	* Pushed parameters are sticky : they are valid for next compressed frame, and any subsequent frame.
	* "sticky" parameters are applicable to `ZSTD_compress2()` and `ZSTD_compressStream*()` !
	* __They do not apply to "simple" one-shot variants such as ZSTD_compressCCtx()__ .
	*
	* It's possible to reset all parameters to "default" using ZSTD_CCtx_reset().
	*
	* This API supercedes all other "advanced" API entry points in the experimental section.
	* In the future, we expect to remove from experimental API entry points which are redundant with this API.
	*/


	/* Compression strategies, listed from fastest to strongest */
	typedef enum { ZSTD_fast=1,
	ZSTD_dfast=2,
	ZSTD_greedy=3,
	ZSTD_lazy=4,
	ZSTD_lazy2=5,
	ZSTD_btlazy2=6,
	ZSTD_btopt=7,
	ZSTD_btultra=8,
	ZSTD_btultra2=9
	/* note : new strategies _might_ be added in the future.
	Only the order (from fast to strong) is guaranteed */
	} ZSTD_strategy;


	typedef enum {

	/* compression parameters
	* Note: When compressing with a ZSTD_CDict these parameters are superseded
	* by the parameters used to construct the ZSTD_CDict.
	* See ZSTD_CCtx_refCDict() for more info (superseded-by-cdict). */
	ZSTD_c_compressionLevel=100, /* Set compression parameters according to pre-defined cLevel table.
	* Note that exact compression parameters are dynamically determined,
	* depending on both compression level and srcSize (when known).
	* Default level is ZSTD_CLEVEL_DEFAULT==3.
	* Special: value 0 means default, which is controlled by ZSTD_CLEVEL_DEFAULT.
	* Note 1 : it's possible to pass a negative compression level.
	* Note 2 : setting a level does not automatically set all other compression parameters
	* to default. Setting this will however eventually dynamically impact the compression
	* parameters which have not been manually set. The manually set
	* ones will 'stick'. */
	/* Advanced compression parameters :
	* It's possible to pin down compression parameters to some specific values.
	* In which case, these values are no longer dynamically selected by the compressor */
	ZSTD_c_windowLog=101, /* Maximum allowed back-reference distance, expressed as power of 2.
	* This will set a memory budget for streaming decompression,
	* with larger values requiring more memory
	* and typically compressing more.
	* Must be clamped between ZSTD_WINDOWLOG_MIN and ZSTD_WINDOWLOG_MAX.
	* Special: value 0 means "use default windowLog".
	* Note: Using a windowLog greater than ZSTD_WINDOWLOG_LIMIT_DEFAULT
	* requires explicitly allowing such size at streaming decompression stage. */
	ZSTD_c_hashLog=102, /* Size of the initial probe table, as a power of 2.
	* Resulting memory usage is (1 << (hashLog+2)).
	* Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX.
	* Larger tables improve compression ratio of strategies <= dFast,
	* and improve speed of strategies > dFast.
	* Special: value 0 means "use default hashLog". */
	ZSTD_c_chainLog=103, /* Size of the multi-probe search table, as a power of 2.
	* Resulting memory usage is (1 << (chainLog+2)).
	* Must be clamped between ZSTD_CHAINLOG_MIN and ZSTD_CHAINLOG_MAX.
	* Larger tables result in better and slower compression.
	* This parameter is useless for "fast" strategy.
	* It's still useful when using "dfast" strategy,
	* in which case it defines a secondary probe table.
	* Special: value 0 means "use default chainLog". */
	ZSTD_c_searchLog=104, /* Number of search attempts, as a power of 2.
	* More attempts result in better and slower compression.
	* This parameter is useless for "fast" and "dFast" strategies.
	* Special: value 0 means "use default searchLog". */
	ZSTD_c_minMatch=105, /* Minimum size of searched matches.
	* Note that Zstandard can still find matches of smaller size,
	* it just tweaks its search algorithm to look for this size and larger.
	* Larger values increase compression and decompression speed, but decrease ratio.
	* Must be clamped between ZSTD_MINMATCH_MIN and ZSTD_MINMATCH_MAX.
	* Note that currently, for all strategies < btopt, effective minimum is 4.
	* , for all strategies > fast, effective maximum is 6.
	* Special: value 0 means "use default minMatchLength". */
	ZSTD_c_targetLength=106, /* Impact of this field depends on strategy.
	* For strategies btopt, btultra & btultra2:
	* Length of Match considered "good enough" to stop search.
	* Larger values make compression stronger, and slower.
	* For strategy fast:
	* Distance between match sampling.
	* Larger values make compression faster, and weaker.
	* Special: value 0 means "use default targetLength". */
	ZSTD_c_strategy=107, /* See ZSTD_strategy enum definition.
	* The higher the value of selected strategy, the more complex it is,
	* resulting in stronger and slower compression.
	* Special: value 0 means "use default strategy". */

	/* LDM mode parameters */
	ZSTD_c_enableLongDistanceMatching=160, /* Enable long distance matching.
	* This parameter is designed to improve compression ratio
	* for large inputs, by finding large matches at long distance.
	* It increases memory usage and window size.
	* Note: enabling this parameter increases default ZSTD_c_windowLog to 128 MB
	* except when expressly set to a different value. */
	ZSTD_c_ldmHashLog=161, /* Size of the table for long distance matching, as a power of 2.
	* Larger values increase memory usage and compression ratio,
	* but decrease compression speed.
	* Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX
	* default: windowlog - 7.
	* Special: value 0 means "automatically determine hashlog". */
	ZSTD_c_ldmMinMatch=162, /* Minimum match size for long distance matcher.
	* Larger/too small values usually decrease compression ratio.
	* Must be clamped between ZSTD_LDM_MINMATCH_MIN and ZSTD_LDM_MINMATCH_MAX.
	* Special: value 0 means "use default value" (default: 64). */
	ZSTD_c_ldmBucketSizeLog=163, /* Log size of each bucket in the LDM hash table for collision resolution.
	* Larger values improve collision resolution but decrease compression speed.
	* The maximum value is ZSTD_LDM_BUCKETSIZELOG_MAX.
	* Special: value 0 means "use default value" (default: 3). */
	ZSTD_c_ldmHashRateLog=164, /* Frequency of inserting/looking up entries into the LDM hash table.
	* Must be clamped between 0 and (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN).
	* Default is MAX(0, (windowLog - ldmHashLog)), optimizing hash table usage.
	* Larger values improve compression speed.
	* Deviating far from default value will likely result in a compression ratio decrease.
	* Special: value 0 means "automatically determine hashRateLog". */

	/* frame parameters */
	ZSTD_c_contentSizeFlag=200, /* Content size will be written into frame header _whenever known_ (default:1)
	* Content size must be known at the beginning of compression.
	* This is automatically the case when using ZSTD_compress2(),
	* For streaming scenarios, content size must be provided with ZSTD_CCtx_setPledgedSrcSize() */
	ZSTD_c_checksumFlag=201, /* A 32-bits checksum of content is written at end of frame (default:0) */
	ZSTD_c_dictIDFlag=202, /* When applicable, dictionary's ID is written into frame header (default:1) */

	/* multi-threading parameters */
	/* These parameters are only useful if multi-threading is enabled (compiled with build macro ZSTD_MULTITHREAD).
	* They return an error otherwise. */
	ZSTD_c_nbWorkers=400, /* Select how many threads will be spawned to compress in parallel.
	* When nbWorkers >= 1, triggers asynchronous mode when used with ZSTD_compressStream*() :
	* ZSTD_compressStream*() consumes input and flush output if possible, but immediately gives back control to caller,
	* while compression work is performed in parallel, within worker threads.
	* (note : a strong exception to this rule is when first invocation of ZSTD_compressStream2() sets ZSTD_e_end :
	* in which case, ZSTD_compressStream2() delegates to ZSTD_compress2(), which is always a blocking call).
	* More workers improve speed, but also increase memory usage.
	* Default value is `0`, aka "single-threaded mode" : no worker is spawned, compression is performed inside Caller's thread, all invocations are blocking */
	ZSTD_c_jobSize=401, /* Size of a compression job. This value is enforced only when nbWorkers >= 1.
	* Each compression job is completed in parallel, so this value can indirectly impact the nb of active threads.
	* 0 means default, which is dynamically determined based on compression parameters.
	* Job size must be a minimum of overlap size, or 1 MB, whichever is largest.
	* The minimum size is automatically and transparently enforced. */
	ZSTD_c_overlapLog=402, /* Control the overlap size, as a fraction of window size.
	* The overlap size is an amount of data reloaded from previous job at the beginning of a new job.
	* It helps preserve compression ratio, while each job is compressed in parallel.
	* This value is enforced only when nbWorkers >= 1.
	* Larger values increase compression ratio, but decrease speed.
	* Possible values range from 0 to 9 :
	* - 0 means "default" : value will be determined by the library, depending on strategy
	* - 1 means "no overlap"
	* - 9 means "full overlap", using a full window size.
	* Each intermediate rank increases/decreases load size by a factor 2 :
	* 9: full window; 8: w/2; 7: w/4; 6: w/8; 5:w/16; 4: w/32; 3:w/64; 2:w/128; 1:no overlap; 0:default
	* default value varies between 6 and 9, depending on strategy */

	/* note : additional experimental parameters are also available
	* within the experimental section of the API.
	* At the time of this writing, they include :
	* ZSTD_c_rsyncable
	* ZSTD_c_format
	* ZSTD_c_forceMaxWindow
	* ZSTD_c_forceAttachDict
	* ZSTD_c_literalCompressionMode
	* ZSTD_c_targetCBlockSize
	* ZSTD_c_srcSizeHint
	* Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them.
	* note : never ever use experimentalParam? names directly;
	* also, the enums values themselves are unstable and can still change.
	*/
	ZSTD_c_experimentalParam1=500,
	ZSTD_c_experimentalParam2=10,
	ZSTD_c_experimentalParam3=1000,
	ZSTD_c_experimentalParam4=1001,
	ZSTD_c_experimentalParam5=1002,
	ZSTD_c_experimentalParam6=1003,
	ZSTD_c_experimentalParam7=1004
	} ZSTD_cParameter;

	typedef struct {
	size_t error;
	int lowerBound;
	int upperBound;
	} ZSTD_bounds;

	/*! ZSTD_cParam_getBounds() :
	* All parameters must belong to an interval with lower and upper bounds,
	* otherwise they will either trigger an error or be automatically clamped.
	* @return : a structure, ZSTD_bounds, which contains
	* - an error status field, which must be tested using ZSTD_isError()
	* - lower and upper bounds, both inclusive
	*/
	ZSTDLIB_API ZSTD_bounds ZSTD_cParam_getBounds(ZSTD_cParameter cParam);

	/*! ZSTD_CCtx_setParameter() :
	* Set one compression parameter, selected by enum ZSTD_cParameter.
	* All parameters have valid bounds. Bounds can be queried using ZSTD_cParam_getBounds().
	* Providing a value beyond bound will either clamp it, or trigger an error (depending on parameter).
	* Setting a parameter is generally only possible during frame initialization (before starting compression).
	* Exception : when using multi-threading mode (nbWorkers >= 1),
	* the following parameters can be updated _during_ compression (within same frame):
	* => compressionLevel, hashLog, chainLog, searchLog, minMatch, targetLength and strategy.
	* new parameters will be active for next job only (after a flush()).
	* @return : an error code (which can be tested using ZSTD_isError()).
	*/
	ZSTDLIB_API size_t ZSTD_CCtx_setParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int value);

	/*! ZSTD_CCtx_setPledgedSrcSize() :
	* Total input data size to be compressed as a single frame.
	* Value will be written in frame header, unless if explicitly forbidden using ZSTD_c_contentSizeFlag.
	* This value will also be controlled at end of frame, and trigger an error if not respected.
	* @result : 0, or an error code (which can be tested with ZSTD_isError()).
	* Note 1 : pledgedSrcSize==0 actually means zero, aka an empty frame.
	* In order to mean "unknown content size", pass constant ZSTD_CONTENTSIZE_UNKNOWN.
	* ZSTD_CONTENTSIZE_UNKNOWN is default value for any new frame.
	* Note 2 : pledgedSrcSize is only valid once, for the next frame.
	* It's discarded at the end of the frame, and replaced by ZSTD_CONTENTSIZE_UNKNOWN.
	* Note 3 : Whenever all input data is provided and consumed in a single round,
	* for example with ZSTD_compress2(),
	* or invoking immediately ZSTD_compressStream2(,,,ZSTD_e_end),
	* this value is automatically overridden by srcSize instead.
	*/
	ZSTDLIB_API size_t ZSTD_CCtx_setPledgedSrcSize(ZSTD_CCtx* cctx, unsigned long long pledgedSrcSize);

	typedef enum {
	ZSTD_reset_session_only = 1,
	ZSTD_reset_parameters = 2,
	ZSTD_reset_session_and_parameters = 3
	} ZSTD_ResetDirective;

	/*! ZSTD_CCtx_reset() :
	* There are 2 different things that can be reset, independently or jointly :
	* - The session : will stop compressing current frame, and make CCtx ready to start a new one.
	* Useful after an error, or to interrupt any ongoing compression.
	* Any internal data not yet flushed is cancelled.
	* Compression parameters and dictionary remain unchanged.
	* They will be used to compress next frame.
	* Resetting session never fails.
	* - The parameters : changes all parameters back to "default".
	* This removes any reference to any dictionary too.
	* Parameters can only be changed between 2 sessions (i.e. no compression is currently ongoing)
	* otherwise the reset fails, and function returns an error value (which can be tested using ZSTD_isError())
	* - Both : similar to resetting the session, followed by resetting parameters.
	*/
	ZSTDLIB_API size_t ZSTD_CCtx_reset(ZSTD_CCtx* cctx, ZSTD_ResetDirective reset);

	/*! ZSTD_compress2() :
	* Behave the same as ZSTD_compressCCtx(), but compression parameters are set using the advanced API.
	* ZSTD_compress2() always starts a new frame.
	* Should cctx hold data from a previously unfinished frame, everything about it is forgotten.
	* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*()
	* - The function is always blocking, returns when compression is completed.
	* Hint : compression runs faster if `dstCapacity` >= `ZSTD_compressBound(srcSize)`.
	* @return : compressed size written into `dst` (<= `dstCapacity),
	* or an error code if it fails (which can be tested using ZSTD_isError()).
	*/
	ZSTDLIB_API size_t ZSTD_compress2( ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize);


	/***************************************
	* Advanced decompression API
	***************************************/

	/* The advanced API pushes parameters one by one into an existing DCtx context.
	* Parameters are sticky, and remain valid for all following frames
	* using the same DCtx context.
	* It's possible to reset parameters to default values using ZSTD_DCtx_reset().
	* Note : This API is compatible with existing ZSTD_decompressDCtx() and ZSTD_decompressStream().
	* Therefore, no new decompression function is necessary.
	*/

	typedef enum {

	ZSTD_d_windowLogMax=100, /* Select a size limit (in power of 2) beyond which
	* the streaming API will refuse to allocate memory buffer
	* in order to protect the host from unreasonable memory requirements.
	* This parameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
	* By default, a decompression context accepts window sizes <= (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT).
	* Special: value 0 means "use default maximum windowLog". */

	/* note : additional experimental parameters are also available
	* within the experimental section of the API.
	* At the time of this writing, they include :
	* ZSTD_d_format
	* ZSTD_d_stableOutBuffer
	* Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them.
	* note : never ever use experimentalParam? names directly
	*/
	ZSTD_d_experimentalParam1=1000,
	ZSTD_d_experimentalParam2=1001

	} ZSTD_dParameter;

	/*! ZSTD_dParam_getBounds() :
	* All parameters must belong to an interval with lower and upper bounds,
	* otherwise they will either trigger an error or be automatically clamped.
	* @return : a structure, ZSTD_bounds, which contains
	* - an error status field, which must be tested using ZSTD_isError()
	* - both lower and upper bounds, inclusive
	*/
	ZSTDLIB_API ZSTD_bounds ZSTD_dParam_getBounds(ZSTD_dParameter dParam);

	/*! ZSTD_DCtx_setParameter() :
	* Set one compression parameter, selected by enum ZSTD_dParameter.
	* All parameters have valid bounds. Bounds can be queried using ZSTD_dParam_getBounds().
	* Providing a value beyond bound will either clamp it, or trigger an error (depending on parameter).
	* Setting a parameter is only possible during frame initialization (before starting decompression).
	* @return : 0, or an error code (which can be tested using ZSTD_isError()).
	*/
	ZSTDLIB_API size_t ZSTD_DCtx_setParameter(ZSTD_DCtx* dctx, ZSTD_dParameter param, int value);

	/*! ZSTD_DCtx_reset() :
	* Return a DCtx to clean state.
	* Session and parameters can be reset jointly or separately.
	* Parameters can only be reset when no active frame is being decompressed.
	* @return : 0, or an error code, which can be tested with ZSTD_isError()
	*/
	ZSTDLIB_API size_t ZSTD_DCtx_reset(ZSTD_DCtx* dctx, ZSTD_ResetDirective reset);


	/****************************
	* Streaming
	****************************/

	typedef struct ZSTD_inBuffer_s {
	const void* src; /*< start of input buffer /
	size_t size; /*< size of input buffer /
	size_t pos; /*< position where reading stopped. Will be updated. Necessarily 0 <= pos <= size /
	} ZSTD_inBuffer;

	typedef struct ZSTD_outBuffer_s {
	void* dst; /*< start of output buffer /
	size_t size; /*< size of output buffer /
	size_t pos; /*< position where writing stopped. Will be updated. Necessarily 0 <= pos <= size /
	} ZSTD_outBuffer;



	/-**********************************************************************
	* Streaming compression - HowTo
	*
	* A ZSTD_CStream object is required to track streaming operation.
	* Use ZSTD_createCStream() and ZSTD_freeCStream() to create/release resources.
	* ZSTD_CStream objects can be reused multiple times on consecutive compression operations.
	* It is recommended to re-use ZSTD_CStream since it will play nicer with system's memory, by re-using already allocated memory.
	*
	* For parallel execution, use one separate ZSTD_CStream per thread.
	*
	* note : since v1.3.0, ZSTD_CStream and ZSTD_CCtx are the same thing.
	*
	* Parameters are sticky : when starting a new compression on the same context,
	* it will re-use the same sticky parameters as previous compression session.
	* When in doubt, it's recommended to fully initialize the context before usage.
	* Use ZSTD_CCtx_reset() to reset the context and ZSTD_CCtx_setParameter(),
	* ZSTD_CCtx_setPledgedSrcSize(), or ZSTD_CCtx_loadDictionary() and friends to
	* set more specific parameters, the pledged source size, or load a dictionary.
	*
	* Use ZSTD_compressStream2() with ZSTD_e_continue as many times as necessary to
	* consume input stream. The function will automatically update both `pos`
	* fields within `input` and `output`.
	* Note that the function may not consume the entire input, for example, because
	* the output buffer is already full, in which case `input.pos < input.size`.
	* The caller must check if input has been entirely consumed.
	* If not, the caller must make some room to receive more compressed data,
	* and then present again remaining input data.
	* note: ZSTD_e_continue is guaranteed to make some forward progress when called,
	* but doesn't guarantee maximal forward progress. This is especially relevant
	* when compressing with multiple threads. The call won't block if it can
	* consume some input, but if it can't it will wait for some, but not all,
	* output to be flushed.
	* @return : provides a minimum amount of data remaining to be flushed from internal buffers
	* or an error code, which can be tested using ZSTD_isError().
	*
	* At any moment, it's possible to flush whatever data might remain stuck within internal buffer,
	* using ZSTD_compressStream2() with ZSTD_e_flush. `output->pos` will be updated.
	* Note that, if `output->size` is too small, a single invocation with ZSTD_e_flush might not be enough (return code > 0).
	* In which case, make some room to receive more compressed data, and call again ZSTD_compressStream2() with ZSTD_e_flush.
	* You must continue calling ZSTD_compressStream2() with ZSTD_e_flush until it returns 0, at which point you can change the
	* operation.
	* note: ZSTD_e_flush will flush as much output as possible, meaning when compressing with multiple threads, it will
	* block until the flush is complete or the output buffer is full.
	* @return : 0 if internal buffers are entirely flushed,
	* >0 if some data still present within internal buffer (the value is minimal estimation of remaining size),
	* or an error code, which can be tested using ZSTD_isError().
	*
	* Calling ZSTD_compressStream2() with ZSTD_e_end instructs to finish a frame.
	* It will perform a flush and write frame epilogue.
	* The epilogue is required for decoders to consider a frame completed.
	* flush operation is the same, and follows same rules as calling ZSTD_compressStream2() with ZSTD_e_flush.
	* You must continue calling ZSTD_compressStream2() with ZSTD_e_end until it returns 0, at which point you are free to
	* start a new frame.
	* note: ZSTD_e_end will flush as much output as possible, meaning when compressing with multiple threads, it will
	* block until the flush is complete or the output buffer is full.
	* @return : 0 if frame fully completed and fully flushed,
	* >0 if some data still present within internal buffer (the value is minimal estimation of remaining size),
	* or an error code, which can be tested using ZSTD_isError().
	*
	* *******************************************************************/

	typedef ZSTD_CCtx ZSTD_CStream; /*< CCtx and CStream are now effectively same object (>= v1.3.0) /
	/* Continue to distinguish them for compatibility with older versions <= v1.2.0 */
	/===== ZSTD_CStream management functions =====/
	ZSTDLIB_API ZSTD_CStream* ZSTD_createCStream(void);
	ZSTDLIB_API size_t ZSTD_freeCStream(ZSTD_CStream* zcs);

	/===== Streaming compression functions =====/
	typedef enum {
	ZSTD_e_continue=0, /* collect more data, encoder decides when to output compressed result, for optimal compression ratio */
	ZSTD_e_flush=1, /* flush any data provided so far,
	* it creates (at least) one new block, that can be decoded immediately on reception;
	* frame will continue: any future data can still reference previously compressed data, improving compression.
	* note : multithreaded compression will block to flush as much output as possible. */
	ZSTD_e_end=2 /* flush any remaining data _and_ close current frame.
	* note that frame is only closed after compressed data is fully flushed (return value == 0).
	* After that point, any additional data starts a new frame.
	* note : each frame is independent (does not reference any content from previous frame).
	: note : multithreaded compression will block to flush as much output as possible. */
	} ZSTD_EndDirective;

	/*! ZSTD_compressStream2() :
	* Behaves about the same as ZSTD_compressStream, with additional control on end directive.
	* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*()
	* - Compression parameters cannot be changed once compression is started (save a list of exceptions in multi-threading mode)
	* - output->pos must be <= dstCapacity, input->pos must be <= srcSize
	* - output->pos and input->pos will be updated. They are guaranteed to remain below their respective limit.
	* - When nbWorkers==0 (default), function is blocking : it completes its job before returning to caller.
	* - When nbWorkers>=1, function is non-blocking : it just acquires a copy of input, and distributes jobs to internal worker threads, flush whatever is available,
	* and then immediately returns, just indicating that there is some data remaining to be flushed.
	* The function nonetheless guarantees forward progress : it will return only after it reads or write at least 1+ byte.
	* - Exception : if the first call requests a ZSTD_e_end directive and provides enough dstCapacity, the function delegates to ZSTD_compress2() which is always blocking.
	* - @return provides a minimum amount of data remaining to be flushed from internal buffers
	* or an error code, which can be tested using ZSTD_isError().
	* if @return != 0, flush is not fully completed, there is still some data left within internal buffers.
	* This is useful for ZSTD_e_flush, since in this case more flushes are necessary to empty all buffers.
	* For ZSTD_e_end, @return == 0 when internal buffers are fully flushed and frame is completed.
	* - after a ZSTD_e_end directive, if internal buffer is not fully flushed (@return != 0),
	* only ZSTD_e_end or ZSTD_e_flush operations are allowed.
	* Before starting a new compression job, or changing compression parameters,
	* it is required to fully flush internal buffers.
	*/
	ZSTDLIB_API size_t ZSTD_compressStream2( ZSTD_CCtx* cctx,
	ZSTD_outBuffer* output,
	ZSTD_inBuffer* input,
	ZSTD_EndDirective endOp);


	/* These buffer sizes are softly recommended.
	* They are not required : ZSTD_compressStream*() happily accepts any buffer size, for both input and output.
	* Respecting the recommended size just makes it a bit easier for ZSTD_compressStream*(),
	* reducing the amount of memory shuffling and buffering, resulting in minor performance savings.
	*
	* However, note that these recommendations are from the perspective of a C caller program.
	* If the streaming interface is invoked from some other language,
	* especially managed ones such as Java or Go, through a foreign function interface such as jni or cgo,
	* a major performance rule is to reduce crossing such interface to an absolute minimum.
	* It's not rare that performance ends being spent more into the interface, rather than compression itself.
	* In which cases, prefer using large buffers, as large as practical,
	* for both input and output, to reduce the nb of roundtrips.
	*/
	ZSTDLIB_API size_t ZSTD_CStreamInSize(void); /*< recommended size for input buffer /
	ZSTDLIB_API size_t ZSTD_CStreamOutSize(void); /*< recommended size for output buffer. Guarantee to successfully flush at least one complete compressed block. /


	/* *****************************************************************************
	* This following is a legacy streaming API.
	* It can be replaced by ZSTD_CCtx_reset() and ZSTD_compressStream2().
	* It is redundant, but remains fully supported.
	* Advanced parameters and dictionary compression can only be used through the
	* new API.
	******************************************************************************/

	/*!
	* Equivalent to:
	*
	* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
	* ZSTD_CCtx_refCDict(zcs, NULL); // clear the dictionary (if any)
	* ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel);
	*/
	ZSTDLIB_API size_t ZSTD_initCStream(ZSTD_CStream* zcs, int compressionLevel);
	/*!
	* Alternative for ZSTD_compressStream2(zcs, output, input, ZSTD_e_continue).
	* NOTE: The return value is different. ZSTD_compressStream() returns a hint for
	* the next read size (if non-zero and not an error). ZSTD_compressStream2()
	* returns the minimum nb of bytes left to flush (if non-zero and not an error).
	*/
	ZSTDLIB_API size_t ZSTD_compressStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input);
	/! Equivalent to ZSTD_compressStream2(zcs, output, &emptyInput, ZSTD_e_flush). /
	ZSTDLIB_API size_t ZSTD_flushStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output);
	/! Equivalent to ZSTD_compressStream2(zcs, output, &emptyInput, ZSTD_e_end). /
	ZSTDLIB_API size_t ZSTD_endStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output);


	/-**************************************************************************
	* Streaming decompression - HowTo
	*
	* A ZSTD_DStream object is required to track streaming operations.
	* Use ZSTD_createDStream() and ZSTD_freeDStream() to create/release resources.
	* ZSTD_DStream objects can be re-used multiple times.
	*
	* Use ZSTD_initDStream() to start a new decompression operation.
	* @return : recommended first input size
	* Alternatively, use advanced API to set specific properties.
	*
	* Use ZSTD_decompressStream() repetitively to consume your input.
	* The function will update both `pos` fields.
	* If `input.pos < input.size`, some input has not been consumed.
	* It's up to the caller to present again remaining data.
	* The function tries to flush all data decoded immediately, respecting output buffer size.
	* If `output.pos < output.size`, decoder has flushed everything it could.
	* But if `output.pos == output.size`, there might be some data left within internal buffers.,
	* In which case, call ZSTD_decompressStream() again to flush whatever remains in the buffer.
	* Note : with no additional input provided, amount of data flushed is necessarily <= ZSTD_BLOCKSIZE_MAX.
	* @return : 0 when a frame is completely decoded and fully flushed,
	* or an error code, which can be tested using ZSTD_isError(),
	* or any other value > 0, which means there is still some decoding or flushing to do to complete current frame :
	* the return value is a suggested next input size (just a hint for better latency)
	* that will never request more than the remaining frame size.
	* *******************************************************************************/

	typedef ZSTD_DCtx ZSTD_DStream; /*< DCtx and DStream are now effectively same object (>= v1.3.0) /
	/* For compatibility with versions <= v1.2.0, prefer differentiating them. */
	/===== ZSTD_DStream management functions =====/
	ZSTDLIB_API ZSTD_DStream* ZSTD_createDStream(void);
	ZSTDLIB_API size_t ZSTD_freeDStream(ZSTD_DStream* zds);

	/===== Streaming decompression functions =====/

	/* This function is redundant with the advanced API and equivalent to:
	*
	* ZSTD_DCtx_reset(zds, ZSTD_reset_session_only);
	* ZSTD_DCtx_refDDict(zds, NULL);
	*/
	ZSTDLIB_API size_t ZSTD_initDStream(ZSTD_DStream* zds);

	ZSTDLIB_API size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inBuffer* input);

	ZSTDLIB_API size_t ZSTD_DStreamInSize(void); /!< recommended size for input buffer /
	ZSTDLIB_API size_t ZSTD_DStreamOutSize(void); /!< recommended size for output buffer. Guarantee to successfully flush at least one complete block in all circumstances. /


	/**************************
	* Simple dictionary API
	***************************/
	/*! ZSTD_compress_usingDict() :
	* Compression at an explicit compression level using a Dictionary.
	* A dictionary can be any arbitrary data segment (also called a prefix),
	* or a buffer with specified information (see dictBuilder/zdict.h).
	* Note : This function loads the dictionary, resulting in significant startup delay.
	* It's intended for a dictionary used only once.
	* Note 2 : When `dict == NULL \|\| dictSize < 8` no dictionary is used. */
	ZSTDLIB_API size_t ZSTD_compress_usingDict(ZSTD_CCtx* ctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const void* dict,size_t dictSize,
	int compressionLevel);

	/*! ZSTD_decompress_usingDict() :
	* Decompression using a known Dictionary.
	* Dictionary must be identical to the one used during compression.
	* Note : This function loads the dictionary, resulting in significant startup delay.
	* It's intended for a dictionary used only once.
	* Note : When `dict == NULL \|\| dictSize < 8` no dictionary is used. */
	ZSTDLIB_API size_t ZSTD_decompress_usingDict(ZSTD_DCtx* dctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const void* dict,size_t dictSize);


	/***********************************
	* Bulk processing dictionary API
	**********************************/
	typedef struct ZSTD_CDict_s ZSTD_CDict;

	/*! ZSTD_createCDict() :
	* When compressing multiple messages or blocks using the same dictionary,
	* it's recommended to digest the dictionary only once, since it's a costly operation.
	* ZSTD_createCDict() will create a state from digesting a dictionary.
	* The resulting state can be used for future compression operations with very limited startup cost.
	* ZSTD_CDict can be created once and shared by multiple threads concurrently, since its usage is read-only.
	* @dictBuffer can be released after ZSTD_CDict creation, because its content is copied within CDict.
	* Note 1 : Consider experimental function `ZSTD_createCDict_byReference()` if you prefer to not duplicate @dictBuffer content.
	* Note 2 : A ZSTD_CDict can be created from an empty @dictBuffer,
	* in which case the only thing that it transports is the @compressionLevel.
	* This can be useful in a pipeline featuring ZSTD_compress_usingCDict() exclusively,
	* expecting a ZSTD_CDict parameter with any data, including those without a known dictionary. */
	ZSTDLIB_API ZSTD_CDict* ZSTD_createCDict(const void* dictBuffer, size_t dictSize,
	int compressionLevel);

	/*! ZSTD_freeCDict() :
	* Function frees memory allocated by ZSTD_createCDict(). */
	ZSTDLIB_API size_t ZSTD_freeCDict(ZSTD_CDict* CDict);

	/*! ZSTD_compress_usingCDict() :
	* Compression using a digested Dictionary.
	* Recommended when same dictionary is used multiple times.
	* Note : compression level is _decided at dictionary creation time_,
	* and frame parameters are hardcoded (dictID=yes, contentSize=yes, checksum=no) */
	ZSTDLIB_API size_t ZSTD_compress_usingCDict(ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const ZSTD_CDict* cdict);


	typedef struct ZSTD_DDict_s ZSTD_DDict;

	/*! ZSTD_createDDict() :
	* Create a digested dictionary, ready to start decompression operation without startup delay.
	* dictBuffer can be released after DDict creation, as its content is copied inside DDict. */
	ZSTDLIB_API ZSTD_DDict* ZSTD_createDDict(const void* dictBuffer, size_t dictSize);

	/*! ZSTD_freeDDict() :
	* Function frees memory allocated with ZSTD_createDDict() */
	ZSTDLIB_API size_t ZSTD_freeDDict(ZSTD_DDict* ddict);

	/*! ZSTD_decompress_usingDDict() :
	* Decompression using a digested Dictionary.
	* Recommended when same dictionary is used multiple times. */
	ZSTDLIB_API size_t ZSTD_decompress_usingDDict(ZSTD_DCtx* dctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const ZSTD_DDict* ddict);


	/********************************
	* Dictionary helper functions
	*******************************/

	/*! ZSTD_getDictID_fromDict() :
	* Provides the dictID stored within dictionary.
	* if @return == 0, the dictionary is not conformant with Zstandard specification.
	* It can still be loaded, but as a content-only dictionary. */
	ZSTDLIB_API unsigned ZSTD_getDictID_fromDict(const void* dict, size_t dictSize);

	/*! ZSTD_getDictID_fromDDict() :
	* Provides the dictID of the dictionary loaded into `ddict`.
	* If @return == 0, the dictionary is not conformant to Zstandard specification, or empty.
	* Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
	ZSTDLIB_API unsigned ZSTD_getDictID_fromDDict(const ZSTD_DDict* ddict);

	/*! ZSTD_getDictID_fromFrame() :
	* Provides the dictID required to decompressed the frame stored within `src`.
	* If @return == 0, the dictID could not be decoded.
	* This could for one of the following reasons :
	* - The frame does not require a dictionary to be decoded (most common case).
	* - The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a hidden information.
	* Note : this use case also happens when using a non-conformant dictionary.
	* - `srcSize` is too small, and as a result, the frame header could not be decoded (only possible if `srcSize < ZSTD_FRAMEHEADERSIZE_MAX`).
	* - This is not a Zstandard frame.
	* When identifying the exact failure cause, it's possible to use ZSTD_getFrameHeader(), which will provide a more precise error code. */
	ZSTDLIB_API unsigned ZSTD_getDictID_fromFrame(const void* src, size_t srcSize);


	/*******************************************************************************
	* Advanced dictionary and prefix API
	*
	* This API allows dictionaries to be used with ZSTD_compress2(),
	* ZSTD_compressStream2(), and ZSTD_decompress(). Dictionaries are sticky, and
	* only reset with the context is reset with ZSTD_reset_parameters or
	* ZSTD_reset_session_and_parameters. Prefixes are single-use.
	******************************************************************************/


	/*! ZSTD_CCtx_loadDictionary() :
	* Create an internal CDict from `dict` buffer.
	* Decompression will have to use same dictionary.
	* @result : 0, or an error code (which can be tested with ZSTD_isError()).
	* Special: Loading a NULL (or 0-size) dictionary invalidates previous dictionary,
	* meaning "return to no-dictionary mode".
	* Note 1 : Dictionary is sticky, it will be used for all future compressed frames.
	* To return to "no-dictionary" situation, load a NULL dictionary (or reset parameters).
	* Note 2 : Loading a dictionary involves building tables.
	* It's also a CPU consuming operation, with non-negligible impact on latency.
	* Tables are dependent on compression parameters, and for this reason,
	* compression parameters can no longer be changed after loading a dictionary.
	* Note 3 :`dict` content will be copied internally.
	* Use experimental ZSTD_CCtx_loadDictionary_byReference() to reference content instead.
	* In such a case, dictionary buffer must outlive its users.
	* Note 4 : Use ZSTD_CCtx_loadDictionary_advanced()
	* to precisely select how dictionary content must be interpreted. */
	ZSTDLIB_API size_t ZSTD_CCtx_loadDictionary(ZSTD_CCtx* cctx, const void* dict, size_t dictSize);

	/*! ZSTD_CCtx_refCDict() :
	* Reference a prepared dictionary, to be used for all next compressed frames.
	* Note that compression parameters are enforced from within CDict,
	* and supersede any compression parameter previously set within CCtx.
	* The parameters ignored are labled as "superseded-by-cdict" in the ZSTD_cParameter enum docs.
	* The ignored parameters will be used again if the CCtx is returned to no-dictionary mode.
	* The dictionary will remain valid for future compressed frames using same CCtx.
	* @result : 0, or an error code (which can be tested with ZSTD_isError()).
	* Special : Referencing a NULL CDict means "return to no-dictionary mode".
	* Note 1 : Currently, only one dictionary can be managed.
	* Referencing a new dictionary effectively "discards" any previous one.
	* Note 2 : CDict is just referenced, its lifetime must outlive its usage within CCtx. */
	ZSTDLIB_API size_t ZSTD_CCtx_refCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict);

	/*! ZSTD_CCtx_refPrefix() :
	* Reference a prefix (single-usage dictionary) for next compressed frame.
	* A prefix is only used once. Tables are discarded at end of frame (ZSTD_e_end).
	* Decompression will need same prefix to properly regenerate data.
	* Compressing with a prefix is similar in outcome as performing a diff and compressing it,
	* but performs much faster, especially during decompression (compression speed is tunable with compression level).
	* @result : 0, or an error code (which can be tested with ZSTD_isError()).
	* Special: Adding any prefix (including NULL) invalidates any previous prefix or dictionary
	* Note 1 : Prefix buffer is referenced. It must outlive compression.
	* Its content must remain unmodified during compression.
	* Note 2 : If the intention is to diff some large src data blob with some prior version of itself,
	* ensure that the window size is large enough to contain the entire source.
	* See ZSTD_c_windowLog.
	* Note 3 : Referencing a prefix involves building tables, which are dependent on compression parameters.
	* It's a CPU consuming operation, with non-negligible impact on latency.
	* If there is a need to use the same prefix multiple times, consider loadDictionary instead.
	* Note 4 : By default, the prefix is interpreted as raw content (ZSTD_dct_rawContent).
	* Use experimental ZSTD_CCtx_refPrefix_advanced() to alter dictionary interpretation. */
	ZSTDLIB_API size_t ZSTD_CCtx_refPrefix(ZSTD_CCtx* cctx,
	const void* prefix, size_t prefixSize);

	/*! ZSTD_DCtx_loadDictionary() :
	* Create an internal DDict from dict buffer,
	* to be used to decompress next frames.
	* The dictionary remains valid for all future frames, until explicitly invalidated.
	* @result : 0, or an error code (which can be tested with ZSTD_isError()).
	* Special : Adding a NULL (or 0-size) dictionary invalidates any previous dictionary,
	* meaning "return to no-dictionary mode".
	* Note 1 : Loading a dictionary involves building tables,
	* which has a non-negligible impact on CPU usage and latency.
	* It's recommended to "load once, use many times", to amortize the cost
	* Note 2 :`dict` content will be copied internally, so `dict` can be released after loading.
	* Use ZSTD_DCtx_loadDictionary_byReference() to reference dictionary content instead.
	* Note 3 : Use ZSTD_DCtx_loadDictionary_advanced() to take control of
	* how dictionary content is loaded and interpreted.
	*/
	ZSTDLIB_API size_t ZSTD_DCtx_loadDictionary(ZSTD_DCtx* dctx, const void* dict, size_t dictSize);

	/*! ZSTD_DCtx_refDDict() :
	* Reference a prepared dictionary, to be used to decompress next frames.
	* The dictionary remains active for decompression of future frames using same DCtx.
	* @result : 0, or an error code (which can be tested with ZSTD_isError()).
	* Note 1 : Currently, only one dictionary can be managed.
	* Referencing a new dictionary effectively "discards" any previous one.
	* Special: referencing a NULL DDict means "return to no-dictionary mode".
	* Note 2 : DDict is just referenced, its lifetime must outlive its usage from DCtx.
	*/
	ZSTDLIB_API size_t ZSTD_DCtx_refDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict);

	/*! ZSTD_DCtx_refPrefix() :
	* Reference a prefix (single-usage dictionary) to decompress next frame.
	* This is the reverse operation of ZSTD_CCtx_refPrefix(),
	* and must use the same prefix as the one used during compression.
	* Prefix is only used once. Reference is discarded at end of frame.
	* End of frame is reached when ZSTD_decompressStream() returns 0.
	* @result : 0, or an error code (which can be tested with ZSTD_isError()).
	* Note 1 : Adding any prefix (including NULL) invalidates any previously set prefix or dictionary
	* Note 2 : Prefix buffer is referenced. It must outlive decompression.
	* Prefix buffer must remain unmodified up to the end of frame,
	* reached when ZSTD_decompressStream() returns 0.
	* Note 3 : By default, the prefix is treated as raw content (ZSTD_dct_rawContent).
	* Use ZSTD_CCtx_refPrefix_advanced() to alter dictMode (Experimental section)
	* Note 4 : Referencing a raw content prefix has almost no cpu nor memory cost.
	* A full dictionary is more costly, as it requires building tables.
	*/
	ZSTDLIB_API size_t ZSTD_DCtx_refPrefix(ZSTD_DCtx* dctx,
	const void* prefix, size_t prefixSize);

	/* === Memory management === */

	/! ZSTD_sizeof_() :
	* These functions give the _current_ memory usage of selected object.
	* Note that object memory usage can evolve (increase or decrease) over time. */
	ZSTDLIB_API size_t ZSTD_sizeof_CCtx(const ZSTD_CCtx* cctx);
	ZSTDLIB_API size_t ZSTD_sizeof_DCtx(const ZSTD_DCtx* dctx);
	ZSTDLIB_API size_t ZSTD_sizeof_CStream(const ZSTD_CStream* zcs);
	ZSTDLIB_API size_t ZSTD_sizeof_DStream(const ZSTD_DStream* zds);
	ZSTDLIB_API size_t ZSTD_sizeof_CDict(const ZSTD_CDict* cdict);
	ZSTDLIB_API size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);

	#endif /* ZSTD_H_235446 */


	/* **************************************************************************************
	* ADVANCED AND EXPERIMENTAL FUNCTIONS
	****************************************************************************************
	* The definitions in the following section are considered experimental.
	* They are provided for advanced scenarios.
	* They should never be used with a dynamic library, as prototypes may change in the future.
	* Use them only in association with static linking.
	* ***************************************************************************************/

	#if defined(ZSTD_STATIC_LINKING_ONLY) && !defined(ZSTD_H_ZSTD_STATIC_LINKING_ONLY)
	#define ZSTD_H_ZSTD_STATIC_LINKING_ONLY

	/****************************************************************************************
	* experimental API (static linking only)
	****************************************************************************************
	* The following symbols and constants
	* are not planned to join "stable API" status in the near future.
	* They can still change in future versions.
	* Some of them are planned to remain in the static_only section indefinitely.
	* Some of them might be removed in the future (especially when redundant with existing stable functions)
	* ***************************************************************************************/

	#define ZSTD_FRAMEHEADERSIZE_PREFIX(format) ((format) == ZSTD_f_zstd1 ? 5 : 1) /* minimum input size required to query frame header size */
	#define ZSTD_FRAMEHEADERSIZE_MIN(format) ((format) == ZSTD_f_zstd1 ? 6 : 2)
	#define ZSTD_FRAMEHEADERSIZE_MAX 18 /* can be useful for static allocation */
	#define ZSTD_SKIPPABLEHEADERSIZE 8

	/* compression parameter bounds */
	#define ZSTD_WINDOWLOG_MAX_32 30
	#define ZSTD_WINDOWLOG_MAX_64 31
	#define ZSTD_WINDOWLOG_MAX ((int)(sizeof(size_t) == 4 ? ZSTD_WINDOWLOG_MAX_32 : ZSTD_WINDOWLOG_MAX_64))
	#define ZSTD_WINDOWLOG_MIN 10
	#define ZSTD_HASHLOG_MAX ((ZSTD_WINDOWLOG_MAX < 30) ? ZSTD_WINDOWLOG_MAX : 30)
	#define ZSTD_HASHLOG_MIN 6
	#define ZSTD_CHAINLOG_MAX_32 29
	#define ZSTD_CHAINLOG_MAX_64 30
	#define ZSTD_CHAINLOG_MAX ((int)(sizeof(size_t) == 4 ? ZSTD_CHAINLOG_MAX_32 : ZSTD_CHAINLOG_MAX_64))
	#define ZSTD_CHAINLOG_MIN ZSTD_HASHLOG_MIN
	#define ZSTD_SEARCHLOG_MAX (ZSTD_WINDOWLOG_MAX-1)
	#define ZSTD_SEARCHLOG_MIN 1
	#define ZSTD_MINMATCH_MAX 7 /* only for ZSTD_fast, other strategies are limited to 6 */
	#define ZSTD_MINMATCH_MIN 3 /* only for ZSTD_btopt+, faster strategies are limited to 4 */
	#define ZSTD_TARGETLENGTH_MAX ZSTD_BLOCKSIZE_MAX
	#define ZSTD_TARGETLENGTH_MIN 0 /* note : comparing this constant to an unsigned results in a tautological test */
	#define ZSTD_STRATEGY_MIN ZSTD_fast
	#define ZSTD_STRATEGY_MAX ZSTD_btultra2


	#define ZSTD_OVERLAPLOG_MIN 0
	#define ZSTD_OVERLAPLOG_MAX 9

	#define ZSTD_WINDOWLOG_LIMIT_DEFAULT 27 /* by default, the streaming decoder will refuse any frame
	* requiring larger than (1<<ZSTD_WINDOWLOG_LIMIT_DEFAULT) window size,
	* to preserve host's memory from unreasonable requirements.
	* This limit can be overridden using ZSTD_DCtx_setParameter(,ZSTD_d_windowLogMax,).
	* The limit does not apply for one-pass decoders (such as ZSTD_decompress()), since no additional memory is allocated */


	/* LDM parameter bounds */
	#define ZSTD_LDM_HASHLOG_MIN ZSTD_HASHLOG_MIN
	#define ZSTD_LDM_HASHLOG_MAX ZSTD_HASHLOG_MAX
	#define ZSTD_LDM_MINMATCH_MIN 4
	#define ZSTD_LDM_MINMATCH_MAX 4096
	#define ZSTD_LDM_BUCKETSIZELOG_MIN 1
	#define ZSTD_LDM_BUCKETSIZELOG_MAX 8
	#define ZSTD_LDM_HASHRATELOG_MIN 0
	#define ZSTD_LDM_HASHRATELOG_MAX (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN)

	/* Advanced parameter bounds */
	#define ZSTD_TARGETCBLOCKSIZE_MIN 64
	#define ZSTD_TARGETCBLOCKSIZE_MAX ZSTD_BLOCKSIZE_MAX
	#define ZSTD_SRCSIZEHINT_MIN 0
	#define ZSTD_SRCSIZEHINT_MAX INT_MAX

	/* internal */
	#define ZSTD_HASHLOG3_MAX 17


	/* --- Advanced types --- */

	typedef struct ZSTD_CCtx_params_s ZSTD_CCtx_params;

	typedef struct {
	unsigned int matchPos; /* Match pos in dst */
	/* If seqDef.offset > 3, then this is seqDef.offset - 3
	* If seqDef.offset < 3, then this is the corresponding repeat offset
	* But if seqDef.offset < 3 and litLength == 0, this is the
	* repeat offset before the corresponding repeat offset
	* And if seqDef.offset == 3 and litLength == 0, this is the
	* most recent repeat offset - 1
	*/
	unsigned int offset;
	unsigned int litLength; /* Literal length */
	unsigned int matchLength; /* Match length */
	/* 0 when seq not rep and seqDef.offset otherwise
	* when litLength == 0 this will be <= 4, otherwise <= 3 like normal
	*/
	unsigned int rep;
	} ZSTD_Sequence;

	typedef struct {
	unsigned windowLog; /*< largest match distance : larger == more compression, more memory needed during decompression /
	unsigned chainLog; /*< fully searched segment : larger == more compression, slower, more memory (useless for fast) /
	unsigned hashLog; /*< dispatch table : larger == faster, more memory /
	unsigned searchLog; /*< nb of searches : larger == more compression, slower /
	unsigned minMatch; /*< match length searched : larger == faster decompression, sometimes less compression /
	unsigned targetLength; /*< acceptable match size for optimal parser (only) : larger == more compression, slower /
	ZSTD_strategy strategy; /*< see ZSTD_strategy definition above /
	} ZSTD_compressionParameters;

	typedef struct {
	int contentSizeFlag; /*< 1: content size will be in frame header (when known) /
	int checksumFlag; /*< 1: generate a 32-bits checksum using XXH64 algorithm at end of frame, for error detection /
	int noDictIDFlag; /*< 1: no dictID will be saved into frame header (dictID is only useful for dictionary compression) /
	} ZSTD_frameParameters;

	typedef struct {
	ZSTD_compressionParameters cParams;
	ZSTD_frameParameters fParams;
	} ZSTD_parameters;

	typedef enum {
	ZSTD_dct_auto = 0, /* dictionary is "full" when starting with ZSTD_MAGIC_DICTIONARY, otherwise it is "rawContent" */
	ZSTD_dct_rawContent = 1, /* ensures dictionary is always loaded as rawContent, even if it starts with ZSTD_MAGIC_DICTIONARY */
	ZSTD_dct_fullDict = 2 /* refuses to load a dictionary if it does not respect Zstandard's specification, starting with ZSTD_MAGIC_DICTIONARY */
	} ZSTD_dictContentType_e;

	typedef enum {
	ZSTD_dlm_byCopy = 0, /*< Copy dictionary content internally /
	ZSTD_dlm_byRef = 1 /*< Reference dictionary content -- the dictionary buffer must outlive its users. /
	} ZSTD_dictLoadMethod_e;

	typedef enum {
	ZSTD_f_zstd1 = 0, /* zstd frame format, specified in zstd_compression_format.md (default) */
	ZSTD_f_zstd1_magicless = 1 /* Variant of zstd frame format, without initial 4-bytes magic number.
	* Useful to save 4 bytes per generated frame.
	* Decoder cannot recognise automatically this format, requiring this instruction. */
	} ZSTD_format_e;

	typedef enum {
	/* Note: this enum and the behavior it controls are effectively internal
	* implementation details of the compressor. They are expected to continue
	* to evolve and should be considered only in the context of extremely
	* advanced performance tuning.
	*
	* Zstd currently supports the use of a CDict in three ways:
	*
	* - The contents of the CDict can be copied into the working context. This
	* means that the compression can search both the dictionary and input
	* while operating on a single set of internal tables. This makes
	* the compression faster per-byte of input. However, the initial copy of
	* the CDict's tables incurs a fixed cost at the beginning of the
	* compression. For small compressions (< 8 KB), that copy can dominate
	* the cost of the compression.
	*
	* - The CDict's tables can be used in-place. In this model, compression is
	* slower per input byte, because the compressor has to search two sets of
	* tables. However, this model incurs no start-up cost (as long as the
	* working context's tables can be reused). For small inputs, this can be
	* faster than copying the CDict's tables.
	*
	* - The CDict's tables are not used at all, and instead we use the working
	* context alone to reload the dictionary and use params based on the source
	* size. See ZSTD_compress_insertDictionary() and ZSTD_compress_usingDict().
	* This method is effective when the dictionary sizes are very small relative
	* to the input size, and the input size is fairly large to begin with.
	*
	* Zstd has a simple internal heuristic that selects which strategy to use
	* at the beginning of a compression. However, if experimentation shows that
	* Zstd is making poor choices, it is possible to override that choice with
	* this enum.
	*/
	ZSTD_dictDefaultAttach = 0, /* Use the default heuristic. */
	ZSTD_dictForceAttach = 1, /* Never copy the dictionary. */
	ZSTD_dictForceCopy = 2, /* Always copy the dictionary. */
	ZSTD_dictForceLoad = 3 /* Always reload the dictionary */
	} ZSTD_dictAttachPref_e;

	typedef enum {
	ZSTD_lcm_auto = 0, /**< Automatically determine the compression mode based on the compression level.
	* Negative compression levels will be uncompressed, and positive compression
	* levels will be compressed. */
	ZSTD_lcm_huffman = 1, /**< Always attempt Huffman compression. Uncompressed literals will still be
	* emitted if Huffman compression is not profitable. */
	ZSTD_lcm_uncompressed = 2 /*< Always emit uncompressed literals. /
	} ZSTD_literalCompressionMode_e;


	/***************************************
	* Frame size functions
	***************************************/

	/*! ZSTD_findDecompressedSize() :
	* `src` should point to the start of a series of ZSTD encoded and/or skippable frames
	* `srcSize` must be the _exact_ size of this series
	* (i.e. there should be a frame boundary at `src + srcSize`)
	* @return : - decompressed size of all data in all successive frames
	* - if the decompressed size cannot be determined: ZSTD_CONTENTSIZE_UNKNOWN
	* - if an error occurred: ZSTD_CONTENTSIZE_ERROR
	*
	* note 1 : decompressed size is an optional field, that may not be present, especially in streaming mode.
	* When `return==ZSTD_CONTENTSIZE_UNKNOWN`, data to decompress could be any size.
	* In which case, it's necessary to use streaming mode to decompress data.
	* note 2 : decompressed size is always present when compression is done with ZSTD_compress()
	* note 3 : decompressed size can be very large (64-bits value),
	* potentially larger than what local system can handle as a single memory segment.
	* In which case, it's necessary to use streaming mode to decompress data.
	* note 4 : If source is untrusted, decompressed size could be wrong or intentionally modified.
	* Always ensure result fits within application's authorized limits.
	* Each application can set its own limits.
	* note 5 : ZSTD_findDecompressedSize handles multiple frames, and so it must traverse the input to
	* read each contained frame header. This is fast as most of the data is skipped,
	* however it does mean that all frame data must be present and valid. */
	ZSTDLIB_API unsigned long long ZSTD_findDecompressedSize(const void* src, size_t srcSize);

	/*! ZSTD_decompressBound() :
	* `src` should point to the start of a series of ZSTD encoded and/or skippable frames
	* `srcSize` must be the _exact_ size of this series
	* (i.e. there should be a frame boundary at `src + srcSize`)
	* @return : - upper-bound for the decompressed size of all data in all successive frames
	* - if an error occured: ZSTD_CONTENTSIZE_ERROR
	*
	* note 1 : an error can occur if `src` contains an invalid or incorrectly formatted frame.
	* note 2 : the upper-bound is exact when the decompressed size field is available in every ZSTD encoded frame of `src`.
	* in this case, `ZSTD_findDecompressedSize` and `ZSTD_decompressBound` return the same value.
	* note 3 : when the decompressed size field isn't available, the upper-bound for that frame is calculated by:
	* upper-bound = # blocks * min(128 KB, Window_Size)
	*/
	ZSTDLIB_API unsigned long long ZSTD_decompressBound(const void* src, size_t srcSize);

	/*! ZSTD_frameHeaderSize() :
	* srcSize must be >= ZSTD_FRAMEHEADERSIZE_PREFIX.
	* @return : size of the Frame Header,
	* or an error code (if srcSize is too small) */
	ZSTDLIB_API size_t ZSTD_frameHeaderSize(const void* src, size_t srcSize);

	/*! ZSTD_getSequences() :
	* Extract sequences from the sequence store
	* zc can be used to insert custom compression params.
	* This function invokes ZSTD_compress2
	* @return : number of sequences extracted
	*/
	ZSTDLIB_API size_t ZSTD_getSequences(ZSTD_CCtx* zc, ZSTD_Sequence* outSeqs,
	size_t outSeqsSize, const void* src, size_t srcSize);


	/***************************************
	* Memory management
	***************************************/

	/! ZSTD_estimate() :
	* These functions make it possible to estimate memory usage
	* of a future {D,C}Ctx, before its creation.
	*
	* ZSTD_estimateCCtxSize() will provide a memory budget large enough
	* for any compression level up to selected one.
	* Note : Unlike ZSTD_estimateCStreamSize*(), this estimate
	* does not include space for a window buffer.
	* Therefore, the estimation is only guaranteed for single-shot compressions, not streaming.
	* The estimate will assume the input may be arbitrarily large,
	* which is the worst case.
	*
	* When srcSize can be bound by a known and rather "small" value,
	* this fact can be used to provide a tighter estimation
	* because the CCtx compression context will need less memory.
	* This tighter estimation can be provided by more advanced functions
	* ZSTD_estimateCCtxSize_usingCParams(), which can be used in tandem with ZSTD_getCParams(),
	* and ZSTD_estimateCCtxSize_usingCCtxParams(), which can be used in tandem with ZSTD_CCtxParams_setParameter().
	* Both can be used to estimate memory using custom compression parameters and arbitrary srcSize limits.
	*
	* Note 2 : only single-threaded compression is supported.
	* ZSTD_estimateCCtxSize_usingCCtxParams() will return an error code if ZSTD_c_nbWorkers is >= 1.
	*/
	ZSTDLIB_API size_t ZSTD_estimateCCtxSize(int compressionLevel);
	ZSTDLIB_API size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams);
	ZSTDLIB_API size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params);
	ZSTDLIB_API size_t ZSTD_estimateDCtxSize(void);

	/*! ZSTD_estimateCStreamSize() :
	* ZSTD_estimateCStreamSize() will provide a budget large enough for any compression level up to selected one.
	* It will also consider src size to be arbitrarily "large", which is worst case.
	* If srcSize is known to always be small, ZSTD_estimateCStreamSize_usingCParams() can provide a tighter estimation.
	* ZSTD_estimateCStreamSize_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel.
	* ZSTD_estimateCStreamSize_usingCCtxParams() can be used in tandem with ZSTD_CCtxParams_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_c_nbWorkers is >= 1.
	* Note : CStream size estimation is only correct for single-threaded compression.
	* ZSTD_DStream memory budget depends on window Size.
	* This information can be passed manually, using ZSTD_estimateDStreamSize,
	* or deducted from a valid frame Header, using ZSTD_estimateDStreamSize_fromFrame();
	* Note : if streaming is init with function ZSTD_init?Stream_usingDict(),
	* an internal ?Dict will be created, which additional size is not estimated here.
	* In this case, get total size by adding ZSTD_estimate?DictSize */
	ZSTDLIB_API size_t ZSTD_estimateCStreamSize(int compressionLevel);
	ZSTDLIB_API size_t ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams);
	ZSTDLIB_API size_t ZSTD_estimateCStreamSize_usingCCtxParams(const ZSTD_CCtx_params* params);
	ZSTDLIB_API size_t ZSTD_estimateDStreamSize(size_t windowSize);
	ZSTDLIB_API size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize);

	/*! ZSTD_estimate?DictSize() :
	* ZSTD_estimateCDictSize() will bet that src size is relatively "small", and content is copied, like ZSTD_createCDict().
	* ZSTD_estimateCDictSize_advanced() makes it possible to control compression parameters precisely, like ZSTD_createCDict_advanced().
	* Note : dictionaries created by reference (`ZSTD_dlm_byRef`) are logically smaller.
	*/
	ZSTDLIB_API size_t ZSTD_estimateCDictSize(size_t dictSize, int compressionLevel);
	ZSTDLIB_API size_t ZSTD_estimateCDictSize_advanced(size_t dictSize, ZSTD_compressionParameters cParams, ZSTD_dictLoadMethod_e dictLoadMethod);
	ZSTDLIB_API size_t ZSTD_estimateDDictSize(size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod);

	/! ZSTD_initStatic() :
	* Initialize an object using a pre-allocated fixed-size buffer.
	* workspace: The memory area to emplace the object into.
	* Provided pointer must be 8-bytes aligned.
	* Buffer must outlive object.
	* workspaceSize: Use ZSTD_estimate*Size() to determine
	* how large workspace must be to support target scenario.
	* @return : pointer to object (same address as workspace, just different type),
	* or NULL if error (size too small, incorrect alignment, etc.)
	* Note : zstd will never resize nor malloc() when using a static buffer.
	* If the object requires more memory than available,
	* zstd will just error out (typically ZSTD_error_memory_allocation).
	* Note 2 : there is no corresponding "free" function.
	* Since workspace is allocated externally, it must be freed externally too.
	* Note 3 : cParams : use ZSTD_getCParams() to convert a compression level
	* into its associated cParams.
	* Limitation 1 : currently not compatible with internal dictionary creation, triggered by
	* ZSTD_CCtx_loadDictionary(), ZSTD_initCStream_usingDict() or ZSTD_initDStream_usingDict().
	* Limitation 2 : static cctx currently not compatible with multi-threading.
	* Limitation 3 : static dctx is incompatible with legacy support.
	*/
	ZSTDLIB_API ZSTD_CCtx* ZSTD_initStaticCCtx(void* workspace, size_t workspaceSize);
	ZSTDLIB_API ZSTD_CStream* ZSTD_initStaticCStream(void* workspace, size_t workspaceSize); /*< same as ZSTD_initStaticCCtx() /

	ZSTDLIB_API ZSTD_DCtx* ZSTD_initStaticDCtx(void* workspace, size_t workspaceSize);
	ZSTDLIB_API ZSTD_DStream* ZSTD_initStaticDStream(void* workspace, size_t workspaceSize); /*< same as ZSTD_initStaticDCtx() /

	ZSTDLIB_API const ZSTD_CDict* ZSTD_initStaticCDict(
	void* workspace, size_t workspaceSize,
	const void* dict, size_t dictSize,
	ZSTD_dictLoadMethod_e dictLoadMethod,
	ZSTD_dictContentType_e dictContentType,
	ZSTD_compressionParameters cParams);

	ZSTDLIB_API const ZSTD_DDict* ZSTD_initStaticDDict(
	void* workspace, size_t workspaceSize,
	const void* dict, size_t dictSize,
	ZSTD_dictLoadMethod_e dictLoadMethod,
	ZSTD_dictContentType_e dictContentType);


	/*! Custom memory allocation :
	* These prototypes make it possible to pass your own allocation/free functions.
	* ZSTD_customMem is provided at creation time, using ZSTD_create*_advanced() variants listed below.
	* All allocation/free operations will be completed using these custom variants instead of regular <stdlib.h> ones.
	*/
	typedef void* (ZSTD_allocFunction) (void opaque, size_t size);
	typedef void (ZSTD_freeFunction) (void opaque, void* address);
	typedef struct { ZSTD_allocFunction customAlloc; ZSTD_freeFunction customFree; void* opaque; } ZSTD_customMem;
	static ZSTD_customMem const ZSTD_defaultCMem = { NULL, NULL, NULL }; /*< this constant defers to stdlib's functions /

	ZSTDLIB_API ZSTD_CCtx* ZSTD_createCCtx_advanced(ZSTD_customMem customMem);
	ZSTDLIB_API ZSTD_CStream* ZSTD_createCStream_advanced(ZSTD_customMem customMem);
	ZSTDLIB_API ZSTD_DCtx* ZSTD_createDCtx_advanced(ZSTD_customMem customMem);
	ZSTDLIB_API ZSTD_DStream* ZSTD_createDStream_advanced(ZSTD_customMem customMem);

	ZSTDLIB_API ZSTD_CDict* ZSTD_createCDict_advanced(const void* dict, size_t dictSize,
	ZSTD_dictLoadMethod_e dictLoadMethod,
	ZSTD_dictContentType_e dictContentType,
	ZSTD_compressionParameters cParams,
	ZSTD_customMem customMem);

	ZSTDLIB_API ZSTD_DDict* ZSTD_createDDict_advanced(const void* dict, size_t dictSize,
	ZSTD_dictLoadMethod_e dictLoadMethod,
	ZSTD_dictContentType_e dictContentType,
	ZSTD_customMem customMem);



	/***************************************
	* Advanced compression functions
	***************************************/

	/*! ZSTD_createCDict_byReference() :
	* Create a digested dictionary for compression
	* Dictionary content is just referenced, not duplicated.
	* As a consequence, `dictBuffer` must outlive CDict,
	* and its content must remain unmodified throughout the lifetime of CDict.
	* note: equivalent to ZSTD_createCDict_advanced(), with dictLoadMethod==ZSTD_dlm_byRef */
	ZSTDLIB_API ZSTD_CDict* ZSTD_createCDict_byReference(const void* dictBuffer, size_t dictSize, int compressionLevel);

	/*! ZSTD_getCParams() :
	* @return ZSTD_compressionParameters structure for a selected compression level and estimated srcSize.
	* `estimatedSrcSize` value is optional, select 0 if not known */
	ZSTDLIB_API ZSTD_compressionParameters ZSTD_getCParams(int compressionLevel, unsigned long long estimatedSrcSize, size_t dictSize);

	/*! ZSTD_getParams() :
	* same as ZSTD_getCParams(), but @return a full `ZSTD_parameters` object instead of sub-component `ZSTD_compressionParameters`.
	* All fields of `ZSTD_frameParameters` are set to default : contentSize=1, checksum=0, noDictID=0 */
	ZSTDLIB_API ZSTD_parameters ZSTD_getParams(int compressionLevel, unsigned long long estimatedSrcSize, size_t dictSize);

	/*! ZSTD_checkCParams() :
	* Ensure param values remain within authorized range.
	* @return 0 on success, or an error code (can be checked with ZSTD_isError()) */
	ZSTDLIB_API size_t ZSTD_checkCParams(ZSTD_compressionParameters params);

	/*! ZSTD_adjustCParams() :
	* optimize params for a given `srcSize` and `dictSize`.
	* `srcSize` can be unknown, in which case use ZSTD_CONTENTSIZE_UNKNOWN.
	* `dictSize` must be `0` when there is no dictionary.
	* cPar can be invalid : all parameters will be clamped within valid range in the @return struct.
	* This function never fails (wide contract) */
	ZSTDLIB_API ZSTD_compressionParameters ZSTD_adjustCParams(ZSTD_compressionParameters cPar, unsigned long long srcSize, size_t dictSize);

	/*! ZSTD_compress_advanced() :
	* Note : this function is now DEPRECATED.
	* It can be replaced by ZSTD_compress2(), in combination with ZSTD_CCtx_setParameter() and other parameter setters.
	* This prototype will be marked as deprecated and generate compilation warning on reaching v1.5.x */
	ZSTDLIB_API size_t ZSTD_compress_advanced(ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const void* dict,size_t dictSize,
	ZSTD_parameters params);

	/*! ZSTD_compress_usingCDict_advanced() :
	* Note : this function is now REDUNDANT.
	* It can be replaced by ZSTD_compress2(), in combination with ZSTD_CCtx_loadDictionary() and other parameter setters.
	* This prototype will be marked as deprecated and generate compilation warning in some future version */
	ZSTDLIB_API size_t ZSTD_compress_usingCDict_advanced(ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const ZSTD_CDict* cdict,
	ZSTD_frameParameters fParams);


	/*! ZSTD_CCtx_loadDictionary_byReference() :
	* Same as ZSTD_CCtx_loadDictionary(), but dictionary content is referenced, instead of being copied into CCtx.
	* It saves some memory, but also requires that `dict` outlives its usage within `cctx` */
	ZSTDLIB_API size_t ZSTD_CCtx_loadDictionary_byReference(ZSTD_CCtx* cctx, const void* dict, size_t dictSize);

	/*! ZSTD_CCtx_loadDictionary_advanced() :
	* Same as ZSTD_CCtx_loadDictionary(), but gives finer control over
	* how to load the dictionary (by copy ? by reference ?)
	* and how to interpret it (automatic ? force raw mode ? full mode only ?) */
	ZSTDLIB_API size_t ZSTD_CCtx_loadDictionary_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType);

	/*! ZSTD_CCtx_refPrefix_advanced() :
	* Same as ZSTD_CCtx_refPrefix(), but gives finer control over
	* how to interpret prefix content (automatic ? force raw mode (default) ? full mode only ?) */
	ZSTDLIB_API size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType);

	/* === experimental parameters === */
	/* these parameters can be used with ZSTD_setParameter()
	* they are not guaranteed to remain supported in the future */

	/* Enables rsyncable mode,
	* which makes compressed files more rsync friendly
	* by adding periodic synchronization points to the compressed data.
	* The target average block size is ZSTD_c_jobSize / 2.
	* It's possible to modify the job size to increase or decrease
	* the granularity of the synchronization point.
	* Once the jobSize is smaller than the window size,
	* it will result in compression ratio degradation.
	* NOTE 1: rsyncable mode only works when multithreading is enabled.
	* NOTE 2: rsyncable performs poorly in combination with long range mode,
	* since it will decrease the effectiveness of synchronization points,
	* though mileage may vary.
	* NOTE 3: Rsyncable mode limits maximum compression speed to ~400 MB/s.
	* If the selected compression level is already running significantly slower,
	* the overall speed won't be significantly impacted.
	*/
	#define ZSTD_c_rsyncable ZSTD_c_experimentalParam1

	/* Select a compression format.
	* The value must be of type ZSTD_format_e.
	* See ZSTD_format_e enum definition for details */
	#define ZSTD_c_format ZSTD_c_experimentalParam2

	/* Force back-reference distances to remain < windowSize,
	* even when referencing into Dictionary content (default:0) */
	#define ZSTD_c_forceMaxWindow ZSTD_c_experimentalParam3

	/* Controls whether the contents of a CDict
	* are used in place, or copied into the working context.
	* Accepts values from the ZSTD_dictAttachPref_e enum.
	* See the comments on that enum for an explanation of the feature. */
	#define ZSTD_c_forceAttachDict ZSTD_c_experimentalParam4

	/* Controls how the literals are compressed (default is auto).
	* The value must be of type ZSTD_literalCompressionMode_e.
	* See ZSTD_literalCompressionMode_t enum definition for details.
	*/
	#define ZSTD_c_literalCompressionMode ZSTD_c_experimentalParam5

	/* Tries to fit compressed block size to be around targetCBlockSize.
	* No target when targetCBlockSize == 0.
	* There is no guarantee on compressed block size (default:0) */
	#define ZSTD_c_targetCBlockSize ZSTD_c_experimentalParam6

	/* User's best guess of source size.
	* Hint is not valid when srcSizeHint == 0.
	* There is no guarantee that hint is close to actual source size,
	* but compression ratio may regress significantly if guess considerably underestimates */
	#define ZSTD_c_srcSizeHint ZSTD_c_experimentalParam7

	/*! ZSTD_CCtx_getParameter() :
	* Get the requested compression parameter value, selected by enum ZSTD_cParameter,
	* and store it into int* value.
	* @return : 0, or an error code (which can be tested with ZSTD_isError()).
	*/
	ZSTDLIB_API size_t ZSTD_CCtx_getParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int* value);


	/*! ZSTD_CCtx_params :
	* Quick howto :
	* - ZSTD_createCCtxParams() : Create a ZSTD_CCtx_params structure
	* - ZSTD_CCtxParams_setParameter() : Push parameters one by one into
	* an existing ZSTD_CCtx_params structure.
	* This is similar to
	* ZSTD_CCtx_setParameter().
	* - ZSTD_CCtx_setParametersUsingCCtxParams() : Apply parameters to
	* an existing CCtx.
	* These parameters will be applied to
	* all subsequent frames.
	* - ZSTD_compressStream2() : Do compression using the CCtx.
	* - ZSTD_freeCCtxParams() : Free the memory.
	*
	* This can be used with ZSTD_estimateCCtxSize_advanced_usingCCtxParams()
	* for static allocation of CCtx for single-threaded compression.
	*/
	ZSTDLIB_API ZSTD_CCtx_params* ZSTD_createCCtxParams(void);
	ZSTDLIB_API size_t ZSTD_freeCCtxParams(ZSTD_CCtx_params* params);

	/*! ZSTD_CCtxParams_reset() :
	* Reset params to default values.
	*/
	ZSTDLIB_API size_t ZSTD_CCtxParams_reset(ZSTD_CCtx_params* params);

	/*! ZSTD_CCtxParams_init() :
	* Initializes the compression parameters of cctxParams according to
	* compression level. All other parameters are reset to their default values.
	*/
	ZSTDLIB_API size_t ZSTD_CCtxParams_init(ZSTD_CCtx_params* cctxParams, int compressionLevel);

	/*! ZSTD_CCtxParams_init_advanced() :
	* Initializes the compression and frame parameters of cctxParams according to
	* params. All other parameters are reset to their default values.
	*/
	ZSTDLIB_API size_t ZSTD_CCtxParams_init_advanced(ZSTD_CCtx_params* cctxParams, ZSTD_parameters params);

	/*! ZSTD_CCtxParams_setParameter() :
	* Similar to ZSTD_CCtx_setParameter.
	* Set one compression parameter, selected by enum ZSTD_cParameter.
	* Parameters must be applied to a ZSTD_CCtx using ZSTD_CCtx_setParametersUsingCCtxParams().
	* @result : 0, or an error code (which can be tested with ZSTD_isError()).
	*/
	ZSTDLIB_API size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* params, ZSTD_cParameter param, int value);

	/*! ZSTD_CCtxParams_getParameter() :
	* Similar to ZSTD_CCtx_getParameter.
	* Get the requested value of one compression parameter, selected by enum ZSTD_cParameter.
	* @result : 0, or an error code (which can be tested with ZSTD_isError()).
	*/
	ZSTDLIB_API size_t ZSTD_CCtxParams_getParameter(ZSTD_CCtx_params* params, ZSTD_cParameter param, int* value);

	/*! ZSTD_CCtx_setParametersUsingCCtxParams() :
	* Apply a set of ZSTD_CCtx_params to the compression context.
	* This can be done even after compression is started,
	* if nbWorkers==0, this will have no impact until a new compression is started.
	* if nbWorkers>=1, new parameters will be picked up at next job,
	* with a few restrictions (windowLog, pledgedSrcSize, nbWorkers, jobSize, and overlapLog are not updated).
	*/
	ZSTDLIB_API size_t ZSTD_CCtx_setParametersUsingCCtxParams(
	ZSTD_CCtx* cctx, const ZSTD_CCtx_params* params);

	/*! ZSTD_compressStream2_simpleArgs() :
	* Same as ZSTD_compressStream2(),
	* but using only integral types as arguments.
	* This variant might be helpful for binders from dynamic languages
	* which have troubles handling structures containing memory pointers.
	*/
	ZSTDLIB_API size_t ZSTD_compressStream2_simpleArgs (
	ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity, size_t* dstPos,
	const void* src, size_t srcSize, size_t* srcPos,
	ZSTD_EndDirective endOp);


	/***************************************
	* Advanced decompression functions
	***************************************/

	/*! ZSTD_isFrame() :
	* Tells if the content of `buffer` starts with a valid Frame Identifier.
	* Note : Frame Identifier is 4 bytes. If `size < 4`, @return will always be 0.
	* Note 2 : Legacy Frame Identifiers are considered valid only if Legacy Support is enabled.
	* Note 3 : Skippable Frame Identifiers are considered valid. */
	ZSTDLIB_API unsigned ZSTD_isFrame(const void* buffer, size_t size);

	/*! ZSTD_createDDict_byReference() :
	* Create a digested dictionary, ready to start decompression operation without startup delay.
	* Dictionary content is referenced, and therefore stays in dictBuffer.
	* It is important that dictBuffer outlives DDict,
	* it must remain read accessible throughout the lifetime of DDict */
	ZSTDLIB_API ZSTD_DDict* ZSTD_createDDict_byReference(const void* dictBuffer, size_t dictSize);

	/*! ZSTD_DCtx_loadDictionary_byReference() :
	* Same as ZSTD_DCtx_loadDictionary(),
	* but references `dict` content instead of copying it into `dctx`.
	* This saves memory if `dict` remains around.,
	* However, it's imperative that `dict` remains accessible (and unmodified) while being used, so it must outlive decompression. */
	ZSTDLIB_API size_t ZSTD_DCtx_loadDictionary_byReference(ZSTD_DCtx* dctx, const void* dict, size_t dictSize);

	/*! ZSTD_DCtx_loadDictionary_advanced() :
	* Same as ZSTD_DCtx_loadDictionary(),
	* but gives direct control over
	* how to load the dictionary (by copy ? by reference ?)
	* and how to interpret it (automatic ? force raw mode ? full mode only ?). */
	ZSTDLIB_API size_t ZSTD_DCtx_loadDictionary_advanced(ZSTD_DCtx* dctx, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType);

	/*! ZSTD_DCtx_refPrefix_advanced() :
	* Same as ZSTD_DCtx_refPrefix(), but gives finer control over
	* how to interpret prefix content (automatic ? force raw mode (default) ? full mode only ?) */
	ZSTDLIB_API size_t ZSTD_DCtx_refPrefix_advanced(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType);

	/*! ZSTD_DCtx_setMaxWindowSize() :
	* Refuses allocating internal buffers for frames requiring a window size larger than provided limit.
	* This protects a decoder context from reserving too much memory for itself (potential attack scenario).
	* This parameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
	* By default, a decompression context accepts all window sizes <= (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT)
	* @return : 0, or an error code (which can be tested using ZSTD_isError()).
	*/
	ZSTDLIB_API size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize);

	/* ZSTD_d_format
	* experimental parameter,
	* allowing selection between ZSTD_format_e input compression formats
	*/
	#define ZSTD_d_format ZSTD_d_experimentalParam1
	/* ZSTD_d_stableOutBuffer
	* Experimental parameter.
	* Default is 0 == disabled. Set to 1 to enable.
	*
	* Tells the decompressor that the ZSTD_outBuffer will ALWAYS be the same
	* between calls, except for the modifications that zstd makes to pos (the
	* caller must not modify pos). This is checked by the decompressor, and
	* decompression will fail if it ever changes. Therefore the ZSTD_outBuffer
	* MUST be large enough to fit the entire decompressed frame. This will be
	* checked when the frame content size is known. The data in the ZSTD_outBuffer
	* in the range [dst, dst + pos) MUST not be modified during decompression
	* or you will get data corruption.
	*
	* When this flags is enabled zstd won't allocate an output buffer, because
	* it can write directly to the ZSTD_outBuffer, but it will still allocate
	* an input buffer large enough to fit any compressed block. This will also
	* avoid the memcpy() from the internal output buffer to the ZSTD_outBuffer.
	* If you need to avoid the input buffer allocation use the buffer-less
	* streaming API.
	*
	* NOTE: So long as the ZSTD_outBuffer always points to valid memory, using
	* this flag is ALWAYS memory safe, and will never access out-of-bounds
	* memory. However, decompression WILL fail if you violate the preconditions.
	*
	* WARNING: The data in the ZSTD_outBuffer in the range [dst, dst + pos) MUST
	* not be modified during decompression or you will get data corruption. This
	* is because zstd needs to reference data in the ZSTD_outBuffer to regenerate
	* matches. Normally zstd maintains its own buffer for this purpose, but passing
	* this flag tells zstd to use the user provided buffer.
	*/
	#define ZSTD_d_stableOutBuffer ZSTD_d_experimentalParam2

	/*! ZSTD_DCtx_setFormat() :
	* Instruct the decoder context about what kind of data to decode next.
	* This instruction is mandatory to decode data without a fully-formed header,
	* such ZSTD_f_zstd1_magicless for example.
	* @return : 0, or an error code (which can be tested using ZSTD_isError()). */
	ZSTDLIB_API size_t ZSTD_DCtx_setFormat(ZSTD_DCtx* dctx, ZSTD_format_e format);

	/*! ZSTD_decompressStream_simpleArgs() :
	* Same as ZSTD_decompressStream(),
	* but using only integral types as arguments.
	* This can be helpful for binders from dynamic languages
	* which have troubles handling structures containing memory pointers.
	*/
	ZSTDLIB_API size_t ZSTD_decompressStream_simpleArgs (
	ZSTD_DCtx* dctx,
	void* dst, size_t dstCapacity, size_t* dstPos,
	const void* src, size_t srcSize, size_t* srcPos);


	/********************************************************************
	* Advanced streaming functions
	* Warning : most of these functions are now redundant with the Advanced API.
	* Once Advanced API reaches "stable" status,
	* redundant functions will be deprecated, and then at some point removed.
	********************************************************************/

	/===== Advanced Streaming compression functions =====/
	/**! ZSTD_initCStream_srcSize() :
	* This function is deprecated, and equivalent to:
	* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
	* ZSTD_CCtx_refCDict(zcs, NULL); // clear the dictionary (if any)
	* ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel);
	* ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize);
	*
	* pledgedSrcSize must be correct. If it is not known at init time, use
	* ZSTD_CONTENTSIZE_UNKNOWN. Note that, for compatibility with older programs,
	* "0" also disables frame content size field. It may be enabled in the future.
	* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
	*/
	ZSTDLIB_API size_t
	ZSTD_initCStream_srcSize(ZSTD_CStream* zcs,
	int compressionLevel,
	unsigned long long pledgedSrcSize);

	/**! ZSTD_initCStream_usingDict() :
	* This function is deprecated, and is equivalent to:
	* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
	* ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel);
	* ZSTD_CCtx_loadDictionary(zcs, dict, dictSize);
	*
	* Creates of an internal CDict (incompatible with static CCtx), except if
	* dict == NULL or dictSize < 8, in which case no dict is used.
	* Note: dict is loaded with ZSTD_dct_auto (treated as a full zstd dictionary if
	* it begins with ZSTD_MAGIC_DICTIONARY, else as raw content) and ZSTD_dlm_byCopy.
	* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
	*/
	ZSTDLIB_API size_t
	ZSTD_initCStream_usingDict(ZSTD_CStream* zcs,
	const void* dict, size_t dictSize,
	int compressionLevel);

	/**! ZSTD_initCStream_advanced() :
	* This function is deprecated, and is approximately equivalent to:
	* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
	* // Pseudocode: Set each zstd parameter and leave the rest as-is.
	* for ((param, value) : params) {
	* ZSTD_CCtx_setParameter(zcs, param, value);
	* }
	* ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize);
	* ZSTD_CCtx_loadDictionary(zcs, dict, dictSize);
	*
	* dict is loaded with ZSTD_dct_auto and ZSTD_dlm_byCopy.
	* pledgedSrcSize must be correct.
	* If srcSize is not known at init time, use value ZSTD_CONTENTSIZE_UNKNOWN.
	* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
	*/
	ZSTDLIB_API size_t
	ZSTD_initCStream_advanced(ZSTD_CStream* zcs,
	const void* dict, size_t dictSize,
	ZSTD_parameters params,
	unsigned long long pledgedSrcSize);

	/**! ZSTD_initCStream_usingCDict() :
	* This function is deprecated, and equivalent to:
	* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
	* ZSTD_CCtx_refCDict(zcs, cdict);
	*
	* note : cdict will just be referenced, and must outlive compression session
	* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
	*/
	ZSTDLIB_API size_t ZSTD_initCStream_usingCDict(ZSTD_CStream* zcs, const ZSTD_CDict* cdict);

	/**! ZSTD_initCStream_usingCDict_advanced() :
	* This function is DEPRECATED, and is approximately equivalent to:
	* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
	* // Pseudocode: Set each zstd frame parameter and leave the rest as-is.
	* for ((fParam, value) : fParams) {
	* ZSTD_CCtx_setParameter(zcs, fParam, value);
	* }
	* ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize);
	* ZSTD_CCtx_refCDict(zcs, cdict);
	*
	* same as ZSTD_initCStream_usingCDict(), with control over frame parameters.
	* pledgedSrcSize must be correct. If srcSize is not known at init time, use
	* value ZSTD_CONTENTSIZE_UNKNOWN.
	* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
	*/
	ZSTDLIB_API size_t
	ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs,
	const ZSTD_CDict* cdict,
	ZSTD_frameParameters fParams,
	unsigned long long pledgedSrcSize);

	/*! ZSTD_resetCStream() :
	* This function is deprecated, and is equivalent to:
	* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
	* ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize);
	*
	* start a new frame, using same parameters from previous frame.
	* This is typically useful to skip dictionary loading stage, since it will re-use it in-place.
	* Note that zcs must be init at least once before using ZSTD_resetCStream().
	* If pledgedSrcSize is not known at reset time, use macro ZSTD_CONTENTSIZE_UNKNOWN.
	* If pledgedSrcSize > 0, its value must be correct, as it will be written in header, and controlled at the end.
	* For the time being, pledgedSrcSize==0 is interpreted as "srcSize unknown" for compatibility with older programs,
	* but it will change to mean "empty" in future version, so use macro ZSTD_CONTENTSIZE_UNKNOWN instead.
	* @return : 0, or an error code (which can be tested using ZSTD_isError())
	* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
	*/
	ZSTDLIB_API size_t ZSTD_resetCStream(ZSTD_CStream* zcs, unsigned long long pledgedSrcSize);


	typedef struct {
	unsigned long long ingested; /* nb input bytes read and buffered */
	unsigned long long consumed; /* nb input bytes actually compressed */
	unsigned long long produced; /* nb of compressed bytes generated and buffered */
	unsigned long long flushed; /* nb of compressed bytes flushed : not provided; can be tracked from caller side */
	unsigned currentJobID; /* MT only : latest started job nb */
	unsigned nbActiveWorkers; /* MT only : nb of workers actively compressing at probe time */
	} ZSTD_frameProgression;

	/* ZSTD_getFrameProgression() :
	* tells how much data has been ingested (read from input)
	* consumed (input actually compressed) and produced (output) for current frame.
	* Note : (ingested - consumed) is amount of input data buffered internally, not yet compressed.
	* Aggregates progression inside active worker threads.
	*/
	ZSTDLIB_API ZSTD_frameProgression ZSTD_getFrameProgression(const ZSTD_CCtx* cctx);

	/*! ZSTD_toFlushNow() :
	* Tell how many bytes are ready to be flushed immediately.
	* Useful for multithreading scenarios (nbWorkers >= 1).
	* Probe the oldest active job, defined as oldest job not yet entirely flushed,
	* and check its output buffer.
	* @return : amount of data stored in oldest job and ready to be flushed immediately.
	* if @return == 0, it means either :
	* + there is no active job (could be checked with ZSTD_frameProgression()), or
	* + oldest job is still actively compressing data,
	* but everything it has produced has also been flushed so far,
	* therefore flush speed is limited by production speed of oldest job
	* irrespective of the speed of concurrent (and newer) jobs.
	*/
	ZSTDLIB_API size_t ZSTD_toFlushNow(ZSTD_CCtx* cctx);


	/===== Advanced Streaming decompression functions =====/
	/**
	* This function is deprecated, and is equivalent to:
	*
	* ZSTD_DCtx_reset(zds, ZSTD_reset_session_only);
	* ZSTD_DCtx_loadDictionary(zds, dict, dictSize);
	*
	* note: no dictionary will be used if dict == NULL or dictSize < 8
	* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
	*/
	ZSTDLIB_API size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize);

	/**
	* This function is deprecated, and is equivalent to:
	*
	* ZSTD_DCtx_reset(zds, ZSTD_reset_session_only);
	* ZSTD_DCtx_refDDict(zds, ddict);
	*
	* note : ddict is referenced, it must outlive decompression session
	* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
	*/
	ZSTDLIB_API size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict);

	/**
	* This function is deprecated, and is equivalent to:
	*
	* ZSTD_DCtx_reset(zds, ZSTD_reset_session_only);
	*
	* re-use decompression parameters from previous init; saves dictionary loading
	* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
	*/
	ZSTDLIB_API size_t ZSTD_resetDStream(ZSTD_DStream* zds);


	/*********************************************************************
	* Buffer-less and synchronous inner streaming functions
	*
	* This is an advanced API, giving full control over buffer management, for users which need direct control over memory.
	* But it's also a complex one, with several restrictions, documented below.
	* Prefer normal streaming API for an easier experience.
	********************************************************************* */

	/**
	Buffer-less streaming compression (synchronous mode)

	A ZSTD_CCtx object is required to track streaming operations.
	Use ZSTD_createCCtx() / ZSTD_freeCCtx() to manage resource.
	ZSTD_CCtx object can be re-used multiple times within successive compression operations.

	Start by initializing a context.
	Use ZSTD_compressBegin(), or ZSTD_compressBegin_usingDict() for dictionary compression,
	or ZSTD_compressBegin_advanced(), for finer parameter control.
	It's also possible to duplicate a reference context which has already been initialized, using ZSTD_copyCCtx()

	Then, consume your input using ZSTD_compressContinue().
	There are some important considerations to keep in mind when using this advanced function :
	- ZSTD_compressContinue() has no internal buffer. It uses externally provided buffers only.
	- Interface is synchronous : input is consumed entirely and produces 1+ compressed blocks.
	- Caller must ensure there is enough space in `dst` to store compressed data under worst case scenario.
	Worst case evaluation is provided by ZSTD_compressBound().
	ZSTD_compressContinue() doesn't guarantee recover after a failed compression.
	- ZSTD_compressContinue() presumes prior input *is still accessible and unmodified* (up to maximum distance size, see WindowLog).
	It remembers all previous contiguous blocks, plus one separated memory segment (which can itself consists of multiple contiguous blocks)
	- ZSTD_compressContinue() detects that prior input has been overwritten when `src` buffer overlaps.
	In which case, it will "discard" the relevant memory section from its history.

	Finish a frame with ZSTD_compressEnd(), which will write the last block(s) and optional checksum.
	It's possible to use srcSize==0, in which case, it will write a final empty block to end the frame.
	Without last block mark, frames are considered unfinished (hence corrupted) by compliant decoders.

	`ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress again.
	*/

	/===== Buffer-less streaming compression functions =====/
	ZSTDLIB_API size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel);
	ZSTDLIB_API size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel);
	ZSTDLIB_API size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize); /*< pledgedSrcSize : If srcSize is not known at init time, use ZSTD_CONTENTSIZE_UNKNOWN /
	ZSTDLIB_API size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict); /*< note: fails if cdict==NULL /
	ZSTDLIB_API size_t ZSTD_compressBegin_usingCDict_advanced(ZSTD_CCtx* const cctx, const ZSTD_CDict* const cdict, ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize); /* compression parameters are already set within cdict. pledgedSrcSize must be correct. If srcSize is not known, use macro ZSTD_CONTENTSIZE_UNKNOWN */
	ZSTDLIB_API size_t ZSTD_copyCCtx(ZSTD_CCtx* cctx, const ZSTD_CCtx* preparedCCtx, unsigned long long pledgedSrcSize); /*< note: if pledgedSrcSize is not known, use ZSTD_CONTENTSIZE_UNKNOWN /

	ZSTDLIB_API size_t ZSTD_compressContinue(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
	ZSTDLIB_API size_t ZSTD_compressEnd(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);


	/*-
	Buffer-less streaming decompression (synchronous mode)

	A ZSTD_DCtx object is required to track streaming operations.
	Use ZSTD_createDCtx() / ZSTD_freeDCtx() to manage it.
	A ZSTD_DCtx object can be re-used multiple times.

	First typical operation is to retrieve frame parameters, using ZSTD_getFrameHeader().
	Frame header is extracted from the beginning of compressed frame, so providing only the frame's beginning is enough.
	Data fragment must be large enough to ensure successful decoding.
	`ZSTD_frameHeaderSize_max` bytes is guaranteed to always be large enough.
	@result : 0 : successful decoding, the `ZSTD_frameHeader` structure is correctly filled.
	>0 : `srcSize` is too small, please provide at least @result bytes on next attempt.
	errorCode, which can be tested using ZSTD_isError().

	It fills a ZSTD_frameHeader structure with important information to correctly decode the frame,
	such as the dictionary ID, content size, or maximum back-reference distance (`windowSize`).
	Note that these values could be wrong, either because of data corruption, or because a 3rd party deliberately spoofs false information.
	As a consequence, check that values remain within valid application range.
	For example, do not allocate memory blindly, check that `windowSize` is within expectation.
	Each application can set its own limits, depending on local restrictions.
	For extended interoperability, it is recommended to support `windowSize` of at least 8 MB.

	ZSTD_decompressContinue() needs previous data blocks during decompression, up to `windowSize` bytes.
	ZSTD_decompressContinue() is very sensitive to contiguity,
	if 2 blocks don't follow each other, make sure that either the compressor breaks contiguity at the same place,
	or that previous contiguous segment is large enough to properly handle maximum back-reference distance.
	There are multiple ways to guarantee this condition.

	The most memory efficient way is to use a round buffer of sufficient size.
	Sufficient size is determined by invoking ZSTD_decodingBufferSize_min(),
	which can @return an error code if required value is too large for current system (in 32-bits mode).
	In a round buffer methodology, ZSTD_decompressContinue() decompresses each block next to previous one,
	up to the moment there is not enough room left in the buffer to guarantee decoding another full block,
	which maximum size is provided in `ZSTD_frameHeader` structure, field `blockSizeMax`.
	At which point, decoding can resume from the beginning of the buffer.
	Note that already decoded data stored in the buffer should be flushed before being overwritten.

	There are alternatives possible, for example using two or more buffers of size `windowSize` each, though they consume more memory.

	Finally, if you control the compression process, you can also ignore all buffer size rules,
	as long as the encoder and decoder progress in "lock-step",
	aka use exactly the same buffer sizes, break contiguity at the same place, etc.

	Once buffers are setup, start decompression, with ZSTD_decompressBegin().
	If decompression requires a dictionary, use ZSTD_decompressBegin_usingDict() or ZSTD_decompressBegin_usingDDict().

	Then use ZSTD_nextSrcSizeToDecompress() and ZSTD_decompressContinue() alternatively.
	ZSTD_nextSrcSizeToDecompress() tells how many bytes to provide as 'srcSize' to ZSTD_decompressContinue().
	ZSTD_decompressContinue() requires this _exact_ amount of bytes, or it will fail.

	@result of ZSTD_decompressContinue() is the number of bytes regenerated within 'dst' (necessarily <= dstCapacity).
	It can be zero : it just means ZSTD_decompressContinue() has decoded some metadata item.
	It can also be an error code, which can be tested with ZSTD_isError().

	A frame is fully decoded when ZSTD_nextSrcSizeToDecompress() returns zero.
	Context can then be reset to start a new decompression.

	Note : it's possible to know if next input to present is a header or a block, using ZSTD_nextInputType().
	This information is not required to properly decode a frame.

	== Special case : skippable frames ==

	Skippable frames allow integration of user-defined data into a flow of concatenated frames.
	Skippable frames will be ignored (skipped) by decompressor.
	The format of skippable frames is as follows :
	a) Skippable frame ID - 4 Bytes, Little endian format, any value from 0x184D2A50 to 0x184D2A5F
	b) Frame Size - 4 Bytes, Little endian format, unsigned 32-bits
	c) Frame Content - any content (User Data) of length equal to Frame Size
	For skippable frames ZSTD_getFrameHeader() returns zfhPtr->frameType==ZSTD_skippableFrame.
	For skippable frames ZSTD_decompressContinue() always returns 0 : it only skips the content.
	*/

	/===== Buffer-less streaming decompression functions =====/
	typedef enum { ZSTD_frame, ZSTD_skippableFrame } ZSTD_frameType_e;
	typedef struct {
	unsigned long long frameContentSize; /* if == ZSTD_CONTENTSIZE_UNKNOWN, it means this field is not available. 0 means "empty" */
	unsigned long long windowSize; /* can be very large, up to <= frameContentSize */
	unsigned blockSizeMax;
	ZSTD_frameType_e frameType; /* if == ZSTD_skippableFrame, frameContentSize is the size of skippable content */
	unsigned headerSize;
	unsigned dictID;
	unsigned checksumFlag;
	} ZSTD_frameHeader;

	/*! ZSTD_getFrameHeader() :
	* decode Frame Header, or requires larger `srcSize`.
	* @return : 0, `zfhPtr` is correctly filled,
	* >0, `srcSize` is too small, value is wanted `srcSize` amount,
	* or an error code, which can be tested using ZSTD_isError() */
	ZSTDLIB_API size_t ZSTD_getFrameHeader(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize); /*< doesn't consume input /
	/*! ZSTD_getFrameHeader_advanced() :
	* same as ZSTD_getFrameHeader(),
	* with added capability to select a format (like ZSTD_f_zstd1_magicless) */
	ZSTDLIB_API size_t ZSTD_getFrameHeader_advanced(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize, ZSTD_format_e format);
	ZSTDLIB_API size_t ZSTD_decodingBufferSize_min(unsigned long long windowSize, unsigned long long frameContentSize); /*< when frame content size is not known, pass in frameContentSize == ZSTD_CONTENTSIZE_UNKNOWN /

	ZSTDLIB_API size_t ZSTD_decompressBegin(ZSTD_DCtx* dctx);
	ZSTDLIB_API size_t ZSTD_decompressBegin_usingDict(ZSTD_DCtx* dctx, const void* dict, size_t dictSize);
	ZSTDLIB_API size_t ZSTD_decompressBegin_usingDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict);

	ZSTDLIB_API size_t ZSTD_nextSrcSizeToDecompress(ZSTD_DCtx* dctx);
	ZSTDLIB_API size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);

	/* misc */
	ZSTDLIB_API void ZSTD_copyDCtx(ZSTD_DCtx* dctx, const ZSTD_DCtx* preparedDCtx);
	typedef enum { ZSTDnit_frameHeader, ZSTDnit_blockHeader, ZSTDnit_block, ZSTDnit_lastBlock, ZSTDnit_checksum, ZSTDnit_skippableFrame } ZSTD_nextInputType_e;
	ZSTDLIB_API ZSTD_nextInputType_e ZSTD_nextInputType(ZSTD_DCtx* dctx);




	/* ============================ */
	/** Block level API */
	/* ============================ */

	/*!
	Block functions produce and decode raw zstd blocks, without frame metadata.
	Frame metadata cost is typically ~12 bytes, which can be non-negligible for very small blocks (< 100 bytes).
	But users will have to take in charge needed metadata to regenerate data, such as compressed and content sizes.

	A few rules to respect :
	- Compressing and decompressing require a context structure
	+ Use ZSTD_createCCtx() and ZSTD_createDCtx()
	- It is necessary to init context before starting
	+ compression : any ZSTD_compressBegin*() variant, including with dictionary
	+ decompression : any ZSTD_decompressBegin*() variant, including with dictionary
	+ copyCCtx() and copyDCtx() can be used too
	- Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX == 128 KB
	+ If input is larger than a block size, it's necessary to split input data into multiple blocks
	+ For inputs larger than a single block, consider using regular ZSTD_compress() instead.
	Frame metadata is not that costly, and quickly becomes negligible as source size grows larger than a block.
	- When a block is considered not compressible enough, ZSTD_compressBlock() result will be 0 (zero) !
	===> In which case, nothing is produced into `dst` !
	+ User __must__ test for such outcome and deal directly with uncompressed data
	+ A block cannot be declared incompressible if ZSTD_compressBlock() return value was != 0.
	Doing so would mess up with statistics history, leading to potential data corruption.
	+ ZSTD_decompressBlock() _doesn't accept uncompressed data as input_ !!
	+ In case of multiple successive blocks, should some of them be uncompressed,
	decoder must be informed of their existence in order to follow proper history.
	Use ZSTD_insertBlock() for such a case.
	*/

	/===== Raw zstd block functions =====/
	ZSTDLIB_API size_t ZSTD_getBlockSize (const ZSTD_CCtx* cctx);
	ZSTDLIB_API size_t ZSTD_compressBlock (ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
	ZSTDLIB_API size_t ZSTD_decompressBlock(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
	ZSTDLIB_API size_t ZSTD_insertBlock (ZSTD_DCtx* dctx, const void* blockStart, size_t blockSize); /*< insert uncompressed block into `dctx` history. Useful for multi-blocks decompression. /


	#endif /* ZSTD_H_ZSTD_STATIC_LINKING_ONLY */

	#if defined (__cplusplus)
	}
	#endif
	/** ended inlining ../zstd.h **/
	#define FSE_STATIC_LINKING_ONLY
	/** skipping file: fse.h **/
	#define HUF_STATIC_LINKING_ONLY
	/** skipping file: huf.h **/
	#ifndef XXH_STATIC_LINKING_ONLY
	# define XXH_STATIC_LINKING_ONLY /* XXH64_state_t */
	#endif
	/** start inlining xxhash.h **/
	/*
	* xxHash - Extremely Fast Hash algorithm
	* Header File
	* Copyright (c) 2012-2020, Yann Collet, Facebook, Inc.
	*
	* You can contact the author at :
	* - xxHash source repository : https://github.com/Cyan4973/xxHash
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	/* Notice extracted from xxHash homepage :

	xxHash is an extremely fast Hash algorithm, running at RAM speed limits.
	It also successfully passes all tests from the SMHasher suite.

	Comparison (single thread, Windows Seven 32 bits, using SMHasher on a Core 2 Duo @3GHz)

	Name Speed Q.Score Author
	xxHash 5.4 GB/s 10
	CrapWow 3.2 GB/s 2 Andrew
	MumurHash 3a 2.7 GB/s 10 Austin Appleby
	SpookyHash 2.0 GB/s 10 Bob Jenkins
	SBox 1.4 GB/s 9 Bret Mulvey
	Lookup3 1.2 GB/s 9 Bob Jenkins
	SuperFastHash 1.2 GB/s 1 Paul Hsieh
	CityHash64 1.05 GB/s 10 Pike & Alakuijala
	FNV 0.55 GB/s 5 Fowler, Noll, Vo
	CRC32 0.43 GB/s 9
	MD5-32 0.33 GB/s 10 Ronald L. Rivest
	SHA1-32 0.28 GB/s 10

	Q.Score is a measure of quality of the hash function.
	It depends on successfully passing SMHasher test set.
	10 is a perfect score.

	A 64-bits version, named XXH64, is available since r35.
	It offers much better speed, but for 64-bits applications only.
	Name Speed on 64 bits Speed on 32 bits
	XXH64 13.8 GB/s 1.9 GB/s
	XXH32 6.8 GB/s 6.0 GB/s
	*/

	#if defined (__cplusplus)
	extern "C" {
	#endif

	#ifndef XXHASH_H_5627135585666179
	#define XXHASH_H_5627135585666179 1


	/* ****************************
	* Definitions
	******************************/
	#include <stddef.h> /* size_t */
	typedef enum { XXH_OK=0, XXH_ERROR } XXH_errorcode;


	/* ****************************
	* API modifier
	******************************/
	/** XXH_PRIVATE_API
	* This is useful if you want to include xxhash functions in `static` mode
	* in order to inline them, and remove their symbol from the public list.
	* Methodology :
	* #define XXH_PRIVATE_API
	* #include "xxhash.h"
	* `xxhash.c` is automatically included.
	* It's not useful to compile and link it as a separate module anymore.
	*/
	#ifdef XXH_PRIVATE_API
	# ifndef XXH_STATIC_LINKING_ONLY
	# define XXH_STATIC_LINKING_ONLY
	# endif
	# if defined(__GNUC__)
	# define XXH_PUBLIC_API static __inline __attribute__((unused))
	# elif defined (__cplusplus) \|\| (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */)
	# define XXH_PUBLIC_API static inline
	# elif defined(_MSC_VER)
	# define XXH_PUBLIC_API static __inline
	# else
	# define XXH_PUBLIC_API static /* this version may generate warnings for unused static functions; disable the relevant warning */
	# endif
	#else
	# define XXH_PUBLIC_API /* do nothing */
	#endif /* XXH_PRIVATE_API */

	/*!XXH_NAMESPACE, aka Namespace Emulation :

	If you want to include _and expose_ xxHash functions from within your own library,
	but also want to avoid symbol collisions with another library which also includes xxHash,

	you can use XXH_NAMESPACE, to automatically prefix any public symbol from xxhash library
	with the value of XXH_NAMESPACE (so avoid to keep it NULL and avoid numeric values).

	Note that no change is required within the calling program as long as it includes `xxhash.h` :
	regular symbol name will be automatically translated by this header.
	*/
	#ifdef XXH_NAMESPACE
	# define XXH_CAT(A,B) A##B
	# define XXH_NAME2(A,B) XXH_CAT(A,B)
	# define XXH32 XXH_NAME2(XXH_NAMESPACE, XXH32)
	# define XXH64 XXH_NAME2(XXH_NAMESPACE, XXH64)
	# define XXH_versionNumber XXH_NAME2(XXH_NAMESPACE, XXH_versionNumber)
	# define XXH32_createState XXH_NAME2(XXH_NAMESPACE, XXH32_createState)
	# define XXH64_createState XXH_NAME2(XXH_NAMESPACE, XXH64_createState)
	# define XXH32_freeState XXH_NAME2(XXH_NAMESPACE, XXH32_freeState)
	# define XXH64_freeState XXH_NAME2(XXH_NAMESPACE, XXH64_freeState)
	# define XXH32_reset XXH_NAME2(XXH_NAMESPACE, XXH32_reset)
	# define XXH64_reset XXH_NAME2(XXH_NAMESPACE, XXH64_reset)
	# define XXH32_update XXH_NAME2(XXH_NAMESPACE, XXH32_update)
	# define XXH64_update XXH_NAME2(XXH_NAMESPACE, XXH64_update)
	# define XXH32_digest XXH_NAME2(XXH_NAMESPACE, XXH32_digest)
	# define XXH64_digest XXH_NAME2(XXH_NAMESPACE, XXH64_digest)
	# define XXH32_copyState XXH_NAME2(XXH_NAMESPACE, XXH32_copyState)
	# define XXH64_copyState XXH_NAME2(XXH_NAMESPACE, XXH64_copyState)
	# define XXH32_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH32_canonicalFromHash)
	# define XXH64_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH64_canonicalFromHash)
	# define XXH32_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH32_hashFromCanonical)
	# define XXH64_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH64_hashFromCanonical)
	#endif


	/* *************************************
	* Version
	***************************************/
	#define XXH_VERSION_MAJOR 0
	#define XXH_VERSION_MINOR 6
	#define XXH_VERSION_RELEASE 2
	#define XXH_VERSION_NUMBER (XXH_VERSION_MAJOR 100100 + XXH_VERSION_MINOR *100 + XXH_VERSION_RELEASE)
	XXH_PUBLIC_API unsigned XXH_versionNumber (void);


	/* ****************************
	* Simple Hash Functions
	******************************/
	typedef unsigned int XXH32_hash_t;
	typedef unsigned long long XXH64_hash_t;

	XXH_PUBLIC_API XXH32_hash_t XXH32 (const void* input, size_t length, unsigned int seed);
	XXH_PUBLIC_API XXH64_hash_t XXH64 (const void* input, size_t length, unsigned long long seed);

	/*!
	XXH32() :
	Calculate the 32-bits hash of sequence "length" bytes stored at memory address "input".
	The memory between input & input+length must be valid (allocated and read-accessible).
	"seed" can be used to alter the result predictably.
	Speed on Core 2 Duo @ 3 GHz (single thread, SMHasher benchmark) : 5.4 GB/s
	XXH64() :
	Calculate the 64-bits hash of sequence of length "len" stored at memory address "input".
	"seed" can be used to alter the result predictably.
	This function runs 2x faster on 64-bits systems, but slower on 32-bits systems (see benchmark).
	*/


	/* ****************************
	* Streaming Hash Functions
	******************************/
	typedef struct XXH32_state_s XXH32_state_t; /* incomplete type */
	typedef struct XXH64_state_s XXH64_state_t; /* incomplete type */

	/! State allocation, compatible with dynamic libraries /

	XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void);
	XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr);

	XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void);
	XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr);


	/* hash streaming */

	XXH_PUBLIC_API XXH_errorcode XXH32_reset (XXH32_state_t* statePtr, unsigned int seed);
	XXH_PUBLIC_API XXH_errorcode XXH32_update (XXH32_state_t* statePtr, const void* input, size_t length);
	XXH_PUBLIC_API XXH32_hash_t XXH32_digest (const XXH32_state_t* statePtr);

	XXH_PUBLIC_API XXH_errorcode XXH64_reset (XXH64_state_t* statePtr, unsigned long long seed);
	XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH64_state_t* statePtr, const void* input, size_t length);
	XXH_PUBLIC_API XXH64_hash_t XXH64_digest (const XXH64_state_t* statePtr);

	/*
	These functions generate the xxHash of an input provided in multiple segments.
	Note that, for small input, they are slower than single-call functions, due to state management.
	For small input, prefer `XXH32()` and `XXH64()` .

	XXH state must first be allocated, using XXH*_createState() .

	Start a new hash by initializing state with a seed, using XXH*_reset().

	Then, feed the hash state by calling XXH*_update() as many times as necessary.
	Obviously, input must be allocated and read accessible.
	The function returns an error code, with 0 meaning OK, and any other value meaning there is an error.

	Finally, a hash value can be produced anytime, by using XXH*_digest().
	This function returns the nn-bits hash as an int or long long.

	It's still possible to continue inserting input into the hash state after a digest,
	and generate some new hashes later on, by calling again XXH*_digest().

	When done, free XXH state space if it was allocated dynamically.
	*/


	/* **************************
	* Utils
	****************************/
	#if !(defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L)) /* ! C99 */
	# define restrict /* disable restrict */
	#endif

	XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* restrict dst_state, const XXH32_state_t* restrict src_state);
	XXH_PUBLIC_API void XXH64_copyState(XXH64_state_t* restrict dst_state, const XXH64_state_t* restrict src_state);


	/* **************************
	* Canonical representation
	****************************/
	/* Default result type for XXH functions are primitive unsigned 32 and 64 bits.
	* The canonical representation uses human-readable write convention, aka big-endian (large digits first).
	* These functions allow transformation of hash result into and from its canonical format.
	* This way, hash values can be written into a file / memory, and remain comparable on different systems and programs.
	*/
	typedef struct { unsigned char digest[4]; } XXH32_canonical_t;
	typedef struct { unsigned char digest[8]; } XXH64_canonical_t;

	XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash);
	XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash);

	XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src);
	XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src);

	#endif /* XXHASH_H_5627135585666179 */



	/* ================================================================================================
	This section contains definitions which are not guaranteed to remain stable.
	They may change in future versions, becoming incompatible with a different version of the library.
	They shall only be used with static linking.
	Never use these definitions in association with dynamic linking !
	=================================================================================================== */
	#if defined(XXH_STATIC_LINKING_ONLY) && !defined(XXH_STATIC_H_3543687687345)
	#define XXH_STATIC_H_3543687687345

	/* These definitions are only meant to allow allocation of XXH state
	statically, on stack, or in a struct for example.
	Do not use members directly. */

	struct XXH32_state_s {
	unsigned total_len_32;
	unsigned large_len;
	unsigned v1;
	unsigned v2;
	unsigned v3;
	unsigned v4;
	unsigned mem32[4]; /* buffer defined as U32 for alignment */
	unsigned memsize;
	unsigned reserved; /* never read nor write, will be removed in a future version */
	}; /* typedef'd to XXH32_state_t */

	struct XXH64_state_s {
	unsigned long long total_len;
	unsigned long long v1;
	unsigned long long v2;
	unsigned long long v3;
	unsigned long long v4;
	unsigned long long mem64[4]; /* buffer defined as U64 for alignment */
	unsigned memsize;
	unsigned reserved[2]; /* never read nor write, will be removed in a future version */
	}; /* typedef'd to XXH64_state_t */


	# ifdef XXH_PRIVATE_API
	/** start inlining xxhash.c **/
	/*
	* xxHash - Fast Hash algorithm
	* Copyright (c) 2012-2020, Yann Collet, Facebook, Inc.
	*
	* You can contact the author at :
	* - xxHash homepage: http://www.xxhash.com
	* - xxHash source repository : https://github.com/Cyan4973/xxHash
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/


	/* *************************************
	* Tuning parameters
	***************************************/
	/*!XXH_FORCE_MEMORY_ACCESS :
	* By default, access to unaligned memory is controlled by `memcpy()`, which is safe and portable.
	* Unfortunately, on some target/compiler combinations, the generated assembly is sub-optimal.
	* The below switch allow to select different access method for improved performance.
	* Method 0 (default) : use `memcpy()`. Safe and portable.
	* Method 1 : `__packed` statement. It depends on compiler extension (ie, not portable).
	* This method is safe if your compiler supports it, and generally as fast or faster than `memcpy`.
	* Method 2 : direct access. This method doesn't depend on compiler but violate C standard.
	* It can generate buggy code on targets which do not support unaligned memory accesses.
	* But in some circumstances, it's the only known way to get the most performance (ie GCC + ARMv6)
	* See http://stackoverflow.com/a/32095106/646947 for details.
	* Prefer these methods in priority order (0 > 1 > 2)
	*/
	#ifndef XXH_FORCE_MEMORY_ACCESS /* can be defined externally, on command line for example */
	# if defined(__GNUC__) && ( defined(__ARM_ARCH_6__) \|\| defined(__ARM_ARCH_6J__) \|\| defined(__ARM_ARCH_6K__) \|\| defined(__ARM_ARCH_6Z__) \|\| defined(__ARM_ARCH_6ZK__) \|\| defined(__ARM_ARCH_6T2__) )
	# define XXH_FORCE_MEMORY_ACCESS 2
	# elif (defined(__INTEL_COMPILER) && !defined(WIN32)) \|\| \
	(defined(__GNUC__) && ( defined(__ARM_ARCH_7__) \|\| defined(__ARM_ARCH_7A__) \|\| defined(__ARM_ARCH_7R__) \|\| defined(__ARM_ARCH_7M__) \|\| defined(__ARM_ARCH_7S__) )) \|\| \
	defined(__ICCARM__)
	# define XXH_FORCE_MEMORY_ACCESS 1
	# endif
	#endif

	/*!XXH_ACCEPT_NULL_INPUT_POINTER :
	* If the input pointer is a null pointer, xxHash default behavior is to trigger a memory access error, since it is a bad pointer.
	* When this option is enabled, xxHash output for null input pointers will be the same as a null-length input.
	* By default, this option is disabled. To enable it, uncomment below define :
	*/
	/* #define XXH_ACCEPT_NULL_INPUT_POINTER 1 */

	/*!XXH_FORCE_NATIVE_FORMAT :
	* By default, xxHash library provides endian-independent Hash values, based on little-endian convention.
	* Results are therefore identical for little-endian and big-endian CPU.
	* This comes at a performance cost for big-endian CPU, since some swapping is required to emulate little-endian format.
	* Should endian-independence be of no importance for your application, you may set the #define below to 1,
	* to improve speed for Big-endian CPU.
	* This option has no impact on Little_Endian CPU.
	*/
	#ifndef XXH_FORCE_NATIVE_FORMAT /* can be defined externally */
	# define XXH_FORCE_NATIVE_FORMAT 0
	#endif

	/*!XXH_FORCE_ALIGN_CHECK :
	* This is a minor performance trick, only useful with lots of very small keys.
	* It means : check for aligned/unaligned input.
	* The check costs one initial branch per hash; set to 0 when the input data
	* is guaranteed to be aligned.
	*/
	#ifndef XXH_FORCE_ALIGN_CHECK /* can be defined externally */
	# if defined(__i386) \|\| defined(_M_IX86) \|\| defined(__x86_64__) \|\| defined(_M_X64)
	# define XXH_FORCE_ALIGN_CHECK 0
	# else
	# define XXH_FORCE_ALIGN_CHECK 1
	# endif
	#endif


	/* *************************************
	* Includes & Memory related functions
	***************************************/
	/* Modify the local functions below should you wish to use some other memory routines */
	/* for malloc(), free() */
	#include <stdlib.h>
	#include <stddef.h> /* size_t */
	static void* XXH_malloc(size_t s) { return malloc(s); }
	static void XXH_free (void* p) { free(p); }
	/* for memcpy() */
	#include <string.h>
	static void* XXH_memcpy(void* dest, const void* src, size_t size) { return memcpy(dest,src,size); }

	#ifndef XXH_STATIC_LINKING_ONLY
	# define XXH_STATIC_LINKING_ONLY
	#endif
	/** skipping file: xxhash.h **/


	/* *************************************
	* Compiler Specific Options
	***************************************/
	#if (defined(__GNUC__) && !defined(__STRICT_ANSI__)) \|\| defined(__cplusplus) \|\| defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */
	# define INLINE_KEYWORD inline
	#else
	# define INLINE_KEYWORD
	#endif

	#if defined(__GNUC__) \|\| defined(__ICCARM__)
	# define FORCE_INLINE_ATTR __attribute__((always_inline))
	#elif defined(_MSC_VER)
	# define FORCE_INLINE_ATTR __forceinline
	#else
	# define FORCE_INLINE_ATTR
	#endif

	#define FORCE_INLINE_TEMPLATE static INLINE_KEYWORD FORCE_INLINE_ATTR


	#ifdef _MSC_VER
	# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */
	#endif


	/* *************************************
	* Basic Types
	***************************************/
	#ifndef MEM_MODULE
	# define MEM_MODULE
	# if !defined (__VMS) && (defined (__cplusplus) \|\| (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
	# include <stdint.h>
	typedef uint8_t BYTE;
	typedef uint16_t U16;
	typedef uint32_t U32;
	typedef int32_t S32;
	typedef uint64_t U64;
	# else
	typedef unsigned char BYTE;
	typedef unsigned short U16;
	typedef unsigned int U32;
	typedef signed int S32;
	typedef unsigned long long U64; /* if your compiler doesn't support unsigned long long, replace by another 64-bit type here. Note that xxhash.h will also need to be updated. */
	# endif
	#endif


	#if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2))

	/* Force direct memory access. Only works on CPU which support unaligned memory access in hardware */
	static U32 XXH_read32(const void* memPtr) { return (const U32) memPtr; }
	static U64 XXH_read64(const void* memPtr) { return (const U64) memPtr; }

	#elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1))

	/* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */
	/* currently only defined for gcc and icc */
	typedef union { U32 u32; U64 u64; } __attribute__((packed)) unalign;

	static U32 XXH_read32(const void* ptr) { return ((const unalign*)ptr)->u32; }
	static U64 XXH_read64(const void* ptr) { return ((const unalign*)ptr)->u64; }

	#else

	/* portable and safe solution. Generally efficient.
	* see : http://stackoverflow.com/a/32095106/646947
	*/

	static U32 XXH_read32(const void* memPtr)
	{
	U32 val;
	memcpy(&val, memPtr, sizeof(val));
	return val;
	}

	static U64 XXH_read64(const void* memPtr)
	{
	U64 val;
	memcpy(&val, memPtr, sizeof(val));
	return val;
	}

	#endif /* XXH_FORCE_DIRECT_MEMORY_ACCESS */


	/* ****************************************
	* Compiler-specific Functions and Macros
	******************************************/
	#define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__)

	/* Note : although _rotl exists for minGW (GCC under windows), performance seems poor */
	#if defined(_MSC_VER)
	# define XXH_rotl32(x,r) _rotl(x,r)
	# define XXH_rotl64(x,r) _rotl64(x,r)
	#else
	#if defined(__ICCARM__)
	# include <intrinsics.h>
	# define XXH_rotl32(x,r) __ROR(x,(32 - r))
	#else
	# define XXH_rotl32(x,r) ((x << r) \| (x >> (32 - r)))
	#endif
	# define XXH_rotl64(x,r) ((x << r) \| (x >> (64 - r)))
	#endif

	#if defined(_MSC_VER) /* Visual Studio */
	# define XXH_swap32 _byteswap_ulong
	# define XXH_swap64 _byteswap_uint64
	#elif GCC_VERSION >= 403
	# define XXH_swap32 __builtin_bswap32
	# define XXH_swap64 __builtin_bswap64
	#else
	static U32 XXH_swap32 (U32 x)
	{
	return ((x << 24) & 0xff000000 ) \|
	((x << 8) & 0x00ff0000 ) \|
	((x >> 8) & 0x0000ff00 ) \|
	((x >> 24) & 0x000000ff );
	}
	static U64 XXH_swap64 (U64 x)
	{
	return ((x << 56) & 0xff00000000000000ULL) \|
	((x << 40) & 0x00ff000000000000ULL) \|
	((x << 24) & 0x0000ff0000000000ULL) \|
	((x << 8) & 0x000000ff00000000ULL) \|
	((x >> 8) & 0x00000000ff000000ULL) \|
	((x >> 24) & 0x0000000000ff0000ULL) \|
	((x >> 40) & 0x000000000000ff00ULL) \|
	((x >> 56) & 0x00000000000000ffULL);
	}
	#endif


	/* *************************************
	* Architecture Macros
	***************************************/
	typedef enum { XXH_bigEndian=0, XXH_littleEndian=1 } XXH_endianess;

	/* XXH_CPU_LITTLE_ENDIAN can be defined externally, for example on the compiler command line */
	#ifndef XXH_CPU_LITTLE_ENDIAN
	static const int g_one = 1;
	# define XXH_CPU_LITTLE_ENDIAN ((const char)(&g_one))
	#endif


	/* ***************************
	* Memory reads
	*****************************/
	typedef enum { XXH_aligned, XXH_unaligned } XXH_alignment;

	FORCE_INLINE_TEMPLATE U32 XXH_readLE32_align(const void* ptr, XXH_endianess endian, XXH_alignment align)
	{
	if (align==XXH_unaligned)
	return endian==XXH_littleEndian ? XXH_read32(ptr) : XXH_swap32(XXH_read32(ptr));
	else
	return endian==XXH_littleEndian ? (const U32)ptr : XXH_swap32((const U32)ptr);
	}

	FORCE_INLINE_TEMPLATE U32 XXH_readLE32(const void* ptr, XXH_endianess endian)
	{
	return XXH_readLE32_align(ptr, endian, XXH_unaligned);
	}

	static U32 XXH_readBE32(const void* ptr)
	{
	return XXH_CPU_LITTLE_ENDIAN ? XXH_swap32(XXH_read32(ptr)) : XXH_read32(ptr);
	}

	FORCE_INLINE_TEMPLATE U64 XXH_readLE64_align(const void* ptr, XXH_endianess endian, XXH_alignment align)
	{
	if (align==XXH_unaligned)
	return endian==XXH_littleEndian ? XXH_read64(ptr) : XXH_swap64(XXH_read64(ptr));
	else
	return endian==XXH_littleEndian ? (const U64)ptr : XXH_swap64((const U64)ptr);
	}

	FORCE_INLINE_TEMPLATE U64 XXH_readLE64(const void* ptr, XXH_endianess endian)
	{
	return XXH_readLE64_align(ptr, endian, XXH_unaligned);
	}

	static U64 XXH_readBE64(const void* ptr)
	{
	return XXH_CPU_LITTLE_ENDIAN ? XXH_swap64(XXH_read64(ptr)) : XXH_read64(ptr);
	}


	/* *************************************
	* Macros
	***************************************/
	#define XXH_STATIC_ASSERT(c) { enum { XXH_static_assert = 1/(int)(!!(c)) }; } /* use only after variable declarations */


	/* *************************************
	* Constants
	***************************************/
	static const U32 PRIME32_1 = 2654435761U;
	static const U32 PRIME32_2 = 2246822519U;
	static const U32 PRIME32_3 = 3266489917U;
	static const U32 PRIME32_4 = 668265263U;
	static const U32 PRIME32_5 = 374761393U;

	static const U64 PRIME64_1 = 11400714785074694791ULL;
	static const U64 PRIME64_2 = 14029467366897019727ULL;
	static const U64 PRIME64_3 = 1609587929392839161ULL;
	static const U64 PRIME64_4 = 9650029242287828579ULL;
	static const U64 PRIME64_5 = 2870177450012600261ULL;

	XXH_PUBLIC_API unsigned XXH_versionNumber (void) { return XXH_VERSION_NUMBER; }


	/* **************************
	* Utils
	****************************/
	XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* restrict dstState, const XXH32_state_t* restrict srcState)
	{
	memcpy(dstState, srcState, sizeof(*dstState));
	}

	XXH_PUBLIC_API void XXH64_copyState(XXH64_state_t* restrict dstState, const XXH64_state_t* restrict srcState)
	{
	memcpy(dstState, srcState, sizeof(*dstState));
	}


	/* ***************************
	* Simple Hash Functions
	*****************************/

	static U32 XXH32_round(U32 seed, U32 input)
	{
	seed += input * PRIME32_2;
	seed = XXH_rotl32(seed, 13);
	seed *= PRIME32_1;
	return seed;
	}

	FORCE_INLINE_TEMPLATE U32 XXH32_endian_align(const void* input, size_t len, U32 seed, XXH_endianess endian, XXH_alignment align)
	{
	const BYTE* p = (const BYTE*)input;
	const BYTE* bEnd = p + len;
	U32 h32;
	#define XXH_get32bits(p) XXH_readLE32_align(p, endian, align)

	#ifdef XXH_ACCEPT_NULL_INPUT_POINTER
	if (p==NULL) {
	len=0;
	bEnd=p=(const BYTE*)(size_t)16;
	}
	#endif

	if (len>=16) {
	const BYTE* const limit = bEnd - 16;
	U32 v1 = seed + PRIME32_1 + PRIME32_2;
	U32 v2 = seed + PRIME32_2;
	U32 v3 = seed + 0;
	U32 v4 = seed - PRIME32_1;

	do {
	v1 = XXH32_round(v1, XXH_get32bits(p)); p+=4;
	v2 = XXH32_round(v2, XXH_get32bits(p)); p+=4;
	v3 = XXH32_round(v3, XXH_get32bits(p)); p+=4;
	v4 = XXH32_round(v4, XXH_get32bits(p)); p+=4;
	} while (p<=limit);

	h32 = XXH_rotl32(v1, 1) + XXH_rotl32(v2, 7) + XXH_rotl32(v3, 12) + XXH_rotl32(v4, 18);
	} else {
	h32 = seed + PRIME32_5;
	}

	h32 += (U32) len;

	while (p+4<=bEnd) {
	h32 += XXH_get32bits(p) * PRIME32_3;
	h32 = XXH_rotl32(h32, 17) * PRIME32_4 ;
	p+=4;
	}

	while (p<bEnd) {
	h32 += (p) PRIME32_5;
	h32 = XXH_rotl32(h32, 11) * PRIME32_1 ;
	p++;
	}

	h32 ^= h32 >> 15;
	h32 *= PRIME32_2;
	h32 ^= h32 >> 13;
	h32 *= PRIME32_3;
	h32 ^= h32 >> 16;

	return h32;
	}


	XXH_PUBLIC_API unsigned int XXH32 (const void* input, size_t len, unsigned int seed)
	{
	#if 0
	/* Simple version, good for code maintenance, but unfortunately slow for small inputs */
	XXH32_CREATESTATE_STATIC(state);
	XXH32_reset(state, seed);
	XXH32_update(state, input, len);
	return XXH32_digest(state);
	#else
	XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN;

	if (XXH_FORCE_ALIGN_CHECK) {
	if ((((size_t)input) & 3) == 0) { /* Input is 4-bytes aligned, leverage the speed benefit */
	if ((endian_detected==XXH_littleEndian) \|\| XXH_FORCE_NATIVE_FORMAT)
	return XXH32_endian_align(input, len, seed, XXH_littleEndian, XXH_aligned);
	else
	return XXH32_endian_align(input, len, seed, XXH_bigEndian, XXH_aligned);
	} }

	if ((endian_detected==XXH_littleEndian) \|\| XXH_FORCE_NATIVE_FORMAT)
	return XXH32_endian_align(input, len, seed, XXH_littleEndian, XXH_unaligned);
	else
	return XXH32_endian_align(input, len, seed, XXH_bigEndian, XXH_unaligned);
	#endif
	}


	static U64 XXH64_round(U64 acc, U64 input)
	{
	acc += input * PRIME64_2;
	acc = XXH_rotl64(acc, 31);
	acc *= PRIME64_1;
	return acc;
	}

	static U64 XXH64_mergeRound(U64 acc, U64 val)
	{
	val = XXH64_round(0, val);
	acc ^= val;
	acc = acc * PRIME64_1 + PRIME64_4;
	return acc;
	}

	FORCE_INLINE_TEMPLATE U64 XXH64_endian_align(const void* input, size_t len, U64 seed, XXH_endianess endian, XXH_alignment align)
	{
	const BYTE* p = (const BYTE*)input;
	const BYTE* const bEnd = p + len;
	U64 h64;
	#define XXH_get64bits(p) XXH_readLE64_align(p, endian, align)

	#ifdef XXH_ACCEPT_NULL_INPUT_POINTER
	if (p==NULL) {
	len=0;
	bEnd=p=(const BYTE*)(size_t)32;
	}
	#endif

	if (len>=32) {
	const BYTE* const limit = bEnd - 32;
	U64 v1 = seed + PRIME64_1 + PRIME64_2;
	U64 v2 = seed + PRIME64_2;
	U64 v3 = seed + 0;
	U64 v4 = seed - PRIME64_1;

	do {
	v1 = XXH64_round(v1, XXH_get64bits(p)); p+=8;
	v2 = XXH64_round(v2, XXH_get64bits(p)); p+=8;
	v3 = XXH64_round(v3, XXH_get64bits(p)); p+=8;
	v4 = XXH64_round(v4, XXH_get64bits(p)); p+=8;
	} while (p<=limit);

	h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18);
	h64 = XXH64_mergeRound(h64, v1);
	h64 = XXH64_mergeRound(h64, v2);
	h64 = XXH64_mergeRound(h64, v3);
	h64 = XXH64_mergeRound(h64, v4);

	} else {
	h64 = seed + PRIME64_5;
	}

	h64 += (U64) len;

	while (p+8<=bEnd) {
	U64 const k1 = XXH64_round(0, XXH_get64bits(p));
	h64 ^= k1;
	h64 = XXH_rotl64(h64,27) * PRIME64_1 + PRIME64_4;
	p+=8;
	}

	if (p+4<=bEnd) {
	h64 ^= (U64)(XXH_get32bits(p)) * PRIME64_1;
	h64 = XXH_rotl64(h64, 23) * PRIME64_2 + PRIME64_3;
	p+=4;
	}

	while (p<bEnd) {
	h64 ^= (p) PRIME64_5;
	h64 = XXH_rotl64(h64, 11) * PRIME64_1;
	p++;
	}

	h64 ^= h64 >> 33;
	h64 *= PRIME64_2;
	h64 ^= h64 >> 29;
	h64 *= PRIME64_3;
	h64 ^= h64 >> 32;

	return h64;
	}


	XXH_PUBLIC_API unsigned long long XXH64 (const void* input, size_t len, unsigned long long seed)
	{
	#if 0
	/* Simple version, good for code maintenance, but unfortunately slow for small inputs */
	XXH64_CREATESTATE_STATIC(state);
	XXH64_reset(state, seed);
	XXH64_update(state, input, len);
	return XXH64_digest(state);
	#else
	XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN;

	if (XXH_FORCE_ALIGN_CHECK) {
	if ((((size_t)input) & 7)==0) { /* Input is aligned, let's leverage the speed advantage */
	if ((endian_detected==XXH_littleEndian) \|\| XXH_FORCE_NATIVE_FORMAT)
	return XXH64_endian_align(input, len, seed, XXH_littleEndian, XXH_aligned);
	else
	return XXH64_endian_align(input, len, seed, XXH_bigEndian, XXH_aligned);
	} }

	if ((endian_detected==XXH_littleEndian) \|\| XXH_FORCE_NATIVE_FORMAT)
	return XXH64_endian_align(input, len, seed, XXH_littleEndian, XXH_unaligned);
	else
	return XXH64_endian_align(input, len, seed, XXH_bigEndian, XXH_unaligned);
	#endif
	}


	/* **************************************************
	* Advanced Hash Functions
	****************************************************/

	XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void)
	{
	return (XXH32_state_t*)XXH_malloc(sizeof(XXH32_state_t));
	}
	XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr)
	{
	XXH_free(statePtr);
	return XXH_OK;
	}

	XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void)
	{
	return (XXH64_state_t*)XXH_malloc(sizeof(XXH64_state_t));
	}
	XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr)
	{
	XXH_free(statePtr);
	return XXH_OK;
	}


	/* Hash feed */

	XXH_PUBLIC_API XXH_errorcode XXH32_reset(XXH32_state_t* statePtr, unsigned int seed)
	{
	XXH32_state_t state; /* using a local state to memcpy() in order to avoid strict-aliasing warnings */
	memset(&state, 0, sizeof(state)-4); /* do not write into reserved, for future removal */
	state.v1 = seed + PRIME32_1 + PRIME32_2;
	state.v2 = seed + PRIME32_2;
	state.v3 = seed + 0;
	state.v4 = seed - PRIME32_1;
	memcpy(statePtr, &state, sizeof(state));
	return XXH_OK;
	}


	XXH_PUBLIC_API XXH_errorcode XXH64_reset(XXH64_state_t* statePtr, unsigned long long seed)
	{
	XXH64_state_t state; /* using a local state to memcpy() in order to avoid strict-aliasing warnings */
	memset(&state, 0, sizeof(state)-8); /* do not write into reserved, for future removal */
	state.v1 = seed + PRIME64_1 + PRIME64_2;
	state.v2 = seed + PRIME64_2;
	state.v3 = seed + 0;
	state.v4 = seed - PRIME64_1;
	memcpy(statePtr, &state, sizeof(state));
	return XXH_OK;
	}


	FORCE_INLINE_TEMPLATE XXH_errorcode XXH32_update_endian (XXH32_state_t* state, const void* input, size_t len, XXH_endianess endian)
	{
	const BYTE* p = (const BYTE*)input;
	const BYTE* const bEnd = p + len;

	#ifdef XXH_ACCEPT_NULL_INPUT_POINTER
	if (input==NULL) return XXH_ERROR;
	#endif

	state->total_len_32 += (unsigned)len;
	state->large_len \|= (len>=16) \| (state->total_len_32>=16);

	if (state->memsize + len < 16) { /* fill in tmp buffer */
	XXH_memcpy((BYTE*)(state->mem32) + state->memsize, input, len);
	state->memsize += (unsigned)len;
	return XXH_OK;
	}

	if (state->memsize) { /* some data left from previous update */
	XXH_memcpy((BYTE*)(state->mem32) + state->memsize, input, 16-state->memsize);
	{ const U32* p32 = state->mem32;
	state->v1 = XXH32_round(state->v1, XXH_readLE32(p32, endian)); p32++;
	state->v2 = XXH32_round(state->v2, XXH_readLE32(p32, endian)); p32++;
	state->v3 = XXH32_round(state->v3, XXH_readLE32(p32, endian)); p32++;
	state->v4 = XXH32_round(state->v4, XXH_readLE32(p32, endian)); p32++;
	}
	p += 16-state->memsize;
	state->memsize = 0;
	}

	if (p <= bEnd-16) {
	const BYTE* const limit = bEnd - 16;
	U32 v1 = state->v1;
	U32 v2 = state->v2;
	U32 v3 = state->v3;
	U32 v4 = state->v4;

	do {
	v1 = XXH32_round(v1, XXH_readLE32(p, endian)); p+=4;
	v2 = XXH32_round(v2, XXH_readLE32(p, endian)); p+=4;
	v3 = XXH32_round(v3, XXH_readLE32(p, endian)); p+=4;
	v4 = XXH32_round(v4, XXH_readLE32(p, endian)); p+=4;
	} while (p<=limit);

	state->v1 = v1;
	state->v2 = v2;
	state->v3 = v3;
	state->v4 = v4;
	}

	if (p < bEnd) {
	XXH_memcpy(state->mem32, p, (size_t)(bEnd-p));
	state->memsize = (unsigned)(bEnd-p);
	}

	return XXH_OK;
	}

	XXH_PUBLIC_API XXH_errorcode XXH32_update (XXH32_state_t* state_in, const void* input, size_t len)
	{
	XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN;

	if ((endian_detected==XXH_littleEndian) \|\| XXH_FORCE_NATIVE_FORMAT)
	return XXH32_update_endian(state_in, input, len, XXH_littleEndian);
	else
	return XXH32_update_endian(state_in, input, len, XXH_bigEndian);
	}



	FORCE_INLINE_TEMPLATE U32 XXH32_digest_endian (const XXH32_state_t* state, XXH_endianess endian)
	{
	const BYTE * p = (const BYTE*)state->mem32;
	const BYTE* const bEnd = (const BYTE*)(state->mem32) + state->memsize;
	U32 h32;

	if (state->large_len) {
	h32 = XXH_rotl32(state->v1, 1) + XXH_rotl32(state->v2, 7) + XXH_rotl32(state->v3, 12) + XXH_rotl32(state->v4, 18);
	} else {
	h32 = state->v3 /* == seed */ + PRIME32_5;
	}

	h32 += state->total_len_32;

	while (p+4<=bEnd) {
	h32 += XXH_readLE32(p, endian) * PRIME32_3;
	h32 = XXH_rotl32(h32, 17) * PRIME32_4;
	p+=4;
	}

	while (p<bEnd) {
	h32 += (p) PRIME32_5;
	h32 = XXH_rotl32(h32, 11) * PRIME32_1;
	p++;
	}

	h32 ^= h32 >> 15;
	h32 *= PRIME32_2;
	h32 ^= h32 >> 13;
	h32 *= PRIME32_3;
	h32 ^= h32 >> 16;

	return h32;
	}


	XXH_PUBLIC_API unsigned int XXH32_digest (const XXH32_state_t* state_in)
	{
	XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN;

	if ((endian_detected==XXH_littleEndian) \|\| XXH_FORCE_NATIVE_FORMAT)
	return XXH32_digest_endian(state_in, XXH_littleEndian);
	else
	return XXH32_digest_endian(state_in, XXH_bigEndian);
	}



	/* ** XXH64 ** */

	FORCE_INLINE_TEMPLATE XXH_errorcode XXH64_update_endian (XXH64_state_t* state, const void* input, size_t len, XXH_endianess endian)
	{
	const BYTE* p = (const BYTE*)input;
	const BYTE* const bEnd = p + len;

	#ifdef XXH_ACCEPT_NULL_INPUT_POINTER
	if (input==NULL) return XXH_ERROR;
	#endif

	state->total_len += len;

	if (state->memsize + len < 32) { /* fill in tmp buffer */
	if (input != NULL) {
	XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, len);
	}
	state->memsize += (U32)len;
	return XXH_OK;
	}

	if (state->memsize) { /* tmp buffer is full */
	XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, 32-state->memsize);
	state->v1 = XXH64_round(state->v1, XXH_readLE64(state->mem64+0, endian));
	state->v2 = XXH64_round(state->v2, XXH_readLE64(state->mem64+1, endian));
	state->v3 = XXH64_round(state->v3, XXH_readLE64(state->mem64+2, endian));
	state->v4 = XXH64_round(state->v4, XXH_readLE64(state->mem64+3, endian));
	p += 32-state->memsize;
	state->memsize = 0;
	}

	if (p+32 <= bEnd) {
	const BYTE* const limit = bEnd - 32;
	U64 v1 = state->v1;
	U64 v2 = state->v2;
	U64 v3 = state->v3;
	U64 v4 = state->v4;

	do {
	v1 = XXH64_round(v1, XXH_readLE64(p, endian)); p+=8;
	v2 = XXH64_round(v2, XXH_readLE64(p, endian)); p+=8;
	v3 = XXH64_round(v3, XXH_readLE64(p, endian)); p+=8;
	v4 = XXH64_round(v4, XXH_readLE64(p, endian)); p+=8;
	} while (p<=limit);

	state->v1 = v1;
	state->v2 = v2;
	state->v3 = v3;
	state->v4 = v4;
	}

	if (p < bEnd) {
	XXH_memcpy(state->mem64, p, (size_t)(bEnd-p));
	state->memsize = (unsigned)(bEnd-p);
	}

	return XXH_OK;
	}

	XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH64_state_t* state_in, const void* input, size_t len)
	{
	XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN;

	if ((endian_detected==XXH_littleEndian) \|\| XXH_FORCE_NATIVE_FORMAT)
	return XXH64_update_endian(state_in, input, len, XXH_littleEndian);
	else
	return XXH64_update_endian(state_in, input, len, XXH_bigEndian);
	}



	FORCE_INLINE_TEMPLATE U64 XXH64_digest_endian (const XXH64_state_t* state, XXH_endianess endian)
	{
	const BYTE * p = (const BYTE*)state->mem64;
	const BYTE* const bEnd = (const BYTE*)state->mem64 + state->memsize;
	U64 h64;

	if (state->total_len >= 32) {
	U64 const v1 = state->v1;
	U64 const v2 = state->v2;
	U64 const v3 = state->v3;
	U64 const v4 = state->v4;

	h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18);
	h64 = XXH64_mergeRound(h64, v1);
	h64 = XXH64_mergeRound(h64, v2);
	h64 = XXH64_mergeRound(h64, v3);
	h64 = XXH64_mergeRound(h64, v4);
	} else {
	h64 = state->v3 + PRIME64_5;
	}

	h64 += (U64) state->total_len;

	while (p+8<=bEnd) {
	U64 const k1 = XXH64_round(0, XXH_readLE64(p, endian));
	h64 ^= k1;
	h64 = XXH_rotl64(h64,27) * PRIME64_1 + PRIME64_4;
	p+=8;
	}

	if (p+4<=bEnd) {
	h64 ^= (U64)(XXH_readLE32(p, endian)) * PRIME64_1;
	h64 = XXH_rotl64(h64, 23) * PRIME64_2 + PRIME64_3;
	p+=4;
	}

	while (p<bEnd) {
	h64 ^= (p) PRIME64_5;
	h64 = XXH_rotl64(h64, 11) * PRIME64_1;
	p++;
	}

	h64 ^= h64 >> 33;
	h64 *= PRIME64_2;
	h64 ^= h64 >> 29;
	h64 *= PRIME64_3;
	h64 ^= h64 >> 32;

	return h64;
	}


	XXH_PUBLIC_API unsigned long long XXH64_digest (const XXH64_state_t* state_in)
	{
	XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN;

	if ((endian_detected==XXH_littleEndian) \|\| XXH_FORCE_NATIVE_FORMAT)
	return XXH64_digest_endian(state_in, XXH_littleEndian);
	else
	return XXH64_digest_endian(state_in, XXH_bigEndian);
	}


	/* **************************
	* Canonical representation
	****************************/

	/*! Default XXH result types are basic unsigned 32 and 64 bits.
	* The canonical representation follows human-readable write convention, aka big-endian (large digits first).
	* These functions allow transformation of hash result into and from its canonical format.
	* This way, hash values can be written into a file or buffer, and remain comparable across different systems and programs.
	*/

	XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash)
	{
	XXH_STATIC_ASSERT(sizeof(XXH32_canonical_t) == sizeof(XXH32_hash_t));
	if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap32(hash);
	memcpy(dst, &hash, sizeof(*dst));
	}

	XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash)
	{
	XXH_STATIC_ASSERT(sizeof(XXH64_canonical_t) == sizeof(XXH64_hash_t));
	if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap64(hash);
	memcpy(dst, &hash, sizeof(*dst));
	}

	XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src)
	{
	return XXH_readBE32(src);
	}

	XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src)
	{
	return XXH_readBE64(src);
	}
	/** ended inlining xxhash.c **/
	# endif

	#endif /* XXH_STATIC_LINKING_ONLY && XXH_STATIC_H_3543687687345 */


	#if defined (__cplusplus)
	}
	#endif
	/** ended inlining xxhash.h **/

	#if defined (__cplusplus)
	extern "C" {
	#endif

	/* ---- static assert (debug) --- */
	#define ZSTD_STATIC_ASSERT(c) DEBUG_STATIC_ASSERT(c)
	#define ZSTD_isError ERR_isError /* for inlining */
	#define FSE_isError ERR_isError
	#define HUF_isError ERR_isError


	/-************************************
	* shared macros
	***************************************/
	#undef MIN
	#undef MAX
	#define MIN(a,b) ((a)<(b) ? (a) : (b))
	#define MAX(a,b) ((a)>(b) ? (a) : (b))

	/**
	* Ignore: this is an internal helper.
	*
	* This is a helper function to help force C99-correctness during compilation.
	* Under strict compilation modes, variadic macro arguments can't be empty.
	* However, variadic function arguments can be. Using a function therefore lets
	* us statically check that at least one (string) argument was passed,
	* independent of the compilation flags.
	*/
	static INLINE_KEYWORD UNUSED_ATTR
	void _force_has_format_string(const char *format, ...) {
	(void)format;
	}

	/**
	* Ignore: this is an internal helper.
	*
	* We want to force this function invocation to be syntactically correct, but
	* we don't want to force runtime evaluation of its arguments.
	*/
	#define _FORCE_HAS_FORMAT_STRING(...) \
	if (0) { \
	_force_has_format_string(__VA_ARGS__); \
	}

	/**
	* Return the specified error if the condition evaluates to true.
	*
	* In debug modes, prints additional information.
	* In order to do that (particularly, printing the conditional that failed),
	* this can't just wrap RETURN_ERROR().
	*/
	#define RETURN_ERROR_IF(cond, err, ...) \
	if (cond) { \
	RAWLOG(3, "%s:%d: ERROR!: check %s failed, returning %s", \
	__FILE__, __LINE__, ZSTD_QUOTE(cond), ZSTD_QUOTE(ERROR(err))); \
	_FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \
	RAWLOG(3, ": " __VA_ARGS__); \
	RAWLOG(3, "\n"); \
	return ERROR(err); \
	}

	/**
	* Unconditionally return the specified error.
	*
	* In debug modes, prints additional information.
	*/
	#define RETURN_ERROR(err, ...) \
	do { \
	RAWLOG(3, "%s:%d: ERROR!: unconditional check failed, returning %s", \
	__FILE__, __LINE__, ZSTD_QUOTE(ERROR(err))); \
	_FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \
	RAWLOG(3, ": " __VA_ARGS__); \
	RAWLOG(3, "\n"); \
	return ERROR(err); \
	} while(0);

	/**
	* If the provided expression evaluates to an error code, returns that error code.
	*
	* In debug modes, prints additional information.
	*/
	#define FORWARD_IF_ERROR(err, ...) \
	do { \
	size_t const err_code = (err); \
	if (ERR_isError(err_code)) { \
	RAWLOG(3, "%s:%d: ERROR!: forwarding error in %s: %s", \
	__FILE__, __LINE__, ZSTD_QUOTE(err), ERR_getErrorName(err_code)); \
	_FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \
	RAWLOG(3, ": " __VA_ARGS__); \
	RAWLOG(3, "\n"); \
	return err_code; \
	} \
	} while(0);


	/-************************************
	* Common constants
	***************************************/
	#define ZSTD_OPT_NUM (1<<12)

	#define ZSTD_REP_NUM 3 /* number of repcodes */
	#define ZSTD_REP_MOVE (ZSTD_REP_NUM-1)
	static const U32 repStartValue[ZSTD_REP_NUM] = { 1, 4, 8 };

	#define KB *(1 <<10)
	#define MB *(1 <<20)
	#define GB *(1U<<30)

	#define BIT7 128
	#define BIT6 64
	#define BIT5 32
	#define BIT4 16
	#define BIT1 2
	#define BIT0 1

	#define ZSTD_WINDOWLOG_ABSOLUTEMIN 10
	static const size_t ZSTD_fcs_fieldSize[4] = { 0, 2, 4, 8 };
	static const size_t ZSTD_did_fieldSize[4] = { 0, 1, 2, 4 };

	#define ZSTD_FRAMEIDSIZE 4 /* magic number size */

	#define ZSTD_BLOCKHEADERSIZE 3 /* C standard doesn't allow `static const` variable to be init using another `static const` variable */
	static const size_t ZSTD_blockHeaderSize = ZSTD_BLOCKHEADERSIZE;
	typedef enum { bt_raw, bt_rle, bt_compressed, bt_reserved } blockType_e;

	#define ZSTD_FRAMECHECKSUMSIZE 4

	#define MIN_SEQUENCES_SIZE 1 /* nbSeq==0 */
	#define MIN_CBLOCK_SIZE (1 /litCSize/ + 1 /* RLE or RAW / + MIN_SEQUENCES_SIZE / nbSeq==0 /) / for a non-null block */

	#define HufLog 12
	typedef enum { set_basic, set_rle, set_compressed, set_repeat } symbolEncodingType_e;

	#define LONGNBSEQ 0x7F00

	#define MINMATCH 3

	#define Litbits 8
	#define MaxLit ((1<<Litbits) - 1)
	#define MaxML 52
	#define MaxLL 35
	#define DefaultMaxOff 28
	#define MaxOff 31
	#define MaxSeq MAX(MaxLL, MaxML) /* Assumption : MaxOff < MaxLL,MaxML */
	#define MLFSELog 9
	#define LLFSELog 9
	#define OffFSELog 8
	#define MaxFSELog MAX(MAX(MLFSELog, LLFSELog), OffFSELog)

	static const U32 LL_bits[MaxLL+1] = { 0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	1, 1, 1, 1, 2, 2, 3, 3,
	4, 6, 7, 8, 9,10,11,12,
	13,14,15,16 };
	static const S16 LL_defaultNorm[MaxLL+1] = { 4, 3, 2, 2, 2, 2, 2, 2,
	2, 2, 2, 2, 2, 1, 1, 1,
	2, 2, 2, 2, 2, 2, 2, 2,
	2, 3, 2, 1, 1, 1, 1, 1,
	-1,-1,-1,-1 };
	#define LL_DEFAULTNORMLOG 6 /* for static allocation */
	static const U32 LL_defaultNormLog = LL_DEFAULTNORMLOG;

	static const U32 ML_bits[MaxML+1] = { 0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	0, 0, 0, 0, 0, 0, 0, 0,
	1, 1, 1, 1, 2, 2, 3, 3,
	4, 4, 5, 7, 8, 9,10,11,
	12,13,14,15,16 };
	static const S16 ML_defaultNorm[MaxML+1] = { 1, 4, 3, 2, 2, 2, 2, 2,
	2, 1, 1, 1, 1, 1, 1, 1,
	1, 1, 1, 1, 1, 1, 1, 1,
	1, 1, 1, 1, 1, 1, 1, 1,
	1, 1, 1, 1, 1, 1, 1, 1,
	1, 1, 1, 1, 1, 1,-1,-1,
	-1,-1,-1,-1,-1 };
	#define ML_DEFAULTNORMLOG 6 /* for static allocation */
	static const U32 ML_defaultNormLog = ML_DEFAULTNORMLOG;

	static const S16 OF_defaultNorm[DefaultMaxOff+1] = { 1, 1, 1, 1, 1, 1, 2, 2,
	2, 1, 1, 1, 1, 1, 1, 1,
	1, 1, 1, 1, 1, 1, 1, 1,
	-1,-1,-1,-1,-1 };
	#define OF_DEFAULTNORMLOG 5 /* for static allocation */
	static const U32 OF_defaultNormLog = OF_DEFAULTNORMLOG;


	/-******************************************
	* Shared functions to include for inlining
	*********************************************/
	static void ZSTD_copy8(void* dst, const void* src) {
	-#ifdef __aarch64__
	+#if !defined(ZSTD_NO_INTRINSICS) && defined(__ARM_NEON)
	vst1_u8((uint8_t)dst, vld1_u8((const uint8_t)src));
	#else
	memcpy(dst, src, 8);
	#endif
	}

	#define COPY8(d,s) { ZSTD_copy8(d,s); d+=8; s+=8; }
	static void ZSTD_copy16(void* dst, const void* src) {
	-#ifdef __aarch64__
	+#if !defined(ZSTD_NO_INTRINSICS) && defined(__ARM_NEON)
	vst1q_u8((uint8_t)dst, vld1q_u8((const uint8_t)src));
	#else
	memcpy(dst, src, 16);
	#endif
	}
	#define COPY16(d,s) { ZSTD_copy16(d,s); d+=16; s+=16; }

	#define WILDCOPY_OVERLENGTH 32
	#define WILDCOPY_VECLEN 16

	typedef enum {
	ZSTD_no_overlap,
	ZSTD_overlap_src_before_dst
	/* ZSTD_overlap_dst_before_src, */
	} ZSTD_overlap_e;

	/*! ZSTD_wildcopy() :
	* Custom version of memcpy(), can over read/write up to WILDCOPY_OVERLENGTH bytes (if length==0)
	* @param ovtype controls the overlap detection
	* - ZSTD_no_overlap: The source and destination are guaranteed to be at least WILDCOPY_VECLEN bytes apart.
	* - ZSTD_overlap_src_before_dst: The src and dst may overlap, but they MUST be at least 8 bytes apart.
	* The src buffer must be before the dst buffer.
	*/
	MEM_STATIC FORCE_INLINE_ATTR
	void ZSTD_wildcopy(void* dst, const void* src, ptrdiff_t length, ZSTD_overlap_e const ovtype)
	{
	ptrdiff_t diff = (BYTE)dst - (const BYTE)src;
	const BYTE* ip = (const BYTE*)src;
	BYTE* op = (BYTE*)dst;
	BYTE* const oend = op + length;

	assert(diff >= 8 \|\| (ovtype == ZSTD_no_overlap && diff <= -WILDCOPY_VECLEN));

	if (ovtype == ZSTD_overlap_src_before_dst && diff < WILDCOPY_VECLEN) {
	/* Handle short offset copies. */
	do {
	COPY8(op, ip)
	} while (op < oend);
	} else {
	assert(diff >= WILDCOPY_VECLEN \|\| diff <= -WILDCOPY_VECLEN);
	/* Separate out the first COPY16() call because the copy length is
	* almost certain to be short, so the branches have different
	* probabilities. Since it is almost certain to be short, only do
	* one COPY16() in the first call. Then, do two calls per loop since
	* at that point it is more likely to have a high trip count.
	*/
	#ifndef __aarch64__
	do {
	COPY16(op, ip);
	}
	while (op < oend);
	#else
	COPY16(op, ip);
	if (op >= oend) return;
	do {
	COPY16(op, ip);
	COPY16(op, ip);
	}
	while (op < oend);
	#endif
	}
	}

	MEM_STATIC size_t ZSTD_limitCopy(void* dst, size_t dstCapacity, const void* src, size_t srcSize)
	{
	size_t const length = MIN(dstCapacity, srcSize);
	if (length > 0) {
	memcpy(dst, src, length);
	}
	return length;
	}

	/* define "workspace is too large" as this number of times larger than needed */
	#define ZSTD_WORKSPACETOOLARGE_FACTOR 3

	/* when workspace is continuously too large
	* during at least this number of times,
	* context's memory usage is considered wasteful,
	* because it's sized to handle a worst case scenario which rarely happens.
	* In which case, resize it down to free some memory */
	#define ZSTD_WORKSPACETOOLARGE_MAXDURATION 128


	/-******************************************
	* Private declarations
	*********************************************/
	typedef struct seqDef_s {
	U32 offset;
	U16 litLength;
	U16 matchLength;
	} seqDef;

	typedef struct {
	seqDef* sequencesStart;
	seqDef* sequences;
	BYTE* litStart;
	BYTE* lit;
	BYTE* llCode;
	BYTE* mlCode;
	BYTE* ofCode;
	size_t maxNbSeq;
	size_t maxNbLit;
	U32 longLengthID; /* 0 == no longLength; 1 == Lit.longLength; 2 == Match.longLength; */
	U32 longLengthPos;
	} seqStore_t;

	typedef struct {
	U32 litLength;
	U32 matchLength;
	} ZSTD_sequenceLength;

	/**
	* Returns the ZSTD_sequenceLength for the given sequences. It handles the decoding of long sequences
	* indicated by longLengthPos and longLengthID, and adds MINMATCH back to matchLength.
	*/
	MEM_STATIC ZSTD_sequenceLength ZSTD_getSequenceLength(seqStore_t const* seqStore, seqDef const* seq)
	{
	ZSTD_sequenceLength seqLen;
	seqLen.litLength = seq->litLength;
	seqLen.matchLength = seq->matchLength + MINMATCH;
	if (seqStore->longLengthPos == (U32)(seq - seqStore->sequencesStart)) {
	if (seqStore->longLengthID == 1) {
	seqLen.litLength += 0xFFFF;
	}
	if (seqStore->longLengthID == 2) {
	seqLen.matchLength += 0xFFFF;
	}
	}
	return seqLen;
	}

	/**
	* Contains the compressed frame size and an upper-bound for the decompressed frame size.
	* Note: before using `compressedSize`, check for errors using ZSTD_isError().
	* similarly, before using `decompressedBound`, check for errors using:
	* `decompressedBound != ZSTD_CONTENTSIZE_ERROR`
	*/
	typedef struct {
	size_t compressedSize;
	unsigned long long decompressedBound;
	} ZSTD_frameSizeInfo; /* decompress & legacy */

	const seqStore_t* ZSTD_getSeqStore(const ZSTD_CCtx* ctx); /* compress & dictBuilder */
	void ZSTD_seqToCodes(const seqStore_t* seqStorePtr); /* compress, dictBuilder, decodeCorpus (shouldn't get its definition from here) */

	/* custom memory allocation functions */
	void* ZSTD_malloc(size_t size, ZSTD_customMem customMem);
	void* ZSTD_calloc(size_t size, ZSTD_customMem customMem);
	void ZSTD_free(void* ptr, ZSTD_customMem customMem);


	MEM_STATIC U32 ZSTD_highbit32(U32 val) /* compress, dictBuilder, decodeCorpus */
	{
	assert(val != 0);
	{
	# if defined(_MSC_VER) /* Visual */
	unsigned long r=0;
	return _BitScanReverse(&r, val) ? (unsigned)r : 0;
	# elif defined(__GNUC__) && (__GNUC__ >= 3) /* GCC Intrinsic */
	return __builtin_clz (val) ^ 31;
	# elif defined(__ICCARM__) /* IAR Intrinsic */
	return 31 - __CLZ(val);
	# else /* Software version */
	static const U32 DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29, 11, 14, 16, 18, 22, 25, 3, 30, 8, 12, 20, 28, 15, 17, 24, 7, 19, 27, 23, 6, 26, 5, 4, 31 };
	U32 v = val;
	v \|= v >> 1;
	v \|= v >> 2;
	v \|= v >> 4;
	v \|= v >> 8;
	v \|= v >> 16;
	return DeBruijnClz[(v * 0x07C4ACDDU) >> 27];
	# endif
	}
	}


	/* ZSTD_invalidateRepCodes() :
	* ensures next compression will not use repcodes from previous block.
	* Note : only works with regular variant;
	* do not use with extDict variant ! */
	void ZSTD_invalidateRepCodes(ZSTD_CCtx* cctx); /* zstdmt, adaptive_compression (shouldn't get this definition from here) */


	typedef struct {
	blockType_e blockType;
	U32 lastBlock;
	U32 origSize;
	} blockProperties_t; /* declared here for decompress and fullbench */

	/*! ZSTD_getcBlockSize() :
	* Provides the size of compressed block from block header `src` */
	/* Used by: decompress, fullbench (does not get its definition from here) */
	size_t ZSTD_getcBlockSize(const void* src, size_t srcSize,
	blockProperties_t* bpPtr);

	/*! ZSTD_decodeSeqHeaders() :
	* decode sequence header from src */
	/* Used by: decompress, fullbench (does not get its definition from here) */
	size_t ZSTD_decodeSeqHeaders(ZSTD_DCtx* dctx, int* nbSeqPtr,
	const void* src, size_t srcSize);


	#if defined (__cplusplus)
	}
	#endif

	#endif /* ZSTD_CCOMMON_H_MODULE */
	/** ended inlining zstd_internal.h **/
	/** start inlining pool.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef POOL_H
	#define POOL_H

	#if defined (__cplusplus)
	extern "C" {
	#endif


	#include <stddef.h> /* size_t */
	#define ZSTD_STATIC_LINKING_ONLY /* ZSTD_customMem */
	/** skipping file: ../zstd.h **/

	typedef struct POOL_ctx_s POOL_ctx;

	/*! POOL_create() :
	* Create a thread pool with at most `numThreads` threads.
	* `numThreads` must be at least 1.
	* The maximum number of queued jobs before blocking is `queueSize`.
	* @return : POOL_ctx pointer on success, else NULL.
	*/
	POOL_ctx* POOL_create(size_t numThreads, size_t queueSize);

	POOL_ctx* POOL_create_advanced(size_t numThreads, size_t queueSize,
	ZSTD_customMem customMem);

	/*! POOL_free() :
	* Free a thread pool returned by POOL_create().
	*/
	void POOL_free(POOL_ctx* ctx);

	/*! POOL_resize() :
	* Expands or shrinks pool's number of threads.
	* This is more efficient than releasing + creating a new context,
	* since it tries to preserve and re-use existing threads.
	* `numThreads` must be at least 1.
	* @return : 0 when resize was successful,
	* !0 (typically 1) if there is an error.
	* note : only numThreads can be resized, queueSize remains unchanged.
	*/
	int POOL_resize(POOL_ctx* ctx, size_t numThreads);

	/*! POOL_sizeof() :
	* @return threadpool memory usage
	* note : compatible with NULL (returns 0 in this case)
	*/
	size_t POOL_sizeof(POOL_ctx* ctx);

	/*! POOL_function :
	* The function type that can be added to a thread pool.
	*/
	typedef void (POOL_function)(void);

	/*! POOL_add() :
	* Add the job `function(opaque)` to the thread pool. `ctx` must be valid.
	* Possibly blocks until there is room in the queue.
	* Note : The function may be executed asynchronously,
	* therefore, `opaque` must live until function has been completed.
	*/
	void POOL_add(POOL_ctx* ctx, POOL_function function, void* opaque);


	/*! POOL_tryAdd() :
	* Add the job `function(opaque)` to thread pool _if_ a worker is available.
	* Returns immediately even if not (does not block).
	* @return : 1 if successful, 0 if not.
	*/
	int POOL_tryAdd(POOL_ctx* ctx, POOL_function function, void* opaque);


	#if defined (__cplusplus)
	}
	#endif

	#endif
	/** ended inlining pool.h **/

	/* ====== Compiler specifics ====== */
	#if defined(_MSC_VER)
	# pragma warning(disable : 4204) /* disable: C4204: non-constant aggregate initializer */
	#endif


	#ifdef ZSTD_MULTITHREAD

	/** start inlining threading.h **/
	/**
	* Copyright (c) 2016 Tino Reichardt
	* All rights reserved.
	*
	* You can contact the author at:
	* - zstdmt source repository: https://github.com/mcmilk/zstdmt
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef THREADING_H_938743
	#define THREADING_H_938743

	/** skipping file: debug.h **/

	#if defined (__cplusplus)
	extern "C" {
	#endif

	#if defined(ZSTD_MULTITHREAD) && defined(_WIN32)

	/**
	* Windows minimalist Pthread Wrapper, based on :
	* http://www.cse.wustl.edu/~schmidt/win32-cv-1.html
	*/
	#ifdef WINVER
	# undef WINVER
	#endif
	#define WINVER 0x0600

	#ifdef _WIN32_WINNT
	# undef _WIN32_WINNT
	#endif
	#define _WIN32_WINNT 0x0600

	#ifndef WIN32_LEAN_AND_MEAN
	# define WIN32_LEAN_AND_MEAN
	#endif

	#undef ERROR /* reported already defined on VS 2015 (Rich Geldreich) */
	#include <windows.h>
	#undef ERROR
	#define ERROR(name) ZSTD_ERROR(name)


	/* mutex */
	#define ZSTD_pthread_mutex_t CRITICAL_SECTION
	#define ZSTD_pthread_mutex_init(a, b) ((void)(b), InitializeCriticalSection((a)), 0)
	#define ZSTD_pthread_mutex_destroy(a) DeleteCriticalSection((a))
	#define ZSTD_pthread_mutex_lock(a) EnterCriticalSection((a))
	#define ZSTD_pthread_mutex_unlock(a) LeaveCriticalSection((a))

	/* condition variable */
	#define ZSTD_pthread_cond_t CONDITION_VARIABLE
	#define ZSTD_pthread_cond_init(a, b) ((void)(b), InitializeConditionVariable((a)), 0)
	#define ZSTD_pthread_cond_destroy(a) ((void)(a))
	#define ZSTD_pthread_cond_wait(a, b) SleepConditionVariableCS((a), (b), INFINITE)
	#define ZSTD_pthread_cond_signal(a) WakeConditionVariable((a))
	#define ZSTD_pthread_cond_broadcast(a) WakeAllConditionVariable((a))

	/* ZSTD_pthread_create() and ZSTD_pthread_join() */
	typedef struct {
	HANDLE handle;
	void* (start_routine)(void);
	void* arg;
	} ZSTD_pthread_t;

	int ZSTD_pthread_create(ZSTD_pthread_t* thread, const void* unused,
	void* (start_routine) (void), void* arg);

	int ZSTD_pthread_join(ZSTD_pthread_t thread, void** value_ptr);

	/**
	* add here more wrappers as required
	*/


	#elif defined(ZSTD_MULTITHREAD) /* posix assumed ; need a better detection method */
	/* === POSIX Systems === */
	# include <pthread.h>

	#if DEBUGLEVEL < 1

	#define ZSTD_pthread_mutex_t pthread_mutex_t
	#define ZSTD_pthread_mutex_init(a, b) pthread_mutex_init((a), (b))
	#define ZSTD_pthread_mutex_destroy(a) pthread_mutex_destroy((a))
	#define ZSTD_pthread_mutex_lock(a) pthread_mutex_lock((a))
	#define ZSTD_pthread_mutex_unlock(a) pthread_mutex_unlock((a))

	#define ZSTD_pthread_cond_t pthread_cond_t
	#define ZSTD_pthread_cond_init(a, b) pthread_cond_init((a), (b))
	#define ZSTD_pthread_cond_destroy(a) pthread_cond_destroy((a))
	#define ZSTD_pthread_cond_wait(a, b) pthread_cond_wait((a), (b))
	#define ZSTD_pthread_cond_signal(a) pthread_cond_signal((a))
	#define ZSTD_pthread_cond_broadcast(a) pthread_cond_broadcast((a))

	#define ZSTD_pthread_t pthread_t
	#define ZSTD_pthread_create(a, b, c, d) pthread_create((a), (b), (c), (d))
	#define ZSTD_pthread_join(a, b) pthread_join((a),(b))

	#else /* DEBUGLEVEL >= 1 */

	/* Debug implementation of threading.
	* In this implementation we use pointers for mutexes and condition variables.
	* This way, if we forget to init/destroy them the program will crash or ASAN
	* will report leaks.
	*/

	#define ZSTD_pthread_mutex_t pthread_mutex_t*
	int ZSTD_pthread_mutex_init(ZSTD_pthread_mutex_t* mutex, pthread_mutexattr_t const* attr);
	int ZSTD_pthread_mutex_destroy(ZSTD_pthread_mutex_t* mutex);
	#define ZSTD_pthread_mutex_lock(a) pthread_mutex_lock(*(a))
	#define ZSTD_pthread_mutex_unlock(a) pthread_mutex_unlock(*(a))

	#define ZSTD_pthread_cond_t pthread_cond_t*
	int ZSTD_pthread_cond_init(ZSTD_pthread_cond_t* cond, pthread_condattr_t const* attr);
	int ZSTD_pthread_cond_destroy(ZSTD_pthread_cond_t* cond);
	#define ZSTD_pthread_cond_wait(a, b) pthread_cond_wait((a), (b))
	#define ZSTD_pthread_cond_signal(a) pthread_cond_signal(*(a))
	#define ZSTD_pthread_cond_broadcast(a) pthread_cond_broadcast(*(a))

	#define ZSTD_pthread_t pthread_t
	#define ZSTD_pthread_create(a, b, c, d) pthread_create((a), (b), (c), (d))
	#define ZSTD_pthread_join(a, b) pthread_join((a),(b))

	#endif

	#else /* ZSTD_MULTITHREAD not defined */
	/* No multithreading support */

	typedef int ZSTD_pthread_mutex_t;
	#define ZSTD_pthread_mutex_init(a, b) ((void)(a), (void)(b), 0)
	#define ZSTD_pthread_mutex_destroy(a) ((void)(a))
	#define ZSTD_pthread_mutex_lock(a) ((void)(a))
	#define ZSTD_pthread_mutex_unlock(a) ((void)(a))

	typedef int ZSTD_pthread_cond_t;
	#define ZSTD_pthread_cond_init(a, b) ((void)(a), (void)(b), 0)
	#define ZSTD_pthread_cond_destroy(a) ((void)(a))
	#define ZSTD_pthread_cond_wait(a, b) ((void)(a), (void)(b))
	#define ZSTD_pthread_cond_signal(a) ((void)(a))
	#define ZSTD_pthread_cond_broadcast(a) ((void)(a))

	/* do not use ZSTD_pthread_t */

	#endif /* ZSTD_MULTITHREAD */

	#if defined (__cplusplus)
	}
	#endif

	#endif /* THREADING_H_938743 */
	/** ended inlining threading.h **/

	/* A job is a function and an opaque argument */
	typedef struct POOL_job_s {
	POOL_function function;
	void *opaque;
	} POOL_job;

	struct POOL_ctx_s {
	ZSTD_customMem customMem;
	/* Keep track of the threads */
	ZSTD_pthread_t* threads;
	size_t threadCapacity;
	size_t threadLimit;

	/* The queue is a circular buffer */
	POOL_job *queue;
	size_t queueHead;
	size_t queueTail;
	size_t queueSize;

	/* The number of threads working on jobs */
	size_t numThreadsBusy;
	/* Indicates if the queue is empty */
	int queueEmpty;

	/* The mutex protects the queue */
	ZSTD_pthread_mutex_t queueMutex;
	/* Condition variable for pushers to wait on when the queue is full */
	ZSTD_pthread_cond_t queuePushCond;
	/* Condition variables for poppers to wait on when the queue is empty */
	ZSTD_pthread_cond_t queuePopCond;
	/* Indicates if the queue is shutting down */
	int shutdown;
	};

	/* POOL_thread() :
	* Work thread for the thread pool.
	* Waits for jobs and executes them.
	* @returns : NULL on failure else non-null.
	*/
	static void* POOL_thread(void* opaque) {
	POOL_ctx* const ctx = (POOL_ctx*)opaque;
	if (!ctx) { return NULL; }
	for (;;) {
	/* Lock the mutex and wait for a non-empty queue or until shutdown */
	ZSTD_pthread_mutex_lock(&ctx->queueMutex);

	while ( ctx->queueEmpty
	\|\| (ctx->numThreadsBusy >= ctx->threadLimit) ) {
	if (ctx->shutdown) {
	/* even if !queueEmpty, (possible if numThreadsBusy >= threadLimit),
	* a few threads will be shutdown while !queueEmpty,
	* but enough threads will remain active to finish the queue */
	ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
	return opaque;
	}
	ZSTD_pthread_cond_wait(&ctx->queuePopCond, &ctx->queueMutex);
	}
	/* Pop a job off the queue */
	{ POOL_job const job = ctx->queue[ctx->queueHead];
	ctx->queueHead = (ctx->queueHead + 1) % ctx->queueSize;
	ctx->numThreadsBusy++;
	ctx->queueEmpty = ctx->queueHead == ctx->queueTail;
	/* Unlock the mutex, signal a pusher, and run the job */
	ZSTD_pthread_cond_signal(&ctx->queuePushCond);
	ZSTD_pthread_mutex_unlock(&ctx->queueMutex);

	job.function(job.opaque);

	/* If the intended queue size was 0, signal after finishing job */
	ZSTD_pthread_mutex_lock(&ctx->queueMutex);
	ctx->numThreadsBusy--;
	if (ctx->queueSize == 1) {
	ZSTD_pthread_cond_signal(&ctx->queuePushCond);
	}
	ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
	}
	} /* for (;;) */
	assert(0); /* Unreachable */
	}

	POOL_ctx* POOL_create(size_t numThreads, size_t queueSize) {
	return POOL_create_advanced(numThreads, queueSize, ZSTD_defaultCMem);
	}

	POOL_ctx* POOL_create_advanced(size_t numThreads, size_t queueSize,
	ZSTD_customMem customMem) {
	POOL_ctx* ctx;
	/* Check parameters */
	if (!numThreads) { return NULL; }
	/* Allocate the context and zero initialize */
	ctx = (POOL_ctx*)ZSTD_calloc(sizeof(POOL_ctx), customMem);
	if (!ctx) { return NULL; }
	/* Initialize the job queue.
	* It needs one extra space since one space is wasted to differentiate
	* empty and full queues.
	*/
	ctx->queueSize = queueSize + 1;
	ctx->queue = (POOL_job)ZSTD_malloc(ctx->queueSize sizeof(POOL_job), customMem);
	ctx->queueHead = 0;
	ctx->queueTail = 0;
	ctx->numThreadsBusy = 0;
	ctx->queueEmpty = 1;
	{
	int error = 0;
	error \|= ZSTD_pthread_mutex_init(&ctx->queueMutex, NULL);
	error \|= ZSTD_pthread_cond_init(&ctx->queuePushCond, NULL);
	error \|= ZSTD_pthread_cond_init(&ctx->queuePopCond, NULL);
	if (error) { POOL_free(ctx); return NULL; }
	}
	ctx->shutdown = 0;
	/* Allocate space for the thread handles */
	ctx->threads = (ZSTD_pthread_t)ZSTD_malloc(numThreads sizeof(ZSTD_pthread_t), customMem);
	ctx->threadCapacity = 0;
	ctx->customMem = customMem;
	/* Check for errors */
	if (!ctx->threads \|\| !ctx->queue) { POOL_free(ctx); return NULL; }
	/* Initialize the threads */
	{ size_t i;
	for (i = 0; i < numThreads; ++i) {
	if (ZSTD_pthread_create(&ctx->threads[i], NULL, &POOL_thread, ctx)) {
	ctx->threadCapacity = i;
	POOL_free(ctx);
	return NULL;
	} }
	ctx->threadCapacity = numThreads;
	ctx->threadLimit = numThreads;
	}
	return ctx;
	}

	/*! POOL_join() :
	Shutdown the queue, wake any sleeping threads, and join all of the threads.
	*/
	static void POOL_join(POOL_ctx* ctx) {
	/* Shut down the queue */
	ZSTD_pthread_mutex_lock(&ctx->queueMutex);
	ctx->shutdown = 1;
	ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
	/* Wake up sleeping threads */
	ZSTD_pthread_cond_broadcast(&ctx->queuePushCond);
	ZSTD_pthread_cond_broadcast(&ctx->queuePopCond);
	/* Join all of the threads */
	{ size_t i;
	for (i = 0; i < ctx->threadCapacity; ++i) {
	ZSTD_pthread_join(ctx->threads[i], NULL); /* note : could fail */
	} }
	}

	void POOL_free(POOL_ctx *ctx) {
	if (!ctx) { return; }
	POOL_join(ctx);
	ZSTD_pthread_mutex_destroy(&ctx->queueMutex);
	ZSTD_pthread_cond_destroy(&ctx->queuePushCond);
	ZSTD_pthread_cond_destroy(&ctx->queuePopCond);
	ZSTD_free(ctx->queue, ctx->customMem);
	ZSTD_free(ctx->threads, ctx->customMem);
	ZSTD_free(ctx, ctx->customMem);
	}



	size_t POOL_sizeof(POOL_ctx *ctx) {
	if (ctx==NULL) return 0; /* supports sizeof NULL */
	return sizeof(*ctx)
	+ ctx->queueSize * sizeof(POOL_job)
	+ ctx->threadCapacity * sizeof(ZSTD_pthread_t);
	}


	/* @return : 0 on success, 1 on error */
	static int POOL_resize_internal(POOL_ctx* ctx, size_t numThreads)
	{
	if (numThreads <= ctx->threadCapacity) {
	if (!numThreads) return 1;
	ctx->threadLimit = numThreads;
	return 0;
	}
	/* numThreads > threadCapacity */
	{ ZSTD_pthread_t* const threadPool = (ZSTD_pthread_t)ZSTD_malloc(numThreads sizeof(ZSTD_pthread_t), ctx->customMem);
	if (!threadPool) return 1;
	/* replace existing thread pool */
	memcpy(threadPool, ctx->threads, ctx->threadCapacity * sizeof(*threadPool));
	ZSTD_free(ctx->threads, ctx->customMem);
	ctx->threads = threadPool;
	/* Initialize additional threads */
	{ size_t threadId;
	for (threadId = ctx->threadCapacity; threadId < numThreads; ++threadId) {
	if (ZSTD_pthread_create(&threadPool[threadId], NULL, &POOL_thread, ctx)) {
	ctx->threadCapacity = threadId;
	return 1;
	} }
	} }
	/* successfully expanded */
	ctx->threadCapacity = numThreads;
	ctx->threadLimit = numThreads;
	return 0;
	}

	/* @return : 0 on success, 1 on error */
	int POOL_resize(POOL_ctx* ctx, size_t numThreads)
	{
	int result;
	if (ctx==NULL) return 1;
	ZSTD_pthread_mutex_lock(&ctx->queueMutex);
	result = POOL_resize_internal(ctx, numThreads);
	ZSTD_pthread_cond_broadcast(&ctx->queuePopCond);
	ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
	return result;
	}

	/**
	* Returns 1 if the queue is full and 0 otherwise.
	*
	* When queueSize is 1 (pool was created with an intended queueSize of 0),
	* then a queue is empty if there is a thread free _and_ no job is waiting.
	*/
	static int isQueueFull(POOL_ctx const* ctx) {
	if (ctx->queueSize > 1) {
	return ctx->queueHead == ((ctx->queueTail + 1) % ctx->queueSize);
	} else {
	return (ctx->numThreadsBusy == ctx->threadLimit) \|\|
	!ctx->queueEmpty;
	}
	}


	static void POOL_add_internal(POOL_ctx* ctx, POOL_function function, void *opaque)
	{
	POOL_job const job = {function, opaque};
	assert(ctx != NULL);
	if (ctx->shutdown) return;

	ctx->queueEmpty = 0;
	ctx->queue[ctx->queueTail] = job;
	ctx->queueTail = (ctx->queueTail + 1) % ctx->queueSize;
	ZSTD_pthread_cond_signal(&ctx->queuePopCond);
	}

	void POOL_add(POOL_ctx* ctx, POOL_function function, void* opaque)
	{
	assert(ctx != NULL);
	ZSTD_pthread_mutex_lock(&ctx->queueMutex);
	/* Wait until there is space in the queue for the new job */
	while (isQueueFull(ctx) && (!ctx->shutdown)) {
	ZSTD_pthread_cond_wait(&ctx->queuePushCond, &ctx->queueMutex);
	}
	POOL_add_internal(ctx, function, opaque);
	ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
	}


	int POOL_tryAdd(POOL_ctx* ctx, POOL_function function, void* opaque)
	{
	assert(ctx != NULL);
	ZSTD_pthread_mutex_lock(&ctx->queueMutex);
	if (isQueueFull(ctx)) {
	ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
	return 0;
	}
	POOL_add_internal(ctx, function, opaque);
	ZSTD_pthread_mutex_unlock(&ctx->queueMutex);
	return 1;
	}


	#else /* ZSTD_MULTITHREAD not defined */

	/* ========================== */
	/* No multi-threading support */
	/* ========================== */


	/* We don't need any data, but if it is empty, malloc() might return NULL. */
	struct POOL_ctx_s {
	int dummy;
	};
	static POOL_ctx g_ctx;

	POOL_ctx* POOL_create(size_t numThreads, size_t queueSize) {
	return POOL_create_advanced(numThreads, queueSize, ZSTD_defaultCMem);
	}

	POOL_ctx* POOL_create_advanced(size_t numThreads, size_t queueSize, ZSTD_customMem customMem) {
	(void)numThreads;
	(void)queueSize;
	(void)customMem;
	return &g_ctx;
	}

	void POOL_free(POOL_ctx* ctx) {
	assert(!ctx \|\| ctx == &g_ctx);
	(void)ctx;
	}

	int POOL_resize(POOL_ctx* ctx, size_t numThreads) {
	(void)ctx; (void)numThreads;
	return 0;
	}

	void POOL_add(POOL_ctx* ctx, POOL_function function, void* opaque) {
	(void)ctx;
	function(opaque);
	}

	int POOL_tryAdd(POOL_ctx* ctx, POOL_function function, void* opaque) {
	(void)ctx;
	function(opaque);
	return 1;
	}

	size_t POOL_sizeof(POOL_ctx* ctx) {
	if (ctx==NULL) return 0; /* supports sizeof NULL */
	assert(ctx == &g_ctx);
	return sizeof(*ctx);
	}

	#endif /* ZSTD_MULTITHREAD */
	/** ended inlining common/pool.c **/
	/** start inlining common/zstd_common.c **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/



	/-************************************
	* Dependencies
	***************************************/
	#include <stdlib.h> /* malloc, calloc, free */
	#include <string.h> /* memset */
	/** skipping file: error_private.h **/
	/** skipping file: zstd_internal.h **/


	/-***************************************
	* Version
	******************************************/
	unsigned ZSTD_versionNumber(void) { return ZSTD_VERSION_NUMBER; }

	const char* ZSTD_versionString(void) { return ZSTD_VERSION_STRING; }


	/-***************************************
	* ZSTD Error Management
	******************************************/
	#undef ZSTD_isError /* defined within zstd_internal.h */
	/*! ZSTD_isError() :
	* tells if a return value is an error code
	* symbol is required for external callers */
	unsigned ZSTD_isError(size_t code) { return ERR_isError(code); }

	/*! ZSTD_getErrorName() :
	* provides error code string from function result (useful for debugging) */
	const char* ZSTD_getErrorName(size_t code) { return ERR_getErrorName(code); }

	/*! ZSTD_getError() :
	* convert a `size_t` function result into a proper ZSTD_errorCode enum */
	ZSTD_ErrorCode ZSTD_getErrorCode(size_t code) { return ERR_getErrorCode(code); }

	/*! ZSTD_getErrorString() :
	* provides error code string from enum */
	const char* ZSTD_getErrorString(ZSTD_ErrorCode code) { return ERR_getErrorString(code); }



	/=*************************************************************
	* Custom allocator
	****************************************************************/
	void* ZSTD_malloc(size_t size, ZSTD_customMem customMem)
	{
	if (customMem.customAlloc)
	return customMem.customAlloc(customMem.opaque, size);
	return malloc(size);
	}

	void* ZSTD_calloc(size_t size, ZSTD_customMem customMem)
	{
	if (customMem.customAlloc) {
	/* calloc implemented as malloc+memset;
	* not as efficient as calloc, but next best guess for custom malloc */
	void* const ptr = customMem.customAlloc(customMem.opaque, size);
	memset(ptr, 0, size);
	return ptr;
	}
	return calloc(1, size);
	}

	void ZSTD_free(void* ptr, ZSTD_customMem customMem)
	{
	if (ptr!=NULL) {
	if (customMem.customFree)
	customMem.customFree(customMem.opaque, ptr);
	else
	free(ptr);
	}
	}
	/** ended inlining common/zstd_common.c **/

	/** start inlining compress/fse_compress.c **/
	/* ******************************************************************
	* FSE : Finite State Entropy encoder
	* Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
	*
	* You can contact the author at :
	* - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
	* - Public forum : https://groups.google.com/forum/#!forum/lz4c
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	****************************************************************** */

	/* **************************************************************
	* Includes
	****************************************************************/
	#include <stdlib.h> /* malloc, free, qsort */
	#include <string.h> /* memcpy, memset */
	/** skipping file: ../common/compiler.h **/
	/** skipping file: ../common/mem.h **/
	/** skipping file: ../common/debug.h **/
	/** start inlining hist.h **/
	/* ******************************************************************
	* hist : Histogram functions
	* part of Finite State Entropy project
	* Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
	*
	* You can contact the author at :
	* - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
	* - Public forum : https://groups.google.com/forum/#!forum/lz4c
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	****************************************************************** */

	/* --- dependencies --- */
	#include <stddef.h> /* size_t */


	/* --- simple histogram functions --- */

	/*! HIST_count():
	* Provides the precise count of each byte within a table 'count'.
	* 'count' is a table of unsigned int, of minimum size (*maxSymbolValuePtr+1).
	* Updates *maxSymbolValuePtr with actual largest symbol value detected.
	* @return : count of the most frequent symbol (which isn't identified).
	* or an error code, which can be tested using HIST_isError().
	* note : if return == srcSize, there is only one symbol.
	*/
	size_t HIST_count(unsigned* count, unsigned* maxSymbolValuePtr,
	const void* src, size_t srcSize);

	unsigned HIST_isError(size_t code); /*< tells if a return value is an error code /


	/* --- advanced histogram functions --- */

	#define HIST_WKSP_SIZE_U32 1024
	#define HIST_WKSP_SIZE (HIST_WKSP_SIZE_U32 * sizeof(unsigned))
	/** HIST_count_wksp() :
	* Same as HIST_count(), but using an externally provided scratch buffer.
	* Benefit is this function will use very little stack space.
	* `workSpace` is a writable buffer which must be 4-bytes aligned,
	* `workSpaceSize` must be >= HIST_WKSP_SIZE
	*/
	size_t HIST_count_wksp(unsigned* count, unsigned* maxSymbolValuePtr,
	const void* src, size_t srcSize,
	void* workSpace, size_t workSpaceSize);

	/** HIST_countFast() :
	* same as HIST_count(), but blindly trusts that all byte values within src are <= *maxSymbolValuePtr.
	* This function is unsafe, and will segfault if any value within `src` is `> *maxSymbolValuePtr`
	*/
	size_t HIST_countFast(unsigned* count, unsigned* maxSymbolValuePtr,
	const void* src, size_t srcSize);

	/** HIST_countFast_wksp() :
	* Same as HIST_countFast(), but using an externally provided scratch buffer.
	* `workSpace` is a writable buffer which must be 4-bytes aligned,
	* `workSpaceSize` must be >= HIST_WKSP_SIZE
	*/
	size_t HIST_countFast_wksp(unsigned* count, unsigned* maxSymbolValuePtr,
	const void* src, size_t srcSize,
	void* workSpace, size_t workSpaceSize);

	/*! HIST_count_simple() :
	* Same as HIST_countFast(), this function is unsafe,
	* and will segfault if any value within `src` is `> *maxSymbolValuePtr`.
	* It is also a bit slower for large inputs.
	* However, it does not need any additional memory (not even on stack).
	* @return : count of the most frequent symbol.
	* Note this function doesn't produce any error (i.e. it must succeed).
	*/
	unsigned HIST_count_simple(unsigned* count, unsigned* maxSymbolValuePtr,
	const void* src, size_t srcSize);
	/** ended inlining hist.h **/
	/** skipping file: ../common/bitstream.h **/
	#define FSE_STATIC_LINKING_ONLY
	/** skipping file: ../common/fse.h **/
	/** skipping file: ../common/error_private.h **/


	/* **************************************************************
	* Error Management
	****************************************************************/
	#define FSE_isError ERR_isError


	/* **************************************************************
	* Templates
	****************************************************************/
	/*
	designed to be included
	for type-specific functions (template emulation in C)
	Objective is to write these functions only once, for improved maintenance
	*/

	/* safety checks */
	#ifndef FSE_FUNCTION_EXTENSION
	# error "FSE_FUNCTION_EXTENSION must be defined"
	#endif
	#ifndef FSE_FUNCTION_TYPE
	# error "FSE_FUNCTION_TYPE must be defined"
	#endif

	/* Function names */
	#define FSE_CAT(X,Y) X##Y
	#define FSE_FUNCTION_NAME(X,Y) FSE_CAT(X,Y)
	#define FSE_TYPE_NAME(X,Y) FSE_CAT(X,Y)


	/* Function templates */

	/* FSE_buildCTable_wksp() :
	* Same as FSE_buildCTable(), but using an externally allocated scratch buffer (`workSpace`).
	* wkspSize should be sized to handle worst case situation, which is `1<<max_tableLog * sizeof(FSE_FUNCTION_TYPE)`
	* workSpace must also be properly aligned with FSE_FUNCTION_TYPE requirements
	*/
	size_t FSE_buildCTable_wksp(FSE_CTable* ct,
	const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog,
	void* workSpace, size_t wkspSize)
	{
	U32 const tableSize = 1 << tableLog;
	U32 const tableMask = tableSize - 1;
	void* const ptr = ct;
	U16* const tableU16 = ( (U16*) ptr) + 2;
	void* const FSCT = ((U32)ptr) + 1 / header */ + (tableLog ? tableSize>>1 : 1) ;
	FSE_symbolCompressionTransform* const symbolTT = (FSE_symbolCompressionTransform*) (FSCT);
	U32 const step = FSE_TABLESTEP(tableSize);
	U32 cumul[FSE_MAX_SYMBOL_VALUE+2];

	FSE_FUNCTION_TYPE* const tableSymbol = (FSE_FUNCTION_TYPE*)workSpace;
	U32 highThreshold = tableSize-1;

	/* CTable header */
	if (((size_t)1 << tableLog) * sizeof(FSE_FUNCTION_TYPE) > wkspSize) return ERROR(tableLog_tooLarge);
	tableU16[-2] = (U16) tableLog;
	tableU16[-1] = (U16) maxSymbolValue;
	assert(tableLog < 16); /* required for threshold strategy to work */

	/* For explanations on how to distribute symbol values over the table :
	* http://fastcompression.blogspot.fr/2014/02/fse-distributing-symbol-values.html */

	#ifdef __clang_analyzer__
	memset(tableSymbol, 0, sizeof(tableSymbol) tableSize); /* useless initialization, just to keep scan-build happy */
	#endif

	/* symbol start positions */
	{ U32 u;
	cumul[0] = 0;
	for (u=1; u <= maxSymbolValue+1; u++) {
	if (normalizedCounter[u-1]==-1) { /* Low proba symbol */
	cumul[u] = cumul[u-1] + 1;
	tableSymbol[highThreshold--] = (FSE_FUNCTION_TYPE)(u-1);
	} else {
	cumul[u] = cumul[u-1] + normalizedCounter[u-1];
	} }
	cumul[maxSymbolValue+1] = tableSize+1;
	}

	/* Spread symbols */
	{ U32 position = 0;
	U32 symbol;
	for (symbol=0; symbol<=maxSymbolValue; symbol++) {
	int nbOccurrences;
	int const freq = normalizedCounter[symbol];
	for (nbOccurrences=0; nbOccurrences<freq; nbOccurrences++) {
	tableSymbol[position] = (FSE_FUNCTION_TYPE)symbol;
	position = (position + step) & tableMask;
	while (position > highThreshold)
	position = (position + step) & tableMask; /* Low proba area */
	} }

	assert(position==0); /* Must have initialized all positions */
	}

	/* Build table */
	{ U32 u; for (u=0; u<tableSize; u++) {
	FSE_FUNCTION_TYPE s = tableSymbol[u]; /* note : static analyzer may not understand tableSymbol is properly initialized */
	tableU16[cumul[s]++] = (U16) (tableSize+u); /* TableU16 : sorted by symbol order; gives next state value */
	} }

	/* Build Symbol Transformation Table */
	{ unsigned total = 0;
	unsigned s;
	for (s=0; s<=maxSymbolValue; s++) {
	switch (normalizedCounter[s])
	{
	case 0:
	/* filling nonetheless, for compatibility with FSE_getMaxNbBits() */
	symbolTT[s].deltaNbBits = ((tableLog+1) << 16) - (1<<tableLog);
	break;

	case -1:
	case 1:
	symbolTT[s].deltaNbBits = (tableLog << 16) - (1<<tableLog);
	symbolTT[s].deltaFindState = total - 1;
	total ++;
	break;
	default :
	{
	U32 const maxBitsOut = tableLog - BIT_highbit32 (normalizedCounter[s]-1);
	U32 const minStatePlus = normalizedCounter[s] << maxBitsOut;
	symbolTT[s].deltaNbBits = (maxBitsOut << 16) - minStatePlus;
	symbolTT[s].deltaFindState = total - normalizedCounter[s];
	total += normalizedCounter[s];
	} } } }

	#if 0 /* debug : symbol costs */
	DEBUGLOG(5, "\n --- table statistics : ");
	{ U32 symbol;
	for (symbol=0; symbol<=maxSymbolValue; symbol++) {
	DEBUGLOG(5, "%3u: w=%3i, maxBits=%u, fracBits=%.2f",
	symbol, normalizedCounter[symbol],
	FSE_getMaxNbBits(symbolTT, symbol),
	(double)FSE_bitCost(symbolTT, tableLog, symbol, 8) / 256);
	}
	}
	#endif

	return 0;
	}


	size_t FSE_buildCTable(FSE_CTable* ct, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog)
	{
	FSE_FUNCTION_TYPE tableSymbol[FSE_MAX_TABLESIZE]; /* memset() is not necessary, even if static analyzer complain about it */
	return FSE_buildCTable_wksp(ct, normalizedCounter, maxSymbolValue, tableLog, tableSymbol, sizeof(tableSymbol));
	}



	#ifndef FSE_COMMONDEFS_ONLY


	/-*************************************************************
	* FSE NCount encoding
	****************************************************************/
	size_t FSE_NCountWriteBound(unsigned maxSymbolValue, unsigned tableLog)
	{
	size_t const maxHeaderSize = (((maxSymbolValue+1) * tableLog) >> 3) + 3;
	return maxSymbolValue ? maxHeaderSize : FSE_NCOUNTBOUND; /* maxSymbolValue==0 ? use default */
	}

	static size_t
	FSE_writeNCount_generic (void* header, size_t headerBufferSize,
	const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog,
	unsigned writeIsSafe)
	{
	BYTE* const ostart = (BYTE*) header;
	BYTE* out = ostart;
	BYTE* const oend = ostart + headerBufferSize;
	int nbBits;
	const int tableSize = 1 << tableLog;
	int remaining;
	int threshold;
	U32 bitStream = 0;
	int bitCount = 0;
	unsigned symbol = 0;
	unsigned const alphabetSize = maxSymbolValue + 1;
	int previousIs0 = 0;

	/* Table Size */
	bitStream += (tableLog-FSE_MIN_TABLELOG) << bitCount;
	bitCount += 4;

	/* Init */
	remaining = tableSize+1; /* +1 for extra accuracy */
	threshold = tableSize;
	nbBits = tableLog+1;

	while ((symbol < alphabetSize) && (remaining>1)) { /* stops at 1 */
	if (previousIs0) {
	unsigned start = symbol;
	while ((symbol < alphabetSize) && !normalizedCounter[symbol]) symbol++;
	if (symbol == alphabetSize) break; /* incorrect distribution */
	while (symbol >= start+24) {
	start+=24;
	bitStream += 0xFFFFU << bitCount;
	if ((!writeIsSafe) && (out > oend-2))
	return ERROR(dstSize_tooSmall); /* Buffer overflow */
	out[0] = (BYTE) bitStream;
	out[1] = (BYTE)(bitStream>>8);
	out+=2;
	bitStream>>=16;
	}
	while (symbol >= start+3) {
	start+=3;
	bitStream += 3 << bitCount;
	bitCount += 2;
	}
	bitStream += (symbol-start) << bitCount;
	bitCount += 2;
	if (bitCount>16) {
	if ((!writeIsSafe) && (out > oend - 2))
	return ERROR(dstSize_tooSmall); /* Buffer overflow */
	out[0] = (BYTE)bitStream;
	out[1] = (BYTE)(bitStream>>8);
	out += 2;
	bitStream >>= 16;
	bitCount -= 16;
	} }
	{ int count = normalizedCounter[symbol++];
	int const max = (2*threshold-1) - remaining;
	remaining -= count < 0 ? -count : count;
	count++; /* +1 for extra accuracy */
	if (count>=threshold)
	count += max; /* [0..max[ [max..threshold[ (...) [threshold+max 2threshold[ /
	bitStream += count << bitCount;
	bitCount += nbBits;
	bitCount -= (count<max);
	previousIs0 = (count==1);
	if (remaining<1) return ERROR(GENERIC);
	while (remaining<threshold) { nbBits--; threshold>>=1; }
	}
	if (bitCount>16) {
	if ((!writeIsSafe) && (out > oend - 2))
	return ERROR(dstSize_tooSmall); /* Buffer overflow */
	out[0] = (BYTE)bitStream;
	out[1] = (BYTE)(bitStream>>8);
	out += 2;
	bitStream >>= 16;
	bitCount -= 16;
	} }

	if (remaining != 1)
	return ERROR(GENERIC); /* incorrect normalized distribution */
	assert(symbol <= alphabetSize);

	/* flush remaining bitStream */
	if ((!writeIsSafe) && (out > oend - 2))
	return ERROR(dstSize_tooSmall); /* Buffer overflow */
	out[0] = (BYTE)bitStream;
	out[1] = (BYTE)(bitStream>>8);
	out+= (bitCount+7) /8;

	return (out-ostart);
	}


	size_t FSE_writeNCount (void* buffer, size_t bufferSize,
	const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog)
	{
	if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge); /* Unsupported */
	if (tableLog < FSE_MIN_TABLELOG) return ERROR(GENERIC); /* Unsupported */

	if (bufferSize < FSE_NCountWriteBound(maxSymbolValue, tableLog))
	return FSE_writeNCount_generic(buffer, bufferSize, normalizedCounter, maxSymbolValue, tableLog, 0);

	return FSE_writeNCount_generic(buffer, bufferSize, normalizedCounter, maxSymbolValue, tableLog, 1 /* write in buffer is safe */);
	}


	/-*************************************************************
	* FSE Compression Code
	****************************************************************/

	FSE_CTable* FSE_createCTable (unsigned maxSymbolValue, unsigned tableLog)
	{
	size_t size;
	if (tableLog > FSE_TABLELOG_ABSOLUTE_MAX) tableLog = FSE_TABLELOG_ABSOLUTE_MAX;
	size = FSE_CTABLE_SIZE_U32 (tableLog, maxSymbolValue) * sizeof(U32);
	return (FSE_CTable*)malloc(size);
	}

	void FSE_freeCTable (FSE_CTable* ct) { free(ct); }

	/* provides the minimum logSize to safely represent a distribution */
	static unsigned FSE_minTableLog(size_t srcSize, unsigned maxSymbolValue)
	{
	U32 minBitsSrc = BIT_highbit32((U32)(srcSize)) + 1;
	U32 minBitsSymbols = BIT_highbit32(maxSymbolValue) + 2;
	U32 minBits = minBitsSrc < minBitsSymbols ? minBitsSrc : minBitsSymbols;
	assert(srcSize > 1); /* Not supported, RLE should be used instead */
	return minBits;
	}

	unsigned FSE_optimalTableLog_internal(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue, unsigned minus)
	{
	U32 maxBitsSrc = BIT_highbit32((U32)(srcSize - 1)) - minus;
	U32 tableLog = maxTableLog;
	U32 minBits = FSE_minTableLog(srcSize, maxSymbolValue);
	assert(srcSize > 1); /* Not supported, RLE should be used instead */
	if (tableLog==0) tableLog = FSE_DEFAULT_TABLELOG;
	if (maxBitsSrc < tableLog) tableLog = maxBitsSrc; /* Accuracy can be reduced */
	if (minBits > tableLog) tableLog = minBits; /* Need a minimum to safely represent all symbol values */
	if (tableLog < FSE_MIN_TABLELOG) tableLog = FSE_MIN_TABLELOG;
	if (tableLog > FSE_MAX_TABLELOG) tableLog = FSE_MAX_TABLELOG;
	return tableLog;
	}

	unsigned FSE_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue)
	{
	return FSE_optimalTableLog_internal(maxTableLog, srcSize, maxSymbolValue, 2);
	}


	/* Secondary normalization method.
	To be used when primary method fails. */

	static size_t FSE_normalizeM2(short* norm, U32 tableLog, const unsigned* count, size_t total, U32 maxSymbolValue)
	{
	short const NOT_YET_ASSIGNED = -2;
	U32 s;
	U32 distributed = 0;
	U32 ToDistribute;

	/* Init */
	U32 const lowThreshold = (U32)(total >> tableLog);
	U32 lowOne = (U32)((total * 3) >> (tableLog + 1));

	for (s=0; s<=maxSymbolValue; s++) {
	if (count[s] == 0) {
	norm[s]=0;
	continue;
	}
	if (count[s] <= lowThreshold) {
	norm[s] = -1;
	distributed++;
	total -= count[s];
	continue;
	}
	if (count[s] <= lowOne) {
	norm[s] = 1;
	distributed++;
	total -= count[s];
	continue;
	}

	norm[s]=NOT_YET_ASSIGNED;
	}
	ToDistribute = (1 << tableLog) - distributed;

	if (ToDistribute == 0)
	return 0;

	if ((total / ToDistribute) > lowOne) {
	/* risk of rounding to zero */
	lowOne = (U32)((total * 3) / (ToDistribute * 2));
	for (s=0; s<=maxSymbolValue; s++) {
	if ((norm[s] == NOT_YET_ASSIGNED) && (count[s] <= lowOne)) {
	norm[s] = 1;
	distributed++;
	total -= count[s];
	continue;
	} }
	ToDistribute = (1 << tableLog) - distributed;
	}

	if (distributed == maxSymbolValue+1) {
	/* all values are pretty poor;
	probably incompressible data (should have already been detected);
	find max, then give all remaining points to max */
	U32 maxV = 0, maxC = 0;
	for (s=0; s<=maxSymbolValue; s++)
	if (count[s] > maxC) { maxV=s; maxC=count[s]; }
	norm[maxV] += (short)ToDistribute;
	return 0;
	}

	if (total == 0) {
	/* all of the symbols were low enough for the lowOne or lowThreshold */
	for (s=0; ToDistribute > 0; s = (s+1)%(maxSymbolValue+1))
	if (norm[s] > 0) { ToDistribute--; norm[s]++; }
	return 0;
	}

	{ U64 const vStepLog = 62 - tableLog;
	U64 const mid = (1ULL << (vStepLog-1)) - 1;
	U64 const rStep = ((((U64)1<<vStepLog) * ToDistribute) + mid) / total; /* scale on remaining */
	U64 tmpTotal = mid;
	for (s=0; s<=maxSymbolValue; s++) {
	if (norm[s]==NOT_YET_ASSIGNED) {
	U64 const end = tmpTotal + (count[s] * rStep);
	U32 const sStart = (U32)(tmpTotal >> vStepLog);
	U32 const sEnd = (U32)(end >> vStepLog);
	U32 const weight = sEnd - sStart;
	if (weight < 1)
	return ERROR(GENERIC);
	norm[s] = (short)weight;
	tmpTotal = end;
	} } }

	return 0;
	}


	size_t FSE_normalizeCount (short* normalizedCounter, unsigned tableLog,
	const unsigned* count, size_t total,
	unsigned maxSymbolValue)
	{
	/* Sanity checks */
	if (tableLog==0) tableLog = FSE_DEFAULT_TABLELOG;
	if (tableLog < FSE_MIN_TABLELOG) return ERROR(GENERIC); /* Unsupported size */
	if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge); /* Unsupported size */
	if (tableLog < FSE_minTableLog(total, maxSymbolValue)) return ERROR(GENERIC); /* Too small tableLog, compression potentially impossible */

	{ static U32 const rtbTable[] = { 0, 473195, 504333, 520860, 550000, 700000, 750000, 830000 };
	U64 const scale = 62 - tableLog;
	U64 const step = ((U64)1<<62) / total; /* <== here, one division ! */
	U64 const vStep = 1ULL<<(scale-20);
	int stillToDistribute = 1<<tableLog;
	unsigned s;
	unsigned largest=0;
	short largestP=0;
	U32 lowThreshold = (U32)(total >> tableLog);

	for (s=0; s<=maxSymbolValue; s++) {
	if (count[s] == total) return 0; /* rle special case */
	if (count[s] == 0) { normalizedCounter[s]=0; continue; }
	if (count[s] <= lowThreshold) {
	normalizedCounter[s] = -1;
	stillToDistribute--;
	} else {
	short proba = (short)((count[s]*step) >> scale);
	if (proba<8) {
	U64 restToBeat = vStep * rtbTable[proba];
	proba += (count[s]*step) - ((U64)proba<<scale) > restToBeat;
	}
	if (proba > largestP) { largestP=proba; largest=s; }
	normalizedCounter[s] = proba;
	stillToDistribute -= proba;
	} }
	if (-stillToDistribute >= (normalizedCounter[largest] >> 1)) {
	/* corner case, need another normalization method */
	size_t const errorCode = FSE_normalizeM2(normalizedCounter, tableLog, count, total, maxSymbolValue);
	if (FSE_isError(errorCode)) return errorCode;
	}
	else normalizedCounter[largest] += (short)stillToDistribute;
	}

	#if 0
	{ /* Print Table (debug) */
	U32 s;
	U32 nTotal = 0;
	for (s=0; s<=maxSymbolValue; s++)
	RAWLOG(2, "%3i: %4i \n", s, normalizedCounter[s]);
	for (s=0; s<=maxSymbolValue; s++)
	nTotal += abs(normalizedCounter[s]);
	if (nTotal != (1U<<tableLog))
	RAWLOG(2, "Warning !!! Total == %u != %u !!!", nTotal, 1U<<tableLog);
	getchar();
	}
	#endif

	return tableLog;
	}


	/* fake FSE_CTable, for raw (uncompressed) input */
	size_t FSE_buildCTable_raw (FSE_CTable* ct, unsigned nbBits)
	{
	const unsigned tableSize = 1 << nbBits;
	const unsigned tableMask = tableSize - 1;
	const unsigned maxSymbolValue = tableMask;
	void* const ptr = ct;
	U16* const tableU16 = ( (U16*) ptr) + 2;
	void* const FSCT = ((U32)ptr) + 1 / header / + (tableSize>>1); / assumption : tableLog >= 1 */
	FSE_symbolCompressionTransform* const symbolTT = (FSE_symbolCompressionTransform*) (FSCT);
	unsigned s;

	/* Sanity checks */
	if (nbBits < 1) return ERROR(GENERIC); /* min size */

	/* header */
	tableU16[-2] = (U16) nbBits;
	tableU16[-1] = (U16) maxSymbolValue;

	/* Build table */
	for (s=0; s<tableSize; s++)
	tableU16[s] = (U16)(tableSize + s);

	/* Build Symbol Transformation Table */
	{ const U32 deltaNbBits = (nbBits << 16) - (1 << nbBits);
	for (s=0; s<=maxSymbolValue; s++) {
	symbolTT[s].deltaNbBits = deltaNbBits;
	symbolTT[s].deltaFindState = s-1;
	} }

	return 0;
	}

	/* fake FSE_CTable, for rle input (always same symbol) */
	size_t FSE_buildCTable_rle (FSE_CTable* ct, BYTE symbolValue)
	{
	void* ptr = ct;
	U16* tableU16 = ( (U16*) ptr) + 2;
	void* FSCTptr = (U32*)ptr + 2;
	FSE_symbolCompressionTransform* symbolTT = (FSE_symbolCompressionTransform*) FSCTptr;

	/* header */
	tableU16[-2] = (U16) 0;
	tableU16[-1] = (U16) symbolValue;

	/* Build table */
	tableU16[0] = 0;
	tableU16[1] = 0; /* just in case */

	/* Build Symbol Transformation Table */
	symbolTT[symbolValue].deltaNbBits = 0;
	symbolTT[symbolValue].deltaFindState = 0;

	return 0;
	}


	static size_t FSE_compress_usingCTable_generic (void* dst, size_t dstSize,
	const void* src, size_t srcSize,
	const FSE_CTable* ct, const unsigned fast)
	{
	const BYTE* const istart = (const BYTE*) src;
	const BYTE* const iend = istart + srcSize;
	const BYTE* ip=iend;

	BIT_CStream_t bitC;
	FSE_CState_t CState1, CState2;

	/* init */
	if (srcSize <= 2) return 0;
	{ size_t const initError = BIT_initCStream(&bitC, dst, dstSize);
	if (FSE_isError(initError)) return 0; /* not enough space available to write a bitstream */ }

	#define FSE_FLUSHBITS(s) (fast ? BIT_flushBitsFast(s) : BIT_flushBits(s))

	if (srcSize & 1) {
	FSE_initCState2(&CState1, ct, *--ip);
	FSE_initCState2(&CState2, ct, *--ip);
	FSE_encodeSymbol(&bitC, &CState1, *--ip);
	FSE_FLUSHBITS(&bitC);
	} else {
	FSE_initCState2(&CState2, ct, *--ip);
	FSE_initCState2(&CState1, ct, *--ip);
	}

	/* join to mod 4 */
	srcSize -= 2;
	if ((sizeof(bitC.bitContainer)8 > FSE_MAX_TABLELOG4+7 ) && (srcSize & 2)) { /* test bit 2 */
	FSE_encodeSymbol(&bitC, &CState2, *--ip);
	FSE_encodeSymbol(&bitC, &CState1, *--ip);
	FSE_FLUSHBITS(&bitC);
	}

	/* 2 or 4 encoding per loop */
	while ( ip>istart ) {

	FSE_encodeSymbol(&bitC, &CState2, *--ip);

	if (sizeof(bitC.bitContainer)8 < FSE_MAX_TABLELOG2+7 ) /* this test must be static */
	FSE_FLUSHBITS(&bitC);

	FSE_encodeSymbol(&bitC, &CState1, *--ip);

	if (sizeof(bitC.bitContainer)8 > FSE_MAX_TABLELOG4+7 ) { /* this test must be static */
	FSE_encodeSymbol(&bitC, &CState2, *--ip);
	FSE_encodeSymbol(&bitC, &CState1, *--ip);
	}

	FSE_FLUSHBITS(&bitC);
	}

	FSE_flushCState(&bitC, &CState2);
	FSE_flushCState(&bitC, &CState1);
	return BIT_closeCStream(&bitC);
	}

	size_t FSE_compress_usingCTable (void* dst, size_t dstSize,
	const void* src, size_t srcSize,
	const FSE_CTable* ct)
	{
	unsigned const fast = (dstSize >= FSE_BLOCKBOUND(srcSize));

	if (fast)
	return FSE_compress_usingCTable_generic(dst, dstSize, src, srcSize, ct, 1);
	else
	return FSE_compress_usingCTable_generic(dst, dstSize, src, srcSize, ct, 0);
	}


	size_t FSE_compressBound(size_t size) { return FSE_COMPRESSBOUND(size); }

	/* FSE_compress_wksp() :
	* Same as FSE_compress2(), but using an externally allocated scratch buffer (`workSpace`).
	* `wkspSize` size must be `(1<<tableLog)`.
	*/
	size_t FSE_compress_wksp (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize)
	{
	BYTE* const ostart = (BYTE*) dst;
	BYTE* op = ostart;
	BYTE* const oend = ostart + dstSize;

	unsigned count[FSE_MAX_SYMBOL_VALUE+1];
	S16 norm[FSE_MAX_SYMBOL_VALUE+1];
	FSE_CTable* CTable = (FSE_CTable*)workSpace;
	size_t const CTableSize = FSE_CTABLE_SIZE_U32(tableLog, maxSymbolValue);
	void* scratchBuffer = (void*)(CTable + CTableSize);
	size_t const scratchBufferSize = wkspSize - (CTableSize * sizeof(FSE_CTable));

	/* init conditions */
	if (wkspSize < FSE_WKSP_SIZE_U32(tableLog, maxSymbolValue)) return ERROR(tableLog_tooLarge);
	if (srcSize <= 1) return 0; /* Not compressible */
	if (!maxSymbolValue) maxSymbolValue = FSE_MAX_SYMBOL_VALUE;
	if (!tableLog) tableLog = FSE_DEFAULT_TABLELOG;

	/* Scan input and build symbol stats */
	{ CHECK_V_F(maxCount, HIST_count_wksp(count, &maxSymbolValue, src, srcSize, scratchBuffer, scratchBufferSize) );
	if (maxCount == srcSize) return 1; /* only a single symbol in src : rle */
	if (maxCount == 1) return 0; /* each symbol present maximum once => not compressible */
	if (maxCount < (srcSize >> 7)) return 0; /* Heuristic : not compressible enough */
	}

	tableLog = FSE_optimalTableLog(tableLog, srcSize, maxSymbolValue);
	CHECK_F( FSE_normalizeCount(norm, tableLog, count, srcSize, maxSymbolValue) );

	/* Write table description header */
	{ CHECK_V_F(nc_err, FSE_writeNCount(op, oend-op, norm, maxSymbolValue, tableLog) );
	op += nc_err;
	}

	/* Compress */
	CHECK_F( FSE_buildCTable_wksp(CTable, norm, maxSymbolValue, tableLog, scratchBuffer, scratchBufferSize) );
	{ CHECK_V_F(cSize, FSE_compress_usingCTable(op, oend - op, src, srcSize, CTable) );
	if (cSize == 0) return 0; /* not enough space for compressed data */
	op += cSize;
	}

	/* check compressibility */
	if ( (size_t)(op-ostart) >= srcSize-1 ) return 0;

	return op-ostart;
	}

	typedef struct {
	FSE_CTable CTable_max[FSE_CTABLE_SIZE_U32(FSE_MAX_TABLELOG, FSE_MAX_SYMBOL_VALUE)];
	BYTE scratchBuffer[1 << FSE_MAX_TABLELOG];
	} fseWkspMax_t;

	size_t FSE_compress2 (void* dst, size_t dstCapacity, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog)
	{
	fseWkspMax_t scratchBuffer;
	DEBUG_STATIC_ASSERT(sizeof(scratchBuffer) >= FSE_WKSP_SIZE_U32(FSE_MAX_TABLELOG, FSE_MAX_SYMBOL_VALUE)); /* compilation failures here means scratchBuffer is not large enough */
	if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge);
	return FSE_compress_wksp(dst, dstCapacity, src, srcSize, maxSymbolValue, tableLog, &scratchBuffer, sizeof(scratchBuffer));
	}

	size_t FSE_compress (void* dst, size_t dstCapacity, const void* src, size_t srcSize)
	{
	return FSE_compress2(dst, dstCapacity, src, srcSize, FSE_MAX_SYMBOL_VALUE, FSE_DEFAULT_TABLELOG);
	}


	#endif /* FSE_COMMONDEFS_ONLY */
	/** ended inlining compress/fse_compress.c **/
	/** start inlining compress/hist.c **/
	/* ******************************************************************
	* hist : Histogram functions
	* part of Finite State Entropy project
	* Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
	*
	* You can contact the author at :
	* - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
	* - Public forum : https://groups.google.com/forum/#!forum/lz4c
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	****************************************************************** */

	/* --- dependencies --- */
	/** skipping file: ../common/mem.h **/
	/** skipping file: ../common/debug.h **/
	/** skipping file: ../common/error_private.h **/
	/** skipping file: hist.h **/


	/* --- Error management --- */
	unsigned HIST_isError(size_t code) { return ERR_isError(code); }

	/-*************************************************************
	* Histogram functions
	****************************************************************/
	unsigned HIST_count_simple(unsigned* count, unsigned* maxSymbolValuePtr,
	const void* src, size_t srcSize)
	{
	const BYTE* ip = (const BYTE*)src;
	const BYTE* const end = ip + srcSize;
	unsigned maxSymbolValue = *maxSymbolValuePtr;
	unsigned largestCount=0;

	memset(count, 0, (maxSymbolValue+1) * sizeof(*count));
	if (srcSize==0) { *maxSymbolValuePtr = 0; return 0; }

	while (ip<end) {
	assert(*ip <= maxSymbolValue);
	count[*ip++]++;
	}

	while (!count[maxSymbolValue]) maxSymbolValue--;
	*maxSymbolValuePtr = maxSymbolValue;

	{ U32 s;
	for (s=0; s<=maxSymbolValue; s++)
	if (count[s] > largestCount) largestCount = count[s];
	}

	return largestCount;
	}

	typedef enum { trustInput, checkMaxSymbolValue } HIST_checkInput_e;

	/* HIST_count_parallel_wksp() :
	* store histogram into 4 intermediate tables, recombined at the end.
	* this design makes better use of OoO cpus,
	* and is noticeably faster when some values are heavily repeated.
	* But it needs some additional workspace for intermediate tables.
	* `workSpace` size must be a table of size >= HIST_WKSP_SIZE_U32.
	* @return : largest histogram frequency,
	* or an error code (notably when histogram would be larger than maxSymbolValuePtr). /
	static size_t HIST_count_parallel_wksp(
	unsigned* count, unsigned* maxSymbolValuePtr,
	const void* source, size_t sourceSize,
	HIST_checkInput_e check,
	U32* const workSpace)
	{
	const BYTE* ip = (const BYTE*)source;
	const BYTE* const iend = ip+sourceSize;
	unsigned maxSymbolValue = *maxSymbolValuePtr;
	unsigned max=0;
	U32* const Counting1 = workSpace;
	U32* const Counting2 = Counting1 + 256;
	U32* const Counting3 = Counting2 + 256;
	U32* const Counting4 = Counting3 + 256;

	memset(workSpace, 0, 4256sizeof(unsigned));

	/* safety checks */
	if (!sourceSize) {
	memset(count, 0, maxSymbolValue + 1);
	*maxSymbolValuePtr = 0;
	return 0;
	}
	if (!maxSymbolValue) maxSymbolValue = 255; /* 0 == default */

	/* by stripes of 16 bytes */
	{ U32 cached = MEM_read32(ip); ip += 4;
	while (ip < iend-15) {
	U32 c = cached; cached = MEM_read32(ip); ip += 4;
	Counting1[(BYTE) c ]++;
	Counting2[(BYTE)(c>>8) ]++;
	Counting3[(BYTE)(c>>16)]++;
	Counting4[ c>>24 ]++;
	c = cached; cached = MEM_read32(ip); ip += 4;
	Counting1[(BYTE) c ]++;
	Counting2[(BYTE)(c>>8) ]++;
	Counting3[(BYTE)(c>>16)]++;
	Counting4[ c>>24 ]++;
	c = cached; cached = MEM_read32(ip); ip += 4;
	Counting1[(BYTE) c ]++;
	Counting2[(BYTE)(c>>8) ]++;
	Counting3[(BYTE)(c>>16)]++;
	Counting4[ c>>24 ]++;
	c = cached; cached = MEM_read32(ip); ip += 4;
	Counting1[(BYTE) c ]++;
	Counting2[(BYTE)(c>>8) ]++;
	Counting3[(BYTE)(c>>16)]++;
	Counting4[ c>>24 ]++;
	}
	ip-=4;
	}

	/* finish last symbols */
	while (ip<iend) Counting1[*ip++]++;

	if (check) { /* verify stats will fit into destination table */
	U32 s; for (s=255; s>maxSymbolValue; s--) {
	Counting1[s] += Counting2[s] + Counting3[s] + Counting4[s];
	if (Counting1[s]) return ERROR(maxSymbolValue_tooSmall);
	} }

	{ U32 s;
	if (maxSymbolValue > 255) maxSymbolValue = 255;
	for (s=0; s<=maxSymbolValue; s++) {
	count[s] = Counting1[s] + Counting2[s] + Counting3[s] + Counting4[s];
	if (count[s] > max) max = count[s];
	} }

	while (!count[maxSymbolValue]) maxSymbolValue--;
	*maxSymbolValuePtr = maxSymbolValue;
	return (size_t)max;
	}

	/* HIST_countFast_wksp() :
	* Same as HIST_countFast(), but using an externally provided scratch buffer.
	* `workSpace` is a writable buffer which must be 4-bytes aligned,
	* `workSpaceSize` must be >= HIST_WKSP_SIZE
	*/
	size_t HIST_countFast_wksp(unsigned* count, unsigned* maxSymbolValuePtr,
	const void* source, size_t sourceSize,
	void* workSpace, size_t workSpaceSize)
	{
	if (sourceSize < 1500) /* heuristic threshold */
	return HIST_count_simple(count, maxSymbolValuePtr, source, sourceSize);
	if ((size_t)workSpace & 3) return ERROR(GENERIC); /* must be aligned on 4-bytes boundaries */
	if (workSpaceSize < HIST_WKSP_SIZE) return ERROR(workSpace_tooSmall);
	return HIST_count_parallel_wksp(count, maxSymbolValuePtr, source, sourceSize, trustInput, (U32*)workSpace);
	}

	/* fast variant (unsafe : won't check if src contains values beyond count[] limit) */
	size_t HIST_countFast(unsigned* count, unsigned* maxSymbolValuePtr,
	const void* source, size_t sourceSize)
	{
	unsigned tmpCounters[HIST_WKSP_SIZE_U32];
	return HIST_countFast_wksp(count, maxSymbolValuePtr, source, sourceSize, tmpCounters, sizeof(tmpCounters));
	}

	/* HIST_count_wksp() :
	* Same as HIST_count(), but using an externally provided scratch buffer.
	* `workSpace` size must be table of >= HIST_WKSP_SIZE_U32 unsigned */
	size_t HIST_count_wksp(unsigned* count, unsigned* maxSymbolValuePtr,
	const void* source, size_t sourceSize,
	void* workSpace, size_t workSpaceSize)
	{
	if ((size_t)workSpace & 3) return ERROR(GENERIC); /* must be aligned on 4-bytes boundaries */
	if (workSpaceSize < HIST_WKSP_SIZE) return ERROR(workSpace_tooSmall);
	if (*maxSymbolValuePtr < 255)
	return HIST_count_parallel_wksp(count, maxSymbolValuePtr, source, sourceSize, checkMaxSymbolValue, (U32*)workSpace);
	*maxSymbolValuePtr = 255;
	return HIST_countFast_wksp(count, maxSymbolValuePtr, source, sourceSize, workSpace, workSpaceSize);
	}

	size_t HIST_count(unsigned* count, unsigned* maxSymbolValuePtr,
	const void* src, size_t srcSize)
	{
	unsigned tmpCounters[HIST_WKSP_SIZE_U32];
	return HIST_count_wksp(count, maxSymbolValuePtr, src, srcSize, tmpCounters, sizeof(tmpCounters));
	}
	/** ended inlining compress/hist.c **/
	/** start inlining compress/huf_compress.c **/
	/* ******************************************************************
	* Huffman encoder, part of New Generation Entropy library
	* Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
	*
	* You can contact the author at :
	* - FSE+HUF source repository : https://github.com/Cyan4973/FiniteStateEntropy
	* - Public forum : https://groups.google.com/forum/#!forum/lz4c
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	****************************************************************** */

	/* **************************************************************
	* Compiler specifics
	****************************************************************/
	#ifdef _MSC_VER /* Visual Studio */
	# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */
	#endif


	/* **************************************************************
	* Includes
	****************************************************************/
	#include <string.h> /* memcpy, memset */
	#include <stdio.h> /* printf (debug) */
	/** skipping file: ../common/compiler.h **/
	/** skipping file: ../common/bitstream.h **/
	/** skipping file: hist.h **/
	#define FSE_STATIC_LINKING_ONLY /* FSE_optimalTableLog_internal */
	/** skipping file: ../common/fse.h **/
	#define HUF_STATIC_LINKING_ONLY
	/** skipping file: ../common/huf.h **/
	/** skipping file: ../common/error_private.h **/


	/* **************************************************************
	* Error Management
	****************************************************************/
	#define HUF_isError ERR_isError
	#define HUF_STATIC_ASSERT(c) DEBUG_STATIC_ASSERT(c) /* use only after variable declarations */


	/* **************************************************************
	* Utils
	****************************************************************/
	unsigned HUF_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue)
	{
	return FSE_optimalTableLog_internal(maxTableLog, srcSize, maxSymbolValue, 1);
	}


	/* *******************************************************
	* HUF : Huffman block compression
	*********************************************************/
	/* HUF_compressWeights() :
	* Same as FSE_compress(), but dedicated to huff0's weights compression.
	* The use case needs much less stack memory.
	* Note : all elements within weightTable are supposed to be <= HUF_TABLELOG_MAX.
	*/
	#define MAX_FSE_TABLELOG_FOR_HUFF_HEADER 6
	static size_t HUF_compressWeights (void* dst, size_t dstSize, const void* weightTable, size_t wtSize)
	{
	BYTE* const ostart = (BYTE*) dst;
	BYTE* op = ostart;
	BYTE* const oend = ostart + dstSize;

	unsigned maxSymbolValue = HUF_TABLELOG_MAX;
	U32 tableLog = MAX_FSE_TABLELOG_FOR_HUFF_HEADER;

	FSE_CTable CTable[FSE_CTABLE_SIZE_U32(MAX_FSE_TABLELOG_FOR_HUFF_HEADER, HUF_TABLELOG_MAX)];
	BYTE scratchBuffer[1<<MAX_FSE_TABLELOG_FOR_HUFF_HEADER];

	unsigned count[HUF_TABLELOG_MAX+1];
	S16 norm[HUF_TABLELOG_MAX+1];

	/* init conditions */
	if (wtSize <= 1) return 0; /* Not compressible */

	/* Scan input and build symbol stats */
	{ unsigned const maxCount = HIST_count_simple(count, &maxSymbolValue, weightTable, wtSize); /* never fails */
	if (maxCount == wtSize) return 1; /* only a single symbol in src : rle */
	if (maxCount == 1) return 0; /* each symbol present maximum once => not compressible */
	}

	tableLog = FSE_optimalTableLog(tableLog, wtSize, maxSymbolValue);
	CHECK_F( FSE_normalizeCount(norm, tableLog, count, wtSize, maxSymbolValue) );

	/* Write table description header */
	{ CHECK_V_F(hSize, FSE_writeNCount(op, (size_t)(oend-op), norm, maxSymbolValue, tableLog) );
	op += hSize;
	}

	/* Compress */
	CHECK_F( FSE_buildCTable_wksp(CTable, norm, maxSymbolValue, tableLog, scratchBuffer, sizeof(scratchBuffer)) );
	{ CHECK_V_F(cSize, FSE_compress_usingCTable(op, (size_t)(oend - op), weightTable, wtSize, CTable) );
	if (cSize == 0) return 0; /* not enough space for compressed data */
	op += cSize;
	}

	return (size_t)(op-ostart);
	}


	struct HUF_CElt_s {
	U16 val;
	BYTE nbBits;
	}; /* typedef'd to HUF_CElt within "huf.h" */

	/*! HUF_writeCTable() :
	`CTable` : Huffman tree to save, using huf representation.
	@return : size of saved CTable */
	size_t HUF_writeCTable (void* dst, size_t maxDstSize,
	const HUF_CElt* CTable, unsigned maxSymbolValue, unsigned huffLog)
	{
	BYTE bitsToWeight[HUF_TABLELOG_MAX + 1]; /* precomputed conversion table */
	BYTE huffWeight[HUF_SYMBOLVALUE_MAX];
	BYTE* op = (BYTE*)dst;
	U32 n;

	/* check conditions */
	if (maxSymbolValue > HUF_SYMBOLVALUE_MAX) return ERROR(maxSymbolValue_tooLarge);

	/* convert to weight */
	bitsToWeight[0] = 0;
	for (n=1; n<huffLog+1; n++)
	bitsToWeight[n] = (BYTE)(huffLog + 1 - n);
	for (n=0; n<maxSymbolValue; n++)
	huffWeight[n] = bitsToWeight[CTable[n].nbBits];

	/* attempt weights compression by FSE */
	{ CHECK_V_F(hSize, HUF_compressWeights(op+1, maxDstSize-1, huffWeight, maxSymbolValue) );
	if ((hSize>1) & (hSize < maxSymbolValue/2)) { /* FSE compressed */
	op[0] = (BYTE)hSize;
	return hSize+1;
	} }

	/* write raw values as 4-bits (max : 15) */
	if (maxSymbolValue > (256-128)) return ERROR(GENERIC); /* should not happen : likely means source cannot be compressed */
	if (((maxSymbolValue+1)/2) + 1 > maxDstSize) return ERROR(dstSize_tooSmall); /* not enough space within dst buffer */
	op[0] = (BYTE)(128 /special case/ + (maxSymbolValue-1));
	huffWeight[maxSymbolValue] = 0; /* to be sure it doesn't cause msan issue in final combination */
	for (n=0; n<maxSymbolValue; n+=2)
	op[(n/2)+1] = (BYTE)((huffWeight[n] << 4) + huffWeight[n+1]);
	return ((maxSymbolValue+1)/2) + 1;
	}


	size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize, unsigned* hasZeroWeights)
	{
	BYTE huffWeight[HUF_SYMBOLVALUE_MAX + 1]; /* init not required, even though some static analyzer may complain */
	U32 rankVal[HUF_TABLELOG_ABSOLUTEMAX + 1]; /* large enough for values from 0 to 16 */
	U32 tableLog = 0;
	U32 nbSymbols = 0;

	/* get symbol weights */
	CHECK_V_F(readSize, HUF_readStats(huffWeight, HUF_SYMBOLVALUE_MAX+1, rankVal, &nbSymbols, &tableLog, src, srcSize));

	/* check result */
	if (tableLog > HUF_TABLELOG_MAX) return ERROR(tableLog_tooLarge);
	if (nbSymbols > *maxSymbolValuePtr+1) return ERROR(maxSymbolValue_tooSmall);

	/* Prepare base value per rank */
	{ U32 n, nextRankStart = 0;
	for (n=1; n<=tableLog; n++) {
	U32 current = nextRankStart;
	nextRankStart += (rankVal[n] << (n-1));
	rankVal[n] = current;
	} }

	/* fill nbBits */
	*hasZeroWeights = 0;
	{ U32 n; for (n=0; n<nbSymbols; n++) {
	const U32 w = huffWeight[n];
	*hasZeroWeights \|= (w == 0);
	CTable[n].nbBits = (BYTE)(tableLog + 1 - w) & -(w != 0);
	} }

	/* fill val */
	{ U16 nbPerRank[HUF_TABLELOG_MAX+2] = {0}; /* support w=0=>n=tableLog+1 */
	U16 valPerRank[HUF_TABLELOG_MAX+2] = {0};
	{ U32 n; for (n=0; n<nbSymbols; n++) nbPerRank[CTable[n].nbBits]++; }
	/* determine stating value per rank */
	valPerRank[tableLog+1] = 0; /* for w==0 */
	{ U16 min = 0;
	U32 n; for (n=tableLog; n>0; n--) { /* start at n=tablelog <-> w=1 */
	valPerRank[n] = min; /* get starting value within each rank */
	min += nbPerRank[n];
	min >>= 1;
	} }
	/* assign value within rank, symbol order */
	{ U32 n; for (n=0; n<nbSymbols; n++) CTable[n].val = valPerRank[CTable[n].nbBits]++; }
	}

	*maxSymbolValuePtr = nbSymbols - 1;
	return readSize;
	}

	U32 HUF_getNbBits(const void* symbolTable, U32 symbolValue)
	{
	const HUF_CElt* table = (const HUF_CElt*)symbolTable;
	assert(symbolValue <= HUF_SYMBOLVALUE_MAX);
	return table[symbolValue].nbBits;
	}


	typedef struct nodeElt_s {
	U32 count;
	U16 parent;
	BYTE byte;
	BYTE nbBits;
	} nodeElt;

	static U32 HUF_setMaxHeight(nodeElt* huffNode, U32 lastNonNull, U32 maxNbBits)
	{
	const U32 largestBits = huffNode[lastNonNull].nbBits;
	if (largestBits <= maxNbBits) return largestBits; /* early exit : no elt > maxNbBits */

	/* there are several too large elements (at least >= 2) */
	{ int totalCost = 0;
	const U32 baseCost = 1 << (largestBits - maxNbBits);
	int n = (int)lastNonNull;

	while (huffNode[n].nbBits > maxNbBits) {
	totalCost += baseCost - (1 << (largestBits - huffNode[n].nbBits));
	huffNode[n].nbBits = (BYTE)maxNbBits;
	n --;
	} /* n stops at huffNode[n].nbBits <= maxNbBits */
	while (huffNode[n].nbBits == maxNbBits) n--; /* n end at index of smallest symbol using < maxNbBits */

	/* renorm totalCost */
	totalCost >>= (largestBits - maxNbBits); /* note : totalCost is necessarily a multiple of baseCost */

	/* repay normalized cost */
	{ U32 const noSymbol = 0xF0F0F0F0;
	U32 rankLast[HUF_TABLELOG_MAX+2];

	/* Get pos of last (smallest) symbol per rank */
	memset(rankLast, 0xF0, sizeof(rankLast));
	{ U32 currentNbBits = maxNbBits;
	int pos;
	for (pos=n ; pos >= 0; pos--) {
	if (huffNode[pos].nbBits >= currentNbBits) continue;
	currentNbBits = huffNode[pos].nbBits; /* < maxNbBits */
	rankLast[maxNbBits-currentNbBits] = (U32)pos;
	} }

	while (totalCost > 0) {
	U32 nBitsToDecrease = BIT_highbit32((U32)totalCost) + 1;
	for ( ; nBitsToDecrease > 1; nBitsToDecrease--) {
	U32 const highPos = rankLast[nBitsToDecrease];
	U32 const lowPos = rankLast[nBitsToDecrease-1];
	if (highPos == noSymbol) continue;
	if (lowPos == noSymbol) break;
	{ U32 const highTotal = huffNode[highPos].count;
	U32 const lowTotal = 2 * huffNode[lowPos].count;
	if (highTotal <= lowTotal) break;
	} }
	/* only triggered when no more rank 1 symbol left => find closest one (note : there is necessarily at least one !) */
	/* HUF_MAX_TABLELOG test just to please gcc 5+; but it should not be necessary */
	while ((nBitsToDecrease<=HUF_TABLELOG_MAX) && (rankLast[nBitsToDecrease] == noSymbol))
	nBitsToDecrease ++;
	totalCost -= 1 << (nBitsToDecrease-1);
	if (rankLast[nBitsToDecrease-1] == noSymbol)
	rankLast[nBitsToDecrease-1] = rankLast[nBitsToDecrease]; /* this rank is no longer empty */
	huffNode[rankLast[nBitsToDecrease]].nbBits ++;
	if (rankLast[nBitsToDecrease] == 0) /* special case, reached largest symbol */
	rankLast[nBitsToDecrease] = noSymbol;
	else {
	rankLast[nBitsToDecrease]--;
	if (huffNode[rankLast[nBitsToDecrease]].nbBits != maxNbBits-nBitsToDecrease)
	rankLast[nBitsToDecrease] = noSymbol; /* this rank is now empty */
	} } /* while (totalCost > 0) */

	while (totalCost < 0) { /* Sometimes, cost correction overshoot */
	if (rankLast[1] == noSymbol) { /* special case : no rank 1 symbol (using maxNbBits-1); let's create one from largest rank 0 (using maxNbBits) */
	while (huffNode[n].nbBits == maxNbBits) n--;
	huffNode[n+1].nbBits--;
	assert(n >= 0);
	rankLast[1] = (U32)(n+1);
	totalCost++;
	continue;
	}
	huffNode[ rankLast[1] + 1 ].nbBits--;
	rankLast[1]++;
	totalCost ++;
	} } } /* there are several too large elements (at least >= 2) */

	return maxNbBits;
	}

	typedef struct {
	U32 base;
	U32 current;
	} rankPos;

	typedef nodeElt huffNodeTable[HUF_CTABLE_WORKSPACE_SIZE_U32];

	#define RANK_POSITION_TABLE_SIZE 32

	typedef struct {
	huffNodeTable huffNodeTbl;
	rankPos rankPosition[RANK_POSITION_TABLE_SIZE];
	} HUF_buildCTable_wksp_tables;

	static void HUF_sort(nodeElt* huffNode, const unsigned* count, U32 maxSymbolValue, rankPos* rankPosition)
	{
	U32 n;

	memset(rankPosition, 0, sizeof(rankPosition) RANK_POSITION_TABLE_SIZE);
	for (n=0; n<=maxSymbolValue; n++) {
	U32 r = BIT_highbit32(count[n] + 1);
	rankPosition[r].base ++;
	}
	for (n=30; n>0; n--) rankPosition[n-1].base += rankPosition[n].base;
	for (n=0; n<32; n++) rankPosition[n].current = rankPosition[n].base;
	for (n=0; n<=maxSymbolValue; n++) {
	U32 const c = count[n];
	U32 const r = BIT_highbit32(c+1) + 1;
	U32 pos = rankPosition[r].current++;
	while ((pos > rankPosition[r].base) && (c > huffNode[pos-1].count)) {
	huffNode[pos] = huffNode[pos-1];
	pos--;
	}
	huffNode[pos].count = c;
	huffNode[pos].byte = (BYTE)n;
	}
	}


	/** HUF_buildCTable_wksp() :
	* Same as HUF_buildCTable(), but using externally allocated scratch buffer.
	* `workSpace` must be aligned on 4-bytes boundaries, and be at least as large as sizeof(HUF_buildCTable_wksp_tables).
	*/
	#define STARTNODE (HUF_SYMBOLVALUE_MAX+1)

	size_t HUF_buildCTable_wksp (HUF_CElt* tree, const unsigned* count, U32 maxSymbolValue, U32 maxNbBits, void* workSpace, size_t wkspSize)
	{
	HUF_buildCTable_wksp_tables* const wksp_tables = (HUF_buildCTable_wksp_tables*)workSpace;
	nodeElt* const huffNode0 = wksp_tables->huffNodeTbl;
	nodeElt* const huffNode = huffNode0+1;
	int nonNullRank;
	int lowS, lowN;
	int nodeNb = STARTNODE;
	int n, nodeRoot;

	/* safety checks */
	if (((size_t)workSpace & 3) != 0) return ERROR(GENERIC); /* must be aligned on 4-bytes boundaries */
	if (wkspSize < sizeof(HUF_buildCTable_wksp_tables))
	return ERROR(workSpace_tooSmall);
	if (maxNbBits == 0) maxNbBits = HUF_TABLELOG_DEFAULT;
	if (maxSymbolValue > HUF_SYMBOLVALUE_MAX)
	return ERROR(maxSymbolValue_tooLarge);
	memset(huffNode0, 0, sizeof(huffNodeTable));

	/* sort, decreasing order */
	HUF_sort(huffNode, count, maxSymbolValue, wksp_tables->rankPosition);

	/* init for parents */
	nonNullRank = (int)maxSymbolValue;
	while(huffNode[nonNullRank].count == 0) nonNullRank--;
	lowS = nonNullRank; nodeRoot = nodeNb + lowS - 1; lowN = nodeNb;
	huffNode[nodeNb].count = huffNode[lowS].count + huffNode[lowS-1].count;
	huffNode[lowS].parent = huffNode[lowS-1].parent = (U16)nodeNb;
	nodeNb++; lowS-=2;
	for (n=nodeNb; n<=nodeRoot; n++) huffNode[n].count = (U32)(1U<<30);
	huffNode0[0].count = (U32)(1U<<31); /* fake entry, strong barrier */

	/* create parents */
	while (nodeNb <= nodeRoot) {
	int const n1 = (huffNode[lowS].count < huffNode[lowN].count) ? lowS-- : lowN++;
	int const n2 = (huffNode[lowS].count < huffNode[lowN].count) ? lowS-- : lowN++;
	huffNode[nodeNb].count = huffNode[n1].count + huffNode[n2].count;
	huffNode[n1].parent = huffNode[n2].parent = (U16)nodeNb;
	nodeNb++;
	}

	/* distribute weights (unlimited tree height) */
	huffNode[nodeRoot].nbBits = 0;
	for (n=nodeRoot-1; n>=STARTNODE; n--)
	huffNode[n].nbBits = huffNode[ huffNode[n].parent ].nbBits + 1;
	for (n=0; n<=nonNullRank; n++)
	huffNode[n].nbBits = huffNode[ huffNode[n].parent ].nbBits + 1;

	/* enforce maxTableLog */
	maxNbBits = HUF_setMaxHeight(huffNode, (U32)nonNullRank, maxNbBits);

	/* fill result into tree (val, nbBits) */
	{ U16 nbPerRank[HUF_TABLELOG_MAX+1] = {0};
	U16 valPerRank[HUF_TABLELOG_MAX+1] = {0};
	int const alphabetSize = (int)(maxSymbolValue + 1);
	if (maxNbBits > HUF_TABLELOG_MAX) return ERROR(GENERIC); /* check fit into table */
	for (n=0; n<=nonNullRank; n++)
	nbPerRank[huffNode[n].nbBits]++;
	/* determine stating value per rank */
	{ U16 min = 0;
	for (n=(int)maxNbBits; n>0; n--) {
	valPerRank[n] = min; /* get starting value within each rank */
	min += nbPerRank[n];
	min >>= 1;
	} }
	for (n=0; n<alphabetSize; n++)
	tree[huffNode[n].byte].nbBits = huffNode[n].nbBits; /* push nbBits per symbol, symbol order */
	for (n=0; n<alphabetSize; n++)
	tree[n].val = valPerRank[tree[n].nbBits]++; /* assign value within rank, symbol order */
	}

	return maxNbBits;
	}

	/** HUF_buildCTable() :
	* @return : maxNbBits
	* Note : count is used before tree is written, so they can safely overlap
	*/
	size_t HUF_buildCTable (HUF_CElt* tree, const unsigned* count, unsigned maxSymbolValue, unsigned maxNbBits)
	{
	HUF_buildCTable_wksp_tables workspace;
	return HUF_buildCTable_wksp(tree, count, maxSymbolValue, maxNbBits, &workspace, sizeof(workspace));
	}

	size_t HUF_estimateCompressedSize(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue)
	{
	size_t nbBits = 0;
	int s;
	for (s = 0; s <= (int)maxSymbolValue; ++s) {
	nbBits += CTable[s].nbBits * count[s];
	}
	return nbBits >> 3;
	}

	int HUF_validateCTable(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue) {
	int bad = 0;
	int s;
	for (s = 0; s <= (int)maxSymbolValue; ++s) {
	bad \|= (count[s] != 0) & (CTable[s].nbBits == 0);
	}
	return !bad;
	}

	size_t HUF_compressBound(size_t size) { return HUF_COMPRESSBOUND(size); }

	FORCE_INLINE_TEMPLATE void
	HUF_encodeSymbol(BIT_CStream_t* bitCPtr, U32 symbol, const HUF_CElt* CTable)
	{
	BIT_addBitsFast(bitCPtr, CTable[symbol].val, CTable[symbol].nbBits);
	}

	#define HUF_FLUSHBITS(s) BIT_flushBits(s)

	#define HUF_FLUSHBITS_1(stream) \
	if (sizeof((stream)->bitContainer)8 < HUF_TABLELOG_MAX2+7) HUF_FLUSHBITS(stream)

	#define HUF_FLUSHBITS_2(stream) \
	if (sizeof((stream)->bitContainer)8 < HUF_TABLELOG_MAX4+7) HUF_FLUSHBITS(stream)

	FORCE_INLINE_TEMPLATE size_t
	HUF_compress1X_usingCTable_internal_body(void* dst, size_t dstSize,
	const void* src, size_t srcSize,
	const HUF_CElt* CTable)
	{
	const BYTE* ip = (const BYTE*) src;
	BYTE* const ostart = (BYTE*)dst;
	BYTE* const oend = ostart + dstSize;
	BYTE* op = ostart;
	size_t n;
	BIT_CStream_t bitC;

	/* init */
	if (dstSize < 8) return 0; /* not enough space to compress */
	{ size_t const initErr = BIT_initCStream(&bitC, op, (size_t)(oend-op));
	if (HUF_isError(initErr)) return 0; }

	n = srcSize & ~3; /* join to mod 4 */
	switch (srcSize & 3)
	{
	case 3 : HUF_encodeSymbol(&bitC, ip[n+ 2], CTable);
	HUF_FLUSHBITS_2(&bitC);
	/* fall-through */
	case 2 : HUF_encodeSymbol(&bitC, ip[n+ 1], CTable);
	HUF_FLUSHBITS_1(&bitC);
	/* fall-through */
	case 1 : HUF_encodeSymbol(&bitC, ip[n+ 0], CTable);
	HUF_FLUSHBITS(&bitC);
	/* fall-through */
	case 0 : /* fall-through */
	default: break;
	}

	for (; n>0; n-=4) { /* note : n&3==0 at this stage */
	HUF_encodeSymbol(&bitC, ip[n- 1], CTable);
	HUF_FLUSHBITS_1(&bitC);
	HUF_encodeSymbol(&bitC, ip[n- 2], CTable);
	HUF_FLUSHBITS_2(&bitC);
	HUF_encodeSymbol(&bitC, ip[n- 3], CTable);
	HUF_FLUSHBITS_1(&bitC);
	HUF_encodeSymbol(&bitC, ip[n- 4], CTable);
	HUF_FLUSHBITS(&bitC);
	}

	return BIT_closeCStream(&bitC);
	}

	#if DYNAMIC_BMI2

	static TARGET_ATTRIBUTE("bmi2") size_t
	HUF_compress1X_usingCTable_internal_bmi2(void* dst, size_t dstSize,
	const void* src, size_t srcSize,
	const HUF_CElt* CTable)
	{
	return HUF_compress1X_usingCTable_internal_body(dst, dstSize, src, srcSize, CTable);
	}

	static size_t
	HUF_compress1X_usingCTable_internal_default(void* dst, size_t dstSize,
	const void* src, size_t srcSize,
	const HUF_CElt* CTable)
	{
	return HUF_compress1X_usingCTable_internal_body(dst, dstSize, src, srcSize, CTable);
	}

	static size_t
	HUF_compress1X_usingCTable_internal(void* dst, size_t dstSize,
	const void* src, size_t srcSize,
	const HUF_CElt* CTable, const int bmi2)
	{
	if (bmi2) {
	return HUF_compress1X_usingCTable_internal_bmi2(dst, dstSize, src, srcSize, CTable);
	}
	return HUF_compress1X_usingCTable_internal_default(dst, dstSize, src, srcSize, CTable);
	}

	#else

	static size_t
	HUF_compress1X_usingCTable_internal(void* dst, size_t dstSize,
	const void* src, size_t srcSize,
	const HUF_CElt* CTable, const int bmi2)
	{
	(void)bmi2;
	return HUF_compress1X_usingCTable_internal_body(dst, dstSize, src, srcSize, CTable);
	}

	#endif

	size_t HUF_compress1X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable)
	{
	return HUF_compress1X_usingCTable_internal(dst, dstSize, src, srcSize, CTable, /* bmi2 */ 0);
	}


	static size_t
	HUF_compress4X_usingCTable_internal(void* dst, size_t dstSize,
	const void* src, size_t srcSize,
	const HUF_CElt* CTable, int bmi2)
	{
	size_t const segmentSize = (srcSize+3)/4; /* first 3 segments */
	const BYTE* ip = (const BYTE*) src;
	const BYTE* const iend = ip + srcSize;
	BYTE* const ostart = (BYTE*) dst;
	BYTE* const oend = ostart + dstSize;
	BYTE* op = ostart;

	if (dstSize < 6 + 1 + 1 + 1 + 8) return 0; /* minimum space to compress successfully */
	if (srcSize < 12) return 0; /* no saving possible : too small input */
	op += 6; /* jumpTable */

	assert(op <= oend);
	{ CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, segmentSize, CTable, bmi2) );
	if (cSize==0) return 0;
	assert(cSize <= 65535);
	MEM_writeLE16(ostart, (U16)cSize);
	op += cSize;
	}

	ip += segmentSize;
	assert(op <= oend);
	{ CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, segmentSize, CTable, bmi2) );
	if (cSize==0) return 0;
	assert(cSize <= 65535);
	MEM_writeLE16(ostart+2, (U16)cSize);
	op += cSize;
	}

	ip += segmentSize;
	assert(op <= oend);
	{ CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, segmentSize, CTable, bmi2) );
	if (cSize==0) return 0;
	assert(cSize <= 65535);
	MEM_writeLE16(ostart+4, (U16)cSize);
	op += cSize;
	}

	ip += segmentSize;
	assert(op <= oend);
	assert(ip <= iend);
	{ CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, (size_t)(iend-ip), CTable, bmi2) );
	if (cSize==0) return 0;
	op += cSize;
	}

	return (size_t)(op-ostart);
	}

	size_t HUF_compress4X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable)
	{
	return HUF_compress4X_usingCTable_internal(dst, dstSize, src, srcSize, CTable, /* bmi2 */ 0);
	}

	typedef enum { HUF_singleStream, HUF_fourStreams } HUF_nbStreams_e;

	static size_t HUF_compressCTable_internal(
	BYTE* const ostart, BYTE* op, BYTE* const oend,
	const void* src, size_t srcSize,
	HUF_nbStreams_e nbStreams, const HUF_CElt* CTable, const int bmi2)
	{
	size_t const cSize = (nbStreams==HUF_singleStream) ?
	HUF_compress1X_usingCTable_internal(op, (size_t)(oend - op), src, srcSize, CTable, bmi2) :
	HUF_compress4X_usingCTable_internal(op, (size_t)(oend - op), src, srcSize, CTable, bmi2);
	if (HUF_isError(cSize)) { return cSize; }
	if (cSize==0) { return 0; } /* uncompressible */
	op += cSize;
	/* check compressibility */
	assert(op >= ostart);
	if ((size_t)(op-ostart) >= srcSize-1) { return 0; }
	return (size_t)(op-ostart);
	}

	typedef struct {
	unsigned count[HUF_SYMBOLVALUE_MAX + 1];
	HUF_CElt CTable[HUF_SYMBOLVALUE_MAX + 1];
	HUF_buildCTable_wksp_tables buildCTable_wksp;
	} HUF_compress_tables_t;

	/* HUF_compress_internal() :
	* `workSpace` must a table of at least HUF_WORKSPACE_SIZE_U32 unsigned */
	static size_t
	HUF_compress_internal (void* dst, size_t dstSize,
	const void* src, size_t srcSize,
	unsigned maxSymbolValue, unsigned huffLog,
	HUF_nbStreams_e nbStreams,
	void* workSpace, size_t wkspSize,
	HUF_CElt* oldHufTable, HUF_repeat* repeat, int preferRepeat,
	const int bmi2)
	{
	HUF_compress_tables_t* const table = (HUF_compress_tables_t*)workSpace;
	BYTE* const ostart = (BYTE*)dst;
	BYTE* const oend = ostart + dstSize;
	BYTE* op = ostart;

	HUF_STATIC_ASSERT(sizeof(*table) <= HUF_WORKSPACE_SIZE);

	/* checks & inits */
	if (((size_t)workSpace & 3) != 0) return ERROR(GENERIC); /* must be aligned on 4-bytes boundaries */
	if (wkspSize < HUF_WORKSPACE_SIZE) return ERROR(workSpace_tooSmall);
	if (!srcSize) return 0; /* Uncompressed */
	if (!dstSize) return 0; /* cannot fit anything within dst budget */
	if (srcSize > HUF_BLOCKSIZE_MAX) return ERROR(srcSize_wrong); /* current block size limit */
	if (huffLog > HUF_TABLELOG_MAX) return ERROR(tableLog_tooLarge);
	if (maxSymbolValue > HUF_SYMBOLVALUE_MAX) return ERROR(maxSymbolValue_tooLarge);
	if (!maxSymbolValue) maxSymbolValue = HUF_SYMBOLVALUE_MAX;
	if (!huffLog) huffLog = HUF_TABLELOG_DEFAULT;

	/* Heuristic : If old table is valid, use it for small inputs */
	if (preferRepeat && repeat && *repeat == HUF_repeat_valid) {
	return HUF_compressCTable_internal(ostart, op, oend,
	src, srcSize,
	nbStreams, oldHufTable, bmi2);
	}

	/* Scan input and build symbol stats */
	{ CHECK_V_F(largest, HIST_count_wksp (table->count, &maxSymbolValue, (const BYTE*)src, srcSize, workSpace, wkspSize) );
	if (largest == srcSize) { ostart = ((const BYTE)src)[0]; return 1; } /* single symbol, rle */
	if (largest <= (srcSize >> 7)+4) return 0; /* heuristic : probably not compressible enough */
	}

	/* Check validity of previous table */
	if ( repeat
	&& *repeat == HUF_repeat_check
	&& !HUF_validateCTable(oldHufTable, table->count, maxSymbolValue)) {
	*repeat = HUF_repeat_none;
	}
	/* Heuristic : use existing table for small inputs */
	if (preferRepeat && repeat && *repeat != HUF_repeat_none) {
	return HUF_compressCTable_internal(ostart, op, oend,
	src, srcSize,
	nbStreams, oldHufTable, bmi2);
	}

	/* Build Huffman Tree */
	huffLog = HUF_optimalTableLog(huffLog, srcSize, maxSymbolValue);
	{ size_t const maxBits = HUF_buildCTable_wksp(table->CTable, table->count,
	maxSymbolValue, huffLog,
	&table->buildCTable_wksp, sizeof(table->buildCTable_wksp));
	CHECK_F(maxBits);
	huffLog = (U32)maxBits;
	/* Zero unused symbols in CTable, so we can check it for validity */
	memset(table->CTable + (maxSymbolValue + 1), 0,
	sizeof(table->CTable) - ((maxSymbolValue + 1) * sizeof(HUF_CElt)));
	}

	/* Write table description header */
	{ CHECK_V_F(hSize, HUF_writeCTable (op, dstSize, table->CTable, maxSymbolValue, huffLog) );
	/* Check if using previous huffman table is beneficial */
	if (repeat && *repeat != HUF_repeat_none) {
	size_t const oldSize = HUF_estimateCompressedSize(oldHufTable, table->count, maxSymbolValue);
	size_t const newSize = HUF_estimateCompressedSize(table->CTable, table->count, maxSymbolValue);
	if (oldSize <= hSize + newSize \|\| hSize + 12 >= srcSize) {
	return HUF_compressCTable_internal(ostart, op, oend,
	src, srcSize,
	nbStreams, oldHufTable, bmi2);
	} }

	/* Use the new huffman table */
	if (hSize + 12ul >= srcSize) { return 0; }
	op += hSize;
	if (repeat) { *repeat = HUF_repeat_none; }
	if (oldHufTable)
	memcpy(oldHufTable, table->CTable, sizeof(table->CTable)); /* Save new table */
	}
	return HUF_compressCTable_internal(ostart, op, oend,
	src, srcSize,
	nbStreams, table->CTable, bmi2);
	}


	size_t HUF_compress1X_wksp (void* dst, size_t dstSize,
	const void* src, size_t srcSize,
	unsigned maxSymbolValue, unsigned huffLog,
	void* workSpace, size_t wkspSize)
	{
	return HUF_compress_internal(dst, dstSize, src, srcSize,
	maxSymbolValue, huffLog, HUF_singleStream,
	workSpace, wkspSize,
	NULL, NULL, 0, 0 /bmi2/);
	}

	size_t HUF_compress1X_repeat (void* dst, size_t dstSize,
	const void* src, size_t srcSize,
	unsigned maxSymbolValue, unsigned huffLog,
	void* workSpace, size_t wkspSize,
	HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat, int bmi2)
	{
	return HUF_compress_internal(dst, dstSize, src, srcSize,
	maxSymbolValue, huffLog, HUF_singleStream,
	workSpace, wkspSize, hufTable,
	repeat, preferRepeat, bmi2);
	}

	size_t HUF_compress1X (void* dst, size_t dstSize,
	const void* src, size_t srcSize,
	unsigned maxSymbolValue, unsigned huffLog)
	{
	unsigned workSpace[HUF_WORKSPACE_SIZE_U32];
	return HUF_compress1X_wksp(dst, dstSize, src, srcSize, maxSymbolValue, huffLog, workSpace, sizeof(workSpace));
	}

	/* HUF_compress4X_repeat():
	* compress input using 4 streams.
	* provide workspace to generate compression tables */
	size_t HUF_compress4X_wksp (void* dst, size_t dstSize,
	const void* src, size_t srcSize,
	unsigned maxSymbolValue, unsigned huffLog,
	void* workSpace, size_t wkspSize)
	{
	return HUF_compress_internal(dst, dstSize, src, srcSize,
	maxSymbolValue, huffLog, HUF_fourStreams,
	workSpace, wkspSize,
	NULL, NULL, 0, 0 /bmi2/);
	}

	/* HUF_compress4X_repeat():
	* compress input using 4 streams.
	* re-use an existing huffman compression table */
	size_t HUF_compress4X_repeat (void* dst, size_t dstSize,
	const void* src, size_t srcSize,
	unsigned maxSymbolValue, unsigned huffLog,
	void* workSpace, size_t wkspSize,
	HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat, int bmi2)
	{
	return HUF_compress_internal(dst, dstSize, src, srcSize,
	maxSymbolValue, huffLog, HUF_fourStreams,
	workSpace, wkspSize,
	hufTable, repeat, preferRepeat, bmi2);
	}

	size_t HUF_compress2 (void* dst, size_t dstSize,
	const void* src, size_t srcSize,
	unsigned maxSymbolValue, unsigned huffLog)
	{
	unsigned workSpace[HUF_WORKSPACE_SIZE_U32];
	return HUF_compress4X_wksp(dst, dstSize, src, srcSize, maxSymbolValue, huffLog, workSpace, sizeof(workSpace));
	}

	size_t HUF_compress (void* dst, size_t maxDstSize, const void* src, size_t srcSize)
	{
	return HUF_compress2(dst, maxDstSize, src, srcSize, 255, HUF_TABLELOG_DEFAULT);
	}
	/** ended inlining compress/huf_compress.c **/
	/** start inlining compress/zstd_compress_literals.c **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	/-************************************
	* Dependencies
	***************************************/
	/** start inlining zstd_compress_literals.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTD_COMPRESS_LITERALS_H
	#define ZSTD_COMPRESS_LITERALS_H

	/** start inlining zstd_compress_internal.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	/* This header contains definitions
	* that shall only be used by modules within lib/compress.
	*/

	#ifndef ZSTD_COMPRESS_H
	#define ZSTD_COMPRESS_H

	/-************************************
	* Dependencies
	***************************************/
	/** skipping file: ../common/zstd_internal.h **/
	/** start inlining zstd_cwksp.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTD_CWKSP_H
	#define ZSTD_CWKSP_H

	/-************************************
	* Dependencies
	***************************************/
	/** skipping file: ../common/zstd_internal.h **/

	#if defined (__cplusplus)
	extern "C" {
	#endif

	/-************************************
	* Constants
	***************************************/

	/* Since the workspace is effectively its own little malloc implementation /
	* arena, when we run under ASAN, we should similarly insert redzones between
	* each internal element of the workspace, so ASAN will catch overruns that
	* reach outside an object but that stay inside the workspace.
	*
	* This defines the size of that redzone.
	*/
	#ifndef ZSTD_CWKSP_ASAN_REDZONE_SIZE
	#define ZSTD_CWKSP_ASAN_REDZONE_SIZE 128
	#endif

	/-************************************
	* Structures
	***************************************/
	typedef enum {
	ZSTD_cwksp_alloc_objects,
	ZSTD_cwksp_alloc_buffers,
	ZSTD_cwksp_alloc_aligned
	} ZSTD_cwksp_alloc_phase_e;

	/**
	* Zstd fits all its internal datastructures into a single continuous buffer,
	* so that it only needs to perform a single OS allocation (or so that a buffer
	* can be provided to it and it can perform no allocations at all). This buffer
	* is called the workspace.
	*
	* Several optimizations complicate that process of allocating memory ranges
	* from this workspace for each internal datastructure:
	*
	* - These different internal datastructures have different setup requirements:
	*
	* - The static objects need to be cleared once and can then be trivially
	* reused for each compression.
	*
	* - Various buffers don't need to be initialized at all--they are always
	* written into before they're read.
	*
	* - The matchstate tables have a unique requirement that they don't need
	* their memory to be totally cleared, but they do need the memory to have
	* some bound, i.e., a guarantee that all values in the memory they've been
	* allocated is less than some maximum value (which is the starting value
	* for the indices that they will then use for compression). When this
	* guarantee is provided to them, they can use the memory without any setup
	* work. When it can't, they have to clear the area.
	*
	* - These buffers also have different alignment requirements.
	*
	* - We would like to reuse the objects in the workspace for multiple
	* compressions without having to perform any expensive reallocation or
	* reinitialization work.
	*
	* - We would like to be able to efficiently reuse the workspace across
	* multiple compressions even when the compression parameters change and
	* we need to resize some of the objects (where possible).
	*
	* To attempt to manage this buffer, given these constraints, the ZSTD_cwksp
	* abstraction was created. It works as follows:
	*
	* Workspace Layout:
	*
	* [ ... workspace ... ]
	* [objects][tables ... ->] free space [<- ... aligned][<- ... buffers]
	*
	* The various objects that live in the workspace are divided into the
	* following categories, and are allocated separately:
	*
	* - Static objects: this is optionally the enclosing ZSTD_CCtx or ZSTD_CDict,
	* so that literally everything fits in a single buffer. Note: if present,
	* this must be the first object in the workspace, since ZSTD_free{CCtx,
	* CDict}() rely on a pointer comparison to see whether one or two frees are
	* required.
	*
	* - Fixed size objects: these are fixed-size, fixed-count objects that are
	* nonetheless "dynamically" allocated in the workspace so that we can
	* control how they're initialized separately from the broader ZSTD_CCtx.
	* Examples:
	* - Entropy Workspace
	* - 2 x ZSTD_compressedBlockState_t
	* - CDict dictionary contents
	*
	* - Tables: these are any of several different datastructures (hash tables,
	* chain tables, binary trees) that all respect a common format: they are
	* uint32_t arrays, all of whose values are between 0 and (nextSrc - base).
	* Their sizes depend on the cparams.
	*
	* - Aligned: these buffers are used for various purposes that require 4 byte
	* alignment, but don't require any initialization before they're used.
	*
	* - Buffers: these buffers are used for various purposes that don't require
	* any alignment or initialization before they're used. This means they can
	* be moved around at no cost for a new compression.
	*
	* Allocating Memory:
	*
	* The various types of objects must be allocated in order, so they can be
	* correctly packed into the workspace buffer. That order is:
	*
	* 1. Objects
	* 2. Buffers
	* 3. Aligned
	* 4. Tables
	*
	* Attempts to reserve objects of different types out of order will fail.
	*/
	typedef struct {
	void* workspace;
	void* workspaceEnd;

	void* objectEnd;
	void* tableEnd;
	void* tableValidEnd;
	void* allocStart;

	int allocFailed;
	int workspaceOversizedDuration;
	ZSTD_cwksp_alloc_phase_e phase;
	} ZSTD_cwksp;

	/-************************************
	* Functions
	***************************************/

	MEM_STATIC size_t ZSTD_cwksp_available_space(ZSTD_cwksp* ws);

	MEM_STATIC void ZSTD_cwksp_assert_internal_consistency(ZSTD_cwksp* ws) {
	(void)ws;
	assert(ws->workspace <= ws->objectEnd);
	assert(ws->objectEnd <= ws->tableEnd);
	assert(ws->objectEnd <= ws->tableValidEnd);
	assert(ws->tableEnd <= ws->allocStart);
	assert(ws->tableValidEnd <= ws->allocStart);
	assert(ws->allocStart <= ws->workspaceEnd);
	}

	/**
	* Align must be a power of 2.
	*/
	MEM_STATIC size_t ZSTD_cwksp_align(size_t size, size_t const align) {
	size_t const mask = align - 1;
	assert((align & mask) == 0);
	return (size + mask) & ~mask;
	}

	/**
	* Use this to determine how much space in the workspace we will consume to
	* allocate this object. (Normally it should be exactly the size of the object,
	* but under special conditions, like ASAN, where we pad each object, it might
	* be larger.)
	*
	* Since tables aren't currently redzoned, you don't need to call through this
	* to figure out how much space you need for the matchState tables. Everything
	* else is though.
	*/
	MEM_STATIC size_t ZSTD_cwksp_alloc_size(size_t size) {
	#if defined (ADDRESS_SANITIZER) && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE)
	return size + 2 * ZSTD_CWKSP_ASAN_REDZONE_SIZE;
	#else
	return size;
	#endif
	}

	MEM_STATIC void ZSTD_cwksp_internal_advance_phase(
	ZSTD_cwksp* ws, ZSTD_cwksp_alloc_phase_e phase) {
	assert(phase >= ws->phase);
	if (phase > ws->phase) {
	if (ws->phase < ZSTD_cwksp_alloc_buffers &&
	phase >= ZSTD_cwksp_alloc_buffers) {
	ws->tableValidEnd = ws->objectEnd;
	}
	if (ws->phase < ZSTD_cwksp_alloc_aligned &&
	phase >= ZSTD_cwksp_alloc_aligned) {
	/* If unaligned allocations down from a too-large top have left us
	* unaligned, we need to realign our alloc ptr. Technically, this
	* can consume space that is unaccounted for in the neededSpace
	* calculation. However, I believe this can only happen when the
	* workspace is too large, and specifically when it is too large
	* by a larger margin than the space that will be consumed. */
	/* TODO: cleaner, compiler warning friendly way to do this??? */
	ws->allocStart = (BYTE*)ws->allocStart - ((size_t)ws->allocStart & (sizeof(U32)-1));
	if (ws->allocStart < ws->tableValidEnd) {
	ws->tableValidEnd = ws->allocStart;
	}
	}
	ws->phase = phase;
	}
	}

	/**
	* Returns whether this object/buffer/etc was allocated in this workspace.
	*/
	MEM_STATIC int ZSTD_cwksp_owns_buffer(const ZSTD_cwksp* ws, const void* ptr) {
	return (ptr != NULL) && (ws->workspace <= ptr) && (ptr <= ws->workspaceEnd);
	}

	/**
	* Internal function. Do not use directly.
	*/
	MEM_STATIC void* ZSTD_cwksp_reserve_internal(
	ZSTD_cwksp* ws, size_t bytes, ZSTD_cwksp_alloc_phase_e phase) {
	void* alloc;
	void* bottom = ws->tableEnd;
	ZSTD_cwksp_internal_advance_phase(ws, phase);
	alloc = (BYTE *)ws->allocStart - bytes;

	#if defined (ADDRESS_SANITIZER) && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE)
	/* over-reserve space */
	alloc = (BYTE )alloc - 2 ZSTD_CWKSP_ASAN_REDZONE_SIZE;
	#endif

	DEBUGLOG(5, "cwksp: reserving %p %zd bytes, %zd bytes remaining",
	alloc, bytes, ZSTD_cwksp_available_space(ws) - bytes);
	ZSTD_cwksp_assert_internal_consistency(ws);
	assert(alloc >= bottom);
	if (alloc < bottom) {
	DEBUGLOG(4, "cwksp: alloc failed!");
	ws->allocFailed = 1;
	return NULL;
	}
	if (alloc < ws->tableValidEnd) {
	ws->tableValidEnd = alloc;
	}
	ws->allocStart = alloc;

	#if defined (ADDRESS_SANITIZER) && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE)
	/* Move alloc so there's ZSTD_CWKSP_ASAN_REDZONE_SIZE unused space on
	* either size. */
	alloc = (BYTE *)alloc + ZSTD_CWKSP_ASAN_REDZONE_SIZE;
	__asan_unpoison_memory_region(alloc, bytes);
	#endif

	return alloc;
	}

	/**
	* Reserves and returns unaligned memory.
	*/
	MEM_STATIC BYTE* ZSTD_cwksp_reserve_buffer(ZSTD_cwksp* ws, size_t bytes) {
	return (BYTE*)ZSTD_cwksp_reserve_internal(ws, bytes, ZSTD_cwksp_alloc_buffers);
	}

	/**
	* Reserves and returns memory sized on and aligned on sizeof(unsigned).
	*/
	MEM_STATIC void* ZSTD_cwksp_reserve_aligned(ZSTD_cwksp* ws, size_t bytes) {
	assert((bytes & (sizeof(U32)-1)) == 0);
	return ZSTD_cwksp_reserve_internal(ws, ZSTD_cwksp_align(bytes, sizeof(U32)), ZSTD_cwksp_alloc_aligned);
	}

	/**
	* Aligned on sizeof(unsigned). These buffers have the special property that
	* their values remain constrained, allowing us to re-use them without
	* memset()-ing them.
	*/
	MEM_STATIC void* ZSTD_cwksp_reserve_table(ZSTD_cwksp* ws, size_t bytes) {
	const ZSTD_cwksp_alloc_phase_e phase = ZSTD_cwksp_alloc_aligned;
	void* alloc = ws->tableEnd;
	void* end = (BYTE *)alloc + bytes;
	void* top = ws->allocStart;

	DEBUGLOG(5, "cwksp: reserving %p table %zd bytes, %zd bytes remaining",
	alloc, bytes, ZSTD_cwksp_available_space(ws) - bytes);
	assert((bytes & (sizeof(U32)-1)) == 0);
	ZSTD_cwksp_internal_advance_phase(ws, phase);
	ZSTD_cwksp_assert_internal_consistency(ws);
	assert(end <= top);
	if (end > top) {
	DEBUGLOG(4, "cwksp: table alloc failed!");
	ws->allocFailed = 1;
	return NULL;
	}
	ws->tableEnd = end;

	#if defined (ADDRESS_SANITIZER) && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE)
	__asan_unpoison_memory_region(alloc, bytes);
	#endif

	return alloc;
	}

	/**
	* Aligned on sizeof(void*).
	*/
	MEM_STATIC void* ZSTD_cwksp_reserve_object(ZSTD_cwksp* ws, size_t bytes) {
	size_t roundedBytes = ZSTD_cwksp_align(bytes, sizeof(void*));
	void* alloc = ws->objectEnd;
	void* end = (BYTE*)alloc + roundedBytes;

	#if defined (ADDRESS_SANITIZER) && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE)
	/* over-reserve space */
	end = (BYTE )end + 2 ZSTD_CWKSP_ASAN_REDZONE_SIZE;
	#endif

	DEBUGLOG(5,
	"cwksp: reserving %p object %zd bytes (rounded to %zd), %zd bytes remaining",
	alloc, bytes, roundedBytes, ZSTD_cwksp_available_space(ws) - roundedBytes);
	assert(((size_t)alloc & (sizeof(void*)-1)) == 0);
	assert((bytes & (sizeof(void*)-1)) == 0);
	ZSTD_cwksp_assert_internal_consistency(ws);
	/* we must be in the first phase, no advance is possible */
	if (ws->phase != ZSTD_cwksp_alloc_objects \|\| end > ws->workspaceEnd) {
	DEBUGLOG(4, "cwksp: object alloc failed!");
	ws->allocFailed = 1;
	return NULL;
	}
	ws->objectEnd = end;
	ws->tableEnd = end;
	ws->tableValidEnd = end;

	#if defined (ADDRESS_SANITIZER) && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE)
	/* Move alloc so there's ZSTD_CWKSP_ASAN_REDZONE_SIZE unused space on
	* either size. */
	alloc = (BYTE *)alloc + ZSTD_CWKSP_ASAN_REDZONE_SIZE;
	__asan_unpoison_memory_region(alloc, bytes);
	#endif

	return alloc;
	}

	MEM_STATIC void ZSTD_cwksp_mark_tables_dirty(ZSTD_cwksp* ws) {
	DEBUGLOG(4, "cwksp: ZSTD_cwksp_mark_tables_dirty");

	#if defined (MEMORY_SANITIZER) && !defined (ZSTD_MSAN_DONT_POISON_WORKSPACE)
	/* To validate that the table re-use logic is sound, and that we don't
	* access table space that we haven't cleaned, we re-"poison" the table
	* space every time we mark it dirty. */
	{
	size_t size = (BYTE)ws->tableValidEnd - (BYTE)ws->objectEnd;
	assert(__msan_test_shadow(ws->objectEnd, size) == -1);
	__msan_poison(ws->objectEnd, size);
	}
	#endif

	assert(ws->tableValidEnd >= ws->objectEnd);
	assert(ws->tableValidEnd <= ws->allocStart);
	ws->tableValidEnd = ws->objectEnd;
	ZSTD_cwksp_assert_internal_consistency(ws);
	}

	MEM_STATIC void ZSTD_cwksp_mark_tables_clean(ZSTD_cwksp* ws) {
	DEBUGLOG(4, "cwksp: ZSTD_cwksp_mark_tables_clean");
	assert(ws->tableValidEnd >= ws->objectEnd);
	assert(ws->tableValidEnd <= ws->allocStart);
	if (ws->tableValidEnd < ws->tableEnd) {
	ws->tableValidEnd = ws->tableEnd;
	}
	ZSTD_cwksp_assert_internal_consistency(ws);
	}

	/**
	* Zero the part of the allocated tables not already marked clean.
	*/
	MEM_STATIC void ZSTD_cwksp_clean_tables(ZSTD_cwksp* ws) {
	DEBUGLOG(4, "cwksp: ZSTD_cwksp_clean_tables");
	assert(ws->tableValidEnd >= ws->objectEnd);
	assert(ws->tableValidEnd <= ws->allocStart);
	if (ws->tableValidEnd < ws->tableEnd) {
	memset(ws->tableValidEnd, 0, (BYTE)ws->tableEnd - (BYTE)ws->tableValidEnd);
	}
	ZSTD_cwksp_mark_tables_clean(ws);
	}

	/**
	* Invalidates table allocations.
	* All other allocations remain valid.
	*/
	MEM_STATIC void ZSTD_cwksp_clear_tables(ZSTD_cwksp* ws) {
	DEBUGLOG(4, "cwksp: clearing tables!");

	#if defined (ADDRESS_SANITIZER) && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE)
	{
	size_t size = (BYTE)ws->tableValidEnd - (BYTE)ws->objectEnd;
	__asan_poison_memory_region(ws->objectEnd, size);
	}
	#endif

	ws->tableEnd = ws->objectEnd;
	ZSTD_cwksp_assert_internal_consistency(ws);
	}

	/**
	* Invalidates all buffer, aligned, and table allocations.
	* Object allocations remain valid.
	*/
	MEM_STATIC void ZSTD_cwksp_clear(ZSTD_cwksp* ws) {
	DEBUGLOG(4, "cwksp: clearing!");

	#if defined (MEMORY_SANITIZER) && !defined (ZSTD_MSAN_DONT_POISON_WORKSPACE)
	/* To validate that the context re-use logic is sound, and that we don't
	* access stuff that this compression hasn't initialized, we re-"poison"
	* the workspace (or at least the non-static, non-table parts of it)
	* every time we start a new compression. */
	{
	size_t size = (BYTE)ws->workspaceEnd - (BYTE)ws->tableValidEnd;
	__msan_poison(ws->tableValidEnd, size);
	}
	#endif

	#if defined (ADDRESS_SANITIZER) && !defined (ZSTD_ASAN_DONT_POISON_WORKSPACE)
	{
	size_t size = (BYTE)ws->workspaceEnd - (BYTE)ws->objectEnd;
	__asan_poison_memory_region(ws->objectEnd, size);
	}
	#endif

	ws->tableEnd = ws->objectEnd;
	ws->allocStart = ws->workspaceEnd;
	ws->allocFailed = 0;
	if (ws->phase > ZSTD_cwksp_alloc_buffers) {
	ws->phase = ZSTD_cwksp_alloc_buffers;
	}
	ZSTD_cwksp_assert_internal_consistency(ws);
	}

	/**
	* The provided workspace takes ownership of the buffer [start, start+size).
	* Any existing values in the workspace are ignored (the previously managed
	* buffer, if present, must be separately freed).
	*/
	MEM_STATIC void ZSTD_cwksp_init(ZSTD_cwksp* ws, void* start, size_t size) {
	DEBUGLOG(4, "cwksp: init'ing workspace with %zd bytes", size);
	assert(((size_t)start & (sizeof(void)-1)) == 0); / ensure correct alignment */
	ws->workspace = start;
	ws->workspaceEnd = (BYTE*)start + size;
	ws->objectEnd = ws->workspace;
	ws->tableValidEnd = ws->objectEnd;
	ws->phase = ZSTD_cwksp_alloc_objects;
	ZSTD_cwksp_clear(ws);
	ws->workspaceOversizedDuration = 0;
	ZSTD_cwksp_assert_internal_consistency(ws);
	}

	MEM_STATIC size_t ZSTD_cwksp_create(ZSTD_cwksp* ws, size_t size, ZSTD_customMem customMem) {
	void* workspace = ZSTD_malloc(size, customMem);
	DEBUGLOG(4, "cwksp: creating new workspace with %zd bytes", size);
	RETURN_ERROR_IF(workspace == NULL, memory_allocation, "NULL pointer!");
	ZSTD_cwksp_init(ws, workspace, size);
	return 0;
	}

	MEM_STATIC void ZSTD_cwksp_free(ZSTD_cwksp* ws, ZSTD_customMem customMem) {
	void *ptr = ws->workspace;
	DEBUGLOG(4, "cwksp: freeing workspace");
	memset(ws, 0, sizeof(ZSTD_cwksp));
	ZSTD_free(ptr, customMem);
	}

	/**
	* Moves the management of a workspace from one cwksp to another. The src cwksp
	* is left in an invalid state (src must be re-init()'ed before its used again).
	*/
	MEM_STATIC void ZSTD_cwksp_move(ZSTD_cwksp* dst, ZSTD_cwksp* src) {
	dst = src;
	memset(src, 0, sizeof(ZSTD_cwksp));
	}

	MEM_STATIC size_t ZSTD_cwksp_sizeof(const ZSTD_cwksp* ws) {
	return (size_t)((BYTE)ws->workspaceEnd - (BYTE)ws->workspace);
	}

	MEM_STATIC int ZSTD_cwksp_reserve_failed(const ZSTD_cwksp* ws) {
	return ws->allocFailed;
	}

	/-************************************
	* Functions Checking Free Space
	***************************************/

	MEM_STATIC size_t ZSTD_cwksp_available_space(ZSTD_cwksp* ws) {
	return (size_t)((BYTE)ws->allocStart - (BYTE)ws->tableEnd);
	}

	MEM_STATIC int ZSTD_cwksp_check_available(ZSTD_cwksp* ws, size_t additionalNeededSpace) {
	return ZSTD_cwksp_available_space(ws) >= additionalNeededSpace;
	}

	MEM_STATIC int ZSTD_cwksp_check_too_large(ZSTD_cwksp* ws, size_t additionalNeededSpace) {
	return ZSTD_cwksp_check_available(
	ws, additionalNeededSpace * ZSTD_WORKSPACETOOLARGE_FACTOR);
	}

	MEM_STATIC int ZSTD_cwksp_check_wasteful(ZSTD_cwksp* ws, size_t additionalNeededSpace) {
	return ZSTD_cwksp_check_too_large(ws, additionalNeededSpace)
	&& ws->workspaceOversizedDuration > ZSTD_WORKSPACETOOLARGE_MAXDURATION;
	}

	MEM_STATIC void ZSTD_cwksp_bump_oversized_duration(
	ZSTD_cwksp* ws, size_t additionalNeededSpace) {
	if (ZSTD_cwksp_check_too_large(ws, additionalNeededSpace)) {
	ws->workspaceOversizedDuration++;
	} else {
	ws->workspaceOversizedDuration = 0;
	}
	}

	#if defined (__cplusplus)
	}
	#endif

	#endif /* ZSTD_CWKSP_H */
	/** ended inlining zstd_cwksp.h **/
	#ifdef ZSTD_MULTITHREAD
	/** start inlining zstdmt_compress.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTDMT_COMPRESS_H
	#define ZSTDMT_COMPRESS_H

	#if defined (__cplusplus)
	extern "C" {
	#endif


	/* Note : This is an internal API.
	* These APIs used to be exposed with ZSTDLIB_API,
	* because it used to be the only way to invoke MT compression.
	* Now, it's recommended to use ZSTD_compress2 and ZSTD_compressStream2()
	* instead.
	*
	* If you depend on these APIs and can't switch, then define
	* ZSTD_LEGACY_MULTITHREADED_API when making the dynamic library.
	* However, we may completely remove these functions in a future
	* release, so please switch soon.
	*
	* This API requires ZSTD_MULTITHREAD to be defined during compilation,
	* otherwise ZSTDMT_createCCtx*() will fail.
	*/

	#ifdef ZSTD_LEGACY_MULTITHREADED_API
	# define ZSTDMT_API ZSTDLIB_API
	#else
	# define ZSTDMT_API
	#endif

	/* === Dependencies === */
	#include <stddef.h> /* size_t */
	#define ZSTD_STATIC_LINKING_ONLY /* ZSTD_parameters */
	/** skipping file: ../zstd.h **/


	/* === Constants === */
	#ifndef ZSTDMT_NBWORKERS_MAX
	# define ZSTDMT_NBWORKERS_MAX 200
	#endif
	#ifndef ZSTDMT_JOBSIZE_MIN
	# define ZSTDMT_JOBSIZE_MIN (1 MB)
	#endif
	#define ZSTDMT_JOBLOG_MAX (MEM_32bits() ? 29 : 30)
	#define ZSTDMT_JOBSIZE_MAX (MEM_32bits() ? (512 MB) : (1024 MB))


	/* === Memory management === */
	typedef struct ZSTDMT_CCtx_s ZSTDMT_CCtx;
	/* Requires ZSTD_MULTITHREAD to be defined during compilation, otherwise it will return NULL. */
	ZSTDMT_API ZSTDMT_CCtx* ZSTDMT_createCCtx(unsigned nbWorkers);
	/* Requires ZSTD_MULTITHREAD to be defined during compilation, otherwise it will return NULL. */
	ZSTDMT_API ZSTDMT_CCtx* ZSTDMT_createCCtx_advanced(unsigned nbWorkers,
	ZSTD_customMem cMem);
	ZSTDMT_API size_t ZSTDMT_freeCCtx(ZSTDMT_CCtx* mtctx);

	ZSTDMT_API size_t ZSTDMT_sizeof_CCtx(ZSTDMT_CCtx* mtctx);


	/* === Simple one-pass compression function === */

	ZSTDMT_API size_t ZSTDMT_compressCCtx(ZSTDMT_CCtx* mtctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	int compressionLevel);



	/* === Streaming functions === */

	ZSTDMT_API size_t ZSTDMT_initCStream(ZSTDMT_CCtx* mtctx, int compressionLevel);
	ZSTDMT_API size_t ZSTDMT_resetCStream(ZSTDMT_CCtx* mtctx, unsigned long long pledgedSrcSize); /*< if srcSize is not known at reset time, use ZSTD_CONTENTSIZE_UNKNOWN. Note: for compatibility with older programs, 0 means the same as ZSTD_CONTENTSIZE_UNKNOWN, but it will change in the future to mean "empty" /

	ZSTDMT_API size_t ZSTDMT_nextInputSizeHint(const ZSTDMT_CCtx* mtctx);
	ZSTDMT_API size_t ZSTDMT_compressStream(ZSTDMT_CCtx* mtctx, ZSTD_outBuffer* output, ZSTD_inBuffer* input);

	ZSTDMT_API size_t ZSTDMT_flushStream(ZSTDMT_CCtx* mtctx, ZSTD_outBuffer* output); /*< @return : 0 == all flushed; >0 : still some data to be flushed; or an error code (ZSTD_isError()) /
	ZSTDMT_API size_t ZSTDMT_endStream(ZSTDMT_CCtx* mtctx, ZSTD_outBuffer* output); /*< @return : 0 == all flushed; >0 : still some data to be flushed; or an error code (ZSTD_isError()) /


	/* === Advanced functions and parameters === */

	ZSTDMT_API size_t ZSTDMT_compress_advanced(ZSTDMT_CCtx* mtctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const ZSTD_CDict* cdict,
	ZSTD_parameters params,
	int overlapLog);

	ZSTDMT_API size_t ZSTDMT_initCStream_advanced(ZSTDMT_CCtx* mtctx,
	const void* dict, size_t dictSize, /* dict can be released after init, a local copy is preserved within zcs */
	ZSTD_parameters params,
	unsigned long long pledgedSrcSize); /* pledgedSrcSize is optional and can be zero == unknown */

	ZSTDMT_API size_t ZSTDMT_initCStream_usingCDict(ZSTDMT_CCtx* mtctx,
	const ZSTD_CDict* cdict,
	ZSTD_frameParameters fparams,
	unsigned long long pledgedSrcSize); /* note : zero means empty */

	/* ZSTDMT_parameter :
	* List of parameters that can be set using ZSTDMT_setMTCtxParameter() */
	typedef enum {
	ZSTDMT_p_jobSize, /* Each job is compressed in parallel. By default, this value is dynamically determined depending on compression parameters. Can be set explicitly here. */
	ZSTDMT_p_overlapLog, /* Each job may reload a part of previous job to enhance compression ratio; 0 == no overlap, 6(default) == use 1/8th of window, >=9 == use full window. This is a "sticky" parameter : its value will be re-used on next compression job */
	ZSTDMT_p_rsyncable /* Enables rsyncable mode. */
	} ZSTDMT_parameter;

	/* ZSTDMT_setMTCtxParameter() :
	* allow setting individual parameters, one at a time, among a list of enums defined in ZSTDMT_parameter.
	* The function must be called typically after ZSTD_createCCtx() but __before ZSTDMT_init*() !__
	* Parameters not explicitly reset by ZSTDMT_init*() remain the same in consecutive compression sessions.
	* @return : 0, or an error code (which can be tested using ZSTD_isError()) */
	ZSTDMT_API size_t ZSTDMT_setMTCtxParameter(ZSTDMT_CCtx* mtctx, ZSTDMT_parameter parameter, int value);

	/* ZSTDMT_getMTCtxParameter() :
	* Query the ZSTDMT_CCtx for a parameter value.
	* @return : 0, or an error code (which can be tested using ZSTD_isError()) */
	ZSTDMT_API size_t ZSTDMT_getMTCtxParameter(ZSTDMT_CCtx* mtctx, ZSTDMT_parameter parameter, int* value);


	/*! ZSTDMT_compressStream_generic() :
	* Combines ZSTDMT_compressStream() with optional ZSTDMT_flushStream() or ZSTDMT_endStream()
	* depending on flush directive.
	* @return : minimum amount of data still to be flushed
	* 0 if fully flushed
	* or an error code
	* note : needs to be init using any ZSTD_initCStream() variant /
	ZSTDMT_API size_t ZSTDMT_compressStream_generic(ZSTDMT_CCtx* mtctx,
	ZSTD_outBuffer* output,
	ZSTD_inBuffer* input,
	ZSTD_EndDirective endOp);


	/* ========================================================
	* === Private interface, for use by ZSTD_compress.c ===
	* === Not exposed in libzstd. Never invoke directly ===
	* ======================================================== */

	/*! ZSTDMT_toFlushNow()
	* Tell how many bytes are ready to be flushed immediately.
	* Probe the oldest active job (not yet entirely flushed) and check its output buffer.
	* If return 0, it means there is no active job,
	* or, it means oldest job is still active, but everything produced has been flushed so far,
	* therefore flushing is limited by speed of oldest job. */
	size_t ZSTDMT_toFlushNow(ZSTDMT_CCtx* mtctx);

	/*! ZSTDMT_CCtxParam_setMTCtxParameter()
	* like ZSTDMT_setMTCtxParameter(), but into a ZSTD_CCtx_Params */
	size_t ZSTDMT_CCtxParam_setMTCtxParameter(ZSTD_CCtx_params* params, ZSTDMT_parameter parameter, int value);

	/*! ZSTDMT_CCtxParam_setNbWorkers()
	* Set nbWorkers, and clamp it.
	* Also reset jobSize and overlapLog */
	size_t ZSTDMT_CCtxParam_setNbWorkers(ZSTD_CCtx_params* params, unsigned nbWorkers);

	/*! ZSTDMT_updateCParams_whileCompressing() :
	* Updates only a selected set of compression parameters, to remain compatible with current frame.
	* New parameters will be applied to next compression job. */
	void ZSTDMT_updateCParams_whileCompressing(ZSTDMT_CCtx* mtctx, const ZSTD_CCtx_params* cctxParams);

	/*! ZSTDMT_getFrameProgression():
	* tells how much data has been consumed (input) and produced (output) for current frame.
	* able to count progression inside worker threads.
	*/
	ZSTD_frameProgression ZSTDMT_getFrameProgression(ZSTDMT_CCtx* mtctx);


	/*! ZSTDMT_initCStream_internal() :
	* Private use only. Init streaming operation.
	* expects params to be valid.
	* must receive dict, or cdict, or none, but not both.
	* @return : 0, or an error code */
	size_t ZSTDMT_initCStream_internal(ZSTDMT_CCtx* zcs,
	const void* dict, size_t dictSize, ZSTD_dictContentType_e dictContentType,
	const ZSTD_CDict* cdict,
	ZSTD_CCtx_params params, unsigned long long pledgedSrcSize);


	#if defined (__cplusplus)
	}
	#endif

	#endif /* ZSTDMT_COMPRESS_H */
	/** ended inlining zstdmt_compress.h **/
	#endif

	#if defined (__cplusplus)
	extern "C" {
	#endif


	/-************************************
	* Constants
	***************************************/
	#define kSearchStrength 8
	#define HASH_READ_SIZE 8
	#define ZSTD_DUBT_UNSORTED_MARK 1 /* For btlazy2 strategy, index ZSTD_DUBT_UNSORTED_MARK==1 means "unsorted".
	It could be confused for a real successor at index "1", if sorted as larger than its predecessor.
	It's not a big deal though : candidate will just be sorted again.
	Additionally, candidate position 1 will be lost.
	But candidate 1 cannot hide a large tree of candidates, so it's a minimal loss.
	The benefit is that ZSTD_DUBT_UNSORTED_MARK cannot be mishandled after table re-use with a different strategy.
	This constant is required by ZSTD_compressBlock_btlazy2() and ZSTD_reduceTable_internal() */


	/-************************************
	* Context memory management
	***************************************/
	typedef enum { ZSTDcs_created=0, ZSTDcs_init, ZSTDcs_ongoing, ZSTDcs_ending } ZSTD_compressionStage_e;
	typedef enum { zcss_init=0, zcss_load, zcss_flush } ZSTD_cStreamStage;

	typedef struct ZSTD_prefixDict_s {
	const void* dict;
	size_t dictSize;
	ZSTD_dictContentType_e dictContentType;
	} ZSTD_prefixDict;

	typedef struct {
	void* dictBuffer;
	void const* dict;
	size_t dictSize;
	ZSTD_dictContentType_e dictContentType;
	ZSTD_CDict* cdict;
	} ZSTD_localDict;

	typedef struct {
	U32 CTable[HUF_CTABLE_SIZE_U32(255)];
	HUF_repeat repeatMode;
	} ZSTD_hufCTables_t;

	typedef struct {
	FSE_CTable offcodeCTable[FSE_CTABLE_SIZE_U32(OffFSELog, MaxOff)];
	FSE_CTable matchlengthCTable[FSE_CTABLE_SIZE_U32(MLFSELog, MaxML)];
	FSE_CTable litlengthCTable[FSE_CTABLE_SIZE_U32(LLFSELog, MaxLL)];
	FSE_repeat offcode_repeatMode;
	FSE_repeat matchlength_repeatMode;
	FSE_repeat litlength_repeatMode;
	} ZSTD_fseCTables_t;

	typedef struct {
	ZSTD_hufCTables_t huf;
	ZSTD_fseCTables_t fse;
	} ZSTD_entropyCTables_t;

	typedef struct {
	U32 off;
	U32 len;
	} ZSTD_match_t;

	typedef struct {
	int price;
	U32 off;
	U32 mlen;
	U32 litlen;
	U32 rep[ZSTD_REP_NUM];
	} ZSTD_optimal_t;

	typedef enum { zop_dynamic=0, zop_predef } ZSTD_OptPrice_e;

	typedef struct {
	/* All tables are allocated inside cctx->workspace by ZSTD_resetCCtx_internal() */
	unsigned* litFreq; /* table of literals statistics, of size 256 */
	unsigned* litLengthFreq; /* table of litLength statistics, of size (MaxLL+1) */
	unsigned* matchLengthFreq; /* table of matchLength statistics, of size (MaxML+1) */
	unsigned* offCodeFreq; /* table of offCode statistics, of size (MaxOff+1) */
	ZSTD_match_t* matchTable; /* list of found matches, of size ZSTD_OPT_NUM+1 */
	ZSTD_optimal_t* priceTable; /* All positions tracked by optimal parser, of size ZSTD_OPT_NUM+1 */

	U32 litSum; /* nb of literals */
	U32 litLengthSum; /* nb of litLength codes */
	U32 matchLengthSum; /* nb of matchLength codes */
	U32 offCodeSum; /* nb of offset codes */
	U32 litSumBasePrice; /* to compare to log2(litfreq) */
	U32 litLengthSumBasePrice; /* to compare to log2(llfreq) */
	U32 matchLengthSumBasePrice;/* to compare to log2(mlfreq) */
	U32 offCodeSumBasePrice; /* to compare to log2(offreq) */
	ZSTD_OptPrice_e priceType; /* prices can be determined dynamically, or follow a pre-defined cost structure */
	const ZSTD_entropyCTables_t* symbolCosts; /* pre-calculated dictionary statistics */
	ZSTD_literalCompressionMode_e literalCompressionMode;
	} optState_t;

	typedef struct {
	ZSTD_entropyCTables_t entropy;
	U32 rep[ZSTD_REP_NUM];
	} ZSTD_compressedBlockState_t;

	typedef struct {
	BYTE const* nextSrc; /* next block here to continue on current prefix */
	BYTE const* base; /* All regular indexes relative to this position */
	BYTE const* dictBase; /* extDict indexes relative to this position */
	U32 dictLimit; /* below that point, need extDict */
	U32 lowLimit; /* below that point, no more valid data */
	} ZSTD_window_t;

	typedef struct ZSTD_matchState_t ZSTD_matchState_t;
	struct ZSTD_matchState_t {
	ZSTD_window_t window; /* State for window round buffer management */
	U32 loadedDictEnd; /* index of end of dictionary, within context's referential.
	* When loadedDictEnd != 0, a dictionary is in use, and still valid.
	* This relies on a mechanism to set loadedDictEnd=0 when dictionary is no longer within distance.
	* Such mechanism is provided within ZSTD_window_enforceMaxDist() and ZSTD_checkDictValidity().
	* When dict referential is copied into active context (i.e. not attached),
	* loadedDictEnd == dictSize, since referential starts from zero.
	*/
	U32 nextToUpdate; /* index from which to continue table update */
	U32 hashLog3; /* dispatch table for matches of len==3 : larger == faster, more memory */
	U32* hashTable;
	U32* hashTable3;
	U32* chainTable;
	optState_t opt; /* optimal parser state */
	const ZSTD_matchState_t* dictMatchState;
	ZSTD_compressionParameters cParams;
	};

	typedef struct {
	ZSTD_compressedBlockState_t* prevCBlock;
	ZSTD_compressedBlockState_t* nextCBlock;
	ZSTD_matchState_t matchState;
	} ZSTD_blockState_t;

	typedef struct {
	U32 offset;
	U32 checksum;
	} ldmEntry_t;

	typedef struct {
	ZSTD_window_t window; /* State for the window round buffer management */
	ldmEntry_t* hashTable;
	U32 loadedDictEnd;
	BYTE* bucketOffsets; /* Next position in bucket to insert entry */
	U64 hashPower; /* Used to compute the rolling hash.
	* Depends on ldmParams.minMatchLength */
	} ldmState_t;

	typedef struct {
	U32 enableLdm; /* 1 if enable long distance matching */
	U32 hashLog; /* Log size of hashTable */
	U32 bucketSizeLog; /* Log bucket size for collision resolution, at most 8 */
	U32 minMatchLength; /* Minimum match length */
	U32 hashRateLog; /* Log number of entries to skip */
	U32 windowLog; /* Window log for the LDM */
	} ldmParams_t;

	typedef struct {
	U32 offset;
	U32 litLength;
	U32 matchLength;
	} rawSeq;

	typedef struct {
	rawSeq* seq; /* The start of the sequences */
	size_t pos; /* The position where reading stopped. <= size. */
	size_t size; /* The number of sequences. <= capacity. */
	size_t capacity; /* The capacity starting from `seq` pointer */
	} rawSeqStore_t;

	typedef struct {
	int collectSequences;
	ZSTD_Sequence* seqStart;
	size_t seqIndex;
	size_t maxSequences;
	} SeqCollector;

	struct ZSTD_CCtx_params_s {
	ZSTD_format_e format;
	ZSTD_compressionParameters cParams;
	ZSTD_frameParameters fParams;

	int compressionLevel;
	int forceWindow; /* force back-references to respect limit of
	* 1<<wLog, even for dictionary */
	size_t targetCBlockSize; /* Tries to fit compressed block size to be around targetCBlockSize.
	* No target when targetCBlockSize == 0.
	* There is no guarantee on compressed block size */
	int srcSizeHint; /* User's best guess of source size.
	* Hint is not valid when srcSizeHint == 0.
	* There is no guarantee that hint is close to actual source size */

	ZSTD_dictAttachPref_e attachDictPref;
	ZSTD_literalCompressionMode_e literalCompressionMode;

	/* Multithreading: used to pass parameters to mtctx */
	int nbWorkers;
	size_t jobSize;
	int overlapLog;
	int rsyncable;

	/* Long distance matching parameters */
	ldmParams_t ldmParams;

	/* Internal use, for createCCtxParams() and freeCCtxParams() only */
	ZSTD_customMem customMem;
	}; /* typedef'd to ZSTD_CCtx_params within "zstd.h" */

	struct ZSTD_CCtx_s {
	ZSTD_compressionStage_e stage;
	int cParamsChanged; /* == 1 if cParams(except wlog) or compression level are changed in requestedParams. Triggers transmission of new params to ZSTDMT (if available) then reset to 0. */
	int bmi2; /* == 1 if the CPU supports BMI2 and 0 otherwise. CPU support is determined dynamically once per context lifetime. */
	ZSTD_CCtx_params requestedParams;
	ZSTD_CCtx_params appliedParams;
	U32 dictID;

	ZSTD_cwksp workspace; /* manages buffer for dynamic allocations */
	size_t blockSize;
	unsigned long long pledgedSrcSizePlusOne; /* this way, 0 (default) == unknown */
	unsigned long long consumedSrcSize;
	unsigned long long producedCSize;
	XXH64_state_t xxhState;
	ZSTD_customMem customMem;
	size_t staticSize;
	SeqCollector seqCollector;
	int isFirstBlock;
	int initialized;

	seqStore_t seqStore; /* sequences storage ptrs */
	ldmState_t ldmState; /* long distance matching state */
	rawSeq* ldmSequences; /* Storage for the ldm output sequences */
	size_t maxNbLdmSequences;
	rawSeqStore_t externSeqStore; /* Mutable reference to external sequences */
	ZSTD_blockState_t blockState;
	U32* entropyWorkspace; /* entropy workspace of HUF_WORKSPACE_SIZE bytes */

	/* streaming */
	char* inBuff;
	size_t inBuffSize;
	size_t inToCompress;
	size_t inBuffPos;
	size_t inBuffTarget;
	char* outBuff;
	size_t outBuffSize;
	size_t outBuffContentSize;
	size_t outBuffFlushedSize;
	ZSTD_cStreamStage streamStage;
	U32 frameEnded;

	/* Dictionary */
	ZSTD_localDict localDict;
	const ZSTD_CDict* cdict;
	ZSTD_prefixDict prefixDict; /* single-usage dictionary */

	/* Multi-threading */
	#ifdef ZSTD_MULTITHREAD
	ZSTDMT_CCtx* mtctx;
	#endif
	};

	typedef enum { ZSTD_dtlm_fast, ZSTD_dtlm_full } ZSTD_dictTableLoadMethod_e;

	typedef enum { ZSTD_noDict = 0, ZSTD_extDict = 1, ZSTD_dictMatchState = 2 } ZSTD_dictMode_e;


	typedef size_t (*ZSTD_blockCompressor) (
	ZSTD_matchState_t* bs, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);
	ZSTD_blockCompressor ZSTD_selectBlockCompressor(ZSTD_strategy strat, ZSTD_dictMode_e dictMode);


	MEM_STATIC U32 ZSTD_LLcode(U32 litLength)
	{
	static const BYTE LL_Code[64] = { 0, 1, 2, 3, 4, 5, 6, 7,
	8, 9, 10, 11, 12, 13, 14, 15,
	16, 16, 17, 17, 18, 18, 19, 19,
	20, 20, 20, 20, 21, 21, 21, 21,
	22, 22, 22, 22, 22, 22, 22, 22,
	23, 23, 23, 23, 23, 23, 23, 23,
	24, 24, 24, 24, 24, 24, 24, 24,
	24, 24, 24, 24, 24, 24, 24, 24 };
	static const U32 LL_deltaCode = 19;
	return (litLength > 63) ? ZSTD_highbit32(litLength) + LL_deltaCode : LL_Code[litLength];
	}

	/* ZSTD_MLcode() :
	* note : mlBase = matchLength - MINMATCH;
	* because it's the format it's stored in seqStore->sequences */
	MEM_STATIC U32 ZSTD_MLcode(U32 mlBase)
	{
	static const BYTE ML_Code[128] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
	16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
	32, 32, 33, 33, 34, 34, 35, 35, 36, 36, 36, 36, 37, 37, 37, 37,
	38, 38, 38, 38, 38, 38, 38, 38, 39, 39, 39, 39, 39, 39, 39, 39,
	40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40,
	41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41,
	42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42,
	42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42 };
	static const U32 ML_deltaCode = 36;
	return (mlBase > 127) ? ZSTD_highbit32(mlBase) + ML_deltaCode : ML_Code[mlBase];
	}

	typedef struct repcodes_s {
	U32 rep[3];
	} repcodes_t;

	MEM_STATIC repcodes_t ZSTD_updateRep(U32 const rep[3], U32 const offset, U32 const ll0)
	{
	repcodes_t newReps;
	if (offset >= ZSTD_REP_NUM) { /* full offset */
	newReps.rep[2] = rep[1];
	newReps.rep[1] = rep[0];
	newReps.rep[0] = offset - ZSTD_REP_MOVE;
	} else { /* repcode */
	U32 const repCode = offset + ll0;
	if (repCode > 0) { /* note : if repCode==0, no change */
	U32 const currentOffset = (repCode==ZSTD_REP_NUM) ? (rep[0] - 1) : rep[repCode];
	newReps.rep[2] = (repCode >= 2) ? rep[1] : rep[2];
	newReps.rep[1] = rep[0];
	newReps.rep[0] = currentOffset;
	} else { /* repCode == 0 */
	memcpy(&newReps, rep, sizeof(newReps));
	}
	}
	return newReps;
	}

	/* ZSTD_cParam_withinBounds:
	* @return 1 if value is within cParam bounds,
	* 0 otherwise */
	MEM_STATIC int ZSTD_cParam_withinBounds(ZSTD_cParameter cParam, int value)
	{
	ZSTD_bounds const bounds = ZSTD_cParam_getBounds(cParam);
	if (ZSTD_isError(bounds.error)) return 0;
	if (value < bounds.lowerBound) return 0;
	if (value > bounds.upperBound) return 0;
	return 1;
	}

	/* ZSTD_noCompressBlock() :
	* Writes uncompressed block to dst buffer from given src.
	* Returns the size of the block */
	MEM_STATIC size_t ZSTD_noCompressBlock (void* dst, size_t dstCapacity, const void* src, size_t srcSize, U32 lastBlock)
	{
	U32 const cBlockHeader24 = lastBlock + (((U32)bt_raw)<<1) + (U32)(srcSize << 3);
	RETURN_ERROR_IF(srcSize + ZSTD_blockHeaderSize > dstCapacity,
	dstSize_tooSmall, "dst buf too small for uncompressed block");
	MEM_writeLE24(dst, cBlockHeader24);
	memcpy((BYTE*)dst + ZSTD_blockHeaderSize, src, srcSize);
	return ZSTD_blockHeaderSize + srcSize;
	}

	MEM_STATIC size_t ZSTD_rleCompressBlock (void* dst, size_t dstCapacity, BYTE src, size_t srcSize, U32 lastBlock)
	{
	BYTE* const op = (BYTE*)dst;
	U32 const cBlockHeader = lastBlock + (((U32)bt_rle)<<1) + (U32)(srcSize << 3);
	RETURN_ERROR_IF(dstCapacity < 4, dstSize_tooSmall, "");
	MEM_writeLE24(op, cBlockHeader);
	op[3] = src;
	return 4;
	}


	/* ZSTD_minGain() :
	* minimum compression required
	* to generate a compress block or a compressed literals section.
	* note : use same formula for both situations */
	MEM_STATIC size_t ZSTD_minGain(size_t srcSize, ZSTD_strategy strat)
	{
	U32 const minlog = (strat>=ZSTD_btultra) ? (U32)(strat) - 1 : 6;
	ZSTD_STATIC_ASSERT(ZSTD_btultra == 8);
	assert(ZSTD_cParam_withinBounds(ZSTD_c_strategy, strat));
	return (srcSize >> minlog) + 2;
	}

	MEM_STATIC int ZSTD_disableLiteralsCompression(const ZSTD_CCtx_params* cctxParams)
	{
	switch (cctxParams->literalCompressionMode) {
	case ZSTD_lcm_huffman:
	return 0;
	case ZSTD_lcm_uncompressed:
	return 1;
	default:
	assert(0 /* impossible: pre-validated */);
	/* fall-through */
	case ZSTD_lcm_auto:
	return (cctxParams->cParams.strategy == ZSTD_fast) && (cctxParams->cParams.targetLength > 0);
	}
	}

	/*! ZSTD_safecopyLiterals() :
	* memcpy() function that won't read beyond more than WILDCOPY_OVERLENGTH bytes past ilimit_w.
	* Only called when the sequence ends past ilimit_w, so it only needs to be optimized for single
	* large copies.
	*/
	static void ZSTD_safecopyLiterals(BYTE* op, BYTE const* ip, BYTE const* const iend, BYTE const* ilimit_w) {
	assert(iend > ilimit_w);
	if (ip <= ilimit_w) {
	ZSTD_wildcopy(op, ip, ilimit_w - ip, ZSTD_no_overlap);
	op += ilimit_w - ip;
	ip = ilimit_w;
	}
	while (ip < iend) op++ = ip++;
	}

	/*! ZSTD_storeSeq() :
	* Store a sequence (litlen, litPtr, offCode and mlBase) into seqStore_t.
	* `offCode` : distance to match + ZSTD_REP_MOVE (values <= ZSTD_REP_MOVE are repCodes).
	* `mlBase` : matchLength - MINMATCH
	* Allowed to overread literals up to litLimit.
	*/
	HINT_INLINE UNUSED_ATTR
	void ZSTD_storeSeq(seqStore_t* seqStorePtr, size_t litLength, const BYTE* literals, const BYTE* litLimit, U32 offCode, size_t mlBase)
	{
	BYTE const* const litLimit_w = litLimit - WILDCOPY_OVERLENGTH;
	BYTE const* const litEnd = literals + litLength;
	#if defined(DEBUGLEVEL) && (DEBUGLEVEL >= 6)
	static const BYTE* g_start = NULL;
	if (g_start==NULL) g_start = (const BYTE)literals; / note : index only works for compression within a single segment */
	{ U32 const pos = (U32)((const BYTE*)literals - g_start);
	DEBUGLOG(6, "Cpos%7u :%3u literals, match%4u bytes at offCode%7u",
	pos, (U32)litLength, (U32)mlBase+MINMATCH, (U32)offCode);
	}
	#endif
	assert((size_t)(seqStorePtr->sequences - seqStorePtr->sequencesStart) < seqStorePtr->maxNbSeq);
	/* copy Literals */
	assert(seqStorePtr->maxNbLit <= 128 KB);
	assert(seqStorePtr->lit + litLength <= seqStorePtr->litStart + seqStorePtr->maxNbLit);
	assert(literals + litLength <= litLimit);
	if (litEnd <= litLimit_w) {
	/* Common case we can use wildcopy.
	* First copy 16 bytes, because literals are likely short.
	*/
	assert(WILDCOPY_OVERLENGTH >= 16);
	ZSTD_copy16(seqStorePtr->lit, literals);
	if (litLength > 16) {
	ZSTD_wildcopy(seqStorePtr->lit+16, literals+16, (ptrdiff_t)litLength-16, ZSTD_no_overlap);
	}
	} else {
	ZSTD_safecopyLiterals(seqStorePtr->lit, literals, litEnd, litLimit_w);
	}
	seqStorePtr->lit += litLength;

	/* literal Length */
	if (litLength>0xFFFF) {
	assert(seqStorePtr->longLengthID == 0); /* there can only be a single long length */
	seqStorePtr->longLengthID = 1;
	seqStorePtr->longLengthPos = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart);
	}
	seqStorePtr->sequences[0].litLength = (U16)litLength;

	/* match offset */
	seqStorePtr->sequences[0].offset = offCode + 1;

	/* match Length */
	if (mlBase>0xFFFF) {
	assert(seqStorePtr->longLengthID == 0); /* there can only be a single long length */
	seqStorePtr->longLengthID = 2;
	seqStorePtr->longLengthPos = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart);
	}
	seqStorePtr->sequences[0].matchLength = (U16)mlBase;

	seqStorePtr->sequences++;
	}


	/-************************************
	* Match length counter
	***************************************/
	static unsigned ZSTD_NbCommonBytes (size_t val)
	{
	if (MEM_isLittleEndian()) {
	if (MEM_64bits()) {
	# if defined(_MSC_VER) && defined(_WIN64)
	unsigned long r = 0;
	return _BitScanForward64( &r, (U64)val ) ? (unsigned)(r >> 3) : 0;
	# elif defined(__GNUC__) && (__GNUC__ >= 4)
	return (__builtin_ctzll((U64)val) >> 3);
	# else
	static const int DeBruijnBytePos[64] = { 0, 0, 0, 0, 0, 1, 1, 2,
	0, 3, 1, 3, 1, 4, 2, 7,
	0, 2, 3, 6, 1, 5, 3, 5,
	1, 3, 4, 4, 2, 5, 6, 7,
	7, 0, 1, 2, 3, 3, 4, 6,
	2, 6, 5, 5, 3, 4, 5, 6,
	7, 1, 2, 4, 6, 4, 4, 5,
	7, 2, 6, 5, 7, 6, 7, 7 };
	return DeBruijnBytePos[((U64)((val & -(long long)val) * 0x0218A392CDABBD3FULL)) >> 58];
	# endif
	} else { /* 32 bits */
	# if defined(_MSC_VER)
	unsigned long r=0;
	return _BitScanForward( &r, (U32)val ) ? (unsigned)(r >> 3) : 0;
	# elif defined(__GNUC__) && (__GNUC__ >= 3)
	return (__builtin_ctz((U32)val) >> 3);
	# else
	static const int DeBruijnBytePos[32] = { 0, 0, 3, 0, 3, 1, 3, 0,
	3, 2, 2, 1, 3, 2, 0, 1,
	3, 3, 1, 2, 2, 2, 2, 0,
	3, 1, 2, 0, 1, 0, 1, 1 };
	return DeBruijnBytePos[((U32)((val & -(S32)val) * 0x077CB531U)) >> 27];
	# endif
	}
	} else { /* Big Endian CPU */
	if (MEM_64bits()) {
	# if defined(_MSC_VER) && defined(_WIN64)
	unsigned long r = 0;
	return _BitScanReverse64( &r, val ) ? (unsigned)(r >> 3) : 0;
	# elif defined(__GNUC__) && (__GNUC__ >= 4)
	return (__builtin_clzll(val) >> 3);
	# else
	unsigned r;
	const unsigned n32 = sizeof(size_t)4; / calculate this way due to compiler complaining in 32-bits mode */
	if (!(val>>n32)) { r=4; } else { r=0; val>>=n32; }
	if (!(val>>16)) { r+=2; val>>=8; } else { val>>=24; }
	r += (!val);
	return r;
	# endif
	} else { /* 32 bits */
	# if defined(_MSC_VER)
	unsigned long r = 0;
	return _BitScanReverse( &r, (unsigned long)val ) ? (unsigned)(r >> 3) : 0;
	# elif defined(__GNUC__) && (__GNUC__ >= 3)
	return (__builtin_clz((U32)val) >> 3);
	# else
	unsigned r;
	if (!(val>>16)) { r=2; val>>=8; } else { r=0; val>>=24; }
	r += (!val);
	return r;
	# endif
	} }
	}


	MEM_STATIC size_t ZSTD_count(const BYTE* pIn, const BYTE* pMatch, const BYTE* const pInLimit)
	{
	const BYTE* const pStart = pIn;
	const BYTE* const pInLoopLimit = pInLimit - (sizeof(size_t)-1);

	if (pIn < pInLoopLimit) {
	{ size_t const diff = MEM_readST(pMatch) ^ MEM_readST(pIn);
	if (diff) return ZSTD_NbCommonBytes(diff); }
	pIn+=sizeof(size_t); pMatch+=sizeof(size_t);
	while (pIn < pInLoopLimit) {
	size_t const diff = MEM_readST(pMatch) ^ MEM_readST(pIn);
	if (!diff) { pIn+=sizeof(size_t); pMatch+=sizeof(size_t); continue; }
	pIn += ZSTD_NbCommonBytes(diff);
	return (size_t)(pIn - pStart);
	} }
	if (MEM_64bits() && (pIn<(pInLimit-3)) && (MEM_read32(pMatch) == MEM_read32(pIn))) { pIn+=4; pMatch+=4; }
	if ((pIn<(pInLimit-1)) && (MEM_read16(pMatch) == MEM_read16(pIn))) { pIn+=2; pMatch+=2; }
	if ((pIn<pInLimit) && (pMatch == pIn)) pIn++;
	return (size_t)(pIn - pStart);
	}

	/** ZSTD_count_2segments() :
	* can count match length with `ip` & `match` in 2 different segments.
	* convention : on reaching mEnd, match count continue starting from iStart
	*/
	MEM_STATIC size_t
	ZSTD_count_2segments(const BYTE* ip, const BYTE* match,
	const BYTE* iEnd, const BYTE* mEnd, const BYTE* iStart)
	{
	const BYTE* const vEnd = MIN( ip + (mEnd - match), iEnd);
	size_t const matchLength = ZSTD_count(ip, match, vEnd);
	if (match + matchLength != mEnd) return matchLength;
	DEBUGLOG(7, "ZSTD_count_2segments: found a 2-parts match (current length==%zu)", matchLength);
	DEBUGLOG(7, "distance from match beginning to end dictionary = %zi", mEnd - match);
	DEBUGLOG(7, "distance from current pos to end buffer = %zi", iEnd - ip);
	DEBUGLOG(7, "next byte : ip==%02X, istart==%02X", ip[matchLength], *iStart);
	DEBUGLOG(7, "final match length = %zu", matchLength + ZSTD_count(ip+matchLength, iStart, iEnd));
	return matchLength + ZSTD_count(ip+matchLength, iStart, iEnd);
	}


	/-************************************
	* Hashes
	***************************************/
	static const U32 prime3bytes = 506832829U;
	static U32 ZSTD_hash3(U32 u, U32 h) { return ((u << (32-24)) * prime3bytes) >> (32-h) ; }
	MEM_STATIC size_t ZSTD_hash3Ptr(const void* ptr, U32 h) { return ZSTD_hash3(MEM_readLE32(ptr), h); } /* only in zstd_opt.h */

	static const U32 prime4bytes = 2654435761U;
	static U32 ZSTD_hash4(U32 u, U32 h) { return (u * prime4bytes) >> (32-h) ; }
	static size_t ZSTD_hash4Ptr(const void* ptr, U32 h) { return ZSTD_hash4(MEM_read32(ptr), h); }

	static const U64 prime5bytes = 889523592379ULL;
	static size_t ZSTD_hash5(U64 u, U32 h) { return (size_t)(((u << (64-40)) * prime5bytes) >> (64-h)) ; }
	static size_t ZSTD_hash5Ptr(const void* p, U32 h) { return ZSTD_hash5(MEM_readLE64(p), h); }

	static const U64 prime6bytes = 227718039650203ULL;
	static size_t ZSTD_hash6(U64 u, U32 h) { return (size_t)(((u << (64-48)) * prime6bytes) >> (64-h)) ; }
	static size_t ZSTD_hash6Ptr(const void* p, U32 h) { return ZSTD_hash6(MEM_readLE64(p), h); }

	static const U64 prime7bytes = 58295818150454627ULL;
	static size_t ZSTD_hash7(U64 u, U32 h) { return (size_t)(((u << (64-56)) * prime7bytes) >> (64-h)) ; }
	static size_t ZSTD_hash7Ptr(const void* p, U32 h) { return ZSTD_hash7(MEM_readLE64(p), h); }

	static const U64 prime8bytes = 0xCF1BBCDCB7A56463ULL;
	static size_t ZSTD_hash8(U64 u, U32 h) { return (size_t)(((u) * prime8bytes) >> (64-h)) ; }
	static size_t ZSTD_hash8Ptr(const void* p, U32 h) { return ZSTD_hash8(MEM_readLE64(p), h); }

	MEM_STATIC size_t ZSTD_hashPtr(const void* p, U32 hBits, U32 mls)
	{
	switch(mls)
	{
	default:
	case 4: return ZSTD_hash4Ptr(p, hBits);
	case 5: return ZSTD_hash5Ptr(p, hBits);
	case 6: return ZSTD_hash6Ptr(p, hBits);
	case 7: return ZSTD_hash7Ptr(p, hBits);
	case 8: return ZSTD_hash8Ptr(p, hBits);
	}
	}

	/** ZSTD_ipow() :
	* Return base^exponent.
	*/
	static U64 ZSTD_ipow(U64 base, U64 exponent)
	{
	U64 power = 1;
	while (exponent) {
	if (exponent & 1) power *= base;
	exponent >>= 1;
	base *= base;
	}
	return power;
	}

	#define ZSTD_ROLL_HASH_CHAR_OFFSET 10

	/** ZSTD_rollingHash_append() :
	* Add the buffer to the hash value.
	*/
	static U64 ZSTD_rollingHash_append(U64 hash, void const* buf, size_t size)
	{
	BYTE const* istart = (BYTE const*)buf;
	size_t pos;
	for (pos = 0; pos < size; ++pos) {
	hash *= prime8bytes;
	hash += istart[pos] + ZSTD_ROLL_HASH_CHAR_OFFSET;
	}
	return hash;
	}

	/** ZSTD_rollingHash_compute() :
	* Compute the rolling hash value of the buffer.
	*/
	MEM_STATIC U64 ZSTD_rollingHash_compute(void const* buf, size_t size)
	{
	return ZSTD_rollingHash_append(0, buf, size);
	}

	/** ZSTD_rollingHash_primePower() :
	* Compute the primePower to be passed to ZSTD_rollingHash_rotate() for a hash
	* over a window of length bytes.
	*/
	MEM_STATIC U64 ZSTD_rollingHash_primePower(U32 length)
	{
	return ZSTD_ipow(prime8bytes, length - 1);
	}

	/** ZSTD_rollingHash_rotate() :
	* Rotate the rolling hash by one byte.
	*/
	MEM_STATIC U64 ZSTD_rollingHash_rotate(U64 hash, BYTE toRemove, BYTE toAdd, U64 primePower)
	{
	hash -= (toRemove + ZSTD_ROLL_HASH_CHAR_OFFSET) * primePower;
	hash *= prime8bytes;
	hash += toAdd + ZSTD_ROLL_HASH_CHAR_OFFSET;
	return hash;
	}

	/-************************************
	* Round buffer management
	***************************************/
	#if (ZSTD_WINDOWLOG_MAX_64 > 31)
	# error "ZSTD_WINDOWLOG_MAX is too large : would overflow ZSTD_CURRENT_MAX"
	#endif
	/* Max current allowed */
	#define ZSTD_CURRENT_MAX ((3U << 29) + (1U << ZSTD_WINDOWLOG_MAX))
	/* Maximum chunk size before overflow correction needs to be called again */
	#define ZSTD_CHUNKSIZE_MAX \
	( ((U32)-1) /* Maximum ending current index */ \
	- ZSTD_CURRENT_MAX) /* Maximum beginning lowLimit */

	/**
	* ZSTD_window_clear():
	* Clears the window containing the history by simply setting it to empty.
	*/
	MEM_STATIC void ZSTD_window_clear(ZSTD_window_t* window)
	{
	size_t const endT = (size_t)(window->nextSrc - window->base);
	U32 const end = (U32)endT;

	window->lowLimit = end;
	window->dictLimit = end;
	}

	/**
	* ZSTD_window_hasExtDict():
	* Returns non-zero if the window has a non-empty extDict.
	*/
	MEM_STATIC U32 ZSTD_window_hasExtDict(ZSTD_window_t const window)
	{
	return window.lowLimit < window.dictLimit;
	}

	/**
	* ZSTD_matchState_dictMode():
	* Inspects the provided matchState and figures out what dictMode should be
	* passed to the compressor.
	*/
	MEM_STATIC ZSTD_dictMode_e ZSTD_matchState_dictMode(const ZSTD_matchState_t *ms)
	{
	return ZSTD_window_hasExtDict(ms->window) ?
	ZSTD_extDict :
	ms->dictMatchState != NULL ?
	ZSTD_dictMatchState :
	ZSTD_noDict;
	}

	/**
	* ZSTD_window_needOverflowCorrection():
	* Returns non-zero if the indices are getting too large and need overflow
	* protection.
	*/
	MEM_STATIC U32 ZSTD_window_needOverflowCorrection(ZSTD_window_t const window,
	void const* srcEnd)
	{
	U32 const current = (U32)((BYTE const*)srcEnd - window.base);
	return current > ZSTD_CURRENT_MAX;
	}

	/**
	* ZSTD_window_correctOverflow():
	* Reduces the indices to protect from index overflow.
	* Returns the correction made to the indices, which must be applied to every
	* stored index.
	*
	* The least significant cycleLog bits of the indices must remain the same,
	* which may be 0. Every index up to maxDist in the past must be valid.
	* NOTE: (maxDist & cycleMask) must be zero.
	*/
	MEM_STATIC U32 ZSTD_window_correctOverflow(ZSTD_window_t* window, U32 cycleLog,
	U32 maxDist, void const* src)
	{
	/* preemptive overflow correction:
	* 1. correction is large enough:
	* lowLimit > (3<<29) ==> current > 3<<29 + 1<<windowLog
	* 1<<windowLog <= newCurrent < 1<<chainLog + 1<<windowLog
	*
	* current - newCurrent
	* > (3<<29 + 1<<windowLog) - (1<<windowLog + 1<<chainLog)
	* > (3<<29) - (1<<chainLog)
	* > (3<<29) - (1<<30) (NOTE: chainLog <= 30)
	* > 1<<29
	*
	* 2. (ip+ZSTD_CHUNKSIZE_MAX - cctx->base) doesn't overflow:
	* After correction, current is less than (1<<chainLog + 1<<windowLog).
	* In 64-bit mode we are safe, because we have 64-bit ptrdiff_t.
	* In 32-bit mode we are safe, because (chainLog <= 29), so
	* ip+ZSTD_CHUNKSIZE_MAX - cctx->base < 1<<32.
	* 3. (cctx->lowLimit + 1<<windowLog) < 1<<32:
	* windowLog <= 31 ==> 3<<29 + 1<<windowLog < 7<<29 < 1<<32.
	*/
	U32 const cycleMask = (1U << cycleLog) - 1;
	U32 const current = (U32)((BYTE const*)src - window->base);
	U32 const currentCycle0 = current & cycleMask;
	/* Exclude zero so that newCurrent - maxDist >= 1. */
	U32 const currentCycle1 = currentCycle0 == 0 ? (1U << cycleLog) : currentCycle0;
	U32 const newCurrent = currentCycle1 + maxDist;
	U32 const correction = current - newCurrent;
	assert((maxDist & cycleMask) == 0);
	assert(current > newCurrent);
	/* Loose bound, should be around 1<<29 (see above) */
	assert(correction > 1<<28);

	window->base += correction;
	window->dictBase += correction;
	if (window->lowLimit <= correction) window->lowLimit = 1;
	else window->lowLimit -= correction;
	if (window->dictLimit <= correction) window->dictLimit = 1;
	else window->dictLimit -= correction;

	/* Ensure we can still reference the full window. */
	assert(newCurrent >= maxDist);
	assert(newCurrent - maxDist >= 1);
	/* Ensure that lowLimit and dictLimit didn't underflow. */
	assert(window->lowLimit <= newCurrent);
	assert(window->dictLimit <= newCurrent);

	DEBUGLOG(4, "Correction of 0x%x bytes to lowLimit=0x%x", correction,
	window->lowLimit);
	return correction;
	}

	/**
	* ZSTD_window_enforceMaxDist():
	* Updates lowLimit so that:
	* (srcEnd - base) - lowLimit == maxDist + loadedDictEnd
	*
	* It ensures index is valid as long as index >= lowLimit.
	* This must be called before a block compression call.
	*
	* loadedDictEnd is only defined if a dictionary is in use for current compression.
	* As the name implies, loadedDictEnd represents the index at end of dictionary.
	* The value lies within context's referential, it can be directly compared to blockEndIdx.
	*
	* If loadedDictEndPtr is NULL, no dictionary is in use, and we use loadedDictEnd == 0.
	* If loadedDictEndPtr is not NULL, we set it to zero after updating lowLimit.
	* This is because dictionaries are allowed to be referenced fully
	* as long as the last byte of the dictionary is in the window.
	* Once input has progressed beyond window size, dictionary cannot be referenced anymore.
	*
	* In normal dict mode, the dictionary lies between lowLimit and dictLimit.
	* In dictMatchState mode, lowLimit and dictLimit are the same,
	* and the dictionary is below them.
	* forceWindow and dictMatchState are therefore incompatible.
	*/
	MEM_STATIC void
	ZSTD_window_enforceMaxDist(ZSTD_window_t* window,
	const void* blockEnd,
	U32 maxDist,
	U32* loadedDictEndPtr,
	const ZSTD_matchState_t** dictMatchStatePtr)
	{
	U32 const blockEndIdx = (U32)((BYTE const*)blockEnd - window->base);
	U32 const loadedDictEnd = (loadedDictEndPtr != NULL) ? *loadedDictEndPtr : 0;
	DEBUGLOG(5, "ZSTD_window_enforceMaxDist: blockEndIdx=%u, maxDist=%u, loadedDictEnd=%u",
	(unsigned)blockEndIdx, (unsigned)maxDist, (unsigned)loadedDictEnd);

	/* - When there is no dictionary : loadedDictEnd == 0.
	In which case, the test (blockEndIdx > maxDist) is merely to avoid
	overflowing next operation `newLowLimit = blockEndIdx - maxDist`.
	- When there is a standard dictionary :
	Index referential is copied from the dictionary,
	which means it starts from 0.
	In which case, loadedDictEnd == dictSize,
	and it makes sense to compare `blockEndIdx > maxDist + dictSize`
	since `blockEndIdx` also starts from zero.
	- When there is an attached dictionary :
	loadedDictEnd is expressed within the referential of the context,
	so it can be directly compared against blockEndIdx.
	*/
	if (blockEndIdx > maxDist + loadedDictEnd) {
	U32 const newLowLimit = blockEndIdx - maxDist;
	if (window->lowLimit < newLowLimit) window->lowLimit = newLowLimit;
	if (window->dictLimit < window->lowLimit) {
	DEBUGLOG(5, "Update dictLimit to match lowLimit, from %u to %u",
	(unsigned)window->dictLimit, (unsigned)window->lowLimit);
	window->dictLimit = window->lowLimit;
	}
	/* On reaching window size, dictionaries are invalidated */
	if (loadedDictEndPtr) *loadedDictEndPtr = 0;
	if (dictMatchStatePtr) *dictMatchStatePtr = NULL;
	}
	}

	/* Similar to ZSTD_window_enforceMaxDist(),
	* but only invalidates dictionary
	* when input progresses beyond window size.
	* assumption : loadedDictEndPtr and dictMatchStatePtr are valid (non NULL)
	* loadedDictEnd uses same referential as window->base
	* maxDist is the window size */
	MEM_STATIC void
	ZSTD_checkDictValidity(const ZSTD_window_t* window,
	const void* blockEnd,
	U32 maxDist,
	U32* loadedDictEndPtr,
	const ZSTD_matchState_t** dictMatchStatePtr)
	{
	assert(loadedDictEndPtr != NULL);
	assert(dictMatchStatePtr != NULL);
	{ U32 const blockEndIdx = (U32)((BYTE const*)blockEnd - window->base);
	U32 const loadedDictEnd = *loadedDictEndPtr;
	DEBUGLOG(5, "ZSTD_checkDictValidity: blockEndIdx=%u, maxDist=%u, loadedDictEnd=%u",
	(unsigned)blockEndIdx, (unsigned)maxDist, (unsigned)loadedDictEnd);
	assert(blockEndIdx >= loadedDictEnd);

	if (blockEndIdx > loadedDictEnd + maxDist) {
	/* On reaching window size, dictionaries are invalidated.
	* For simplification, if window size is reached anywhere within next block,
	* the dictionary is invalidated for the full block.
	*/
	DEBUGLOG(6, "invalidating dictionary for current block (distance > windowSize)");
	*loadedDictEndPtr = 0;
	*dictMatchStatePtr = NULL;
	} else {
	if (*loadedDictEndPtr != 0) {
	DEBUGLOG(6, "dictionary considered valid for current block");
	} } }
	}

	MEM_STATIC void ZSTD_window_init(ZSTD_window_t* window) {
	memset(window, 0, sizeof(*window));
	window->base = (BYTE const*)"";
	window->dictBase = (BYTE const*)"";
	window->dictLimit = 1; /* start from 1, so that 1st position is valid */
	window->lowLimit = 1; /* it ensures first and later CCtx usages compress the same */
	window->nextSrc = window->base + 1; /* see issue #1241 */
	}

	/**
	* ZSTD_window_update():
	* Updates the window by appending [src, src + srcSize) to the window.
	* If it is not contiguous, the current prefix becomes the extDict, and we
	* forget about the extDict. Handles overlap of the prefix and extDict.
	* Returns non-zero if the segment is contiguous.
	*/
	MEM_STATIC U32 ZSTD_window_update(ZSTD_window_t* window,
	void const* src, size_t srcSize)
	{
	BYTE const* const ip = (BYTE const*)src;
	U32 contiguous = 1;
	DEBUGLOG(5, "ZSTD_window_update");
	if (srcSize == 0)
	return contiguous;
	assert(window->base != NULL);
	assert(window->dictBase != NULL);
	/* Check if blocks follow each other */
	if (src != window->nextSrc) {
	/* not contiguous */
	size_t const distanceFromBase = (size_t)(window->nextSrc - window->base);
	DEBUGLOG(5, "Non contiguous blocks, new segment starts at %u", window->dictLimit);
	window->lowLimit = window->dictLimit;
	assert(distanceFromBase == (size_t)(U32)distanceFromBase); /* should never overflow */
	window->dictLimit = (U32)distanceFromBase;
	window->dictBase = window->base;
	window->base = ip - distanceFromBase;
	/* ms->nextToUpdate = window->dictLimit; */
	if (window->dictLimit - window->lowLimit < HASH_READ_SIZE) window->lowLimit = window->dictLimit; /* too small extDict */
	contiguous = 0;
	}
	window->nextSrc = ip + srcSize;
	/* if input and dictionary overlap : reduce dictionary (area presumed modified by input) */
	if ( (ip+srcSize > window->dictBase + window->lowLimit)
	& (ip < window->dictBase + window->dictLimit)) {
	ptrdiff_t const highInputIdx = (ip + srcSize) - window->dictBase;
	U32 const lowLimitMax = (highInputIdx > (ptrdiff_t)window->dictLimit) ? window->dictLimit : (U32)highInputIdx;
	window->lowLimit = lowLimitMax;
	DEBUGLOG(5, "Overlapping extDict and input : new lowLimit = %u", window->lowLimit);
	}
	return contiguous;
	}

	/**
	* Returns the lowest allowed match index. It may either be in the ext-dict or the prefix.
	*/
	MEM_STATIC U32 ZSTD_getLowestMatchIndex(const ZSTD_matchState_t* ms, U32 current, unsigned windowLog)
	{
	U32 const maxDistance = 1U << windowLog;
	U32 const lowestValid = ms->window.lowLimit;
	U32 const withinWindow = (current - lowestValid > maxDistance) ? current - maxDistance : lowestValid;
	U32 const isDictionary = (ms->loadedDictEnd != 0);
	U32 const matchLowest = isDictionary ? lowestValid : withinWindow;
	return matchLowest;
	}

	/**
	* Returns the lowest allowed match index in the prefix.
	*/
	MEM_STATIC U32 ZSTD_getLowestPrefixIndex(const ZSTD_matchState_t* ms, U32 current, unsigned windowLog)
	{
	U32 const maxDistance = 1U << windowLog;
	U32 const lowestValid = ms->window.dictLimit;
	U32 const withinWindow = (current - lowestValid > maxDistance) ? current - maxDistance : lowestValid;
	U32 const isDictionary = (ms->loadedDictEnd != 0);
	U32 const matchLowest = isDictionary ? lowestValid : withinWindow;
	return matchLowest;
	}



	/* debug functions */
	#if (DEBUGLEVEL>=2)

	MEM_STATIC double ZSTD_fWeight(U32 rawStat)
	{
	U32 const fp_accuracy = 8;
	U32 const fp_multiplier = (1 << fp_accuracy);
	U32 const newStat = rawStat + 1;
	U32 const hb = ZSTD_highbit32(newStat);
	U32 const BWeight = hb * fp_multiplier;
	U32 const FWeight = (newStat << fp_accuracy) >> hb;
	U32 const weight = BWeight + FWeight;
	assert(hb + fp_accuracy < 31);
	return (double)weight / fp_multiplier;
	}

	/* display a table content,
	* listing each element, its frequency, and its predicted bit cost */
	MEM_STATIC void ZSTD_debugTable(const U32* table, U32 max)
	{
	unsigned u, sum;
	for (u=0, sum=0; u<=max; u++) sum += table[u];
	DEBUGLOG(2, "total nb elts: %u", sum);
	for (u=0; u<=max; u++) {
	DEBUGLOG(2, "%2u: %5u (%.2f)",
	u, table[u], ZSTD_fWeight(sum) - ZSTD_fWeight(table[u]) );
	}
	}

	#endif


	#if defined (__cplusplus)
	}
	#endif

	/* ===============================================================
	* Shared internal declarations
	* These prototypes may be called from sources not in lib/compress
	* =============================================================== */

	/* ZSTD_loadCEntropy() :
	* dict : must point at beginning of a valid zstd dictionary.
	* return : size of dictionary header (size of magic number + dict ID + entropy tables)
	* assumptions : magic number supposed already checked
	* and dictSize >= 8 */
	size_t ZSTD_loadCEntropy(ZSTD_compressedBlockState_t* bs, void* workspace,
	short* offcodeNCount, unsigned* offcodeMaxValue,
	const void* const dict, size_t dictSize);

	void ZSTD_reset_compressedBlockState(ZSTD_compressedBlockState_t* bs);

	/* ==============================================================
	* Private declarations
	* These prototypes shall only be called from within lib/compress
	* ============================================================== */

	/* ZSTD_getCParamsFromCCtxParams() :
	* cParams are built depending on compressionLevel, src size hints,
	* LDM and manually set compression parameters.
	* Note: srcSizeHint == 0 means 0!
	*/
	ZSTD_compressionParameters ZSTD_getCParamsFromCCtxParams(
	const ZSTD_CCtx_params* CCtxParams, U64 srcSizeHint, size_t dictSize);

	/*! ZSTD_initCStream_internal() :
	* Private use only. Init streaming operation.
	* expects params to be valid.
	* must receive dict, or cdict, or none, but not both.
	* @return : 0, or an error code */
	size_t ZSTD_initCStream_internal(ZSTD_CStream* zcs,
	const void* dict, size_t dictSize,
	const ZSTD_CDict* cdict,
	const ZSTD_CCtx_params* params, unsigned long long pledgedSrcSize);

	void ZSTD_resetSeqStore(seqStore_t* ssPtr);

	/*! ZSTD_getCParamsFromCDict() :
	* as the name implies */
	ZSTD_compressionParameters ZSTD_getCParamsFromCDict(const ZSTD_CDict* cdict);

	/* ZSTD_compressBegin_advanced_internal() :
	* Private use only. To be called from zstdmt_compress.c. */
	size_t ZSTD_compressBegin_advanced_internal(ZSTD_CCtx* cctx,
	const void* dict, size_t dictSize,
	ZSTD_dictContentType_e dictContentType,
	ZSTD_dictTableLoadMethod_e dtlm,
	const ZSTD_CDict* cdict,
	const ZSTD_CCtx_params* params,
	unsigned long long pledgedSrcSize);

	/* ZSTD_compress_advanced_internal() :
	* Private use only. To be called from zstdmt_compress.c. */
	size_t ZSTD_compress_advanced_internal(ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const void* dict,size_t dictSize,
	const ZSTD_CCtx_params* params);


	/* ZSTD_writeLastEmptyBlock() :
	* output an empty Block with end-of-frame mark to complete a frame
	* @return : size of data written into `dst` (== ZSTD_blockHeaderSize (defined in zstd_internal.h))
	* or an error code if `dstCapacity` is too small (<ZSTD_blockHeaderSize)
	*/
	size_t ZSTD_writeLastEmptyBlock(void* dst, size_t dstCapacity);


	/* ZSTD_referenceExternalSequences() :
	* Must be called before starting a compression operation.
	* seqs must parse a prefix of the source.
	* This cannot be used when long range matching is enabled.
	* Zstd will use these sequences, and pass the literals to a secondary block
	* compressor.
	* @return : An error code on failure.
	* NOTE: seqs are not verified! Invalid sequences can cause out-of-bounds memory
	* access and data corruption.
	*/
	size_t ZSTD_referenceExternalSequences(ZSTD_CCtx* cctx, rawSeq* seq, size_t nbSeq);

	/** ZSTD_cycleLog() :
	* condition for correct operation : hashLog > 1 */
	U32 ZSTD_cycleLog(U32 hashLog, ZSTD_strategy strat);

	#endif /* ZSTD_COMPRESS_H */
	/** ended inlining zstd_compress_internal.h **/


	size_t ZSTD_noCompressLiterals (void* dst, size_t dstCapacity, const void* src, size_t srcSize);

	size_t ZSTD_compressRleLiteralsBlock (void* dst, size_t dstCapacity, const void* src, size_t srcSize);

	size_t ZSTD_compressLiterals (ZSTD_hufCTables_t const* prevHuf,
	ZSTD_hufCTables_t* nextHuf,
	ZSTD_strategy strategy, int disableLiteralCompression,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	void* entropyWorkspace, size_t entropyWorkspaceSize,
	const int bmi2);

	#endif /* ZSTD_COMPRESS_LITERALS_H */
	/** ended inlining zstd_compress_literals.h **/

	size_t ZSTD_noCompressLiterals (void* dst, size_t dstCapacity, const void* src, size_t srcSize)
	{
	BYTE* const ostart = (BYTE* const)dst;
	U32 const flSize = 1 + (srcSize>31) + (srcSize>4095);

	RETURN_ERROR_IF(srcSize + flSize > dstCapacity, dstSize_tooSmall, "");

	switch(flSize)
	{
	case 1: /* 2 - 1 - 5 */
	ostart[0] = (BYTE)((U32)set_basic + (srcSize<<3));
	break;
	case 2: /* 2 - 2 - 12 */
	MEM_writeLE16(ostart, (U16)((U32)set_basic + (1<<2) + (srcSize<<4)));
	break;
	case 3: /* 2 - 2 - 20 */
	MEM_writeLE32(ostart, (U32)((U32)set_basic + (3<<2) + (srcSize<<4)));
	break;
	default: /* not necessary : flSize is {1,2,3} */
	assert(0);
	}

	memcpy(ostart + flSize, src, srcSize);
	DEBUGLOG(5, "Raw literals: %u -> %u", (U32)srcSize, (U32)(srcSize + flSize));
	return srcSize + flSize;
	}

	size_t ZSTD_compressRleLiteralsBlock (void* dst, size_t dstCapacity, const void* src, size_t srcSize)
	{
	BYTE* const ostart = (BYTE* const)dst;
	U32 const flSize = 1 + (srcSize>31) + (srcSize>4095);

	(void)dstCapacity; /* dstCapacity already guaranteed to be >=4, hence large enough */

	switch(flSize)
	{
	case 1: /* 2 - 1 - 5 */
	ostart[0] = (BYTE)((U32)set_rle + (srcSize<<3));
	break;
	case 2: /* 2 - 2 - 12 */
	MEM_writeLE16(ostart, (U16)((U32)set_rle + (1<<2) + (srcSize<<4)));
	break;
	case 3: /* 2 - 2 - 20 */
	MEM_writeLE32(ostart, (U32)((U32)set_rle + (3<<2) + (srcSize<<4)));
	break;
	default: /* not necessary : flSize is {1,2,3} */
	assert(0);
	}

	ostart[flSize] = (const BYTE)src;
	DEBUGLOG(5, "RLE literals: %u -> %u", (U32)srcSize, (U32)flSize + 1);
	return flSize+1;
	}

	size_t ZSTD_compressLiterals (ZSTD_hufCTables_t const* prevHuf,
	ZSTD_hufCTables_t* nextHuf,
	ZSTD_strategy strategy, int disableLiteralCompression,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	void* entropyWorkspace, size_t entropyWorkspaceSize,
	const int bmi2)
	{
	size_t const minGain = ZSTD_minGain(srcSize, strategy);
	size_t const lhSize = 3 + (srcSize >= 1 KB) + (srcSize >= 16 KB);
	BYTE* const ostart = (BYTE*)dst;
	U32 singleStream = srcSize < 256;
	symbolEncodingType_e hType = set_compressed;
	size_t cLitSize;

	DEBUGLOG(5,"ZSTD_compressLiterals (disableLiteralCompression=%i srcSize=%u)",
	disableLiteralCompression, (U32)srcSize);

	/* Prepare nextEntropy assuming reusing the existing table */
	memcpy(nextHuf, prevHuf, sizeof(*prevHuf));

	if (disableLiteralCompression)
	return ZSTD_noCompressLiterals(dst, dstCapacity, src, srcSize);

	/* small ? don't even attempt compression (speed opt) */
	# define COMPRESS_LITERALS_SIZE_MIN 63
	{ size_t const minLitSize = (prevHuf->repeatMode == HUF_repeat_valid) ? 6 : COMPRESS_LITERALS_SIZE_MIN;
	if (srcSize <= minLitSize) return ZSTD_noCompressLiterals(dst, dstCapacity, src, srcSize);
	}

	RETURN_ERROR_IF(dstCapacity < lhSize+1, dstSize_tooSmall, "not enough space for compression");
	{ HUF_repeat repeat = prevHuf->repeatMode;
	int const preferRepeat = strategy < ZSTD_lazy ? srcSize <= 1024 : 0;
	if (repeat == HUF_repeat_valid && lhSize == 3) singleStream = 1;
	cLitSize = singleStream ?
	HUF_compress1X_repeat(
	ostart+lhSize, dstCapacity-lhSize, src, srcSize,
	HUF_SYMBOLVALUE_MAX, HUF_TABLELOG_DEFAULT, entropyWorkspace, entropyWorkspaceSize,
	(HUF_CElt*)nextHuf->CTable, &repeat, preferRepeat, bmi2) :
	HUF_compress4X_repeat(
	ostart+lhSize, dstCapacity-lhSize, src, srcSize,
	HUF_SYMBOLVALUE_MAX, HUF_TABLELOG_DEFAULT, entropyWorkspace, entropyWorkspaceSize,
	(HUF_CElt*)nextHuf->CTable, &repeat, preferRepeat, bmi2);
	if (repeat != HUF_repeat_none) {
	/* reused the existing table */
	DEBUGLOG(5, "Reusing previous huffman table");
	hType = set_repeat;
	}
	}

	if ((cLitSize==0) \| (cLitSize >= srcSize - minGain) \| ERR_isError(cLitSize)) {
	memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
	return ZSTD_noCompressLiterals(dst, dstCapacity, src, srcSize);
	}
	if (cLitSize==1) {
	memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
	return ZSTD_compressRleLiteralsBlock(dst, dstCapacity, src, srcSize);
	}

	if (hType == set_compressed) {
	/* using a newly constructed table */
	nextHuf->repeatMode = HUF_repeat_check;
	}

	/* Build header */
	switch(lhSize)
	{
	case 3: /* 2 - 2 - 10 - 10 */
	{ U32 const lhc = hType + ((!singleStream) << 2) + ((U32)srcSize<<4) + ((U32)cLitSize<<14);
	MEM_writeLE24(ostart, lhc);
	break;
	}
	case 4: /* 2 - 2 - 14 - 14 */
	{ U32 const lhc = hType + (2 << 2) + ((U32)srcSize<<4) + ((U32)cLitSize<<18);
	MEM_writeLE32(ostart, lhc);
	break;
	}
	case 5: /* 2 - 2 - 18 - 18 */
	{ U32 const lhc = hType + (3 << 2) + ((U32)srcSize<<4) + ((U32)cLitSize<<22);
	MEM_writeLE32(ostart, lhc);
	ostart[4] = (BYTE)(cLitSize >> 10);
	break;
	}
	default: /* not possible : lhSize is {3,4,5} */
	assert(0);
	}
	DEBUGLOG(5, "Compressed literals: %u -> %u", (U32)srcSize, (U32)(lhSize+cLitSize));
	return lhSize+cLitSize;
	}
	/** ended inlining compress/zstd_compress_literals.c **/
	/** start inlining compress/zstd_compress_sequences.c **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	/-************************************
	* Dependencies
	***************************************/
	/** start inlining zstd_compress_sequences.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTD_COMPRESS_SEQUENCES_H
	#define ZSTD_COMPRESS_SEQUENCES_H

	/** skipping file: ../common/fse.h **/
	/** skipping file: ../common/zstd_internal.h **/

	typedef enum {
	ZSTD_defaultDisallowed = 0,
	ZSTD_defaultAllowed = 1
	} ZSTD_defaultPolicy_e;

	symbolEncodingType_e
	ZSTD_selectEncodingType(
	FSE_repeat* repeatMode, unsigned const* count, unsigned const max,
	size_t const mostFrequent, size_t nbSeq, unsigned const FSELog,
	FSE_CTable const* prevCTable,
	short const* defaultNorm, U32 defaultNormLog,
	ZSTD_defaultPolicy_e const isDefaultAllowed,
	ZSTD_strategy const strategy);

	size_t
	ZSTD_buildCTable(void* dst, size_t dstCapacity,
	FSE_CTable* nextCTable, U32 FSELog, symbolEncodingType_e type,
	unsigned* count, U32 max,
	const BYTE* codeTable, size_t nbSeq,
	const S16* defaultNorm, U32 defaultNormLog, U32 defaultMax,
	const FSE_CTable* prevCTable, size_t prevCTableSize,
	void* entropyWorkspace, size_t entropyWorkspaceSize);

	size_t ZSTD_encodeSequences(
	void* dst, size_t dstCapacity,
	FSE_CTable const* CTable_MatchLength, BYTE const* mlCodeTable,
	FSE_CTable const* CTable_OffsetBits, BYTE const* ofCodeTable,
	FSE_CTable const* CTable_LitLength, BYTE const* llCodeTable,
	seqDef const* sequences, size_t nbSeq, int longOffsets, int bmi2);

	size_t ZSTD_fseBitCost(
	FSE_CTable const* ctable,
	unsigned const* count,
	unsigned const max);

	size_t ZSTD_crossEntropyCost(short const* norm, unsigned accuracyLog,
	unsigned const* count, unsigned const max);
	#endif /* ZSTD_COMPRESS_SEQUENCES_H */
	/** ended inlining zstd_compress_sequences.h **/

	/**
	* -log2(x / 256) lookup table for x in [0, 256).
	* If x == 0: Return 0
	* Else: Return floor(-log2(x / 256) * 256)
	*/
	static unsigned const kInverseProbabilityLog256[256] = {
	0, 2048, 1792, 1642, 1536, 1453, 1386, 1329, 1280, 1236, 1197, 1162,
	1130, 1100, 1073, 1047, 1024, 1001, 980, 960, 941, 923, 906, 889,
	874, 859, 844, 830, 817, 804, 791, 779, 768, 756, 745, 734,
	724, 714, 704, 694, 685, 676, 667, 658, 650, 642, 633, 626,
	618, 610, 603, 595, 588, 581, 574, 567, 561, 554, 548, 542,
	535, 529, 523, 517, 512, 506, 500, 495, 489, 484, 478, 473,
	468, 463, 458, 453, 448, 443, 438, 434, 429, 424, 420, 415,
	411, 407, 402, 398, 394, 390, 386, 382, 377, 373, 370, 366,
	362, 358, 354, 350, 347, 343, 339, 336, 332, 329, 325, 322,
	318, 315, 311, 308, 305, 302, 298, 295, 292, 289, 286, 282,
	279, 276, 273, 270, 267, 264, 261, 258, 256, 253, 250, 247,
	244, 241, 239, 236, 233, 230, 228, 225, 222, 220, 217, 215,
	212, 209, 207, 204, 202, 199, 197, 194, 192, 190, 187, 185,
	182, 180, 178, 175, 173, 171, 168, 166, 164, 162, 159, 157,
	155, 153, 151, 149, 146, 144, 142, 140, 138, 136, 134, 132,
	130, 128, 126, 123, 121, 119, 117, 115, 114, 112, 110, 108,
	106, 104, 102, 100, 98, 96, 94, 93, 91, 89, 87, 85,
	83, 82, 80, 78, 76, 74, 73, 71, 69, 67, 66, 64,
	62, 61, 59, 57, 55, 54, 52, 50, 49, 47, 46, 44,
	42, 41, 39, 37, 36, 34, 33, 31, 30, 28, 26, 25,
	23, 22, 20, 19, 17, 16, 14, 13, 11, 10, 8, 7,
	5, 4, 2, 1,
	};

	static unsigned ZSTD_getFSEMaxSymbolValue(FSE_CTable const* ctable) {
	void const* ptr = ctable;
	U16 const* u16ptr = (U16 const*)ptr;
	U32 const maxSymbolValue = MEM_read16(u16ptr + 1);
	return maxSymbolValue;
	}

	/**
	* Returns the cost in bytes of encoding the normalized count header.
	* Returns an error if any of the helper functions return an error.
	*/
	static size_t ZSTD_NCountCost(unsigned const* count, unsigned const max,
	size_t const nbSeq, unsigned const FSELog)
	{
	BYTE wksp[FSE_NCOUNTBOUND];
	S16 norm[MaxSeq + 1];
	const U32 tableLog = FSE_optimalTableLog(FSELog, nbSeq, max);
	FORWARD_IF_ERROR(FSE_normalizeCount(norm, tableLog, count, nbSeq, max), "");
	return FSE_writeNCount(wksp, sizeof(wksp), norm, max, tableLog);
	}

	/**
	* Returns the cost in bits of encoding the distribution described by count
	* using the entropy bound.
	*/
	static size_t ZSTD_entropyCost(unsigned const* count, unsigned const max, size_t const total)
	{
	unsigned cost = 0;
	unsigned s;
	for (s = 0; s <= max; ++s) {
	unsigned norm = (unsigned)((256 * count[s]) / total);
	if (count[s] != 0 && norm == 0)
	norm = 1;
	assert(count[s] < total);
	cost += count[s] * kInverseProbabilityLog256[norm];
	}
	return cost >> 8;
	}

	/**
	* Returns the cost in bits of encoding the distribution in count using ctable.
	* Returns an error if ctable cannot represent all the symbols in count.
	*/
	size_t ZSTD_fseBitCost(
	FSE_CTable const* ctable,
	unsigned const* count,
	unsigned const max)
	{
	unsigned const kAccuracyLog = 8;
	size_t cost = 0;
	unsigned s;
	FSE_CState_t cstate;
	FSE_initCState(&cstate, ctable);
	if (ZSTD_getFSEMaxSymbolValue(ctable) < max) {
	DEBUGLOG(5, "Repeat FSE_CTable has maxSymbolValue %u < %u",
	ZSTD_getFSEMaxSymbolValue(ctable), max);
	return ERROR(GENERIC);
	}
	for (s = 0; s <= max; ++s) {
	unsigned const tableLog = cstate.stateLog;
	unsigned const badCost = (tableLog + 1) << kAccuracyLog;
	unsigned const bitCost = FSE_bitCost(cstate.symbolTT, tableLog, s, kAccuracyLog);
	if (count[s] == 0)
	continue;
	if (bitCost >= badCost) {
	DEBUGLOG(5, "Repeat FSE_CTable has Prob[%u] == 0", s);
	return ERROR(GENERIC);
	}
	cost += (size_t)count[s] * bitCost;
	}
	return cost >> kAccuracyLog;
	}

	/**
	* Returns the cost in bits of encoding the distribution in count using the
	* table described by norm. The max symbol support by norm is assumed >= max.
	* norm must be valid for every symbol with non-zero probability in count.
	*/
	size_t ZSTD_crossEntropyCost(short const* norm, unsigned accuracyLog,
	unsigned const* count, unsigned const max)
	{
	unsigned const shift = 8 - accuracyLog;
	size_t cost = 0;
	unsigned s;
	assert(accuracyLog <= 8);
	for (s = 0; s <= max; ++s) {
	unsigned const normAcc = (norm[s] != -1) ? (unsigned)norm[s] : 1;
	unsigned const norm256 = normAcc << shift;
	assert(norm256 > 0);
	assert(norm256 < 256);
	cost += count[s] * kInverseProbabilityLog256[norm256];
	}
	return cost >> 8;
	}

	symbolEncodingType_e
	ZSTD_selectEncodingType(
	FSE_repeat* repeatMode, unsigned const* count, unsigned const max,
	size_t const mostFrequent, size_t nbSeq, unsigned const FSELog,
	FSE_CTable const* prevCTable,
	short const* defaultNorm, U32 defaultNormLog,
	ZSTD_defaultPolicy_e const isDefaultAllowed,
	ZSTD_strategy const strategy)
	{
	ZSTD_STATIC_ASSERT(ZSTD_defaultDisallowed == 0 && ZSTD_defaultAllowed != 0);
	if (mostFrequent == nbSeq) {
	*repeatMode = FSE_repeat_none;
	if (isDefaultAllowed && nbSeq <= 2) {
	/* Prefer set_basic over set_rle when there are 2 or less symbols,
	* since RLE uses 1 byte, but set_basic uses 5-6 bits per symbol.
	* If basic encoding isn't possible, always choose RLE.
	*/
	DEBUGLOG(5, "Selected set_basic");
	return set_basic;
	}
	DEBUGLOG(5, "Selected set_rle");
	return set_rle;
	}
	if (strategy < ZSTD_lazy) {
	if (isDefaultAllowed) {
	size_t const staticFse_nbSeq_max = 1000;
	size_t const mult = 10 - strategy;
	size_t const baseLog = 3;
	size_t const dynamicFse_nbSeq_min = (((size_t)1 << defaultNormLog) * mult) >> baseLog; /* 28-36 for offset, 56-72 for lengths */
	assert(defaultNormLog >= 5 && defaultNormLog <= 6); /* xx_DEFAULTNORMLOG */
	assert(mult <= 9 && mult >= 7);
	if ( (*repeatMode == FSE_repeat_valid)
	&& (nbSeq < staticFse_nbSeq_max) ) {
	DEBUGLOG(5, "Selected set_repeat");
	return set_repeat;
	}
	if ( (nbSeq < dynamicFse_nbSeq_min)
	\|\| (mostFrequent < (nbSeq >> (defaultNormLog-1))) ) {
	DEBUGLOG(5, "Selected set_basic");
	/* The format allows default tables to be repeated, but it isn't useful.
	* When using simple heuristics to select encoding type, we don't want
	* to confuse these tables with dictionaries. When running more careful
	* analysis, we don't need to waste time checking both repeating tables
	* and default tables.
	*/
	*repeatMode = FSE_repeat_none;
	return set_basic;
	}
	}
	} else {
	size_t const basicCost = isDefaultAllowed ? ZSTD_crossEntropyCost(defaultNorm, defaultNormLog, count, max) : ERROR(GENERIC);
	size_t const repeatCost = *repeatMode != FSE_repeat_none ? ZSTD_fseBitCost(prevCTable, count, max) : ERROR(GENERIC);
	size_t const NCountCost = ZSTD_NCountCost(count, max, nbSeq, FSELog);
	size_t const compressedCost = (NCountCost << 3) + ZSTD_entropyCost(count, max, nbSeq);

	if (isDefaultAllowed) {
	assert(!ZSTD_isError(basicCost));
	assert(!(*repeatMode == FSE_repeat_valid && ZSTD_isError(repeatCost)));
	}
	assert(!ZSTD_isError(NCountCost));
	assert(compressedCost < ERROR(maxCode));
	DEBUGLOG(5, "Estimated bit costs: basic=%u\trepeat=%u\tcompressed=%u",
	(unsigned)basicCost, (unsigned)repeatCost, (unsigned)compressedCost);
	if (basicCost <= repeatCost && basicCost <= compressedCost) {
	DEBUGLOG(5, "Selected set_basic");
	assert(isDefaultAllowed);
	*repeatMode = FSE_repeat_none;
	return set_basic;
	}
	if (repeatCost <= compressedCost) {
	DEBUGLOG(5, "Selected set_repeat");
	assert(!ZSTD_isError(repeatCost));
	return set_repeat;
	}
	assert(compressedCost < basicCost && compressedCost < repeatCost);
	}
	DEBUGLOG(5, "Selected set_compressed");
	*repeatMode = FSE_repeat_check;
	return set_compressed;
	}

	size_t
	ZSTD_buildCTable(void* dst, size_t dstCapacity,
	FSE_CTable* nextCTable, U32 FSELog, symbolEncodingType_e type,
	unsigned* count, U32 max,
	const BYTE* codeTable, size_t nbSeq,
	const S16* defaultNorm, U32 defaultNormLog, U32 defaultMax,
	const FSE_CTable* prevCTable, size_t prevCTableSize,
	void* entropyWorkspace, size_t entropyWorkspaceSize)
	{
	BYTE* op = (BYTE*)dst;
	const BYTE* const oend = op + dstCapacity;
	DEBUGLOG(6, "ZSTD_buildCTable (dstCapacity=%u)", (unsigned)dstCapacity);

	switch (type) {
	case set_rle:
	FORWARD_IF_ERROR(FSE_buildCTable_rle(nextCTable, (BYTE)max), "");
	RETURN_ERROR_IF(dstCapacity==0, dstSize_tooSmall, "not enough space");
	*op = codeTable[0];
	return 1;
	case set_repeat:
	memcpy(nextCTable, prevCTable, prevCTableSize);
	return 0;
	case set_basic:
	FORWARD_IF_ERROR(FSE_buildCTable_wksp(nextCTable, defaultNorm, defaultMax, defaultNormLog, entropyWorkspace, entropyWorkspaceSize), ""); /* note : could be pre-calculated */
	return 0;
	case set_compressed: {
	S16 norm[MaxSeq + 1];
	size_t nbSeq_1 = nbSeq;
	const U32 tableLog = FSE_optimalTableLog(FSELog, nbSeq, max);
	if (count[codeTable[nbSeq-1]] > 1) {
	count[codeTable[nbSeq-1]]--;
	nbSeq_1--;
	}
	assert(nbSeq_1 > 1);
	FORWARD_IF_ERROR(FSE_normalizeCount(norm, tableLog, count, nbSeq_1, max), "");
	{ size_t const NCountSize = FSE_writeNCount(op, oend - op, norm, max, tableLog); /* overflow protected */
	FORWARD_IF_ERROR(NCountSize, "FSE_writeNCount failed");
	FORWARD_IF_ERROR(FSE_buildCTable_wksp(nextCTable, norm, max, tableLog, entropyWorkspace, entropyWorkspaceSize), "");
	return NCountSize;
	}
	}
	default: assert(0); RETURN_ERROR(GENERIC, "impossible to reach");
	}
	}

	FORCE_INLINE_TEMPLATE size_t
	ZSTD_encodeSequences_body(
	void* dst, size_t dstCapacity,
	FSE_CTable const* CTable_MatchLength, BYTE const* mlCodeTable,
	FSE_CTable const* CTable_OffsetBits, BYTE const* ofCodeTable,
	FSE_CTable const* CTable_LitLength, BYTE const* llCodeTable,
	seqDef const* sequences, size_t nbSeq, int longOffsets)
	{
	BIT_CStream_t blockStream;
	FSE_CState_t stateMatchLength;
	FSE_CState_t stateOffsetBits;
	FSE_CState_t stateLitLength;

	RETURN_ERROR_IF(
	ERR_isError(BIT_initCStream(&blockStream, dst, dstCapacity)),
	dstSize_tooSmall, "not enough space remaining");
	DEBUGLOG(6, "available space for bitstream : %i (dstCapacity=%u)",
	(int)(blockStream.endPtr - blockStream.startPtr),
	(unsigned)dstCapacity);

	/* first symbols */
	FSE_initCState2(&stateMatchLength, CTable_MatchLength, mlCodeTable[nbSeq-1]);
	FSE_initCState2(&stateOffsetBits, CTable_OffsetBits, ofCodeTable[nbSeq-1]);
	FSE_initCState2(&stateLitLength, CTable_LitLength, llCodeTable[nbSeq-1]);
	BIT_addBits(&blockStream, sequences[nbSeq-1].litLength, LL_bits[llCodeTable[nbSeq-1]]);
	if (MEM_32bits()) BIT_flushBits(&blockStream);
	BIT_addBits(&blockStream, sequences[nbSeq-1].matchLength, ML_bits[mlCodeTable[nbSeq-1]]);
	if (MEM_32bits()) BIT_flushBits(&blockStream);
	if (longOffsets) {
	U32 const ofBits = ofCodeTable[nbSeq-1];
	unsigned const extraBits = ofBits - MIN(ofBits, STREAM_ACCUMULATOR_MIN-1);
	if (extraBits) {
	BIT_addBits(&blockStream, sequences[nbSeq-1].offset, extraBits);
	BIT_flushBits(&blockStream);
	}
	BIT_addBits(&blockStream, sequences[nbSeq-1].offset >> extraBits,
	ofBits - extraBits);
	} else {
	BIT_addBits(&blockStream, sequences[nbSeq-1].offset, ofCodeTable[nbSeq-1]);
	}
	BIT_flushBits(&blockStream);

	{ size_t n;
	for (n=nbSeq-2 ; n<nbSeq ; n--) { /* intentional underflow */
	BYTE const llCode = llCodeTable[n];
	BYTE const ofCode = ofCodeTable[n];
	BYTE const mlCode = mlCodeTable[n];
	U32 const llBits = LL_bits[llCode];
	U32 const ofBits = ofCode;
	U32 const mlBits = ML_bits[mlCode];
	DEBUGLOG(6, "encoding: litlen:%2u - matchlen:%2u - offCode:%7u",
	(unsigned)sequences[n].litLength,
	(unsigned)sequences[n].matchLength + MINMATCH,
	(unsigned)sequences[n].offset);
	/* 32b/ / 64b*/
	/* (7)/ / (7)*/
	FSE_encodeSymbol(&blockStream, &stateOffsetBits, ofCode); /* 15 / / 15 */
	FSE_encodeSymbol(&blockStream, &stateMatchLength, mlCode); /* 24 / / 24 */
	if (MEM_32bits()) BIT_flushBits(&blockStream); /* (7)*/
	FSE_encodeSymbol(&blockStream, &stateLitLength, llCode); /* 16 / / 33 */
	if (MEM_32bits() \|\| (ofBits+mlBits+llBits >= 64-7-(LLFSELog+MLFSELog+OffFSELog)))
	BIT_flushBits(&blockStream); /* (7)*/
	BIT_addBits(&blockStream, sequences[n].litLength, llBits);
	if (MEM_32bits() && ((llBits+mlBits)>24)) BIT_flushBits(&blockStream);
	BIT_addBits(&blockStream, sequences[n].matchLength, mlBits);
	if (MEM_32bits() \|\| (ofBits+mlBits+llBits > 56)) BIT_flushBits(&blockStream);
	if (longOffsets) {
	unsigned const extraBits = ofBits - MIN(ofBits, STREAM_ACCUMULATOR_MIN-1);
	if (extraBits) {
	BIT_addBits(&blockStream, sequences[n].offset, extraBits);
	BIT_flushBits(&blockStream); /* (7)*/
	}
	BIT_addBits(&blockStream, sequences[n].offset >> extraBits,
	ofBits - extraBits); /* 31 */
	} else {
	BIT_addBits(&blockStream, sequences[n].offset, ofBits); /* 31 */
	}
	BIT_flushBits(&blockStream); /* (7)*/
	DEBUGLOG(7, "remaining space : %i", (int)(blockStream.endPtr - blockStream.ptr));
	} }

	DEBUGLOG(6, "ZSTD_encodeSequences: flushing ML state with %u bits", stateMatchLength.stateLog);
	FSE_flushCState(&blockStream, &stateMatchLength);
	DEBUGLOG(6, "ZSTD_encodeSequences: flushing Off state with %u bits", stateOffsetBits.stateLog);
	FSE_flushCState(&blockStream, &stateOffsetBits);
	DEBUGLOG(6, "ZSTD_encodeSequences: flushing LL state with %u bits", stateLitLength.stateLog);
	FSE_flushCState(&blockStream, &stateLitLength);

	{ size_t const streamSize = BIT_closeCStream(&blockStream);
	RETURN_ERROR_IF(streamSize==0, dstSize_tooSmall, "not enough space");
	return streamSize;
	}
	}

	static size_t
	ZSTD_encodeSequences_default(
	void* dst, size_t dstCapacity,
	FSE_CTable const* CTable_MatchLength, BYTE const* mlCodeTable,
	FSE_CTable const* CTable_OffsetBits, BYTE const* ofCodeTable,
	FSE_CTable const* CTable_LitLength, BYTE const* llCodeTable,
	seqDef const* sequences, size_t nbSeq, int longOffsets)
	{
	return ZSTD_encodeSequences_body(dst, dstCapacity,
	CTable_MatchLength, mlCodeTable,
	CTable_OffsetBits, ofCodeTable,
	CTable_LitLength, llCodeTable,
	sequences, nbSeq, longOffsets);
	}


	#if DYNAMIC_BMI2

	static TARGET_ATTRIBUTE("bmi2") size_t
	ZSTD_encodeSequences_bmi2(
	void* dst, size_t dstCapacity,
	FSE_CTable const* CTable_MatchLength, BYTE const* mlCodeTable,
	FSE_CTable const* CTable_OffsetBits, BYTE const* ofCodeTable,
	FSE_CTable const* CTable_LitLength, BYTE const* llCodeTable,
	seqDef const* sequences, size_t nbSeq, int longOffsets)
	{
	return ZSTD_encodeSequences_body(dst, dstCapacity,
	CTable_MatchLength, mlCodeTable,
	CTable_OffsetBits, ofCodeTable,
	CTable_LitLength, llCodeTable,
	sequences, nbSeq, longOffsets);
	}

	#endif

	size_t ZSTD_encodeSequences(
	void* dst, size_t dstCapacity,
	FSE_CTable const* CTable_MatchLength, BYTE const* mlCodeTable,
	FSE_CTable const* CTable_OffsetBits, BYTE const* ofCodeTable,
	FSE_CTable const* CTable_LitLength, BYTE const* llCodeTable,
	seqDef const* sequences, size_t nbSeq, int longOffsets, int bmi2)
	{
	DEBUGLOG(5, "ZSTD_encodeSequences: dstCapacity = %u", (unsigned)dstCapacity);
	#if DYNAMIC_BMI2
	if (bmi2) {
	return ZSTD_encodeSequences_bmi2(dst, dstCapacity,
	CTable_MatchLength, mlCodeTable,
	CTable_OffsetBits, ofCodeTable,
	CTable_LitLength, llCodeTable,
	sequences, nbSeq, longOffsets);
	}
	#endif
	(void)bmi2;
	return ZSTD_encodeSequences_default(dst, dstCapacity,
	CTable_MatchLength, mlCodeTable,
	CTable_OffsetBits, ofCodeTable,
	CTable_LitLength, llCodeTable,
	sequences, nbSeq, longOffsets);
	}
	/** ended inlining compress/zstd_compress_sequences.c **/
	/** start inlining compress/zstd_compress_superblock.c **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	/-************************************
	* Dependencies
	***************************************/
	/** start inlining zstd_compress_superblock.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTD_COMPRESS_ADVANCED_H
	#define ZSTD_COMPRESS_ADVANCED_H

	/-************************************
	* Dependencies
	***************************************/

	/** skipping file: ../zstd.h **/

	/-************************************
	* Target Compressed Block Size
	***************************************/

	/* ZSTD_compressSuperBlock() :
	* Used to compress a super block when targetCBlockSize is being used.
	* The given block will be compressed into multiple sub blocks that are around targetCBlockSize. */
	size_t ZSTD_compressSuperBlock(ZSTD_CCtx* zc,
	void* dst, size_t dstCapacity,
	void const* src, size_t srcSize,
	unsigned lastBlock);

	#endif /* ZSTD_COMPRESS_ADVANCED_H */
	/** ended inlining zstd_compress_superblock.h **/

	/** skipping file: ../common/zstd_internal.h **/
	/** skipping file: hist.h **/
	/** skipping file: zstd_compress_internal.h **/
	/** skipping file: zstd_compress_sequences.h **/
	/** skipping file: zstd_compress_literals.h **/

	/-************************************
	* Superblock entropy buffer structs
	***************************************/
	/** ZSTD_hufCTablesMetadata_t :
	* Stores Literals Block Type for a super-block in hType, and
	* huffman tree description in hufDesBuffer.
	* hufDesSize refers to the size of huffman tree description in bytes.
	* This metadata is populated in ZSTD_buildSuperBlockEntropy_literal() */
	typedef struct {
	symbolEncodingType_e hType;
	BYTE hufDesBuffer[500]; /* TODO give name to this value */
	size_t hufDesSize;
	} ZSTD_hufCTablesMetadata_t;

	/** ZSTD_fseCTablesMetadata_t :
	* Stores symbol compression modes for a super-block in {ll, ol, ml}Type, and
	* fse tables in fseTablesBuffer.
	* fseTablesSize refers to the size of fse tables in bytes.
	* This metadata is populated in ZSTD_buildSuperBlockEntropy_sequences() */
	typedef struct {
	symbolEncodingType_e llType;
	symbolEncodingType_e ofType;
	symbolEncodingType_e mlType;
	BYTE fseTablesBuffer[500]; /* TODO give name to this value */
	size_t fseTablesSize;
	size_t lastCountSize; /* This is to account for bug in 1.3.4. More detail in ZSTD_compressSubBlock_sequences() */
	} ZSTD_fseCTablesMetadata_t;

	typedef struct {
	ZSTD_hufCTablesMetadata_t hufMetadata;
	ZSTD_fseCTablesMetadata_t fseMetadata;
	} ZSTD_entropyCTablesMetadata_t;


	/** ZSTD_buildSuperBlockEntropy_literal() :
	* Builds entropy for the super-block literals.
	* Stores literals block type (raw, rle, compressed, repeat) and
	* huffman description table to hufMetadata.
	* @return : size of huffman description table or error code */
	static size_t ZSTD_buildSuperBlockEntropy_literal(void* const src, size_t srcSize,
	const ZSTD_hufCTables_t* prevHuf,
	ZSTD_hufCTables_t* nextHuf,
	ZSTD_hufCTablesMetadata_t* hufMetadata,
	const int disableLiteralsCompression,
	void* workspace, size_t wkspSize)
	{
	BYTE* const wkspStart = (BYTE*)workspace;
	BYTE* const wkspEnd = wkspStart + wkspSize;
	BYTE* const countWkspStart = wkspStart;
	unsigned* const countWksp = (unsigned*)workspace;
	const size_t countWkspSize = (HUF_SYMBOLVALUE_MAX + 1) * sizeof(unsigned);
	BYTE* const nodeWksp = countWkspStart + countWkspSize;
	const size_t nodeWkspSize = wkspEnd-nodeWksp;
	unsigned maxSymbolValue = 255;
	unsigned huffLog = HUF_TABLELOG_DEFAULT;
	HUF_repeat repeat = prevHuf->repeatMode;

	DEBUGLOG(5, "ZSTD_buildSuperBlockEntropy_literal (srcSize=%zu)", srcSize);

	/* Prepare nextEntropy assuming reusing the existing table */
	memcpy(nextHuf, prevHuf, sizeof(*prevHuf));

	if (disableLiteralsCompression) {
	DEBUGLOG(5, "set_basic - disabled");
	hufMetadata->hType = set_basic;
	return 0;
	}

	/* small ? don't even attempt compression (speed opt) */
	# define COMPRESS_LITERALS_SIZE_MIN 63
	{ size_t const minLitSize = (prevHuf->repeatMode == HUF_repeat_valid) ? 6 : COMPRESS_LITERALS_SIZE_MIN;
	if (srcSize <= minLitSize) {
	DEBUGLOG(5, "set_basic - too small");
	hufMetadata->hType = set_basic;
	return 0;
	}
	}

	/* Scan input and build symbol stats */
	{ size_t const largest = HIST_count_wksp (countWksp, &maxSymbolValue, (const BYTE*)src, srcSize, workspace, wkspSize);
	FORWARD_IF_ERROR(largest, "HIST_count_wksp failed");
	if (largest == srcSize) {
	DEBUGLOG(5, "set_rle");
	hufMetadata->hType = set_rle;
	return 0;
	}
	if (largest <= (srcSize >> 7)+4) {
	DEBUGLOG(5, "set_basic - no gain");
	hufMetadata->hType = set_basic;
	return 0;
	}
	}

	/* Validate the previous Huffman table */
	if (repeat == HUF_repeat_check && !HUF_validateCTable((HUF_CElt const*)prevHuf->CTable, countWksp, maxSymbolValue)) {
	repeat = HUF_repeat_none;
	}

	/* Build Huffman Tree */
	memset(nextHuf->CTable, 0, sizeof(nextHuf->CTable));
	huffLog = HUF_optimalTableLog(huffLog, srcSize, maxSymbolValue);
	{ size_t const maxBits = HUF_buildCTable_wksp((HUF_CElt*)nextHuf->CTable, countWksp,
	maxSymbolValue, huffLog,
	nodeWksp, nodeWkspSize);
	FORWARD_IF_ERROR(maxBits, "HUF_buildCTable_wksp");
	huffLog = (U32)maxBits;
	{ /* Build and write the CTable */
	size_t const newCSize = HUF_estimateCompressedSize(
	(HUF_CElt*)nextHuf->CTable, countWksp, maxSymbolValue);
	size_t const hSize = HUF_writeCTable(
	hufMetadata->hufDesBuffer, sizeof(hufMetadata->hufDesBuffer),
	(HUF_CElt*)nextHuf->CTable, maxSymbolValue, huffLog);
	/* Check against repeating the previous CTable */
	if (repeat != HUF_repeat_none) {
	size_t const oldCSize = HUF_estimateCompressedSize(
	(HUF_CElt const*)prevHuf->CTable, countWksp, maxSymbolValue);
	if (oldCSize < srcSize && (oldCSize <= hSize + newCSize \|\| hSize + 12 >= srcSize)) {
	DEBUGLOG(5, "set_repeat - smaller");
	memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
	hufMetadata->hType = set_repeat;
	return 0;
	}
	}
	if (newCSize + hSize >= srcSize) {
	DEBUGLOG(5, "set_basic - no gains");
	memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
	hufMetadata->hType = set_basic;
	return 0;
	}
	DEBUGLOG(5, "set_compressed (hSize=%u)", (U32)hSize);
	hufMetadata->hType = set_compressed;
	nextHuf->repeatMode = HUF_repeat_check;
	return hSize;
	}
	}
	}

	/** ZSTD_buildSuperBlockEntropy_sequences() :
	* Builds entropy for the super-block sequences.
	* Stores symbol compression modes and fse table to fseMetadata.
	* @return : size of fse tables or error code */
	static size_t ZSTD_buildSuperBlockEntropy_sequences(seqStore_t* seqStorePtr,
	const ZSTD_fseCTables_t* prevEntropy,
	ZSTD_fseCTables_t* nextEntropy,
	const ZSTD_CCtx_params* cctxParams,
	ZSTD_fseCTablesMetadata_t* fseMetadata,
	void* workspace, size_t wkspSize)
	{
	BYTE* const wkspStart = (BYTE*)workspace;
	BYTE* const wkspEnd = wkspStart + wkspSize;
	BYTE* const countWkspStart = wkspStart;
	unsigned* const countWksp = (unsigned*)workspace;
	const size_t countWkspSize = (MaxSeq + 1) * sizeof(unsigned);
	BYTE* const cTableWksp = countWkspStart + countWkspSize;
	const size_t cTableWkspSize = wkspEnd-cTableWksp;
	ZSTD_strategy const strategy = cctxParams->cParams.strategy;
	FSE_CTable* CTable_LitLength = nextEntropy->litlengthCTable;
	FSE_CTable* CTable_OffsetBits = nextEntropy->offcodeCTable;
	FSE_CTable* CTable_MatchLength = nextEntropy->matchlengthCTable;
	const BYTE* const ofCodeTable = seqStorePtr->ofCode;
	const BYTE* const llCodeTable = seqStorePtr->llCode;
	const BYTE* const mlCodeTable = seqStorePtr->mlCode;
	size_t const nbSeq = seqStorePtr->sequences - seqStorePtr->sequencesStart;
	BYTE* const ostart = fseMetadata->fseTablesBuffer;
	BYTE* const oend = ostart + sizeof(fseMetadata->fseTablesBuffer);
	BYTE* op = ostart;

	assert(cTableWkspSize >= (1 << MaxFSELog) * sizeof(FSE_FUNCTION_TYPE));
	DEBUGLOG(5, "ZSTD_buildSuperBlockEntropy_sequences (nbSeq=%zu)", nbSeq);
	memset(workspace, 0, wkspSize);

	fseMetadata->lastCountSize = 0;
	/* convert length/distances into codes */
	ZSTD_seqToCodes(seqStorePtr);
	/* build CTable for Literal Lengths */
	{ U32 LLtype;
	unsigned max = MaxLL;
	size_t const mostFrequent = HIST_countFast_wksp(countWksp, &max, llCodeTable, nbSeq, workspace, wkspSize); /* can't fail */
	DEBUGLOG(5, "Building LL table");
	nextEntropy->litlength_repeatMode = prevEntropy->litlength_repeatMode;
	LLtype = ZSTD_selectEncodingType(&nextEntropy->litlength_repeatMode,
	countWksp, max, mostFrequent, nbSeq,
	LLFSELog, prevEntropy->litlengthCTable,
	LL_defaultNorm, LL_defaultNormLog,
	ZSTD_defaultAllowed, strategy);
	assert(set_basic < set_compressed && set_rle < set_compressed);
	assert(!(LLtype < set_compressed && nextEntropy->litlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */
	{ size_t const countSize = ZSTD_buildCTable(op, oend - op, CTable_LitLength, LLFSELog, (symbolEncodingType_e)LLtype,
	countWksp, max, llCodeTable, nbSeq, LL_defaultNorm, LL_defaultNormLog, MaxLL,
	prevEntropy->litlengthCTable, sizeof(prevEntropy->litlengthCTable),
	cTableWksp, cTableWkspSize);
	FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for LitLens failed");
	if (LLtype == set_compressed)
	fseMetadata->lastCountSize = countSize;
	op += countSize;
	fseMetadata->llType = (symbolEncodingType_e) LLtype;
	} }
	/* build CTable for Offsets */
	{ U32 Offtype;
	unsigned max = MaxOff;
	size_t const mostFrequent = HIST_countFast_wksp(countWksp, &max, ofCodeTable, nbSeq, workspace, wkspSize); /* can't fail */
	/* We can only use the basic table if max <= DefaultMaxOff, otherwise the offsets are too large */
	ZSTD_defaultPolicy_e const defaultPolicy = (max <= DefaultMaxOff) ? ZSTD_defaultAllowed : ZSTD_defaultDisallowed;
	DEBUGLOG(5, "Building OF table");
	nextEntropy->offcode_repeatMode = prevEntropy->offcode_repeatMode;
	Offtype = ZSTD_selectEncodingType(&nextEntropy->offcode_repeatMode,
	countWksp, max, mostFrequent, nbSeq,
	OffFSELog, prevEntropy->offcodeCTable,
	OF_defaultNorm, OF_defaultNormLog,
	defaultPolicy, strategy);
	assert(!(Offtype < set_compressed && nextEntropy->offcode_repeatMode != FSE_repeat_none)); /* We don't copy tables */
	{ size_t const countSize = ZSTD_buildCTable(op, oend - op, CTable_OffsetBits, OffFSELog, (symbolEncodingType_e)Offtype,
	countWksp, max, ofCodeTable, nbSeq, OF_defaultNorm, OF_defaultNormLog, DefaultMaxOff,
	prevEntropy->offcodeCTable, sizeof(prevEntropy->offcodeCTable),
	cTableWksp, cTableWkspSize);
	FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for Offsets failed");
	if (Offtype == set_compressed)
	fseMetadata->lastCountSize = countSize;
	op += countSize;
	fseMetadata->ofType = (symbolEncodingType_e) Offtype;
	} }
	/* build CTable for MatchLengths */
	{ U32 MLtype;
	unsigned max = MaxML;
	size_t const mostFrequent = HIST_countFast_wksp(countWksp, &max, mlCodeTable, nbSeq, workspace, wkspSize); /* can't fail */
	DEBUGLOG(5, "Building ML table (remaining space : %i)", (int)(oend-op));
	nextEntropy->matchlength_repeatMode = prevEntropy->matchlength_repeatMode;
	MLtype = ZSTD_selectEncodingType(&nextEntropy->matchlength_repeatMode,
	countWksp, max, mostFrequent, nbSeq,
	MLFSELog, prevEntropy->matchlengthCTable,
	ML_defaultNorm, ML_defaultNormLog,
	ZSTD_defaultAllowed, strategy);
	assert(!(MLtype < set_compressed && nextEntropy->matchlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */
	{ size_t const countSize = ZSTD_buildCTable(op, oend - op, CTable_MatchLength, MLFSELog, (symbolEncodingType_e)MLtype,
	countWksp, max, mlCodeTable, nbSeq, ML_defaultNorm, ML_defaultNormLog, MaxML,
	prevEntropy->matchlengthCTable, sizeof(prevEntropy->matchlengthCTable),
	cTableWksp, cTableWkspSize);
	FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for MatchLengths failed");
	if (MLtype == set_compressed)
	fseMetadata->lastCountSize = countSize;
	op += countSize;
	fseMetadata->mlType = (symbolEncodingType_e) MLtype;
	} }
	assert((size_t) (op-ostart) <= sizeof(fseMetadata->fseTablesBuffer));
	return op-ostart;
	}


	/** ZSTD_buildSuperBlockEntropy() :
	* Builds entropy for the super-block.
	* @return : 0 on success or error code */
	static size_t
	ZSTD_buildSuperBlockEntropy(seqStore_t* seqStorePtr,
	const ZSTD_entropyCTables_t* prevEntropy,
	ZSTD_entropyCTables_t* nextEntropy,
	const ZSTD_CCtx_params* cctxParams,
	ZSTD_entropyCTablesMetadata_t* entropyMetadata,
	void* workspace, size_t wkspSize)
	{
	size_t const litSize = seqStorePtr->lit - seqStorePtr->litStart;
	DEBUGLOG(5, "ZSTD_buildSuperBlockEntropy");
	entropyMetadata->hufMetadata.hufDesSize =
	ZSTD_buildSuperBlockEntropy_literal(seqStorePtr->litStart, litSize,
	&prevEntropy->huf, &nextEntropy->huf,
	&entropyMetadata->hufMetadata,
	ZSTD_disableLiteralsCompression(cctxParams),
	workspace, wkspSize);
	FORWARD_IF_ERROR(entropyMetadata->hufMetadata.hufDesSize, "ZSTD_buildSuperBlockEntropy_literal failed");
	entropyMetadata->fseMetadata.fseTablesSize =
	ZSTD_buildSuperBlockEntropy_sequences(seqStorePtr,
	&prevEntropy->fse, &nextEntropy->fse,
	cctxParams,
	&entropyMetadata->fseMetadata,
	workspace, wkspSize);
	FORWARD_IF_ERROR(entropyMetadata->fseMetadata.fseTablesSize, "ZSTD_buildSuperBlockEntropy_sequences failed");
	return 0;
	}

	/** ZSTD_compressSubBlock_literal() :
	* Compresses literals section for a sub-block.
	* When we have to write the Huffman table we will sometimes choose a header
	* size larger than necessary. This is because we have to pick the header size
	* before we know the table size + compressed size, so we have a bound on the
	* table size. If we guessed incorrectly, we fall back to uncompressed literals.
	*
	* We write the header when writeEntropy=1 and set entropyWrriten=1 when we succeeded
	* in writing the header, otherwise it is set to 0.
	*
	* hufMetadata->hType has literals block type info.
	* If it is set_basic, all sub-blocks literals section will be Raw_Literals_Block.
	* If it is set_rle, all sub-blocks literals section will be RLE_Literals_Block.
	* If it is set_compressed, first sub-block's literals section will be Compressed_Literals_Block
	* If it is set_compressed, first sub-block's literals section will be Treeless_Literals_Block
	* and the following sub-blocks' literals sections will be Treeless_Literals_Block.
	* @return : compressed size of literals section of a sub-block
	* Or 0 if it unable to compress.
	* Or error code */
	static size_t ZSTD_compressSubBlock_literal(const HUF_CElt* hufTable,
	const ZSTD_hufCTablesMetadata_t* hufMetadata,
	const BYTE* literals, size_t litSize,
	void* dst, size_t dstSize,
	const int bmi2, int writeEntropy, int* entropyWritten)
	{
	size_t const header = writeEntropy ? 200 : 0;
	size_t const lhSize = 3 + (litSize >= (1 KB - header)) + (litSize >= (16 KB - header));
	BYTE* const ostart = (BYTE*)dst;
	BYTE* const oend = ostart + dstSize;
	BYTE* op = ostart + lhSize;
	U32 const singleStream = lhSize == 3;
	symbolEncodingType_e hType = writeEntropy ? hufMetadata->hType : set_repeat;
	size_t cLitSize = 0;

	(void)bmi2; /* TODO bmi2... */

	DEBUGLOG(5, "ZSTD_compressSubBlock_literal (litSize=%zu, lhSize=%zu, writeEntropy=%d)", litSize, lhSize, writeEntropy);

	*entropyWritten = 0;
	if (litSize == 0 \|\| hufMetadata->hType == set_basic) {
	DEBUGLOG(5, "ZSTD_compressSubBlock_literal using raw literal");
	return ZSTD_noCompressLiterals(dst, dstSize, literals, litSize);
	} else if (hufMetadata->hType == set_rle) {
	DEBUGLOG(5, "ZSTD_compressSubBlock_literal using rle literal");
	return ZSTD_compressRleLiteralsBlock(dst, dstSize, literals, litSize);
	}

	assert(litSize > 0);
	assert(hufMetadata->hType == set_compressed \|\| hufMetadata->hType == set_repeat);

	if (writeEntropy && hufMetadata->hType == set_compressed) {
	memcpy(op, hufMetadata->hufDesBuffer, hufMetadata->hufDesSize);
	op += hufMetadata->hufDesSize;
	cLitSize += hufMetadata->hufDesSize;
	DEBUGLOG(5, "ZSTD_compressSubBlock_literal (hSize=%zu)", hufMetadata->hufDesSize);
	}

	/* TODO bmi2 */
	{ const size_t cSize = singleStream ? HUF_compress1X_usingCTable(op, oend-op, literals, litSize, hufTable)
	: HUF_compress4X_usingCTable(op, oend-op, literals, litSize, hufTable);
	op += cSize;
	cLitSize += cSize;
	if (cSize == 0 \|\| ERR_isError(cSize)) {
	DEBUGLOG(5, "Failed to write entropy tables %s", ZSTD_getErrorName(cSize));
	return 0;
	}
	/* If we expand and we aren't writing a header then emit uncompressed */
	if (!writeEntropy && cLitSize >= litSize) {
	DEBUGLOG(5, "ZSTD_compressSubBlock_literal using raw literal because uncompressible");
	return ZSTD_noCompressLiterals(dst, dstSize, literals, litSize);
	}
	/* If we are writing headers then allow expansion that doesn't change our header size. */
	if (lhSize < (size_t)(3 + (cLitSize >= 1 KB) + (cLitSize >= 16 KB))) {
	assert(cLitSize > litSize);
	DEBUGLOG(5, "Literals expanded beyond allowed header size");
	return ZSTD_noCompressLiterals(dst, dstSize, literals, litSize);
	}
	DEBUGLOG(5, "ZSTD_compressSubBlock_literal (cSize=%zu)", cSize);
	}

	/* Build header */
	switch(lhSize)
	{
	case 3: /* 2 - 2 - 10 - 10 */
	{ U32 const lhc = hType + ((!singleStream) << 2) + ((U32)litSize<<4) + ((U32)cLitSize<<14);
	MEM_writeLE24(ostart, lhc);
	break;
	}
	case 4: /* 2 - 2 - 14 - 14 */
	{ U32 const lhc = hType + (2 << 2) + ((U32)litSize<<4) + ((U32)cLitSize<<18);
	MEM_writeLE32(ostart, lhc);
	break;
	}
	case 5: /* 2 - 2 - 18 - 18 */
	{ U32 const lhc = hType + (3 << 2) + ((U32)litSize<<4) + ((U32)cLitSize<<22);
	MEM_writeLE32(ostart, lhc);
	ostart[4] = (BYTE)(cLitSize >> 10);
	break;
	}
	default: /* not possible : lhSize is {3,4,5} */
	assert(0);
	}
	*entropyWritten = 1;
	DEBUGLOG(5, "Compressed literals: %u -> %u", (U32)litSize, (U32)(op-ostart));
	return op-ostart;
	}

	static size_t ZSTD_seqDecompressedSize(seqStore_t const* seqStore, const seqDef* sequences, size_t nbSeq, size_t litSize, int lastSequence) {
	const seqDef* const sstart = sequences;
	const seqDef* const send = sequences + nbSeq;
	const seqDef* sp = sstart;
	size_t matchLengthSum = 0;
	size_t litLengthSum = 0;
	while (send-sp > 0) {
	ZSTD_sequenceLength const seqLen = ZSTD_getSequenceLength(seqStore, sp);
	litLengthSum += seqLen.litLength;
	matchLengthSum += seqLen.matchLength;
	sp++;
	}
	assert(litLengthSum <= litSize);
	if (!lastSequence) {
	assert(litLengthSum == litSize);
	}
	return matchLengthSum + litSize;
	}

	/** ZSTD_compressSubBlock_sequences() :
	* Compresses sequences section for a sub-block.
	* fseMetadata->llType, fseMetadata->ofType, and fseMetadata->mlType have
	* symbol compression modes for the super-block.
	* The first successfully compressed block will have these in its header.
	* We set entropyWritten=1 when we succeed in compressing the sequences.
	* The following sub-blocks will always have repeat mode.
	* @return : compressed size of sequences section of a sub-block
	* Or 0 if it is unable to compress
	* Or error code. */
	static size_t ZSTD_compressSubBlock_sequences(const ZSTD_fseCTables_t* fseTables,
	const ZSTD_fseCTablesMetadata_t* fseMetadata,
	const seqDef* sequences, size_t nbSeq,
	const BYTE* llCode, const BYTE* mlCode, const BYTE* ofCode,
	const ZSTD_CCtx_params* cctxParams,
	void* dst, size_t dstCapacity,
	const int bmi2, int writeEntropy, int* entropyWritten)
	{
	const int longOffsets = cctxParams->cParams.windowLog > STREAM_ACCUMULATOR_MIN;
	BYTE* const ostart = (BYTE*)dst;
	BYTE* const oend = ostart + dstCapacity;
	BYTE* op = ostart;
	BYTE* seqHead;

	DEBUGLOG(5, "ZSTD_compressSubBlock_sequences (nbSeq=%zu, writeEntropy=%d, longOffsets=%d)", nbSeq, writeEntropy, longOffsets);

	*entropyWritten = 0;
	/* Sequences Header */
	RETURN_ERROR_IF((oend-op) < 3 /max nbSeq Size/ + 1 /seqHead/,
	dstSize_tooSmall, "");
	if (nbSeq < 0x7F)
	*op++ = (BYTE)nbSeq;
	else if (nbSeq < LONGNBSEQ)
	op[0] = (BYTE)((nbSeq>>8) + 0x80), op[1] = (BYTE)nbSeq, op+=2;
	else
	op[0]=0xFF, MEM_writeLE16(op+1, (U16)(nbSeq - LONGNBSEQ)), op+=3;
	if (nbSeq==0) {
	return op - ostart;
	}

	/* seqHead : flags for FSE encoding type */
	seqHead = op++;

	DEBUGLOG(5, "ZSTD_compressSubBlock_sequences (seqHeadSize=%u)", (unsigned)(op-ostart));

	if (writeEntropy) {
	const U32 LLtype = fseMetadata->llType;
	const U32 Offtype = fseMetadata->ofType;
	const U32 MLtype = fseMetadata->mlType;
	DEBUGLOG(5, "ZSTD_compressSubBlock_sequences (fseTablesSize=%zu)", fseMetadata->fseTablesSize);
	*seqHead = (BYTE)((LLtype<<6) + (Offtype<<4) + (MLtype<<2));
	memcpy(op, fseMetadata->fseTablesBuffer, fseMetadata->fseTablesSize);
	op += fseMetadata->fseTablesSize;
	} else {
	const U32 repeat = set_repeat;
	*seqHead = (BYTE)((repeat<<6) + (repeat<<4) + (repeat<<2));
	}

	{ size_t const bitstreamSize = ZSTD_encodeSequences(
	op, oend - op,
	fseTables->matchlengthCTable, mlCode,
	fseTables->offcodeCTable, ofCode,
	fseTables->litlengthCTable, llCode,
	sequences, nbSeq,
	longOffsets, bmi2);
	FORWARD_IF_ERROR(bitstreamSize, "ZSTD_encodeSequences failed");
	op += bitstreamSize;
	/* zstd versions <= 1.3.4 mistakenly report corruption when
	* FSE_readNCount() receives a buffer < 4 bytes.
	* Fixed by https://github.com/facebook/zstd/pull/1146.
	* This can happen when the last set_compressed table present is 2
	* bytes and the bitstream is only one byte.
	* In this exceedingly rare case, we will simply emit an uncompressed
	* block, since it isn't worth optimizing.
	*/
	#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
	if (writeEntropy && fseMetadata->lastCountSize && fseMetadata->lastCountSize + bitstreamSize < 4) {
	/* NCountSize >= 2 && bitstreamSize > 0 ==> lastCountSize == 3 */
	assert(fseMetadata->lastCountSize + bitstreamSize == 3);
	DEBUGLOG(5, "Avoiding bug in zstd decoder in versions <= 1.3.4 by "
	"emitting an uncompressed block.");
	return 0;
	}
	#endif
	DEBUGLOG(5, "ZSTD_compressSubBlock_sequences (bitstreamSize=%zu)", bitstreamSize);
	}

	/* zstd versions <= 1.4.0 mistakenly report error when
	* sequences section body size is less than 3 bytes.
	* Fixed by https://github.com/facebook/zstd/pull/1664.
	* This can happen when the previous sequences section block is compressed
	* with rle mode and the current block's sequences section is compressed
	* with repeat mode where sequences section body size can be 1 byte.
	*/
	#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
	if (op-seqHead < 4) {
	DEBUGLOG(5, "Avoiding bug in zstd decoder in versions <= 1.4.0 by emitting "
	"an uncompressed block when sequences are < 4 bytes");
	return 0;
	}
	#endif

	*entropyWritten = 1;
	return op - ostart;
	}

	/** ZSTD_compressSubBlock() :
	* Compresses a single sub-block.
	* @return : compressed size of the sub-block
	* Or 0 if it failed to compress. */
	static size_t ZSTD_compressSubBlock(const ZSTD_entropyCTables_t* entropy,
	const ZSTD_entropyCTablesMetadata_t* entropyMetadata,
	const seqDef* sequences, size_t nbSeq,
	const BYTE* literals, size_t litSize,
	const BYTE* llCode, const BYTE* mlCode, const BYTE* ofCode,
	const ZSTD_CCtx_params* cctxParams,
	void* dst, size_t dstCapacity,
	const int bmi2,
	int writeLitEntropy, int writeSeqEntropy,
	int* litEntropyWritten, int* seqEntropyWritten,
	U32 lastBlock)
	{
	BYTE* const ostart = (BYTE*)dst;
	BYTE* const oend = ostart + dstCapacity;
	BYTE* op = ostart + ZSTD_blockHeaderSize;
	DEBUGLOG(5, "ZSTD_compressSubBlock (litSize=%zu, nbSeq=%zu, writeLitEntropy=%d, writeSeqEntropy=%d, lastBlock=%d)",
	litSize, nbSeq, writeLitEntropy, writeSeqEntropy, lastBlock);
	{ size_t cLitSize = ZSTD_compressSubBlock_literal((const HUF_CElt*)entropy->huf.CTable,
	&entropyMetadata->hufMetadata, literals, litSize,
	op, oend-op, bmi2, writeLitEntropy, litEntropyWritten);
	FORWARD_IF_ERROR(cLitSize, "ZSTD_compressSubBlock_literal failed");
	if (cLitSize == 0) return 0;
	op += cLitSize;
	}
	{ size_t cSeqSize = ZSTD_compressSubBlock_sequences(&entropy->fse,
	&entropyMetadata->fseMetadata,
	sequences, nbSeq,
	llCode, mlCode, ofCode,
	cctxParams,
	op, oend-op,
	bmi2, writeSeqEntropy, seqEntropyWritten);
	FORWARD_IF_ERROR(cSeqSize, "ZSTD_compressSubBlock_sequences failed");
	if (cSeqSize == 0) return 0;
	op += cSeqSize;
	}
	/* Write block header */
	{ size_t cSize = (op-ostart)-ZSTD_blockHeaderSize;
	U32 const cBlockHeader24 = lastBlock + (((U32)bt_compressed)<<1) + (U32)(cSize << 3);
	MEM_writeLE24(ostart, cBlockHeader24);
	}
	return op-ostart;
	}

	static size_t ZSTD_estimateSubBlockSize_literal(const BYTE* literals, size_t litSize,
	const ZSTD_hufCTables_t* huf,
	const ZSTD_hufCTablesMetadata_t* hufMetadata,
	void* workspace, size_t wkspSize,
	int writeEntropy)
	{
	unsigned* const countWksp = (unsigned*)workspace;
	unsigned maxSymbolValue = 255;
	size_t literalSectionHeaderSize = 3; /* Use hard coded size of 3 bytes */

	if (hufMetadata->hType == set_basic) return litSize;
	else if (hufMetadata->hType == set_rle) return 1;
	else if (hufMetadata->hType == set_compressed \|\| hufMetadata->hType == set_repeat) {
	size_t const largest = HIST_count_wksp (countWksp, &maxSymbolValue, (const BYTE*)literals, litSize, workspace, wkspSize);
	if (ZSTD_isError(largest)) return litSize;
	{ size_t cLitSizeEstimate = HUF_estimateCompressedSize((const HUF_CElt*)huf->CTable, countWksp, maxSymbolValue);
	if (writeEntropy) cLitSizeEstimate += hufMetadata->hufDesSize;
	return cLitSizeEstimate + literalSectionHeaderSize;
	} }
	assert(0); /* impossible */
	return 0;
	}

	static size_t ZSTD_estimateSubBlockSize_symbolType(symbolEncodingType_e type,
	const BYTE* codeTable, unsigned maxCode,
	size_t nbSeq, const FSE_CTable* fseCTable,
	const U32* additionalBits,
	short const* defaultNorm, U32 defaultNormLog,
	void* workspace, size_t wkspSize)
	{
	unsigned* const countWksp = (unsigned*)workspace;
	const BYTE* ctp = codeTable;
	const BYTE* const ctStart = ctp;
	const BYTE* const ctEnd = ctStart + nbSeq;
	size_t cSymbolTypeSizeEstimateInBits = 0;
	unsigned max = maxCode;

	HIST_countFast_wksp(countWksp, &max, codeTable, nbSeq, workspace, wkspSize); /* can't fail */
	if (type == set_basic) {
	cSymbolTypeSizeEstimateInBits = ZSTD_crossEntropyCost(defaultNorm, defaultNormLog, countWksp, max);
	} else if (type == set_rle) {
	cSymbolTypeSizeEstimateInBits = 0;
	} else if (type == set_compressed \|\| type == set_repeat) {
	cSymbolTypeSizeEstimateInBits = ZSTD_fseBitCost(fseCTable, countWksp, max);
	}
	if (ZSTD_isError(cSymbolTypeSizeEstimateInBits)) return nbSeq * 10;
	while (ctp < ctEnd) {
	if (additionalBits) cSymbolTypeSizeEstimateInBits += additionalBits[*ctp];
	else cSymbolTypeSizeEstimateInBits += ctp; / for offset, offset code is also the number of additional bits */
	ctp++;
	}
	return cSymbolTypeSizeEstimateInBits / 8;
	}

	static size_t ZSTD_estimateSubBlockSize_sequences(const BYTE* ofCodeTable,
	const BYTE* llCodeTable,
	const BYTE* mlCodeTable,
	size_t nbSeq,
	const ZSTD_fseCTables_t* fseTables,
	const ZSTD_fseCTablesMetadata_t* fseMetadata,
	void* workspace, size_t wkspSize,
	int writeEntropy)
	{
	size_t sequencesSectionHeaderSize = 3; /* Use hard coded size of 3 bytes */
	size_t cSeqSizeEstimate = 0;
	cSeqSizeEstimate += ZSTD_estimateSubBlockSize_symbolType(fseMetadata->ofType, ofCodeTable, MaxOff,
	nbSeq, fseTables->offcodeCTable, NULL,
	OF_defaultNorm, OF_defaultNormLog,
	workspace, wkspSize);
	cSeqSizeEstimate += ZSTD_estimateSubBlockSize_symbolType(fseMetadata->llType, llCodeTable, MaxLL,
	nbSeq, fseTables->litlengthCTable, LL_bits,
	LL_defaultNorm, LL_defaultNormLog,
	workspace, wkspSize);
	cSeqSizeEstimate += ZSTD_estimateSubBlockSize_symbolType(fseMetadata->mlType, mlCodeTable, MaxML,
	nbSeq, fseTables->matchlengthCTable, ML_bits,
	ML_defaultNorm, ML_defaultNormLog,
	workspace, wkspSize);
	if (writeEntropy) cSeqSizeEstimate += fseMetadata->fseTablesSize;
	return cSeqSizeEstimate + sequencesSectionHeaderSize;
	}

	static size_t ZSTD_estimateSubBlockSize(const BYTE* literals, size_t litSize,
	const BYTE* ofCodeTable,
	const BYTE* llCodeTable,
	const BYTE* mlCodeTable,
	size_t nbSeq,
	const ZSTD_entropyCTables_t* entropy,
	const ZSTD_entropyCTablesMetadata_t* entropyMetadata,
	void* workspace, size_t wkspSize,
	int writeLitEntropy, int writeSeqEntropy) {
	size_t cSizeEstimate = 0;
	cSizeEstimate += ZSTD_estimateSubBlockSize_literal(literals, litSize,
	&entropy->huf, &entropyMetadata->hufMetadata,
	workspace, wkspSize, writeLitEntropy);
	cSizeEstimate += ZSTD_estimateSubBlockSize_sequences(ofCodeTable, llCodeTable, mlCodeTable,
	nbSeq, &entropy->fse, &entropyMetadata->fseMetadata,
	workspace, wkspSize, writeSeqEntropy);
	return cSizeEstimate + ZSTD_blockHeaderSize;
	}

	static int ZSTD_needSequenceEntropyTables(ZSTD_fseCTablesMetadata_t const* fseMetadata)
	{
	if (fseMetadata->llType == set_compressed \|\| fseMetadata->llType == set_rle)
	return 1;
	if (fseMetadata->mlType == set_compressed \|\| fseMetadata->mlType == set_rle)
	return 1;
	if (fseMetadata->ofType == set_compressed \|\| fseMetadata->ofType == set_rle)
	return 1;
	return 0;
	}

	/** ZSTD_compressSubBlock_multi() :
	* Breaks super-block into multiple sub-blocks and compresses them.
	* Entropy will be written to the first block.
	* The following blocks will use repeat mode to compress.
	* All sub-blocks are compressed blocks (no raw or rle blocks).
	* @return : compressed size of the super block (which is multiple ZSTD blocks)
	* Or 0 if it failed to compress. */
	static size_t ZSTD_compressSubBlock_multi(const seqStore_t* seqStorePtr,
	const ZSTD_compressedBlockState_t* prevCBlock,
	ZSTD_compressedBlockState_t* nextCBlock,
	const ZSTD_entropyCTablesMetadata_t* entropyMetadata,
	const ZSTD_CCtx_params* cctxParams,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const int bmi2, U32 lastBlock,
	void* workspace, size_t wkspSize)
	{
	const seqDef* const sstart = seqStorePtr->sequencesStart;
	const seqDef* const send = seqStorePtr->sequences;
	const seqDef* sp = sstart;
	const BYTE* const lstart = seqStorePtr->litStart;
	const BYTE* const lend = seqStorePtr->lit;
	const BYTE* lp = lstart;
	BYTE const* ip = (BYTE const*)src;
	BYTE const* const iend = ip + srcSize;
	BYTE* const ostart = (BYTE*)dst;
	BYTE* const oend = ostart + dstCapacity;
	BYTE* op = ostart;
	const BYTE* llCodePtr = seqStorePtr->llCode;
	const BYTE* mlCodePtr = seqStorePtr->mlCode;
	const BYTE* ofCodePtr = seqStorePtr->ofCode;
	size_t targetCBlockSize = cctxParams->targetCBlockSize;
	size_t litSize, seqCount;
	int writeLitEntropy = entropyMetadata->hufMetadata.hType == set_compressed;
	int writeSeqEntropy = 1;
	int lastSequence = 0;

	DEBUGLOG(5, "ZSTD_compressSubBlock_multi (litSize=%u, nbSeq=%u)",
	(unsigned)(lend-lp), (unsigned)(send-sstart));

	litSize = 0;
	seqCount = 0;
	do {
	size_t cBlockSizeEstimate = 0;
	if (sstart == send) {
	lastSequence = 1;
	} else {
	const seqDef* const sequence = sp + seqCount;
	lastSequence = sequence == send - 1;
	litSize += ZSTD_getSequenceLength(seqStorePtr, sequence).litLength;
	seqCount++;
	}
	if (lastSequence) {
	assert(lp <= lend);
	assert(litSize <= (size_t)(lend - lp));
	litSize = (size_t)(lend - lp);
	}
	/* I think there is an optimization opportunity here.
	* Calling ZSTD_estimateSubBlockSize for every sequence can be wasteful
	* since it recalculates estimate from scratch.
	* For example, it would recount literal distribution and symbol codes everytime.
	*/
	cBlockSizeEstimate = ZSTD_estimateSubBlockSize(lp, litSize, ofCodePtr, llCodePtr, mlCodePtr, seqCount,
	&nextCBlock->entropy, entropyMetadata,
	workspace, wkspSize, writeLitEntropy, writeSeqEntropy);
	if (cBlockSizeEstimate > targetCBlockSize \|\| lastSequence) {
	int litEntropyWritten = 0;
	int seqEntropyWritten = 0;
	const size_t decompressedSize = ZSTD_seqDecompressedSize(seqStorePtr, sp, seqCount, litSize, lastSequence);
	const size_t cSize = ZSTD_compressSubBlock(&nextCBlock->entropy, entropyMetadata,
	sp, seqCount,
	lp, litSize,
	llCodePtr, mlCodePtr, ofCodePtr,
	cctxParams,
	op, oend-op,
	bmi2, writeLitEntropy, writeSeqEntropy,
	&litEntropyWritten, &seqEntropyWritten,
	lastBlock && lastSequence);
	FORWARD_IF_ERROR(cSize, "ZSTD_compressSubBlock failed");
	if (cSize > 0 && cSize < decompressedSize) {
	DEBUGLOG(5, "Committed the sub-block");
	assert(ip + decompressedSize <= iend);
	ip += decompressedSize;
	sp += seqCount;
	lp += litSize;
	op += cSize;
	llCodePtr += seqCount;
	mlCodePtr += seqCount;
	ofCodePtr += seqCount;
	litSize = 0;
	seqCount = 0;
	/* Entropy only needs to be written once */
	if (litEntropyWritten) {
	writeLitEntropy = 0;
	}
	if (seqEntropyWritten) {
	writeSeqEntropy = 0;
	}
	}
	}
	} while (!lastSequence);
	if (writeLitEntropy) {
	DEBUGLOG(5, "ZSTD_compressSubBlock_multi has literal entropy tables unwritten");
	memcpy(&nextCBlock->entropy.huf, &prevCBlock->entropy.huf, sizeof(prevCBlock->entropy.huf));
	}
	if (writeSeqEntropy && ZSTD_needSequenceEntropyTables(&entropyMetadata->fseMetadata)) {
	/* If we haven't written our entropy tables, then we've violated our contract and
	* must emit an uncompressed block.
	*/
	DEBUGLOG(5, "ZSTD_compressSubBlock_multi has sequence entropy tables unwritten");
	return 0;
	}
	if (ip < iend) {
	size_t const cSize = ZSTD_noCompressBlock(op, oend - op, ip, iend - ip, lastBlock);
	DEBUGLOG(5, "ZSTD_compressSubBlock_multi last sub-block uncompressed, %zu bytes", (size_t)(iend - ip));
	FORWARD_IF_ERROR(cSize, "ZSTD_noCompressBlock failed");
	assert(cSize != 0);
	op += cSize;
	/* We have to regenerate the repcodes because we've skipped some sequences */
	if (sp < send) {
	seqDef const* seq;
	repcodes_t rep;
	memcpy(&rep, prevCBlock->rep, sizeof(rep));
	for (seq = sstart; seq < sp; ++seq) {
	rep = ZSTD_updateRep(rep.rep, seq->offset - 1, ZSTD_getSequenceLength(seqStorePtr, seq).litLength == 0);
	}
	memcpy(nextCBlock->rep, &rep, sizeof(rep));
	}
	}
	DEBUGLOG(5, "ZSTD_compressSubBlock_multi compressed");
	return op-ostart;
	}

	size_t ZSTD_compressSuperBlock(ZSTD_CCtx* zc,
	void* dst, size_t dstCapacity,
	void const* src, size_t srcSize,
	unsigned lastBlock) {
	ZSTD_entropyCTablesMetadata_t entropyMetadata;

	FORWARD_IF_ERROR(ZSTD_buildSuperBlockEntropy(&zc->seqStore,
	&zc->blockState.prevCBlock->entropy,
	&zc->blockState.nextCBlock->entropy,
	&zc->appliedParams,
	&entropyMetadata,
	zc->entropyWorkspace, HUF_WORKSPACE_SIZE /* statically allocated in resetCCtx */), "");

	return ZSTD_compressSubBlock_multi(&zc->seqStore,
	zc->blockState.prevCBlock,
	zc->blockState.nextCBlock,
	&entropyMetadata,
	&zc->appliedParams,
	dst, dstCapacity,
	src, srcSize,
	zc->bmi2, lastBlock,
	zc->entropyWorkspace, HUF_WORKSPACE_SIZE /* statically allocated in resetCCtx */);
	}
	/** ended inlining compress/zstd_compress_superblock.c **/
	/** start inlining compress/zstd_compress.c **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	/-************************************
	* Dependencies
	***************************************/
	#include <limits.h> /* INT_MAX */
	#include <string.h> /* memset */
	/** start inlining ../common/cpu.h **/
	/*
	* Copyright (c) 2018-2020, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTD_COMMON_CPU_H
	#define ZSTD_COMMON_CPU_H

	/**
	* Implementation taken from folly/CpuId.h
	* https://github.com/facebook/folly/blob/master/folly/CpuId.h
	*/

	#include <string.h>

	/** skipping file: mem.h **/

	#ifdef _MSC_VER
	#include <intrin.h>
	#endif

	typedef struct {
	U32 f1c;
	U32 f1d;
	U32 f7b;
	U32 f7c;
	} ZSTD_cpuid_t;

	MEM_STATIC ZSTD_cpuid_t ZSTD_cpuid(void) {
	U32 f1c = 0;
	U32 f1d = 0;
	U32 f7b = 0;
	U32 f7c = 0;
	#if defined(_MSC_VER) && (defined(_M_X64) \|\| defined(_M_IX86))
	int reg[4];
	__cpuid((int*)reg, 0);
	{
	int const n = reg[0];
	if (n >= 1) {
	__cpuid((int*)reg, 1);
	f1c = (U32)reg[2];
	f1d = (U32)reg[3];
	}
	if (n >= 7) {
	__cpuidex((int*)reg, 7, 0);
	f7b = (U32)reg[1];
	f7c = (U32)reg[2];
	}
	}
	#elif defined(__i386__) && defined(__PIC__) && !defined(__clang__) && defined(__GNUC__)
	/* The following block like the normal cpuid branch below, but gcc
	* reserves ebx for use of its pic register so we must specially
	* handle the save and restore to avoid clobbering the register
	*/
	U32 n;
	__asm__(
	"pushl %%ebx\n\t"
	"cpuid\n\t"
	"popl %%ebx\n\t"
	: "=a"(n)
	: "a"(0)
	: "ecx", "edx");
	if (n >= 1) {
	U32 f1a;
	__asm__(
	"pushl %%ebx\n\t"
	"cpuid\n\t"
	"popl %%ebx\n\t"
	: "=a"(f1a), "=c"(f1c), "=d"(f1d)
	: "a"(1));
	}
	if (n >= 7) {
	__asm__(
	"pushl %%ebx\n\t"
	"cpuid\n\t"
	"movl %%ebx, %%eax\n\t"
	"popl %%ebx"
	: "=a"(f7b), "=c"(f7c)
	: "a"(7), "c"(0)
	: "edx");
	}
	#elif defined(__x86_64__) \|\| defined(_M_X64) \|\| defined(__i386__)
	U32 n;
	__asm__("cpuid" : "=a"(n) : "a"(0) : "ebx", "ecx", "edx");
	if (n >= 1) {
	U32 f1a;
	__asm__("cpuid" : "=a"(f1a), "=c"(f1c), "=d"(f1d) : "a"(1) : "ebx");
	}
	if (n >= 7) {
	U32 f7a;
	__asm__("cpuid"
	: "=a"(f7a), "=b"(f7b), "=c"(f7c)
	: "a"(7), "c"(0)
	: "edx");
	}
	#endif
	{
	ZSTD_cpuid_t cpuid;
	cpuid.f1c = f1c;
	cpuid.f1d = f1d;
	cpuid.f7b = f7b;
	cpuid.f7c = f7c;
	return cpuid;
	}
	}

	#define X(name, r, bit) \
	MEM_STATIC int ZSTD_cpuid_##name(ZSTD_cpuid_t const cpuid) { \
	return ((cpuid.r) & (1U << bit)) != 0; \
	}

	/* cpuid(1): Processor Info and Feature Bits. */
	#define C(name, bit) X(name, f1c, bit)
	C(sse3, 0)
	C(pclmuldq, 1)
	C(dtes64, 2)
	C(monitor, 3)
	C(dscpl, 4)
	C(vmx, 5)
	C(smx, 6)
	C(eist, 7)
	C(tm2, 8)
	C(ssse3, 9)
	C(cnxtid, 10)
	C(fma, 12)
	C(cx16, 13)
	C(xtpr, 14)
	C(pdcm, 15)
	C(pcid, 17)
	C(dca, 18)
	C(sse41, 19)
	C(sse42, 20)
	C(x2apic, 21)
	C(movbe, 22)
	C(popcnt, 23)
	C(tscdeadline, 24)
	C(aes, 25)
	C(xsave, 26)
	C(osxsave, 27)
	C(avx, 28)
	C(f16c, 29)
	C(rdrand, 30)
	#undef C
	#define D(name, bit) X(name, f1d, bit)
	D(fpu, 0)
	D(vme, 1)
	D(de, 2)
	D(pse, 3)
	D(tsc, 4)
	D(msr, 5)
	D(pae, 6)
	D(mce, 7)
	D(cx8, 8)
	D(apic, 9)
	D(sep, 11)
	D(mtrr, 12)
	D(pge, 13)
	D(mca, 14)
	D(cmov, 15)
	D(pat, 16)
	D(pse36, 17)
	D(psn, 18)
	D(clfsh, 19)
	D(ds, 21)
	D(acpi, 22)
	D(mmx, 23)
	D(fxsr, 24)
	D(sse, 25)
	D(sse2, 26)
	D(ss, 27)
	D(htt, 28)
	D(tm, 29)
	D(pbe, 31)
	#undef D

	/* cpuid(7): Extended Features. */
	#define B(name, bit) X(name, f7b, bit)
	B(bmi1, 3)
	B(hle, 4)
	B(avx2, 5)
	B(smep, 7)
	B(bmi2, 8)
	B(erms, 9)
	B(invpcid, 10)
	B(rtm, 11)
	B(mpx, 14)
	B(avx512f, 16)
	B(avx512dq, 17)
	B(rdseed, 18)
	B(adx, 19)
	B(smap, 20)
	B(avx512ifma, 21)
	B(pcommit, 22)
	B(clflushopt, 23)
	B(clwb, 24)
	B(avx512pf, 26)
	B(avx512er, 27)
	B(avx512cd, 28)
	B(sha, 29)
	B(avx512bw, 30)
	B(avx512vl, 31)
	#undef B
	#define C(name, bit) X(name, f7c, bit)
	C(prefetchwt1, 0)
	C(avx512vbmi, 1)
	#undef C

	#undef X

	#endif /* ZSTD_COMMON_CPU_H */
	/** ended inlining ../common/cpu.h **/
	/** skipping file: ../common/mem.h **/
	/** skipping file: hist.h **/
	#define FSE_STATIC_LINKING_ONLY /* FSE_encodeSymbol */
	/** skipping file: ../common/fse.h **/
	#define HUF_STATIC_LINKING_ONLY
	/** skipping file: ../common/huf.h **/
	/** skipping file: zstd_compress_internal.h **/
	/** skipping file: zstd_compress_sequences.h **/
	/** skipping file: zstd_compress_literals.h **/
	/** start inlining zstd_fast.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTD_FAST_H
	#define ZSTD_FAST_H

	#if defined (__cplusplus)
	extern "C" {
	#endif

	/** skipping file: ../common/mem.h **/
	/** skipping file: zstd_compress_internal.h **/

	void ZSTD_fillHashTable(ZSTD_matchState_t* ms,
	void const* end, ZSTD_dictTableLoadMethod_e dtlm);
	size_t ZSTD_compressBlock_fast(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);
	size_t ZSTD_compressBlock_fast_dictMatchState(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);
	size_t ZSTD_compressBlock_fast_extDict(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);

	#if defined (__cplusplus)
	}
	#endif

	#endif /* ZSTD_FAST_H */
	/** ended inlining zstd_fast.h **/
	/** start inlining zstd_double_fast.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTD_DOUBLE_FAST_H
	#define ZSTD_DOUBLE_FAST_H

	#if defined (__cplusplus)
	extern "C" {
	#endif

	/** skipping file: ../common/mem.h **/
	/** skipping file: zstd_compress_internal.h **/

	void ZSTD_fillDoubleHashTable(ZSTD_matchState_t* ms,
	void const* end, ZSTD_dictTableLoadMethod_e dtlm);
	size_t ZSTD_compressBlock_doubleFast(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);
	size_t ZSTD_compressBlock_doubleFast_dictMatchState(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);
	size_t ZSTD_compressBlock_doubleFast_extDict(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);


	#if defined (__cplusplus)
	}
	#endif

	#endif /* ZSTD_DOUBLE_FAST_H */
	/** ended inlining zstd_double_fast.h **/
	/** start inlining zstd_lazy.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTD_LAZY_H
	#define ZSTD_LAZY_H

	#if defined (__cplusplus)
	extern "C" {
	#endif

	/** skipping file: zstd_compress_internal.h **/

	U32 ZSTD_insertAndFindFirstIndex(ZSTD_matchState_t* ms, const BYTE* ip);

	void ZSTD_preserveUnsortedMark (U32* const table, U32 const size, U32 const reducerValue); /! used in ZSTD_reduceIndex(). preemptively increase value of ZSTD_DUBT_UNSORTED_MARK /

	size_t ZSTD_compressBlock_btlazy2(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);
	size_t ZSTD_compressBlock_lazy2(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);
	size_t ZSTD_compressBlock_lazy(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);
	size_t ZSTD_compressBlock_greedy(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);

	size_t ZSTD_compressBlock_btlazy2_dictMatchState(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);
	size_t ZSTD_compressBlock_lazy2_dictMatchState(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);
	size_t ZSTD_compressBlock_lazy_dictMatchState(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);
	size_t ZSTD_compressBlock_greedy_dictMatchState(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);

	size_t ZSTD_compressBlock_greedy_extDict(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);
	size_t ZSTD_compressBlock_lazy_extDict(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);
	size_t ZSTD_compressBlock_lazy2_extDict(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);
	size_t ZSTD_compressBlock_btlazy2_extDict(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);

	#if defined (__cplusplus)
	}
	#endif

	#endif /* ZSTD_LAZY_H */
	/** ended inlining zstd_lazy.h **/
	/** start inlining zstd_opt.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTD_OPT_H
	#define ZSTD_OPT_H

	#if defined (__cplusplus)
	extern "C" {
	#endif

	/** skipping file: zstd_compress_internal.h **/

	/* used in ZSTD_loadDictionaryContent() */
	void ZSTD_updateTree(ZSTD_matchState_t* ms, const BYTE* ip, const BYTE* iend);

	size_t ZSTD_compressBlock_btopt(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);
	size_t ZSTD_compressBlock_btultra(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);
	size_t ZSTD_compressBlock_btultra2(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);


	size_t ZSTD_compressBlock_btopt_dictMatchState(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);
	size_t ZSTD_compressBlock_btultra_dictMatchState(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);

	size_t ZSTD_compressBlock_btopt_extDict(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);
	size_t ZSTD_compressBlock_btultra_extDict(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);

	/* note : no btultra2 variant for extDict nor dictMatchState,
	* because btultra2 is not meant to work with dictionaries
	* and is only specific for the first block (no prefix) */

	#if defined (__cplusplus)
	}
	#endif

	#endif /* ZSTD_OPT_H */
	/** ended inlining zstd_opt.h **/
	/** start inlining zstd_ldm.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTD_LDM_H
	#define ZSTD_LDM_H

	#if defined (__cplusplus)
	extern "C" {
	#endif

	/** skipping file: zstd_compress_internal.h **/
	/** skipping file: ../zstd.h **/

	/-************************************
	* Long distance matching
	***************************************/

	#define ZSTD_LDM_DEFAULT_WINDOW_LOG ZSTD_WINDOWLOG_LIMIT_DEFAULT

	void ZSTD_ldm_fillHashTable(
	ldmState_t* state, const BYTE* ip,
	const BYTE* iend, ldmParams_t const* params);

	/**
	* ZSTD_ldm_generateSequences():
	*
	* Generates the sequences using the long distance match finder.
	* Generates long range matching sequences in `sequences`, which parse a prefix
	* of the source. `sequences` must be large enough to store every sequence,
	* which can be checked with `ZSTD_ldm_getMaxNbSeq()`.
	* @returns 0 or an error code.
	*
	* NOTE: The user must have called ZSTD_window_update() for all of the input
	* they have, even if they pass it to ZSTD_ldm_generateSequences() in chunks.
	* NOTE: This function returns an error if it runs out of space to store
	* sequences.
	*/
	size_t ZSTD_ldm_generateSequences(
	ldmState_t* ldms, rawSeqStore_t* sequences,
	ldmParams_t const* params, void const* src, size_t srcSize);

	/**
	* ZSTD_ldm_blockCompress():
	*
	* Compresses a block using the predefined sequences, along with a secondary
	* block compressor. The literals section of every sequence is passed to the
	* secondary block compressor, and those sequences are interspersed with the
	* predefined sequences. Returns the length of the last literals.
	* Updates `rawSeqStore.pos` to indicate how many sequences have been consumed.
	* `rawSeqStore.seq` may also be updated to split the last sequence between two
	* blocks.
	* @return The length of the last literals.
	*
	* NOTE: The source must be at most the maximum block size, but the predefined
	* sequences can be any size, and may be longer than the block. In the case that
	* they are longer than the block, the last sequences may need to be split into
	* two. We handle that case correctly, and update `rawSeqStore` appropriately.
	* NOTE: This function does not return any errors.
	*/
	size_t ZSTD_ldm_blockCompress(rawSeqStore_t* rawSeqStore,
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize);

	/**
	* ZSTD_ldm_skipSequences():
	*
	* Skip past `srcSize` bytes worth of sequences in `rawSeqStore`.
	* Avoids emitting matches less than `minMatch` bytes.
	* Must be called for data with is not passed to ZSTD_ldm_blockCompress().
	*/
	void ZSTD_ldm_skipSequences(rawSeqStore_t* rawSeqStore, size_t srcSize,
	U32 const minMatch);


	/** ZSTD_ldm_getTableSize() :
	* Estimate the space needed for long distance matching tables or 0 if LDM is
	* disabled.
	*/
	size_t ZSTD_ldm_getTableSize(ldmParams_t params);

	/** ZSTD_ldm_getSeqSpace() :
	* Return an upper bound on the number of sequences that can be produced by
	* the long distance matcher, or 0 if LDM is disabled.
	*/
	size_t ZSTD_ldm_getMaxNbSeq(ldmParams_t params, size_t maxChunkSize);

	/** ZSTD_ldm_adjustParameters() :
	* If the params->hashRateLog is not set, set it to its default value based on
	* windowLog and params->hashLog.
	*
	* Ensures that params->bucketSizeLog is <= params->hashLog (setting it to
	* params->hashLog if it is not).
	*
	* Ensures that the minMatchLength >= targetLength during optimal parsing.
	*/
	void ZSTD_ldm_adjustParameters(ldmParams_t* params,
	ZSTD_compressionParameters const* cParams);

	#if defined (__cplusplus)
	}
	#endif

	#endif /* ZSTD_FAST_H */
	/** ended inlining zstd_ldm.h **/
	/** skipping file: zstd_compress_superblock.h **/


	/-************************************
	* Helper functions
	***************************************/
	/* ZSTD_compressBound()
	* Note that the result from this function is only compatible with the "normal"
	* full-block strategy.
	* When there are a lot of small blocks due to frequent flush in streaming mode
	* the overhead of headers can make the compressed data to be larger than the
	* return value of ZSTD_compressBound().
	*/
	size_t ZSTD_compressBound(size_t srcSize) {
	return ZSTD_COMPRESSBOUND(srcSize);
	}


	/-************************************
	* Context memory management
	***************************************/
	struct ZSTD_CDict_s {
	const void* dictContent;
	size_t dictContentSize;
	U32* entropyWorkspace; /* entropy workspace of HUF_WORKSPACE_SIZE bytes */
	ZSTD_cwksp workspace;
	ZSTD_matchState_t matchState;
	ZSTD_compressedBlockState_t cBlockState;
	ZSTD_customMem customMem;
	U32 dictID;
	int compressionLevel; /* 0 indicates that advanced API was used to select CDict params */
	}; /* typedef'd to ZSTD_CDict within "zstd.h" */

	ZSTD_CCtx* ZSTD_createCCtx(void)
	{
	return ZSTD_createCCtx_advanced(ZSTD_defaultCMem);
	}

	static void ZSTD_initCCtx(ZSTD_CCtx* cctx, ZSTD_customMem memManager)
	{
	assert(cctx != NULL);
	memset(cctx, 0, sizeof(*cctx));
	cctx->customMem = memManager;
	cctx->bmi2 = ZSTD_cpuid_bmi2(ZSTD_cpuid());
	{ size_t const err = ZSTD_CCtx_reset(cctx, ZSTD_reset_parameters);
	assert(!ZSTD_isError(err));
	(void)err;
	}
	}

	ZSTD_CCtx* ZSTD_createCCtx_advanced(ZSTD_customMem customMem)
	{
	ZSTD_STATIC_ASSERT(zcss_init==0);
	ZSTD_STATIC_ASSERT(ZSTD_CONTENTSIZE_UNKNOWN==(0ULL - 1));
	if (!customMem.customAlloc ^ !customMem.customFree) return NULL;
	{ ZSTD_CCtx* const cctx = (ZSTD_CCtx*)ZSTD_malloc(sizeof(ZSTD_CCtx), customMem);
	if (!cctx) return NULL;
	ZSTD_initCCtx(cctx, customMem);
	return cctx;
	}
	}

	ZSTD_CCtx* ZSTD_initStaticCCtx(void* workspace, size_t workspaceSize)
	{
	ZSTD_cwksp ws;
	ZSTD_CCtx* cctx;
	if (workspaceSize <= sizeof(ZSTD_CCtx)) return NULL; /* minimum size */
	if ((size_t)workspace & 7) return NULL; /* must be 8-aligned */
	ZSTD_cwksp_init(&ws, workspace, workspaceSize);

	cctx = (ZSTD_CCtx*)ZSTD_cwksp_reserve_object(&ws, sizeof(ZSTD_CCtx));
	if (cctx == NULL) return NULL;

	memset(cctx, 0, sizeof(ZSTD_CCtx));
	ZSTD_cwksp_move(&cctx->workspace, &ws);
	cctx->staticSize = workspaceSize;

	/* statically sized space. entropyWorkspace never moves (but prev/next block swap places) */
	if (!ZSTD_cwksp_check_available(&cctx->workspace, HUF_WORKSPACE_SIZE + 2 * sizeof(ZSTD_compressedBlockState_t))) return NULL;
	cctx->blockState.prevCBlock = (ZSTD_compressedBlockState_t*)ZSTD_cwksp_reserve_object(&cctx->workspace, sizeof(ZSTD_compressedBlockState_t));
	cctx->blockState.nextCBlock = (ZSTD_compressedBlockState_t*)ZSTD_cwksp_reserve_object(&cctx->workspace, sizeof(ZSTD_compressedBlockState_t));
	cctx->entropyWorkspace = (U32*)ZSTD_cwksp_reserve_object(&cctx->workspace, HUF_WORKSPACE_SIZE);
	cctx->bmi2 = ZSTD_cpuid_bmi2(ZSTD_cpuid());
	return cctx;
	}

	/**
	* Clears and frees all of the dictionaries in the CCtx.
	*/
	static void ZSTD_clearAllDicts(ZSTD_CCtx* cctx)
	{
	ZSTD_free(cctx->localDict.dictBuffer, cctx->customMem);
	ZSTD_freeCDict(cctx->localDict.cdict);
	memset(&cctx->localDict, 0, sizeof(cctx->localDict));
	memset(&cctx->prefixDict, 0, sizeof(cctx->prefixDict));
	cctx->cdict = NULL;
	}

	static size_t ZSTD_sizeof_localDict(ZSTD_localDict dict)
	{
	size_t const bufferSize = dict.dictBuffer != NULL ? dict.dictSize : 0;
	size_t const cdictSize = ZSTD_sizeof_CDict(dict.cdict);
	return bufferSize + cdictSize;
	}

	static void ZSTD_freeCCtxContent(ZSTD_CCtx* cctx)
	{
	assert(cctx != NULL);
	assert(cctx->staticSize == 0);
	ZSTD_clearAllDicts(cctx);
	#ifdef ZSTD_MULTITHREAD
	ZSTDMT_freeCCtx(cctx->mtctx); cctx->mtctx = NULL;
	#endif
	ZSTD_cwksp_free(&cctx->workspace, cctx->customMem);
	}

	size_t ZSTD_freeCCtx(ZSTD_CCtx* cctx)
	{
	if (cctx==NULL) return 0; /* support free on NULL */
	RETURN_ERROR_IF(cctx->staticSize, memory_allocation,
	"not compatible with static CCtx");
	{
	int cctxInWorkspace = ZSTD_cwksp_owns_buffer(&cctx->workspace, cctx);
	ZSTD_freeCCtxContent(cctx);
	if (!cctxInWorkspace) {
	ZSTD_free(cctx, cctx->customMem);
	}
	}
	return 0;
	}


	static size_t ZSTD_sizeof_mtctx(const ZSTD_CCtx* cctx)
	{
	#ifdef ZSTD_MULTITHREAD
	return ZSTDMT_sizeof_CCtx(cctx->mtctx);
	#else
	(void)cctx;
	return 0;
	#endif
	}


	size_t ZSTD_sizeof_CCtx(const ZSTD_CCtx* cctx)
	{
	if (cctx==NULL) return 0; /* support sizeof on NULL */
	/* cctx may be in the workspace */
	return (cctx->workspace.workspace == cctx ? 0 : sizeof(*cctx))
	+ ZSTD_cwksp_sizeof(&cctx->workspace)
	+ ZSTD_sizeof_localDict(cctx->localDict)
	+ ZSTD_sizeof_mtctx(cctx);
	}

	size_t ZSTD_sizeof_CStream(const ZSTD_CStream* zcs)
	{
	return ZSTD_sizeof_CCtx(zcs); /* same object */
	}

	/* private API call, for dictBuilder only */
	const seqStore_t* ZSTD_getSeqStore(const ZSTD_CCtx* ctx) { return &(ctx->seqStore); }

	static ZSTD_CCtx_params ZSTD_makeCCtxParamsFromCParams(
	ZSTD_compressionParameters cParams)
	{
	ZSTD_CCtx_params cctxParams;
	memset(&cctxParams, 0, sizeof(cctxParams));
	cctxParams.cParams = cParams;
	cctxParams.compressionLevel = ZSTD_CLEVEL_DEFAULT; /* should not matter, as all cParams are presumed properly defined */
	assert(!ZSTD_checkCParams(cParams));
	cctxParams.fParams.contentSizeFlag = 1;
	return cctxParams;
	}

	static ZSTD_CCtx_params* ZSTD_createCCtxParams_advanced(
	ZSTD_customMem customMem)
	{
	ZSTD_CCtx_params* params;
	if (!customMem.customAlloc ^ !customMem.customFree) return NULL;
	params = (ZSTD_CCtx_params*)ZSTD_calloc(
	sizeof(ZSTD_CCtx_params), customMem);
	if (!params) { return NULL; }
	params->customMem = customMem;
	params->compressionLevel = ZSTD_CLEVEL_DEFAULT;
	params->fParams.contentSizeFlag = 1;
	return params;
	}

	ZSTD_CCtx_params* ZSTD_createCCtxParams(void)
	{
	return ZSTD_createCCtxParams_advanced(ZSTD_defaultCMem);
	}

	size_t ZSTD_freeCCtxParams(ZSTD_CCtx_params* params)
	{
	if (params == NULL) { return 0; }
	ZSTD_free(params, params->customMem);
	return 0;
	}

	size_t ZSTD_CCtxParams_reset(ZSTD_CCtx_params* params)
	{
	return ZSTD_CCtxParams_init(params, ZSTD_CLEVEL_DEFAULT);
	}

	size_t ZSTD_CCtxParams_init(ZSTD_CCtx_params* cctxParams, int compressionLevel) {
	RETURN_ERROR_IF(!cctxParams, GENERIC, "NULL pointer!");
	memset(cctxParams, 0, sizeof(*cctxParams));
	cctxParams->compressionLevel = compressionLevel;
	cctxParams->fParams.contentSizeFlag = 1;
	return 0;
	}

	size_t ZSTD_CCtxParams_init_advanced(ZSTD_CCtx_params* cctxParams, ZSTD_parameters params)
	{
	RETURN_ERROR_IF(!cctxParams, GENERIC, "NULL pointer!");
	FORWARD_IF_ERROR( ZSTD_checkCParams(params.cParams) , "");
	memset(cctxParams, 0, sizeof(*cctxParams));
	assert(!ZSTD_checkCParams(params.cParams));
	cctxParams->cParams = params.cParams;
	cctxParams->fParams = params.fParams;
	cctxParams->compressionLevel = ZSTD_CLEVEL_DEFAULT; /* should not matter, as all cParams are presumed properly defined */
	return 0;
	}

	/* ZSTD_assignParamsToCCtxParams() :
	* params is presumed valid at this stage */
	static ZSTD_CCtx_params ZSTD_assignParamsToCCtxParams(
	const ZSTD_CCtx_params* cctxParams, const ZSTD_parameters* params)
	{
	ZSTD_CCtx_params ret = *cctxParams;
	assert(!ZSTD_checkCParams(params->cParams));
	ret.cParams = params->cParams;
	ret.fParams = params->fParams;
	ret.compressionLevel = ZSTD_CLEVEL_DEFAULT; /* should not matter, as all cParams are presumed properly defined */
	return ret;
	}

	ZSTD_bounds ZSTD_cParam_getBounds(ZSTD_cParameter param)
	{
	ZSTD_bounds bounds = { 0, 0, 0 };

	switch(param)
	{
	case ZSTD_c_compressionLevel:
	bounds.lowerBound = ZSTD_minCLevel();
	bounds.upperBound = ZSTD_maxCLevel();
	return bounds;

	case ZSTD_c_windowLog:
	bounds.lowerBound = ZSTD_WINDOWLOG_MIN;
	bounds.upperBound = ZSTD_WINDOWLOG_MAX;
	return bounds;

	case ZSTD_c_hashLog:
	bounds.lowerBound = ZSTD_HASHLOG_MIN;
	bounds.upperBound = ZSTD_HASHLOG_MAX;
	return bounds;

	case ZSTD_c_chainLog:
	bounds.lowerBound = ZSTD_CHAINLOG_MIN;
	bounds.upperBound = ZSTD_CHAINLOG_MAX;
	return bounds;

	case ZSTD_c_searchLog:
	bounds.lowerBound = ZSTD_SEARCHLOG_MIN;
	bounds.upperBound = ZSTD_SEARCHLOG_MAX;
	return bounds;

	case ZSTD_c_minMatch:
	bounds.lowerBound = ZSTD_MINMATCH_MIN;
	bounds.upperBound = ZSTD_MINMATCH_MAX;
	return bounds;

	case ZSTD_c_targetLength:
	bounds.lowerBound = ZSTD_TARGETLENGTH_MIN;
	bounds.upperBound = ZSTD_TARGETLENGTH_MAX;
	return bounds;

	case ZSTD_c_strategy:
	bounds.lowerBound = ZSTD_STRATEGY_MIN;
	bounds.upperBound = ZSTD_STRATEGY_MAX;
	return bounds;

	case ZSTD_c_contentSizeFlag:
	bounds.lowerBound = 0;
	bounds.upperBound = 1;
	return bounds;

	case ZSTD_c_checksumFlag:
	bounds.lowerBound = 0;
	bounds.upperBound = 1;
	return bounds;

	case ZSTD_c_dictIDFlag:
	bounds.lowerBound = 0;
	bounds.upperBound = 1;
	return bounds;

	case ZSTD_c_nbWorkers:
	bounds.lowerBound = 0;
	#ifdef ZSTD_MULTITHREAD
	bounds.upperBound = ZSTDMT_NBWORKERS_MAX;
	#else
	bounds.upperBound = 0;
	#endif
	return bounds;

	case ZSTD_c_jobSize:
	bounds.lowerBound = 0;
	#ifdef ZSTD_MULTITHREAD
	bounds.upperBound = ZSTDMT_JOBSIZE_MAX;
	#else
	bounds.upperBound = 0;
	#endif
	return bounds;

	case ZSTD_c_overlapLog:
	#ifdef ZSTD_MULTITHREAD
	bounds.lowerBound = ZSTD_OVERLAPLOG_MIN;
	bounds.upperBound = ZSTD_OVERLAPLOG_MAX;
	#else
	bounds.lowerBound = 0;
	bounds.upperBound = 0;
	#endif
	return bounds;

	case ZSTD_c_enableLongDistanceMatching:
	bounds.lowerBound = 0;
	bounds.upperBound = 1;
	return bounds;

	case ZSTD_c_ldmHashLog:
	bounds.lowerBound = ZSTD_LDM_HASHLOG_MIN;
	bounds.upperBound = ZSTD_LDM_HASHLOG_MAX;
	return bounds;

	case ZSTD_c_ldmMinMatch:
	bounds.lowerBound = ZSTD_LDM_MINMATCH_MIN;
	bounds.upperBound = ZSTD_LDM_MINMATCH_MAX;
	return bounds;

	case ZSTD_c_ldmBucketSizeLog:
	bounds.lowerBound = ZSTD_LDM_BUCKETSIZELOG_MIN;
	bounds.upperBound = ZSTD_LDM_BUCKETSIZELOG_MAX;
	return bounds;

	case ZSTD_c_ldmHashRateLog:
	bounds.lowerBound = ZSTD_LDM_HASHRATELOG_MIN;
	bounds.upperBound = ZSTD_LDM_HASHRATELOG_MAX;
	return bounds;

	/* experimental parameters */
	case ZSTD_c_rsyncable:
	bounds.lowerBound = 0;
	bounds.upperBound = 1;
	return bounds;

	case ZSTD_c_forceMaxWindow :
	bounds.lowerBound = 0;
	bounds.upperBound = 1;
	return bounds;

	case ZSTD_c_format:
	ZSTD_STATIC_ASSERT(ZSTD_f_zstd1 < ZSTD_f_zstd1_magicless);
	bounds.lowerBound = ZSTD_f_zstd1;
	bounds.upperBound = ZSTD_f_zstd1_magicless; /* note : how to ensure at compile time that this is the highest value enum ? */
	return bounds;

	case ZSTD_c_forceAttachDict:
	ZSTD_STATIC_ASSERT(ZSTD_dictDefaultAttach < ZSTD_dictForceCopy);
	bounds.lowerBound = ZSTD_dictDefaultAttach;
	bounds.upperBound = ZSTD_dictForceLoad; /* note : how to ensure at compile time that this is the highest value enum ? */
	return bounds;

	case ZSTD_c_literalCompressionMode:
	ZSTD_STATIC_ASSERT(ZSTD_lcm_auto < ZSTD_lcm_huffman && ZSTD_lcm_huffman < ZSTD_lcm_uncompressed);
	bounds.lowerBound = ZSTD_lcm_auto;
	bounds.upperBound = ZSTD_lcm_uncompressed;
	return bounds;

	case ZSTD_c_targetCBlockSize:
	bounds.lowerBound = ZSTD_TARGETCBLOCKSIZE_MIN;
	bounds.upperBound = ZSTD_TARGETCBLOCKSIZE_MAX;
	return bounds;

	case ZSTD_c_srcSizeHint:
	bounds.lowerBound = ZSTD_SRCSIZEHINT_MIN;
	bounds.upperBound = ZSTD_SRCSIZEHINT_MAX;
	return bounds;

	default:
	bounds.error = ERROR(parameter_unsupported);
	return bounds;
	}
	}

	/* ZSTD_cParam_clampBounds:
	* Clamps the value into the bounded range.
	*/
	static size_t ZSTD_cParam_clampBounds(ZSTD_cParameter cParam, int* value)
	{
	ZSTD_bounds const bounds = ZSTD_cParam_getBounds(cParam);
	if (ZSTD_isError(bounds.error)) return bounds.error;
	if (value < bounds.lowerBound) value = bounds.lowerBound;
	if (value > bounds.upperBound) value = bounds.upperBound;
	return 0;
	}

	#define BOUNDCHECK(cParam, val) { \
	RETURN_ERROR_IF(!ZSTD_cParam_withinBounds(cParam,val), \
	parameter_outOfBound, "Param out of bounds"); \
	}


	static int ZSTD_isUpdateAuthorized(ZSTD_cParameter param)
	{
	switch(param)
	{
	case ZSTD_c_compressionLevel:
	case ZSTD_c_hashLog:
	case ZSTD_c_chainLog:
	case ZSTD_c_searchLog:
	case ZSTD_c_minMatch:
	case ZSTD_c_targetLength:
	case ZSTD_c_strategy:
	return 1;

	case ZSTD_c_format:
	case ZSTD_c_windowLog:
	case ZSTD_c_contentSizeFlag:
	case ZSTD_c_checksumFlag:
	case ZSTD_c_dictIDFlag:
	case ZSTD_c_forceMaxWindow :
	case ZSTD_c_nbWorkers:
	case ZSTD_c_jobSize:
	case ZSTD_c_overlapLog:
	case ZSTD_c_rsyncable:
	case ZSTD_c_enableLongDistanceMatching:
	case ZSTD_c_ldmHashLog:
	case ZSTD_c_ldmMinMatch:
	case ZSTD_c_ldmBucketSizeLog:
	case ZSTD_c_ldmHashRateLog:
	case ZSTD_c_forceAttachDict:
	case ZSTD_c_literalCompressionMode:
	case ZSTD_c_targetCBlockSize:
	case ZSTD_c_srcSizeHint:
	default:
	return 0;
	}
	}

	size_t ZSTD_CCtx_setParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int value)
	{
	DEBUGLOG(4, "ZSTD_CCtx_setParameter (%i, %i)", (int)param, value);
	if (cctx->streamStage != zcss_init) {
	if (ZSTD_isUpdateAuthorized(param)) {
	cctx->cParamsChanged = 1;
	} else {
	RETURN_ERROR(stage_wrong, "can only set params in ctx init stage");
	} }

	switch(param)
	{
	case ZSTD_c_nbWorkers:
	RETURN_ERROR_IF((value!=0) && cctx->staticSize, parameter_unsupported,
	"MT not compatible with static alloc");
	break;

	case ZSTD_c_compressionLevel:
	case ZSTD_c_windowLog:
	case ZSTD_c_hashLog:
	case ZSTD_c_chainLog:
	case ZSTD_c_searchLog:
	case ZSTD_c_minMatch:
	case ZSTD_c_targetLength:
	case ZSTD_c_strategy:
	case ZSTD_c_ldmHashRateLog:
	case ZSTD_c_format:
	case ZSTD_c_contentSizeFlag:
	case ZSTD_c_checksumFlag:
	case ZSTD_c_dictIDFlag:
	case ZSTD_c_forceMaxWindow:
	case ZSTD_c_forceAttachDict:
	case ZSTD_c_literalCompressionMode:
	case ZSTD_c_jobSize:
	case ZSTD_c_overlapLog:
	case ZSTD_c_rsyncable:
	case ZSTD_c_enableLongDistanceMatching:
	case ZSTD_c_ldmHashLog:
	case ZSTD_c_ldmMinMatch:
	case ZSTD_c_ldmBucketSizeLog:
	case ZSTD_c_targetCBlockSize:
	case ZSTD_c_srcSizeHint:
	break;

	default: RETURN_ERROR(parameter_unsupported, "unknown parameter");
	}
	return ZSTD_CCtxParams_setParameter(&cctx->requestedParams, param, value);
	}

	size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* CCtxParams,
	ZSTD_cParameter param, int value)
	{
	DEBUGLOG(4, "ZSTD_CCtxParams_setParameter (%i, %i)", (int)param, value);
	switch(param)
	{
	case ZSTD_c_format :
	BOUNDCHECK(ZSTD_c_format, value);
	CCtxParams->format = (ZSTD_format_e)value;
	return (size_t)CCtxParams->format;

	case ZSTD_c_compressionLevel : {
	FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(param, &value), "");
	if (value) { /* 0 : does not change current level */
	CCtxParams->compressionLevel = value;
	}
	if (CCtxParams->compressionLevel >= 0) return (size_t)CCtxParams->compressionLevel;
	return 0; /* return type (size_t) cannot represent negative values */
	}

	case ZSTD_c_windowLog :
	if (value!=0) /* 0 => use default */
	BOUNDCHECK(ZSTD_c_windowLog, value);
	CCtxParams->cParams.windowLog = (U32)value;
	return CCtxParams->cParams.windowLog;

	case ZSTD_c_hashLog :
	if (value!=0) /* 0 => use default */
	BOUNDCHECK(ZSTD_c_hashLog, value);
	CCtxParams->cParams.hashLog = (U32)value;
	return CCtxParams->cParams.hashLog;

	case ZSTD_c_chainLog :
	if (value!=0) /* 0 => use default */
	BOUNDCHECK(ZSTD_c_chainLog, value);
	CCtxParams->cParams.chainLog = (U32)value;
	return CCtxParams->cParams.chainLog;

	case ZSTD_c_searchLog :
	if (value!=0) /* 0 => use default */
	BOUNDCHECK(ZSTD_c_searchLog, value);
	CCtxParams->cParams.searchLog = (U32)value;
	return (size_t)value;

	case ZSTD_c_minMatch :
	if (value!=0) /* 0 => use default */
	BOUNDCHECK(ZSTD_c_minMatch, value);
	CCtxParams->cParams.minMatch = value;
	return CCtxParams->cParams.minMatch;

	case ZSTD_c_targetLength :
	BOUNDCHECK(ZSTD_c_targetLength, value);
	CCtxParams->cParams.targetLength = value;
	return CCtxParams->cParams.targetLength;

	case ZSTD_c_strategy :
	if (value!=0) /* 0 => use default */
	BOUNDCHECK(ZSTD_c_strategy, value);
	CCtxParams->cParams.strategy = (ZSTD_strategy)value;
	return (size_t)CCtxParams->cParams.strategy;

	case ZSTD_c_contentSizeFlag :
	/* Content size written in frame header _when known_ (default:1) */
	DEBUGLOG(4, "set content size flag = %u", (value!=0));
	CCtxParams->fParams.contentSizeFlag = value != 0;
	return CCtxParams->fParams.contentSizeFlag;

	case ZSTD_c_checksumFlag :
	/* A 32-bits content checksum will be calculated and written at end of frame (default:0) */
	CCtxParams->fParams.checksumFlag = value != 0;
	return CCtxParams->fParams.checksumFlag;

	case ZSTD_c_dictIDFlag : /* When applicable, dictionary's dictID is provided in frame header (default:1) */
	DEBUGLOG(4, "set dictIDFlag = %u", (value!=0));
	CCtxParams->fParams.noDictIDFlag = !value;
	return !CCtxParams->fParams.noDictIDFlag;

	case ZSTD_c_forceMaxWindow :
	CCtxParams->forceWindow = (value != 0);
	return CCtxParams->forceWindow;

	case ZSTD_c_forceAttachDict : {
	const ZSTD_dictAttachPref_e pref = (ZSTD_dictAttachPref_e)value;
	BOUNDCHECK(ZSTD_c_forceAttachDict, pref);
	CCtxParams->attachDictPref = pref;
	return CCtxParams->attachDictPref;
	}

	case ZSTD_c_literalCompressionMode : {
	const ZSTD_literalCompressionMode_e lcm = (ZSTD_literalCompressionMode_e)value;
	BOUNDCHECK(ZSTD_c_literalCompressionMode, lcm);
	CCtxParams->literalCompressionMode = lcm;
	return CCtxParams->literalCompressionMode;
	}

	case ZSTD_c_nbWorkers :
	#ifndef ZSTD_MULTITHREAD
	RETURN_ERROR_IF(value!=0, parameter_unsupported, "not compiled with multithreading");
	return 0;
	#else
	FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(param, &value), "");
	CCtxParams->nbWorkers = value;
	return CCtxParams->nbWorkers;
	#endif

	case ZSTD_c_jobSize :
	#ifndef ZSTD_MULTITHREAD
	RETURN_ERROR_IF(value!=0, parameter_unsupported, "not compiled with multithreading");
	return 0;
	#else
	/* Adjust to the minimum non-default value. */
	if (value != 0 && value < ZSTDMT_JOBSIZE_MIN)
	value = ZSTDMT_JOBSIZE_MIN;
	FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(param, &value), "");
	assert(value >= 0);
	CCtxParams->jobSize = value;
	return CCtxParams->jobSize;
	#endif

	case ZSTD_c_overlapLog :
	#ifndef ZSTD_MULTITHREAD
	RETURN_ERROR_IF(value!=0, parameter_unsupported, "not compiled with multithreading");
	return 0;
	#else
	FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(ZSTD_c_overlapLog, &value), "");
	CCtxParams->overlapLog = value;
	return CCtxParams->overlapLog;
	#endif

	case ZSTD_c_rsyncable :
	#ifndef ZSTD_MULTITHREAD
	RETURN_ERROR_IF(value!=0, parameter_unsupported, "not compiled with multithreading");
	return 0;
	#else
	FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(ZSTD_c_overlapLog, &value), "");
	CCtxParams->rsyncable = value;
	return CCtxParams->rsyncable;
	#endif

	case ZSTD_c_enableLongDistanceMatching :
	CCtxParams->ldmParams.enableLdm = (value!=0);
	return CCtxParams->ldmParams.enableLdm;

	case ZSTD_c_ldmHashLog :
	if (value!=0) /* 0 ==> auto */
	BOUNDCHECK(ZSTD_c_ldmHashLog, value);
	CCtxParams->ldmParams.hashLog = value;
	return CCtxParams->ldmParams.hashLog;

	case ZSTD_c_ldmMinMatch :
	if (value!=0) /* 0 ==> default */
	BOUNDCHECK(ZSTD_c_ldmMinMatch, value);
	CCtxParams->ldmParams.minMatchLength = value;
	return CCtxParams->ldmParams.minMatchLength;

	case ZSTD_c_ldmBucketSizeLog :
	if (value!=0) /* 0 ==> default */
	BOUNDCHECK(ZSTD_c_ldmBucketSizeLog, value);
	CCtxParams->ldmParams.bucketSizeLog = value;
	return CCtxParams->ldmParams.bucketSizeLog;

	case ZSTD_c_ldmHashRateLog :
	RETURN_ERROR_IF(value > ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN,
	parameter_outOfBound, "Param out of bounds!");
	CCtxParams->ldmParams.hashRateLog = value;
	return CCtxParams->ldmParams.hashRateLog;

	case ZSTD_c_targetCBlockSize :
	if (value!=0) /* 0 ==> default */
	BOUNDCHECK(ZSTD_c_targetCBlockSize, value);
	CCtxParams->targetCBlockSize = value;
	return CCtxParams->targetCBlockSize;

	case ZSTD_c_srcSizeHint :
	if (value!=0) /* 0 ==> default */
	BOUNDCHECK(ZSTD_c_srcSizeHint, value);
	CCtxParams->srcSizeHint = value;
	return CCtxParams->srcSizeHint;

	default: RETURN_ERROR(parameter_unsupported, "unknown parameter");
	}
	}

	size_t ZSTD_CCtx_getParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int* value)
	{
	return ZSTD_CCtxParams_getParameter(&cctx->requestedParams, param, value);
	}

	size_t ZSTD_CCtxParams_getParameter(
	ZSTD_CCtx_params* CCtxParams, ZSTD_cParameter param, int* value)
	{
	switch(param)
	{
	case ZSTD_c_format :
	*value = CCtxParams->format;
	break;
	case ZSTD_c_compressionLevel :
	*value = CCtxParams->compressionLevel;
	break;
	case ZSTD_c_windowLog :
	*value = (int)CCtxParams->cParams.windowLog;
	break;
	case ZSTD_c_hashLog :
	*value = (int)CCtxParams->cParams.hashLog;
	break;
	case ZSTD_c_chainLog :
	*value = (int)CCtxParams->cParams.chainLog;
	break;
	case ZSTD_c_searchLog :
	*value = CCtxParams->cParams.searchLog;
	break;
	case ZSTD_c_minMatch :
	*value = CCtxParams->cParams.minMatch;
	break;
	case ZSTD_c_targetLength :
	*value = CCtxParams->cParams.targetLength;
	break;
	case ZSTD_c_strategy :
	*value = (unsigned)CCtxParams->cParams.strategy;
	break;
	case ZSTD_c_contentSizeFlag :
	*value = CCtxParams->fParams.contentSizeFlag;
	break;
	case ZSTD_c_checksumFlag :
	*value = CCtxParams->fParams.checksumFlag;
	break;
	case ZSTD_c_dictIDFlag :
	*value = !CCtxParams->fParams.noDictIDFlag;
	break;
	case ZSTD_c_forceMaxWindow :
	*value = CCtxParams->forceWindow;
	break;
	case ZSTD_c_forceAttachDict :
	*value = CCtxParams->attachDictPref;
	break;
	case ZSTD_c_literalCompressionMode :
	*value = CCtxParams->literalCompressionMode;
	break;
	case ZSTD_c_nbWorkers :
	#ifndef ZSTD_MULTITHREAD
	assert(CCtxParams->nbWorkers == 0);
	#endif
	*value = CCtxParams->nbWorkers;
	break;
	case ZSTD_c_jobSize :
	#ifndef ZSTD_MULTITHREAD
	RETURN_ERROR(parameter_unsupported, "not compiled with multithreading");
	#else
	assert(CCtxParams->jobSize <= INT_MAX);
	*value = (int)CCtxParams->jobSize;
	break;
	#endif
	case ZSTD_c_overlapLog :
	#ifndef ZSTD_MULTITHREAD
	RETURN_ERROR(parameter_unsupported, "not compiled with multithreading");
	#else
	*value = CCtxParams->overlapLog;
	break;
	#endif
	case ZSTD_c_rsyncable :
	#ifndef ZSTD_MULTITHREAD
	RETURN_ERROR(parameter_unsupported, "not compiled with multithreading");
	#else
	*value = CCtxParams->rsyncable;
	break;
	#endif
	case ZSTD_c_enableLongDistanceMatching :
	*value = CCtxParams->ldmParams.enableLdm;
	break;
	case ZSTD_c_ldmHashLog :
	*value = CCtxParams->ldmParams.hashLog;
	break;
	case ZSTD_c_ldmMinMatch :
	*value = CCtxParams->ldmParams.minMatchLength;
	break;
	case ZSTD_c_ldmBucketSizeLog :
	*value = CCtxParams->ldmParams.bucketSizeLog;
	break;
	case ZSTD_c_ldmHashRateLog :
	*value = CCtxParams->ldmParams.hashRateLog;
	break;
	case ZSTD_c_targetCBlockSize :
	*value = (int)CCtxParams->targetCBlockSize;
	break;
	case ZSTD_c_srcSizeHint :
	*value = (int)CCtxParams->srcSizeHint;
	break;
	default: RETURN_ERROR(parameter_unsupported, "unknown parameter");
	}
	return 0;
	}

	/** ZSTD_CCtx_setParametersUsingCCtxParams() :
	* just applies `params` into `cctx`
	* no action is performed, parameters are merely stored.
	* If ZSTDMT is enabled, parameters are pushed to cctx->mtctx.
	* This is possible even if a compression is ongoing.
	* In which case, new parameters will be applied on the fly, starting with next compression job.
	*/
	size_t ZSTD_CCtx_setParametersUsingCCtxParams(
	ZSTD_CCtx* cctx, const ZSTD_CCtx_params* params)
	{
	DEBUGLOG(4, "ZSTD_CCtx_setParametersUsingCCtxParams");
	RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
	"The context is in the wrong stage!");
	RETURN_ERROR_IF(cctx->cdict, stage_wrong,
	"Can't override parameters with cdict attached (some must "
	"be inherited from the cdict).");

	cctx->requestedParams = *params;
	return 0;
	}

	ZSTDLIB_API size_t ZSTD_CCtx_setPledgedSrcSize(ZSTD_CCtx* cctx, unsigned long long pledgedSrcSize)
	{
	DEBUGLOG(4, "ZSTD_CCtx_setPledgedSrcSize to %u bytes", (U32)pledgedSrcSize);
	RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
	"Can't set pledgedSrcSize when not in init stage.");
	cctx->pledgedSrcSizePlusOne = pledgedSrcSize+1;
	return 0;
	}

	/**
	* Initializes the local dict using the requested parameters.
	* NOTE: This does not use the pledged src size, because it may be used for more
	* than one compression.
	*/
	static size_t ZSTD_initLocalDict(ZSTD_CCtx* cctx)
	{
	ZSTD_localDict* const dl = &cctx->localDict;
	ZSTD_compressionParameters const cParams = ZSTD_getCParamsFromCCtxParams(
	&cctx->requestedParams, ZSTD_CONTENTSIZE_UNKNOWN, dl->dictSize);
	if (dl->dict == NULL) {
	/* No local dictionary. */
	assert(dl->dictBuffer == NULL);
	assert(dl->cdict == NULL);
	assert(dl->dictSize == 0);
	return 0;
	}
	if (dl->cdict != NULL) {
	assert(cctx->cdict == dl->cdict);
	/* Local dictionary already initialized. */
	return 0;
	}
	assert(dl->dictSize > 0);
	assert(cctx->cdict == NULL);
	assert(cctx->prefixDict.dict == NULL);

	dl->cdict = ZSTD_createCDict_advanced(
	dl->dict,
	dl->dictSize,
	ZSTD_dlm_byRef,
	dl->dictContentType,
	cParams,
	cctx->customMem);
	RETURN_ERROR_IF(!dl->cdict, memory_allocation, "ZSTD_createCDict_advanced failed");
	cctx->cdict = dl->cdict;
	return 0;
	}

	size_t ZSTD_CCtx_loadDictionary_advanced(
	ZSTD_CCtx* cctx, const void* dict, size_t dictSize,
	ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType)
	{
	RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
	"Can't load a dictionary when ctx is not in init stage.");
	RETURN_ERROR_IF(cctx->staticSize, memory_allocation,
	"no malloc for static CCtx");
	DEBUGLOG(4, "ZSTD_CCtx_loadDictionary_advanced (size: %u)", (U32)dictSize);
	ZSTD_clearAllDicts(cctx); /* in case one already exists */
	if (dict == NULL \|\| dictSize == 0) /* no dictionary mode */
	return 0;
	if (dictLoadMethod == ZSTD_dlm_byRef) {
	cctx->localDict.dict = dict;
	} else {
	void* dictBuffer = ZSTD_malloc(dictSize, cctx->customMem);
	RETURN_ERROR_IF(!dictBuffer, memory_allocation, "NULL pointer!");
	memcpy(dictBuffer, dict, dictSize);
	cctx->localDict.dictBuffer = dictBuffer;
	cctx->localDict.dict = dictBuffer;
	}
	cctx->localDict.dictSize = dictSize;
	cctx->localDict.dictContentType = dictContentType;
	return 0;
	}

	ZSTDLIB_API size_t ZSTD_CCtx_loadDictionary_byReference(
	ZSTD_CCtx* cctx, const void* dict, size_t dictSize)
	{
	return ZSTD_CCtx_loadDictionary_advanced(
	cctx, dict, dictSize, ZSTD_dlm_byRef, ZSTD_dct_auto);
	}

	ZSTDLIB_API size_t ZSTD_CCtx_loadDictionary(ZSTD_CCtx* cctx, const void* dict, size_t dictSize)
	{
	return ZSTD_CCtx_loadDictionary_advanced(
	cctx, dict, dictSize, ZSTD_dlm_byCopy, ZSTD_dct_auto);
	}


	size_t ZSTD_CCtx_refCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict)
	{
	RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
	"Can't ref a dict when ctx not in init stage.");
	/* Free the existing local cdict (if any) to save memory. */
	ZSTD_clearAllDicts(cctx);
	cctx->cdict = cdict;
	return 0;
	}

	size_t ZSTD_CCtx_refPrefix(ZSTD_CCtx* cctx, const void* prefix, size_t prefixSize)
	{
	return ZSTD_CCtx_refPrefix_advanced(cctx, prefix, prefixSize, ZSTD_dct_rawContent);
	}

	size_t ZSTD_CCtx_refPrefix_advanced(
	ZSTD_CCtx* cctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType)
	{
	RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
	"Can't ref a prefix when ctx not in init stage.");
	ZSTD_clearAllDicts(cctx);
	if (prefix != NULL && prefixSize > 0) {
	cctx->prefixDict.dict = prefix;
	cctx->prefixDict.dictSize = prefixSize;
	cctx->prefixDict.dictContentType = dictContentType;
	}
	return 0;
	}

	/*! ZSTD_CCtx_reset() :
	* Also dumps dictionary */
	size_t ZSTD_CCtx_reset(ZSTD_CCtx* cctx, ZSTD_ResetDirective reset)
	{
	if ( (reset == ZSTD_reset_session_only)
	\|\| (reset == ZSTD_reset_session_and_parameters) ) {
	cctx->streamStage = zcss_init;
	cctx->pledgedSrcSizePlusOne = 0;
	}
	if ( (reset == ZSTD_reset_parameters)
	\|\| (reset == ZSTD_reset_session_and_parameters) ) {
	RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
	"Can't reset parameters only when not in init stage.");
	ZSTD_clearAllDicts(cctx);
	return ZSTD_CCtxParams_reset(&cctx->requestedParams);
	}
	return 0;
	}


	/** ZSTD_checkCParams() :
	control CParam values remain within authorized range.
	@return : 0, or an error code if one value is beyond authorized range */
	size_t ZSTD_checkCParams(ZSTD_compressionParameters cParams)
	{
	BOUNDCHECK(ZSTD_c_windowLog, (int)cParams.windowLog);
	BOUNDCHECK(ZSTD_c_chainLog, (int)cParams.chainLog);
	BOUNDCHECK(ZSTD_c_hashLog, (int)cParams.hashLog);
	BOUNDCHECK(ZSTD_c_searchLog, (int)cParams.searchLog);
	BOUNDCHECK(ZSTD_c_minMatch, (int)cParams.minMatch);
	BOUNDCHECK(ZSTD_c_targetLength,(int)cParams.targetLength);
	BOUNDCHECK(ZSTD_c_strategy, cParams.strategy);
	return 0;
	}

	/** ZSTD_clampCParams() :
	* make CParam values within valid range.
	* @return : valid CParams */
	static ZSTD_compressionParameters
	ZSTD_clampCParams(ZSTD_compressionParameters cParams)
	{
	# define CLAMP_TYPE(cParam, val, type) { \
	ZSTD_bounds const bounds = ZSTD_cParam_getBounds(cParam); \
	if ((int)val<bounds.lowerBound) val=(type)bounds.lowerBound; \
	else if ((int)val>bounds.upperBound) val=(type)bounds.upperBound; \
	}
	# define CLAMP(cParam, val) CLAMP_TYPE(cParam, val, unsigned)
	CLAMP(ZSTD_c_windowLog, cParams.windowLog);
	CLAMP(ZSTD_c_chainLog, cParams.chainLog);
	CLAMP(ZSTD_c_hashLog, cParams.hashLog);
	CLAMP(ZSTD_c_searchLog, cParams.searchLog);
	CLAMP(ZSTD_c_minMatch, cParams.minMatch);
	CLAMP(ZSTD_c_targetLength,cParams.targetLength);
	CLAMP_TYPE(ZSTD_c_strategy,cParams.strategy, ZSTD_strategy);
	return cParams;
	}

	/** ZSTD_cycleLog() :
	* condition for correct operation : hashLog > 1 */
	U32 ZSTD_cycleLog(U32 hashLog, ZSTD_strategy strat)
	{
	U32 const btScale = ((U32)strat >= (U32)ZSTD_btlazy2);
	return hashLog - btScale;
	}

	/** ZSTD_adjustCParams_internal() :
	* optimize `cPar` for a specified input (`srcSize` and `dictSize`).
	* mostly downsize to reduce memory consumption and initialization latency.
	* `srcSize` can be ZSTD_CONTENTSIZE_UNKNOWN when not known.
	* note : `srcSize==0` means 0!
	* condition : cPar is presumed validated (can be checked using ZSTD_checkCParams()). */
	static ZSTD_compressionParameters
	ZSTD_adjustCParams_internal(ZSTD_compressionParameters cPar,
	unsigned long long srcSize,
	size_t dictSize)
	{
	static const U64 minSrcSize = 513; /* (1<<9) + 1 */
	static const U64 maxWindowResize = 1ULL << (ZSTD_WINDOWLOG_MAX-1);
	assert(ZSTD_checkCParams(cPar)==0);

	if (dictSize && srcSize == ZSTD_CONTENTSIZE_UNKNOWN)
	srcSize = minSrcSize;

	/* resize windowLog if input is small enough, to use less memory */
	if ( (srcSize < maxWindowResize)
	&& (dictSize < maxWindowResize) ) {
	U32 const tSize = (U32)(srcSize + dictSize);
	static U32 const hashSizeMin = 1 << ZSTD_HASHLOG_MIN;
	U32 const srcLog = (tSize < hashSizeMin) ? ZSTD_HASHLOG_MIN :
	ZSTD_highbit32(tSize-1) + 1;
	if (cPar.windowLog > srcLog) cPar.windowLog = srcLog;
	}
	if (cPar.hashLog > cPar.windowLog+1) cPar.hashLog = cPar.windowLog+1;
	{ U32 const cycleLog = ZSTD_cycleLog(cPar.chainLog, cPar.strategy);
	if (cycleLog > cPar.windowLog)
	cPar.chainLog -= (cycleLog - cPar.windowLog);
	}

	if (cPar.windowLog < ZSTD_WINDOWLOG_ABSOLUTEMIN)
	cPar.windowLog = ZSTD_WINDOWLOG_ABSOLUTEMIN; /* minimum wlog required for valid frame header */

	return cPar;
	}

	ZSTD_compressionParameters
	ZSTD_adjustCParams(ZSTD_compressionParameters cPar,
	unsigned long long srcSize,
	size_t dictSize)
	{
	cPar = ZSTD_clampCParams(cPar); /* resulting cPar is necessarily valid (all parameters within range) */
	if (srcSize == 0) srcSize = ZSTD_CONTENTSIZE_UNKNOWN;
	return ZSTD_adjustCParams_internal(cPar, srcSize, dictSize);
	}

	static ZSTD_compressionParameters ZSTD_getCParams_internal(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize);
	static ZSTD_parameters ZSTD_getParams_internal(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize);

	ZSTD_compressionParameters ZSTD_getCParamsFromCCtxParams(
	const ZSTD_CCtx_params* CCtxParams, U64 srcSizeHint, size_t dictSize)
	{
	ZSTD_compressionParameters cParams;
	if (srcSizeHint == ZSTD_CONTENTSIZE_UNKNOWN && CCtxParams->srcSizeHint > 0) {
	srcSizeHint = CCtxParams->srcSizeHint;
	}
	cParams = ZSTD_getCParams_internal(CCtxParams->compressionLevel, srcSizeHint, dictSize);
	if (CCtxParams->ldmParams.enableLdm) cParams.windowLog = ZSTD_LDM_DEFAULT_WINDOW_LOG;
	if (CCtxParams->cParams.windowLog) cParams.windowLog = CCtxParams->cParams.windowLog;
	if (CCtxParams->cParams.hashLog) cParams.hashLog = CCtxParams->cParams.hashLog;
	if (CCtxParams->cParams.chainLog) cParams.chainLog = CCtxParams->cParams.chainLog;
	if (CCtxParams->cParams.searchLog) cParams.searchLog = CCtxParams->cParams.searchLog;
	if (CCtxParams->cParams.minMatch) cParams.minMatch = CCtxParams->cParams.minMatch;
	if (CCtxParams->cParams.targetLength) cParams.targetLength = CCtxParams->cParams.targetLength;
	if (CCtxParams->cParams.strategy) cParams.strategy = CCtxParams->cParams.strategy;
	assert(!ZSTD_checkCParams(cParams));
	/* srcSizeHint == 0 means 0 */
	return ZSTD_adjustCParams_internal(cParams, srcSizeHint, dictSize);
	}

	static size_t
	ZSTD_sizeof_matchState(const ZSTD_compressionParameters* const cParams,
	const U32 forCCtx)
	{
	size_t const chainSize = (cParams->strategy == ZSTD_fast) ? 0 : ((size_t)1 << cParams->chainLog);
	size_t const hSize = ((size_t)1) << cParams->hashLog;
	U32 const hashLog3 = (forCCtx && cParams->minMatch==3) ? MIN(ZSTD_HASHLOG3_MAX, cParams->windowLog) : 0;
	size_t const h3Size = hashLog3 ? ((size_t)1) << hashLog3 : 0;
	/* We don't use ZSTD_cwksp_alloc_size() here because the tables aren't
	* surrounded by redzones in ASAN. */
	size_t const tableSpace = chainSize * sizeof(U32)
	+ hSize * sizeof(U32)
	+ h3Size * sizeof(U32);
	size_t const optPotentialSpace =
	ZSTD_cwksp_alloc_size((MaxML+1) * sizeof(U32))
	+ ZSTD_cwksp_alloc_size((MaxLL+1) * sizeof(U32))
	+ ZSTD_cwksp_alloc_size((MaxOff+1) * sizeof(U32))
	+ ZSTD_cwksp_alloc_size((1<<Litbits) * sizeof(U32))
	+ ZSTD_cwksp_alloc_size((ZSTD_OPT_NUM+1) * sizeof(ZSTD_match_t))
	+ ZSTD_cwksp_alloc_size((ZSTD_OPT_NUM+1) * sizeof(ZSTD_optimal_t));
	size_t const optSpace = (forCCtx && (cParams->strategy >= ZSTD_btopt))
	? optPotentialSpace
	: 0;
	DEBUGLOG(4, "chainSize: %u - hSize: %u - h3Size: %u",
	(U32)chainSize, (U32)hSize, (U32)h3Size);
	return tableSpace + optSpace;
	}

	size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params)
	{
	RETURN_ERROR_IF(params->nbWorkers > 0, GENERIC, "Estimate CCtx size is supported for single-threaded compression only.");
	{ ZSTD_compressionParameters const cParams =
	ZSTD_getCParamsFromCCtxParams(params, ZSTD_CONTENTSIZE_UNKNOWN, 0);
	size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, (size_t)1 << cParams.windowLog);
	U32 const divider = (cParams.minMatch==3) ? 3 : 4;
	size_t const maxNbSeq = blockSize / divider;
	size_t const tokenSpace = ZSTD_cwksp_alloc_size(WILDCOPY_OVERLENGTH + blockSize)
	+ ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(seqDef))
	+ 3 * ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(BYTE));
	size_t const entropySpace = ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE);
	size_t const blockStateSpace = 2 * ZSTD_cwksp_alloc_size(sizeof(ZSTD_compressedBlockState_t));
	size_t const matchStateSize = ZSTD_sizeof_matchState(&cParams, /* forCCtx */ 1);

	size_t const ldmSpace = ZSTD_ldm_getTableSize(params->ldmParams);
	size_t const ldmSeqSpace = ZSTD_cwksp_alloc_size(ZSTD_ldm_getMaxNbSeq(params->ldmParams, blockSize) * sizeof(rawSeq));

	/* estimateCCtxSize is for one-shot compression. So no buffers should
	* be needed. However, we still allocate two 0-sized buffers, which can
	* take space under ASAN. */
	size_t const bufferSpace = ZSTD_cwksp_alloc_size(0)
	+ ZSTD_cwksp_alloc_size(0);

	size_t const cctxSpace = ZSTD_cwksp_alloc_size(sizeof(ZSTD_CCtx));

	size_t const neededSpace =
	cctxSpace +
	entropySpace +
	blockStateSpace +
	ldmSpace +
	ldmSeqSpace +
	matchStateSize +
	tokenSpace +
	bufferSpace;

	DEBUGLOG(5, "estimate workspace : %u", (U32)neededSpace);
	return neededSpace;
	}
	}

	size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams)
	{
	ZSTD_CCtx_params const params = ZSTD_makeCCtxParamsFromCParams(cParams);
	return ZSTD_estimateCCtxSize_usingCCtxParams(&params);
	}

	static size_t ZSTD_estimateCCtxSize_internal(int compressionLevel)
	{
	ZSTD_compressionParameters const cParams = ZSTD_getCParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, 0);
	return ZSTD_estimateCCtxSize_usingCParams(cParams);
	}

	size_t ZSTD_estimateCCtxSize(int compressionLevel)
	{
	int level;
	size_t memBudget = 0;
	for (level=MIN(compressionLevel, 1); level<=compressionLevel; level++) {
	size_t const newMB = ZSTD_estimateCCtxSize_internal(level);
	if (newMB > memBudget) memBudget = newMB;
	}
	return memBudget;
	}

	size_t ZSTD_estimateCStreamSize_usingCCtxParams(const ZSTD_CCtx_params* params)
	{
	RETURN_ERROR_IF(params->nbWorkers > 0, GENERIC, "Estimate CCtx size is supported for single-threaded compression only.");
	{ ZSTD_compressionParameters const cParams =
	ZSTD_getCParamsFromCCtxParams(params, ZSTD_CONTENTSIZE_UNKNOWN, 0);
	size_t const CCtxSize = ZSTD_estimateCCtxSize_usingCCtxParams(params);
	size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, (size_t)1 << cParams.windowLog);
	size_t const inBuffSize = ((size_t)1 << cParams.windowLog) + blockSize;
	size_t const outBuffSize = ZSTD_compressBound(blockSize) + 1;
	size_t const streamingSize = ZSTD_cwksp_alloc_size(inBuffSize)
	+ ZSTD_cwksp_alloc_size(outBuffSize);

	return CCtxSize + streamingSize;
	}
	}

	size_t ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams)
	{
	ZSTD_CCtx_params const params = ZSTD_makeCCtxParamsFromCParams(cParams);
	return ZSTD_estimateCStreamSize_usingCCtxParams(&params);
	}

	static size_t ZSTD_estimateCStreamSize_internal(int compressionLevel)
	{
	ZSTD_compressionParameters const cParams = ZSTD_getCParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, 0);
	return ZSTD_estimateCStreamSize_usingCParams(cParams);
	}

	size_t ZSTD_estimateCStreamSize(int compressionLevel)
	{
	int level;
	size_t memBudget = 0;
	for (level=MIN(compressionLevel, 1); level<=compressionLevel; level++) {
	size_t const newMB = ZSTD_estimateCStreamSize_internal(level);
	if (newMB > memBudget) memBudget = newMB;
	}
	return memBudget;
	}

	/* ZSTD_getFrameProgression():
	* tells how much data has been consumed (input) and produced (output) for current frame.
	* able to count progression inside worker threads (non-blocking mode).
	*/
	ZSTD_frameProgression ZSTD_getFrameProgression(const ZSTD_CCtx* cctx)
	{
	#ifdef ZSTD_MULTITHREAD
	if (cctx->appliedParams.nbWorkers > 0) {
	return ZSTDMT_getFrameProgression(cctx->mtctx);
	}
	#endif
	{ ZSTD_frameProgression fp;
	size_t const buffered = (cctx->inBuff == NULL) ? 0 :
	cctx->inBuffPos - cctx->inToCompress;
	if (buffered) assert(cctx->inBuffPos >= cctx->inToCompress);
	assert(buffered <= ZSTD_BLOCKSIZE_MAX);
	fp.ingested = cctx->consumedSrcSize + buffered;
	fp.consumed = cctx->consumedSrcSize;
	fp.produced = cctx->producedCSize;
	fp.flushed = cctx->producedCSize; /* simplified; some data might still be left within streaming output buffer */
	fp.currentJobID = 0;
	fp.nbActiveWorkers = 0;
	return fp;
	} }

	/*! ZSTD_toFlushNow()
	* Only useful for multithreading scenarios currently (nbWorkers >= 1).
	*/
	size_t ZSTD_toFlushNow(ZSTD_CCtx* cctx)
	{
	#ifdef ZSTD_MULTITHREAD
	if (cctx->appliedParams.nbWorkers > 0) {
	return ZSTDMT_toFlushNow(cctx->mtctx);
	}
	#endif
	(void)cctx;
	return 0; /* over-simplification; could also check if context is currently running in streaming mode, and in which case, report how many bytes are left to be flushed within output buffer */
	}

	static void ZSTD_assertEqualCParams(ZSTD_compressionParameters cParams1,
	ZSTD_compressionParameters cParams2)
	{
	(void)cParams1;
	(void)cParams2;
	assert(cParams1.windowLog == cParams2.windowLog);
	assert(cParams1.chainLog == cParams2.chainLog);
	assert(cParams1.hashLog == cParams2.hashLog);
	assert(cParams1.searchLog == cParams2.searchLog);
	assert(cParams1.minMatch == cParams2.minMatch);
	assert(cParams1.targetLength == cParams2.targetLength);
	assert(cParams1.strategy == cParams2.strategy);
	}

	void ZSTD_reset_compressedBlockState(ZSTD_compressedBlockState_t* bs)
	{
	int i;
	for (i = 0; i < ZSTD_REP_NUM; ++i)
	bs->rep[i] = repStartValue[i];
	bs->entropy.huf.repeatMode = HUF_repeat_none;
	bs->entropy.fse.offcode_repeatMode = FSE_repeat_none;
	bs->entropy.fse.matchlength_repeatMode = FSE_repeat_none;
	bs->entropy.fse.litlength_repeatMode = FSE_repeat_none;
	}

	/*! ZSTD_invalidateMatchState()
	* Invalidate all the matches in the match finder tables.
	* Requires nextSrc and base to be set (can be NULL).
	*/
	static void ZSTD_invalidateMatchState(ZSTD_matchState_t* ms)
	{
	ZSTD_window_clear(&ms->window);

	ms->nextToUpdate = ms->window.dictLimit;
	ms->loadedDictEnd = 0;
	ms->opt.litLengthSum = 0; /* force reset of btopt stats */
	ms->dictMatchState = NULL;
	}

	/**
	* Indicates whether this compression proceeds directly from user-provided
	* source buffer to user-provided destination buffer (ZSTDb_not_buffered), or
	* whether the context needs to buffer the input/output (ZSTDb_buffered).
	*/
	typedef enum {
	ZSTDb_not_buffered,
	ZSTDb_buffered
	} ZSTD_buffered_policy_e;

	/**
	* Controls, for this matchState reset, whether the tables need to be cleared /
	* prepared for the coming compression (ZSTDcrp_makeClean), or whether the
	* tables can be left unclean (ZSTDcrp_leaveDirty), because we know that a
	* subsequent operation will overwrite the table space anyways (e.g., copying
	* the matchState contents in from a CDict).
	*/
	typedef enum {
	ZSTDcrp_makeClean,
	ZSTDcrp_leaveDirty
	} ZSTD_compResetPolicy_e;

	/**
	* Controls, for this matchState reset, whether indexing can continue where it
	* left off (ZSTDirp_continue), or whether it needs to be restarted from zero
	* (ZSTDirp_reset).
	*/
	typedef enum {
	ZSTDirp_continue,
	ZSTDirp_reset
	} ZSTD_indexResetPolicy_e;

	typedef enum {
	ZSTD_resetTarget_CDict,
	ZSTD_resetTarget_CCtx
	} ZSTD_resetTarget_e;

	static size_t
	ZSTD_reset_matchState(ZSTD_matchState_t* ms,
	ZSTD_cwksp* ws,
	const ZSTD_compressionParameters* cParams,
	const ZSTD_compResetPolicy_e crp,
	const ZSTD_indexResetPolicy_e forceResetIndex,
	const ZSTD_resetTarget_e forWho)
	{
	size_t const chainSize = (cParams->strategy == ZSTD_fast) ? 0 : ((size_t)1 << cParams->chainLog);
	size_t const hSize = ((size_t)1) << cParams->hashLog;
	U32 const hashLog3 = ((forWho == ZSTD_resetTarget_CCtx) && cParams->minMatch==3) ? MIN(ZSTD_HASHLOG3_MAX, cParams->windowLog) : 0;
	size_t const h3Size = hashLog3 ? ((size_t)1) << hashLog3 : 0;

	DEBUGLOG(4, "reset indices : %u", forceResetIndex == ZSTDirp_reset);
	if (forceResetIndex == ZSTDirp_reset) {
	ZSTD_window_init(&ms->window);
	ZSTD_cwksp_mark_tables_dirty(ws);
	}

	ms->hashLog3 = hashLog3;

	ZSTD_invalidateMatchState(ms);

	assert(!ZSTD_cwksp_reserve_failed(ws)); /* check that allocation hasn't already failed */

	ZSTD_cwksp_clear_tables(ws);

	DEBUGLOG(5, "reserving table space");
	/* table Space */
	ms->hashTable = (U32)ZSTD_cwksp_reserve_table(ws, hSize sizeof(U32));
	ms->chainTable = (U32)ZSTD_cwksp_reserve_table(ws, chainSize sizeof(U32));
	ms->hashTable3 = (U32)ZSTD_cwksp_reserve_table(ws, h3Size sizeof(U32));
	RETURN_ERROR_IF(ZSTD_cwksp_reserve_failed(ws), memory_allocation,
	"failed a workspace allocation in ZSTD_reset_matchState");

	DEBUGLOG(4, "reset table : %u", crp!=ZSTDcrp_leaveDirty);
	if (crp!=ZSTDcrp_leaveDirty) {
	/* reset tables only */
	ZSTD_cwksp_clean_tables(ws);
	}

	/* opt parser space */
	if ((forWho == ZSTD_resetTarget_CCtx) && (cParams->strategy >= ZSTD_btopt)) {
	DEBUGLOG(4, "reserving optimal parser space");
	ms->opt.litFreq = (unsigned)ZSTD_cwksp_reserve_aligned(ws, (1<<Litbits) sizeof(unsigned));
	ms->opt.litLengthFreq = (unsigned)ZSTD_cwksp_reserve_aligned(ws, (MaxLL+1) sizeof(unsigned));
	ms->opt.matchLengthFreq = (unsigned)ZSTD_cwksp_reserve_aligned(ws, (MaxML+1) sizeof(unsigned));
	ms->opt.offCodeFreq = (unsigned)ZSTD_cwksp_reserve_aligned(ws, (MaxOff+1) sizeof(unsigned));
	ms->opt.matchTable = (ZSTD_match_t)ZSTD_cwksp_reserve_aligned(ws, (ZSTD_OPT_NUM+1) sizeof(ZSTD_match_t));
	ms->opt.priceTable = (ZSTD_optimal_t)ZSTD_cwksp_reserve_aligned(ws, (ZSTD_OPT_NUM+1) sizeof(ZSTD_optimal_t));
	}

	ms->cParams = *cParams;

	RETURN_ERROR_IF(ZSTD_cwksp_reserve_failed(ws), memory_allocation,
	"failed a workspace allocation in ZSTD_reset_matchState");

	return 0;
	}

	/* ZSTD_indexTooCloseToMax() :
	* minor optimization : prefer memset() rather than reduceIndex()
	* which is measurably slow in some circumstances (reported for Visual Studio).
	* Works when re-using a context for a lot of smallish inputs :
	* if all inputs are smaller than ZSTD_INDEXOVERFLOW_MARGIN,
	* memset() will be triggered before reduceIndex().
	*/
	#define ZSTD_INDEXOVERFLOW_MARGIN (16 MB)
	static int ZSTD_indexTooCloseToMax(ZSTD_window_t w)
	{
	return (size_t)(w.nextSrc - w.base) > (ZSTD_CURRENT_MAX - ZSTD_INDEXOVERFLOW_MARGIN);
	}

	/*! ZSTD_resetCCtx_internal() :
	note : `params` are assumed fully validated at this stage */
	static size_t ZSTD_resetCCtx_internal(ZSTD_CCtx* zc,
	ZSTD_CCtx_params params,
	U64 const pledgedSrcSize,
	ZSTD_compResetPolicy_e const crp,
	ZSTD_buffered_policy_e const zbuff)
	{
	ZSTD_cwksp* const ws = &zc->workspace;
	DEBUGLOG(4, "ZSTD_resetCCtx_internal: pledgedSrcSize=%u, wlog=%u",
	(U32)pledgedSrcSize, params.cParams.windowLog);
	assert(!ZSTD_isError(ZSTD_checkCParams(params.cParams)));

	zc->isFirstBlock = 1;

	if (params.ldmParams.enableLdm) {
	/* Adjust long distance matching parameters */
	ZSTD_ldm_adjustParameters(&params.ldmParams, &params.cParams);
	assert(params.ldmParams.hashLog >= params.ldmParams.bucketSizeLog);
	assert(params.ldmParams.hashRateLog < 32);
	zc->ldmState.hashPower = ZSTD_rollingHash_primePower(params.ldmParams.minMatchLength);
	}

	{ size_t const windowSize = MAX(1, (size_t)MIN(((U64)1 << params.cParams.windowLog), pledgedSrcSize));
	size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, windowSize);
	U32 const divider = (params.cParams.minMatch==3) ? 3 : 4;
	size_t const maxNbSeq = blockSize / divider;
	size_t const tokenSpace = ZSTD_cwksp_alloc_size(WILDCOPY_OVERLENGTH + blockSize)
	+ ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(seqDef))
	+ 3 * ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(BYTE));
	size_t const buffOutSize = (zbuff==ZSTDb_buffered) ? ZSTD_compressBound(blockSize)+1 : 0;
	size_t const buffInSize = (zbuff==ZSTDb_buffered) ? windowSize + blockSize : 0;
	size_t const matchStateSize = ZSTD_sizeof_matchState(&params.cParams, /* forCCtx */ 1);
	size_t const maxNbLdmSeq = ZSTD_ldm_getMaxNbSeq(params.ldmParams, blockSize);

	ZSTD_indexResetPolicy_e needsIndexReset = zc->initialized ? ZSTDirp_continue : ZSTDirp_reset;

	if (ZSTD_indexTooCloseToMax(zc->blockState.matchState.window)) {
	needsIndexReset = ZSTDirp_reset;
	}

	if (!zc->staticSize) ZSTD_cwksp_bump_oversized_duration(ws, 0);

	/* Check if workspace is large enough, alloc a new one if needed */
	{ size_t const cctxSpace = zc->staticSize ? ZSTD_cwksp_alloc_size(sizeof(ZSTD_CCtx)) : 0;
	size_t const entropySpace = ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE);
	size_t const blockStateSpace = 2 * ZSTD_cwksp_alloc_size(sizeof(ZSTD_compressedBlockState_t));
	size_t const bufferSpace = ZSTD_cwksp_alloc_size(buffInSize) + ZSTD_cwksp_alloc_size(buffOutSize);
	size_t const ldmSpace = ZSTD_ldm_getTableSize(params.ldmParams);
	size_t const ldmSeqSpace = ZSTD_cwksp_alloc_size(maxNbLdmSeq * sizeof(rawSeq));

	size_t const neededSpace =
	cctxSpace +
	entropySpace +
	blockStateSpace +
	ldmSpace +
	ldmSeqSpace +
	matchStateSize +
	tokenSpace +
	bufferSpace;

	int const workspaceTooSmall = ZSTD_cwksp_sizeof(ws) < neededSpace;
	int const workspaceWasteful = ZSTD_cwksp_check_wasteful(ws, neededSpace);

	DEBUGLOG(4, "Need %zuKB workspace, including %zuKB for match state, and %zuKB for buffers",
	neededSpace>>10, matchStateSize>>10, bufferSpace>>10);
	DEBUGLOG(4, "windowSize: %zu - blockSize: %zu", windowSize, blockSize);

	if (workspaceTooSmall \|\| workspaceWasteful) {
	DEBUGLOG(4, "Resize workspaceSize from %zuKB to %zuKB",
	ZSTD_cwksp_sizeof(ws) >> 10,
	neededSpace >> 10);

	RETURN_ERROR_IF(zc->staticSize, memory_allocation, "static cctx : no resize");

	needsIndexReset = ZSTDirp_reset;

	ZSTD_cwksp_free(ws, zc->customMem);
	FORWARD_IF_ERROR(ZSTD_cwksp_create(ws, neededSpace, zc->customMem), "");

	DEBUGLOG(5, "reserving object space");
	/* Statically sized space.
	* entropyWorkspace never moves,
	* though prev/next block swap places */
	assert(ZSTD_cwksp_check_available(ws, 2 * sizeof(ZSTD_compressedBlockState_t)));
	zc->blockState.prevCBlock = (ZSTD_compressedBlockState_t*) ZSTD_cwksp_reserve_object(ws, sizeof(ZSTD_compressedBlockState_t));
	RETURN_ERROR_IF(zc->blockState.prevCBlock == NULL, memory_allocation, "couldn't allocate prevCBlock");
	zc->blockState.nextCBlock = (ZSTD_compressedBlockState_t*) ZSTD_cwksp_reserve_object(ws, sizeof(ZSTD_compressedBlockState_t));
	RETURN_ERROR_IF(zc->blockState.nextCBlock == NULL, memory_allocation, "couldn't allocate nextCBlock");
	zc->entropyWorkspace = (U32*) ZSTD_cwksp_reserve_object(ws, HUF_WORKSPACE_SIZE);
	RETURN_ERROR_IF(zc->blockState.nextCBlock == NULL, memory_allocation, "couldn't allocate entropyWorkspace");
	} }

	ZSTD_cwksp_clear(ws);

	/* init params */
	zc->appliedParams = params;
	zc->blockState.matchState.cParams = params.cParams;
	zc->pledgedSrcSizePlusOne = pledgedSrcSize+1;
	zc->consumedSrcSize = 0;
	zc->producedCSize = 0;
	if (pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN)
	zc->appliedParams.fParams.contentSizeFlag = 0;
	DEBUGLOG(4, "pledged content size : %u ; flag : %u",
	(unsigned)pledgedSrcSize, zc->appliedParams.fParams.contentSizeFlag);
	zc->blockSize = blockSize;

	XXH64_reset(&zc->xxhState, 0);
	zc->stage = ZSTDcs_init;
	zc->dictID = 0;

	ZSTD_reset_compressedBlockState(zc->blockState.prevCBlock);

	/* ZSTD_wildcopy() is used to copy into the literals buffer,
	* so we have to oversize the buffer by WILDCOPY_OVERLENGTH bytes.
	*/
	zc->seqStore.litStart = ZSTD_cwksp_reserve_buffer(ws, blockSize + WILDCOPY_OVERLENGTH);
	zc->seqStore.maxNbLit = blockSize;

	/* buffers */
	zc->inBuffSize = buffInSize;
	zc->inBuff = (char*)ZSTD_cwksp_reserve_buffer(ws, buffInSize);
	zc->outBuffSize = buffOutSize;
	zc->outBuff = (char*)ZSTD_cwksp_reserve_buffer(ws, buffOutSize);

	/* ldm bucketOffsets table */
	if (params.ldmParams.enableLdm) {
	/* TODO: avoid memset? */
	size_t const ldmBucketSize =
	((size_t)1) << (params.ldmParams.hashLog -
	params.ldmParams.bucketSizeLog);
	zc->ldmState.bucketOffsets = ZSTD_cwksp_reserve_buffer(ws, ldmBucketSize);
	memset(zc->ldmState.bucketOffsets, 0, ldmBucketSize);
	}

	/* sequences storage */
	ZSTD_referenceExternalSequences(zc, NULL, 0);
	zc->seqStore.maxNbSeq = maxNbSeq;
	zc->seqStore.llCode = ZSTD_cwksp_reserve_buffer(ws, maxNbSeq * sizeof(BYTE));
	zc->seqStore.mlCode = ZSTD_cwksp_reserve_buffer(ws, maxNbSeq * sizeof(BYTE));
	zc->seqStore.ofCode = ZSTD_cwksp_reserve_buffer(ws, maxNbSeq * sizeof(BYTE));
	zc->seqStore.sequencesStart = (seqDef)ZSTD_cwksp_reserve_aligned(ws, maxNbSeq sizeof(seqDef));

	FORWARD_IF_ERROR(ZSTD_reset_matchState(
	&zc->blockState.matchState,
	ws,
	&params.cParams,
	crp,
	needsIndexReset,
	ZSTD_resetTarget_CCtx), "");

	/* ldm hash table */
	if (params.ldmParams.enableLdm) {
	/* TODO: avoid memset? */
	size_t const ldmHSize = ((size_t)1) << params.ldmParams.hashLog;
	zc->ldmState.hashTable = (ldmEntry_t)ZSTD_cwksp_reserve_aligned(ws, ldmHSize sizeof(ldmEntry_t));
	memset(zc->ldmState.hashTable, 0, ldmHSize * sizeof(ldmEntry_t));
	zc->ldmSequences = (rawSeq)ZSTD_cwksp_reserve_aligned(ws, maxNbLdmSeq sizeof(rawSeq));
	zc->maxNbLdmSequences = maxNbLdmSeq;

	ZSTD_window_init(&zc->ldmState.window);
	ZSTD_window_clear(&zc->ldmState.window);
	zc->ldmState.loadedDictEnd = 0;
	}

	DEBUGLOG(3, "wksp: finished allocating, %zd bytes remain available", ZSTD_cwksp_available_space(ws));
	zc->initialized = 1;

	return 0;
	}
	}

	/* ZSTD_invalidateRepCodes() :
	* ensures next compression will not use repcodes from previous block.
	* Note : only works with regular variant;
	* do not use with extDict variant ! */
	void ZSTD_invalidateRepCodes(ZSTD_CCtx* cctx) {
	int i;
	for (i=0; i<ZSTD_REP_NUM; i++) cctx->blockState.prevCBlock->rep[i] = 0;
	assert(!ZSTD_window_hasExtDict(cctx->blockState.matchState.window));
	}

	/* These are the approximate sizes for each strategy past which copying the
	* dictionary tables into the working context is faster than using them
	* in-place.
	*/
	static const size_t attachDictSizeCutoffs[ZSTD_STRATEGY_MAX+1] = {
	8 KB, /* unused */
	8 KB, /* ZSTD_fast */
	16 KB, /* ZSTD_dfast */
	32 KB, /* ZSTD_greedy */
	32 KB, /* ZSTD_lazy */
	32 KB, /* ZSTD_lazy2 */
	32 KB, /* ZSTD_btlazy2 */
	32 KB, /* ZSTD_btopt */
	8 KB, /* ZSTD_btultra */
	8 KB /* ZSTD_btultra2 */
	};

	static int ZSTD_shouldAttachDict(const ZSTD_CDict* cdict,
	const ZSTD_CCtx_params* params,
	U64 pledgedSrcSize)
	{
	size_t cutoff = attachDictSizeCutoffs[cdict->matchState.cParams.strategy];
	return ( pledgedSrcSize <= cutoff
	\|\| pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN
	\|\| params->attachDictPref == ZSTD_dictForceAttach )
	&& params->attachDictPref != ZSTD_dictForceCopy
	&& !params->forceWindow; /* dictMatchState isn't correctly
	* handled in _enforceMaxDist */
	}

	static size_t
	ZSTD_resetCCtx_byAttachingCDict(ZSTD_CCtx* cctx,
	const ZSTD_CDict* cdict,
	ZSTD_CCtx_params params,
	U64 pledgedSrcSize,
	ZSTD_buffered_policy_e zbuff)
	{
	{ const ZSTD_compressionParameters* const cdict_cParams = &cdict->matchState.cParams;
	unsigned const windowLog = params.cParams.windowLog;
	assert(windowLog != 0);
	/* Resize working context table params for input only, since the dict
	* has its own tables. */
	/* pledgeSrcSize == 0 means 0! */
	params.cParams = ZSTD_adjustCParams_internal(*cdict_cParams, pledgedSrcSize, 0);
	params.cParams.windowLog = windowLog;
	FORWARD_IF_ERROR(ZSTD_resetCCtx_internal(cctx, params, pledgedSrcSize,
	ZSTDcrp_makeClean, zbuff), "");
	assert(cctx->appliedParams.cParams.strategy == cdict_cParams->strategy);
	}

	{ const U32 cdictEnd = (U32)( cdict->matchState.window.nextSrc
	- cdict->matchState.window.base);
	const U32 cdictLen = cdictEnd - cdict->matchState.window.dictLimit;
	if (cdictLen == 0) {
	/* don't even attach dictionaries with no contents */
	DEBUGLOG(4, "skipping attaching empty dictionary");
	} else {
	DEBUGLOG(4, "attaching dictionary into context");
	cctx->blockState.matchState.dictMatchState = &cdict->matchState;

	/* prep working match state so dict matches never have negative indices
	* when they are translated to the working context's index space. */
	if (cctx->blockState.matchState.window.dictLimit < cdictEnd) {
	cctx->blockState.matchState.window.nextSrc =
	cctx->blockState.matchState.window.base + cdictEnd;
	ZSTD_window_clear(&cctx->blockState.matchState.window);
	}
	/* loadedDictEnd is expressed within the referential of the active context */
	cctx->blockState.matchState.loadedDictEnd = cctx->blockState.matchState.window.dictLimit;
	} }

	cctx->dictID = cdict->dictID;

	/* copy block state */
	memcpy(cctx->blockState.prevCBlock, &cdict->cBlockState, sizeof(cdict->cBlockState));

	return 0;
	}

	static size_t ZSTD_resetCCtx_byCopyingCDict(ZSTD_CCtx* cctx,
	const ZSTD_CDict* cdict,
	ZSTD_CCtx_params params,
	U64 pledgedSrcSize,
	ZSTD_buffered_policy_e zbuff)
	{
	const ZSTD_compressionParameters *cdict_cParams = &cdict->matchState.cParams;

	DEBUGLOG(4, "copying dictionary into context");

	{ unsigned const windowLog = params.cParams.windowLog;
	assert(windowLog != 0);
	/* Copy only compression parameters related to tables. */
	params.cParams = *cdict_cParams;
	params.cParams.windowLog = windowLog;
	FORWARD_IF_ERROR(ZSTD_resetCCtx_internal(cctx, params, pledgedSrcSize,
	ZSTDcrp_leaveDirty, zbuff), "");
	assert(cctx->appliedParams.cParams.strategy == cdict_cParams->strategy);
	assert(cctx->appliedParams.cParams.hashLog == cdict_cParams->hashLog);
	assert(cctx->appliedParams.cParams.chainLog == cdict_cParams->chainLog);
	}

	ZSTD_cwksp_mark_tables_dirty(&cctx->workspace);

	/* copy tables */
	{ size_t const chainSize = (cdict_cParams->strategy == ZSTD_fast) ? 0 : ((size_t)1 << cdict_cParams->chainLog);
	size_t const hSize = (size_t)1 << cdict_cParams->hashLog;

	memcpy(cctx->blockState.matchState.hashTable,
	cdict->matchState.hashTable,
	hSize * sizeof(U32));
	memcpy(cctx->blockState.matchState.chainTable,
	cdict->matchState.chainTable,
	chainSize * sizeof(U32));
	}

	/* Zero the hashTable3, since the cdict never fills it */
	{ int const h3log = cctx->blockState.matchState.hashLog3;
	size_t const h3Size = h3log ? ((size_t)1 << h3log) : 0;
	assert(cdict->matchState.hashLog3 == 0);
	memset(cctx->blockState.matchState.hashTable3, 0, h3Size * sizeof(U32));
	}

	ZSTD_cwksp_mark_tables_clean(&cctx->workspace);

	/* copy dictionary offsets */
	{ ZSTD_matchState_t const* srcMatchState = &cdict->matchState;
	ZSTD_matchState_t* dstMatchState = &cctx->blockState.matchState;
	dstMatchState->window = srcMatchState->window;
	dstMatchState->nextToUpdate = srcMatchState->nextToUpdate;
	dstMatchState->loadedDictEnd= srcMatchState->loadedDictEnd;
	}

	cctx->dictID = cdict->dictID;

	/* copy block state */
	memcpy(cctx->blockState.prevCBlock, &cdict->cBlockState, sizeof(cdict->cBlockState));

	return 0;
	}

	/* We have a choice between copying the dictionary context into the working
	* context, or referencing the dictionary context from the working context
	* in-place. We decide here which strategy to use. */
	static size_t ZSTD_resetCCtx_usingCDict(ZSTD_CCtx* cctx,
	const ZSTD_CDict* cdict,
	const ZSTD_CCtx_params* params,
	U64 pledgedSrcSize,
	ZSTD_buffered_policy_e zbuff)
	{

	DEBUGLOG(4, "ZSTD_resetCCtx_usingCDict (pledgedSrcSize=%u)",
	(unsigned)pledgedSrcSize);

	if (ZSTD_shouldAttachDict(cdict, params, pledgedSrcSize)) {
	return ZSTD_resetCCtx_byAttachingCDict(
	cctx, cdict, *params, pledgedSrcSize, zbuff);
	} else {
	return ZSTD_resetCCtx_byCopyingCDict(
	cctx, cdict, *params, pledgedSrcSize, zbuff);
	}
	}

	/*! ZSTD_copyCCtx_internal() :
	* Duplicate an existing context `srcCCtx` into another one `dstCCtx`.
	* Only works during stage ZSTDcs_init (i.e. after creation, but before first call to ZSTD_compressContinue()).
	* The "context", in this case, refers to the hash and chain tables,
	* entropy tables, and dictionary references.
	* `windowLog` value is enforced if != 0, otherwise value is copied from srcCCtx.
	* @return : 0, or an error code */
	static size_t ZSTD_copyCCtx_internal(ZSTD_CCtx* dstCCtx,
	const ZSTD_CCtx* srcCCtx,
	ZSTD_frameParameters fParams,
	U64 pledgedSrcSize,
	ZSTD_buffered_policy_e zbuff)
	{
	DEBUGLOG(5, "ZSTD_copyCCtx_internal");
	RETURN_ERROR_IF(srcCCtx->stage!=ZSTDcs_init, stage_wrong,
	"Can't copy a ctx that's not in init stage.");

	memcpy(&dstCCtx->customMem, &srcCCtx->customMem, sizeof(ZSTD_customMem));
	{ ZSTD_CCtx_params params = dstCCtx->requestedParams;
	/* Copy only compression parameters related to tables. */
	params.cParams = srcCCtx->appliedParams.cParams;
	params.fParams = fParams;
	ZSTD_resetCCtx_internal(dstCCtx, params, pledgedSrcSize,
	ZSTDcrp_leaveDirty, zbuff);
	assert(dstCCtx->appliedParams.cParams.windowLog == srcCCtx->appliedParams.cParams.windowLog);
	assert(dstCCtx->appliedParams.cParams.strategy == srcCCtx->appliedParams.cParams.strategy);
	assert(dstCCtx->appliedParams.cParams.hashLog == srcCCtx->appliedParams.cParams.hashLog);
	assert(dstCCtx->appliedParams.cParams.chainLog == srcCCtx->appliedParams.cParams.chainLog);
	assert(dstCCtx->blockState.matchState.hashLog3 == srcCCtx->blockState.matchState.hashLog3);
	}

	ZSTD_cwksp_mark_tables_dirty(&dstCCtx->workspace);

	/* copy tables */
	{ size_t const chainSize = (srcCCtx->appliedParams.cParams.strategy == ZSTD_fast) ? 0 : ((size_t)1 << srcCCtx->appliedParams.cParams.chainLog);
	size_t const hSize = (size_t)1 << srcCCtx->appliedParams.cParams.hashLog;
	int const h3log = srcCCtx->blockState.matchState.hashLog3;
	size_t const h3Size = h3log ? ((size_t)1 << h3log) : 0;

	memcpy(dstCCtx->blockState.matchState.hashTable,
	srcCCtx->blockState.matchState.hashTable,
	hSize * sizeof(U32));
	memcpy(dstCCtx->blockState.matchState.chainTable,
	srcCCtx->blockState.matchState.chainTable,
	chainSize * sizeof(U32));
	memcpy(dstCCtx->blockState.matchState.hashTable3,
	srcCCtx->blockState.matchState.hashTable3,
	h3Size * sizeof(U32));
	}

	ZSTD_cwksp_mark_tables_clean(&dstCCtx->workspace);

	/* copy dictionary offsets */
	{
	const ZSTD_matchState_t* srcMatchState = &srcCCtx->blockState.matchState;
	ZSTD_matchState_t* dstMatchState = &dstCCtx->blockState.matchState;
	dstMatchState->window = srcMatchState->window;
	dstMatchState->nextToUpdate = srcMatchState->nextToUpdate;
	dstMatchState->loadedDictEnd= srcMatchState->loadedDictEnd;
	}
	dstCCtx->dictID = srcCCtx->dictID;

	/* copy block state */
	memcpy(dstCCtx->blockState.prevCBlock, srcCCtx->blockState.prevCBlock, sizeof(*srcCCtx->blockState.prevCBlock));

	return 0;
	}

	/*! ZSTD_copyCCtx() :
	* Duplicate an existing context `srcCCtx` into another one `dstCCtx`.
	* Only works during stage ZSTDcs_init (i.e. after creation, but before first call to ZSTD_compressContinue()).
	* pledgedSrcSize==0 means "unknown".
	* @return : 0, or an error code */
	size_t ZSTD_copyCCtx(ZSTD_CCtx* dstCCtx, const ZSTD_CCtx* srcCCtx, unsigned long long pledgedSrcSize)
	{
	ZSTD_frameParameters fParams = { 1 /content/, 0 /checksum/, 0 /noDictID/ };
	ZSTD_buffered_policy_e const zbuff = (ZSTD_buffered_policy_e)(srcCCtx->inBuffSize>0);
	ZSTD_STATIC_ASSERT((U32)ZSTDb_buffered==1);
	if (pledgedSrcSize==0) pledgedSrcSize = ZSTD_CONTENTSIZE_UNKNOWN;
	fParams.contentSizeFlag = (pledgedSrcSize != ZSTD_CONTENTSIZE_UNKNOWN);

	return ZSTD_copyCCtx_internal(dstCCtx, srcCCtx,
	fParams, pledgedSrcSize,
	zbuff);
	}


	#define ZSTD_ROWSIZE 16
	/*! ZSTD_reduceTable() :
	* reduce table indexes by `reducerValue`, or squash to zero.
	* PreserveMark preserves "unsorted mark" for btlazy2 strategy.
	* It must be set to a clear 0/1 value, to remove branch during inlining.
	* Presume table size is a multiple of ZSTD_ROWSIZE
	* to help auto-vectorization */
	FORCE_INLINE_TEMPLATE void
	ZSTD_reduceTable_internal (U32* const table, U32 const size, U32 const reducerValue, int const preserveMark)
	{
	int const nbRows = (int)size / ZSTD_ROWSIZE;
	int cellNb = 0;
	int rowNb;
	assert((size & (ZSTD_ROWSIZE-1)) == 0); /* multiple of ZSTD_ROWSIZE */
	assert(size < (1U<<31)); /* can be casted to int */

	#if defined (MEMORY_SANITIZER) && !defined (ZSTD_MSAN_DONT_POISON_WORKSPACE)
	/* To validate that the table re-use logic is sound, and that we don't
	* access table space that we haven't cleaned, we re-"poison" the table
	* space every time we mark it dirty.
	*
	* This function however is intended to operate on those dirty tables and
	* re-clean them. So when this function is used correctly, we can unpoison
	* the memory it operated on. This introduces a blind spot though, since
	* if we now try to operate on __actually__ poisoned memory, we will not
	* detect that. */
	__msan_unpoison(table, size * sizeof(U32));
	#endif

	for (rowNb=0 ; rowNb < nbRows ; rowNb++) {
	int column;
	for (column=0; column<ZSTD_ROWSIZE; column++) {
	if (preserveMark) {
	U32 const adder = (table[cellNb] == ZSTD_DUBT_UNSORTED_MARK) ? reducerValue : 0;
	table[cellNb] += adder;
	}
	if (table[cellNb] < reducerValue) table[cellNb] = 0;
	else table[cellNb] -= reducerValue;
	cellNb++;
	} }
	}

	static void ZSTD_reduceTable(U32* const table, U32 const size, U32 const reducerValue)
	{
	ZSTD_reduceTable_internal(table, size, reducerValue, 0);
	}

	static void ZSTD_reduceTable_btlazy2(U32* const table, U32 const size, U32 const reducerValue)
	{
	ZSTD_reduceTable_internal(table, size, reducerValue, 1);
	}

	/*! ZSTD_reduceIndex() :
	* rescale all indexes to avoid future overflow (indexes are U32) */
	static void ZSTD_reduceIndex (ZSTD_matchState_t* ms, ZSTD_CCtx_params const* params, const U32 reducerValue)
	{
	{ U32 const hSize = (U32)1 << params->cParams.hashLog;
	ZSTD_reduceTable(ms->hashTable, hSize, reducerValue);
	}

	if (params->cParams.strategy != ZSTD_fast) {
	U32 const chainSize = (U32)1 << params->cParams.chainLog;
	if (params->cParams.strategy == ZSTD_btlazy2)
	ZSTD_reduceTable_btlazy2(ms->chainTable, chainSize, reducerValue);
	else
	ZSTD_reduceTable(ms->chainTable, chainSize, reducerValue);
	}

	if (ms->hashLog3) {
	U32 const h3Size = (U32)1 << ms->hashLog3;
	ZSTD_reduceTable(ms->hashTable3, h3Size, reducerValue);
	}
	}


	/-******************************************************
	* Block entropic compression
	*********************************************************/

	/* See doc/zstd_compression_format.md for detailed format description */

	void ZSTD_seqToCodes(const seqStore_t* seqStorePtr)
	{
	const seqDef* const sequences = seqStorePtr->sequencesStart;
	BYTE* const llCodeTable = seqStorePtr->llCode;
	BYTE* const ofCodeTable = seqStorePtr->ofCode;
	BYTE* const mlCodeTable = seqStorePtr->mlCode;
	U32 const nbSeq = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart);
	U32 u;
	assert(nbSeq <= seqStorePtr->maxNbSeq);
	for (u=0; u<nbSeq; u++) {
	U32 const llv = sequences[u].litLength;
	U32 const mlv = sequences[u].matchLength;
	llCodeTable[u] = (BYTE)ZSTD_LLcode(llv);
	ofCodeTable[u] = (BYTE)ZSTD_highbit32(sequences[u].offset);
	mlCodeTable[u] = (BYTE)ZSTD_MLcode(mlv);
	}
	if (seqStorePtr->longLengthID==1)
	llCodeTable[seqStorePtr->longLengthPos] = MaxLL;
	if (seqStorePtr->longLengthID==2)
	mlCodeTable[seqStorePtr->longLengthPos] = MaxML;
	}

	/* ZSTD_useTargetCBlockSize():
	* Returns if target compressed block size param is being used.
	* If used, compression will do best effort to make a compressed block size to be around targetCBlockSize.
	* Returns 1 if true, 0 otherwise. */
	static int ZSTD_useTargetCBlockSize(const ZSTD_CCtx_params* cctxParams)
	{
	DEBUGLOG(5, "ZSTD_useTargetCBlockSize (targetCBlockSize=%zu)", cctxParams->targetCBlockSize);
	return (cctxParams->targetCBlockSize != 0);
	}

	/* ZSTD_compressSequences_internal():
	* actually compresses both literals and sequences */
	MEM_STATIC size_t
	ZSTD_compressSequences_internal(seqStore_t* seqStorePtr,
	const ZSTD_entropyCTables_t* prevEntropy,
	ZSTD_entropyCTables_t* nextEntropy,
	const ZSTD_CCtx_params* cctxParams,
	void* dst, size_t dstCapacity,
	void* entropyWorkspace, size_t entropyWkspSize,
	const int bmi2)
	{
	const int longOffsets = cctxParams->cParams.windowLog > STREAM_ACCUMULATOR_MIN;
	ZSTD_strategy const strategy = cctxParams->cParams.strategy;
	unsigned count[MaxSeq+1];
	FSE_CTable* CTable_LitLength = nextEntropy->fse.litlengthCTable;
	FSE_CTable* CTable_OffsetBits = nextEntropy->fse.offcodeCTable;
	FSE_CTable* CTable_MatchLength = nextEntropy->fse.matchlengthCTable;
	U32 LLtype, Offtype, MLtype; /* compressed, raw or rle */
	const seqDef* const sequences = seqStorePtr->sequencesStart;
	const BYTE* const ofCodeTable = seqStorePtr->ofCode;
	const BYTE* const llCodeTable = seqStorePtr->llCode;
	const BYTE* const mlCodeTable = seqStorePtr->mlCode;
	BYTE* const ostart = (BYTE*)dst;
	BYTE* const oend = ostart + dstCapacity;
	BYTE* op = ostart;
	size_t const nbSeq = (size_t)(seqStorePtr->sequences - seqStorePtr->sequencesStart);
	BYTE* seqHead;
	BYTE* lastNCount = NULL;

	DEBUGLOG(5, "ZSTD_compressSequences_internal (nbSeq=%zu)", nbSeq);
	ZSTD_STATIC_ASSERT(HUF_WORKSPACE_SIZE >= (1<<MAX(MLFSELog,LLFSELog)));

	/* Compress literals */
	{ const BYTE* const literals = seqStorePtr->litStart;
	size_t const litSize = (size_t)(seqStorePtr->lit - literals);
	size_t const cSize = ZSTD_compressLiterals(
	&prevEntropy->huf, &nextEntropy->huf,
	cctxParams->cParams.strategy,
	ZSTD_disableLiteralsCompression(cctxParams),
	op, dstCapacity,
	literals, litSize,
	entropyWorkspace, entropyWkspSize,
	bmi2);
	FORWARD_IF_ERROR(cSize, "ZSTD_compressLiterals failed");
	assert(cSize <= dstCapacity);
	op += cSize;
	}

	/* Sequences Header */
	RETURN_ERROR_IF((oend-op) < 3 /max nbSeq Size/ + 1 /seqHead/,
	dstSize_tooSmall, "Can't fit seq hdr in output buf!");
	if (nbSeq < 128) {
	*op++ = (BYTE)nbSeq;
	} else if (nbSeq < LONGNBSEQ) {
	op[0] = (BYTE)((nbSeq>>8) + 0x80);
	op[1] = (BYTE)nbSeq;
	op+=2;
	} else {
	op[0]=0xFF;
	MEM_writeLE16(op+1, (U16)(nbSeq - LONGNBSEQ));
	op+=3;
	}
	assert(op <= oend);
	if (nbSeq==0) {
	/* Copy the old tables over as if we repeated them */
	memcpy(&nextEntropy->fse, &prevEntropy->fse, sizeof(prevEntropy->fse));
	return (size_t)(op - ostart);
	}

	/* seqHead : flags for FSE encoding type */
	seqHead = op++;
	assert(op <= oend);

	/* convert length/distances into codes */
	ZSTD_seqToCodes(seqStorePtr);
	/* build CTable for Literal Lengths */
	{ unsigned max = MaxLL;
	size_t const mostFrequent = HIST_countFast_wksp(count, &max, llCodeTable, nbSeq, entropyWorkspace, entropyWkspSize); /* can't fail */
	DEBUGLOG(5, "Building LL table");
	nextEntropy->fse.litlength_repeatMode = prevEntropy->fse.litlength_repeatMode;
	LLtype = ZSTD_selectEncodingType(&nextEntropy->fse.litlength_repeatMode,
	count, max, mostFrequent, nbSeq,
	LLFSELog, prevEntropy->fse.litlengthCTable,
	LL_defaultNorm, LL_defaultNormLog,
	ZSTD_defaultAllowed, strategy);
	assert(set_basic < set_compressed && set_rle < set_compressed);
	assert(!(LLtype < set_compressed && nextEntropy->fse.litlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */
	{ size_t const countSize = ZSTD_buildCTable(
	op, (size_t)(oend - op),
	CTable_LitLength, LLFSELog, (symbolEncodingType_e)LLtype,
	count, max, llCodeTable, nbSeq,
	LL_defaultNorm, LL_defaultNormLog, MaxLL,
	prevEntropy->fse.litlengthCTable,
	sizeof(prevEntropy->fse.litlengthCTable),
	entropyWorkspace, entropyWkspSize);
	FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for LitLens failed");
	if (LLtype == set_compressed)
	lastNCount = op;
	op += countSize;
	assert(op <= oend);
	} }
	/* build CTable for Offsets */
	{ unsigned max = MaxOff;
	size_t const mostFrequent = HIST_countFast_wksp(
	count, &max, ofCodeTable, nbSeq, entropyWorkspace, entropyWkspSize); /* can't fail */
	/* We can only use the basic table if max <= DefaultMaxOff, otherwise the offsets are too large */
	ZSTD_defaultPolicy_e const defaultPolicy = (max <= DefaultMaxOff) ? ZSTD_defaultAllowed : ZSTD_defaultDisallowed;
	DEBUGLOG(5, "Building OF table");
	nextEntropy->fse.offcode_repeatMode = prevEntropy->fse.offcode_repeatMode;
	Offtype = ZSTD_selectEncodingType(&nextEntropy->fse.offcode_repeatMode,
	count, max, mostFrequent, nbSeq,
	OffFSELog, prevEntropy->fse.offcodeCTable,
	OF_defaultNorm, OF_defaultNormLog,
	defaultPolicy, strategy);
	assert(!(Offtype < set_compressed && nextEntropy->fse.offcode_repeatMode != FSE_repeat_none)); /* We don't copy tables */
	{ size_t const countSize = ZSTD_buildCTable(
	op, (size_t)(oend - op),
	CTable_OffsetBits, OffFSELog, (symbolEncodingType_e)Offtype,
	count, max, ofCodeTable, nbSeq,
	OF_defaultNorm, OF_defaultNormLog, DefaultMaxOff,
	prevEntropy->fse.offcodeCTable,
	sizeof(prevEntropy->fse.offcodeCTable),
	entropyWorkspace, entropyWkspSize);
	FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for Offsets failed");
	if (Offtype == set_compressed)
	lastNCount = op;
	op += countSize;
	assert(op <= oend);
	} }
	/* build CTable for MatchLengths */
	{ unsigned max = MaxML;
	size_t const mostFrequent = HIST_countFast_wksp(
	count, &max, mlCodeTable, nbSeq, entropyWorkspace, entropyWkspSize); /* can't fail */
	DEBUGLOG(5, "Building ML table (remaining space : %i)", (int)(oend-op));
	nextEntropy->fse.matchlength_repeatMode = prevEntropy->fse.matchlength_repeatMode;
	MLtype = ZSTD_selectEncodingType(&nextEntropy->fse.matchlength_repeatMode,
	count, max, mostFrequent, nbSeq,
	MLFSELog, prevEntropy->fse.matchlengthCTable,
	ML_defaultNorm, ML_defaultNormLog,
	ZSTD_defaultAllowed, strategy);
	assert(!(MLtype < set_compressed && nextEntropy->fse.matchlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */
	{ size_t const countSize = ZSTD_buildCTable(
	op, (size_t)(oend - op),
	CTable_MatchLength, MLFSELog, (symbolEncodingType_e)MLtype,
	count, max, mlCodeTable, nbSeq,
	ML_defaultNorm, ML_defaultNormLog, MaxML,
	prevEntropy->fse.matchlengthCTable,
	sizeof(prevEntropy->fse.matchlengthCTable),
	entropyWorkspace, entropyWkspSize);
	FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for MatchLengths failed");
	if (MLtype == set_compressed)
	lastNCount = op;
	op += countSize;
	assert(op <= oend);
	} }

	*seqHead = (BYTE)((LLtype<<6) + (Offtype<<4) + (MLtype<<2));

	{ size_t const bitstreamSize = ZSTD_encodeSequences(
	op, (size_t)(oend - op),
	CTable_MatchLength, mlCodeTable,
	CTable_OffsetBits, ofCodeTable,
	CTable_LitLength, llCodeTable,
	sequences, nbSeq,
	longOffsets, bmi2);
	FORWARD_IF_ERROR(bitstreamSize, "ZSTD_encodeSequences failed");
	op += bitstreamSize;
	assert(op <= oend);
	/* zstd versions <= 1.3.4 mistakenly report corruption when
	* FSE_readNCount() receives a buffer < 4 bytes.
	* Fixed by https://github.com/facebook/zstd/pull/1146.
	* This can happen when the last set_compressed table present is 2
	* bytes and the bitstream is only one byte.
	* In this exceedingly rare case, we will simply emit an uncompressed
	* block, since it isn't worth optimizing.
	*/
	if (lastNCount && (op - lastNCount) < 4) {
	/* NCountSize >= 2 && bitstreamSize > 0 ==> lastCountSize == 3 */
	assert(op - lastNCount == 3);
	DEBUGLOG(5, "Avoiding bug in zstd decoder in versions <= 1.3.4 by "
	"emitting an uncompressed block.");
	return 0;
	}
	}

	DEBUGLOG(5, "compressed block size : %u", (unsigned)(op - ostart));
	return (size_t)(op - ostart);
	}

	MEM_STATIC size_t
	ZSTD_compressSequences(seqStore_t* seqStorePtr,
	const ZSTD_entropyCTables_t* prevEntropy,
	ZSTD_entropyCTables_t* nextEntropy,
	const ZSTD_CCtx_params* cctxParams,
	void* dst, size_t dstCapacity,
	size_t srcSize,
	void* entropyWorkspace, size_t entropyWkspSize,
	int bmi2)
	{
	size_t const cSize = ZSTD_compressSequences_internal(
	seqStorePtr, prevEntropy, nextEntropy, cctxParams,
	dst, dstCapacity,
	entropyWorkspace, entropyWkspSize, bmi2);
	if (cSize == 0) return 0;
	/* When srcSize <= dstCapacity, there is enough space to write a raw uncompressed block.
	* Since we ran out of space, block must be not compressible, so fall back to raw uncompressed block.
	*/
	if ((cSize == ERROR(dstSize_tooSmall)) & (srcSize <= dstCapacity))
	return 0; /* block not compressed */
	FORWARD_IF_ERROR(cSize, "ZSTD_compressSequences_internal failed");

	/* Check compressibility */
	{ size_t const maxCSize = srcSize - ZSTD_minGain(srcSize, cctxParams->cParams.strategy);
	if (cSize >= maxCSize) return 0; /* block not compressed */
	}

	return cSize;
	}

	/* ZSTD_selectBlockCompressor() :
	* Not static, but internal use only (used by long distance matcher)
	* assumption : strat is a valid strategy */
	ZSTD_blockCompressor ZSTD_selectBlockCompressor(ZSTD_strategy strat, ZSTD_dictMode_e dictMode)
	{
	static const ZSTD_blockCompressor blockCompressor[3][ZSTD_STRATEGY_MAX+1] = {
	{ ZSTD_compressBlock_fast /* default for 0 */,
	ZSTD_compressBlock_fast,
	ZSTD_compressBlock_doubleFast,
	ZSTD_compressBlock_greedy,
	ZSTD_compressBlock_lazy,
	ZSTD_compressBlock_lazy2,
	ZSTD_compressBlock_btlazy2,
	ZSTD_compressBlock_btopt,
	ZSTD_compressBlock_btultra,
	ZSTD_compressBlock_btultra2 },
	{ ZSTD_compressBlock_fast_extDict /* default for 0 */,
	ZSTD_compressBlock_fast_extDict,
	ZSTD_compressBlock_doubleFast_extDict,
	ZSTD_compressBlock_greedy_extDict,
	ZSTD_compressBlock_lazy_extDict,
	ZSTD_compressBlock_lazy2_extDict,
	ZSTD_compressBlock_btlazy2_extDict,
	ZSTD_compressBlock_btopt_extDict,
	ZSTD_compressBlock_btultra_extDict,
	ZSTD_compressBlock_btultra_extDict },
	{ ZSTD_compressBlock_fast_dictMatchState /* default for 0 */,
	ZSTD_compressBlock_fast_dictMatchState,
	ZSTD_compressBlock_doubleFast_dictMatchState,
	ZSTD_compressBlock_greedy_dictMatchState,
	ZSTD_compressBlock_lazy_dictMatchState,
	ZSTD_compressBlock_lazy2_dictMatchState,
	ZSTD_compressBlock_btlazy2_dictMatchState,
	ZSTD_compressBlock_btopt_dictMatchState,
	ZSTD_compressBlock_btultra_dictMatchState,
	ZSTD_compressBlock_btultra_dictMatchState }
	};
	ZSTD_blockCompressor selectedCompressor;
	ZSTD_STATIC_ASSERT((unsigned)ZSTD_fast == 1);

	assert(ZSTD_cParam_withinBounds(ZSTD_c_strategy, strat));
	selectedCompressor = blockCompressor[(int)dictMode][(int)strat];
	assert(selectedCompressor != NULL);
	return selectedCompressor;
	}

	static void ZSTD_storeLastLiterals(seqStore_t* seqStorePtr,
	const BYTE* anchor, size_t lastLLSize)
	{
	memcpy(seqStorePtr->lit, anchor, lastLLSize);
	seqStorePtr->lit += lastLLSize;
	}

	void ZSTD_resetSeqStore(seqStore_t* ssPtr)
	{
	ssPtr->lit = ssPtr->litStart;
	ssPtr->sequences = ssPtr->sequencesStart;
	ssPtr->longLengthID = 0;
	}

	typedef enum { ZSTDbss_compress, ZSTDbss_noCompress } ZSTD_buildSeqStore_e;

	static size_t ZSTD_buildSeqStore(ZSTD_CCtx* zc, const void* src, size_t srcSize)
	{
	ZSTD_matchState_t* const ms = &zc->blockState.matchState;
	DEBUGLOG(5, "ZSTD_buildSeqStore (srcSize=%zu)", srcSize);
	assert(srcSize <= ZSTD_BLOCKSIZE_MAX);
	/* Assert that we have correctly flushed the ctx params into the ms's copy */
	ZSTD_assertEqualCParams(zc->appliedParams.cParams, ms->cParams);
	if (srcSize < MIN_CBLOCK_SIZE+ZSTD_blockHeaderSize+1) {
	ZSTD_ldm_skipSequences(&zc->externSeqStore, srcSize, zc->appliedParams.cParams.minMatch);
	return ZSTDbss_noCompress; /* don't even attempt compression below a certain srcSize */
	}
	ZSTD_resetSeqStore(&(zc->seqStore));
	/* required for optimal parser to read stats from dictionary */
	ms->opt.symbolCosts = &zc->blockState.prevCBlock->entropy;
	/* tell the optimal parser how we expect to compress literals */
	ms->opt.literalCompressionMode = zc->appliedParams.literalCompressionMode;
	/* a gap between an attached dict and the current window is not safe,
	* they must remain adjacent,
	* and when that stops being the case, the dict must be unset */
	assert(ms->dictMatchState == NULL \|\| ms->loadedDictEnd == ms->window.dictLimit);

	/* limited update after a very long match */
	{ const BYTE* const base = ms->window.base;
	const BYTE* const istart = (const BYTE*)src;
	const U32 current = (U32)(istart-base);
	if (sizeof(ptrdiff_t)==8) assert(istart - base < (ptrdiff_t)(U32)(-1)); /* ensure no overflow */
	if (current > ms->nextToUpdate + 384)
	ms->nextToUpdate = current - MIN(192, (U32)(current - ms->nextToUpdate - 384));
	}

	/* select and store sequences */
	{ ZSTD_dictMode_e const dictMode = ZSTD_matchState_dictMode(ms);
	size_t lastLLSize;
	{ int i;
	for (i = 0; i < ZSTD_REP_NUM; ++i)
	zc->blockState.nextCBlock->rep[i] = zc->blockState.prevCBlock->rep[i];
	}
	if (zc->externSeqStore.pos < zc->externSeqStore.size) {
	assert(!zc->appliedParams.ldmParams.enableLdm);
	/* Updates ldmSeqStore.pos */
	lastLLSize =
	ZSTD_ldm_blockCompress(&zc->externSeqStore,
	ms, &zc->seqStore,
	zc->blockState.nextCBlock->rep,
	src, srcSize);
	assert(zc->externSeqStore.pos <= zc->externSeqStore.size);
	} else if (zc->appliedParams.ldmParams.enableLdm) {
	rawSeqStore_t ldmSeqStore = {NULL, 0, 0, 0};

	ldmSeqStore.seq = zc->ldmSequences;
	ldmSeqStore.capacity = zc->maxNbLdmSequences;
	/* Updates ldmSeqStore.size */
	FORWARD_IF_ERROR(ZSTD_ldm_generateSequences(&zc->ldmState, &ldmSeqStore,
	&zc->appliedParams.ldmParams,
	src, srcSize), "");
	/* Updates ldmSeqStore.pos */
	lastLLSize =
	ZSTD_ldm_blockCompress(&ldmSeqStore,
	ms, &zc->seqStore,
	zc->blockState.nextCBlock->rep,
	src, srcSize);
	assert(ldmSeqStore.pos == ldmSeqStore.size);
	} else { /* not long range mode */
	ZSTD_blockCompressor const blockCompressor = ZSTD_selectBlockCompressor(zc->appliedParams.cParams.strategy, dictMode);
	lastLLSize = blockCompressor(ms, &zc->seqStore, zc->blockState.nextCBlock->rep, src, srcSize);
	}
	{ const BYTE* const lastLiterals = (const BYTE*)src + srcSize - lastLLSize;
	ZSTD_storeLastLiterals(&zc->seqStore, lastLiterals, lastLLSize);
	} }
	return ZSTDbss_compress;
	}

	static void ZSTD_copyBlockSequences(ZSTD_CCtx* zc)
	{
	const seqStore_t* seqStore = ZSTD_getSeqStore(zc);
	const seqDef* seqs = seqStore->sequencesStart;
	size_t seqsSize = seqStore->sequences - seqs;

	ZSTD_Sequence* outSeqs = &zc->seqCollector.seqStart[zc->seqCollector.seqIndex];
	size_t i; size_t position; int repIdx;

	assert(zc->seqCollector.seqIndex + 1 < zc->seqCollector.maxSequences);
	for (i = 0, position = 0; i < seqsSize; ++i) {
	outSeqs[i].offset = seqs[i].offset;
	outSeqs[i].litLength = seqs[i].litLength;
	outSeqs[i].matchLength = seqs[i].matchLength + MINMATCH;

	if (i == seqStore->longLengthPos) {
	if (seqStore->longLengthID == 1) {
	outSeqs[i].litLength += 0x10000;
	} else if (seqStore->longLengthID == 2) {
	outSeqs[i].matchLength += 0x10000;
	}
	}

	if (outSeqs[i].offset <= ZSTD_REP_NUM) {
	outSeqs[i].rep = outSeqs[i].offset;
	repIdx = (unsigned int)i - outSeqs[i].offset;

	if (outSeqs[i].litLength == 0) {
	if (outSeqs[i].offset < 3) {
	--repIdx;
	} else {
	repIdx = (unsigned int)i - 1;
	}
	++outSeqs[i].rep;
	}
	assert(repIdx >= -3);
	outSeqs[i].offset = repIdx >= 0 ? outSeqs[repIdx].offset : repStartValue[-repIdx - 1];
	if (outSeqs[i].rep == 4) {
	--outSeqs[i].offset;
	}
	} else {
	outSeqs[i].offset -= ZSTD_REP_NUM;
	}

	position += outSeqs[i].litLength;
	outSeqs[i].matchPos = (unsigned int)position;
	position += outSeqs[i].matchLength;
	}
	zc->seqCollector.seqIndex += seqsSize;
	}

	size_t ZSTD_getSequences(ZSTD_CCtx* zc, ZSTD_Sequence* outSeqs,
	size_t outSeqsSize, const void* src, size_t srcSize)
	{
	const size_t dstCapacity = ZSTD_compressBound(srcSize);
	void* dst = ZSTD_malloc(dstCapacity, ZSTD_defaultCMem);
	SeqCollector seqCollector;

	RETURN_ERROR_IF(dst == NULL, memory_allocation, "NULL pointer!");

	seqCollector.collectSequences = 1;
	seqCollector.seqStart = outSeqs;
	seqCollector.seqIndex = 0;
	seqCollector.maxSequences = outSeqsSize;
	zc->seqCollector = seqCollector;

	ZSTD_compress2(zc, dst, dstCapacity, src, srcSize);
	ZSTD_free(dst, ZSTD_defaultCMem);
	return zc->seqCollector.seqIndex;
	}

	/* Returns true if the given block is a RLE block */
	static int ZSTD_isRLE(const BYTE *ip, size_t length) {
	size_t i;
	if (length < 2) return 1;
	for (i = 1; i < length; ++i) {
	if (ip[0] != ip[i]) return 0;
	}
	return 1;
	}

	/* Returns true if the given block may be RLE.
	* This is just a heuristic based on the compressibility.
	* It may return both false positives and false negatives.
	*/
	static int ZSTD_maybeRLE(seqStore_t const* seqStore)
	{
	size_t const nbSeqs = (size_t)(seqStore->sequences - seqStore->sequencesStart);
	size_t const nbLits = (size_t)(seqStore->lit - seqStore->litStart);

	return nbSeqs < 4 && nbLits < 10;
	}

	static void ZSTD_confirmRepcodesAndEntropyTables(ZSTD_CCtx* zc)
	{
	ZSTD_compressedBlockState_t* const tmp = zc->blockState.prevCBlock;
	zc->blockState.prevCBlock = zc->blockState.nextCBlock;
	zc->blockState.nextCBlock = tmp;
	}

	static size_t ZSTD_compressBlock_internal(ZSTD_CCtx* zc,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize, U32 frame)
	{
	/* This the upper bound for the length of an rle block.
	* This isn't the actual upper bound. Finding the real threshold
	* needs further investigation.
	*/
	const U32 rleMaxLength = 25;
	size_t cSize;
	const BYTE* ip = (const BYTE*)src;
	BYTE* op = (BYTE*)dst;
	DEBUGLOG(5, "ZSTD_compressBlock_internal (dstCapacity=%u, dictLimit=%u, nextToUpdate=%u)",
	(unsigned)dstCapacity, (unsigned)zc->blockState.matchState.window.dictLimit,
	(unsigned)zc->blockState.matchState.nextToUpdate);

	{ const size_t bss = ZSTD_buildSeqStore(zc, src, srcSize);
	FORWARD_IF_ERROR(bss, "ZSTD_buildSeqStore failed");
	if (bss == ZSTDbss_noCompress) { cSize = 0; goto out; }
	}

	if (zc->seqCollector.collectSequences) {
	ZSTD_copyBlockSequences(zc);
	return 0;
	}

	/* encode sequences and literals */
	cSize = ZSTD_compressSequences(&zc->seqStore,
	&zc->blockState.prevCBlock->entropy, &zc->blockState.nextCBlock->entropy,
	&zc->appliedParams,
	dst, dstCapacity,
	srcSize,
	zc->entropyWorkspace, HUF_WORKSPACE_SIZE /* statically allocated in resetCCtx */,
	zc->bmi2);

	if (frame &&
	/* We don't want to emit our first block as a RLE even if it qualifies because
	* doing so will cause the decoder (cli only) to throw a "should consume all input error."
	* This is only an issue for zstd <= v1.4.3
	*/
	!zc->isFirstBlock &&
	cSize < rleMaxLength &&
	ZSTD_isRLE(ip, srcSize))
	{
	cSize = 1;
	op[0] = ip[0];
	}

	out:
	if (!ZSTD_isError(cSize) && cSize > 1) {
	ZSTD_confirmRepcodesAndEntropyTables(zc);
	}
	/* We check that dictionaries have offset codes available for the first
	* block. After the first block, the offcode table might not have large
	* enough codes to represent the offsets in the data.
	*/
	if (zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode == FSE_repeat_valid)
	zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode = FSE_repeat_check;

	return cSize;
	}

	static size_t ZSTD_compressBlock_targetCBlockSize_body(ZSTD_CCtx* zc,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const size_t bss, U32 lastBlock)
	{
	DEBUGLOG(6, "Attempting ZSTD_compressSuperBlock()");
	if (bss == ZSTDbss_compress) {
	if (/* We don't want to emit our first block as a RLE even if it qualifies because
	* doing so will cause the decoder (cli only) to throw a "should consume all input error."
	* This is only an issue for zstd <= v1.4.3
	*/
	!zc->isFirstBlock &&
	ZSTD_maybeRLE(&zc->seqStore) &&
	ZSTD_isRLE((BYTE const*)src, srcSize))
	{
	return ZSTD_rleCompressBlock(dst, dstCapacity, (BYTE const)src, srcSize, lastBlock);
	}
	/* Attempt superblock compression.
	*
	* Note that compressed size of ZSTD_compressSuperBlock() is not bound by the
	* standard ZSTD_compressBound(). This is a problem, because even if we have
	* space now, taking an extra byte now could cause us to run out of space later
	* and violate ZSTD_compressBound().
	*
	* Define blockBound(blockSize) = blockSize + ZSTD_blockHeaderSize.
	*
	* In order to respect ZSTD_compressBound() we must attempt to emit a raw
	* uncompressed block in these cases:
	* * cSize == 0: Return code for an uncompressed block.
	* * cSize == dstSize_tooSmall: We may have expanded beyond blockBound(srcSize).
	* ZSTD_noCompressBlock() will return dstSize_tooSmall if we are really out of
	* output space.
	* * cSize >= blockBound(srcSize): We have expanded the block too much so
	* emit an uncompressed block.
	*/
	{
	size_t const cSize = ZSTD_compressSuperBlock(zc, dst, dstCapacity, src, srcSize, lastBlock);
	if (cSize != ERROR(dstSize_tooSmall)) {
	size_t const maxCSize = srcSize - ZSTD_minGain(srcSize, zc->appliedParams.cParams.strategy);
	FORWARD_IF_ERROR(cSize, "ZSTD_compressSuperBlock failed");
	if (cSize != 0 && cSize < maxCSize + ZSTD_blockHeaderSize) {
	ZSTD_confirmRepcodesAndEntropyTables(zc);
	return cSize;
	}
	}
	}
	}

	DEBUGLOG(6, "Resorting to ZSTD_noCompressBlock()");
	/* Superblock compression failed, attempt to emit a single no compress block.
	* The decoder will be able to stream this block since it is uncompressed.
	*/
	return ZSTD_noCompressBlock(dst, dstCapacity, src, srcSize, lastBlock);
	}

	static size_t ZSTD_compressBlock_targetCBlockSize(ZSTD_CCtx* zc,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	U32 lastBlock)
	{
	size_t cSize = 0;
	const size_t bss = ZSTD_buildSeqStore(zc, src, srcSize);
	DEBUGLOG(5, "ZSTD_compressBlock_targetCBlockSize (dstCapacity=%u, dictLimit=%u, nextToUpdate=%u, srcSize=%zu)",
	(unsigned)dstCapacity, (unsigned)zc->blockState.matchState.window.dictLimit, (unsigned)zc->blockState.matchState.nextToUpdate, srcSize);
	FORWARD_IF_ERROR(bss, "ZSTD_buildSeqStore failed");

	cSize = ZSTD_compressBlock_targetCBlockSize_body(zc, dst, dstCapacity, src, srcSize, bss, lastBlock);
	FORWARD_IF_ERROR(cSize, "ZSTD_compressBlock_targetCBlockSize_body failed");

	if (zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode == FSE_repeat_valid)
	zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode = FSE_repeat_check;

	return cSize;
	}

	static void ZSTD_overflowCorrectIfNeeded(ZSTD_matchState_t* ms,
	ZSTD_cwksp* ws,
	ZSTD_CCtx_params const* params,
	void const* ip,
	void const* iend)
	{
	if (ZSTD_window_needOverflowCorrection(ms->window, iend)) {
	U32 const maxDist = (U32)1 << params->cParams.windowLog;
	U32 const cycleLog = ZSTD_cycleLog(params->cParams.chainLog, params->cParams.strategy);
	U32 const correction = ZSTD_window_correctOverflow(&ms->window, cycleLog, maxDist, ip);
	ZSTD_STATIC_ASSERT(ZSTD_CHAINLOG_MAX <= 30);
	ZSTD_STATIC_ASSERT(ZSTD_WINDOWLOG_MAX_32 <= 30);
	ZSTD_STATIC_ASSERT(ZSTD_WINDOWLOG_MAX <= 31);
	ZSTD_cwksp_mark_tables_dirty(ws);
	ZSTD_reduceIndex(ms, params, correction);
	ZSTD_cwksp_mark_tables_clean(ws);
	if (ms->nextToUpdate < correction) ms->nextToUpdate = 0;
	else ms->nextToUpdate -= correction;
	/* invalidate dictionaries on overflow correction */
	ms->loadedDictEnd = 0;
	ms->dictMatchState = NULL;
	}
	}

	/*! ZSTD_compress_frameChunk() :
	* Compress a chunk of data into one or multiple blocks.
	* All blocks will be terminated, all input will be consumed.
	* Function will issue an error if there is not enough `dstCapacity` to hold the compressed content.
	* Frame is supposed already started (header already produced)
	* @return : compressed size, or an error code
	*/
	static size_t ZSTD_compress_frameChunk (ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	U32 lastFrameChunk)
	{
	size_t blockSize = cctx->blockSize;
	size_t remaining = srcSize;
	const BYTE* ip = (const BYTE*)src;
	BYTE* const ostart = (BYTE*)dst;
	BYTE* op = ostart;
	U32 const maxDist = (U32)1 << cctx->appliedParams.cParams.windowLog;

	assert(cctx->appliedParams.cParams.windowLog <= ZSTD_WINDOWLOG_MAX);

	DEBUGLOG(5, "ZSTD_compress_frameChunk (blockSize=%u)", (unsigned)blockSize);
	if (cctx->appliedParams.fParams.checksumFlag && srcSize)
	XXH64_update(&cctx->xxhState, src, srcSize);

	while (remaining) {
	ZSTD_matchState_t* const ms = &cctx->blockState.matchState;
	U32 const lastBlock = lastFrameChunk & (blockSize >= remaining);

	RETURN_ERROR_IF(dstCapacity < ZSTD_blockHeaderSize + MIN_CBLOCK_SIZE,
	dstSize_tooSmall,
	"not enough space to store compressed block");
	if (remaining < blockSize) blockSize = remaining;

	ZSTD_overflowCorrectIfNeeded(
	ms, &cctx->workspace, &cctx->appliedParams, ip, ip + blockSize);
	ZSTD_checkDictValidity(&ms->window, ip + blockSize, maxDist, &ms->loadedDictEnd, &ms->dictMatchState);

	/* Ensure hash/chain table insertion resumes no sooner than lowlimit */
	if (ms->nextToUpdate < ms->window.lowLimit) ms->nextToUpdate = ms->window.lowLimit;

	{ size_t cSize;
	if (ZSTD_useTargetCBlockSize(&cctx->appliedParams)) {
	cSize = ZSTD_compressBlock_targetCBlockSize(cctx, op, dstCapacity, ip, blockSize, lastBlock);
	FORWARD_IF_ERROR(cSize, "ZSTD_compressBlock_targetCBlockSize failed");
	assert(cSize > 0);
	assert(cSize <= blockSize + ZSTD_blockHeaderSize);
	} else {
	cSize = ZSTD_compressBlock_internal(cctx,
	op+ZSTD_blockHeaderSize, dstCapacity-ZSTD_blockHeaderSize,
	ip, blockSize, 1 /* frame */);
	FORWARD_IF_ERROR(cSize, "ZSTD_compressBlock_internal failed");

	if (cSize == 0) { /* block is not compressible */
	cSize = ZSTD_noCompressBlock(op, dstCapacity, ip, blockSize, lastBlock);
	FORWARD_IF_ERROR(cSize, "ZSTD_noCompressBlock failed");
	} else {
	U32 const cBlockHeader = cSize == 1 ?
	lastBlock + (((U32)bt_rle)<<1) + (U32)(blockSize << 3) :
	lastBlock + (((U32)bt_compressed)<<1) + (U32)(cSize << 3);
	MEM_writeLE24(op, cBlockHeader);
	cSize += ZSTD_blockHeaderSize;
	}
	}


	ip += blockSize;
	assert(remaining >= blockSize);
	remaining -= blockSize;
	op += cSize;
	assert(dstCapacity >= cSize);
	dstCapacity -= cSize;
	cctx->isFirstBlock = 0;
	DEBUGLOG(5, "ZSTD_compress_frameChunk: adding a block of size %u",
	(unsigned)cSize);
	} }

	if (lastFrameChunk && (op>ostart)) cctx->stage = ZSTDcs_ending;
	return (size_t)(op-ostart);
	}


	static size_t ZSTD_writeFrameHeader(void* dst, size_t dstCapacity,
	const ZSTD_CCtx_params* params, U64 pledgedSrcSize, U32 dictID)
	{ BYTE* const op = (BYTE*)dst;
	U32 const dictIDSizeCodeLength = (dictID>0) + (dictID>=256) + (dictID>=65536); /* 0-3 */
	U32 const dictIDSizeCode = params->fParams.noDictIDFlag ? 0 : dictIDSizeCodeLength; /* 0-3 */
	U32 const checksumFlag = params->fParams.checksumFlag>0;
	U32 const windowSize = (U32)1 << params->cParams.windowLog;
	U32 const singleSegment = params->fParams.contentSizeFlag && (windowSize >= pledgedSrcSize);
	BYTE const windowLogByte = (BYTE)((params->cParams.windowLog - ZSTD_WINDOWLOG_ABSOLUTEMIN) << 3);
	U32 const fcsCode = params->fParams.contentSizeFlag ?
	(pledgedSrcSize>=256) + (pledgedSrcSize>=65536+256) + (pledgedSrcSize>=0xFFFFFFFFU) : 0; /* 0-3 */
	BYTE const frameHeaderDescriptionByte = (BYTE)(dictIDSizeCode + (checksumFlag<<2) + (singleSegment<<5) + (fcsCode<<6) );
	size_t pos=0;

	assert(!(params->fParams.contentSizeFlag && pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN));
	RETURN_ERROR_IF(dstCapacity < ZSTD_FRAMEHEADERSIZE_MAX, dstSize_tooSmall,
	"dst buf is too small to fit worst-case frame header size.");
	DEBUGLOG(4, "ZSTD_writeFrameHeader : dictIDFlag : %u ; dictID : %u ; dictIDSizeCode : %u",
	!params->fParams.noDictIDFlag, (unsigned)dictID, (unsigned)dictIDSizeCode);

	if (params->format == ZSTD_f_zstd1) {
	MEM_writeLE32(dst, ZSTD_MAGICNUMBER);
	pos = 4;
	}
	op[pos++] = frameHeaderDescriptionByte;
	if (!singleSegment) op[pos++] = windowLogByte;
	switch(dictIDSizeCode)
	{
	default: assert(0); /* impossible */
	case 0 : break;
	case 1 : op[pos] = (BYTE)(dictID); pos++; break;
	case 2 : MEM_writeLE16(op+pos, (U16)dictID); pos+=2; break;
	case 3 : MEM_writeLE32(op+pos, dictID); pos+=4; break;
	}
	switch(fcsCode)
	{
	default: assert(0); /* impossible */
	case 0 : if (singleSegment) op[pos++] = (BYTE)(pledgedSrcSize); break;
	case 1 : MEM_writeLE16(op+pos, (U16)(pledgedSrcSize-256)); pos+=2; break;
	case 2 : MEM_writeLE32(op+pos, (U32)(pledgedSrcSize)); pos+=4; break;
	case 3 : MEM_writeLE64(op+pos, (U64)(pledgedSrcSize)); pos+=8; break;
	}
	return pos;
	}

	/* ZSTD_writeLastEmptyBlock() :
	* output an empty Block with end-of-frame mark to complete a frame
	* @return : size of data written into `dst` (== ZSTD_blockHeaderSize (defined in zstd_internal.h))
	* or an error code if `dstCapacity` is too small (<ZSTD_blockHeaderSize)
	*/
	size_t ZSTD_writeLastEmptyBlock(void* dst, size_t dstCapacity)
	{
	RETURN_ERROR_IF(dstCapacity < ZSTD_blockHeaderSize, dstSize_tooSmall,
	"dst buf is too small to write frame trailer empty block.");
	{ U32 const cBlockHeader24 = 1 /lastBlock/ + (((U32)bt_raw)<<1); /* 0 size */
	MEM_writeLE24(dst, cBlockHeader24);
	return ZSTD_blockHeaderSize;
	}
	}

	size_t ZSTD_referenceExternalSequences(ZSTD_CCtx* cctx, rawSeq* seq, size_t nbSeq)
	{
	RETURN_ERROR_IF(cctx->stage != ZSTDcs_init, stage_wrong,
	"wrong cctx stage");
	RETURN_ERROR_IF(cctx->appliedParams.ldmParams.enableLdm,
	parameter_unsupported,
	"incompatible with ldm");
	cctx->externSeqStore.seq = seq;
	cctx->externSeqStore.size = nbSeq;
	cctx->externSeqStore.capacity = nbSeq;
	cctx->externSeqStore.pos = 0;
	return 0;
	}


	static size_t ZSTD_compressContinue_internal (ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	U32 frame, U32 lastFrameChunk)
	{
	ZSTD_matchState_t* const ms = &cctx->blockState.matchState;
	size_t fhSize = 0;

	DEBUGLOG(5, "ZSTD_compressContinue_internal, stage: %u, srcSize: %u",
	cctx->stage, (unsigned)srcSize);
	RETURN_ERROR_IF(cctx->stage==ZSTDcs_created, stage_wrong,
	"missing init (ZSTD_compressBegin)");

	if (frame && (cctx->stage==ZSTDcs_init)) {
	fhSize = ZSTD_writeFrameHeader(dst, dstCapacity, &cctx->appliedParams,
	cctx->pledgedSrcSizePlusOne-1, cctx->dictID);
	FORWARD_IF_ERROR(fhSize, "ZSTD_writeFrameHeader failed");
	assert(fhSize <= dstCapacity);
	dstCapacity -= fhSize;
	dst = (char*)dst + fhSize;
	cctx->stage = ZSTDcs_ongoing;
	}

	if (!srcSize) return fhSize; /* do not generate an empty block if no input */

	if (!ZSTD_window_update(&ms->window, src, srcSize)) {
	ms->nextToUpdate = ms->window.dictLimit;
	}
	if (cctx->appliedParams.ldmParams.enableLdm) {
	ZSTD_window_update(&cctx->ldmState.window, src, srcSize);
	}

	if (!frame) {
	/* overflow check and correction for block mode */
	ZSTD_overflowCorrectIfNeeded(
	ms, &cctx->workspace, &cctx->appliedParams,
	src, (BYTE const*)src + srcSize);
	}

	DEBUGLOG(5, "ZSTD_compressContinue_internal (blockSize=%u)", (unsigned)cctx->blockSize);
	{ size_t const cSize = frame ?
	ZSTD_compress_frameChunk (cctx, dst, dstCapacity, src, srcSize, lastFrameChunk) :
	ZSTD_compressBlock_internal (cctx, dst, dstCapacity, src, srcSize, 0 /* frame */);
	FORWARD_IF_ERROR(cSize, "%s", frame ? "ZSTD_compress_frameChunk failed" : "ZSTD_compressBlock_internal failed");
	cctx->consumedSrcSize += srcSize;
	cctx->producedCSize += (cSize + fhSize);
	assert(!(cctx->appliedParams.fParams.contentSizeFlag && cctx->pledgedSrcSizePlusOne == 0));
	if (cctx->pledgedSrcSizePlusOne != 0) { /* control src size */
	ZSTD_STATIC_ASSERT(ZSTD_CONTENTSIZE_UNKNOWN == (unsigned long long)-1);
	RETURN_ERROR_IF(
	cctx->consumedSrcSize+1 > cctx->pledgedSrcSizePlusOne,
	srcSize_wrong,
	"error : pledgedSrcSize = %u, while realSrcSize >= %u",
	(unsigned)cctx->pledgedSrcSizePlusOne-1,
	(unsigned)cctx->consumedSrcSize);
	}
	return cSize + fhSize;
	}
	}

	size_t ZSTD_compressContinue (ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize)
	{
	DEBUGLOG(5, "ZSTD_compressContinue (srcSize=%u)", (unsigned)srcSize);
	return ZSTD_compressContinue_internal(cctx, dst, dstCapacity, src, srcSize, 1 /* frame mode /, 0 / last chunk */);
	}


	size_t ZSTD_getBlockSize(const ZSTD_CCtx* cctx)
	{
	ZSTD_compressionParameters const cParams = cctx->appliedParams.cParams;
	assert(!ZSTD_checkCParams(cParams));
	return MIN (ZSTD_BLOCKSIZE_MAX, (U32)1 << cParams.windowLog);
	}

	size_t ZSTD_compressBlock(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize)
	{
	DEBUGLOG(5, "ZSTD_compressBlock: srcSize = %u", (unsigned)srcSize);
	{ size_t const blockSizeMax = ZSTD_getBlockSize(cctx);
	RETURN_ERROR_IF(srcSize > blockSizeMax, srcSize_wrong, "input is larger than a block"); }

	return ZSTD_compressContinue_internal(cctx, dst, dstCapacity, src, srcSize, 0 /* frame mode /, 0 / last chunk */);
	}

	/*! ZSTD_loadDictionaryContent() :
	* @return : 0, or an error code
	*/
	static size_t ZSTD_loadDictionaryContent(ZSTD_matchState_t* ms,
	ldmState_t* ls,
	ZSTD_cwksp* ws,
	ZSTD_CCtx_params const* params,
	const void* src, size_t srcSize,
	ZSTD_dictTableLoadMethod_e dtlm)
	{
	const BYTE* ip = (const BYTE*) src;
	const BYTE* const iend = ip + srcSize;

	ZSTD_window_update(&ms->window, src, srcSize);
	ms->loadedDictEnd = params->forceWindow ? 0 : (U32)(iend - ms->window.base);

	if (params->ldmParams.enableLdm && ls != NULL) {
	ZSTD_window_update(&ls->window, src, srcSize);
	ls->loadedDictEnd = params->forceWindow ? 0 : (U32)(iend - ls->window.base);
	}

	/* Assert that we the ms params match the params we're being given */
	ZSTD_assertEqualCParams(params->cParams, ms->cParams);

	if (srcSize <= HASH_READ_SIZE) return 0;

	while (iend - ip > HASH_READ_SIZE) {
	size_t const remaining = (size_t)(iend - ip);
	size_t const chunk = MIN(remaining, ZSTD_CHUNKSIZE_MAX);
	const BYTE* const ichunk = ip + chunk;

	ZSTD_overflowCorrectIfNeeded(ms, ws, params, ip, ichunk);

	if (params->ldmParams.enableLdm && ls != NULL)
	ZSTD_ldm_fillHashTable(ls, (const BYTE)src, (const BYTE)src + srcSize, &params->ldmParams);

	switch(params->cParams.strategy)
	{
	case ZSTD_fast:
	ZSTD_fillHashTable(ms, ichunk, dtlm);
	break;
	case ZSTD_dfast:
	ZSTD_fillDoubleHashTable(ms, ichunk, dtlm);
	break;

	case ZSTD_greedy:
	case ZSTD_lazy:
	case ZSTD_lazy2:
	if (chunk >= HASH_READ_SIZE)
	ZSTD_insertAndFindFirstIndex(ms, ichunk-HASH_READ_SIZE);
	break;

	case ZSTD_btlazy2: /* we want the dictionary table fully sorted */
	case ZSTD_btopt:
	case ZSTD_btultra:
	case ZSTD_btultra2:
	if (chunk >= HASH_READ_SIZE)
	ZSTD_updateTree(ms, ichunk-HASH_READ_SIZE, ichunk);
	break;

	default:
	assert(0); /* not possible : not a valid strategy id */
	}

	ip = ichunk;
	}

	ms->nextToUpdate = (U32)(iend - ms->window.base);
	return 0;
	}


	/* Dictionaries that assign zero probability to symbols that show up causes problems
	when FSE encoding. Refuse dictionaries that assign zero probability to symbols
	that we may encounter during compression.
	NOTE: This behavior is not standard and could be improved in the future. */
	static size_t ZSTD_checkDictNCount(short* normalizedCounter, unsigned dictMaxSymbolValue, unsigned maxSymbolValue) {
	U32 s;
	RETURN_ERROR_IF(dictMaxSymbolValue < maxSymbolValue, dictionary_corrupted, "dict fse tables don't have all symbols");
	for (s = 0; s <= maxSymbolValue; ++s) {
	RETURN_ERROR_IF(normalizedCounter[s] == 0, dictionary_corrupted, "dict fse tables don't have all symbols");
	}
	return 0;
	}

	size_t ZSTD_loadCEntropy(ZSTD_compressedBlockState_t* bs, void* workspace,
	short* offcodeNCount, unsigned* offcodeMaxValue,
	const void* const dict, size_t dictSize)
	{
	const BYTE* dictPtr = (const BYTE)dict; / skip magic num and dict ID */
	const BYTE* const dictEnd = dictPtr + dictSize;
	dictPtr += 8;
	bs->entropy.huf.repeatMode = HUF_repeat_check;

	{ unsigned maxSymbolValue = 255;
	unsigned hasZeroWeights = 1;
	size_t const hufHeaderSize = HUF_readCTable((HUF_CElt*)bs->entropy.huf.CTable, &maxSymbolValue, dictPtr,
	dictEnd-dictPtr, &hasZeroWeights);

	/* We only set the loaded table as valid if it contains all non-zero
	* weights. Otherwise, we set it to check */
	if (!hasZeroWeights)
	bs->entropy.huf.repeatMode = HUF_repeat_valid;

	RETURN_ERROR_IF(HUF_isError(hufHeaderSize), dictionary_corrupted, "");
	RETURN_ERROR_IF(maxSymbolValue < 255, dictionary_corrupted, "");
	dictPtr += hufHeaderSize;
	}

	{ unsigned offcodeLog;
	size_t const offcodeHeaderSize = FSE_readNCount(offcodeNCount, offcodeMaxValue, &offcodeLog, dictPtr, dictEnd-dictPtr);
	RETURN_ERROR_IF(FSE_isError(offcodeHeaderSize), dictionary_corrupted, "");
	RETURN_ERROR_IF(offcodeLog > OffFSELog, dictionary_corrupted, "");
	/* Defer checking offcodeMaxValue because we need to know the size of the dictionary content */
	/* fill all offset symbols to avoid garbage at end of table */
	RETURN_ERROR_IF(FSE_isError(FSE_buildCTable_wksp(
	bs->entropy.fse.offcodeCTable,
	offcodeNCount, MaxOff, offcodeLog,
	workspace, HUF_WORKSPACE_SIZE)),
	dictionary_corrupted, "");
	dictPtr += offcodeHeaderSize;
	}

	{ short matchlengthNCount[MaxML+1];
	unsigned matchlengthMaxValue = MaxML, matchlengthLog;
	size_t const matchlengthHeaderSize = FSE_readNCount(matchlengthNCount, &matchlengthMaxValue, &matchlengthLog, dictPtr, dictEnd-dictPtr);
	RETURN_ERROR_IF(FSE_isError(matchlengthHeaderSize), dictionary_corrupted, "");
	RETURN_ERROR_IF(matchlengthLog > MLFSELog, dictionary_corrupted, "");
	/* Every match length code must have non-zero probability */
	FORWARD_IF_ERROR( ZSTD_checkDictNCount(matchlengthNCount, matchlengthMaxValue, MaxML), "");
	RETURN_ERROR_IF(FSE_isError(FSE_buildCTable_wksp(
	bs->entropy.fse.matchlengthCTable,
	matchlengthNCount, matchlengthMaxValue, matchlengthLog,
	workspace, HUF_WORKSPACE_SIZE)),
	dictionary_corrupted, "");
	dictPtr += matchlengthHeaderSize;
	}

	{ short litlengthNCount[MaxLL+1];
	unsigned litlengthMaxValue = MaxLL, litlengthLog;
	size_t const litlengthHeaderSize = FSE_readNCount(litlengthNCount, &litlengthMaxValue, &litlengthLog, dictPtr, dictEnd-dictPtr);
	RETURN_ERROR_IF(FSE_isError(litlengthHeaderSize), dictionary_corrupted, "");
	RETURN_ERROR_IF(litlengthLog > LLFSELog, dictionary_corrupted, "");
	/* Every literal length code must have non-zero probability */
	FORWARD_IF_ERROR( ZSTD_checkDictNCount(litlengthNCount, litlengthMaxValue, MaxLL), "");
	RETURN_ERROR_IF(FSE_isError(FSE_buildCTable_wksp(
	bs->entropy.fse.litlengthCTable,
	litlengthNCount, litlengthMaxValue, litlengthLog,
	workspace, HUF_WORKSPACE_SIZE)),
	dictionary_corrupted, "");
	dictPtr += litlengthHeaderSize;
	}

	RETURN_ERROR_IF(dictPtr+12 > dictEnd, dictionary_corrupted, "");
	bs->rep[0] = MEM_readLE32(dictPtr+0);
	bs->rep[1] = MEM_readLE32(dictPtr+4);
	bs->rep[2] = MEM_readLE32(dictPtr+8);
	dictPtr += 12;

	return dictPtr - (const BYTE*)dict;
	}

	/* Dictionary format :
	* See :
	* https://github.com/facebook/zstd/blob/master/doc/zstd_compression_format.md#dictionary-format
	*/
	/*! ZSTD_loadZstdDictionary() :
	* @return : dictID, or an error code
	* assumptions : magic number supposed already checked
	* dictSize supposed >= 8
	*/
	static size_t ZSTD_loadZstdDictionary(ZSTD_compressedBlockState_t* bs,
	ZSTD_matchState_t* ms,
	ZSTD_cwksp* ws,
	ZSTD_CCtx_params const* params,
	const void* dict, size_t dictSize,
	ZSTD_dictTableLoadMethod_e dtlm,
	void* workspace)
	{
	const BYTE* dictPtr = (const BYTE*)dict;
	const BYTE* const dictEnd = dictPtr + dictSize;
	short offcodeNCount[MaxOff+1];
	unsigned offcodeMaxValue = MaxOff;
	size_t dictID;
	size_t eSize;

	ZSTD_STATIC_ASSERT(HUF_WORKSPACE_SIZE >= (1<<MAX(MLFSELog,LLFSELog)));
	assert(dictSize >= 8);
	assert(MEM_readLE32(dictPtr) == ZSTD_MAGIC_DICTIONARY);

	dictID = params->fParams.noDictIDFlag ? 0 : MEM_readLE32(dictPtr + 4 /* skip magic number */ );
	eSize = ZSTD_loadCEntropy(bs, workspace, offcodeNCount, &offcodeMaxValue, dict, dictSize);
	FORWARD_IF_ERROR(eSize, "ZSTD_loadCEntropy failed");
	dictPtr += eSize;

	{ size_t const dictContentSize = (size_t)(dictEnd - dictPtr);
	U32 offcodeMax = MaxOff;
	if (dictContentSize <= ((U32)-1) - 128 KB) {
	U32 const maxOffset = (U32)dictContentSize + 128 KB; /* The maximum offset that must be supported */
	offcodeMax = ZSTD_highbit32(maxOffset); /* Calculate minimum offset code required to represent maxOffset */
	}
	/* All offset values <= dictContentSize + 128 KB must be representable */
	FORWARD_IF_ERROR(ZSTD_checkDictNCount(offcodeNCount, offcodeMaxValue, MIN(offcodeMax, MaxOff)), "");
	/* All repCodes must be <= dictContentSize and != 0*/
	{ U32 u;
	for (u=0; u<3; u++) {
	RETURN_ERROR_IF(bs->rep[u] == 0, dictionary_corrupted, "");
	RETURN_ERROR_IF(bs->rep[u] > dictContentSize, dictionary_corrupted, "");
	} }

	bs->entropy.fse.offcode_repeatMode = FSE_repeat_valid;
	bs->entropy.fse.matchlength_repeatMode = FSE_repeat_valid;
	bs->entropy.fse.litlength_repeatMode = FSE_repeat_valid;
	FORWARD_IF_ERROR(ZSTD_loadDictionaryContent(
	ms, NULL, ws, params, dictPtr, dictContentSize, dtlm), "");
	return dictID;
	}
	}

	/** ZSTD_compress_insertDictionary() :
	* @return : dictID, or an error code */
	static size_t
	ZSTD_compress_insertDictionary(ZSTD_compressedBlockState_t* bs,
	ZSTD_matchState_t* ms,
	ldmState_t* ls,
	ZSTD_cwksp* ws,
	const ZSTD_CCtx_params* params,
	const void* dict, size_t dictSize,
	ZSTD_dictContentType_e dictContentType,
	ZSTD_dictTableLoadMethod_e dtlm,
	void* workspace)
	{
	DEBUGLOG(4, "ZSTD_compress_insertDictionary (dictSize=%u)", (U32)dictSize);
	if ((dict==NULL) \|\| (dictSize<8)) {
	RETURN_ERROR_IF(dictContentType == ZSTD_dct_fullDict, dictionary_wrong, "");
	return 0;
	}

	ZSTD_reset_compressedBlockState(bs);

	/* dict restricted modes */
	if (dictContentType == ZSTD_dct_rawContent)
	return ZSTD_loadDictionaryContent(ms, ls, ws, params, dict, dictSize, dtlm);

	if (MEM_readLE32(dict) != ZSTD_MAGIC_DICTIONARY) {
	if (dictContentType == ZSTD_dct_auto) {
	DEBUGLOG(4, "raw content dictionary detected");
	return ZSTD_loadDictionaryContent(
	ms, ls, ws, params, dict, dictSize, dtlm);
	}
	RETURN_ERROR_IF(dictContentType == ZSTD_dct_fullDict, dictionary_wrong, "");
	assert(0); /* impossible */
	}

	/* dict as full zstd dictionary */
	return ZSTD_loadZstdDictionary(
	bs, ms, ws, params, dict, dictSize, dtlm, workspace);
	}

	#define ZSTD_USE_CDICT_PARAMS_SRCSIZE_CUTOFF (128 KB)
	#define ZSTD_USE_CDICT_PARAMS_DICTSIZE_MULTIPLIER (6)

	/*! ZSTD_compressBegin_internal() :
	* @return : 0, or an error code */
	static size_t ZSTD_compressBegin_internal(ZSTD_CCtx* cctx,
	const void* dict, size_t dictSize,
	ZSTD_dictContentType_e dictContentType,
	ZSTD_dictTableLoadMethod_e dtlm,
	const ZSTD_CDict* cdict,
	const ZSTD_CCtx_params* params, U64 pledgedSrcSize,
	ZSTD_buffered_policy_e zbuff)
	{
	DEBUGLOG(4, "ZSTD_compressBegin_internal: wlog=%u", params->cParams.windowLog);
	/* params are supposed to be fully validated at this point */
	assert(!ZSTD_isError(ZSTD_checkCParams(params->cParams)));
	assert(!((dict) && (cdict))); /* either dict or cdict, not both */
	if ( (cdict)
	&& (cdict->dictContentSize > 0)
	&& ( pledgedSrcSize < ZSTD_USE_CDICT_PARAMS_SRCSIZE_CUTOFF
	\|\| pledgedSrcSize < cdict->dictContentSize * ZSTD_USE_CDICT_PARAMS_DICTSIZE_MULTIPLIER
	\|\| pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN
	\|\| cdict->compressionLevel == 0)
	&& (params->attachDictPref != ZSTD_dictForceLoad) ) {
	return ZSTD_resetCCtx_usingCDict(cctx, cdict, params, pledgedSrcSize, zbuff);
	}

	FORWARD_IF_ERROR( ZSTD_resetCCtx_internal(cctx, *params, pledgedSrcSize,
	ZSTDcrp_makeClean, zbuff) , "");
	{ size_t const dictID = cdict ?
	ZSTD_compress_insertDictionary(
	cctx->blockState.prevCBlock, &cctx->blockState.matchState,
	&cctx->ldmState, &cctx->workspace, &cctx->appliedParams, cdict->dictContent,
	cdict->dictContentSize, dictContentType, dtlm,
	cctx->entropyWorkspace)
	: ZSTD_compress_insertDictionary(
	cctx->blockState.prevCBlock, &cctx->blockState.matchState,
	&cctx->ldmState, &cctx->workspace, &cctx->appliedParams, dict, dictSize,
	dictContentType, dtlm, cctx->entropyWorkspace);
	FORWARD_IF_ERROR(dictID, "ZSTD_compress_insertDictionary failed");
	assert(dictID <= UINT_MAX);
	cctx->dictID = (U32)dictID;
	}
	return 0;
	}

	size_t ZSTD_compressBegin_advanced_internal(ZSTD_CCtx* cctx,
	const void* dict, size_t dictSize,
	ZSTD_dictContentType_e dictContentType,
	ZSTD_dictTableLoadMethod_e dtlm,
	const ZSTD_CDict* cdict,
	const ZSTD_CCtx_params* params,
	unsigned long long pledgedSrcSize)
	{
	DEBUGLOG(4, "ZSTD_compressBegin_advanced_internal: wlog=%u", params->cParams.windowLog);
	/* compression parameters verification and optimization */
	FORWARD_IF_ERROR( ZSTD_checkCParams(params->cParams) , "");
	return ZSTD_compressBegin_internal(cctx,
	dict, dictSize, dictContentType, dtlm,
	cdict,
	params, pledgedSrcSize,
	ZSTDb_not_buffered);
	}

	/*! ZSTD_compressBegin_advanced() :
	* @return : 0, or an error code */
	size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx,
	const void* dict, size_t dictSize,
	ZSTD_parameters params, unsigned long long pledgedSrcSize)
	{
	ZSTD_CCtx_params const cctxParams =
	ZSTD_assignParamsToCCtxParams(&cctx->requestedParams, &params);
	return ZSTD_compressBegin_advanced_internal(cctx,
	dict, dictSize, ZSTD_dct_auto, ZSTD_dtlm_fast,
	NULL /cdict/,
	&cctxParams, pledgedSrcSize);
	}

	size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel)
	{
	ZSTD_parameters const params = ZSTD_getParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize);
	ZSTD_CCtx_params const cctxParams =
	ZSTD_assignParamsToCCtxParams(&cctx->requestedParams, &params);
	DEBUGLOG(4, "ZSTD_compressBegin_usingDict (dictSize=%u)", (unsigned)dictSize);
	return ZSTD_compressBegin_internal(cctx, dict, dictSize, ZSTD_dct_auto, ZSTD_dtlm_fast, NULL,
	&cctxParams, ZSTD_CONTENTSIZE_UNKNOWN, ZSTDb_not_buffered);
	}

	size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel)
	{
	return ZSTD_compressBegin_usingDict(cctx, NULL, 0, compressionLevel);
	}


	/*! ZSTD_writeEpilogue() :
	* Ends a frame.
	* @return : nb of bytes written into dst (or an error code) */
	static size_t ZSTD_writeEpilogue(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity)
	{
	BYTE* const ostart = (BYTE*)dst;
	BYTE* op = ostart;
	size_t fhSize = 0;

	DEBUGLOG(4, "ZSTD_writeEpilogue");
	RETURN_ERROR_IF(cctx->stage == ZSTDcs_created, stage_wrong, "init missing");

	/* special case : empty frame */
	if (cctx->stage == ZSTDcs_init) {
	fhSize = ZSTD_writeFrameHeader(dst, dstCapacity, &cctx->appliedParams, 0, 0);
	FORWARD_IF_ERROR(fhSize, "ZSTD_writeFrameHeader failed");
	dstCapacity -= fhSize;
	op += fhSize;
	cctx->stage = ZSTDcs_ongoing;
	}

	if (cctx->stage != ZSTDcs_ending) {
	/* write one last empty block, make it the "last" block */
	U32 const cBlockHeader24 = 1 /* last block */ + (((U32)bt_raw)<<1) + 0;
	RETURN_ERROR_IF(dstCapacity<4, dstSize_tooSmall, "no room for epilogue");
	MEM_writeLE32(op, cBlockHeader24);
	op += ZSTD_blockHeaderSize;
	dstCapacity -= ZSTD_blockHeaderSize;
	}

	if (cctx->appliedParams.fParams.checksumFlag) {
	U32 const checksum = (U32) XXH64_digest(&cctx->xxhState);
	RETURN_ERROR_IF(dstCapacity<4, dstSize_tooSmall, "no room for checksum");
	DEBUGLOG(4, "ZSTD_writeEpilogue: write checksum : %08X", (unsigned)checksum);
	MEM_writeLE32(op, checksum);
	op += 4;
	}

	cctx->stage = ZSTDcs_created; /* return to "created but no init" status */
	return op-ostart;
	}

	size_t ZSTD_compressEnd (ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize)
	{
	size_t endResult;
	size_t const cSize = ZSTD_compressContinue_internal(cctx,
	dst, dstCapacity, src, srcSize,
	1 /* frame mode /, 1 / last chunk */);
	FORWARD_IF_ERROR(cSize, "ZSTD_compressContinue_internal failed");
	endResult = ZSTD_writeEpilogue(cctx, (char*)dst + cSize, dstCapacity-cSize);
	FORWARD_IF_ERROR(endResult, "ZSTD_writeEpilogue failed");
	assert(!(cctx->appliedParams.fParams.contentSizeFlag && cctx->pledgedSrcSizePlusOne == 0));
	if (cctx->pledgedSrcSizePlusOne != 0) { /* control src size */
	ZSTD_STATIC_ASSERT(ZSTD_CONTENTSIZE_UNKNOWN == (unsigned long long)-1);
	DEBUGLOG(4, "end of frame : controlling src size");
	RETURN_ERROR_IF(
	cctx->pledgedSrcSizePlusOne != cctx->consumedSrcSize+1,
	srcSize_wrong,
	"error : pledgedSrcSize = %u, while realSrcSize = %u",
	(unsigned)cctx->pledgedSrcSizePlusOne-1,
	(unsigned)cctx->consumedSrcSize);
	}
	return cSize + endResult;
	}


	static size_t ZSTD_compress_internal (ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const void* dict,size_t dictSize,
	const ZSTD_parameters* params)
	{
	ZSTD_CCtx_params const cctxParams =
	ZSTD_assignParamsToCCtxParams(&cctx->requestedParams, params);
	DEBUGLOG(4, "ZSTD_compress_internal");
	return ZSTD_compress_advanced_internal(cctx,
	dst, dstCapacity,
	src, srcSize,
	dict, dictSize,
	&cctxParams);
	}

	size_t ZSTD_compress_advanced (ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const void* dict,size_t dictSize,
	ZSTD_parameters params)
	{
	DEBUGLOG(4, "ZSTD_compress_advanced");
	FORWARD_IF_ERROR(ZSTD_checkCParams(params.cParams), "");
	return ZSTD_compress_internal(cctx,
	dst, dstCapacity,
	src, srcSize,
	dict, dictSize,
	&params);
	}

	/* Internal */
	size_t ZSTD_compress_advanced_internal(
	ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const void* dict,size_t dictSize,
	const ZSTD_CCtx_params* params)
	{
	DEBUGLOG(4, "ZSTD_compress_advanced_internal (srcSize:%u)", (unsigned)srcSize);
	FORWARD_IF_ERROR( ZSTD_compressBegin_internal(cctx,
	dict, dictSize, ZSTD_dct_auto, ZSTD_dtlm_fast, NULL,
	params, srcSize, ZSTDb_not_buffered) , "");
	return ZSTD_compressEnd(cctx, dst, dstCapacity, src, srcSize);
	}

	size_t ZSTD_compress_usingDict(ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const void* dict, size_t dictSize,
	int compressionLevel)
	{
	ZSTD_parameters const params = ZSTD_getParams_internal(compressionLevel, srcSize, dict ? dictSize : 0);
	ZSTD_CCtx_params cctxParams = ZSTD_assignParamsToCCtxParams(&cctx->requestedParams, &params);
	DEBUGLOG(4, "ZSTD_compress_usingDict (srcSize=%u)", (unsigned)srcSize);
	assert(params.fParams.contentSizeFlag == 1);
	return ZSTD_compress_advanced_internal(cctx, dst, dstCapacity, src, srcSize, dict, dictSize, &cctxParams);
	}

	size_t ZSTD_compressCCtx(ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	int compressionLevel)
	{
	DEBUGLOG(4, "ZSTD_compressCCtx (srcSize=%u)", (unsigned)srcSize);
	assert(cctx != NULL);
	return ZSTD_compress_usingDict(cctx, dst, dstCapacity, src, srcSize, NULL, 0, compressionLevel);
	}

	size_t ZSTD_compress(void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	int compressionLevel)
	{
	size_t result;
	ZSTD_CCtx ctxBody;
	ZSTD_initCCtx(&ctxBody, ZSTD_defaultCMem);
	result = ZSTD_compressCCtx(&ctxBody, dst, dstCapacity, src, srcSize, compressionLevel);
	ZSTD_freeCCtxContent(&ctxBody); /* can't free ctxBody itself, as it's on stack; free only heap content */
	return result;
	}


	/* ===== Dictionary API ===== */

	/*! ZSTD_estimateCDictSize_advanced() :
	* Estimate amount of memory that will be needed to create a dictionary with following arguments */
	size_t ZSTD_estimateCDictSize_advanced(
	size_t dictSize, ZSTD_compressionParameters cParams,
	ZSTD_dictLoadMethod_e dictLoadMethod)
	{
	DEBUGLOG(5, "sizeof(ZSTD_CDict) : %u", (unsigned)sizeof(ZSTD_CDict));
	return ZSTD_cwksp_alloc_size(sizeof(ZSTD_CDict))
	+ ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE)
	+ ZSTD_sizeof_matchState(&cParams, /* forCCtx */ 0)
	+ (dictLoadMethod == ZSTD_dlm_byRef ? 0
	: ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(dictSize, sizeof(void *))));
	}

	size_t ZSTD_estimateCDictSize(size_t dictSize, int compressionLevel)
	{
	ZSTD_compressionParameters const cParams = ZSTD_getCParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize);
	return ZSTD_estimateCDictSize_advanced(dictSize, cParams, ZSTD_dlm_byCopy);
	}

	size_t ZSTD_sizeof_CDict(const ZSTD_CDict* cdict)
	{
	if (cdict==NULL) return 0; /* support sizeof on NULL */
	DEBUGLOG(5, "sizeof(cdict) : %u", (unsigned)sizeof(cdict));
	/* cdict may be in the workspace */
	return (cdict->workspace.workspace == cdict ? 0 : sizeof(*cdict))
	+ ZSTD_cwksp_sizeof(&cdict->workspace);
	}

	static size_t ZSTD_initCDict_internal(
	ZSTD_CDict* cdict,
	const void* dictBuffer, size_t dictSize,
	ZSTD_dictLoadMethod_e dictLoadMethod,
	ZSTD_dictContentType_e dictContentType,
	ZSTD_compressionParameters cParams)
	{
	DEBUGLOG(3, "ZSTD_initCDict_internal (dictContentType:%u)", (unsigned)dictContentType);
	assert(!ZSTD_checkCParams(cParams));
	cdict->matchState.cParams = cParams;
	if ((dictLoadMethod == ZSTD_dlm_byRef) \|\| (!dictBuffer) \|\| (!dictSize)) {
	cdict->dictContent = dictBuffer;
	} else {
	void internalBuffer = ZSTD_cwksp_reserve_object(&cdict->workspace, ZSTD_cwksp_align(dictSize, sizeof(void)));
	RETURN_ERROR_IF(!internalBuffer, memory_allocation, "NULL pointer!");
	cdict->dictContent = internalBuffer;
	memcpy(internalBuffer, dictBuffer, dictSize);
	}
	cdict->dictContentSize = dictSize;

	cdict->entropyWorkspace = (U32*)ZSTD_cwksp_reserve_object(&cdict->workspace, HUF_WORKSPACE_SIZE);


	/* Reset the state to no dictionary */
	ZSTD_reset_compressedBlockState(&cdict->cBlockState);
	FORWARD_IF_ERROR(ZSTD_reset_matchState(
	&cdict->matchState,
	&cdict->workspace,
	&cParams,
	ZSTDcrp_makeClean,
	ZSTDirp_reset,
	ZSTD_resetTarget_CDict), "");
	/* (Maybe) load the dictionary
	* Skips loading the dictionary if it is < 8 bytes.
	*/
	{ ZSTD_CCtx_params params;
	memset(&params, 0, sizeof(params));
	params.compressionLevel = ZSTD_CLEVEL_DEFAULT;
	params.fParams.contentSizeFlag = 1;
	params.cParams = cParams;
	{ size_t const dictID = ZSTD_compress_insertDictionary(
	&cdict->cBlockState, &cdict->matchState, NULL, &cdict->workspace,
	&params, cdict->dictContent, cdict->dictContentSize,
	dictContentType, ZSTD_dtlm_full, cdict->entropyWorkspace);
	FORWARD_IF_ERROR(dictID, "ZSTD_compress_insertDictionary failed");
	assert(dictID <= (size_t)(U32)-1);
	cdict->dictID = (U32)dictID;
	}
	}

	return 0;
	}

	ZSTD_CDict* ZSTD_createCDict_advanced(const void* dictBuffer, size_t dictSize,
	ZSTD_dictLoadMethod_e dictLoadMethod,
	ZSTD_dictContentType_e dictContentType,
	ZSTD_compressionParameters cParams, ZSTD_customMem customMem)
	{
	DEBUGLOG(3, "ZSTD_createCDict_advanced, mode %u", (unsigned)dictContentType);
	if (!customMem.customAlloc ^ !customMem.customFree) return NULL;

	{ size_t const workspaceSize =
	ZSTD_cwksp_alloc_size(sizeof(ZSTD_CDict)) +
	ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE) +
	ZSTD_sizeof_matchState(&cParams, /* forCCtx */ 0) +
	(dictLoadMethod == ZSTD_dlm_byRef ? 0
	: ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(dictSize, sizeof(void*))));
	void* const workspace = ZSTD_malloc(workspaceSize, customMem);
	ZSTD_cwksp ws;
	ZSTD_CDict* cdict;

	if (!workspace) {
	ZSTD_free(workspace, customMem);
	return NULL;
	}

	ZSTD_cwksp_init(&ws, workspace, workspaceSize);

	cdict = (ZSTD_CDict*)ZSTD_cwksp_reserve_object(&ws, sizeof(ZSTD_CDict));
	assert(cdict != NULL);
	ZSTD_cwksp_move(&cdict->workspace, &ws);
	cdict->customMem = customMem;
	cdict->compressionLevel = 0; /* signals advanced API usage */

	if (ZSTD_isError( ZSTD_initCDict_internal(cdict,
	dictBuffer, dictSize,
	dictLoadMethod, dictContentType,
	cParams) )) {
	ZSTD_freeCDict(cdict);
	return NULL;
	}

	return cdict;
	}
	}

	ZSTD_CDict* ZSTD_createCDict(const void* dict, size_t dictSize, int compressionLevel)
	{
	ZSTD_compressionParameters cParams = ZSTD_getCParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize);
	ZSTD_CDict* cdict = ZSTD_createCDict_advanced(dict, dictSize,
	ZSTD_dlm_byCopy, ZSTD_dct_auto,
	cParams, ZSTD_defaultCMem);
	if (cdict)
	cdict->compressionLevel = compressionLevel == 0 ? ZSTD_CLEVEL_DEFAULT : compressionLevel;
	return cdict;
	}

	ZSTD_CDict* ZSTD_createCDict_byReference(const void* dict, size_t dictSize, int compressionLevel)
	{
	ZSTD_compressionParameters cParams = ZSTD_getCParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize);
	return ZSTD_createCDict_advanced(dict, dictSize,
	ZSTD_dlm_byRef, ZSTD_dct_auto,
	cParams, ZSTD_defaultCMem);
	}

	size_t ZSTD_freeCDict(ZSTD_CDict* cdict)
	{
	if (cdict==NULL) return 0; /* support free on NULL */
	{ ZSTD_customMem const cMem = cdict->customMem;
	int cdictInWorkspace = ZSTD_cwksp_owns_buffer(&cdict->workspace, cdict);
	ZSTD_cwksp_free(&cdict->workspace, cMem);
	if (!cdictInWorkspace) {
	ZSTD_free(cdict, cMem);
	}
	return 0;
	}
	}

	/*! ZSTD_initStaticCDict_advanced() :
	* Generate a digested dictionary in provided memory area.
	* workspace: The memory area to emplace the dictionary into.
	* Provided pointer must 8-bytes aligned.
	* It must outlive dictionary usage.
	* workspaceSize: Use ZSTD_estimateCDictSize()
	* to determine how large workspace must be.
	* cParams : use ZSTD_getCParams() to transform a compression level
	* into its relevants cParams.
	* @return : pointer to ZSTD_CDict*, or NULL if error (size too small)
	* Note : there is no corresponding "free" function.
	* Since workspace was allocated externally, it must be freed externally.
	*/
	const ZSTD_CDict* ZSTD_initStaticCDict(
	void* workspace, size_t workspaceSize,
	const void* dict, size_t dictSize,
	ZSTD_dictLoadMethod_e dictLoadMethod,
	ZSTD_dictContentType_e dictContentType,
	ZSTD_compressionParameters cParams)
	{
	size_t const matchStateSize = ZSTD_sizeof_matchState(&cParams, /* forCCtx */ 0);
	size_t const neededSize = ZSTD_cwksp_alloc_size(sizeof(ZSTD_CDict))
	+ (dictLoadMethod == ZSTD_dlm_byRef ? 0
	: ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(dictSize, sizeof(void*))))
	+ ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE)
	+ matchStateSize;
	ZSTD_CDict* cdict;

	if ((size_t)workspace & 7) return NULL; /* 8-aligned */

	{
	ZSTD_cwksp ws;
	ZSTD_cwksp_init(&ws, workspace, workspaceSize);
	cdict = (ZSTD_CDict*)ZSTD_cwksp_reserve_object(&ws, sizeof(ZSTD_CDict));
	if (cdict == NULL) return NULL;
	ZSTD_cwksp_move(&cdict->workspace, &ws);
	}

	DEBUGLOG(4, "(workspaceSize < neededSize) : (%u < %u) => %u",
	(unsigned)workspaceSize, (unsigned)neededSize, (unsigned)(workspaceSize < neededSize));
	if (workspaceSize < neededSize) return NULL;

	if (ZSTD_isError( ZSTD_initCDict_internal(cdict,
	dict, dictSize,
	dictLoadMethod, dictContentType,
	cParams) ))
	return NULL;

	return cdict;
	}

	ZSTD_compressionParameters ZSTD_getCParamsFromCDict(const ZSTD_CDict* cdict)
	{
	assert(cdict != NULL);
	return cdict->matchState.cParams;
	}

	/* ZSTD_compressBegin_usingCDict_advanced() :
	* cdict must be != NULL */
	size_t ZSTD_compressBegin_usingCDict_advanced(
	ZSTD_CCtx* const cctx, const ZSTD_CDict* const cdict,
	ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize)
	{
	DEBUGLOG(4, "ZSTD_compressBegin_usingCDict_advanced");
	RETURN_ERROR_IF(cdict==NULL, dictionary_wrong, "NULL pointer!");
	{ ZSTD_CCtx_params params = cctx->requestedParams;
	params.cParams = ( pledgedSrcSize < ZSTD_USE_CDICT_PARAMS_SRCSIZE_CUTOFF
	\|\| pledgedSrcSize < cdict->dictContentSize * ZSTD_USE_CDICT_PARAMS_DICTSIZE_MULTIPLIER
	\|\| pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN
	\|\| cdict->compressionLevel == 0 )
	&& (params.attachDictPref != ZSTD_dictForceLoad) ?
	ZSTD_getCParamsFromCDict(cdict)
	: ZSTD_getCParams(cdict->compressionLevel,
	pledgedSrcSize,
	cdict->dictContentSize);
	/* Increase window log to fit the entire dictionary and source if the
	* source size is known. Limit the increase to 19, which is the
	* window log for compression level 1 with the largest source size.
	*/
	if (pledgedSrcSize != ZSTD_CONTENTSIZE_UNKNOWN) {
	U32 const limitedSrcSize = (U32)MIN(pledgedSrcSize, 1U << 19);
	U32 const limitedSrcLog = limitedSrcSize > 1 ? ZSTD_highbit32(limitedSrcSize - 1) + 1 : 1;
	params.cParams.windowLog = MAX(params.cParams.windowLog, limitedSrcLog);
	}
	params.fParams = fParams;
	return ZSTD_compressBegin_internal(cctx,
	NULL, 0, ZSTD_dct_auto, ZSTD_dtlm_fast,
	cdict,
	&params, pledgedSrcSize,
	ZSTDb_not_buffered);
	}
	}

	/* ZSTD_compressBegin_usingCDict() :
	* pledgedSrcSize=0 means "unknown"
	* if pledgedSrcSize>0, it will enable contentSizeFlag */
	size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict)
	{
	ZSTD_frameParameters const fParams = { 0 /content/, 0 /checksum/, 0 /noDictID/ };
	DEBUGLOG(4, "ZSTD_compressBegin_usingCDict : dictIDFlag == %u", !fParams.noDictIDFlag);
	return ZSTD_compressBegin_usingCDict_advanced(cctx, cdict, fParams, ZSTD_CONTENTSIZE_UNKNOWN);
	}

	size_t ZSTD_compress_usingCDict_advanced(ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const ZSTD_CDict* cdict, ZSTD_frameParameters fParams)
	{
	FORWARD_IF_ERROR(ZSTD_compressBegin_usingCDict_advanced(cctx, cdict, fParams, srcSize), ""); /* will check if cdict != NULL */
	return ZSTD_compressEnd(cctx, dst, dstCapacity, src, srcSize);
	}

	/*! ZSTD_compress_usingCDict() :
	* Compression using a digested Dictionary.
	* Faster startup than ZSTD_compress_usingDict(), recommended when same dictionary is used multiple times.
	* Note that compression parameters are decided at CDict creation time
	* while frame parameters are hardcoded */
	size_t ZSTD_compress_usingCDict(ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const ZSTD_CDict* cdict)
	{
	ZSTD_frameParameters const fParams = { 1 /content/, 0 /checksum/, 0 /noDictID/ };
	return ZSTD_compress_usingCDict_advanced(cctx, dst, dstCapacity, src, srcSize, cdict, fParams);
	}



	/* ******************************************************************
	* Streaming
	********************************************************************/

	ZSTD_CStream* ZSTD_createCStream(void)
	{
	DEBUGLOG(3, "ZSTD_createCStream");
	return ZSTD_createCStream_advanced(ZSTD_defaultCMem);
	}

	ZSTD_CStream* ZSTD_initStaticCStream(void *workspace, size_t workspaceSize)
	{
	return ZSTD_initStaticCCtx(workspace, workspaceSize);
	}

	ZSTD_CStream* ZSTD_createCStream_advanced(ZSTD_customMem customMem)
	{ /* CStream and CCtx are now same object */
	return ZSTD_createCCtx_advanced(customMem);
	}

	size_t ZSTD_freeCStream(ZSTD_CStream* zcs)
	{
	return ZSTD_freeCCtx(zcs); /* same object */
	}



	/====== Initialization ======/

	size_t ZSTD_CStreamInSize(void) { return ZSTD_BLOCKSIZE_MAX; }

	size_t ZSTD_CStreamOutSize(void)
	{
	return ZSTD_compressBound(ZSTD_BLOCKSIZE_MAX) + ZSTD_blockHeaderSize + 4 /* 32-bits hash */ ;
	}

	static size_t ZSTD_resetCStream_internal(ZSTD_CStream* cctx,
	const void* const dict, size_t const dictSize, ZSTD_dictContentType_e const dictContentType,
	const ZSTD_CDict* const cdict,
	ZSTD_CCtx_params params, unsigned long long const pledgedSrcSize)
	{
	DEBUGLOG(4, "ZSTD_resetCStream_internal");
	/* Finalize the compression parameters */
	params.cParams = ZSTD_getCParamsFromCCtxParams(&params, pledgedSrcSize, dictSize);
	/* params are supposed to be fully validated at this point */
	assert(!ZSTD_isError(ZSTD_checkCParams(params.cParams)));
	assert(!((dict) && (cdict))); /* either dict or cdict, not both */

	FORWARD_IF_ERROR( ZSTD_compressBegin_internal(cctx,
	dict, dictSize, dictContentType, ZSTD_dtlm_fast,
	cdict,
	&params, pledgedSrcSize,
	ZSTDb_buffered) , "");

	cctx->inToCompress = 0;
	cctx->inBuffPos = 0;
	cctx->inBuffTarget = cctx->blockSize
	+ (cctx->blockSize == pledgedSrcSize); /* for small input: avoid automatic flush on reaching end of block, since it would require to add a 3-bytes null block to end frame */
	cctx->outBuffContentSize = cctx->outBuffFlushedSize = 0;
	cctx->streamStage = zcss_load;
	cctx->frameEnded = 0;
	return 0; /* ready to go */
	}

	/* ZSTD_resetCStream():
	* pledgedSrcSize == 0 means "unknown" */
	size_t ZSTD_resetCStream(ZSTD_CStream* zcs, unsigned long long pss)
	{
	/* temporary : 0 interpreted as "unknown" during transition period.
	* Users willing to specify "unknown" must use ZSTD_CONTENTSIZE_UNKNOWN.
	* 0 will be interpreted as "empty" in the future.
	*/
	U64 const pledgedSrcSize = (pss==0) ? ZSTD_CONTENTSIZE_UNKNOWN : pss;
	DEBUGLOG(4, "ZSTD_resetCStream: pledgedSrcSize = %u", (unsigned)pledgedSrcSize);
	FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
	FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , "");
	return 0;
	}

	/*! ZSTD_initCStream_internal() :
	* Note : for lib/compress only. Used by zstdmt_compress.c.
	* Assumption 1 : params are valid
	* Assumption 2 : either dict, or cdict, is defined, not both */
	size_t ZSTD_initCStream_internal(ZSTD_CStream* zcs,
	const void* dict, size_t dictSize, const ZSTD_CDict* cdict,
	const ZSTD_CCtx_params* params,
	unsigned long long pledgedSrcSize)
	{
	DEBUGLOG(4, "ZSTD_initCStream_internal");
	FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
	FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , "");
	assert(!ZSTD_isError(ZSTD_checkCParams(params->cParams)));
	zcs->requestedParams = *params;
	assert(!((dict) && (cdict))); /* either dict or cdict, not both */
	if (dict) {
	FORWARD_IF_ERROR( ZSTD_CCtx_loadDictionary(zcs, dict, dictSize) , "");
	} else {
	/* Dictionary is cleared if !cdict */
	FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, cdict) , "");
	}
	return 0;
	}

	/* ZSTD_initCStream_usingCDict_advanced() :
	* same as ZSTD_initCStream_usingCDict(), with control over frame parameters */
	size_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs,
	const ZSTD_CDict* cdict,
	ZSTD_frameParameters fParams,
	unsigned long long pledgedSrcSize)
	{
	DEBUGLOG(4, "ZSTD_initCStream_usingCDict_advanced");
	FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
	FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , "");
	zcs->requestedParams.fParams = fParams;
	FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, cdict) , "");
	return 0;
	}

	/* note : cdict must outlive compression session */
	size_t ZSTD_initCStream_usingCDict(ZSTD_CStream* zcs, const ZSTD_CDict* cdict)
	{
	DEBUGLOG(4, "ZSTD_initCStream_usingCDict");
	FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
	FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, cdict) , "");
	return 0;
	}


	/* ZSTD_initCStream_advanced() :
	* pledgedSrcSize must be exact.
	* if srcSize is not known at init time, use value ZSTD_CONTENTSIZE_UNKNOWN.
	* dict is loaded with default parameters ZSTD_dct_auto and ZSTD_dlm_byCopy. */
	size_t ZSTD_initCStream_advanced(ZSTD_CStream* zcs,
	const void* dict, size_t dictSize,
	ZSTD_parameters params, unsigned long long pss)
	{
	/* for compatibility with older programs relying on this behavior.
	* Users should now specify ZSTD_CONTENTSIZE_UNKNOWN.
	* This line will be removed in the future.
	*/
	U64 const pledgedSrcSize = (pss==0 && params.fParams.contentSizeFlag==0) ? ZSTD_CONTENTSIZE_UNKNOWN : pss;
	DEBUGLOG(4, "ZSTD_initCStream_advanced");
	FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
	FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , "");
	FORWARD_IF_ERROR( ZSTD_checkCParams(params.cParams) , "");
	zcs->requestedParams = ZSTD_assignParamsToCCtxParams(&zcs->requestedParams, &params);
	FORWARD_IF_ERROR( ZSTD_CCtx_loadDictionary(zcs, dict, dictSize) , "");
	return 0;
	}

	size_t ZSTD_initCStream_usingDict(ZSTD_CStream* zcs, const void* dict, size_t dictSize, int compressionLevel)
	{
	DEBUGLOG(4, "ZSTD_initCStream_usingDict");
	FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
	FORWARD_IF_ERROR( ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel) , "");
	FORWARD_IF_ERROR( ZSTD_CCtx_loadDictionary(zcs, dict, dictSize) , "");
	return 0;
	}

	size_t ZSTD_initCStream_srcSize(ZSTD_CStream* zcs, int compressionLevel, unsigned long long pss)
	{
	/* temporary : 0 interpreted as "unknown" during transition period.
	* Users willing to specify "unknown" must use ZSTD_CONTENTSIZE_UNKNOWN.
	* 0 will be interpreted as "empty" in the future.
	*/
	U64 const pledgedSrcSize = (pss==0) ? ZSTD_CONTENTSIZE_UNKNOWN : pss;
	DEBUGLOG(4, "ZSTD_initCStream_srcSize");
	FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
	FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, NULL) , "");
	FORWARD_IF_ERROR( ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel) , "");
	FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , "");
	return 0;
	}

	size_t ZSTD_initCStream(ZSTD_CStream* zcs, int compressionLevel)
	{
	DEBUGLOG(4, "ZSTD_initCStream");
	FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
	FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, NULL) , "");
	FORWARD_IF_ERROR( ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel) , "");
	return 0;
	}

	/====== Compression ======/

	static size_t ZSTD_nextInputSizeHint(const ZSTD_CCtx* cctx)
	{
	size_t hintInSize = cctx->inBuffTarget - cctx->inBuffPos;
	if (hintInSize==0) hintInSize = cctx->blockSize;
	return hintInSize;
	}

	/** ZSTD_compressStream_generic():
	* internal function for all compressStream() variants
	* non-static, because can be called from zstdmt_compress.c
	* @return : hint size for next input */
	static size_t ZSTD_compressStream_generic(ZSTD_CStream* zcs,
	ZSTD_outBuffer* output,
	ZSTD_inBuffer* input,
	ZSTD_EndDirective const flushMode)
	{
	const char* const istart = (const char*)input->src;
	const char* const iend = input->size != 0 ? istart + input->size : istart;
	const char* ip = input->pos != 0 ? istart + input->pos : istart;
	char* const ostart = (char*)output->dst;
	char* const oend = output->size != 0 ? ostart + output->size : ostart;
	char* op = output->pos != 0 ? ostart + output->pos : ostart;
	U32 someMoreWork = 1;

	/* check expectations */
	DEBUGLOG(5, "ZSTD_compressStream_generic, flush=%u", (unsigned)flushMode);
	assert(zcs->inBuff != NULL);
	assert(zcs->inBuffSize > 0);
	assert(zcs->outBuff != NULL);
	assert(zcs->outBuffSize > 0);
	assert(output->pos <= output->size);
	assert(input->pos <= input->size);

	while (someMoreWork) {
	switch(zcs->streamStage)
	{
	case zcss_init:
	RETURN_ERROR(init_missing, "call ZSTD_initCStream() first!");

	case zcss_load:
	if ( (flushMode == ZSTD_e_end)
	&& ((size_t)(oend-op) >= ZSTD_compressBound(iend-ip)) /* enough dstCapacity */
	&& (zcs->inBuffPos == 0) ) {
	/* shortcut to compression pass directly into output buffer */
	size_t const cSize = ZSTD_compressEnd(zcs,
	op, oend-op, ip, iend-ip);
	DEBUGLOG(4, "ZSTD_compressEnd : cSize=%u", (unsigned)cSize);
	FORWARD_IF_ERROR(cSize, "ZSTD_compressEnd failed");
	ip = iend;
	op += cSize;
	zcs->frameEnded = 1;
	ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
	someMoreWork = 0; break;
	}
	/* complete loading into inBuffer */
	{ size_t const toLoad = zcs->inBuffTarget - zcs->inBuffPos;
	size_t const loaded = ZSTD_limitCopy(
	zcs->inBuff + zcs->inBuffPos, toLoad,
	ip, iend-ip);
	zcs->inBuffPos += loaded;
	if (loaded != 0)
	ip += loaded;
	if ( (flushMode == ZSTD_e_continue)
	&& (zcs->inBuffPos < zcs->inBuffTarget) ) {
	/* not enough input to fill full block : stop here */
	someMoreWork = 0; break;
	}
	if ( (flushMode == ZSTD_e_flush)
	&& (zcs->inBuffPos == zcs->inToCompress) ) {
	/* empty */
	someMoreWork = 0; break;
	}
	}
	/* compress current block (note : this stage cannot be stopped in the middle) */
	DEBUGLOG(5, "stream compression stage (flushMode==%u)", flushMode);
	{ void* cDst;
	size_t cSize;
	size_t const iSize = zcs->inBuffPos - zcs->inToCompress;
	size_t oSize = oend-op;
	unsigned const lastBlock = (flushMode == ZSTD_e_end) && (ip==iend);
	if (oSize >= ZSTD_compressBound(iSize))
	cDst = op; /* compress into output buffer, to skip flush stage */
	else
	cDst = zcs->outBuff, oSize = zcs->outBuffSize;
	cSize = lastBlock ?
	ZSTD_compressEnd(zcs, cDst, oSize,
	zcs->inBuff + zcs->inToCompress, iSize) :
	ZSTD_compressContinue(zcs, cDst, oSize,
	zcs->inBuff + zcs->inToCompress, iSize);
	FORWARD_IF_ERROR(cSize, "%s", lastBlock ? "ZSTD_compressEnd failed" : "ZSTD_compressContinue failed");
	zcs->frameEnded = lastBlock;
	/* prepare next block */
	zcs->inBuffTarget = zcs->inBuffPos + zcs->blockSize;
	if (zcs->inBuffTarget > zcs->inBuffSize)
	zcs->inBuffPos = 0, zcs->inBuffTarget = zcs->blockSize;
	DEBUGLOG(5, "inBuffTarget:%u / inBuffSize:%u",
	(unsigned)zcs->inBuffTarget, (unsigned)zcs->inBuffSize);
	if (!lastBlock)
	assert(zcs->inBuffTarget <= zcs->inBuffSize);
	zcs->inToCompress = zcs->inBuffPos;
	if (cDst == op) { /* no need to flush */
	op += cSize;
	if (zcs->frameEnded) {
	DEBUGLOG(5, "Frame completed directly in outBuffer");
	someMoreWork = 0;
	ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
	}
	break;
	}
	zcs->outBuffContentSize = cSize;
	zcs->outBuffFlushedSize = 0;
	zcs->streamStage = zcss_flush; /* pass-through to flush stage */
	}
	/* fall-through */
	case zcss_flush:
	DEBUGLOG(5, "flush stage");
	{ size_t const toFlush = zcs->outBuffContentSize - zcs->outBuffFlushedSize;
	size_t const flushed = ZSTD_limitCopy(op, (size_t)(oend-op),
	zcs->outBuff + zcs->outBuffFlushedSize, toFlush);
	DEBUGLOG(5, "toFlush: %u into %u ==> flushed: %u",
	(unsigned)toFlush, (unsigned)(oend-op), (unsigned)flushed);
	if (flushed)
	op += flushed;
	zcs->outBuffFlushedSize += flushed;
	if (toFlush!=flushed) {
	/* flush not fully completed, presumably because dst is too small */
	assert(op==oend);
	someMoreWork = 0;
	break;
	}
	zcs->outBuffContentSize = zcs->outBuffFlushedSize = 0;
	if (zcs->frameEnded) {
	DEBUGLOG(5, "Frame completed on flush");
	someMoreWork = 0;
	ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
	break;
	}
	zcs->streamStage = zcss_load;
	break;
	}

	default: /* impossible */
	assert(0);
	}
	}

	input->pos = ip - istart;
	output->pos = op - ostart;
	if (zcs->frameEnded) return 0;
	return ZSTD_nextInputSizeHint(zcs);
	}

	static size_t ZSTD_nextInputSizeHint_MTorST(const ZSTD_CCtx* cctx)
	{
	#ifdef ZSTD_MULTITHREAD
	if (cctx->appliedParams.nbWorkers >= 1) {
	assert(cctx->mtctx != NULL);
	return ZSTDMT_nextInputSizeHint(cctx->mtctx);
	}
	#endif
	return ZSTD_nextInputSizeHint(cctx);

	}

	size_t ZSTD_compressStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input)
	{
	FORWARD_IF_ERROR( ZSTD_compressStream2(zcs, output, input, ZSTD_e_continue) , "");
	return ZSTD_nextInputSizeHint_MTorST(zcs);
	}


	size_t ZSTD_compressStream2( ZSTD_CCtx* cctx,
	ZSTD_outBuffer* output,
	ZSTD_inBuffer* input,
	ZSTD_EndDirective endOp)
	{
	DEBUGLOG(5, "ZSTD_compressStream2, endOp=%u ", (unsigned)endOp);
	/* check conditions */
	RETURN_ERROR_IF(output->pos > output->size, GENERIC, "invalid buffer");
	RETURN_ERROR_IF(input->pos > input->size, GENERIC, "invalid buffer");
	assert(cctx!=NULL);

	/* transparent initialization stage */
	if (cctx->streamStage == zcss_init) {
	ZSTD_CCtx_params params = cctx->requestedParams;
	ZSTD_prefixDict const prefixDict = cctx->prefixDict;
	FORWARD_IF_ERROR( ZSTD_initLocalDict(cctx) , ""); /* Init the local dict if present. */
	memset(&cctx->prefixDict, 0, sizeof(cctx->prefixDict)); /* single usage */
	assert(prefixDict.dict==NULL \|\| cctx->cdict==NULL); /* only one can be set */
	DEBUGLOG(4, "ZSTD_compressStream2 : transparent init stage");
	if (endOp == ZSTD_e_end) cctx->pledgedSrcSizePlusOne = input->size + 1; /* auto-fix pledgedSrcSize */
	params.cParams = ZSTD_getCParamsFromCCtxParams(
	&cctx->requestedParams, cctx->pledgedSrcSizePlusOne-1, 0 /dictSize/);


	#ifdef ZSTD_MULTITHREAD
	if ((cctx->pledgedSrcSizePlusOne-1) <= ZSTDMT_JOBSIZE_MIN) {
	params.nbWorkers = 0; /* do not invoke multi-threading when src size is too small */
	}
	if (params.nbWorkers > 0) {
	/* mt context creation */
	if (cctx->mtctx == NULL) {
	DEBUGLOG(4, "ZSTD_compressStream2: creating new mtctx for nbWorkers=%u",
	params.nbWorkers);
	cctx->mtctx = ZSTDMT_createCCtx_advanced((U32)params.nbWorkers, cctx->customMem);
	RETURN_ERROR_IF(cctx->mtctx == NULL, memory_allocation, "NULL pointer!");
	}
	/* mt compression */
	DEBUGLOG(4, "call ZSTDMT_initCStream_internal as nbWorkers=%u", params.nbWorkers);
	FORWARD_IF_ERROR( ZSTDMT_initCStream_internal(
	cctx->mtctx,
	prefixDict.dict, prefixDict.dictSize, prefixDict.dictContentType,
	cctx->cdict, params, cctx->pledgedSrcSizePlusOne-1) , "");
	cctx->streamStage = zcss_load;
	cctx->appliedParams.nbWorkers = params.nbWorkers;
	} else
	#endif
	{ FORWARD_IF_ERROR( ZSTD_resetCStream_internal(cctx,
	prefixDict.dict, prefixDict.dictSize, prefixDict.dictContentType,
	cctx->cdict,
	params, cctx->pledgedSrcSizePlusOne-1) , "");
	assert(cctx->streamStage == zcss_load);
	assert(cctx->appliedParams.nbWorkers == 0);
	} }
	/* end of transparent initialization stage */

	/* compression stage */
	#ifdef ZSTD_MULTITHREAD
	if (cctx->appliedParams.nbWorkers > 0) {
	int const forceMaxProgress = (endOp == ZSTD_e_flush \|\| endOp == ZSTD_e_end);
	size_t flushMin;
	assert(forceMaxProgress \|\| endOp == ZSTD_e_continue /* Protection for a new flush type */);
	if (cctx->cParamsChanged) {
	ZSTDMT_updateCParams_whileCompressing(cctx->mtctx, &cctx->requestedParams);
	cctx->cParamsChanged = 0;
	}
	do {
	flushMin = ZSTDMT_compressStream_generic(cctx->mtctx, output, input, endOp);
	if ( ZSTD_isError(flushMin)
	\|\| (endOp == ZSTD_e_end && flushMin == 0) ) { /* compression completed */
	ZSTD_CCtx_reset(cctx, ZSTD_reset_session_only);
	}
	FORWARD_IF_ERROR(flushMin, "ZSTDMT_compressStream_generic failed");
	} while (forceMaxProgress && flushMin != 0 && output->pos < output->size);
	DEBUGLOG(5, "completed ZSTD_compressStream2 delegating to ZSTDMT_compressStream_generic");
	/* Either we don't require maximum forward progress, we've finished the
	* flush, or we are out of output space.
	*/
	assert(!forceMaxProgress \|\| flushMin == 0 \|\| output->pos == output->size);
	return flushMin;
	}
	#endif
	FORWARD_IF_ERROR( ZSTD_compressStream_generic(cctx, output, input, endOp) , "");
	DEBUGLOG(5, "completed ZSTD_compressStream2");
	return cctx->outBuffContentSize - cctx->outBuffFlushedSize; /* remaining to flush */
	}

	size_t ZSTD_compressStream2_simpleArgs (
	ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity, size_t* dstPos,
	const void* src, size_t srcSize, size_t* srcPos,
	ZSTD_EndDirective endOp)
	{
	ZSTD_outBuffer output = { dst, dstCapacity, *dstPos };
	ZSTD_inBuffer input = { src, srcSize, *srcPos };
	/* ZSTD_compressStream2() will check validity of dstPos and srcPos */
	size_t const cErr = ZSTD_compressStream2(cctx, &output, &input, endOp);
	*dstPos = output.pos;
	*srcPos = input.pos;
	return cErr;
	}

	size_t ZSTD_compress2(ZSTD_CCtx* cctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize)
	{
	DEBUGLOG(4, "ZSTD_compress2 (srcSize=%u)", (unsigned)srcSize);
	ZSTD_CCtx_reset(cctx, ZSTD_reset_session_only);
	{ size_t oPos = 0;
	size_t iPos = 0;
	size_t const result = ZSTD_compressStream2_simpleArgs(cctx,
	dst, dstCapacity, &oPos,
	src, srcSize, &iPos,
	ZSTD_e_end);
	FORWARD_IF_ERROR(result, "ZSTD_compressStream2_simpleArgs failed");
	if (result != 0) { /* compression not completed, due to lack of output space */
	assert(oPos == dstCapacity);
	RETURN_ERROR(dstSize_tooSmall, "");
	}
	assert(iPos == srcSize); /* all input is expected consumed */
	return oPos;
	}
	}

	/====== Finalize ======/

	/*! ZSTD_flushStream() :
	* @return : amount of data remaining to flush */
	size_t ZSTD_flushStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output)
	{
	ZSTD_inBuffer input = { NULL, 0, 0 };
	return ZSTD_compressStream2(zcs, output, &input, ZSTD_e_flush);
	}


	size_t ZSTD_endStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output)
	{
	ZSTD_inBuffer input = { NULL, 0, 0 };
	size_t const remainingToFlush = ZSTD_compressStream2(zcs, output, &input, ZSTD_e_end);
	FORWARD_IF_ERROR( remainingToFlush , "ZSTD_compressStream2 failed");
	if (zcs->appliedParams.nbWorkers > 0) return remainingToFlush; /* minimal estimation */
	/* single thread mode : attempt to calculate remaining to flush more precisely */
	{ size_t const lastBlockSize = zcs->frameEnded ? 0 : ZSTD_BLOCKHEADERSIZE;
	size_t const checksumSize = (size_t)(zcs->frameEnded ? 0 : zcs->appliedParams.fParams.checksumFlag * 4);
	size_t const toFlush = remainingToFlush + lastBlockSize + checksumSize;
	DEBUGLOG(4, "ZSTD_endStream : remaining to flush : %u", (unsigned)toFlush);
	return toFlush;
	}
	}


	/-===== Pre-defined compression levels =====-/

	#define ZSTD_MAX_CLEVEL 22
	int ZSTD_maxCLevel(void) { return ZSTD_MAX_CLEVEL; }
	int ZSTD_minCLevel(void) { return (int)-ZSTD_TARGETLENGTH_MAX; }

	static const ZSTD_compressionParameters ZSTD_defaultCParameters[4][ZSTD_MAX_CLEVEL+1] = {
	{ /* "default" - for any srcSize > 256 KB */
	/* W, C, H, S, L, TL, strat */
	{ 19, 12, 13, 1, 6, 1, ZSTD_fast }, /* base for negative levels */
	{ 19, 13, 14, 1, 7, 0, ZSTD_fast }, /* level 1 */
	{ 20, 15, 16, 1, 6, 0, ZSTD_fast }, /* level 2 */
	{ 21, 16, 17, 1, 5, 0, ZSTD_dfast }, /* level 3 */
	{ 21, 18, 18, 1, 5, 0, ZSTD_dfast }, /* level 4 */
	{ 21, 18, 19, 2, 5, 2, ZSTD_greedy }, /* level 5 */
	{ 21, 19, 19, 3, 5, 4, ZSTD_greedy }, /* level 6 */
	{ 21, 19, 19, 3, 5, 8, ZSTD_lazy }, /* level 7 */
	{ 21, 19, 19, 3, 5, 16, ZSTD_lazy2 }, /* level 8 */
	{ 21, 19, 20, 4, 5, 16, ZSTD_lazy2 }, /* level 9 */
	{ 22, 20, 21, 4, 5, 16, ZSTD_lazy2 }, /* level 10 */
	{ 22, 21, 22, 4, 5, 16, ZSTD_lazy2 }, /* level 11 */
	{ 22, 21, 22, 5, 5, 16, ZSTD_lazy2 }, /* level 12 */
	{ 22, 21, 22, 5, 5, 32, ZSTD_btlazy2 }, /* level 13 */
	{ 22, 22, 23, 5, 5, 32, ZSTD_btlazy2 }, /* level 14 */
	{ 22, 23, 23, 6, 5, 32, ZSTD_btlazy2 }, /* level 15 */
	{ 22, 22, 22, 5, 5, 48, ZSTD_btopt }, /* level 16 */
	{ 23, 23, 22, 5, 4, 64, ZSTD_btopt }, /* level 17 */
	{ 23, 23, 22, 6, 3, 64, ZSTD_btultra }, /* level 18 */
	{ 23, 24, 22, 7, 3,256, ZSTD_btultra2}, /* level 19 */
	{ 25, 25, 23, 7, 3,256, ZSTD_btultra2}, /* level 20 */
	{ 26, 26, 24, 7, 3,512, ZSTD_btultra2}, /* level 21 */
	{ 27, 27, 25, 9, 3,999, ZSTD_btultra2}, /* level 22 */
	},
	{ /* for srcSize <= 256 KB */
	/* W, C, H, S, L, T, strat */
	{ 18, 12, 13, 1, 5, 1, ZSTD_fast }, /* base for negative levels */
	{ 18, 13, 14, 1, 6, 0, ZSTD_fast }, /* level 1 */
	{ 18, 14, 14, 1, 5, 0, ZSTD_dfast }, /* level 2 */
	{ 18, 16, 16, 1, 4, 0, ZSTD_dfast }, /* level 3 */
	{ 18, 16, 17, 2, 5, 2, ZSTD_greedy }, /* level 4.*/
	{ 18, 18, 18, 3, 5, 2, ZSTD_greedy }, /* level 5.*/
	{ 18, 18, 19, 3, 5, 4, ZSTD_lazy }, /* level 6.*/
	{ 18, 18, 19, 4, 4, 4, ZSTD_lazy }, /* level 7 */
	{ 18, 18, 19, 4, 4, 8, ZSTD_lazy2 }, /* level 8 */
	{ 18, 18, 19, 5, 4, 8, ZSTD_lazy2 }, /* level 9 */
	{ 18, 18, 19, 6, 4, 8, ZSTD_lazy2 }, /* level 10 */
	{ 18, 18, 19, 5, 4, 12, ZSTD_btlazy2 }, /* level 11.*/
	{ 18, 19, 19, 7, 4, 12, ZSTD_btlazy2 }, /* level 12.*/
	{ 18, 18, 19, 4, 4, 16, ZSTD_btopt }, /* level 13 */
	{ 18, 18, 19, 4, 3, 32, ZSTD_btopt }, /* level 14.*/
	{ 18, 18, 19, 6, 3,128, ZSTD_btopt }, /* level 15.*/
	{ 18, 19, 19, 6, 3,128, ZSTD_btultra }, /* level 16.*/
	{ 18, 19, 19, 8, 3,256, ZSTD_btultra }, /* level 17.*/
	{ 18, 19, 19, 6, 3,128, ZSTD_btultra2}, /* level 18.*/
	{ 18, 19, 19, 8, 3,256, ZSTD_btultra2}, /* level 19.*/
	{ 18, 19, 19, 10, 3,512, ZSTD_btultra2}, /* level 20.*/
	{ 18, 19, 19, 12, 3,512, ZSTD_btultra2}, /* level 21.*/
	{ 18, 19, 19, 13, 3,999, ZSTD_btultra2}, /* level 22.*/
	},
	{ /* for srcSize <= 128 KB */
	/* W, C, H, S, L, T, strat */
	{ 17, 12, 12, 1, 5, 1, ZSTD_fast }, /* base for negative levels */
	{ 17, 12, 13, 1, 6, 0, ZSTD_fast }, /* level 1 */
	{ 17, 13, 15, 1, 5, 0, ZSTD_fast }, /* level 2 */
	{ 17, 15, 16, 2, 5, 0, ZSTD_dfast }, /* level 3 */
	{ 17, 17, 17, 2, 4, 0, ZSTD_dfast }, /* level 4 */
	{ 17, 16, 17, 3, 4, 2, ZSTD_greedy }, /* level 5 */
	{ 17, 17, 17, 3, 4, 4, ZSTD_lazy }, /* level 6 */
	{ 17, 17, 17, 3, 4, 8, ZSTD_lazy2 }, /* level 7 */
	{ 17, 17, 17, 4, 4, 8, ZSTD_lazy2 }, /* level 8 */
	{ 17, 17, 17, 5, 4, 8, ZSTD_lazy2 }, /* level 9 */
	{ 17, 17, 17, 6, 4, 8, ZSTD_lazy2 }, /* level 10 */
	{ 17, 17, 17, 5, 4, 8, ZSTD_btlazy2 }, /* level 11 */
	{ 17, 18, 17, 7, 4, 12, ZSTD_btlazy2 }, /* level 12 */
	{ 17, 18, 17, 3, 4, 12, ZSTD_btopt }, /* level 13.*/
	{ 17, 18, 17, 4, 3, 32, ZSTD_btopt }, /* level 14.*/
	{ 17, 18, 17, 6, 3,256, ZSTD_btopt }, /* level 15.*/
	{ 17, 18, 17, 6, 3,128, ZSTD_btultra }, /* level 16.*/
	{ 17, 18, 17, 8, 3,256, ZSTD_btultra }, /* level 17.*/
	{ 17, 18, 17, 10, 3,512, ZSTD_btultra }, /* level 18.*/
	{ 17, 18, 17, 5, 3,256, ZSTD_btultra2}, /* level 19.*/
	{ 17, 18, 17, 7, 3,512, ZSTD_btultra2}, /* level 20.*/
	{ 17, 18, 17, 9, 3,512, ZSTD_btultra2}, /* level 21.*/
	{ 17, 18, 17, 11, 3,999, ZSTD_btultra2}, /* level 22.*/
	},
	{ /* for srcSize <= 16 KB */
	/* W, C, H, S, L, T, strat */
	{ 14, 12, 13, 1, 5, 1, ZSTD_fast }, /* base for negative levels */
	{ 14, 14, 15, 1, 5, 0, ZSTD_fast }, /* level 1 */
	{ 14, 14, 15, 1, 4, 0, ZSTD_fast }, /* level 2 */
	{ 14, 14, 15, 2, 4, 0, ZSTD_dfast }, /* level 3 */
	{ 14, 14, 14, 4, 4, 2, ZSTD_greedy }, /* level 4 */
	{ 14, 14, 14, 3, 4, 4, ZSTD_lazy }, /* level 5.*/
	{ 14, 14, 14, 4, 4, 8, ZSTD_lazy2 }, /* level 6 */
	{ 14, 14, 14, 6, 4, 8, ZSTD_lazy2 }, /* level 7 */
	{ 14, 14, 14, 8, 4, 8, ZSTD_lazy2 }, /* level 8.*/
	{ 14, 15, 14, 5, 4, 8, ZSTD_btlazy2 }, /* level 9.*/
	{ 14, 15, 14, 9, 4, 8, ZSTD_btlazy2 }, /* level 10.*/
	{ 14, 15, 14, 3, 4, 12, ZSTD_btopt }, /* level 11.*/
	{ 14, 15, 14, 4, 3, 24, ZSTD_btopt }, /* level 12.*/
	{ 14, 15, 14, 5, 3, 32, ZSTD_btultra }, /* level 13.*/
	{ 14, 15, 15, 6, 3, 64, ZSTD_btultra }, /* level 14.*/
	{ 14, 15, 15, 7, 3,256, ZSTD_btultra }, /* level 15.*/
	{ 14, 15, 15, 5, 3, 48, ZSTD_btultra2}, /* level 16.*/
	{ 14, 15, 15, 6, 3,128, ZSTD_btultra2}, /* level 17.*/
	{ 14, 15, 15, 7, 3,256, ZSTD_btultra2}, /* level 18.*/
	{ 14, 15, 15, 8, 3,256, ZSTD_btultra2}, /* level 19.*/
	{ 14, 15, 15, 8, 3,512, ZSTD_btultra2}, /* level 20.*/
	{ 14, 15, 15, 9, 3,512, ZSTD_btultra2}, /* level 21.*/
	{ 14, 15, 15, 10, 3,999, ZSTD_btultra2}, /* level 22.*/
	},
	};

	/*! ZSTD_getCParams_internal() :
	* @return ZSTD_compressionParameters structure for a selected compression level, srcSize and dictSize.
	* Note: srcSizeHint 0 means 0, use ZSTD_CONTENTSIZE_UNKNOWN for unknown.
	* Use dictSize == 0 for unknown or unused. */
	static ZSTD_compressionParameters ZSTD_getCParams_internal(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize)
	{
	int const unknown = srcSizeHint == ZSTD_CONTENTSIZE_UNKNOWN;
	size_t const addedSize = unknown && dictSize > 0 ? 500 : 0;
	U64 const rSize = unknown && dictSize == 0 ? ZSTD_CONTENTSIZE_UNKNOWN : srcSizeHint+dictSize+addedSize;
	U32 const tableID = (rSize <= 256 KB) + (rSize <= 128 KB) + (rSize <= 16 KB);
	int row = compressionLevel;
	DEBUGLOG(5, "ZSTD_getCParams_internal (cLevel=%i)", compressionLevel);
	if (compressionLevel == 0) row = ZSTD_CLEVEL_DEFAULT; /* 0 == default */
	if (compressionLevel < 0) row = 0; /* entry 0 is baseline for fast mode */
	if (compressionLevel > ZSTD_MAX_CLEVEL) row = ZSTD_MAX_CLEVEL;
	{ ZSTD_compressionParameters cp = ZSTD_defaultCParameters[tableID][row];
	if (compressionLevel < 0) cp.targetLength = (unsigned)(-compressionLevel); /* acceleration factor */
	/* refine parameters based on srcSize & dictSize */
	return ZSTD_adjustCParams_internal(cp, srcSizeHint, dictSize);
	}
	}

	/*! ZSTD_getCParams() :
	* @return ZSTD_compressionParameters structure for a selected compression level, srcSize and dictSize.
	* Size values are optional, provide 0 if not known or unused */
	ZSTD_compressionParameters ZSTD_getCParams(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize)
	{
	if (srcSizeHint == 0) srcSizeHint = ZSTD_CONTENTSIZE_UNKNOWN;
	return ZSTD_getCParams_internal(compressionLevel, srcSizeHint, dictSize);
	}

	/*! ZSTD_getParams() :
	* same idea as ZSTD_getCParams()
	* @return a `ZSTD_parameters` structure (instead of `ZSTD_compressionParameters`).
	* Fields of `ZSTD_frameParameters` are set to default values */
	static ZSTD_parameters ZSTD_getParams_internal(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize) {
	ZSTD_parameters params;
	ZSTD_compressionParameters const cParams = ZSTD_getCParams_internal(compressionLevel, srcSizeHint, dictSize);
	DEBUGLOG(5, "ZSTD_getParams (cLevel=%i)", compressionLevel);
	memset(&params, 0, sizeof(params));
	params.cParams = cParams;
	params.fParams.contentSizeFlag = 1;
	return params;
	}

	/*! ZSTD_getParams() :
	* same idea as ZSTD_getCParams()
	* @return a `ZSTD_parameters` structure (instead of `ZSTD_compressionParameters`).
	* Fields of `ZSTD_frameParameters` are set to default values */
	ZSTD_parameters ZSTD_getParams(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize) {
	if (srcSizeHint == 0) srcSizeHint = ZSTD_CONTENTSIZE_UNKNOWN;
	return ZSTD_getParams_internal(compressionLevel, srcSizeHint, dictSize);
	}
	/** ended inlining compress/zstd_compress.c **/
	/** start inlining compress/zstd_double_fast.c **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	/** skipping file: zstd_compress_internal.h **/
	/** skipping file: zstd_double_fast.h **/


	void ZSTD_fillDoubleHashTable(ZSTD_matchState_t* ms,
	void const* end, ZSTD_dictTableLoadMethod_e dtlm)
	{
	const ZSTD_compressionParameters* const cParams = &ms->cParams;
	U32* const hashLarge = ms->hashTable;
	U32 const hBitsL = cParams->hashLog;
	U32 const mls = cParams->minMatch;
	U32* const hashSmall = ms->chainTable;
	U32 const hBitsS = cParams->chainLog;
	const BYTE* const base = ms->window.base;
	const BYTE* ip = base + ms->nextToUpdate;
	const BYTE* const iend = ((const BYTE*)end) - HASH_READ_SIZE;
	const U32 fastHashFillStep = 3;

	/* Always insert every fastHashFillStep position into the hash tables.
	* Insert the other positions into the large hash table if their entry
	* is empty.
	*/
	for (; ip + fastHashFillStep - 1 <= iend; ip += fastHashFillStep) {
	U32 const current = (U32)(ip - base);
	U32 i;
	for (i = 0; i < fastHashFillStep; ++i) {
	size_t const smHash = ZSTD_hashPtr(ip + i, hBitsS, mls);
	size_t const lgHash = ZSTD_hashPtr(ip + i, hBitsL, 8);
	if (i == 0)
	hashSmall[smHash] = current + i;
	if (i == 0 \|\| hashLarge[lgHash] == 0)
	hashLarge[lgHash] = current + i;
	/* Only load extra positions for ZSTD_dtlm_full */
	if (dtlm == ZSTD_dtlm_fast)
	break;
	} }
	}


	FORCE_INLINE_TEMPLATE
	size_t ZSTD_compressBlock_doubleFast_generic(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize,
	U32 const mls /* template */, ZSTD_dictMode_e const dictMode)
	{
	ZSTD_compressionParameters const* cParams = &ms->cParams;
	U32* const hashLong = ms->hashTable;
	const U32 hBitsL = cParams->hashLog;
	U32* const hashSmall = ms->chainTable;
	const U32 hBitsS = cParams->chainLog;
	const BYTE* const base = ms->window.base;
	const BYTE* const istart = (const BYTE*)src;
	const BYTE* ip = istart;
	const BYTE* anchor = istart;
	const U32 endIndex = (U32)((size_t)(istart - base) + srcSize);
	/* presumes that, if there is a dictionary, it must be using Attach mode */
	const U32 prefixLowestIndex = ZSTD_getLowestPrefixIndex(ms, endIndex, cParams->windowLog);
	const BYTE* const prefixLowest = base + prefixLowestIndex;
	const BYTE* const iend = istart + srcSize;
	const BYTE* const ilimit = iend - HASH_READ_SIZE;
	U32 offset_1=rep[0], offset_2=rep[1];
	U32 offsetSaved = 0;

	const ZSTD_matchState_t* const dms = ms->dictMatchState;
	const ZSTD_compressionParameters* const dictCParams =
	dictMode == ZSTD_dictMatchState ?
	&dms->cParams : NULL;
	const U32* const dictHashLong = dictMode == ZSTD_dictMatchState ?
	dms->hashTable : NULL;
	const U32* const dictHashSmall = dictMode == ZSTD_dictMatchState ?
	dms->chainTable : NULL;
	const U32 dictStartIndex = dictMode == ZSTD_dictMatchState ?
	dms->window.dictLimit : 0;
	const BYTE* const dictBase = dictMode == ZSTD_dictMatchState ?
	dms->window.base : NULL;
	const BYTE* const dictStart = dictMode == ZSTD_dictMatchState ?
	dictBase + dictStartIndex : NULL;
	const BYTE* const dictEnd = dictMode == ZSTD_dictMatchState ?
	dms->window.nextSrc : NULL;
	const U32 dictIndexDelta = dictMode == ZSTD_dictMatchState ?
	prefixLowestIndex - (U32)(dictEnd - dictBase) :
	0;
	const U32 dictHBitsL = dictMode == ZSTD_dictMatchState ?
	dictCParams->hashLog : hBitsL;
	const U32 dictHBitsS = dictMode == ZSTD_dictMatchState ?
	dictCParams->chainLog : hBitsS;
	const U32 dictAndPrefixLength = (U32)((ip - prefixLowest) + (dictEnd - dictStart));

	DEBUGLOG(5, "ZSTD_compressBlock_doubleFast_generic");

	assert(dictMode == ZSTD_noDict \|\| dictMode == ZSTD_dictMatchState);

	/* if a dictionary is attached, it must be within window range */
	if (dictMode == ZSTD_dictMatchState) {
	assert(ms->window.dictLimit + (1U << cParams->windowLog) >= endIndex);
	}

	/* init */
	ip += (dictAndPrefixLength == 0);
	if (dictMode == ZSTD_noDict) {
	U32 const current = (U32)(ip - base);
	U32 const windowLow = ZSTD_getLowestPrefixIndex(ms, current, cParams->windowLog);
	U32 const maxRep = current - windowLow;
	if (offset_2 > maxRep) offsetSaved = offset_2, offset_2 = 0;
	if (offset_1 > maxRep) offsetSaved = offset_1, offset_1 = 0;
	}
	if (dictMode == ZSTD_dictMatchState) {
	/* dictMatchState repCode checks don't currently handle repCode == 0
	* disabling. */
	assert(offset_1 <= dictAndPrefixLength);
	assert(offset_2 <= dictAndPrefixLength);
	}

	/* Main Search Loop */
	while (ip < ilimit) { /* < instead of <=, because repcode check at (ip+1) */
	size_t mLength;
	U32 offset;
	size_t const h2 = ZSTD_hashPtr(ip, hBitsL, 8);
	size_t const h = ZSTD_hashPtr(ip, hBitsS, mls);
	size_t const dictHL = ZSTD_hashPtr(ip, dictHBitsL, 8);
	size_t const dictHS = ZSTD_hashPtr(ip, dictHBitsS, mls);
	U32 const current = (U32)(ip-base);
	U32 const matchIndexL = hashLong[h2];
	U32 matchIndexS = hashSmall[h];
	const BYTE* matchLong = base + matchIndexL;
	const BYTE* match = base + matchIndexS;
	const U32 repIndex = current + 1 - offset_1;
	const BYTE* repMatch = (dictMode == ZSTD_dictMatchState
	&& repIndex < prefixLowestIndex) ?
	dictBase + (repIndex - dictIndexDelta) :
	base + repIndex;
	hashLong[h2] = hashSmall[h] = current; /* update hash tables */

	/* check dictMatchState repcode */
	if (dictMode == ZSTD_dictMatchState
	&& ((U32)((prefixLowestIndex-1) - repIndex) >= 3 /* intentional underflow */)
	&& (MEM_read32(repMatch) == MEM_read32(ip+1)) ) {
	const BYTE* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend;
	mLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixLowest) + 4;
	ip++;
	ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, mLength-MINMATCH);
	goto _match_stored;
	}

	/* check noDict repcode */
	if ( dictMode == ZSTD_noDict
	&& ((offset_1 > 0) & (MEM_read32(ip+1-offset_1) == MEM_read32(ip+1)))) {
	mLength = ZSTD_count(ip+1+4, ip+1+4-offset_1, iend) + 4;
	ip++;
	ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, mLength-MINMATCH);
	goto _match_stored;
	}

	if (matchIndexL > prefixLowestIndex) {
	/* check prefix long match */
	if (MEM_read64(matchLong) == MEM_read64(ip)) {
	mLength = ZSTD_count(ip+8, matchLong+8, iend) + 8;
	offset = (U32)(ip-matchLong);
	while (((ip>anchor) & (matchLong>prefixLowest)) && (ip[-1] == matchLong[-1])) { ip--; matchLong--; mLength++; } /* catch up */
	goto _match_found;
	}
	} else if (dictMode == ZSTD_dictMatchState) {
	/* check dictMatchState long match */
	U32 const dictMatchIndexL = dictHashLong[dictHL];
	const BYTE* dictMatchL = dictBase + dictMatchIndexL;
	assert(dictMatchL < dictEnd);

	if (dictMatchL > dictStart && MEM_read64(dictMatchL) == MEM_read64(ip)) {
	mLength = ZSTD_count_2segments(ip+8, dictMatchL+8, iend, dictEnd, prefixLowest) + 8;
	offset = (U32)(current - dictMatchIndexL - dictIndexDelta);
	while (((ip>anchor) & (dictMatchL>dictStart)) && (ip[-1] == dictMatchL[-1])) { ip--; dictMatchL--; mLength++; } /* catch up */
	goto _match_found;
	} }

	if (matchIndexS > prefixLowestIndex) {
	/* check prefix short match */
	if (MEM_read32(match) == MEM_read32(ip)) {
	goto _search_next_long;
	}
	} else if (dictMode == ZSTD_dictMatchState) {
	/* check dictMatchState short match */
	U32 const dictMatchIndexS = dictHashSmall[dictHS];
	match = dictBase + dictMatchIndexS;
	matchIndexS = dictMatchIndexS + dictIndexDelta;

	if (match > dictStart && MEM_read32(match) == MEM_read32(ip)) {
	goto _search_next_long;
	} }

	ip += ((ip-anchor) >> kSearchStrength) + 1;
	#if defined(__aarch64__)
	PREFETCH_L1(ip+256);
	#endif
	continue;

	_search_next_long:

	{ size_t const hl3 = ZSTD_hashPtr(ip+1, hBitsL, 8);
	size_t const dictHLNext = ZSTD_hashPtr(ip+1, dictHBitsL, 8);
	U32 const matchIndexL3 = hashLong[hl3];
	const BYTE* matchL3 = base + matchIndexL3;
	hashLong[hl3] = current + 1;

	/* check prefix long +1 match */
	if (matchIndexL3 > prefixLowestIndex) {
	if (MEM_read64(matchL3) == MEM_read64(ip+1)) {
	mLength = ZSTD_count(ip+9, matchL3+8, iend) + 8;
	ip++;
	offset = (U32)(ip-matchL3);
	while (((ip>anchor) & (matchL3>prefixLowest)) && (ip[-1] == matchL3[-1])) { ip--; matchL3--; mLength++; } /* catch up */
	goto _match_found;
	}
	} else if (dictMode == ZSTD_dictMatchState) {
	/* check dict long +1 match */
	U32 const dictMatchIndexL3 = dictHashLong[dictHLNext];
	const BYTE* dictMatchL3 = dictBase + dictMatchIndexL3;
	assert(dictMatchL3 < dictEnd);
	if (dictMatchL3 > dictStart && MEM_read64(dictMatchL3) == MEM_read64(ip+1)) {
	mLength = ZSTD_count_2segments(ip+1+8, dictMatchL3+8, iend, dictEnd, prefixLowest) + 8;
	ip++;
	offset = (U32)(current + 1 - dictMatchIndexL3 - dictIndexDelta);
	while (((ip>anchor) & (dictMatchL3>dictStart)) && (ip[-1] == dictMatchL3[-1])) { ip--; dictMatchL3--; mLength++; } /* catch up */
	goto _match_found;
	} } }

	/* if no long +1 match, explore the short match we found */
	if (dictMode == ZSTD_dictMatchState && matchIndexS < prefixLowestIndex) {
	mLength = ZSTD_count_2segments(ip+4, match+4, iend, dictEnd, prefixLowest) + 4;
	offset = (U32)(current - matchIndexS);
	while (((ip>anchor) & (match>dictStart)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
	} else {
	mLength = ZSTD_count(ip+4, match+4, iend) + 4;
	offset = (U32)(ip - match);
	while (((ip>anchor) & (match>prefixLowest)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
	}

	/* fall-through */

	_match_found:
	offset_2 = offset_1;
	offset_1 = offset;

	ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);

	_match_stored:
	/* match found */
	ip += mLength;
	anchor = ip;

	if (ip <= ilimit) {
	/* Complementary insertion */
	/* done after iLimit test, as candidates could be > iend-8 */
	{ U32 const indexToInsert = current+2;
	hashLong[ZSTD_hashPtr(base+indexToInsert, hBitsL, 8)] = indexToInsert;
	hashLong[ZSTD_hashPtr(ip-2, hBitsL, 8)] = (U32)(ip-2-base);
	hashSmall[ZSTD_hashPtr(base+indexToInsert, hBitsS, mls)] = indexToInsert;
	hashSmall[ZSTD_hashPtr(ip-1, hBitsS, mls)] = (U32)(ip-1-base);
	}

	/* check immediate repcode */
	if (dictMode == ZSTD_dictMatchState) {
	while (ip <= ilimit) {
	U32 const current2 = (U32)(ip-base);
	U32 const repIndex2 = current2 - offset_2;
	const BYTE* repMatch2 = dictMode == ZSTD_dictMatchState
	&& repIndex2 < prefixLowestIndex ?
	dictBase + repIndex2 - dictIndexDelta :
	base + repIndex2;
	if ( ((U32)((prefixLowestIndex-1) - (U32)repIndex2) >= 3 /* intentional overflow */)
	&& (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
	const BYTE* const repEnd2 = repIndex2 < prefixLowestIndex ? dictEnd : iend;
	size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixLowest) + 4;
	U32 tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */
	ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, repLength2-MINMATCH);
	hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = current2;
	hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = current2;
	ip += repLength2;
	anchor = ip;
	continue;
	}
	break;
	} }

	if (dictMode == ZSTD_noDict) {
	while ( (ip <= ilimit)
	&& ( (offset_2>0)
	& (MEM_read32(ip) == MEM_read32(ip - offset_2)) )) {
	/* store sequence */
	size_t const rLength = ZSTD_count(ip+4, ip+4-offset_2, iend) + 4;
	U32 const tmpOff = offset_2; offset_2 = offset_1; offset_1 = tmpOff; /* swap offset_2 <=> offset_1 */
	hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = (U32)(ip-base);
	hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = (U32)(ip-base);
	ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, rLength-MINMATCH);
	ip += rLength;
	anchor = ip;
	continue; /* faster when present ... (?) */
	} } }
	} /* while (ip < ilimit) */

	/* save reps for next block */
	rep[0] = offset_1 ? offset_1 : offsetSaved;
	rep[1] = offset_2 ? offset_2 : offsetSaved;

	/* Return the last literals size */
	return (size_t)(iend - anchor);
	}


	size_t ZSTD_compressBlock_doubleFast(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize)
	{
	const U32 mls = ms->cParams.minMatch;
	switch(mls)
	{
	default: /* includes case 3 */
	case 4 :
	return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 4, ZSTD_noDict);
	case 5 :
	return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 5, ZSTD_noDict);
	case 6 :
	return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 6, ZSTD_noDict);
	case 7 :
	return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 7, ZSTD_noDict);
	}
	}


	size_t ZSTD_compressBlock_doubleFast_dictMatchState(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize)
	{
	const U32 mls = ms->cParams.minMatch;
	switch(mls)
	{
	default: /* includes case 3 */
	case 4 :
	return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 4, ZSTD_dictMatchState);
	case 5 :
	return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 5, ZSTD_dictMatchState);
	case 6 :
	return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 6, ZSTD_dictMatchState);
	case 7 :
	return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 7, ZSTD_dictMatchState);
	}
	}


	static size_t ZSTD_compressBlock_doubleFast_extDict_generic(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize,
	U32 const mls /* template */)
	{
	ZSTD_compressionParameters const* cParams = &ms->cParams;
	U32* const hashLong = ms->hashTable;
	U32 const hBitsL = cParams->hashLog;
	U32* const hashSmall = ms->chainTable;
	U32 const hBitsS = cParams->chainLog;
	const BYTE* const istart = (const BYTE*)src;
	const BYTE* ip = istart;
	const BYTE* anchor = istart;
	const BYTE* const iend = istart + srcSize;
	const BYTE* const ilimit = iend - 8;
	const BYTE* const base = ms->window.base;
	const U32 endIndex = (U32)((size_t)(istart - base) + srcSize);
	const U32 lowLimit = ZSTD_getLowestMatchIndex(ms, endIndex, cParams->windowLog);
	const U32 dictStartIndex = lowLimit;
	const U32 dictLimit = ms->window.dictLimit;
	const U32 prefixStartIndex = (dictLimit > lowLimit) ? dictLimit : lowLimit;
	const BYTE* const prefixStart = base + prefixStartIndex;
	const BYTE* const dictBase = ms->window.dictBase;
	const BYTE* const dictStart = dictBase + dictStartIndex;
	const BYTE* const dictEnd = dictBase + prefixStartIndex;
	U32 offset_1=rep[0], offset_2=rep[1];

	DEBUGLOG(5, "ZSTD_compressBlock_doubleFast_extDict_generic (srcSize=%zu)", srcSize);

	/* if extDict is invalidated due to maxDistance, switch to "regular" variant */
	if (prefixStartIndex == dictStartIndex)
	return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, mls, ZSTD_noDict);

	/* Search Loop */
	while (ip < ilimit) { /* < instead of <=, because (ip+1) */
	const size_t hSmall = ZSTD_hashPtr(ip, hBitsS, mls);
	const U32 matchIndex = hashSmall[hSmall];
	const BYTE* const matchBase = matchIndex < prefixStartIndex ? dictBase : base;
	const BYTE* match = matchBase + matchIndex;

	const size_t hLong = ZSTD_hashPtr(ip, hBitsL, 8);
	const U32 matchLongIndex = hashLong[hLong];
	const BYTE* const matchLongBase = matchLongIndex < prefixStartIndex ? dictBase : base;
	const BYTE* matchLong = matchLongBase + matchLongIndex;

	const U32 current = (U32)(ip-base);
	const U32 repIndex = current + 1 - offset_1; /* offset_1 expected <= current +1 */
	const BYTE* const repBase = repIndex < prefixStartIndex ? dictBase : base;
	const BYTE* const repMatch = repBase + repIndex;
	size_t mLength;
	hashSmall[hSmall] = hashLong[hLong] = current; /* update hash table */

	if ((((U32)((prefixStartIndex-1) - repIndex) >= 3) /* intentional underflow : ensure repIndex doesn't overlap dict + prefix */
	& (repIndex > dictStartIndex))
	&& (MEM_read32(repMatch) == MEM_read32(ip+1)) ) {
	const BYTE* repMatchEnd = repIndex < prefixStartIndex ? dictEnd : iend;
	mLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixStart) + 4;
	ip++;
	ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, mLength-MINMATCH);
	} else {
	if ((matchLongIndex > dictStartIndex) && (MEM_read64(matchLong) == MEM_read64(ip))) {
	const BYTE* const matchEnd = matchLongIndex < prefixStartIndex ? dictEnd : iend;
	const BYTE* const lowMatchPtr = matchLongIndex < prefixStartIndex ? dictStart : prefixStart;
	U32 offset;
	mLength = ZSTD_count_2segments(ip+8, matchLong+8, iend, matchEnd, prefixStart) + 8;
	offset = current - matchLongIndex;
	while (((ip>anchor) & (matchLong>lowMatchPtr)) && (ip[-1] == matchLong[-1])) { ip--; matchLong--; mLength++; } /* catch up */
	offset_2 = offset_1;
	offset_1 = offset;
	ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);

	} else if ((matchIndex > dictStartIndex) && (MEM_read32(match) == MEM_read32(ip))) {
	size_t const h3 = ZSTD_hashPtr(ip+1, hBitsL, 8);
	U32 const matchIndex3 = hashLong[h3];
	const BYTE* const match3Base = matchIndex3 < prefixStartIndex ? dictBase : base;
	const BYTE* match3 = match3Base + matchIndex3;
	U32 offset;
	hashLong[h3] = current + 1;
	if ( (matchIndex3 > dictStartIndex) && (MEM_read64(match3) == MEM_read64(ip+1)) ) {
	const BYTE* const matchEnd = matchIndex3 < prefixStartIndex ? dictEnd : iend;
	const BYTE* const lowMatchPtr = matchIndex3 < prefixStartIndex ? dictStart : prefixStart;
	mLength = ZSTD_count_2segments(ip+9, match3+8, iend, matchEnd, prefixStart) + 8;
	ip++;
	offset = current+1 - matchIndex3;
	while (((ip>anchor) & (match3>lowMatchPtr)) && (ip[-1] == match3[-1])) { ip--; match3--; mLength++; } /* catch up */
	} else {
	const BYTE* const matchEnd = matchIndex < prefixStartIndex ? dictEnd : iend;
	const BYTE* const lowMatchPtr = matchIndex < prefixStartIndex ? dictStart : prefixStart;
	mLength = ZSTD_count_2segments(ip+4, match+4, iend, matchEnd, prefixStart) + 4;
	offset = current - matchIndex;
	while (((ip>anchor) & (match>lowMatchPtr)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
	}
	offset_2 = offset_1;
	offset_1 = offset;
	ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);

	} else {
	ip += ((ip-anchor) >> kSearchStrength) + 1;
	continue;
	} }

	/* move to next sequence start */
	ip += mLength;
	anchor = ip;

	if (ip <= ilimit) {
	/* Complementary insertion */
	/* done after iLimit test, as candidates could be > iend-8 */
	{ U32 const indexToInsert = current+2;
	hashLong[ZSTD_hashPtr(base+indexToInsert, hBitsL, 8)] = indexToInsert;
	hashLong[ZSTD_hashPtr(ip-2, hBitsL, 8)] = (U32)(ip-2-base);
	hashSmall[ZSTD_hashPtr(base+indexToInsert, hBitsS, mls)] = indexToInsert;
	hashSmall[ZSTD_hashPtr(ip-1, hBitsS, mls)] = (U32)(ip-1-base);
	}

	/* check immediate repcode */
	while (ip <= ilimit) {
	U32 const current2 = (U32)(ip-base);
	U32 const repIndex2 = current2 - offset_2;
	const BYTE* repMatch2 = repIndex2 < prefixStartIndex ? dictBase + repIndex2 : base + repIndex2;
	if ( (((U32)((prefixStartIndex-1) - repIndex2) >= 3) /* intentional overflow : ensure repIndex2 doesn't overlap dict + prefix */
	& (repIndex2 > dictStartIndex))
	&& (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
	const BYTE* const repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend;
	size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixStart) + 4;
	U32 const tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */
	ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, repLength2-MINMATCH);
	hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = current2;
	hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = current2;
	ip += repLength2;
	anchor = ip;
	continue;
	}
	break;
	} } }

	/* save reps for next block */
	rep[0] = offset_1;
	rep[1] = offset_2;

	/* Return the last literals size */
	return (size_t)(iend - anchor);
	}


	size_t ZSTD_compressBlock_doubleFast_extDict(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize)
	{
	U32 const mls = ms->cParams.minMatch;
	switch(mls)
	{
	default: /* includes case 3 */
	case 4 :
	return ZSTD_compressBlock_doubleFast_extDict_generic(ms, seqStore, rep, src, srcSize, 4);
	case 5 :
	return ZSTD_compressBlock_doubleFast_extDict_generic(ms, seqStore, rep, src, srcSize, 5);
	case 6 :
	return ZSTD_compressBlock_doubleFast_extDict_generic(ms, seqStore, rep, src, srcSize, 6);
	case 7 :
	return ZSTD_compressBlock_doubleFast_extDict_generic(ms, seqStore, rep, src, srcSize, 7);
	}
	}
	/** ended inlining compress/zstd_double_fast.c **/
	/** start inlining compress/zstd_fast.c **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	/** skipping file: zstd_compress_internal.h **/
	/** skipping file: zstd_fast.h **/


	void ZSTD_fillHashTable(ZSTD_matchState_t* ms,
	const void* const end,
	ZSTD_dictTableLoadMethod_e dtlm)
	{
	const ZSTD_compressionParameters* const cParams = &ms->cParams;
	U32* const hashTable = ms->hashTable;
	U32 const hBits = cParams->hashLog;
	U32 const mls = cParams->minMatch;
	const BYTE* const base = ms->window.base;
	const BYTE* ip = base + ms->nextToUpdate;
	const BYTE* const iend = ((const BYTE*)end) - HASH_READ_SIZE;
	const U32 fastHashFillStep = 3;

	/* Always insert every fastHashFillStep position into the hash table.
	* Insert the other positions if their hash entry is empty.
	*/
	for ( ; ip + fastHashFillStep < iend + 2; ip += fastHashFillStep) {
	U32 const current = (U32)(ip - base);
	size_t const hash0 = ZSTD_hashPtr(ip, hBits, mls);
	hashTable[hash0] = current;
	if (dtlm == ZSTD_dtlm_fast) continue;
	/* Only load extra positions for ZSTD_dtlm_full */
	{ U32 p;
	for (p = 1; p < fastHashFillStep; ++p) {
	size_t const hash = ZSTD_hashPtr(ip + p, hBits, mls);
	if (hashTable[hash] == 0) { /* not yet filled */
	hashTable[hash] = current + p;
	} } } }
	}


	FORCE_INLINE_TEMPLATE size_t
	ZSTD_compressBlock_fast_generic(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize,
	U32 const mls)
	{
	const ZSTD_compressionParameters* const cParams = &ms->cParams;
	U32* const hashTable = ms->hashTable;
	U32 const hlog = cParams->hashLog;
	/* support stepSize of 0 */
	size_t const stepSize = cParams->targetLength + !(cParams->targetLength) + 1;
	const BYTE* const base = ms->window.base;
	const BYTE* const istart = (const BYTE*)src;
	/* We check ip0 (ip + 0) and ip1 (ip + 1) each loop */
	const BYTE* ip0 = istart;
	const BYTE* ip1;
	const BYTE* anchor = istart;
	const U32 endIndex = (U32)((size_t)(istart - base) + srcSize);
	const U32 prefixStartIndex = ZSTD_getLowestPrefixIndex(ms, endIndex, cParams->windowLog);
	const BYTE* const prefixStart = base + prefixStartIndex;
	const BYTE* const iend = istart + srcSize;
	const BYTE* const ilimit = iend - HASH_READ_SIZE;
	U32 offset_1=rep[0], offset_2=rep[1];
	U32 offsetSaved = 0;

	/* init */
	DEBUGLOG(5, "ZSTD_compressBlock_fast_generic");
	ip0 += (ip0 == prefixStart);
	ip1 = ip0 + 1;
	{ U32 const current = (U32)(ip0 - base);
	U32 const windowLow = ZSTD_getLowestPrefixIndex(ms, current, cParams->windowLog);
	U32 const maxRep = current - windowLow;
	if (offset_2 > maxRep) offsetSaved = offset_2, offset_2 = 0;
	if (offset_1 > maxRep) offsetSaved = offset_1, offset_1 = 0;
	}

	/* Main Search Loop */
	#ifdef __INTEL_COMPILER
	/* From intel 'The vector pragma indicates that the loop should be
	* vectorized if it is legal to do so'. Can be used together with
	* #pragma ivdep (but have opted to exclude that because intel
	* warns against using it).*/
	#pragma vector always
	#endif
	while (ip1 < ilimit) { /* < instead of <=, because check at ip0+2 */
	size_t mLength;
	BYTE const* ip2 = ip0 + 2;
	size_t const h0 = ZSTD_hashPtr(ip0, hlog, mls);
	U32 const val0 = MEM_read32(ip0);
	size_t const h1 = ZSTD_hashPtr(ip1, hlog, mls);
	U32 const val1 = MEM_read32(ip1);
	U32 const current0 = (U32)(ip0-base);
	U32 const current1 = (U32)(ip1-base);
	U32 const matchIndex0 = hashTable[h0];
	U32 const matchIndex1 = hashTable[h1];
	BYTE const* repMatch = ip2 - offset_1;
	const BYTE* match0 = base + matchIndex0;
	const BYTE* match1 = base + matchIndex1;
	U32 offcode;

	#if defined(__aarch64__)
	PREFETCH_L1(ip0+256);
	#endif

	hashTable[h0] = current0; /* update hash table */
	hashTable[h1] = current1; /* update hash table */

	assert(ip0 + 1 == ip1);

	if ((offset_1 > 0) & (MEM_read32(repMatch) == MEM_read32(ip2))) {
	mLength = (ip2[-1] == repMatch[-1]) ? 1 : 0;
	ip0 = ip2 - mLength;
	match0 = repMatch - mLength;
	mLength += 4;
	offcode = 0;
	goto _match;
	}
	if ((matchIndex0 > prefixStartIndex) && MEM_read32(match0) == val0) {
	/* found a regular match */
	goto _offset;
	}
	if ((matchIndex1 > prefixStartIndex) && MEM_read32(match1) == val1) {
	/* found a regular match after one literal */
	ip0 = ip1;
	match0 = match1;
	goto _offset;
	}
	{ size_t const step = ((size_t)(ip0-anchor) >> (kSearchStrength - 1)) + stepSize;
	assert(step >= 2);
	ip0 += step;
	ip1 += step;
	continue;
	}
	_offset: /* Requires: ip0, match0 */
	/* Compute the offset code */
	offset_2 = offset_1;
	offset_1 = (U32)(ip0-match0);
	offcode = offset_1 + ZSTD_REP_MOVE;
	mLength = 4;
	/* Count the backwards match length */
	while (((ip0>anchor) & (match0>prefixStart))
	&& (ip0[-1] == match0[-1])) { ip0--; match0--; mLength++; } /* catch up */

	_match: /* Requires: ip0, match0, offcode */
	/* Count the forward length */
	mLength += ZSTD_count(ip0+mLength, match0+mLength, iend);
	ZSTD_storeSeq(seqStore, (size_t)(ip0-anchor), anchor, iend, offcode, mLength-MINMATCH);
	/* match found */
	ip0 += mLength;
	anchor = ip0;

	if (ip0 <= ilimit) {
	/* Fill Table */
	assert(base+current0+2 > istart); /* check base overflow */
	hashTable[ZSTD_hashPtr(base+current0+2, hlog, mls)] = current0+2; /* here because current+2 could be > iend-8 */
	hashTable[ZSTD_hashPtr(ip0-2, hlog, mls)] = (U32)(ip0-2-base);

	if (offset_2 > 0) { /* offset_2==0 means offset_2 is invalidated */
	while ( (ip0 <= ilimit) && (MEM_read32(ip0) == MEM_read32(ip0 - offset_2)) ) {
	/* store sequence */
	size_t const rLength = ZSTD_count(ip0+4, ip0+4-offset_2, iend) + 4;
	{ U32 const tmpOff = offset_2; offset_2 = offset_1; offset_1 = tmpOff; } /* swap offset_2 <=> offset_1 */
	hashTable[ZSTD_hashPtr(ip0, hlog, mls)] = (U32)(ip0-base);
	ip0 += rLength;
	ZSTD_storeSeq(seqStore, 0 /litLen/, anchor, iend, 0 /offCode/, rLength-MINMATCH);
	anchor = ip0;
	continue; /* faster when present (confirmed on gcc-8) ... (?) */
	} } }
	ip1 = ip0 + 1;
	}

	/* save reps for next block */
	rep[0] = offset_1 ? offset_1 : offsetSaved;
	rep[1] = offset_2 ? offset_2 : offsetSaved;

	/* Return the last literals size */
	return (size_t)(iend - anchor);
	}


	size_t ZSTD_compressBlock_fast(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize)
	{
	U32 const mls = ms->cParams.minMatch;
	assert(ms->dictMatchState == NULL);
	switch(mls)
	{
	default: /* includes case 3 */
	case 4 :
	return ZSTD_compressBlock_fast_generic(ms, seqStore, rep, src, srcSize, 4);
	case 5 :
	return ZSTD_compressBlock_fast_generic(ms, seqStore, rep, src, srcSize, 5);
	case 6 :
	return ZSTD_compressBlock_fast_generic(ms, seqStore, rep, src, srcSize, 6);
	case 7 :
	return ZSTD_compressBlock_fast_generic(ms, seqStore, rep, src, srcSize, 7);
	}
	}

	FORCE_INLINE_TEMPLATE
	size_t ZSTD_compressBlock_fast_dictMatchState_generic(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize, U32 const mls)
	{
	const ZSTD_compressionParameters* const cParams = &ms->cParams;
	U32* const hashTable = ms->hashTable;
	U32 const hlog = cParams->hashLog;
	/* support stepSize of 0 */
	U32 const stepSize = cParams->targetLength + !(cParams->targetLength);
	const BYTE* const base = ms->window.base;
	const BYTE* const istart = (const BYTE*)src;
	const BYTE* ip = istart;
	const BYTE* anchor = istart;
	const U32 prefixStartIndex = ms->window.dictLimit;
	const BYTE* const prefixStart = base + prefixStartIndex;
	const BYTE* const iend = istart + srcSize;
	const BYTE* const ilimit = iend - HASH_READ_SIZE;
	U32 offset_1=rep[0], offset_2=rep[1];
	U32 offsetSaved = 0;

	const ZSTD_matchState_t* const dms = ms->dictMatchState;
	const ZSTD_compressionParameters* const dictCParams = &dms->cParams ;
	const U32* const dictHashTable = dms->hashTable;
	const U32 dictStartIndex = dms->window.dictLimit;
	const BYTE* const dictBase = dms->window.base;
	const BYTE* const dictStart = dictBase + dictStartIndex;
	const BYTE* const dictEnd = dms->window.nextSrc;
	const U32 dictIndexDelta = prefixStartIndex - (U32)(dictEnd - dictBase);
	const U32 dictAndPrefixLength = (U32)(ip - prefixStart + dictEnd - dictStart);
	const U32 dictHLog = dictCParams->hashLog;

	/* if a dictionary is still attached, it necessarily means that
	* it is within window size. So we just check it. */
	const U32 maxDistance = 1U << cParams->windowLog;
	const U32 endIndex = (U32)((size_t)(ip - base) + srcSize);
	assert(endIndex - prefixStartIndex <= maxDistance);
	(void)maxDistance; (void)endIndex; /* these variables are not used when assert() is disabled */

	/* ensure there will be no no underflow
	* when translating a dict index into a local index */
	assert(prefixStartIndex >= (U32)(dictEnd - dictBase));

	/* init */
	DEBUGLOG(5, "ZSTD_compressBlock_fast_dictMatchState_generic");
	ip += (dictAndPrefixLength == 0);
	/* dictMatchState repCode checks don't currently handle repCode == 0
	* disabling. */
	assert(offset_1 <= dictAndPrefixLength);
	assert(offset_2 <= dictAndPrefixLength);

	/* Main Search Loop */
	while (ip < ilimit) { /* < instead of <=, because repcode check at (ip+1) */
	size_t mLength;
	size_t const h = ZSTD_hashPtr(ip, hlog, mls);
	U32 const current = (U32)(ip-base);
	U32 const matchIndex = hashTable[h];
	const BYTE* match = base + matchIndex;
	const U32 repIndex = current + 1 - offset_1;
	const BYTE* repMatch = (repIndex < prefixStartIndex) ?
	dictBase + (repIndex - dictIndexDelta) :
	base + repIndex;
	hashTable[h] = current; /* update hash table */

	if ( ((U32)((prefixStartIndex-1) - repIndex) >= 3) /* intentional underflow : ensure repIndex isn't overlapping dict + prefix */
	&& (MEM_read32(repMatch) == MEM_read32(ip+1)) ) {
	const BYTE* const repMatchEnd = repIndex < prefixStartIndex ? dictEnd : iend;
	mLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixStart) + 4;
	ip++;
	ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, mLength-MINMATCH);
	} else if ( (matchIndex <= prefixStartIndex) ) {
	size_t const dictHash = ZSTD_hashPtr(ip, dictHLog, mls);
	U32 const dictMatchIndex = dictHashTable[dictHash];
	const BYTE* dictMatch = dictBase + dictMatchIndex;
	if (dictMatchIndex <= dictStartIndex \|\|
	MEM_read32(dictMatch) != MEM_read32(ip)) {
	assert(stepSize >= 1);
	ip += ((ip-anchor) >> kSearchStrength) + stepSize;
	continue;
	} else {
	/* found a dict match */
	U32 const offset = (U32)(current-dictMatchIndex-dictIndexDelta);
	mLength = ZSTD_count_2segments(ip+4, dictMatch+4, iend, dictEnd, prefixStart) + 4;
	while (((ip>anchor) & (dictMatch>dictStart))
	&& (ip[-1] == dictMatch[-1])) {
	ip--; dictMatch--; mLength++;
	} /* catch up */
	offset_2 = offset_1;
	offset_1 = offset;
	ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
	}
	} else if (MEM_read32(match) != MEM_read32(ip)) {
	/* it's not a match, and we're not going to check the dictionary */
	assert(stepSize >= 1);
	ip += ((ip-anchor) >> kSearchStrength) + stepSize;
	continue;
	} else {
	/* found a regular match */
	U32 const offset = (U32)(ip-match);
	mLength = ZSTD_count(ip+4, match+4, iend) + 4;
	while (((ip>anchor) & (match>prefixStart))
	&& (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
	offset_2 = offset_1;
	offset_1 = offset;
	ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
	}

	/* match found */
	ip += mLength;
	anchor = ip;

	if (ip <= ilimit) {
	/* Fill Table */
	assert(base+current+2 > istart); /* check base overflow */
	hashTable[ZSTD_hashPtr(base+current+2, hlog, mls)] = current+2; /* here because current+2 could be > iend-8 */
	hashTable[ZSTD_hashPtr(ip-2, hlog, mls)] = (U32)(ip-2-base);

	/* check immediate repcode */
	while (ip <= ilimit) {
	U32 const current2 = (U32)(ip-base);
	U32 const repIndex2 = current2 - offset_2;
	const BYTE* repMatch2 = repIndex2 < prefixStartIndex ?
	dictBase - dictIndexDelta + repIndex2 :
	base + repIndex2;
	if ( ((U32)((prefixStartIndex-1) - (U32)repIndex2) >= 3 /* intentional overflow */)
	&& (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
	const BYTE* const repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend;
	size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixStart) + 4;
	U32 tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */
	ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, repLength2-MINMATCH);
	hashTable[ZSTD_hashPtr(ip, hlog, mls)] = current2;
	ip += repLength2;
	anchor = ip;
	continue;
	}
	break;
	}
	}
	}

	/* save reps for next block */
	rep[0] = offset_1 ? offset_1 : offsetSaved;
	rep[1] = offset_2 ? offset_2 : offsetSaved;

	/* Return the last literals size */
	return (size_t)(iend - anchor);
	}

	size_t ZSTD_compressBlock_fast_dictMatchState(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize)
	{
	U32 const mls = ms->cParams.minMatch;
	assert(ms->dictMatchState != NULL);
	switch(mls)
	{
	default: /* includes case 3 */
	case 4 :
	return ZSTD_compressBlock_fast_dictMatchState_generic(ms, seqStore, rep, src, srcSize, 4);
	case 5 :
	return ZSTD_compressBlock_fast_dictMatchState_generic(ms, seqStore, rep, src, srcSize, 5);
	case 6 :
	return ZSTD_compressBlock_fast_dictMatchState_generic(ms, seqStore, rep, src, srcSize, 6);
	case 7 :
	return ZSTD_compressBlock_fast_dictMatchState_generic(ms, seqStore, rep, src, srcSize, 7);
	}
	}


	static size_t ZSTD_compressBlock_fast_extDict_generic(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize, U32 const mls)
	{
	const ZSTD_compressionParameters* const cParams = &ms->cParams;
	U32* const hashTable = ms->hashTable;
	U32 const hlog = cParams->hashLog;
	/* support stepSize of 0 */
	U32 const stepSize = cParams->targetLength + !(cParams->targetLength);
	const BYTE* const base = ms->window.base;
	const BYTE* const dictBase = ms->window.dictBase;
	const BYTE* const istart = (const BYTE*)src;
	const BYTE* ip = istart;
	const BYTE* anchor = istart;
	const U32 endIndex = (U32)((size_t)(istart - base) + srcSize);
	const U32 lowLimit = ZSTD_getLowestMatchIndex(ms, endIndex, cParams->windowLog);
	const U32 dictStartIndex = lowLimit;
	const BYTE* const dictStart = dictBase + dictStartIndex;
	const U32 dictLimit = ms->window.dictLimit;
	const U32 prefixStartIndex = dictLimit < lowLimit ? lowLimit : dictLimit;
	const BYTE* const prefixStart = base + prefixStartIndex;
	const BYTE* const dictEnd = dictBase + prefixStartIndex;
	const BYTE* const iend = istart + srcSize;
	const BYTE* const ilimit = iend - 8;
	U32 offset_1=rep[0], offset_2=rep[1];

	DEBUGLOG(5, "ZSTD_compressBlock_fast_extDict_generic (offset_1=%u)", offset_1);

	/* switch to "regular" variant if extDict is invalidated due to maxDistance */
	if (prefixStartIndex == dictStartIndex)
	return ZSTD_compressBlock_fast_generic(ms, seqStore, rep, src, srcSize, mls);

	/* Search Loop */
	while (ip < ilimit) { /* < instead of <=, because (ip+1) */
	const size_t h = ZSTD_hashPtr(ip, hlog, mls);
	const U32 matchIndex = hashTable[h];
	const BYTE* const matchBase = matchIndex < prefixStartIndex ? dictBase : base;
	const BYTE* match = matchBase + matchIndex;
	const U32 current = (U32)(ip-base);
	const U32 repIndex = current + 1 - offset_1;
	const BYTE* const repBase = repIndex < prefixStartIndex ? dictBase : base;
	const BYTE* const repMatch = repBase + repIndex;
	hashTable[h] = current; /* update hash table */
	DEBUGLOG(7, "offset_1 = %u , current = %u", offset_1, current);
	assert(offset_1 <= current +1); /* check repIndex */

	if ( (((U32)((prefixStartIndex-1) - repIndex) >= 3) /* intentional underflow */ & (repIndex > dictStartIndex))
	&& (MEM_read32(repMatch) == MEM_read32(ip+1)) ) {
	const BYTE* const repMatchEnd = repIndex < prefixStartIndex ? dictEnd : iend;
	size_t const rLength = ZSTD_count_2segments(ip+1 +4, repMatch +4, iend, repMatchEnd, prefixStart) + 4;
	ip++;
	ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, rLength-MINMATCH);
	ip += rLength;
	anchor = ip;
	} else {
	if ( (matchIndex < dictStartIndex) \|\|
	(MEM_read32(match) != MEM_read32(ip)) ) {
	assert(stepSize >= 1);
	ip += ((ip-anchor) >> kSearchStrength) + stepSize;
	continue;
	}
	{ const BYTE* const matchEnd = matchIndex < prefixStartIndex ? dictEnd : iend;
	const BYTE* const lowMatchPtr = matchIndex < prefixStartIndex ? dictStart : prefixStart;
	U32 const offset = current - matchIndex;
	size_t mLength = ZSTD_count_2segments(ip+4, match+4, iend, matchEnd, prefixStart) + 4;
	while (((ip>anchor) & (match>lowMatchPtr)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
	offset_2 = offset_1; offset_1 = offset; /* update offset history */
	ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
	ip += mLength;
	anchor = ip;
	} }

	if (ip <= ilimit) {
	/* Fill Table */
	hashTable[ZSTD_hashPtr(base+current+2, hlog, mls)] = current+2;
	hashTable[ZSTD_hashPtr(ip-2, hlog, mls)] = (U32)(ip-2-base);
	/* check immediate repcode */
	while (ip <= ilimit) {
	U32 const current2 = (U32)(ip-base);
	U32 const repIndex2 = current2 - offset_2;
	const BYTE* const repMatch2 = repIndex2 < prefixStartIndex ? dictBase + repIndex2 : base + repIndex2;
	if ( (((U32)((prefixStartIndex-1) - repIndex2) >= 3) & (repIndex2 > dictStartIndex)) /* intentional overflow */
	&& (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
	const BYTE* const repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend;
	size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixStart) + 4;
	{ U32 const tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; } /* swap offset_2 <=> offset_1 */
	ZSTD_storeSeq(seqStore, 0 /litlen/, anchor, iend, 0 /offcode/, repLength2-MINMATCH);
	hashTable[ZSTD_hashPtr(ip, hlog, mls)] = current2;
	ip += repLength2;
	anchor = ip;
	continue;
	}
	break;
	} } }

	/* save reps for next block */
	rep[0] = offset_1;
	rep[1] = offset_2;

	/* Return the last literals size */
	return (size_t)(iend - anchor);
	}


	size_t ZSTD_compressBlock_fast_extDict(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize)
	{
	U32 const mls = ms->cParams.minMatch;
	switch(mls)
	{
	default: /* includes case 3 */
	case 4 :
	return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 4);
	case 5 :
	return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 5);
	case 6 :
	return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 6);
	case 7 :
	return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 7);
	}
	}
	/** ended inlining compress/zstd_fast.c **/
	/** start inlining compress/zstd_lazy.c **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	/** skipping file: zstd_compress_internal.h **/
	/** skipping file: zstd_lazy.h **/


	/-************************************
	* Binary Tree search
	***************************************/

	static void
	ZSTD_updateDUBT(ZSTD_matchState_t* ms,
	const BYTE* ip, const BYTE* iend,
	U32 mls)
	{
	const ZSTD_compressionParameters* const cParams = &ms->cParams;
	U32* const hashTable = ms->hashTable;
	U32 const hashLog = cParams->hashLog;

	U32* const bt = ms->chainTable;
	U32 const btLog = cParams->chainLog - 1;
	U32 const btMask = (1 << btLog) - 1;

	const BYTE* const base = ms->window.base;
	U32 const target = (U32)(ip - base);
	U32 idx = ms->nextToUpdate;

	if (idx != target)
	DEBUGLOG(7, "ZSTD_updateDUBT, from %u to %u (dictLimit:%u)",
	idx, target, ms->window.dictLimit);
	assert(ip + 8 <= iend); /* condition for ZSTD_hashPtr */
	(void)iend;

	assert(idx >= ms->window.dictLimit); /* condition for valid base+idx */
	for ( ; idx < target ; idx++) {
	size_t const h = ZSTD_hashPtr(base + idx, hashLog, mls); /* assumption : ip + 8 <= iend */
	U32 const matchIndex = hashTable[h];

	U32* const nextCandidatePtr = bt + 2*(idx&btMask);
	U32* const sortMarkPtr = nextCandidatePtr + 1;

	DEBUGLOG(8, "ZSTD_updateDUBT: insert %u", idx);
	hashTable[h] = idx; /* Update Hash Table */
	nextCandidatePtr = matchIndex; / update BT like a chain */
	*sortMarkPtr = ZSTD_DUBT_UNSORTED_MARK;
	}
	ms->nextToUpdate = target;
	}


	/** ZSTD_insertDUBT1() :
	* sort one already inserted but unsorted position
	* assumption : current >= btlow == (current - btmask)
	* doesn't fail */
	static void
	ZSTD_insertDUBT1(ZSTD_matchState_t* ms,
	U32 current, const BYTE* inputEnd,
	U32 nbCompares, U32 btLow,
	const ZSTD_dictMode_e dictMode)
	{
	const ZSTD_compressionParameters* const cParams = &ms->cParams;
	U32* const bt = ms->chainTable;
	U32 const btLog = cParams->chainLog - 1;
	U32 const btMask = (1 << btLog) - 1;
	size_t commonLengthSmaller=0, commonLengthLarger=0;
	const BYTE* const base = ms->window.base;
	const BYTE* const dictBase = ms->window.dictBase;
	const U32 dictLimit = ms->window.dictLimit;
	const BYTE* const ip = (current>=dictLimit) ? base + current : dictBase + current;
	const BYTE* const iend = (current>=dictLimit) ? inputEnd : dictBase + dictLimit;
	const BYTE* const dictEnd = dictBase + dictLimit;
	const BYTE* const prefixStart = base + dictLimit;
	const BYTE* match;
	U32* smallerPtr = bt + 2*(current&btMask);
	U32* largerPtr = smallerPtr + 1;
	U32 matchIndex = smallerPtr; / this candidate is unsorted : next sorted candidate is reached through smallerPtr, while largerPtr contains previous unsorted candidate (which is already saved and can be overwritten) */
	U32 dummy32; /* to be nullified at the end */
	U32 const windowValid = ms->window.lowLimit;
	U32 const maxDistance = 1U << cParams->windowLog;
	U32 const windowLow = (current - windowValid > maxDistance) ? current - maxDistance : windowValid;


	DEBUGLOG(8, "ZSTD_insertDUBT1(%u) (dictLimit=%u, lowLimit=%u)",
	current, dictLimit, windowLow);
	assert(current >= btLow);
	assert(ip < iend); /* condition for ZSTD_count */

	while (nbCompares-- && (matchIndex > windowLow)) {
	U32* const nextPtr = bt + 2*(matchIndex & btMask);
	size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */
	assert(matchIndex < current);
	/* note : all candidates are now supposed sorted,
	* but it's still possible to have nextPtr[1] == ZSTD_DUBT_UNSORTED_MARK
	* when a real index has the same value as ZSTD_DUBT_UNSORTED_MARK */

	if ( (dictMode != ZSTD_extDict)
	\|\| (matchIndex+matchLength >= dictLimit) /* both in current segment*/
	\|\| (current < dictLimit) /* both in extDict */) {
	const BYTE* const mBase = ( (dictMode != ZSTD_extDict)
	\|\| (matchIndex+matchLength >= dictLimit)) ?
	base : dictBase;
	assert( (matchIndex+matchLength >= dictLimit) /* might be wrong if extDict is incorrectly set to 0 */
	\|\| (current < dictLimit) );
	match = mBase + matchIndex;
	matchLength += ZSTD_count(ip+matchLength, match+matchLength, iend);
	} else {
	match = dictBase + matchIndex;
	matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iend, dictEnd, prefixStart);
	if (matchIndex+matchLength >= dictLimit)
	match = base + matchIndex; /* preparation for next read of match[matchLength] */
	}

	DEBUGLOG(8, "ZSTD_insertDUBT1: comparing %u with %u : found %u common bytes ",
	current, matchIndex, (U32)matchLength);

	if (ip+matchLength == iend) { /* equal : no way to know if inf or sup */
	break; /* drop , to guarantee consistency ; miss a bit of compression, but other solutions can corrupt tree */
	}

	if (match[matchLength] < ip[matchLength]) { /* necessarily within buffer */
	/* match is smaller than current */
	smallerPtr = matchIndex; / update smaller idx */
	commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */
	if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop searching */
	DEBUGLOG(8, "ZSTD_insertDUBT1: %u (>btLow=%u) is smaller : next => %u",
	matchIndex, btLow, nextPtr[1]);
	smallerPtr = nextPtr+1; /* new "candidate" => larger than match, which was smaller than target */
	matchIndex = nextPtr[1]; /* new matchIndex, larger than previous and closer to current */
	} else {
	/* match is larger than current */
	*largerPtr = matchIndex;
	commonLengthLarger = matchLength;
	if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop searching */
	DEBUGLOG(8, "ZSTD_insertDUBT1: %u (>btLow=%u) is larger => %u",
	matchIndex, btLow, nextPtr[0]);
	largerPtr = nextPtr;
	matchIndex = nextPtr[0];
	} }

	smallerPtr = largerPtr = 0;
	}


	static size_t
	ZSTD_DUBT_findBetterDictMatch (
	ZSTD_matchState_t* ms,
	const BYTE* const ip, const BYTE* const iend,
	size_t* offsetPtr,
	size_t bestLength,
	U32 nbCompares,
	U32 const mls,
	const ZSTD_dictMode_e dictMode)
	{
	const ZSTD_matchState_t * const dms = ms->dictMatchState;
	const ZSTD_compressionParameters* const dmsCParams = &dms->cParams;
	const U32 * const dictHashTable = dms->hashTable;
	U32 const hashLog = dmsCParams->hashLog;
	size_t const h = ZSTD_hashPtr(ip, hashLog, mls);
	U32 dictMatchIndex = dictHashTable[h];

	const BYTE* const base = ms->window.base;
	const BYTE* const prefixStart = base + ms->window.dictLimit;
	U32 const current = (U32)(ip-base);
	const BYTE* const dictBase = dms->window.base;
	const BYTE* const dictEnd = dms->window.nextSrc;
	U32 const dictHighLimit = (U32)(dms->window.nextSrc - dms->window.base);
	U32 const dictLowLimit = dms->window.lowLimit;
	U32 const dictIndexDelta = ms->window.lowLimit - dictHighLimit;

	U32* const dictBt = dms->chainTable;
	U32 const btLog = dmsCParams->chainLog - 1;
	U32 const btMask = (1 << btLog) - 1;
	U32 const btLow = (btMask >= dictHighLimit - dictLowLimit) ? dictLowLimit : dictHighLimit - btMask;

	size_t commonLengthSmaller=0, commonLengthLarger=0;

	(void)dictMode;
	assert(dictMode == ZSTD_dictMatchState);

	while (nbCompares-- && (dictMatchIndex > dictLowLimit)) {
	U32* const nextPtr = dictBt + 2*(dictMatchIndex & btMask);
	size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */
	const BYTE* match = dictBase + dictMatchIndex;
	matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iend, dictEnd, prefixStart);
	if (dictMatchIndex+matchLength >= dictHighLimit)
	match = base + dictMatchIndex + dictIndexDelta; /* to prepare for next usage of match[matchLength] */

	if (matchLength > bestLength) {
	U32 matchIndex = dictMatchIndex + dictIndexDelta;
	if ( (4*(int)(matchLength-bestLength)) > (int)(ZSTD_highbit32(current-matchIndex+1) - ZSTD_highbit32((U32)offsetPtr[0]+1)) ) {
	DEBUGLOG(9, "ZSTD_DUBT_findBetterDictMatch(%u) : found better match length %u -> %u and offsetCode %u -> %u (dictMatchIndex %u, matchIndex %u)",
	current, (U32)bestLength, (U32)matchLength, (U32)*offsetPtr, ZSTD_REP_MOVE + current - matchIndex, dictMatchIndex, matchIndex);
	bestLength = matchLength, *offsetPtr = ZSTD_REP_MOVE + current - matchIndex;
	}
	if (ip+matchLength == iend) { /* reached end of input : ip[matchLength] is not valid, no way to know if it's larger or smaller than match */
	break; /* drop, to guarantee consistency (miss a little bit of compression) */
	}
	}

	if (match[matchLength] < ip[matchLength]) {
	if (dictMatchIndex <= btLow) { break; } /* beyond tree size, stop the search */
	commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */
	dictMatchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to current) */
	} else {
	/* match is larger than current */
	if (dictMatchIndex <= btLow) { break; } /* beyond tree size, stop the search */
	commonLengthLarger = matchLength;
	dictMatchIndex = nextPtr[0];
	}
	}

	if (bestLength >= MINMATCH) {
	U32 const mIndex = current - ((U32)*offsetPtr - ZSTD_REP_MOVE); (void)mIndex;
	DEBUGLOG(8, "ZSTD_DUBT_findBetterDictMatch(%u) : found match of length %u and offsetCode %u (pos %u)",
	current, (U32)bestLength, (U32)*offsetPtr, mIndex);
	}
	return bestLength;

	}


	static size_t
	ZSTD_DUBT_findBestMatch(ZSTD_matchState_t* ms,
	const BYTE* const ip, const BYTE* const iend,
	size_t* offsetPtr,
	U32 const mls,
	const ZSTD_dictMode_e dictMode)
	{
	const ZSTD_compressionParameters* const cParams = &ms->cParams;
	U32* const hashTable = ms->hashTable;
	U32 const hashLog = cParams->hashLog;
	size_t const h = ZSTD_hashPtr(ip, hashLog, mls);
	U32 matchIndex = hashTable[h];

	const BYTE* const base = ms->window.base;
	U32 const current = (U32)(ip-base);
	U32 const windowLow = ZSTD_getLowestMatchIndex(ms, current, cParams->windowLog);

	U32* const bt = ms->chainTable;
	U32 const btLog = cParams->chainLog - 1;
	U32 const btMask = (1 << btLog) - 1;
	U32 const btLow = (btMask >= current) ? 0 : current - btMask;
	U32 const unsortLimit = MAX(btLow, windowLow);

	U32* nextCandidate = bt + 2*(matchIndex&btMask);
	U32* unsortedMark = bt + 2*(matchIndex&btMask) + 1;
	U32 nbCompares = 1U << cParams->searchLog;
	U32 nbCandidates = nbCompares;
	U32 previousCandidate = 0;

	DEBUGLOG(7, "ZSTD_DUBT_findBestMatch (%u) ", current);
	assert(ip <= iend-8); /* required for h calculation */

	/* reach end of unsorted candidates list */
	while ( (matchIndex > unsortLimit)
	&& (*unsortedMark == ZSTD_DUBT_UNSORTED_MARK)
	&& (nbCandidates > 1) ) {
	DEBUGLOG(8, "ZSTD_DUBT_findBestMatch: candidate %u is unsorted",
	matchIndex);
	unsortedMark = previousCandidate; / the unsortedMark becomes a reversed chain, to move up back to original position */
	previousCandidate = matchIndex;
	matchIndex = *nextCandidate;
	nextCandidate = bt + 2*(matchIndex&btMask);
	unsortedMark = bt + 2*(matchIndex&btMask) + 1;
	nbCandidates --;
	}

	/* nullify last candidate if it's still unsorted
	* simplification, detrimental to compression ratio, beneficial for speed */
	if ( (matchIndex > unsortLimit)
	&& (*unsortedMark==ZSTD_DUBT_UNSORTED_MARK) ) {
	DEBUGLOG(7, "ZSTD_DUBT_findBestMatch: nullify last unsorted candidate %u",
	matchIndex);
	nextCandidate = unsortedMark = 0;
	}

	/* batch sort stacked candidates */
	matchIndex = previousCandidate;
	while (matchIndex) { /* will end on matchIndex == 0 */
	U32* const nextCandidateIdxPtr = bt + 2*(matchIndex&btMask) + 1;
	U32 const nextCandidateIdx = *nextCandidateIdxPtr;
	ZSTD_insertDUBT1(ms, matchIndex, iend,
	nbCandidates, unsortLimit, dictMode);
	matchIndex = nextCandidateIdx;
	nbCandidates++;
	}

	/* find longest match */
	{ size_t commonLengthSmaller = 0, commonLengthLarger = 0;
	const BYTE* const dictBase = ms->window.dictBase;
	const U32 dictLimit = ms->window.dictLimit;
	const BYTE* const dictEnd = dictBase + dictLimit;
	const BYTE* const prefixStart = base + dictLimit;
	U32* smallerPtr = bt + 2*(current&btMask);
	U32* largerPtr = bt + 2*(current&btMask) + 1;
	U32 matchEndIdx = current + 8 + 1;
	U32 dummy32; /* to be nullified at the end */
	size_t bestLength = 0;

	matchIndex = hashTable[h];
	hashTable[h] = current; /* Update Hash Table */

	while (nbCompares-- && (matchIndex > windowLow)) {
	U32* const nextPtr = bt + 2*(matchIndex & btMask);
	size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */
	const BYTE* match;

	if ((dictMode != ZSTD_extDict) \|\| (matchIndex+matchLength >= dictLimit)) {
	match = base + matchIndex;
	matchLength += ZSTD_count(ip+matchLength, match+matchLength, iend);
	} else {
	match = dictBase + matchIndex;
	matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iend, dictEnd, prefixStart);
	if (matchIndex+matchLength >= dictLimit)
	match = base + matchIndex; /* to prepare for next usage of match[matchLength] */
	}

	if (matchLength > bestLength) {
	if (matchLength > matchEndIdx - matchIndex)
	matchEndIdx = matchIndex + (U32)matchLength;
	if ( (4*(int)(matchLength-bestLength)) > (int)(ZSTD_highbit32(current-matchIndex+1) - ZSTD_highbit32((U32)offsetPtr[0]+1)) )
	bestLength = matchLength, *offsetPtr = ZSTD_REP_MOVE + current - matchIndex;
	if (ip+matchLength == iend) { /* equal : no way to know if inf or sup */
	if (dictMode == ZSTD_dictMatchState) {
	nbCompares = 0; /* in addition to avoiding checking any
	* further in this loop, make sure we
	* skip checking in the dictionary. */
	}
	break; /* drop, to guarantee consistency (miss a little bit of compression) */
	}
	}

	if (match[matchLength] < ip[matchLength]) {
	/* match is smaller than current */
	smallerPtr = matchIndex; / update smaller idx */
	commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */
	if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop the search */
	smallerPtr = nextPtr+1; /* new "smaller" => larger of match */
	matchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to current) */
	} else {
	/* match is larger than current */
	*largerPtr = matchIndex;
	commonLengthLarger = matchLength;
	if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop the search */
	largerPtr = nextPtr;
	matchIndex = nextPtr[0];
	} }

	smallerPtr = largerPtr = 0;

	if (dictMode == ZSTD_dictMatchState && nbCompares) {
	bestLength = ZSTD_DUBT_findBetterDictMatch(
	ms, ip, iend,
	offsetPtr, bestLength, nbCompares,
	mls, dictMode);
	}

	assert(matchEndIdx > current+8); /* ensure nextToUpdate is increased */
	ms->nextToUpdate = matchEndIdx - 8; /* skip repetitive patterns */
	if (bestLength >= MINMATCH) {
	U32 const mIndex = current - ((U32)*offsetPtr - ZSTD_REP_MOVE); (void)mIndex;
	DEBUGLOG(8, "ZSTD_DUBT_findBestMatch(%u) : found match of length %u and offsetCode %u (pos %u)",
	current, (U32)bestLength, (U32)*offsetPtr, mIndex);
	}
	return bestLength;
	}
	}


	/** ZSTD_BtFindBestMatch() : Tree updater, providing best match */
	FORCE_INLINE_TEMPLATE size_t
	ZSTD_BtFindBestMatch( ZSTD_matchState_t* ms,
	const BYTE* const ip, const BYTE* const iLimit,
	size_t* offsetPtr,
	const U32 mls /* template */,
	const ZSTD_dictMode_e dictMode)
	{
	DEBUGLOG(7, "ZSTD_BtFindBestMatch");
	if (ip < ms->window.base + ms->nextToUpdate) return 0; /* skipped area */
	ZSTD_updateDUBT(ms, ip, iLimit, mls);
	return ZSTD_DUBT_findBestMatch(ms, ip, iLimit, offsetPtr, mls, dictMode);
	}


	static size_t
	ZSTD_BtFindBestMatch_selectMLS ( ZSTD_matchState_t* ms,
	const BYTE* ip, const BYTE* const iLimit,
	size_t* offsetPtr)
	{
	switch(ms->cParams.minMatch)
	{
	default : /* includes case 3 */
	case 4 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 4, ZSTD_noDict);
	case 5 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 5, ZSTD_noDict);
	case 7 :
	case 6 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 6, ZSTD_noDict);
	}
	}


	static size_t ZSTD_BtFindBestMatch_dictMatchState_selectMLS (
	ZSTD_matchState_t* ms,
	const BYTE* ip, const BYTE* const iLimit,
	size_t* offsetPtr)
	{
	switch(ms->cParams.minMatch)
	{
	default : /* includes case 3 */
	case 4 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 4, ZSTD_dictMatchState);
	case 5 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 5, ZSTD_dictMatchState);
	case 7 :
	case 6 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 6, ZSTD_dictMatchState);
	}
	}


	static size_t ZSTD_BtFindBestMatch_extDict_selectMLS (
	ZSTD_matchState_t* ms,
	const BYTE* ip, const BYTE* const iLimit,
	size_t* offsetPtr)
	{
	switch(ms->cParams.minMatch)
	{
	default : /* includes case 3 */
	case 4 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 4, ZSTD_extDict);
	case 5 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 5, ZSTD_extDict);
	case 7 :
	case 6 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 6, ZSTD_extDict);
	}
	}



	/* *********************************
	* Hash Chain
	***********************************/
	#define NEXT_IN_CHAIN(d, mask) chainTable[(d) & (mask)]

	/* Update chains up to ip (excluded)
	Assumption : always within prefix (i.e. not within extDict) */
	static U32 ZSTD_insertAndFindFirstIndex_internal(
	ZSTD_matchState_t* ms,
	const ZSTD_compressionParameters* const cParams,
	const BYTE* ip, U32 const mls)
	{
	U32* const hashTable = ms->hashTable;
	const U32 hashLog = cParams->hashLog;
	U32* const chainTable = ms->chainTable;
	const U32 chainMask = (1 << cParams->chainLog) - 1;
	const BYTE* const base = ms->window.base;
	const U32 target = (U32)(ip - base);
	U32 idx = ms->nextToUpdate;

	while(idx < target) { /* catch up */
	size_t const h = ZSTD_hashPtr(base+idx, hashLog, mls);
	NEXT_IN_CHAIN(idx, chainMask) = hashTable[h];
	hashTable[h] = idx;
	idx++;
	}

	ms->nextToUpdate = target;
	return hashTable[ZSTD_hashPtr(ip, hashLog, mls)];
	}

	U32 ZSTD_insertAndFindFirstIndex(ZSTD_matchState_t* ms, const BYTE* ip) {
	const ZSTD_compressionParameters* const cParams = &ms->cParams;
	return ZSTD_insertAndFindFirstIndex_internal(ms, cParams, ip, ms->cParams.minMatch);
	}


	/* inlining is important to hardwire a hot branch (template emulation) */
	FORCE_INLINE_TEMPLATE
	size_t ZSTD_HcFindBestMatch_generic (
	ZSTD_matchState_t* ms,
	const BYTE* const ip, const BYTE* const iLimit,
	size_t* offsetPtr,
	const U32 mls, const ZSTD_dictMode_e dictMode)
	{
	const ZSTD_compressionParameters* const cParams = &ms->cParams;
	U32* const chainTable = ms->chainTable;
	const U32 chainSize = (1 << cParams->chainLog);
	const U32 chainMask = chainSize-1;
	const BYTE* const base = ms->window.base;
	const BYTE* const dictBase = ms->window.dictBase;
	const U32 dictLimit = ms->window.dictLimit;
	const BYTE* const prefixStart = base + dictLimit;
	const BYTE* const dictEnd = dictBase + dictLimit;
	const U32 current = (U32)(ip-base);
	const U32 maxDistance = 1U << cParams->windowLog;
	const U32 lowestValid = ms->window.lowLimit;
	const U32 withinMaxDistance = (current - lowestValid > maxDistance) ? current - maxDistance : lowestValid;
	const U32 isDictionary = (ms->loadedDictEnd != 0);
	const U32 lowLimit = isDictionary ? lowestValid : withinMaxDistance;
	const U32 minChain = current > chainSize ? current - chainSize : 0;
	U32 nbAttempts = 1U << cParams->searchLog;
	size_t ml=4-1;

	/* HC4 match finder */
	U32 matchIndex = ZSTD_insertAndFindFirstIndex_internal(ms, cParams, ip, mls);

	for ( ; (matchIndex>lowLimit) & (nbAttempts>0) ; nbAttempts--) {
	size_t currentMl=0;
	if ((dictMode != ZSTD_extDict) \|\| matchIndex >= dictLimit) {
	const BYTE* const match = base + matchIndex;
	assert(matchIndex >= dictLimit); /* ensures this is true if dictMode != ZSTD_extDict */
	if (match[ml] == ip[ml]) /* potentially better */
	currentMl = ZSTD_count(ip, match, iLimit);
	} else {
	const BYTE* const match = dictBase + matchIndex;
	assert(match+4 <= dictEnd);
	if (MEM_read32(match) == MEM_read32(ip)) /* assumption : matchIndex <= dictLimit-4 (by table construction) */
	currentMl = ZSTD_count_2segments(ip+4, match+4, iLimit, dictEnd, prefixStart) + 4;
	}

	/* save best solution */
	if (currentMl > ml) {
	ml = currentMl;
	*offsetPtr = current - matchIndex + ZSTD_REP_MOVE;
	if (ip+currentMl == iLimit) break; /* best possible, avoids read overflow on next attempt */
	}

	if (matchIndex <= minChain) break;
	matchIndex = NEXT_IN_CHAIN(matchIndex, chainMask);
	}

	if (dictMode == ZSTD_dictMatchState) {
	const ZSTD_matchState_t* const dms = ms->dictMatchState;
	const U32* const dmsChainTable = dms->chainTable;
	const U32 dmsChainSize = (1 << dms->cParams.chainLog);
	const U32 dmsChainMask = dmsChainSize - 1;
	const U32 dmsLowestIndex = dms->window.dictLimit;
	const BYTE* const dmsBase = dms->window.base;
	const BYTE* const dmsEnd = dms->window.nextSrc;
	const U32 dmsSize = (U32)(dmsEnd - dmsBase);
	const U32 dmsIndexDelta = dictLimit - dmsSize;
	const U32 dmsMinChain = dmsSize > dmsChainSize ? dmsSize - dmsChainSize : 0;

	matchIndex = dms->hashTable[ZSTD_hashPtr(ip, dms->cParams.hashLog, mls)];

	for ( ; (matchIndex>dmsLowestIndex) & (nbAttempts>0) ; nbAttempts--) {
	size_t currentMl=0;
	const BYTE* const match = dmsBase + matchIndex;
	assert(match+4 <= dmsEnd);
	if (MEM_read32(match) == MEM_read32(ip)) /* assumption : matchIndex <= dictLimit-4 (by table construction) */
	currentMl = ZSTD_count_2segments(ip+4, match+4, iLimit, dmsEnd, prefixStart) + 4;

	/* save best solution */
	if (currentMl > ml) {
	ml = currentMl;
	*offsetPtr = current - (matchIndex + dmsIndexDelta) + ZSTD_REP_MOVE;
	if (ip+currentMl == iLimit) break; /* best possible, avoids read overflow on next attempt */
	}

	if (matchIndex <= dmsMinChain) break;
	matchIndex = dmsChainTable[matchIndex & dmsChainMask];
	}
	}

	return ml;
	}


	FORCE_INLINE_TEMPLATE size_t ZSTD_HcFindBestMatch_selectMLS (
	ZSTD_matchState_t* ms,
	const BYTE* ip, const BYTE* const iLimit,
	size_t* offsetPtr)
	{
	switch(ms->cParams.minMatch)
	{
	default : /* includes case 3 */
	case 4 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 4, ZSTD_noDict);
	case 5 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 5, ZSTD_noDict);
	case 7 :
	case 6 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 6, ZSTD_noDict);
	}
	}


	static size_t ZSTD_HcFindBestMatch_dictMatchState_selectMLS (
	ZSTD_matchState_t* ms,
	const BYTE* ip, const BYTE* const iLimit,
	size_t* offsetPtr)
	{
	switch(ms->cParams.minMatch)
	{
	default : /* includes case 3 */
	case 4 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 4, ZSTD_dictMatchState);
	case 5 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 5, ZSTD_dictMatchState);
	case 7 :
	case 6 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 6, ZSTD_dictMatchState);
	}
	}


	FORCE_INLINE_TEMPLATE size_t ZSTD_HcFindBestMatch_extDict_selectMLS (
	ZSTD_matchState_t* ms,
	const BYTE* ip, const BYTE* const iLimit,
	size_t* offsetPtr)
	{
	switch(ms->cParams.minMatch)
	{
	default : /* includes case 3 */
	case 4 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 4, ZSTD_extDict);
	case 5 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 5, ZSTD_extDict);
	case 7 :
	case 6 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 6, ZSTD_extDict);
	}
	}


	/* *******************************
	* Common parser - lazy strategy
	*********************************/
	typedef enum { search_hashChain, search_binaryTree } searchMethod_e;

	FORCE_INLINE_TEMPLATE size_t
	ZSTD_compressBlock_lazy_generic(
	ZSTD_matchState_t* ms, seqStore_t* seqStore,
	U32 rep[ZSTD_REP_NUM],
	const void* src, size_t srcSize,
	const searchMethod_e searchMethod, const U32 depth,
	ZSTD_dictMode_e const dictMode)
	{
	const BYTE* const istart = (const BYTE*)src;
	const BYTE* ip = istart;
	const BYTE* anchor = istart;
	const BYTE* const iend = istart + srcSize;
	const BYTE* const ilimit = iend - 8;
	const BYTE* const base = ms->window.base;
	const U32 prefixLowestIndex = ms->window.dictLimit;
	const BYTE* const prefixLowest = base + prefixLowestIndex;

	typedef size_t (*searchMax_f)(
	ZSTD_matchState_t* ms,
	const BYTE* ip, const BYTE* iLimit, size_t* offsetPtr);
	searchMax_f const searchMax = dictMode == ZSTD_dictMatchState ?
	(searchMethod==search_binaryTree ? ZSTD_BtFindBestMatch_dictMatchState_selectMLS
	: ZSTD_HcFindBestMatch_dictMatchState_selectMLS) :
	(searchMethod==search_binaryTree ? ZSTD_BtFindBestMatch_selectMLS
	: ZSTD_HcFindBestMatch_selectMLS);
	U32 offset_1 = rep[0], offset_2 = rep[1], savedOffset=0;

	const ZSTD_matchState_t* const dms = ms->dictMatchState;
	const U32 dictLowestIndex = dictMode == ZSTD_dictMatchState ?
	dms->window.dictLimit : 0;
	const BYTE* const dictBase = dictMode == ZSTD_dictMatchState ?
	dms->window.base : NULL;
	const BYTE* const dictLowest = dictMode == ZSTD_dictMatchState ?
	dictBase + dictLowestIndex : NULL;
	const BYTE* const dictEnd = dictMode == ZSTD_dictMatchState ?
	dms->window.nextSrc : NULL;
	const U32 dictIndexDelta = dictMode == ZSTD_dictMatchState ?
	prefixLowestIndex - (U32)(dictEnd - dictBase) :
	0;
	const U32 dictAndPrefixLength = (U32)((ip - prefixLowest) + (dictEnd - dictLowest));

	DEBUGLOG(5, "ZSTD_compressBlock_lazy_generic (dictMode=%u)", (U32)dictMode);

	/* init */
	ip += (dictAndPrefixLength == 0);
	if (dictMode == ZSTD_noDict) {
	U32 const current = (U32)(ip - base);
	U32 const windowLow = ZSTD_getLowestPrefixIndex(ms, current, ms->cParams.windowLog);
	U32 const maxRep = current - windowLow;
	if (offset_2 > maxRep) savedOffset = offset_2, offset_2 = 0;
	if (offset_1 > maxRep) savedOffset = offset_1, offset_1 = 0;
	}
	if (dictMode == ZSTD_dictMatchState) {
	/* dictMatchState repCode checks don't currently handle repCode == 0
	* disabling. */
	assert(offset_1 <= dictAndPrefixLength);
	assert(offset_2 <= dictAndPrefixLength);
	}

	/* Match Loop */
	#if defined(__GNUC__) && defined(__x86_64__)
	/* I've measured random a 5% speed loss on levels 5 & 6 (greedy) when the
	* code alignment is perturbed. To fix the instability align the loop on 32-bytes.
	*/
	__asm__(".p2align 5");
	#endif
	while (ip < ilimit) {
	size_t matchLength=0;
	size_t offset=0;
	const BYTE* start=ip+1;

	/* check repCode */
	if (dictMode == ZSTD_dictMatchState) {
	const U32 repIndex = (U32)(ip - base) + 1 - offset_1;
	const BYTE* repMatch = (dictMode == ZSTD_dictMatchState
	&& repIndex < prefixLowestIndex) ?
	dictBase + (repIndex - dictIndexDelta) :
	base + repIndex;
	if (((U32)((prefixLowestIndex-1) - repIndex) >= 3 /* intentional underflow */)
	&& (MEM_read32(repMatch) == MEM_read32(ip+1)) ) {
	const BYTE* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend;
	matchLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixLowest) + 4;
	if (depth==0) goto _storeSequence;
	}
	}
	if ( dictMode == ZSTD_noDict
	&& ((offset_1 > 0) & (MEM_read32(ip+1-offset_1) == MEM_read32(ip+1)))) {
	matchLength = ZSTD_count(ip+1+4, ip+1+4-offset_1, iend) + 4;
	if (depth==0) goto _storeSequence;
	}

	/* first search (depth 0) */
	{ size_t offsetFound = 999999999;
	size_t const ml2 = searchMax(ms, ip, iend, &offsetFound);
	if (ml2 > matchLength)
	matchLength = ml2, start = ip, offset=offsetFound;
	}

	if (matchLength < 4) {
	ip += ((ip-anchor) >> kSearchStrength) + 1; /* jump faster over incompressible sections */
	continue;
	}

	/* let's try to find a better solution */
	if (depth>=1)
	while (ip<ilimit) {
	ip ++;
	if ( (dictMode == ZSTD_noDict)
	&& (offset) && ((offset_1>0) & (MEM_read32(ip) == MEM_read32(ip - offset_1)))) {
	size_t const mlRep = ZSTD_count(ip+4, ip+4-offset_1, iend) + 4;
	int const gain2 = (int)(mlRep * 3);
	int const gain1 = (int)(matchLength*3 - ZSTD_highbit32((U32)offset+1) + 1);
	if ((mlRep >= 4) && (gain2 > gain1))
	matchLength = mlRep, offset = 0, start = ip;
	}
	if (dictMode == ZSTD_dictMatchState) {
	const U32 repIndex = (U32)(ip - base) - offset_1;
	const BYTE* repMatch = repIndex < prefixLowestIndex ?
	dictBase + (repIndex - dictIndexDelta) :
	base + repIndex;
	if (((U32)((prefixLowestIndex-1) - repIndex) >= 3 /* intentional underflow */)
	&& (MEM_read32(repMatch) == MEM_read32(ip)) ) {
	const BYTE* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend;
	size_t const mlRep = ZSTD_count_2segments(ip+4, repMatch+4, iend, repMatchEnd, prefixLowest) + 4;
	int const gain2 = (int)(mlRep * 3);
	int const gain1 = (int)(matchLength*3 - ZSTD_highbit32((U32)offset+1) + 1);
	if ((mlRep >= 4) && (gain2 > gain1))
	matchLength = mlRep, offset = 0, start = ip;
	}
	}
	{ size_t offset2=999999999;
	size_t const ml2 = searchMax(ms, ip, iend, &offset2);
	int const gain2 = (int)(ml24 - ZSTD_highbit32((U32)offset2+1)); / raw approx */
	int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 4);
	if ((ml2 >= 4) && (gain2 > gain1)) {
	matchLength = ml2, offset = offset2, start = ip;
	continue; /* search a better one */
	} }

	/* let's find an even better one */
	if ((depth==2) && (ip<ilimit)) {
	ip ++;
	if ( (dictMode == ZSTD_noDict)
	&& (offset) && ((offset_1>0) & (MEM_read32(ip) == MEM_read32(ip - offset_1)))) {
	size_t const mlRep = ZSTD_count(ip+4, ip+4-offset_1, iend) + 4;
	int const gain2 = (int)(mlRep * 4);
	int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 1);
	if ((mlRep >= 4) && (gain2 > gain1))
	matchLength = mlRep, offset = 0, start = ip;
	}
	if (dictMode == ZSTD_dictMatchState) {
	const U32 repIndex = (U32)(ip - base) - offset_1;
	const BYTE* repMatch = repIndex < prefixLowestIndex ?
	dictBase + (repIndex - dictIndexDelta) :
	base + repIndex;
	if (((U32)((prefixLowestIndex-1) - repIndex) >= 3 /* intentional underflow */)
	&& (MEM_read32(repMatch) == MEM_read32(ip)) ) {
	const BYTE* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend;
	size_t const mlRep = ZSTD_count_2segments(ip+4, repMatch+4, iend, repMatchEnd, prefixLowest) + 4;
	int const gain2 = (int)(mlRep * 4);
	int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 1);
	if ((mlRep >= 4) && (gain2 > gain1))
	matchLength = mlRep, offset = 0, start = ip;
	}
	}
	{ size_t offset2=999999999;
	size_t const ml2 = searchMax(ms, ip, iend, &offset2);
	int const gain2 = (int)(ml24 - ZSTD_highbit32((U32)offset2+1)); / raw approx */
	int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 7);
	if ((ml2 >= 4) && (gain2 > gain1)) {
	matchLength = ml2, offset = offset2, start = ip;
	continue;
	} } }
	break; /* nothing found : store previous solution */
	}

	/* NOTE:
	* start[-offset+ZSTD_REP_MOVE-1] is undefined behavior.
	* (-offset+ZSTD_REP_MOVE-1) is unsigned, and is added to start, which
	* overflows the pointer, which is undefined behavior.
	*/
	/* catch up */
	if (offset) {
	if (dictMode == ZSTD_noDict) {
	while ( ((start > anchor) & (start - (offset-ZSTD_REP_MOVE) > prefixLowest))
	&& (start[-1] == (start-(offset-ZSTD_REP_MOVE))[-1]) ) /* only search for offset within prefix */
	{ start--; matchLength++; }
	}
	if (dictMode == ZSTD_dictMatchState) {
	U32 const matchIndex = (U32)((start-base) - (offset - ZSTD_REP_MOVE));
	const BYTE* match = (matchIndex < prefixLowestIndex) ? dictBase + matchIndex - dictIndexDelta : base + matchIndex;
	const BYTE* const mStart = (matchIndex < prefixLowestIndex) ? dictLowest : prefixLowest;
	while ((start>anchor) && (match>mStart) && (start[-1] == match[-1])) { start--; match--; matchLength++; } /* catch up */
	}
	offset_2 = offset_1; offset_1 = (U32)(offset - ZSTD_REP_MOVE);
	}
	/* store sequence */
	_storeSequence:
	{ size_t const litLength = start - anchor;
	ZSTD_storeSeq(seqStore, litLength, anchor, iend, (U32)offset, matchLength-MINMATCH);
	anchor = ip = start + matchLength;
	}

	/* check immediate repcode */
	if (dictMode == ZSTD_dictMatchState) {
	while (ip <= ilimit) {
	U32 const current2 = (U32)(ip-base);
	U32 const repIndex = current2 - offset_2;
	const BYTE* repMatch = dictMode == ZSTD_dictMatchState
	&& repIndex < prefixLowestIndex ?
	dictBase - dictIndexDelta + repIndex :
	base + repIndex;
	if ( ((U32)((prefixLowestIndex-1) - (U32)repIndex) >= 3 /* intentional overflow */)
	&& (MEM_read32(repMatch) == MEM_read32(ip)) ) {
	const BYTE* const repEnd2 = repIndex < prefixLowestIndex ? dictEnd : iend;
	matchLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd2, prefixLowest) + 4;
	offset = offset_2; offset_2 = offset_1; offset_1 = (U32)offset; /* swap offset_2 <=> offset_1 */
	ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, matchLength-MINMATCH);
	ip += matchLength;
	anchor = ip;
	continue;
	}
	break;
	}
	}

	if (dictMode == ZSTD_noDict) {
	while ( ((ip <= ilimit) & (offset_2>0))
	&& (MEM_read32(ip) == MEM_read32(ip - offset_2)) ) {
	/* store sequence */
	matchLength = ZSTD_count(ip+4, ip+4-offset_2, iend) + 4;
	offset = offset_2; offset_2 = offset_1; offset_1 = (U32)offset; /* swap repcodes */
	ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, matchLength-MINMATCH);
	ip += matchLength;
	anchor = ip;
	continue; /* faster when present ... (?) */
	} } }

	/* Save reps for next block */
	rep[0] = offset_1 ? offset_1 : savedOffset;
	rep[1] = offset_2 ? offset_2 : savedOffset;

	/* Return the last literals size */
	return (size_t)(iend - anchor);
	}


	size_t ZSTD_compressBlock_btlazy2(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize)
	{
	return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_binaryTree, 2, ZSTD_noDict);
	}

	size_t ZSTD_compressBlock_lazy2(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize)
	{
	return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 2, ZSTD_noDict);
	}

	size_t ZSTD_compressBlock_lazy(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize)
	{
	return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 1, ZSTD_noDict);
	}

	size_t ZSTD_compressBlock_greedy(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize)
	{
	return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 0, ZSTD_noDict);
	}

	size_t ZSTD_compressBlock_btlazy2_dictMatchState(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize)
	{
	return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_binaryTree, 2, ZSTD_dictMatchState);
	}

	size_t ZSTD_compressBlock_lazy2_dictMatchState(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize)
	{
	return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 2, ZSTD_dictMatchState);
	}

	size_t ZSTD_compressBlock_lazy_dictMatchState(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize)
	{
	return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 1, ZSTD_dictMatchState);
	}

	size_t ZSTD_compressBlock_greedy_dictMatchState(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize)
	{
	return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 0, ZSTD_dictMatchState);
	}


	FORCE_INLINE_TEMPLATE
	size_t ZSTD_compressBlock_lazy_extDict_generic(
	ZSTD_matchState_t* ms, seqStore_t* seqStore,
	U32 rep[ZSTD_REP_NUM],
	const void* src, size_t srcSize,
	const searchMethod_e searchMethod, const U32 depth)
	{
	const BYTE* const istart = (const BYTE*)src;
	const BYTE* ip = istart;
	const BYTE* anchor = istart;
	const BYTE* const iend = istart + srcSize;
	const BYTE* const ilimit = iend - 8;
	const BYTE* const base = ms->window.base;
	const U32 dictLimit = ms->window.dictLimit;
	const BYTE* const prefixStart = base + dictLimit;
	const BYTE* const dictBase = ms->window.dictBase;
	const BYTE* const dictEnd = dictBase + dictLimit;
	const BYTE* const dictStart = dictBase + ms->window.lowLimit;
	const U32 windowLog = ms->cParams.windowLog;

	typedef size_t (*searchMax_f)(
	ZSTD_matchState_t* ms,
	const BYTE* ip, const BYTE* iLimit, size_t* offsetPtr);
	searchMax_f searchMax = searchMethod==search_binaryTree ? ZSTD_BtFindBestMatch_extDict_selectMLS : ZSTD_HcFindBestMatch_extDict_selectMLS;

	U32 offset_1 = rep[0], offset_2 = rep[1];

	DEBUGLOG(5, "ZSTD_compressBlock_lazy_extDict_generic");

	/* init */
	ip += (ip == prefixStart);

	/* Match Loop */
	#if defined(__GNUC__) && defined(__x86_64__)
	/* I've measured random a 5% speed loss on levels 5 & 6 (greedy) when the
	* code alignment is perturbed. To fix the instability align the loop on 32-bytes.
	*/
	__asm__(".p2align 5");
	#endif
	while (ip < ilimit) {
	size_t matchLength=0;
	size_t offset=0;
	const BYTE* start=ip+1;
	U32 current = (U32)(ip-base);

	/* check repCode */
	{ const U32 windowLow = ZSTD_getLowestMatchIndex(ms, current+1, windowLog);
	const U32 repIndex = (U32)(current+1 - offset_1);
	const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
	const BYTE* const repMatch = repBase + repIndex;
	if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > windowLow)) /* intentional overflow */
	if (MEM_read32(ip+1) == MEM_read32(repMatch)) {
	/* repcode detected we should take it */
	const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
	matchLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repEnd, prefixStart) + 4;
	if (depth==0) goto _storeSequence;
	} }

	/* first search (depth 0) */
	{ size_t offsetFound = 999999999;
	size_t const ml2 = searchMax(ms, ip, iend, &offsetFound);
	if (ml2 > matchLength)
	matchLength = ml2, start = ip, offset=offsetFound;
	}

	if (matchLength < 4) {
	ip += ((ip-anchor) >> kSearchStrength) + 1; /* jump faster over incompressible sections */
	continue;
	}

	/* let's try to find a better solution */
	if (depth>=1)
	while (ip<ilimit) {
	ip ++;
	current++;
	/* check repCode */
	if (offset) {
	const U32 windowLow = ZSTD_getLowestMatchIndex(ms, current, windowLog);
	const U32 repIndex = (U32)(current - offset_1);
	const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
	const BYTE* const repMatch = repBase + repIndex;
	if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > windowLow)) /* intentional overflow */
	if (MEM_read32(ip) == MEM_read32(repMatch)) {
	/* repcode detected */
	const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
	size_t const repLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd, prefixStart) + 4;
	int const gain2 = (int)(repLength * 3);
	int const gain1 = (int)(matchLength*3 - ZSTD_highbit32((U32)offset+1) + 1);
	if ((repLength >= 4) && (gain2 > gain1))
	matchLength = repLength, offset = 0, start = ip;
	} }

	/* search match, depth 1 */
	{ size_t offset2=999999999;
	size_t const ml2 = searchMax(ms, ip, iend, &offset2);
	int const gain2 = (int)(ml24 - ZSTD_highbit32((U32)offset2+1)); / raw approx */
	int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 4);
	if ((ml2 >= 4) && (gain2 > gain1)) {
	matchLength = ml2, offset = offset2, start = ip;
	continue; /* search a better one */
	} }

	/* let's find an even better one */
	if ((depth==2) && (ip<ilimit)) {
	ip ++;
	current++;
	/* check repCode */
	if (offset) {
	const U32 windowLow = ZSTD_getLowestMatchIndex(ms, current, windowLog);
	const U32 repIndex = (U32)(current - offset_1);
	const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
	const BYTE* const repMatch = repBase + repIndex;
	if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > windowLow)) /* intentional overflow */
	if (MEM_read32(ip) == MEM_read32(repMatch)) {
	/* repcode detected */
	const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
	size_t const repLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd, prefixStart) + 4;
	int const gain2 = (int)(repLength * 4);
	int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 1);
	if ((repLength >= 4) && (gain2 > gain1))
	matchLength = repLength, offset = 0, start = ip;
	} }

	/* search match, depth 2 */
	{ size_t offset2=999999999;
	size_t const ml2 = searchMax(ms, ip, iend, &offset2);
	int const gain2 = (int)(ml24 - ZSTD_highbit32((U32)offset2+1)); / raw approx */
	int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 7);
	if ((ml2 >= 4) && (gain2 > gain1)) {
	matchLength = ml2, offset = offset2, start = ip;
	continue;
	} } }
	break; /* nothing found : store previous solution */
	}

	/* catch up */
	if (offset) {
	U32 const matchIndex = (U32)((start-base) - (offset - ZSTD_REP_MOVE));
	const BYTE* match = (matchIndex < dictLimit) ? dictBase + matchIndex : base + matchIndex;
	const BYTE* const mStart = (matchIndex < dictLimit) ? dictStart : prefixStart;
	while ((start>anchor) && (match>mStart) && (start[-1] == match[-1])) { start--; match--; matchLength++; } /* catch up */
	offset_2 = offset_1; offset_1 = (U32)(offset - ZSTD_REP_MOVE);
	}

	/* store sequence */
	_storeSequence:
	{ size_t const litLength = start - anchor;
	ZSTD_storeSeq(seqStore, litLength, anchor, iend, (U32)offset, matchLength-MINMATCH);
	anchor = ip = start + matchLength;
	}

	/* check immediate repcode */
	while (ip <= ilimit) {
	const U32 repCurrent = (U32)(ip-base);
	const U32 windowLow = ZSTD_getLowestMatchIndex(ms, repCurrent, windowLog);
	const U32 repIndex = repCurrent - offset_2;
	const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
	const BYTE* const repMatch = repBase + repIndex;
	if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > windowLow)) /* intentional overflow */
	if (MEM_read32(ip) == MEM_read32(repMatch)) {
	/* repcode detected we should take it */
	const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
	matchLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd, prefixStart) + 4;
	offset = offset_2; offset_2 = offset_1; offset_1 = (U32)offset; /* swap offset history */
	ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, matchLength-MINMATCH);
	ip += matchLength;
	anchor = ip;
	continue; /* faster when present ... (?) */
	}
	break;
	} }

	/* Save reps for next block */
	rep[0] = offset_1;
	rep[1] = offset_2;

	/* Return the last literals size */
	return (size_t)(iend - anchor);
	}


	size_t ZSTD_compressBlock_greedy_extDict(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize)
	{
	return ZSTD_compressBlock_lazy_extDict_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 0);
	}

	size_t ZSTD_compressBlock_lazy_extDict(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize)

	{
	return ZSTD_compressBlock_lazy_extDict_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 1);
	}

	size_t ZSTD_compressBlock_lazy2_extDict(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize)

	{
	return ZSTD_compressBlock_lazy_extDict_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 2);
	}

	size_t ZSTD_compressBlock_btlazy2_extDict(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize)

	{
	return ZSTD_compressBlock_lazy_extDict_generic(ms, seqStore, rep, src, srcSize, search_binaryTree, 2);
	}
	/** ended inlining compress/zstd_lazy.c **/
	/** start inlining compress/zstd_ldm.c **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	/** skipping file: zstd_ldm.h **/

	/** skipping file: ../common/debug.h **/
	/** skipping file: zstd_fast.h **/
	/** skipping file: zstd_double_fast.h **/

	#define LDM_BUCKET_SIZE_LOG 3
	#define LDM_MIN_MATCH_LENGTH 64
	#define LDM_HASH_RLOG 7
	#define LDM_HASH_CHAR_OFFSET 10

	void ZSTD_ldm_adjustParameters(ldmParams_t* params,
	ZSTD_compressionParameters const* cParams)
	{
	params->windowLog = cParams->windowLog;
	ZSTD_STATIC_ASSERT(LDM_BUCKET_SIZE_LOG <= ZSTD_LDM_BUCKETSIZELOG_MAX);
	DEBUGLOG(4, "ZSTD_ldm_adjustParameters");
	if (!params->bucketSizeLog) params->bucketSizeLog = LDM_BUCKET_SIZE_LOG;
	if (!params->minMatchLength) params->minMatchLength = LDM_MIN_MATCH_LENGTH;
	if (cParams->strategy >= ZSTD_btopt) {
	/* Get out of the way of the optimal parser */
	U32 const minMatch = MAX(cParams->targetLength, params->minMatchLength);
	assert(minMatch >= ZSTD_LDM_MINMATCH_MIN);
	assert(minMatch <= ZSTD_LDM_MINMATCH_MAX);
	params->minMatchLength = minMatch;
	}
	if (params->hashLog == 0) {
	params->hashLog = MAX(ZSTD_HASHLOG_MIN, params->windowLog - LDM_HASH_RLOG);
	assert(params->hashLog <= ZSTD_HASHLOG_MAX);
	}
	if (params->hashRateLog == 0) {
	params->hashRateLog = params->windowLog < params->hashLog
	? 0
	: params->windowLog - params->hashLog;
	}
	params->bucketSizeLog = MIN(params->bucketSizeLog, params->hashLog);
	}

	size_t ZSTD_ldm_getTableSize(ldmParams_t params)
	{
	size_t const ldmHSize = ((size_t)1) << params.hashLog;
	size_t const ldmBucketSizeLog = MIN(params.bucketSizeLog, params.hashLog);
	size_t const ldmBucketSize = ((size_t)1) << (params.hashLog - ldmBucketSizeLog);
	size_t const totalSize = ZSTD_cwksp_alloc_size(ldmBucketSize)
	+ ZSTD_cwksp_alloc_size(ldmHSize * sizeof(ldmEntry_t));
	return params.enableLdm ? totalSize : 0;
	}

	size_t ZSTD_ldm_getMaxNbSeq(ldmParams_t params, size_t maxChunkSize)
	{
	return params.enableLdm ? (maxChunkSize / params.minMatchLength) : 0;
	}

	/** ZSTD_ldm_getSmallHash() :
	* numBits should be <= 32
	* If numBits==0, returns 0.
	* @return : the most significant numBits of value. */
	static U32 ZSTD_ldm_getSmallHash(U64 value, U32 numBits)
	{
	assert(numBits <= 32);
	return numBits == 0 ? 0 : (U32)(value >> (64 - numBits));
	}

	/** ZSTD_ldm_getChecksum() :
	* numBitsToDiscard should be <= 32
	* @return : the next most significant 32 bits after numBitsToDiscard */
	static U32 ZSTD_ldm_getChecksum(U64 hash, U32 numBitsToDiscard)
	{
	assert(numBitsToDiscard <= 32);
	return (hash >> (64 - 32 - numBitsToDiscard)) & 0xFFFFFFFF;
	}

	/** ZSTD_ldm_getTag() ;
	* Given the hash, returns the most significant numTagBits bits
	* after (32 + hbits) bits.
	*
	* If there are not enough bits remaining, return the last
	* numTagBits bits. */
	static U32 ZSTD_ldm_getTag(U64 hash, U32 hbits, U32 numTagBits)
	{
	assert(numTagBits < 32 && hbits <= 32);
	if (32 - hbits < numTagBits) {
	return hash & (((U32)1 << numTagBits) - 1);
	} else {
	return (hash >> (32 - hbits - numTagBits)) & (((U32)1 << numTagBits) - 1);
	}
	}

	/** ZSTD_ldm_getBucket() :
	* Returns a pointer to the start of the bucket associated with hash. */
	static ldmEntry_t* ZSTD_ldm_getBucket(
	ldmState_t* ldmState, size_t hash, ldmParams_t const ldmParams)
	{
	return ldmState->hashTable + (hash << ldmParams.bucketSizeLog);
	}

	/** ZSTD_ldm_insertEntry() :
	* Insert the entry with corresponding hash into the hash table */
	static void ZSTD_ldm_insertEntry(ldmState_t* ldmState,
	size_t const hash, const ldmEntry_t entry,
	ldmParams_t const ldmParams)
	{
	BYTE* const bucketOffsets = ldmState->bucketOffsets;
	*(ZSTD_ldm_getBucket(ldmState, hash, ldmParams) + bucketOffsets[hash]) = entry;
	bucketOffsets[hash]++;
	bucketOffsets[hash] &= ((U32)1 << ldmParams.bucketSizeLog) - 1;
	}

	/** ZSTD_ldm_makeEntryAndInsertByTag() :
	*
	* Gets the small hash, checksum, and tag from the rollingHash.
	*
	* If the tag matches (1 << ldmParams.hashRateLog)-1, then
	* creates an ldmEntry from the offset, and inserts it into the hash table.
	*
	* hBits is the length of the small hash, which is the most significant hBits
	* of rollingHash. The checksum is the next 32 most significant bits, followed
	* by ldmParams.hashRateLog bits that make up the tag. */
	static void ZSTD_ldm_makeEntryAndInsertByTag(ldmState_t* ldmState,
	U64 const rollingHash,
	U32 const hBits,
	U32 const offset,
	ldmParams_t const ldmParams)
	{
	U32 const tag = ZSTD_ldm_getTag(rollingHash, hBits, ldmParams.hashRateLog);
	U32 const tagMask = ((U32)1 << ldmParams.hashRateLog) - 1;
	if (tag == tagMask) {
	U32 const hash = ZSTD_ldm_getSmallHash(rollingHash, hBits);
	U32 const checksum = ZSTD_ldm_getChecksum(rollingHash, hBits);
	ldmEntry_t entry;
	entry.offset = offset;
	entry.checksum = checksum;
	ZSTD_ldm_insertEntry(ldmState, hash, entry, ldmParams);
	}
	}

	/** ZSTD_ldm_countBackwardsMatch() :
	* Returns the number of bytes that match backwards before pIn and pMatch.
	*
	* We count only bytes where pMatch >= pBase and pIn >= pAnchor. */
	static size_t ZSTD_ldm_countBackwardsMatch(
	const BYTE* pIn, const BYTE* pAnchor,
	const BYTE* pMatch, const BYTE* pBase)
	{
	size_t matchLength = 0;
	while (pIn > pAnchor && pMatch > pBase && pIn[-1] == pMatch[-1]) {
	pIn--;
	pMatch--;
	matchLength++;
	}
	return matchLength;
	}

	/** ZSTD_ldm_fillFastTables() :
	*
	* Fills the relevant tables for the ZSTD_fast and ZSTD_dfast strategies.
	* This is similar to ZSTD_loadDictionaryContent.
	*
	* The tables for the other strategies are filled within their
	* block compressors. */
	static size_t ZSTD_ldm_fillFastTables(ZSTD_matchState_t* ms,
	void const* end)
	{
	const BYTE* const iend = (const BYTE*)end;

	switch(ms->cParams.strategy)
	{
	case ZSTD_fast:
	ZSTD_fillHashTable(ms, iend, ZSTD_dtlm_fast);
	break;

	case ZSTD_dfast:
	ZSTD_fillDoubleHashTable(ms, iend, ZSTD_dtlm_fast);
	break;

	case ZSTD_greedy:
	case ZSTD_lazy:
	case ZSTD_lazy2:
	case ZSTD_btlazy2:
	case ZSTD_btopt:
	case ZSTD_btultra:
	case ZSTD_btultra2:
	break;
	default:
	assert(0); /* not possible : not a valid strategy id */
	}

	return 0;
	}

	/** ZSTD_ldm_fillLdmHashTable() :
	*
	* Fills hashTable from (lastHashed + 1) to iend (non-inclusive).
	* lastHash is the rolling hash that corresponds to lastHashed.
	*
	* Returns the rolling hash corresponding to position iend-1. */
	static U64 ZSTD_ldm_fillLdmHashTable(ldmState_t* state,
	U64 lastHash, const BYTE* lastHashed,
	const BYTE* iend, const BYTE* base,
	U32 hBits, ldmParams_t const ldmParams)
	{
	U64 rollingHash = lastHash;
	const BYTE* cur = lastHashed + 1;

	while (cur < iend) {
	rollingHash = ZSTD_rollingHash_rotate(rollingHash, cur[-1],
	cur[ldmParams.minMatchLength-1],
	state->hashPower);
	ZSTD_ldm_makeEntryAndInsertByTag(state,
	rollingHash, hBits,
	(U32)(cur - base), ldmParams);
	++cur;
	}
	return rollingHash;
	}

	void ZSTD_ldm_fillHashTable(
	ldmState_t* state, const BYTE* ip,
	const BYTE* iend, ldmParams_t const* params)
	{
	DEBUGLOG(5, "ZSTD_ldm_fillHashTable");
	if ((size_t)(iend - ip) >= params->minMatchLength) {
	U64 startingHash = ZSTD_rollingHash_compute(ip, params->minMatchLength);
	ZSTD_ldm_fillLdmHashTable(
	state, startingHash, ip, iend - params->minMatchLength, state->window.base,
	params->hashLog - params->bucketSizeLog,
	*params);
	}
	}


	/** ZSTD_ldm_limitTableUpdate() :
	*
	* Sets cctx->nextToUpdate to a position corresponding closer to anchor
	* if it is far way
	* (after a long match, only update tables a limited amount). */
	static void ZSTD_ldm_limitTableUpdate(ZSTD_matchState_t* ms, const BYTE* anchor)
	{
	U32 const current = (U32)(anchor - ms->window.base);
	if (current > ms->nextToUpdate + 1024) {
	ms->nextToUpdate =
	current - MIN(512, current - ms->nextToUpdate - 1024);
	}
	}

	static size_t ZSTD_ldm_generateSequences_internal(
	ldmState_t* ldmState, rawSeqStore_t* rawSeqStore,
	ldmParams_t const* params, void const* src, size_t srcSize)
	{
	/* LDM parameters */
	int const extDict = ZSTD_window_hasExtDict(ldmState->window);
	U32 const minMatchLength = params->minMatchLength;
	U64 const hashPower = ldmState->hashPower;
	U32 const hBits = params->hashLog - params->bucketSizeLog;
	U32 const ldmBucketSize = 1U << params->bucketSizeLog;
	U32 const hashRateLog = params->hashRateLog;
	U32 const ldmTagMask = (1U << params->hashRateLog) - 1;
	/* Prefix and extDict parameters */
	U32 const dictLimit = ldmState->window.dictLimit;
	U32 const lowestIndex = extDict ? ldmState->window.lowLimit : dictLimit;
	BYTE const* const base = ldmState->window.base;
	BYTE const* const dictBase = extDict ? ldmState->window.dictBase : NULL;
	BYTE const* const dictStart = extDict ? dictBase + lowestIndex : NULL;
	BYTE const* const dictEnd = extDict ? dictBase + dictLimit : NULL;
	BYTE const* const lowPrefixPtr = base + dictLimit;
	/* Input bounds */
	BYTE const* const istart = (BYTE const*)src;
	BYTE const* const iend = istart + srcSize;
	BYTE const* const ilimit = iend - MAX(minMatchLength, HASH_READ_SIZE);
	/* Input positions */
	BYTE const* anchor = istart;
	BYTE const* ip = istart;
	/* Rolling hash */
	BYTE const* lastHashed = NULL;
	U64 rollingHash = 0;

	while (ip <= ilimit) {
	size_t mLength;
	U32 const current = (U32)(ip - base);
	size_t forwardMatchLength = 0, backwardMatchLength = 0;
	ldmEntry_t* bestEntry = NULL;
	if (ip != istart) {
	rollingHash = ZSTD_rollingHash_rotate(rollingHash, lastHashed[0],
	lastHashed[minMatchLength],
	hashPower);
	} else {
	rollingHash = ZSTD_rollingHash_compute(ip, minMatchLength);
	}
	lastHashed = ip;

	/* Do not insert and do not look for a match */
	if (ZSTD_ldm_getTag(rollingHash, hBits, hashRateLog) != ldmTagMask) {
	ip++;
	continue;
	}

	/* Get the best entry and compute the match lengths */
	{
	ldmEntry_t* const bucket =
	ZSTD_ldm_getBucket(ldmState,
	ZSTD_ldm_getSmallHash(rollingHash, hBits),
	*params);
	ldmEntry_t* cur;
	size_t bestMatchLength = 0;
	U32 const checksum = ZSTD_ldm_getChecksum(rollingHash, hBits);

	for (cur = bucket; cur < bucket + ldmBucketSize; ++cur) {
	size_t curForwardMatchLength, curBackwardMatchLength,
	curTotalMatchLength;
	if (cur->checksum != checksum \|\| cur->offset <= lowestIndex) {
	continue;
	}
	if (extDict) {
	BYTE const* const curMatchBase =
	cur->offset < dictLimit ? dictBase : base;
	BYTE const* const pMatch = curMatchBase + cur->offset;
	BYTE const* const matchEnd =
	cur->offset < dictLimit ? dictEnd : iend;
	BYTE const* const lowMatchPtr =
	cur->offset < dictLimit ? dictStart : lowPrefixPtr;

	curForwardMatchLength = ZSTD_count_2segments(
	ip, pMatch, iend,
	matchEnd, lowPrefixPtr);
	if (curForwardMatchLength < minMatchLength) {
	continue;
	}
	curBackwardMatchLength =
	ZSTD_ldm_countBackwardsMatch(ip, anchor, pMatch,
	lowMatchPtr);
	curTotalMatchLength = curForwardMatchLength +
	curBackwardMatchLength;
	} else { /* !extDict */
	BYTE const* const pMatch = base + cur->offset;
	curForwardMatchLength = ZSTD_count(ip, pMatch, iend);
	if (curForwardMatchLength < minMatchLength) {
	continue;
	}
	curBackwardMatchLength =
	ZSTD_ldm_countBackwardsMatch(ip, anchor, pMatch,
	lowPrefixPtr);
	curTotalMatchLength = curForwardMatchLength +
	curBackwardMatchLength;
	}

	if (curTotalMatchLength > bestMatchLength) {
	bestMatchLength = curTotalMatchLength;
	forwardMatchLength = curForwardMatchLength;
	backwardMatchLength = curBackwardMatchLength;
	bestEntry = cur;
	}
	}
	}

	/* No match found -- continue searching */
	if (bestEntry == NULL) {
	ZSTD_ldm_makeEntryAndInsertByTag(ldmState, rollingHash,
	hBits, current,
	*params);
	ip++;
	continue;
	}

	/* Match found */
	mLength = forwardMatchLength + backwardMatchLength;
	ip -= backwardMatchLength;

	{
	/* Store the sequence:
	* ip = current - backwardMatchLength
	* The match is at (bestEntry->offset - backwardMatchLength)
	*/
	U32 const matchIndex = bestEntry->offset;
	U32 const offset = current - matchIndex;
	rawSeq* const seq = rawSeqStore->seq + rawSeqStore->size;

	/* Out of sequence storage */
	if (rawSeqStore->size == rawSeqStore->capacity)
	return ERROR(dstSize_tooSmall);
	seq->litLength = (U32)(ip - anchor);
	seq->matchLength = (U32)mLength;
	seq->offset = offset;
	rawSeqStore->size++;
	}

	/* Insert the current entry into the hash table */
	ZSTD_ldm_makeEntryAndInsertByTag(ldmState, rollingHash, hBits,
	(U32)(lastHashed - base),
	*params);

	assert(ip + backwardMatchLength == lastHashed);

	/* Fill the hash table from lastHashed+1 to ip+mLength*/
	/* Heuristic: don't need to fill the entire table at end of block */
	if (ip + mLength <= ilimit) {
	rollingHash = ZSTD_ldm_fillLdmHashTable(
	ldmState, rollingHash, lastHashed,
	ip + mLength, base, hBits, *params);
	lastHashed = ip + mLength - 1;
	}
	ip += mLength;
	anchor = ip;
	}
	return iend - anchor;
	}

	/*! ZSTD_ldm_reduceTable() :
	* reduce table indexes by `reducerValue` */
	static void ZSTD_ldm_reduceTable(ldmEntry_t* const table, U32 const size,
	U32 const reducerValue)
	{
	U32 u;
	for (u = 0; u < size; u++) {
	if (table[u].offset < reducerValue) table[u].offset = 0;
	else table[u].offset -= reducerValue;
	}
	}

	size_t ZSTD_ldm_generateSequences(
	ldmState_t* ldmState, rawSeqStore_t* sequences,
	ldmParams_t const* params, void const* src, size_t srcSize)
	{
	U32 const maxDist = 1U << params->windowLog;
	BYTE const* const istart = (BYTE const*)src;
	BYTE const* const iend = istart + srcSize;
	size_t const kMaxChunkSize = 1 << 20;
	size_t const nbChunks = (srcSize / kMaxChunkSize) + ((srcSize % kMaxChunkSize) != 0);
	size_t chunk;
	size_t leftoverSize = 0;

	assert(ZSTD_CHUNKSIZE_MAX >= kMaxChunkSize);
	/* Check that ZSTD_window_update() has been called for this chunk prior
	* to passing it to this function.
	*/
	assert(ldmState->window.nextSrc >= (BYTE const*)src + srcSize);
	/* The input could be very large (in zstdmt), so it must be broken up into
	* chunks to enforce the maximum distance and handle overflow correction.
	*/
	assert(sequences->pos <= sequences->size);
	assert(sequences->size <= sequences->capacity);
	for (chunk = 0; chunk < nbChunks && sequences->size < sequences->capacity; ++chunk) {
	BYTE const* const chunkStart = istart + chunk * kMaxChunkSize;
	size_t const remaining = (size_t)(iend - chunkStart);
	BYTE const *const chunkEnd =
	(remaining < kMaxChunkSize) ? iend : chunkStart + kMaxChunkSize;
	size_t const chunkSize = chunkEnd - chunkStart;
	size_t newLeftoverSize;
	size_t const prevSize = sequences->size;

	assert(chunkStart < iend);
	/* 1. Perform overflow correction if necessary. */
	if (ZSTD_window_needOverflowCorrection(ldmState->window, chunkEnd)) {
	U32 const ldmHSize = 1U << params->hashLog;
	U32 const correction = ZSTD_window_correctOverflow(
	&ldmState->window, /* cycleLog */ 0, maxDist, chunkStart);
	ZSTD_ldm_reduceTable(ldmState->hashTable, ldmHSize, correction);
	/* invalidate dictionaries on overflow correction */
	ldmState->loadedDictEnd = 0;
	}
	/* 2. We enforce the maximum offset allowed.
	*
	* kMaxChunkSize should be small enough that we don't lose too much of
	* the window through early invalidation.
	* TODO: * Test the chunk size.
	* * Try invalidation after the sequence generation and test the
	* the offset against maxDist directly.
	*
	* NOTE: Because of dictionaries + sequence splitting we MUST make sure
	* that any offset used is valid at the END of the sequence, since it may
	* be split into two sequences. This condition holds when using
	* ZSTD_window_enforceMaxDist(), but if we move to checking offsets
	* against maxDist directly, we'll have to carefully handle that case.
	*/
	ZSTD_window_enforceMaxDist(&ldmState->window, chunkEnd, maxDist, &ldmState->loadedDictEnd, NULL);
	/* 3. Generate the sequences for the chunk, and get newLeftoverSize. */
	newLeftoverSize = ZSTD_ldm_generateSequences_internal(
	ldmState, sequences, params, chunkStart, chunkSize);
	if (ZSTD_isError(newLeftoverSize))
	return newLeftoverSize;
	/* 4. We add the leftover literals from previous iterations to the first
	* newly generated sequence, or add the `newLeftoverSize` if none are
	* generated.
	*/
	/* Prepend the leftover literals from the last call */
	if (prevSize < sequences->size) {
	sequences->seq[prevSize].litLength += (U32)leftoverSize;
	leftoverSize = newLeftoverSize;
	} else {
	assert(newLeftoverSize == chunkSize);
	leftoverSize += chunkSize;
	}
	}
	return 0;
	}

	void ZSTD_ldm_skipSequences(rawSeqStore_t* rawSeqStore, size_t srcSize, U32 const minMatch) {
	while (srcSize > 0 && rawSeqStore->pos < rawSeqStore->size) {
	rawSeq* seq = rawSeqStore->seq + rawSeqStore->pos;
	if (srcSize <= seq->litLength) {
	/* Skip past srcSize literals */
	seq->litLength -= (U32)srcSize;
	return;
	}
	srcSize -= seq->litLength;
	seq->litLength = 0;
	if (srcSize < seq->matchLength) {
	/* Skip past the first srcSize of the match */
	seq->matchLength -= (U32)srcSize;
	if (seq->matchLength < minMatch) {
	/* The match is too short, omit it */
	if (rawSeqStore->pos + 1 < rawSeqStore->size) {
	seq[1].litLength += seq[0].matchLength;
	}
	rawSeqStore->pos++;
	}
	return;
	}
	srcSize -= seq->matchLength;
	seq->matchLength = 0;
	rawSeqStore->pos++;
	}
	}

	/**
	* If the sequence length is longer than remaining then the sequence is split
	* between this block and the next.
	*
	* Returns the current sequence to handle, or if the rest of the block should
	* be literals, it returns a sequence with offset == 0.
	*/
	static rawSeq maybeSplitSequence(rawSeqStore_t* rawSeqStore,
	U32 const remaining, U32 const minMatch)
	{
	rawSeq sequence = rawSeqStore->seq[rawSeqStore->pos];
	assert(sequence.offset > 0);
	/* Likely: No partial sequence */
	if (remaining >= sequence.litLength + sequence.matchLength) {
	rawSeqStore->pos++;
	return sequence;
	}
	/* Cut the sequence short (offset == 0 ==> rest is literals). */
	if (remaining <= sequence.litLength) {
	sequence.offset = 0;
	} else if (remaining < sequence.litLength + sequence.matchLength) {
	sequence.matchLength = remaining - sequence.litLength;
	if (sequence.matchLength < minMatch) {
	sequence.offset = 0;
	}
	}
	/* Skip past `remaining` bytes for the future sequences. */
	ZSTD_ldm_skipSequences(rawSeqStore, remaining, minMatch);
	return sequence;
	}

	size_t ZSTD_ldm_blockCompress(rawSeqStore_t* rawSeqStore,
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	void const* src, size_t srcSize)
	{
	const ZSTD_compressionParameters* const cParams = &ms->cParams;
	unsigned const minMatch = cParams->minMatch;
	ZSTD_blockCompressor const blockCompressor =
	ZSTD_selectBlockCompressor(cParams->strategy, ZSTD_matchState_dictMode(ms));
	/* Input bounds */
	BYTE const* const istart = (BYTE const*)src;
	BYTE const* const iend = istart + srcSize;
	/* Input positions */
	BYTE const* ip = istart;

	DEBUGLOG(5, "ZSTD_ldm_blockCompress: srcSize=%zu", srcSize);
	assert(rawSeqStore->pos <= rawSeqStore->size);
	assert(rawSeqStore->size <= rawSeqStore->capacity);
	/* Loop through each sequence and apply the block compressor to the lits */
	while (rawSeqStore->pos < rawSeqStore->size && ip < iend) {
	/* maybeSplitSequence updates rawSeqStore->pos */
	rawSeq const sequence = maybeSplitSequence(rawSeqStore,
	(U32)(iend - ip), minMatch);
	int i;
	/* End signal */
	if (sequence.offset == 0)
	break;

	assert(ip + sequence.litLength + sequence.matchLength <= iend);

	/* Fill tables for block compressor */
	ZSTD_ldm_limitTableUpdate(ms, ip);
	ZSTD_ldm_fillFastTables(ms, ip);
	/* Run the block compressor */
	DEBUGLOG(5, "pos %u : calling block compressor on segment of size %u", (unsigned)(ip-istart), sequence.litLength);
	{
	size_t const newLitLength =
	blockCompressor(ms, seqStore, rep, ip, sequence.litLength);
	ip += sequence.litLength;
	/* Update the repcodes */
	for (i = ZSTD_REP_NUM - 1; i > 0; i--)
	rep[i] = rep[i-1];
	rep[0] = sequence.offset;
	/* Store the sequence */
	ZSTD_storeSeq(seqStore, newLitLength, ip - newLitLength, iend,
	sequence.offset + ZSTD_REP_MOVE,
	sequence.matchLength - MINMATCH);
	ip += sequence.matchLength;
	}
	}
	/* Fill the tables for the block compressor */
	ZSTD_ldm_limitTableUpdate(ms, ip);
	ZSTD_ldm_fillFastTables(ms, ip);
	/* Compress the last literals */
	return blockCompressor(ms, seqStore, rep, ip, iend - ip);
	}
	/** ended inlining compress/zstd_ldm.c **/
	/** start inlining compress/zstd_opt.c **/
	/*
	* Copyright (c) 2016-2020, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	/** skipping file: zstd_compress_internal.h **/
	/** skipping file: hist.h **/
	/** skipping file: zstd_opt.h **/


	#define ZSTD_LITFREQ_ADD 2 /* scaling factor for litFreq, so that frequencies adapt faster to new stats */
	#define ZSTD_FREQ_DIV 4 /* log factor when using previous stats to init next stats */
	#define ZSTD_MAX_PRICE (1<<30)

	#define ZSTD_PREDEF_THRESHOLD 1024 /* if srcSize < ZSTD_PREDEF_THRESHOLD, symbols' cost is assumed static, directly determined by pre-defined distributions */


	/-************************************
	* Price functions for optimal parser
	***************************************/

	#if 0 /* approximation at bit level */
	# define BITCOST_ACCURACY 0
	# define BITCOST_MULTIPLIER (1 << BITCOST_ACCURACY)
	# define WEIGHT(stat) ((void)opt, ZSTD_bitWeight(stat))
	#elif 0 /* fractional bit accuracy */
	# define BITCOST_ACCURACY 8
	# define BITCOST_MULTIPLIER (1 << BITCOST_ACCURACY)
	# define WEIGHT(stat,opt) ((void)opt, ZSTD_fracWeight(stat))
	#else /* opt==approx, ultra==accurate */
	# define BITCOST_ACCURACY 8
	# define BITCOST_MULTIPLIER (1 << BITCOST_ACCURACY)
	# define WEIGHT(stat,opt) (opt ? ZSTD_fracWeight(stat) : ZSTD_bitWeight(stat))
	#endif

	MEM_STATIC U32 ZSTD_bitWeight(U32 stat)
	{
	return (ZSTD_highbit32(stat+1) * BITCOST_MULTIPLIER);
	}

	MEM_STATIC U32 ZSTD_fracWeight(U32 rawStat)
	{
	U32 const stat = rawStat + 1;
	U32 const hb = ZSTD_highbit32(stat);
	U32 const BWeight = hb * BITCOST_MULTIPLIER;
	U32 const FWeight = (stat << BITCOST_ACCURACY) >> hb;
	U32 const weight = BWeight + FWeight;
	assert(hb + BITCOST_ACCURACY < 31);
	return weight;
	}

	#if (DEBUGLEVEL>=2)
	/* debugging function,
	* @return price in bytes as fractional value
	* for debug messages only */
	MEM_STATIC double ZSTD_fCost(U32 price)
	{
	return (double)price / (BITCOST_MULTIPLIER*8);
	}
	#endif

	static int ZSTD_compressedLiterals(optState_t const* const optPtr)
	{
	return optPtr->literalCompressionMode != ZSTD_lcm_uncompressed;
	}

	static void ZSTD_setBasePrices(optState_t* optPtr, int optLevel)
	{
	if (ZSTD_compressedLiterals(optPtr))
	optPtr->litSumBasePrice = WEIGHT(optPtr->litSum, optLevel);
	optPtr->litLengthSumBasePrice = WEIGHT(optPtr->litLengthSum, optLevel);
	optPtr->matchLengthSumBasePrice = WEIGHT(optPtr->matchLengthSum, optLevel);
	optPtr->offCodeSumBasePrice = WEIGHT(optPtr->offCodeSum, optLevel);
	}


	/* ZSTD_downscaleStat() :
	* reduce all elements in table by a factor 2^(ZSTD_FREQ_DIV+malus)
	* return the resulting sum of elements */
	static U32 ZSTD_downscaleStat(unsigned* table, U32 lastEltIndex, int malus)
	{
	U32 s, sum=0;
	DEBUGLOG(5, "ZSTD_downscaleStat (nbElts=%u)", (unsigned)lastEltIndex+1);
	assert(ZSTD_FREQ_DIV+malus > 0 && ZSTD_FREQ_DIV+malus < 31);
	for (s=0; s<lastEltIndex+1; s++) {
	table[s] = 1 + (table[s] >> (ZSTD_FREQ_DIV+malus));
	sum += table[s];
	}
	return sum;
	}

	/* ZSTD_rescaleFreqs() :
	* if first block (detected by optPtr->litLengthSum == 0) : init statistics
	* take hints from dictionary if there is one
	* or init from zero, using src for literals stats, or flat 1 for match symbols
	* otherwise downscale existing stats, to be used as seed for next block.
	*/
	static void
	ZSTD_rescaleFreqs(optState_t* const optPtr,
	const BYTE* const src, size_t const srcSize,
	int const optLevel)
	{
	int const compressedLiterals = ZSTD_compressedLiterals(optPtr);
	DEBUGLOG(5, "ZSTD_rescaleFreqs (srcSize=%u)", (unsigned)srcSize);
	optPtr->priceType = zop_dynamic;

	if (optPtr->litLengthSum == 0) { /* first block : init */
	if (srcSize <= ZSTD_PREDEF_THRESHOLD) { /* heuristic */
	DEBUGLOG(5, "(srcSize <= ZSTD_PREDEF_THRESHOLD) => zop_predef");
	optPtr->priceType = zop_predef;
	}

	assert(optPtr->symbolCosts != NULL);
	if (optPtr->symbolCosts->huf.repeatMode == HUF_repeat_valid) {
	/* huffman table presumed generated by dictionary */
	optPtr->priceType = zop_dynamic;

	if (compressedLiterals) {
	unsigned lit;
	assert(optPtr->litFreq != NULL);
	optPtr->litSum = 0;
	for (lit=0; lit<=MaxLit; lit++) {
	U32 const scaleLog = 11; /* scale to 2K */
	U32 const bitCost = HUF_getNbBits(optPtr->symbolCosts->huf.CTable, lit);
	assert(bitCost <= scaleLog);
	optPtr->litFreq[lit] = bitCost ? 1 << (scaleLog-bitCost) : 1 /minimum to calculate cost/;
	optPtr->litSum += optPtr->litFreq[lit];
	} }

	{ unsigned ll;
	FSE_CState_t llstate;
	FSE_initCState(&llstate, optPtr->symbolCosts->fse.litlengthCTable);
	optPtr->litLengthSum = 0;
	for (ll=0; ll<=MaxLL; ll++) {
	U32 const scaleLog = 10; /* scale to 1K */
	U32 const bitCost = FSE_getMaxNbBits(llstate.symbolTT, ll);
	assert(bitCost < scaleLog);
	optPtr->litLengthFreq[ll] = bitCost ? 1 << (scaleLog-bitCost) : 1 /minimum to calculate cost/;
	optPtr->litLengthSum += optPtr->litLengthFreq[ll];
	} }

	{ unsigned ml;
	FSE_CState_t mlstate;
	FSE_initCState(&mlstate, optPtr->symbolCosts->fse.matchlengthCTable);
	optPtr->matchLengthSum = 0;
	for (ml=0; ml<=MaxML; ml++) {
	U32 const scaleLog = 10;
	U32 const bitCost = FSE_getMaxNbBits(mlstate.symbolTT, ml);
	assert(bitCost < scaleLog);
	optPtr->matchLengthFreq[ml] = bitCost ? 1 << (scaleLog-bitCost) : 1 /minimum to calculate cost/;
	optPtr->matchLengthSum += optPtr->matchLengthFreq[ml];
	} }

	{ unsigned of;
	FSE_CState_t ofstate;
	FSE_initCState(&ofstate, optPtr->symbolCosts->fse.offcodeCTable);
	optPtr->offCodeSum = 0;
	for (of=0; of<=MaxOff; of++) {
	U32 const scaleLog = 10;
	U32 const bitCost = FSE_getMaxNbBits(ofstate.symbolTT, of);
	assert(bitCost < scaleLog);
	optPtr->offCodeFreq[of] = bitCost ? 1 << (scaleLog-bitCost) : 1 /minimum to calculate cost/;
	optPtr->offCodeSum += optPtr->offCodeFreq[of];
	} }

	} else { /* not a dictionary */

	assert(optPtr->litFreq != NULL);
	if (compressedLiterals) {
	unsigned lit = MaxLit;
	HIST_count_simple(optPtr->litFreq, &lit, src, srcSize); /* use raw first block to init statistics */
	optPtr->litSum = ZSTD_downscaleStat(optPtr->litFreq, MaxLit, 1);
	}

	{ unsigned ll;
	for (ll=0; ll<=MaxLL; ll++)
	optPtr->litLengthFreq[ll] = 1;
	}
	optPtr->litLengthSum = MaxLL+1;

	{ unsigned ml;
	for (ml=0; ml<=MaxML; ml++)
	optPtr->matchLengthFreq[ml] = 1;
	}
	optPtr->matchLengthSum = MaxML+1;

	{ unsigned of;
	for (of=0; of<=MaxOff; of++)
	optPtr->offCodeFreq[of] = 1;
	}
	optPtr->offCodeSum = MaxOff+1;

	}

	} else { /* new block : re-use previous statistics, scaled down */

	if (compressedLiterals)
	optPtr->litSum = ZSTD_downscaleStat(optPtr->litFreq, MaxLit, 1);
	optPtr->litLengthSum = ZSTD_downscaleStat(optPtr->litLengthFreq, MaxLL, 0);
	optPtr->matchLengthSum = ZSTD_downscaleStat(optPtr->matchLengthFreq, MaxML, 0);
	optPtr->offCodeSum = ZSTD_downscaleStat(optPtr->offCodeFreq, MaxOff, 0);
	}

	ZSTD_setBasePrices(optPtr, optLevel);
	}

	/* ZSTD_rawLiteralsCost() :
	* price of literals (only) in specified segment (which length can be 0).
	* does not include price of literalLength symbol */
	static U32 ZSTD_rawLiteralsCost(const BYTE* const literals, U32 const litLength,
	const optState_t* const optPtr,
	int optLevel)
	{
	if (litLength == 0) return 0;

	if (!ZSTD_compressedLiterals(optPtr))
	return (litLength << 3) * BITCOST_MULTIPLIER; /* Uncompressed - 8 bytes per literal. */

	if (optPtr->priceType == zop_predef)
	return (litLength6) BITCOST_MULTIPLIER; /* 6 bit per literal - no statistic used */

	/* dynamic statistics */
	{ U32 price = litLength * optPtr->litSumBasePrice;
	U32 u;
	for (u=0; u < litLength; u++) {
	assert(WEIGHT(optPtr->litFreq[literals[u]], optLevel) <= optPtr->litSumBasePrice); /* literal cost should never be negative */
	price -= WEIGHT(optPtr->litFreq[literals[u]], optLevel);
	}
	return price;
	}
	}

	/* ZSTD_litLengthPrice() :
	* cost of literalLength symbol */
	static U32 ZSTD_litLengthPrice(U32 const litLength, const optState_t* const optPtr, int optLevel)
	{
	if (optPtr->priceType == zop_predef) return WEIGHT(litLength, optLevel);

	/* dynamic statistics */
	{ U32 const llCode = ZSTD_LLcode(litLength);
	return (LL_bits[llCode] * BITCOST_MULTIPLIER)
	+ optPtr->litLengthSumBasePrice
	- WEIGHT(optPtr->litLengthFreq[llCode], optLevel);
	}
	}

	/* ZSTD_getMatchPrice() :
	* Provides the cost of the match part (offset + matchLength) of a sequence
	* Must be combined with ZSTD_fullLiteralsCost() to get the full cost of a sequence.
	* optLevel: when <2, favors small offset for decompression speed (improved cache efficiency) */
	FORCE_INLINE_TEMPLATE U32
	ZSTD_getMatchPrice(U32 const offset,
	U32 const matchLength,
	const optState_t* const optPtr,
	int const optLevel)
	{
	U32 price;
	U32 const offCode = ZSTD_highbit32(offset+1);
	U32 const mlBase = matchLength - MINMATCH;
	assert(matchLength >= MINMATCH);

	if (optPtr->priceType == zop_predef) /* fixed scheme, do not use statistics */
	return WEIGHT(mlBase, optLevel) + ((16 + offCode) * BITCOST_MULTIPLIER);

	/* dynamic statistics */
	price = (offCode * BITCOST_MULTIPLIER) + (optPtr->offCodeSumBasePrice - WEIGHT(optPtr->offCodeFreq[offCode], optLevel));
	if ((optLevel<2) /static/ && offCode >= 20)
	price += (offCode-19)2 BITCOST_MULTIPLIER; /* handicap for long distance offsets, favor decompression speed */

	/* match Length */
	{ U32 const mlCode = ZSTD_MLcode(mlBase);
	price += (ML_bits[mlCode] * BITCOST_MULTIPLIER) + (optPtr->matchLengthSumBasePrice - WEIGHT(optPtr->matchLengthFreq[mlCode], optLevel));
	}

	price += BITCOST_MULTIPLIER / 5; /* heuristic : make matches a bit more costly to favor less sequences -> faster decompression speed */

	DEBUGLOG(8, "ZSTD_getMatchPrice(ml:%u) = %u", matchLength, price);
	return price;
	}

	/* ZSTD_updateStats() :
	* assumption : literals + litLengtn <= iend */
	static void ZSTD_updateStats(optState_t* const optPtr,
	U32 litLength, const BYTE* literals,
	U32 offsetCode, U32 matchLength)
	{
	/* literals */
	if (ZSTD_compressedLiterals(optPtr)) {
	U32 u;
	for (u=0; u < litLength; u++)
	optPtr->litFreq[literals[u]] += ZSTD_LITFREQ_ADD;
	optPtr->litSum += litLength*ZSTD_LITFREQ_ADD;
	}

	/* literal Length */
	{ U32 const llCode = ZSTD_LLcode(litLength);
	optPtr->litLengthFreq[llCode]++;
	optPtr->litLengthSum++;
	}

	/* match offset code (0-2=>repCode; 3+=>offset+2) */
	{ U32 const offCode = ZSTD_highbit32(offsetCode+1);
	assert(offCode <= MaxOff);
	optPtr->offCodeFreq[offCode]++;
	optPtr->offCodeSum++;
	}

	/* match Length */
	{ U32 const mlBase = matchLength - MINMATCH;
	U32 const mlCode = ZSTD_MLcode(mlBase);
	optPtr->matchLengthFreq[mlCode]++;
	optPtr->matchLengthSum++;
	}
	}


	/* ZSTD_readMINMATCH() :
	* function safe only for comparisons
	* assumption : memPtr must be at least 4 bytes before end of buffer */
	MEM_STATIC U32 ZSTD_readMINMATCH(const void* memPtr, U32 length)
	{
	switch (length)
	{
	default :
	case 4 : return MEM_read32(memPtr);
	case 3 : if (MEM_isLittleEndian())
	return MEM_read32(memPtr)<<8;
	else
	return MEM_read32(memPtr)>>8;
	}
	}


	/* Update hashTable3 up to ip (excluded)
	Assumption : always within prefix (i.e. not within extDict) */
	static U32 ZSTD_insertAndFindFirstIndexHash3 (ZSTD_matchState_t* ms,
	U32* nextToUpdate3,
	const BYTE* const ip)
	{
	U32* const hashTable3 = ms->hashTable3;
	U32 const hashLog3 = ms->hashLog3;
	const BYTE* const base = ms->window.base;
	U32 idx = *nextToUpdate3;
	U32 const target = (U32)(ip - base);
	size_t const hash3 = ZSTD_hash3Ptr(ip, hashLog3);
	assert(hashLog3 > 0);

	while(idx < target) {
	hashTable3[ZSTD_hash3Ptr(base+idx, hashLog3)] = idx;
	idx++;
	}

	*nextToUpdate3 = target;
	return hashTable3[hash3];
	}


	/-************************************
	* Binary Tree search
	***************************************/
	/** ZSTD_insertBt1() : add one or multiple positions to tree.
	* ip : assumed <= iend-8 .
	* @return : nb of positions added */
	static U32 ZSTD_insertBt1(
	ZSTD_matchState_t* ms,
	const BYTE* const ip, const BYTE* const iend,
	U32 const mls, const int extDict)
	{
	const ZSTD_compressionParameters* const cParams = &ms->cParams;
	U32* const hashTable = ms->hashTable;
	U32 const hashLog = cParams->hashLog;
	size_t const h = ZSTD_hashPtr(ip, hashLog, mls);
	U32* const bt = ms->chainTable;
	U32 const btLog = cParams->chainLog - 1;
	U32 const btMask = (1 << btLog) - 1;
	U32 matchIndex = hashTable[h];
	size_t commonLengthSmaller=0, commonLengthLarger=0;
	const BYTE* const base = ms->window.base;
	const BYTE* const dictBase = ms->window.dictBase;
	const U32 dictLimit = ms->window.dictLimit;
	const BYTE* const dictEnd = dictBase + dictLimit;
	const BYTE* const prefixStart = base + dictLimit;
	const BYTE* match;
	const U32 current = (U32)(ip-base);
	const U32 btLow = btMask >= current ? 0 : current - btMask;
	U32* smallerPtr = bt + 2*(current&btMask);
	U32* largerPtr = smallerPtr + 1;
	U32 dummy32; /* to be nullified at the end */
	U32 const windowLow = ms->window.lowLimit;
	U32 matchEndIdx = current+8+1;
	size_t bestLength = 8;
	U32 nbCompares = 1U << cParams->searchLog;
	#ifdef ZSTD_C_PREDICT
	U32 predictedSmall = (bt + 2((current-1)&btMask) + 0);
	U32 predictedLarge = (bt + 2((current-1)&btMask) + 1);
	predictedSmall += (predictedSmall>0);
	predictedLarge += (predictedLarge>0);
	#endif /* ZSTD_C_PREDICT */

	DEBUGLOG(8, "ZSTD_insertBt1 (%u)", current);

	assert(ip <= iend-8); /* required for h calculation */
	hashTable[h] = current; /* Update Hash Table */

	assert(windowLow > 0);
	while (nbCompares-- && (matchIndex >= windowLow)) {
	U32* const nextPtr = bt + 2*(matchIndex & btMask);
	size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */
	assert(matchIndex < current);

	#ifdef ZSTD_C_PREDICT /* note : can create issues when hlog small <= 11 */
	const U32* predictPtr = bt + 2((matchIndex-1) & btMask); / written this way, as bt is a roll buffer */
	if (matchIndex == predictedSmall) {
	/* no need to check length, result known */
	*smallerPtr = matchIndex;
	if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop the search */
	smallerPtr = nextPtr+1; /* new "smaller" => larger of match */
	matchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to current) */
	predictedSmall = predictPtr[1] + (predictPtr[1]>0);
	continue;
	}
	if (matchIndex == predictedLarge) {
	*largerPtr = matchIndex;
	if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop the search */
	largerPtr = nextPtr;
	matchIndex = nextPtr[0];
	predictedLarge = predictPtr[0] + (predictPtr[0]>0);
	continue;
	}
	#endif

	if (!extDict \|\| (matchIndex+matchLength >= dictLimit)) {
	assert(matchIndex+matchLength >= dictLimit); /* might be wrong if actually extDict */
	match = base + matchIndex;
	matchLength += ZSTD_count(ip+matchLength, match+matchLength, iend);
	} else {
	match = dictBase + matchIndex;
	matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iend, dictEnd, prefixStart);
	if (matchIndex+matchLength >= dictLimit)
	match = base + matchIndex; /* to prepare for next usage of match[matchLength] */
	}

	if (matchLength > bestLength) {
	bestLength = matchLength;
	if (matchLength > matchEndIdx - matchIndex)
	matchEndIdx = matchIndex + (U32)matchLength;
	}

	if (ip+matchLength == iend) { /* equal : no way to know if inf or sup */
	break; /* drop , to guarantee consistency ; miss a bit of compression, but other solutions can corrupt tree */
	}

	if (match[matchLength] < ip[matchLength]) { /* necessarily within buffer */
	/* match is smaller than current */
	smallerPtr = matchIndex; / update smaller idx */
	commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */
	if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop searching */
	smallerPtr = nextPtr+1; /* new "candidate" => larger than match, which was smaller than target */
	matchIndex = nextPtr[1]; /* new matchIndex, larger than previous and closer to current */
	} else {
	/* match is larger than current */
	*largerPtr = matchIndex;
	commonLengthLarger = matchLength;
	if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop searching */
	largerPtr = nextPtr;
	matchIndex = nextPtr[0];
	} }

	smallerPtr = largerPtr = 0;
	{ U32 positions = 0;
	if (bestLength > 384) positions = MIN(192, (U32)(bestLength - 384)); /* speed optimization */
	assert(matchEndIdx > current + 8);
	return MAX(positions, matchEndIdx - (current + 8));
	}
	}

	FORCE_INLINE_TEMPLATE
	void ZSTD_updateTree_internal(
	ZSTD_matchState_t* ms,
	const BYTE* const ip, const BYTE* const iend,
	const U32 mls, const ZSTD_dictMode_e dictMode)
	{
	const BYTE* const base = ms->window.base;
	U32 const target = (U32)(ip - base);
	U32 idx = ms->nextToUpdate;
	DEBUGLOG(6, "ZSTD_updateTree_internal, from %u to %u (dictMode:%u)",
	idx, target, dictMode);

	while(idx < target) {
	U32 const forward = ZSTD_insertBt1(ms, base+idx, iend, mls, dictMode == ZSTD_extDict);
	assert(idx < (U32)(idx + forward));
	idx += forward;
	}
	assert((size_t)(ip - base) <= (size_t)(U32)(-1));
	assert((size_t)(iend - base) <= (size_t)(U32)(-1));
	ms->nextToUpdate = target;
	}

	void ZSTD_updateTree(ZSTD_matchState_t* ms, const BYTE* ip, const BYTE* iend) {
	ZSTD_updateTree_internal(ms, ip, iend, ms->cParams.minMatch, ZSTD_noDict);
	}

	FORCE_INLINE_TEMPLATE
	U32 ZSTD_insertBtAndGetAllMatches (
	ZSTD_match_t* matches, /* store result (found matches) in this table (presumed large enough) */
	ZSTD_matchState_t* ms,
	U32* nextToUpdate3,
	const BYTE* const ip, const BYTE* const iLimit, const ZSTD_dictMode_e dictMode,
	const U32 rep[ZSTD_REP_NUM],
	U32 const ll0, /* tells if associated literal length is 0 or not. This value must be 0 or 1 */
	const U32 lengthToBeat,
	U32 const mls /* template */)
	{
	const ZSTD_compressionParameters* const cParams = &ms->cParams;
	U32 const sufficient_len = MIN(cParams->targetLength, ZSTD_OPT_NUM -1);
	const BYTE* const base = ms->window.base;
	U32 const current = (U32)(ip-base);
	U32 const hashLog = cParams->hashLog;
	U32 const minMatch = (mls==3) ? 3 : 4;
	U32* const hashTable = ms->hashTable;
	size_t const h = ZSTD_hashPtr(ip, hashLog, mls);
	U32 matchIndex = hashTable[h];
	U32* const bt = ms->chainTable;
	U32 const btLog = cParams->chainLog - 1;
	U32 const btMask= (1U << btLog) - 1;
	size_t commonLengthSmaller=0, commonLengthLarger=0;
	const BYTE* const dictBase = ms->window.dictBase;
	U32 const dictLimit = ms->window.dictLimit;
	const BYTE* const dictEnd = dictBase + dictLimit;
	const BYTE* const prefixStart = base + dictLimit;
	U32 const btLow = (btMask >= current) ? 0 : current - btMask;
	U32 const windowLow = ZSTD_getLowestMatchIndex(ms, current, cParams->windowLog);
	U32 const matchLow = windowLow ? windowLow : 1;
	U32* smallerPtr = bt + 2*(current&btMask);
	U32* largerPtr = bt + 2*(current&btMask) + 1;
	U32 matchEndIdx = current+8+1; /* farthest referenced position of any match => detects repetitive patterns */
	U32 dummy32; /* to be nullified at the end */
	U32 mnum = 0;
	U32 nbCompares = 1U << cParams->searchLog;

	const ZSTD_matchState_t* dms = dictMode == ZSTD_dictMatchState ? ms->dictMatchState : NULL;
	const ZSTD_compressionParameters* const dmsCParams =
	dictMode == ZSTD_dictMatchState ? &dms->cParams : NULL;
	const BYTE* const dmsBase = dictMode == ZSTD_dictMatchState ? dms->window.base : NULL;
	const BYTE* const dmsEnd = dictMode == ZSTD_dictMatchState ? dms->window.nextSrc : NULL;
	U32 const dmsHighLimit = dictMode == ZSTD_dictMatchState ? (U32)(dmsEnd - dmsBase) : 0;
	U32 const dmsLowLimit = dictMode == ZSTD_dictMatchState ? dms->window.lowLimit : 0;
	U32 const dmsIndexDelta = dictMode == ZSTD_dictMatchState ? windowLow - dmsHighLimit : 0;
	U32 const dmsHashLog = dictMode == ZSTD_dictMatchState ? dmsCParams->hashLog : hashLog;
	U32 const dmsBtLog = dictMode == ZSTD_dictMatchState ? dmsCParams->chainLog - 1 : btLog;
	U32 const dmsBtMask = dictMode == ZSTD_dictMatchState ? (1U << dmsBtLog) - 1 : 0;
	U32 const dmsBtLow = dictMode == ZSTD_dictMatchState && dmsBtMask < dmsHighLimit - dmsLowLimit ? dmsHighLimit - dmsBtMask : dmsLowLimit;

	size_t bestLength = lengthToBeat-1;
	DEBUGLOG(8, "ZSTD_insertBtAndGetAllMatches: current=%u", current);

	/* check repCode */
	assert(ll0 <= 1); /* necessarily 1 or 0 */
	{ U32 const lastR = ZSTD_REP_NUM + ll0;
	U32 repCode;
	for (repCode = ll0; repCode < lastR; repCode++) {
	U32 const repOffset = (repCode==ZSTD_REP_NUM) ? (rep[0] - 1) : rep[repCode];
	U32 const repIndex = current - repOffset;
	U32 repLen = 0;
	assert(current >= dictLimit);
	if (repOffset-1 /* intentional overflow, discards 0 and -1 / < current-dictLimit) { / equivalent to `current > repIndex >= dictLimit` */
	/* We must validate the repcode offset because when we're using a dictionary the
	* valid offset range shrinks when the dictionary goes out of bounds.
	*/
	if ((repIndex >= windowLow) & (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(ip - repOffset, minMatch))) {
	repLen = (U32)ZSTD_count(ip+minMatch, ip+minMatch-repOffset, iLimit) + minMatch;
	}
	} else { /* repIndex < dictLimit \|\| repIndex >= current */
	const BYTE* const repMatch = dictMode == ZSTD_dictMatchState ?
	dmsBase + repIndex - dmsIndexDelta :
	dictBase + repIndex;
	assert(current >= windowLow);
	if ( dictMode == ZSTD_extDict
	&& ( ((repOffset-1) /intentional overflow/ < current - windowLow) /* equivalent to `current > repIndex >= windowLow` */
	& (((U32)((dictLimit-1) - repIndex) >= 3) ) /* intentional overflow : do not test positions overlapping 2 memory segments */)
	&& (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(repMatch, minMatch)) ) {
	repLen = (U32)ZSTD_count_2segments(ip+minMatch, repMatch+minMatch, iLimit, dictEnd, prefixStart) + minMatch;
	}
	if (dictMode == ZSTD_dictMatchState
	&& ( ((repOffset-1) /intentional overflow/ < current - (dmsLowLimit + dmsIndexDelta)) /* equivalent to `current > repIndex >= dmsLowLimit` */
	& ((U32)((dictLimit-1) - repIndex) >= 3) ) /* intentional overflow : do not test positions overlapping 2 memory segments */
	&& (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(repMatch, minMatch)) ) {
	repLen = (U32)ZSTD_count_2segments(ip+minMatch, repMatch+minMatch, iLimit, dmsEnd, prefixStart) + minMatch;
	} }
	/* save longer solution */
	if (repLen > bestLength) {
	DEBUGLOG(8, "found repCode %u (ll0:%u, offset:%u) of length %u",
	repCode, ll0, repOffset, repLen);
	bestLength = repLen;
	matches[mnum].off = repCode - ll0;
	matches[mnum].len = (U32)repLen;
	mnum++;
	if ( (repLen > sufficient_len)
	\| (ip+repLen == iLimit) ) { /* best possible */
	return mnum;
	} } } }

	/* HC3 match finder */
	if ((mls == 3) /static/ && (bestLength < mls)) {
	U32 const matchIndex3 = ZSTD_insertAndFindFirstIndexHash3(ms, nextToUpdate3, ip);
	if ((matchIndex3 >= matchLow)
	& (current - matchIndex3 < (1<<18)) /heuristic : longer distance likely too expensive/ ) {
	size_t mlen;
	if ((dictMode == ZSTD_noDict) /static/ \|\| (dictMode == ZSTD_dictMatchState) /static/ \|\| (matchIndex3 >= dictLimit)) {
	const BYTE* const match = base + matchIndex3;
	mlen = ZSTD_count(ip, match, iLimit);
	} else {
	const BYTE* const match = dictBase + matchIndex3;
	mlen = ZSTD_count_2segments(ip, match, iLimit, dictEnd, prefixStart);
	}

	/* save best solution */
	if (mlen >= mls /* == 3 > bestLength */) {
	DEBUGLOG(8, "found small match with hlog3, of length %u",
	(U32)mlen);
	bestLength = mlen;
	assert(current > matchIndex3);
	assert(mnum==0); /* no prior solution */
	matches[0].off = (current - matchIndex3) + ZSTD_REP_MOVE;
	matches[0].len = (U32)mlen;
	mnum = 1;
	if ( (mlen > sufficient_len) \|
	(ip+mlen == iLimit) ) { /* best possible length */
	ms->nextToUpdate = current+1; /* skip insertion */
	return 1;
	} } }
	/* no dictMatchState lookup: dicts don't have a populated HC3 table */
	}

	hashTable[h] = current; /* Update Hash Table */

	while (nbCompares-- && (matchIndex >= matchLow)) {
	U32* const nextPtr = bt + 2*(matchIndex & btMask);
	const BYTE* match;
	size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */
	assert(current > matchIndex);

	if ((dictMode == ZSTD_noDict) \|\| (dictMode == ZSTD_dictMatchState) \|\| (matchIndex+matchLength >= dictLimit)) {
	assert(matchIndex+matchLength >= dictLimit); /* ensure the condition is correct when !extDict */
	match = base + matchIndex;
	if (matchIndex >= dictLimit) assert(memcmp(match, ip, matchLength) == 0); /* ensure early section of match is equal as expected */
	matchLength += ZSTD_count(ip+matchLength, match+matchLength, iLimit);
	} else {
	match = dictBase + matchIndex;
	assert(memcmp(match, ip, matchLength) == 0); /* ensure early section of match is equal as expected */
	matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iLimit, dictEnd, prefixStart);
	if (matchIndex+matchLength >= dictLimit)
	match = base + matchIndex; /* prepare for match[matchLength] read */
	}

	if (matchLength > bestLength) {
	DEBUGLOG(8, "found match of length %u at distance %u (offCode=%u)",
	(U32)matchLength, current - matchIndex, current - matchIndex + ZSTD_REP_MOVE);
	assert(matchEndIdx > matchIndex);
	if (matchLength > matchEndIdx - matchIndex)
	matchEndIdx = matchIndex + (U32)matchLength;
	bestLength = matchLength;
	matches[mnum].off = (current - matchIndex) + ZSTD_REP_MOVE;
	matches[mnum].len = (U32)matchLength;
	mnum++;
	if ( (matchLength > ZSTD_OPT_NUM)
	\| (ip+matchLength == iLimit) /* equal : no way to know if inf or sup */) {
	if (dictMode == ZSTD_dictMatchState) nbCompares = 0; /* break should also skip searching dms */
	break; /* drop, to preserve bt consistency (miss a little bit of compression) */
	}
	}

	if (match[matchLength] < ip[matchLength]) {
	/* match smaller than current */
	smallerPtr = matchIndex; / update smaller idx */
	commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */
	if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop the search */
	smallerPtr = nextPtr+1; /* new candidate => larger than match, which was smaller than current */
	matchIndex = nextPtr[1]; /* new matchIndex, larger than previous, closer to current */
	} else {
	*largerPtr = matchIndex;
	commonLengthLarger = matchLength;
	if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop the search */
	largerPtr = nextPtr;
	matchIndex = nextPtr[0];
	} }

	smallerPtr = largerPtr = 0;

	if (dictMode == ZSTD_dictMatchState && nbCompares) {
	size_t const dmsH = ZSTD_hashPtr(ip, dmsHashLog, mls);
	U32 dictMatchIndex = dms->hashTable[dmsH];
	const U32* const dmsBt = dms->chainTable;
	commonLengthSmaller = commonLengthLarger = 0;
	while (nbCompares-- && (dictMatchIndex > dmsLowLimit)) {
	const U32* const nextPtr = dmsBt + 2*(dictMatchIndex & dmsBtMask);
	size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */
	const BYTE* match = dmsBase + dictMatchIndex;
	matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iLimit, dmsEnd, prefixStart);
	if (dictMatchIndex+matchLength >= dmsHighLimit)
	match = base + dictMatchIndex + dmsIndexDelta; /* to prepare for next usage of match[matchLength] */

	if (matchLength > bestLength) {
	matchIndex = dictMatchIndex + dmsIndexDelta;
	DEBUGLOG(8, "found dms match of length %u at distance %u (offCode=%u)",
	(U32)matchLength, current - matchIndex, current - matchIndex + ZSTD_REP_MOVE);
	if (matchLength > matchEndIdx - matchIndex)
	matchEndIdx = matchIndex + (U32)matchLength;
	bestLength = matchLength;
	matches[mnum].off = (current - matchIndex) + ZSTD_REP_MOVE;
	matches[mnum].len = (U32)matchLength;
	mnum++;
	if ( (matchLength > ZSTD_OPT_NUM)
	\| (ip+matchLength == iLimit) /* equal : no way to know if inf or sup */) {
	break; /* drop, to guarantee consistency (miss a little bit of compression) */
	}
	}

	if (dictMatchIndex <= dmsBtLow) { break; } /* beyond tree size, stop the search */
	if (match[matchLength] < ip[matchLength]) {
	commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */
	dictMatchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to current) */
	} else {
	/* match is larger than current */
	commonLengthLarger = matchLength;
	dictMatchIndex = nextPtr[0];
	}
	}
	}

	assert(matchEndIdx > current+8);
	ms->nextToUpdate = matchEndIdx - 8; /* skip repetitive patterns */
	return mnum;
	}


	FORCE_INLINE_TEMPLATE U32 ZSTD_BtGetAllMatches (
	ZSTD_match_t* matches, /* store result (match found, increasing size) in this table */
	ZSTD_matchState_t* ms,
	U32* nextToUpdate3,
	const BYTE* ip, const BYTE* const iHighLimit, const ZSTD_dictMode_e dictMode,
	const U32 rep[ZSTD_REP_NUM],
	U32 const ll0,
	U32 const lengthToBeat)
	{
	const ZSTD_compressionParameters* const cParams = &ms->cParams;
	U32 const matchLengthSearch = cParams->minMatch;
	DEBUGLOG(8, "ZSTD_BtGetAllMatches");
	if (ip < ms->window.base + ms->nextToUpdate) return 0; /* skipped area */
	ZSTD_updateTree_internal(ms, ip, iHighLimit, matchLengthSearch, dictMode);
	switch(matchLengthSearch)
	{
	case 3 : return ZSTD_insertBtAndGetAllMatches(matches, ms, nextToUpdate3, ip, iHighLimit, dictMode, rep, ll0, lengthToBeat, 3);
	default :
	case 4 : return ZSTD_insertBtAndGetAllMatches(matches, ms, nextToUpdate3, ip, iHighLimit, dictMode, rep, ll0, lengthToBeat, 4);
	case 5 : return ZSTD_insertBtAndGetAllMatches(matches, ms, nextToUpdate3, ip, iHighLimit, dictMode, rep, ll0, lengthToBeat, 5);
	case 7 :
	case 6 : return ZSTD_insertBtAndGetAllMatches(matches, ms, nextToUpdate3, ip, iHighLimit, dictMode, rep, ll0, lengthToBeat, 6);
	}
	}


	/-******************************
	* Optimal parser
	*********************************/


	static U32 ZSTD_totalLen(ZSTD_optimal_t sol)
	{
	return sol.litlen + sol.mlen;
	}

	#if 0 /* debug */

	static void
	listStats(const U32* table, int lastEltID)
	{
	int const nbElts = lastEltID + 1;
	int enb;
	for (enb=0; enb < nbElts; enb++) {
	(void)table;
	/* RAWLOG(2, "%3i:%3i, ", enb, table[enb]); */
	RAWLOG(2, "%4i,", table[enb]);
	}
	RAWLOG(2, " \n");
	}

	#endif

	FORCE_INLINE_TEMPLATE size_t
	ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
	seqStore_t* seqStore,
	U32 rep[ZSTD_REP_NUM],
	const void* src, size_t srcSize,
	const int optLevel,
	const ZSTD_dictMode_e dictMode)
	{
	optState_t* const optStatePtr = &ms->opt;
	const BYTE* const istart = (const BYTE*)src;
	const BYTE* ip = istart;
	const BYTE* anchor = istart;
	const BYTE* const iend = istart + srcSize;
	const BYTE* const ilimit = iend - 8;
	const BYTE* const base = ms->window.base;
	const BYTE* const prefixStart = base + ms->window.dictLimit;
	const ZSTD_compressionParameters* const cParams = &ms->cParams;

	U32 const sufficient_len = MIN(cParams->targetLength, ZSTD_OPT_NUM -1);
	U32 const minMatch = (cParams->minMatch == 3) ? 3 : 4;
	U32 nextToUpdate3 = ms->nextToUpdate;

	ZSTD_optimal_t* const opt = optStatePtr->priceTable;
	ZSTD_match_t* const matches = optStatePtr->matchTable;
	ZSTD_optimal_t lastSequence;

	/* init */
	DEBUGLOG(5, "ZSTD_compressBlock_opt_generic: current=%u, prefix=%u, nextToUpdate=%u",
	(U32)(ip - base), ms->window.dictLimit, ms->nextToUpdate);
	assert(optLevel <= 2);
	ZSTD_rescaleFreqs(optStatePtr, (const BYTE*)src, srcSize, optLevel);
	ip += (ip==prefixStart);

	/* Match Loop */
	while (ip < ilimit) {
	U32 cur, last_pos = 0;

	/* find first match */
	{ U32 const litlen = (U32)(ip - anchor);
	U32 const ll0 = !litlen;
	U32 const nbMatches = ZSTD_BtGetAllMatches(matches, ms, &nextToUpdate3, ip, iend, dictMode, rep, ll0, minMatch);
	if (!nbMatches) { ip++; continue; }

	/* initialize opt[0] */
	{ U32 i ; for (i=0; i<ZSTD_REP_NUM; i++) opt[0].rep[i] = rep[i]; }
	opt[0].mlen = 0; /* means is_a_literal */
	opt[0].litlen = litlen;
	/* We don't need to include the actual price of the literals because
	* it is static for the duration of the forward pass, and is included
	* in every price. We include the literal length to avoid negative
	* prices when we subtract the previous literal length.
	*/
	opt[0].price = ZSTD_litLengthPrice(litlen, optStatePtr, optLevel);

	/* large match -> immediate encoding */
	{ U32 const maxML = matches[nbMatches-1].len;
	U32 const maxOffset = matches[nbMatches-1].off;
	DEBUGLOG(6, "found %u matches of maxLength=%u and maxOffCode=%u at cPos=%u => start new series",
	nbMatches, maxML, maxOffset, (U32)(ip-prefixStart));

	if (maxML > sufficient_len) {
	lastSequence.litlen = litlen;
	lastSequence.mlen = maxML;
	lastSequence.off = maxOffset;
	DEBUGLOG(6, "large match (%u>%u), immediate encoding",
	maxML, sufficient_len);
	cur = 0;
	last_pos = ZSTD_totalLen(lastSequence);
	goto _shortestPath;
	} }

	/* set prices for first matches starting position == 0 */
	{ U32 const literalsPrice = opt[0].price + ZSTD_litLengthPrice(0, optStatePtr, optLevel);
	U32 pos;
	U32 matchNb;
	for (pos = 1; pos < minMatch; pos++) {
	opt[pos].price = ZSTD_MAX_PRICE; /* mlen, litlen and price will be fixed during forward scanning */
	}
	for (matchNb = 0; matchNb < nbMatches; matchNb++) {
	U32 const offset = matches[matchNb].off;
	U32 const end = matches[matchNb].len;
	for ( ; pos <= end ; pos++ ) {
	U32 const matchPrice = ZSTD_getMatchPrice(offset, pos, optStatePtr, optLevel);
	U32 const sequencePrice = literalsPrice + matchPrice;
	DEBUGLOG(7, "rPos:%u => set initial price : %.2f",
	pos, ZSTD_fCost(sequencePrice));
	opt[pos].mlen = pos;
	opt[pos].off = offset;
	opt[pos].litlen = litlen;
	opt[pos].price = sequencePrice;
	} }
	last_pos = pos-1;
	}
	}

	/* check further positions */
	for (cur = 1; cur <= last_pos; cur++) {
	const BYTE* const inr = ip + cur;
	assert(cur < ZSTD_OPT_NUM);
	DEBUGLOG(7, "cPos:%zi==rPos:%u", inr-istart, cur)

	/* Fix current position with one literal if cheaper */
	{ U32 const litlen = (opt[cur-1].mlen == 0) ? opt[cur-1].litlen + 1 : 1;
	int const price = opt[cur-1].price
	+ ZSTD_rawLiteralsCost(ip+cur-1, 1, optStatePtr, optLevel)
	+ ZSTD_litLengthPrice(litlen, optStatePtr, optLevel)
	- ZSTD_litLengthPrice(litlen-1, optStatePtr, optLevel);
	assert(price < 1000000000); /* overflow check */
	if (price <= opt[cur].price) {
	DEBUGLOG(7, "cPos:%zi==rPos:%u : better price (%.2f<=%.2f) using literal (ll==%u) (hist:%u,%u,%u)",
	inr-istart, cur, ZSTD_fCost(price), ZSTD_fCost(opt[cur].price), litlen,
	opt[cur-1].rep[0], opt[cur-1].rep[1], opt[cur-1].rep[2]);
	opt[cur].mlen = 0;
	opt[cur].off = 0;
	opt[cur].litlen = litlen;
	opt[cur].price = price;
	} else {
	DEBUGLOG(7, "cPos:%zi==rPos:%u : literal would cost more (%.2f>%.2f) (hist:%u,%u,%u)",
	inr-istart, cur, ZSTD_fCost(price), ZSTD_fCost(opt[cur].price),
	opt[cur].rep[0], opt[cur].rep[1], opt[cur].rep[2]);
	}
	}

	/* Set the repcodes of the current position. We must do it here
	* because we rely on the repcodes of the 2nd to last sequence being
	* correct to set the next chunks repcodes during the backward
	* traversal.
	*/
	ZSTD_STATIC_ASSERT(sizeof(opt[cur].rep) == sizeof(repcodes_t));
	assert(cur >= opt[cur].mlen);
	if (opt[cur].mlen != 0) {
	U32 const prev = cur - opt[cur].mlen;
	repcodes_t newReps = ZSTD_updateRep(opt[prev].rep, opt[cur].off, opt[cur].litlen==0);
	memcpy(opt[cur].rep, &newReps, sizeof(repcodes_t));
	} else {
	memcpy(opt[cur].rep, opt[cur - 1].rep, sizeof(repcodes_t));
	}

	/* last match must start at a minimum distance of 8 from oend */
	if (inr > ilimit) continue;

	if (cur == last_pos) break;

	if ( (optLevel==0) /static_test/
	&& (opt[cur+1].price <= opt[cur].price + (BITCOST_MULTIPLIER/2)) ) {
	DEBUGLOG(7, "move to next rPos:%u : price is <=", cur+1);
	continue; /* skip unpromising positions; about ~+6% speed, -0.01 ratio */
	}

	{ U32 const ll0 = (opt[cur].mlen != 0);
	U32 const litlen = (opt[cur].mlen == 0) ? opt[cur].litlen : 0;
	U32 const previousPrice = opt[cur].price;
	U32 const basePrice = previousPrice + ZSTD_litLengthPrice(0, optStatePtr, optLevel);
	U32 const nbMatches = ZSTD_BtGetAllMatches(matches, ms, &nextToUpdate3, inr, iend, dictMode, opt[cur].rep, ll0, minMatch);
	U32 matchNb;
	if (!nbMatches) {
	DEBUGLOG(7, "rPos:%u : no match found", cur);
	continue;
	}

	{ U32 const maxML = matches[nbMatches-1].len;
	DEBUGLOG(7, "cPos:%zi==rPos:%u, found %u matches, of maxLength=%u",
	inr-istart, cur, nbMatches, maxML);

	if ( (maxML > sufficient_len)
	\|\| (cur + maxML >= ZSTD_OPT_NUM) ) {
	lastSequence.mlen = maxML;
	lastSequence.off = matches[nbMatches-1].off;
	lastSequence.litlen = litlen;
	cur -= (opt[cur].mlen==0) ? opt[cur].litlen : 0; /* last sequence is actually only literals, fix cur to last match - note : may underflow, in which case, it's first sequence, and it's okay */
	last_pos = cur + ZSTD_totalLen(lastSequence);
	if (cur > ZSTD_OPT_NUM) cur = 0; /* underflow => first match */
	goto _shortestPath;
	} }

	/* set prices using matches found at position == cur */
	for (matchNb = 0; matchNb < nbMatches; matchNb++) {
	U32 const offset = matches[matchNb].off;
	U32 const lastML = matches[matchNb].len;
	U32 const startML = (matchNb>0) ? matches[matchNb-1].len+1 : minMatch;
	U32 mlen;

	DEBUGLOG(7, "testing match %u => offCode=%4u, mlen=%2u, llen=%2u",
	matchNb, matches[matchNb].off, lastML, litlen);

	for (mlen = lastML; mlen >= startML; mlen--) { /* scan downward */
	U32 const pos = cur + mlen;
	int const price = basePrice + ZSTD_getMatchPrice(offset, mlen, optStatePtr, optLevel);

	if ((pos > last_pos) \|\| (price < opt[pos].price)) {
	DEBUGLOG(7, "rPos:%u (ml=%2u) => new better price (%.2f<%.2f)",
	pos, mlen, ZSTD_fCost(price), ZSTD_fCost(opt[pos].price));
	while (last_pos < pos) { opt[last_pos+1].price = ZSTD_MAX_PRICE; last_pos++; } /* fill empty positions */
	opt[pos].mlen = mlen;
	opt[pos].off = offset;
	opt[pos].litlen = litlen;
	opt[pos].price = price;
	} else {
	DEBUGLOG(7, "rPos:%u (ml=%2u) => new price is worse (%.2f>=%.2f)",
	pos, mlen, ZSTD_fCost(price), ZSTD_fCost(opt[pos].price));
	if (optLevel==0) break; /* early update abort; gets ~+10% speed for about -0.01 ratio loss */
	}
	} } }
	} /* for (cur = 1; cur <= last_pos; cur++) */

	lastSequence = opt[last_pos];
	cur = last_pos > ZSTD_totalLen(lastSequence) ? last_pos - ZSTD_totalLen(lastSequence) : 0; /* single sequence, and it starts before `ip` */
	assert(cur < ZSTD_OPT_NUM); /* control overflow*/

	_shortestPath: /* cur, last_pos, best_mlen, best_off have to be set */
	assert(opt[0].mlen == 0);

	/* Set the next chunk's repcodes based on the repcodes of the beginning
	* of the last match, and the last sequence. This avoids us having to
	* update them while traversing the sequences.
	*/
	if (lastSequence.mlen != 0) {
	repcodes_t reps = ZSTD_updateRep(opt[cur].rep, lastSequence.off, lastSequence.litlen==0);
	memcpy(rep, &reps, sizeof(reps));
	} else {
	memcpy(rep, opt[cur].rep, sizeof(repcodes_t));
	}

	{ U32 const storeEnd = cur + 1;
	U32 storeStart = storeEnd;
	U32 seqPos = cur;

	DEBUGLOG(6, "start reverse traversal (last_pos:%u, cur:%u)",
	last_pos, cur); (void)last_pos;
	assert(storeEnd < ZSTD_OPT_NUM);
	DEBUGLOG(6, "last sequence copied into pos=%u (llen=%u,mlen=%u,ofc=%u)",
	storeEnd, lastSequence.litlen, lastSequence.mlen, lastSequence.off);
	opt[storeEnd] = lastSequence;
	while (seqPos > 0) {
	U32 const backDist = ZSTD_totalLen(opt[seqPos]);
	storeStart--;
	DEBUGLOG(6, "sequence from rPos=%u copied into pos=%u (llen=%u,mlen=%u,ofc=%u)",
	seqPos, storeStart, opt[seqPos].litlen, opt[seqPos].mlen, opt[seqPos].off);
	opt[storeStart] = opt[seqPos];
	seqPos = (seqPos > backDist) ? seqPos - backDist : 0;
	}

	/* save sequences */
	DEBUGLOG(6, "sending selected sequences into seqStore")
	{ U32 storePos;
	for (storePos=storeStart; storePos <= storeEnd; storePos++) {
	U32 const llen = opt[storePos].litlen;
	U32 const mlen = opt[storePos].mlen;
	U32 const offCode = opt[storePos].off;
	U32 const advance = llen + mlen;
	DEBUGLOG(6, "considering seq starting at %zi, llen=%u, mlen=%u",
	anchor - istart, (unsigned)llen, (unsigned)mlen);

	if (mlen==0) { /* only literals => must be last "sequence", actually starting a new stream of sequences */
	assert(storePos == storeEnd); /* must be last sequence */
	ip = anchor + llen; /* last "sequence" is a bunch of literals => don't progress anchor */
	continue; /* will finish */
	}

	assert(anchor + llen <= iend);
	ZSTD_updateStats(optStatePtr, llen, anchor, offCode, mlen);
	ZSTD_storeSeq(seqStore, llen, anchor, iend, offCode, mlen-MINMATCH);
	anchor += advance;
	ip = anchor;
	} }
	ZSTD_setBasePrices(optStatePtr, optLevel);
	}
	} /* while (ip < ilimit) */

	/* Return the last literals size */
	return (size_t)(iend - anchor);
	}


	size_t ZSTD_compressBlock_btopt(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	const void* src, size_t srcSize)
	{
	DEBUGLOG(5, "ZSTD_compressBlock_btopt");
	return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 0 /optLevel/, ZSTD_noDict);
	}


	/* used in 2-pass strategy */
	static U32 ZSTD_upscaleStat(unsigned* table, U32 lastEltIndex, int bonus)
	{
	U32 s, sum=0;
	assert(ZSTD_FREQ_DIV+bonus >= 0);
	for (s=0; s<lastEltIndex+1; s++) {
	table[s] <<= ZSTD_FREQ_DIV+bonus;
	table[s]--;
	sum += table[s];
	}
	return sum;
	}

	/* used in 2-pass strategy */
	MEM_STATIC void ZSTD_upscaleStats(optState_t* optPtr)
	{
	if (ZSTD_compressedLiterals(optPtr))
	optPtr->litSum = ZSTD_upscaleStat(optPtr->litFreq, MaxLit, 0);
	optPtr->litLengthSum = ZSTD_upscaleStat(optPtr->litLengthFreq, MaxLL, 0);
	optPtr->matchLengthSum = ZSTD_upscaleStat(optPtr->matchLengthFreq, MaxML, 0);
	optPtr->offCodeSum = ZSTD_upscaleStat(optPtr->offCodeFreq, MaxOff, 0);
	}

	/* ZSTD_initStats_ultra():
	* make a first compression pass, just to seed stats with more accurate starting values.
	* only works on first block, with no dictionary and no ldm.
	* this function cannot error, hence its contract must be respected.
	*/
	static void
	ZSTD_initStats_ultra(ZSTD_matchState_t* ms,
	seqStore_t* seqStore,
	U32 rep[ZSTD_REP_NUM],
	const void* src, size_t srcSize)
	{
	U32 tmpRep[ZSTD_REP_NUM]; /* updated rep codes will sink here */
	memcpy(tmpRep, rep, sizeof(tmpRep));

	DEBUGLOG(4, "ZSTD_initStats_ultra (srcSize=%zu)", srcSize);
	assert(ms->opt.litLengthSum == 0); /* first block */
	assert(seqStore->sequences == seqStore->sequencesStart); /* no ldm */
	assert(ms->window.dictLimit == ms->window.lowLimit); /* no dictionary */
	assert(ms->window.dictLimit - ms->nextToUpdate <= 1); /* no prefix (note: intentional overflow, defined as 2-complement) */

	ZSTD_compressBlock_opt_generic(ms, seqStore, tmpRep, src, srcSize, 2 /optLevel/, ZSTD_noDict); /* generate stats into ms->opt*/

	/* invalidate first scan from history */
	ZSTD_resetSeqStore(seqStore);
	ms->window.base -= srcSize;
	ms->window.dictLimit += (U32)srcSize;
	ms->window.lowLimit = ms->window.dictLimit;
	ms->nextToUpdate = ms->window.dictLimit;

	/* re-inforce weight of collected statistics */
	ZSTD_upscaleStats(&ms->opt);
	}

	size_t ZSTD_compressBlock_btultra(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	const void* src, size_t srcSize)
	{
	DEBUGLOG(5, "ZSTD_compressBlock_btultra (srcSize=%zu)", srcSize);
	return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 2 /optLevel/, ZSTD_noDict);
	}

	size_t ZSTD_compressBlock_btultra2(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	const void* src, size_t srcSize)
	{
	U32 const current = (U32)((const BYTE*)src - ms->window.base);
	DEBUGLOG(5, "ZSTD_compressBlock_btultra2 (srcSize=%zu)", srcSize);

	/* 2-pass strategy:
	* this strategy makes a first pass over first block to collect statistics
	* and seed next round's statistics with it.
	* After 1st pass, function forgets everything, and starts a new block.
	* Consequently, this can only work if no data has been previously loaded in tables,
	* aka, no dictionary, no prefix, no ldm preprocessing.
	* The compression ratio gain is generally small (~0.5% on first block),
	* the cost is 2x cpu time on first block. */
	assert(srcSize <= ZSTD_BLOCKSIZE_MAX);
	if ( (ms->opt.litLengthSum==0) /* first block */
	&& (seqStore->sequences == seqStore->sequencesStart) /* no ldm */
	&& (ms->window.dictLimit == ms->window.lowLimit) /* no dictionary */
	&& (current == ms->window.dictLimit) /* start of frame, nothing already loaded nor skipped */
	&& (srcSize > ZSTD_PREDEF_THRESHOLD)
	) {
	ZSTD_initStats_ultra(ms, seqStore, rep, src, srcSize);
	}

	return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 2 /optLevel/, ZSTD_noDict);
	}

	size_t ZSTD_compressBlock_btopt_dictMatchState(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	const void* src, size_t srcSize)
	{
	return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 0 /optLevel/, ZSTD_dictMatchState);
	}

	size_t ZSTD_compressBlock_btultra_dictMatchState(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	const void* src, size_t srcSize)
	{
	return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 2 /optLevel/, ZSTD_dictMatchState);
	}

	size_t ZSTD_compressBlock_btopt_extDict(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	const void* src, size_t srcSize)
	{
	return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 0 /optLevel/, ZSTD_extDict);
	}

	size_t ZSTD_compressBlock_btultra_extDict(
	ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
	const void* src, size_t srcSize)
	{
	return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 2 /optLevel/, ZSTD_extDict);
	}

	/* note : no btultra2 variant for extDict nor dictMatchState,
	* because btultra2 is not meant to work with dictionaries
	* and is only specific for the first block (no prefix) */
	/** ended inlining compress/zstd_opt.c **/

	/** start inlining decompress/huf_decompress.c **/
	/* ******************************************************************
	* huff0 huffman decoder,
	* part of Finite State Entropy library
	* Copyright (c) 2013-2020, Yann Collet, Facebook, Inc.
	*
	* You can contact the author at :
	* - FSE+HUF source repository : https://github.com/Cyan4973/FiniteStateEntropy
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	****************************************************************** */

	/* **************************************************************
	* Dependencies
	****************************************************************/
	#include <string.h> /* memcpy, memset */
	/** skipping file: ../common/compiler.h **/
	/** skipping file: ../common/bitstream.h **/
	/** skipping file: ../common/fse.h **/
	#define HUF_STATIC_LINKING_ONLY
	/** skipping file: ../common/huf.h **/
	/** skipping file: ../common/error_private.h **/

	/* **************************************************************
	* Macros
	****************************************************************/

	/* These two optional macros force the use one way or another of the two
	* Huffman decompression implementations. You can't force in both directions
	* at the same time.
	*/
	#if defined(HUF_FORCE_DECOMPRESS_X1) && \
	defined(HUF_FORCE_DECOMPRESS_X2)
	#error "Cannot force the use of the X1 and X2 decoders at the same time!"
	#endif


	/* **************************************************************
	* Error Management
	****************************************************************/
	#define HUF_isError ERR_isError


	/* **************************************************************
	* Byte alignment for workSpace management
	****************************************************************/
	#define HUF_ALIGN(x, a) HUF_ALIGN_MASK((x), (a) - 1)
	#define HUF_ALIGN_MASK(x, mask) (((x) + (mask)) & ~(mask))


	/* **************************************************************
	* BMI2 Variant Wrappers
	****************************************************************/
	#if DYNAMIC_BMI2

	#define HUF_DGEN(fn) \
	\
	static size_t fn##_default( \
	void* dst, size_t dstSize, \
	const void* cSrc, size_t cSrcSize, \
	const HUF_DTable* DTable) \
	{ \
	return fn##_body(dst, dstSize, cSrc, cSrcSize, DTable); \
	} \
	\
	static TARGET_ATTRIBUTE("bmi2") size_t fn##_bmi2( \
	void* dst, size_t dstSize, \
	const void* cSrc, size_t cSrcSize, \
	const HUF_DTable* DTable) \
	{ \
	return fn##_body(dst, dstSize, cSrc, cSrcSize, DTable); \
	} \
	\
	static size_t fn(void* dst, size_t dstSize, void const* cSrc, \
	size_t cSrcSize, HUF_DTable const* DTable, int bmi2) \
	{ \
	if (bmi2) { \
	return fn##_bmi2(dst, dstSize, cSrc, cSrcSize, DTable); \
	} \
	return fn##_default(dst, dstSize, cSrc, cSrcSize, DTable); \
	}

	#else

	#define HUF_DGEN(fn) \
	static size_t fn(void* dst, size_t dstSize, void const* cSrc, \
	size_t cSrcSize, HUF_DTable const* DTable, int bmi2) \
	{ \
	(void)bmi2; \
	return fn##_body(dst, dstSize, cSrc, cSrcSize, DTable); \
	}

	#endif


	/-**************************/
	/* generic DTableDesc */
	/-**************************/
	typedef struct { BYTE maxTableLog; BYTE tableType; BYTE tableLog; BYTE reserved; } DTableDesc;

	static DTableDesc HUF_getDTableDesc(const HUF_DTable* table)
	{
	DTableDesc dtd;
	memcpy(&dtd, table, sizeof(dtd));
	return dtd;
	}


	#ifndef HUF_FORCE_DECOMPRESS_X2

	/-**************************/
	/* single-symbol decoding */
	/-**************************/
	typedef struct { BYTE byte; BYTE nbBits; } HUF_DEltX1; /* single-symbol decoding */

	size_t HUF_readDTableX1_wksp(HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize)
	{
	U32 tableLog = 0;
	U32 nbSymbols = 0;
	size_t iSize;
	void* const dtPtr = DTable + 1;
	HUF_DEltX1* const dt = (HUF_DEltX1*)dtPtr;

	U32* rankVal;
	BYTE* huffWeight;
	size_t spaceUsed32 = 0;

	rankVal = (U32 *)workSpace + spaceUsed32;
	spaceUsed32 += HUF_TABLELOG_ABSOLUTEMAX + 1;
	huffWeight = (BYTE )((U32 )workSpace + spaceUsed32);
	spaceUsed32 += HUF_ALIGN(HUF_SYMBOLVALUE_MAX + 1, sizeof(U32)) >> 2;

	if ((spaceUsed32 << 2) > wkspSize) return ERROR(tableLog_tooLarge);

	DEBUG_STATIC_ASSERT(sizeof(DTableDesc) == sizeof(HUF_DTable));
	/* memset(huffWeight, 0, sizeof(huffWeight)); / / is not necessary, even though some analyzer complain ... */

	iSize = HUF_readStats(huffWeight, HUF_SYMBOLVALUE_MAX + 1, rankVal, &nbSymbols, &tableLog, src, srcSize);
	if (HUF_isError(iSize)) return iSize;

	/* Table header */
	{ DTableDesc dtd = HUF_getDTableDesc(DTable);
	if (tableLog > (U32)(dtd.maxTableLog+1)) return ERROR(tableLog_tooLarge); /* DTable too small, Huffman tree cannot fit in */
	dtd.tableType = 0;
	dtd.tableLog = (BYTE)tableLog;
	memcpy(DTable, &dtd, sizeof(dtd));
	}

	/* Calculate starting value for each rank */
	{ U32 n, nextRankStart = 0;
	for (n=1; n<tableLog+1; n++) {
	U32 const current = nextRankStart;
	nextRankStart += (rankVal[n] << (n-1));
	rankVal[n] = current;
	} }

	/* fill DTable */
	{ U32 n;
	size_t const nEnd = nbSymbols;
	for (n=0; n<nEnd; n++) {
	size_t const w = huffWeight[n];
	size_t const length = (1 << w) >> 1;
	size_t const uStart = rankVal[w];
	size_t const uEnd = uStart + length;
	size_t u;
	HUF_DEltX1 D;
	D.byte = (BYTE)n;
	D.nbBits = (BYTE)(tableLog + 1 - w);
	rankVal[w] = (U32)uEnd;
	if (length < 4) {
	/* Use length in the loop bound so the compiler knows it is short. */
	for (u = 0; u < length; ++u)
	dt[uStart + u] = D;
	} else {
	/* Unroll the loop 4 times, we know it is a power of 2. */
	for (u = uStart; u < uEnd; u += 4) {
	dt[u + 0] = D;
	dt[u + 1] = D;
	dt[u + 2] = D;
	dt[u + 3] = D;
	} } } }
	return iSize;
	}

	size_t HUF_readDTableX1(HUF_DTable* DTable, const void* src, size_t srcSize)
	{
	U32 workSpace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32];
	return HUF_readDTableX1_wksp(DTable, src, srcSize,
	workSpace, sizeof(workSpace));
	}

	FORCE_INLINE_TEMPLATE BYTE
	HUF_decodeSymbolX1(BIT_DStream_t* Dstream, const HUF_DEltX1* dt, const U32 dtLog)
	{
	size_t const val = BIT_lookBitsFast(Dstream, dtLog); /* note : dtLog >= 1 */
	BYTE const c = dt[val].byte;
	BIT_skipBits(Dstream, dt[val].nbBits);
	return c;
	}

	#define HUF_DECODE_SYMBOLX1_0(ptr, DStreamPtr) \
	*ptr++ = HUF_decodeSymbolX1(DStreamPtr, dt, dtLog)

	#define HUF_DECODE_SYMBOLX1_1(ptr, DStreamPtr) \
	if (MEM_64bits() \|\| (HUF_TABLELOG_MAX<=12)) \
	HUF_DECODE_SYMBOLX1_0(ptr, DStreamPtr)

	#define HUF_DECODE_SYMBOLX1_2(ptr, DStreamPtr) \
	if (MEM_64bits()) \
	HUF_DECODE_SYMBOLX1_0(ptr, DStreamPtr)

	HINT_INLINE size_t
	HUF_decodeStreamX1(BYTE* p, BIT_DStream_t* const bitDPtr, BYTE* const pEnd, const HUF_DEltX1* const dt, const U32 dtLog)
	{
	BYTE* const pStart = p;

	/* up to 4 symbols at a time */
	while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p < pEnd-3)) {
	HUF_DECODE_SYMBOLX1_2(p, bitDPtr);
	HUF_DECODE_SYMBOLX1_1(p, bitDPtr);
	HUF_DECODE_SYMBOLX1_2(p, bitDPtr);
	HUF_DECODE_SYMBOLX1_0(p, bitDPtr);
	}

	/* [0-3] symbols remaining */
	if (MEM_32bits())
	while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p < pEnd))
	HUF_DECODE_SYMBOLX1_0(p, bitDPtr);

	/* no more data to retrieve from bitstream, no need to reload */
	while (p < pEnd)
	HUF_DECODE_SYMBOLX1_0(p, bitDPtr);

	return pEnd-pStart;
	}

	FORCE_INLINE_TEMPLATE size_t
	HUF_decompress1X1_usingDTable_internal_body(
	void* dst, size_t dstSize,
	const void* cSrc, size_t cSrcSize,
	const HUF_DTable* DTable)
	{
	BYTE* op = (BYTE*)dst;
	BYTE* const oend = op + dstSize;
	const void* dtPtr = DTable + 1;
	const HUF_DEltX1* const dt = (const HUF_DEltX1*)dtPtr;
	BIT_DStream_t bitD;
	DTableDesc const dtd = HUF_getDTableDesc(DTable);
	U32 const dtLog = dtd.tableLog;

	CHECK_F( BIT_initDStream(&bitD, cSrc, cSrcSize) );

	HUF_decodeStreamX1(op, &bitD, oend, dt, dtLog);

	if (!BIT_endOfDStream(&bitD)) return ERROR(corruption_detected);

	return dstSize;
	}

	FORCE_INLINE_TEMPLATE size_t
	HUF_decompress4X1_usingDTable_internal_body(
	void* dst, size_t dstSize,
	const void* cSrc, size_t cSrcSize,
	const HUF_DTable* DTable)
	{
	/* Check */
	if (cSrcSize < 10) return ERROR(corruption_detected); /* strict minimum : jump table + 1 byte per stream */

	{ const BYTE* const istart = (const BYTE*) cSrc;
	BYTE* const ostart = (BYTE*) dst;
	BYTE* const oend = ostart + dstSize;
	BYTE* const olimit = oend - 3;
	const void* const dtPtr = DTable + 1;
	const HUF_DEltX1* const dt = (const HUF_DEltX1*)dtPtr;

	/* Init */
	BIT_DStream_t bitD1;
	BIT_DStream_t bitD2;
	BIT_DStream_t bitD3;
	BIT_DStream_t bitD4;
	size_t const length1 = MEM_readLE16(istart);
	size_t const length2 = MEM_readLE16(istart+2);
	size_t const length3 = MEM_readLE16(istart+4);
	size_t const length4 = cSrcSize - (length1 + length2 + length3 + 6);
	const BYTE* const istart1 = istart + 6; /* jumpTable */
	const BYTE* const istart2 = istart1 + length1;
	const BYTE* const istart3 = istart2 + length2;
	const BYTE* const istart4 = istart3 + length3;
	const size_t segmentSize = (dstSize+3) / 4;
	BYTE* const opStart2 = ostart + segmentSize;
	BYTE* const opStart3 = opStart2 + segmentSize;
	BYTE* const opStart4 = opStart3 + segmentSize;
	BYTE* op1 = ostart;
	BYTE* op2 = opStart2;
	BYTE* op3 = opStart3;
	BYTE* op4 = opStart4;
	DTableDesc const dtd = HUF_getDTableDesc(DTable);
	U32 const dtLog = dtd.tableLog;
	U32 endSignal = 1;

	if (length4 > cSrcSize) return ERROR(corruption_detected); /* overflow */
	CHECK_F( BIT_initDStream(&bitD1, istart1, length1) );
	CHECK_F( BIT_initDStream(&bitD2, istart2, length2) );
	CHECK_F( BIT_initDStream(&bitD3, istart3, length3) );
	CHECK_F( BIT_initDStream(&bitD4, istart4, length4) );

	/* up to 16 symbols per loop (4 symbols per stream) in 64-bit mode */
	for ( ; (endSignal) & (op4 < olimit) ; ) {
	HUF_DECODE_SYMBOLX1_2(op1, &bitD1);
	HUF_DECODE_SYMBOLX1_2(op2, &bitD2);
	HUF_DECODE_SYMBOLX1_2(op3, &bitD3);
	HUF_DECODE_SYMBOLX1_2(op4, &bitD4);
	HUF_DECODE_SYMBOLX1_1(op1, &bitD1);
	HUF_DECODE_SYMBOLX1_1(op2, &bitD2);
	HUF_DECODE_SYMBOLX1_1(op3, &bitD3);
	HUF_DECODE_SYMBOLX1_1(op4, &bitD4);
	HUF_DECODE_SYMBOLX1_2(op1, &bitD1);
	HUF_DECODE_SYMBOLX1_2(op2, &bitD2);
	HUF_DECODE_SYMBOLX1_2(op3, &bitD3);
	HUF_DECODE_SYMBOLX1_2(op4, &bitD4);
	HUF_DECODE_SYMBOLX1_0(op1, &bitD1);
	HUF_DECODE_SYMBOLX1_0(op2, &bitD2);
	HUF_DECODE_SYMBOLX1_0(op3, &bitD3);
	HUF_DECODE_SYMBOLX1_0(op4, &bitD4);
	endSignal &= BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished;
	endSignal &= BIT_reloadDStreamFast(&bitD2) == BIT_DStream_unfinished;
	endSignal &= BIT_reloadDStreamFast(&bitD3) == BIT_DStream_unfinished;
	endSignal &= BIT_reloadDStreamFast(&bitD4) == BIT_DStream_unfinished;
	}

	/* check corruption */
	/* note : should not be necessary : op# advance in lock step, and we control op4.
	* but curiously, binary generated by gcc 7.2 & 7.3 with -mbmi2 runs faster when >=1 test is present */
	if (op1 > opStart2) return ERROR(corruption_detected);
	if (op2 > opStart3) return ERROR(corruption_detected);
	if (op3 > opStart4) return ERROR(corruption_detected);
	/* note : op4 supposed already verified within main loop */

	/* finish bitStreams one by one */
	HUF_decodeStreamX1(op1, &bitD1, opStart2, dt, dtLog);
	HUF_decodeStreamX1(op2, &bitD2, opStart3, dt, dtLog);
	HUF_decodeStreamX1(op3, &bitD3, opStart4, dt, dtLog);
	HUF_decodeStreamX1(op4, &bitD4, oend, dt, dtLog);

	/* check */
	{ U32 const endCheck = BIT_endOfDStream(&bitD1) & BIT_endOfDStream(&bitD2) & BIT_endOfDStream(&bitD3) & BIT_endOfDStream(&bitD4);
	if (!endCheck) return ERROR(corruption_detected); }

	/* decoded size */
	return dstSize;
	}
	}


	typedef size_t (HUF_decompress_usingDTable_t)(void dst, size_t dstSize,
	const void *cSrc,
	size_t cSrcSize,
	const HUF_DTable *DTable);

	HUF_DGEN(HUF_decompress1X1_usingDTable_internal)
	HUF_DGEN(HUF_decompress4X1_usingDTable_internal)



	size_t HUF_decompress1X1_usingDTable(
	void* dst, size_t dstSize,
	const void* cSrc, size_t cSrcSize,
	const HUF_DTable* DTable)
	{
	DTableDesc dtd = HUF_getDTableDesc(DTable);
	if (dtd.tableType != 0) return ERROR(GENERIC);
	return HUF_decompress1X1_usingDTable_internal(dst, dstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
	}

	size_t HUF_decompress1X1_DCtx_wksp(HUF_DTable* DCtx, void* dst, size_t dstSize,
	const void* cSrc, size_t cSrcSize,
	void* workSpace, size_t wkspSize)
	{
	const BYTE* ip = (const BYTE*) cSrc;

	size_t const hSize = HUF_readDTableX1_wksp(DCtx, cSrc, cSrcSize, workSpace, wkspSize);
	if (HUF_isError(hSize)) return hSize;
	if (hSize >= cSrcSize) return ERROR(srcSize_wrong);
	ip += hSize; cSrcSize -= hSize;

	return HUF_decompress1X1_usingDTable_internal(dst, dstSize, ip, cSrcSize, DCtx, /* bmi2 */ 0);
	}


	size_t HUF_decompress1X1_DCtx(HUF_DTable* DCtx, void* dst, size_t dstSize,
	const void* cSrc, size_t cSrcSize)
	{
	U32 workSpace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32];
	return HUF_decompress1X1_DCtx_wksp(DCtx, dst, dstSize, cSrc, cSrcSize,
	workSpace, sizeof(workSpace));
	}

	size_t HUF_decompress1X1 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize)
	{
	HUF_CREATE_STATIC_DTABLEX1(DTable, HUF_TABLELOG_MAX);
	return HUF_decompress1X1_DCtx (DTable, dst, dstSize, cSrc, cSrcSize);
	}

	size_t HUF_decompress4X1_usingDTable(
	void* dst, size_t dstSize,
	const void* cSrc, size_t cSrcSize,
	const HUF_DTable* DTable)
	{
	DTableDesc dtd = HUF_getDTableDesc(DTable);
	if (dtd.tableType != 0) return ERROR(GENERIC);
	return HUF_decompress4X1_usingDTable_internal(dst, dstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
	}

	static size_t HUF_decompress4X1_DCtx_wksp_bmi2(HUF_DTable* dctx, void* dst, size_t dstSize,
	const void* cSrc, size_t cSrcSize,
	void* workSpace, size_t wkspSize, int bmi2)
	{
	const BYTE* ip = (const BYTE*) cSrc;

	size_t const hSize = HUF_readDTableX1_wksp (dctx, cSrc, cSrcSize,
	workSpace, wkspSize);
	if (HUF_isError(hSize)) return hSize;
	if (hSize >= cSrcSize) return ERROR(srcSize_wrong);
	ip += hSize; cSrcSize -= hSize;

	return HUF_decompress4X1_usingDTable_internal(dst, dstSize, ip, cSrcSize, dctx, bmi2);
	}

	size_t HUF_decompress4X1_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize,
	const void* cSrc, size_t cSrcSize,
	void* workSpace, size_t wkspSize)
	{
	return HUF_decompress4X1_DCtx_wksp_bmi2(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, 0);
	}


	size_t HUF_decompress4X1_DCtx (HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize)
	{
	U32 workSpace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32];
	return HUF_decompress4X1_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize,
	workSpace, sizeof(workSpace));
	}
	size_t HUF_decompress4X1 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize)
	{
	HUF_CREATE_STATIC_DTABLEX1(DTable, HUF_TABLELOG_MAX);
	return HUF_decompress4X1_DCtx(DTable, dst, dstSize, cSrc, cSrcSize);
	}

	#endif /* HUF_FORCE_DECOMPRESS_X2 */


	#ifndef HUF_FORCE_DECOMPRESS_X1

	/* *************************/
	/* double-symbols decoding */
	/* *************************/

	typedef struct { U16 sequence; BYTE nbBits; BYTE length; } HUF_DEltX2; /* double-symbols decoding */
	typedef struct { BYTE symbol; BYTE weight; } sortedSymbol_t;
	typedef U32 rankValCol_t[HUF_TABLELOG_MAX + 1];
	typedef rankValCol_t rankVal_t[HUF_TABLELOG_MAX];


	/* HUF_fillDTableX2Level2() :
	* `rankValOrigin` must be a table of at least (HUF_TABLELOG_MAX + 1) U32 */
	static void HUF_fillDTableX2Level2(HUF_DEltX2* DTable, U32 sizeLog, const U32 consumed,
	const U32* rankValOrigin, const int minWeight,
	const sortedSymbol_t* sortedSymbols, const U32 sortedListSize,
	U32 nbBitsBaseline, U16 baseSeq)
	{
	HUF_DEltX2 DElt;
	U32 rankVal[HUF_TABLELOG_MAX + 1];

	/* get pre-calculated rankVal */
	memcpy(rankVal, rankValOrigin, sizeof(rankVal));

	/* fill skipped values */
	if (minWeight>1) {
	U32 i, skipSize = rankVal[minWeight];
	MEM_writeLE16(&(DElt.sequence), baseSeq);
	DElt.nbBits = (BYTE)(consumed);
	DElt.length = 1;
	for (i = 0; i < skipSize; i++)
	DTable[i] = DElt;
	}

	/* fill DTable */
	{ U32 s; for (s=0; s<sortedListSize; s++) { /* note : sortedSymbols already skipped */
	const U32 symbol = sortedSymbols[s].symbol;
	const U32 weight = sortedSymbols[s].weight;
	const U32 nbBits = nbBitsBaseline - weight;
	const U32 length = 1 << (sizeLog-nbBits);
	const U32 start = rankVal[weight];
	U32 i = start;
	const U32 end = start + length;

	MEM_writeLE16(&(DElt.sequence), (U16)(baseSeq + (symbol << 8)));
	DElt.nbBits = (BYTE)(nbBits + consumed);
	DElt.length = 2;
	do { DTable[i++] = DElt; } while (i<end); /* since length >= 1 */

	rankVal[weight] += length;
	} }
	}


	static void HUF_fillDTableX2(HUF_DEltX2* DTable, const U32 targetLog,
	const sortedSymbol_t* sortedList, const U32 sortedListSize,
	const U32* rankStart, rankVal_t rankValOrigin, const U32 maxWeight,
	const U32 nbBitsBaseline)
	{
	U32 rankVal[HUF_TABLELOG_MAX + 1];
	const int scaleLog = nbBitsBaseline - targetLog; /* note : targetLog >= srcLog, hence scaleLog <= 1 */
	const U32 minBits = nbBitsBaseline - maxWeight;
	U32 s;

	memcpy(rankVal, rankValOrigin, sizeof(rankVal));

	/* fill DTable */
	for (s=0; s<sortedListSize; s++) {
	const U16 symbol = sortedList[s].symbol;
	const U32 weight = sortedList[s].weight;
	const U32 nbBits = nbBitsBaseline - weight;
	const U32 start = rankVal[weight];
	const U32 length = 1 << (targetLog-nbBits);

	if (targetLog-nbBits >= minBits) { /* enough room for a second symbol */
	U32 sortedRank;
	int minWeight = nbBits + scaleLog;
	if (minWeight < 1) minWeight = 1;
	sortedRank = rankStart[minWeight];
	HUF_fillDTableX2Level2(DTable+start, targetLog-nbBits, nbBits,
	rankValOrigin[nbBits], minWeight,
	sortedList+sortedRank, sortedListSize-sortedRank,
	nbBitsBaseline, symbol);
	} else {
	HUF_DEltX2 DElt;
	MEM_writeLE16(&(DElt.sequence), symbol);
	DElt.nbBits = (BYTE)(nbBits);
	DElt.length = 1;
	{ U32 const end = start + length;
	U32 u;
	for (u = start; u < end; u++) DTable[u] = DElt;
	} }
	rankVal[weight] += length;
	}
	}

	size_t HUF_readDTableX2_wksp(HUF_DTable* DTable,
	const void* src, size_t srcSize,
	void* workSpace, size_t wkspSize)
	{
	U32 tableLog, maxW, sizeOfSort, nbSymbols;
	DTableDesc dtd = HUF_getDTableDesc(DTable);
	U32 const maxTableLog = dtd.maxTableLog;
	size_t iSize;
	void* dtPtr = DTable+1; /* force compiler to avoid strict-aliasing */
	HUF_DEltX2* const dt = (HUF_DEltX2*)dtPtr;
	U32 *rankStart;

	rankValCol_t* rankVal;
	U32* rankStats;
	U32* rankStart0;
	sortedSymbol_t* sortedSymbol;
	BYTE* weightList;
	size_t spaceUsed32 = 0;

	rankVal = (rankValCol_t )((U32 )workSpace + spaceUsed32);
	spaceUsed32 += (sizeof(rankValCol_t) * HUF_TABLELOG_MAX) >> 2;
	rankStats = (U32 *)workSpace + spaceUsed32;
	spaceUsed32 += HUF_TABLELOG_MAX + 1;
	rankStart0 = (U32 *)workSpace + spaceUsed32;
	spaceUsed32 += HUF_TABLELOG_MAX + 2;
	sortedSymbol = (sortedSymbol_t )workSpace + (spaceUsed32 sizeof(U32)) / sizeof(sortedSymbol_t);
	spaceUsed32 += HUF_ALIGN(sizeof(sortedSymbol_t) * (HUF_SYMBOLVALUE_MAX + 1), sizeof(U32)) >> 2;
	weightList = (BYTE )((U32 )workSpace + spaceUsed32);
	spaceUsed32 += HUF_ALIGN(HUF_SYMBOLVALUE_MAX + 1, sizeof(U32)) >> 2;

	if ((spaceUsed32 << 2) > wkspSize) return ERROR(tableLog_tooLarge);

	rankStart = rankStart0 + 1;
	memset(rankStats, 0, sizeof(U32) * (2 * HUF_TABLELOG_MAX + 2 + 1));

	DEBUG_STATIC_ASSERT(sizeof(HUF_DEltX2) == sizeof(HUF_DTable)); /* if compiler fails here, assertion is wrong */
	if (maxTableLog > HUF_TABLELOG_MAX) return ERROR(tableLog_tooLarge);
	/* memset(weightList, 0, sizeof(weightList)); / / is not necessary, even though some analyzer complain ... */

	iSize = HUF_readStats(weightList, HUF_SYMBOLVALUE_MAX + 1, rankStats, &nbSymbols, &tableLog, src, srcSize);
	if (HUF_isError(iSize)) return iSize;

	/* check result */
	if (tableLog > maxTableLog) return ERROR(tableLog_tooLarge); /* DTable can't fit code depth */

	/* find maxWeight */
	for (maxW = tableLog; rankStats[maxW]==0; maxW--) {} /* necessarily finds a solution before 0 */

	/* Get start index of each weight */
	{ U32 w, nextRankStart = 0;
	for (w=1; w<maxW+1; w++) {
	U32 current = nextRankStart;
	nextRankStart += rankStats[w];
	rankStart[w] = current;
	}
	rankStart[0] = nextRankStart; /* put all 0w symbols at the end of sorted list*/
	sizeOfSort = nextRankStart;
	}

	/* sort symbols by weight */
	{ U32 s;
	for (s=0; s<nbSymbols; s++) {
	U32 const w = weightList[s];
	U32 const r = rankStart[w]++;
	sortedSymbol[r].symbol = (BYTE)s;
	sortedSymbol[r].weight = (BYTE)w;
	}
	rankStart[0] = 0; /* forget 0w symbols; this is beginning of weight(1) */
	}

	/* Build rankVal */
	{ U32* const rankVal0 = rankVal[0];
	{ int const rescale = (maxTableLog-tableLog) - 1; /* tableLog <= maxTableLog */
	U32 nextRankVal = 0;
	U32 w;
	for (w=1; w<maxW+1; w++) {
	U32 current = nextRankVal;
	nextRankVal += rankStats[w] << (w+rescale);
	rankVal0[w] = current;
	} }
	{ U32 const minBits = tableLog+1 - maxW;
	U32 consumed;
	for (consumed = minBits; consumed < maxTableLog - minBits + 1; consumed++) {
	U32* const rankValPtr = rankVal[consumed];
	U32 w;
	for (w = 1; w < maxW+1; w++) {
	rankValPtr[w] = rankVal0[w] >> consumed;
	} } } }

	HUF_fillDTableX2(dt, maxTableLog,
	sortedSymbol, sizeOfSort,
	rankStart0, rankVal, maxW,
	tableLog+1);

	dtd.tableLog = (BYTE)maxTableLog;
	dtd.tableType = 1;
	memcpy(DTable, &dtd, sizeof(dtd));
	return iSize;
	}

	size_t HUF_readDTableX2(HUF_DTable* DTable, const void* src, size_t srcSize)
	{
	U32 workSpace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32];
	return HUF_readDTableX2_wksp(DTable, src, srcSize,
	workSpace, sizeof(workSpace));
	}


	FORCE_INLINE_TEMPLATE U32
	HUF_decodeSymbolX2(void* op, BIT_DStream_t* DStream, const HUF_DEltX2* dt, const U32 dtLog)
	{
	size_t const val = BIT_lookBitsFast(DStream, dtLog); /* note : dtLog >= 1 */
	memcpy(op, dt+val, 2);
	BIT_skipBits(DStream, dt[val].nbBits);
	return dt[val].length;
	}

	FORCE_INLINE_TEMPLATE U32
	HUF_decodeLastSymbolX2(void* op, BIT_DStream_t* DStream, const HUF_DEltX2* dt, const U32 dtLog)
	{
	size_t const val = BIT_lookBitsFast(DStream, dtLog); /* note : dtLog >= 1 */
	memcpy(op, dt+val, 1);
	if (dt[val].length==1) BIT_skipBits(DStream, dt[val].nbBits);
	else {
	if (DStream->bitsConsumed < (sizeof(DStream->bitContainer)*8)) {
	BIT_skipBits(DStream, dt[val].nbBits);
	if (DStream->bitsConsumed > (sizeof(DStream->bitContainer)*8))
	/* ugly hack; works only because it's the last symbol. Note : can't easily extract nbBits from just this symbol */
	DStream->bitsConsumed = (sizeof(DStream->bitContainer)*8);
	} }
	return 1;
	}

	#define HUF_DECODE_SYMBOLX2_0(ptr, DStreamPtr) \
	ptr += HUF_decodeSymbolX2(ptr, DStreamPtr, dt, dtLog)

	#define HUF_DECODE_SYMBOLX2_1(ptr, DStreamPtr) \
	if (MEM_64bits() \|\| (HUF_TABLELOG_MAX<=12)) \
	ptr += HUF_decodeSymbolX2(ptr, DStreamPtr, dt, dtLog)

	#define HUF_DECODE_SYMBOLX2_2(ptr, DStreamPtr) \
	if (MEM_64bits()) \
	ptr += HUF_decodeSymbolX2(ptr, DStreamPtr, dt, dtLog)

	HINT_INLINE size_t
	HUF_decodeStreamX2(BYTE* p, BIT_DStream_t* bitDPtr, BYTE* const pEnd,
	const HUF_DEltX2* const dt, const U32 dtLog)
	{
	BYTE* const pStart = p;

	/* up to 8 symbols at a time */
	while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p < pEnd-(sizeof(bitDPtr->bitContainer)-1))) {
	HUF_DECODE_SYMBOLX2_2(p, bitDPtr);
	HUF_DECODE_SYMBOLX2_1(p, bitDPtr);
	HUF_DECODE_SYMBOLX2_2(p, bitDPtr);
	HUF_DECODE_SYMBOLX2_0(p, bitDPtr);
	}

	/* closer to end : up to 2 symbols at a time */
	while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p <= pEnd-2))
	HUF_DECODE_SYMBOLX2_0(p, bitDPtr);

	while (p <= pEnd-2)
	HUF_DECODE_SYMBOLX2_0(p, bitDPtr); /* no need to reload : reached the end of DStream */

	if (p < pEnd)
	p += HUF_decodeLastSymbolX2(p, bitDPtr, dt, dtLog);

	return p-pStart;
	}

	FORCE_INLINE_TEMPLATE size_t
	HUF_decompress1X2_usingDTable_internal_body(
	void* dst, size_t dstSize,
	const void* cSrc, size_t cSrcSize,
	const HUF_DTable* DTable)
	{
	BIT_DStream_t bitD;

	/* Init */
	CHECK_F( BIT_initDStream(&bitD, cSrc, cSrcSize) );

	/* decode */
	{ BYTE* const ostart = (BYTE*) dst;
	BYTE* const oend = ostart + dstSize;
	const void* const dtPtr = DTable+1; /* force compiler to not use strict-aliasing */
	const HUF_DEltX2* const dt = (const HUF_DEltX2*)dtPtr;
	DTableDesc const dtd = HUF_getDTableDesc(DTable);
	HUF_decodeStreamX2(ostart, &bitD, oend, dt, dtd.tableLog);
	}

	/* check */
	if (!BIT_endOfDStream(&bitD)) return ERROR(corruption_detected);

	/* decoded size */
	return dstSize;
	}

	FORCE_INLINE_TEMPLATE size_t
	HUF_decompress4X2_usingDTable_internal_body(
	void* dst, size_t dstSize,
	const void* cSrc, size_t cSrcSize,
	const HUF_DTable* DTable)
	{
	if (cSrcSize < 10) return ERROR(corruption_detected); /* strict minimum : jump table + 1 byte per stream */

	{ const BYTE* const istart = (const BYTE*) cSrc;
	BYTE* const ostart = (BYTE*) dst;
	BYTE* const oend = ostart + dstSize;
	BYTE* const olimit = oend - (sizeof(size_t)-1);
	const void* const dtPtr = DTable+1;
	const HUF_DEltX2* const dt = (const HUF_DEltX2*)dtPtr;

	/* Init */
	BIT_DStream_t bitD1;
	BIT_DStream_t bitD2;
	BIT_DStream_t bitD3;
	BIT_DStream_t bitD4;
	size_t const length1 = MEM_readLE16(istart);
	size_t const length2 = MEM_readLE16(istart+2);
	size_t const length3 = MEM_readLE16(istart+4);
	size_t const length4 = cSrcSize - (length1 + length2 + length3 + 6);
	const BYTE* const istart1 = istart + 6; /* jumpTable */
	const BYTE* const istart2 = istart1 + length1;
	const BYTE* const istart3 = istart2 + length2;
	const BYTE* const istart4 = istart3 + length3;
	size_t const segmentSize = (dstSize+3) / 4;
	BYTE* const opStart2 = ostart + segmentSize;
	BYTE* const opStart3 = opStart2 + segmentSize;
	BYTE* const opStart4 = opStart3 + segmentSize;
	BYTE* op1 = ostart;
	BYTE* op2 = opStart2;
	BYTE* op3 = opStart3;
	BYTE* op4 = opStart4;
	U32 endSignal = 1;
	DTableDesc const dtd = HUF_getDTableDesc(DTable);
	U32 const dtLog = dtd.tableLog;

	if (length4 > cSrcSize) return ERROR(corruption_detected); /* overflow */
	CHECK_F( BIT_initDStream(&bitD1, istart1, length1) );
	CHECK_F( BIT_initDStream(&bitD2, istart2, length2) );
	CHECK_F( BIT_initDStream(&bitD3, istart3, length3) );
	CHECK_F( BIT_initDStream(&bitD4, istart4, length4) );

	/* 16-32 symbols per loop (4-8 symbols per stream) */
	for ( ; (endSignal) & (op4 < olimit); ) {
	#if defined(__clang__) && (defined(__x86_64__) \|\| defined(__i386__))
	HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
	HUF_DECODE_SYMBOLX2_1(op1, &bitD1);
	HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
	HUF_DECODE_SYMBOLX2_0(op1, &bitD1);
	HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
	HUF_DECODE_SYMBOLX2_1(op2, &bitD2);
	HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
	HUF_DECODE_SYMBOLX2_0(op2, &bitD2);
	endSignal &= BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished;
	endSignal &= BIT_reloadDStreamFast(&bitD2) == BIT_DStream_unfinished;
	HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
	HUF_DECODE_SYMBOLX2_1(op3, &bitD3);
	HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
	HUF_DECODE_SYMBOLX2_0(op3, &bitD3);
	HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
	HUF_DECODE_SYMBOLX2_1(op4, &bitD4);
	HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
	HUF_DECODE_SYMBOLX2_0(op4, &bitD4);
	endSignal &= BIT_reloadDStreamFast(&bitD3) == BIT_DStream_unfinished;
	endSignal &= BIT_reloadDStreamFast(&bitD4) == BIT_DStream_unfinished;
	#else
	HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
	HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
	HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
	HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
	HUF_DECODE_SYMBOLX2_1(op1, &bitD1);
	HUF_DECODE_SYMBOLX2_1(op2, &bitD2);
	HUF_DECODE_SYMBOLX2_1(op3, &bitD3);
	HUF_DECODE_SYMBOLX2_1(op4, &bitD4);
	HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
	HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
	HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
	HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
	HUF_DECODE_SYMBOLX2_0(op1, &bitD1);
	HUF_DECODE_SYMBOLX2_0(op2, &bitD2);
	HUF_DECODE_SYMBOLX2_0(op3, &bitD3);
	HUF_DECODE_SYMBOLX2_0(op4, &bitD4);
	endSignal = (U32)LIKELY(
	(BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished)
	& (BIT_reloadDStreamFast(&bitD2) == BIT_DStream_unfinished)
	& (BIT_reloadDStreamFast(&bitD3) == BIT_DStream_unfinished)
	& (BIT_reloadDStreamFast(&bitD4) == BIT_DStream_unfinished));
	#endif
	}

	/* check corruption */
	if (op1 > opStart2) return ERROR(corruption_detected);
	if (op2 > opStart3) return ERROR(corruption_detected);
	if (op3 > opStart4) return ERROR(corruption_detected);
	/* note : op4 already verified within main loop */

	/* finish bitStreams one by one */
	HUF_decodeStreamX2(op1, &bitD1, opStart2, dt, dtLog);
	HUF_decodeStreamX2(op2, &bitD2, opStart3, dt, dtLog);
	HUF_decodeStreamX2(op3, &bitD3, opStart4, dt, dtLog);
	HUF_decodeStreamX2(op4, &bitD4, oend, dt, dtLog);

	/* check */
	{ U32 const endCheck = BIT_endOfDStream(&bitD1) & BIT_endOfDStream(&bitD2) & BIT_endOfDStream(&bitD3) & BIT_endOfDStream(&bitD4);
	if (!endCheck) return ERROR(corruption_detected); }

	/* decoded size */
	return dstSize;
	}
	}

	HUF_DGEN(HUF_decompress1X2_usingDTable_internal)
	HUF_DGEN(HUF_decompress4X2_usingDTable_internal)

	size_t HUF_decompress1X2_usingDTable(
	void* dst, size_t dstSize,
	const void* cSrc, size_t cSrcSize,
	const HUF_DTable* DTable)
	{
	DTableDesc dtd = HUF_getDTableDesc(DTable);
	if (dtd.tableType != 1) return ERROR(GENERIC);
	return HUF_decompress1X2_usingDTable_internal(dst, dstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
	}

	size_t HUF_decompress1X2_DCtx_wksp(HUF_DTable* DCtx, void* dst, size_t dstSize,
	const void* cSrc, size_t cSrcSize,
	void* workSpace, size_t wkspSize)
	{
	const BYTE* ip = (const BYTE*) cSrc;

	size_t const hSize = HUF_readDTableX2_wksp(DCtx, cSrc, cSrcSize,
	workSpace, wkspSize);
	if (HUF_isError(hSize)) return hSize;
	if (hSize >= cSrcSize) return ERROR(srcSize_wrong);
	ip += hSize; cSrcSize -= hSize;

	return HUF_decompress1X2_usingDTable_internal(dst, dstSize, ip, cSrcSize, DCtx, /* bmi2 */ 0);
	}


	size_t HUF_decompress1X2_DCtx(HUF_DTable* DCtx, void* dst, size_t dstSize,
	const void* cSrc, size_t cSrcSize)
	{
	U32 workSpace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32];
	return HUF_decompress1X2_DCtx_wksp(DCtx, dst, dstSize, cSrc, cSrcSize,
	workSpace, sizeof(workSpace));
	}

	size_t HUF_decompress1X2 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize)
	{
	HUF_CREATE_STATIC_DTABLEX2(DTable, HUF_TABLELOG_MAX);
	return HUF_decompress1X2_DCtx(DTable, dst, dstSize, cSrc, cSrcSize);
	}

	size_t HUF_decompress4X2_usingDTable(
	void* dst, size_t dstSize,
	const void* cSrc, size_t cSrcSize,
	const HUF_DTable* DTable)
	{
	DTableDesc dtd = HUF_getDTableDesc(DTable);
	if (dtd.tableType != 1) return ERROR(GENERIC);
	return HUF_decompress4X2_usingDTable_internal(dst, dstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
	}

	static size_t HUF_decompress4X2_DCtx_wksp_bmi2(HUF_DTable* dctx, void* dst, size_t dstSize,
	const void* cSrc, size_t cSrcSize,
	void* workSpace, size_t wkspSize, int bmi2)
	{
	const BYTE* ip = (const BYTE*) cSrc;

	size_t hSize = HUF_readDTableX2_wksp(dctx, cSrc, cSrcSize,
	workSpace, wkspSize);
	if (HUF_isError(hSize)) return hSize;
	if (hSize >= cSrcSize) return ERROR(srcSize_wrong);
	ip += hSize; cSrcSize -= hSize;

	return HUF_decompress4X2_usingDTable_internal(dst, dstSize, ip, cSrcSize, dctx, bmi2);
	}

	size_t HUF_decompress4X2_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize,
	const void* cSrc, size_t cSrcSize,
	void* workSpace, size_t wkspSize)
	{
	return HUF_decompress4X2_DCtx_wksp_bmi2(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, /* bmi2 */ 0);
	}


	size_t HUF_decompress4X2_DCtx(HUF_DTable* dctx, void* dst, size_t dstSize,
	const void* cSrc, size_t cSrcSize)
	{
	U32 workSpace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32];
	return HUF_decompress4X2_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize,
	workSpace, sizeof(workSpace));
	}

	size_t HUF_decompress4X2 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize)
	{
	HUF_CREATE_STATIC_DTABLEX2(DTable, HUF_TABLELOG_MAX);
	return HUF_decompress4X2_DCtx(DTable, dst, dstSize, cSrc, cSrcSize);
	}

	#endif /* HUF_FORCE_DECOMPRESS_X1 */


	/* ***********************************/
	/* Universal decompression selectors */
	/* ***********************************/

	size_t HUF_decompress1X_usingDTable(void* dst, size_t maxDstSize,
	const void* cSrc, size_t cSrcSize,
	const HUF_DTable* DTable)
	{
	DTableDesc const dtd = HUF_getDTableDesc(DTable);
	#if defined(HUF_FORCE_DECOMPRESS_X1)
	(void)dtd;
	assert(dtd.tableType == 0);
	return HUF_decompress1X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
	#elif defined(HUF_FORCE_DECOMPRESS_X2)
	(void)dtd;
	assert(dtd.tableType == 1);
	return HUF_decompress1X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
	#else
	return dtd.tableType ? HUF_decompress1X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0) :
	HUF_decompress1X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
	#endif
	}

	size_t HUF_decompress4X_usingDTable(void* dst, size_t maxDstSize,
	const void* cSrc, size_t cSrcSize,
	const HUF_DTable* DTable)
	{
	DTableDesc const dtd = HUF_getDTableDesc(DTable);
	#if defined(HUF_FORCE_DECOMPRESS_X1)
	(void)dtd;
	assert(dtd.tableType == 0);
	return HUF_decompress4X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
	#elif defined(HUF_FORCE_DECOMPRESS_X2)
	(void)dtd;
	assert(dtd.tableType == 1);
	return HUF_decompress4X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
	#else
	return dtd.tableType ? HUF_decompress4X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0) :
	HUF_decompress4X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
	#endif
	}


	#if !defined(HUF_FORCE_DECOMPRESS_X1) && !defined(HUF_FORCE_DECOMPRESS_X2)
	typedef struct { U32 tableTime; U32 decode256Time; } algo_time_t;
	static const algo_time_t algoTime[16 /* Quantization /][3 / single, double, quad */] =
	{
	/* single, double, quad */
	{{0,0}, {1,1}, {2,2}}, /* Q==0 : impossible */
	{{0,0}, {1,1}, {2,2}}, /* Q==1 : impossible */
	{{ 38,130}, {1313, 74}, {2151, 38}}, /* Q == 2 : 12-18% */
	{{ 448,128}, {1353, 74}, {2238, 41}}, /* Q == 3 : 18-25% */
	{{ 556,128}, {1353, 74}, {2238, 47}}, /* Q == 4 : 25-32% */
	{{ 714,128}, {1418, 74}, {2436, 53}}, /* Q == 5 : 32-38% */
	{{ 883,128}, {1437, 74}, {2464, 61}}, /* Q == 6 : 38-44% */
	{{ 897,128}, {1515, 75}, {2622, 68}}, /* Q == 7 : 44-50% */
	{{ 926,128}, {1613, 75}, {2730, 75}}, /* Q == 8 : 50-56% */
	{{ 947,128}, {1729, 77}, {3359, 77}}, /* Q == 9 : 56-62% */
	{{1107,128}, {2083, 81}, {4006, 84}}, /* Q ==10 : 62-69% */
	{{1177,128}, {2379, 87}, {4785, 88}}, /* Q ==11 : 69-75% */
	{{1242,128}, {2415, 93}, {5155, 84}}, /* Q ==12 : 75-81% */
	{{1349,128}, {2644,106}, {5260,106}}, /* Q ==13 : 81-87% */
	{{1455,128}, {2422,124}, {4174,124}}, /* Q ==14 : 87-93% */
	{{ 722,128}, {1891,145}, {1936,146}}, /* Q ==15 : 93-99% */
	};
	#endif

	/** HUF_selectDecoder() :
	* Tells which decoder is likely to decode faster,
	* based on a set of pre-computed metrics.
	* @return : 0==HUF_decompress4X1, 1==HUF_decompress4X2 .
	* Assumption : 0 < dstSize <= 128 KB */
	U32 HUF_selectDecoder (size_t dstSize, size_t cSrcSize)
	{
	assert(dstSize > 0);
	assert(dstSize <= 128*1024);
	#if defined(HUF_FORCE_DECOMPRESS_X1)
	(void)dstSize;
	(void)cSrcSize;
	return 0;
	#elif defined(HUF_FORCE_DECOMPRESS_X2)
	(void)dstSize;
	(void)cSrcSize;
	return 1;
	#else
	/* decoder timing evaluation */
	{ U32 const Q = (cSrcSize >= dstSize) ? 15 : (U32)(cSrcSize * 16 / dstSize); /* Q < 16 */
	U32 const D256 = (U32)(dstSize >> 8);
	U32 const DTime0 = algoTime[Q][0].tableTime + (algoTime[Q][0].decode256Time * D256);
	U32 DTime1 = algoTime[Q][1].tableTime + (algoTime[Q][1].decode256Time * D256);
	DTime1 += DTime1 >> 3; /* advantage to algorithm using less memory, to reduce cache eviction */
	return DTime1 < DTime0;
	}
	#endif
	}


	typedef size_t (decompressionAlgo)(void dst, size_t dstSize, const void* cSrc, size_t cSrcSize);

	size_t HUF_decompress (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize)
	{
	#if !defined(HUF_FORCE_DECOMPRESS_X1) && !defined(HUF_FORCE_DECOMPRESS_X2)
	static const decompressionAlgo decompress[2] = { HUF_decompress4X1, HUF_decompress4X2 };
	#endif

	/* validation checks */
	if (dstSize == 0) return ERROR(dstSize_tooSmall);
	if (cSrcSize > dstSize) return ERROR(corruption_detected); /* invalid */
	if (cSrcSize == dstSize) { memcpy(dst, cSrc, dstSize); return dstSize; } /* not compressed */
	if (cSrcSize == 1) { memset(dst, (const BYTE)cSrc, dstSize); return dstSize; } /* RLE */

	{ U32 const algoNb = HUF_selectDecoder(dstSize, cSrcSize);
	#if defined(HUF_FORCE_DECOMPRESS_X1)
	(void)algoNb;
	assert(algoNb == 0);
	return HUF_decompress4X1(dst, dstSize, cSrc, cSrcSize);
	#elif defined(HUF_FORCE_DECOMPRESS_X2)
	(void)algoNb;
	assert(algoNb == 1);
	return HUF_decompress4X2(dst, dstSize, cSrc, cSrcSize);
	#else
	return decompress[algoNb](dst, dstSize, cSrc, cSrcSize);
	#endif
	}
	}

	size_t HUF_decompress4X_DCtx (HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize)
	{
	/* validation checks */
	if (dstSize == 0) return ERROR(dstSize_tooSmall);
	if (cSrcSize > dstSize) return ERROR(corruption_detected); /* invalid */
	if (cSrcSize == dstSize) { memcpy(dst, cSrc, dstSize); return dstSize; } /* not compressed */
	if (cSrcSize == 1) { memset(dst, (const BYTE)cSrc, dstSize); return dstSize; } /* RLE */

	{ U32 const algoNb = HUF_selectDecoder(dstSize, cSrcSize);
	#if defined(HUF_FORCE_DECOMPRESS_X1)
	(void)algoNb;
	assert(algoNb == 0);
	return HUF_decompress4X1_DCtx(dctx, dst, dstSize, cSrc, cSrcSize);
	#elif defined(HUF_FORCE_DECOMPRESS_X2)
	(void)algoNb;
	assert(algoNb == 1);
	return HUF_decompress4X2_DCtx(dctx, dst, dstSize, cSrc, cSrcSize);
	#else
	return algoNb ? HUF_decompress4X2_DCtx(dctx, dst, dstSize, cSrc, cSrcSize) :
	HUF_decompress4X1_DCtx(dctx, dst, dstSize, cSrc, cSrcSize) ;
	#endif
	}
	}

	size_t HUF_decompress4X_hufOnly(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize)
	{
	U32 workSpace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32];
	return HUF_decompress4X_hufOnly_wksp(dctx, dst, dstSize, cSrc, cSrcSize,
	workSpace, sizeof(workSpace));
	}


	size_t HUF_decompress4X_hufOnly_wksp(HUF_DTable* dctx, void* dst,
	size_t dstSize, const void* cSrc,
	size_t cSrcSize, void* workSpace,
	size_t wkspSize)
	{
	/* validation checks */
	if (dstSize == 0) return ERROR(dstSize_tooSmall);
	if (cSrcSize == 0) return ERROR(corruption_detected);

	{ U32 const algoNb = HUF_selectDecoder(dstSize, cSrcSize);
	#if defined(HUF_FORCE_DECOMPRESS_X1)
	(void)algoNb;
	assert(algoNb == 0);
	return HUF_decompress4X1_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize);
	#elif defined(HUF_FORCE_DECOMPRESS_X2)
	(void)algoNb;
	assert(algoNb == 1);
	return HUF_decompress4X2_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize);
	#else
	return algoNb ? HUF_decompress4X2_DCtx_wksp(dctx, dst, dstSize, cSrc,
	cSrcSize, workSpace, wkspSize):
	HUF_decompress4X1_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize);
	#endif
	}
	}

	size_t HUF_decompress1X_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize,
	const void* cSrc, size_t cSrcSize,
	void* workSpace, size_t wkspSize)
	{
	/* validation checks */
	if (dstSize == 0) return ERROR(dstSize_tooSmall);
	if (cSrcSize > dstSize) return ERROR(corruption_detected); /* invalid */
	if (cSrcSize == dstSize) { memcpy(dst, cSrc, dstSize); return dstSize; } /* not compressed */
	if (cSrcSize == 1) { memset(dst, (const BYTE)cSrc, dstSize); return dstSize; } /* RLE */

	{ U32 const algoNb = HUF_selectDecoder(dstSize, cSrcSize);
	#if defined(HUF_FORCE_DECOMPRESS_X1)
	(void)algoNb;
	assert(algoNb == 0);
	return HUF_decompress1X1_DCtx_wksp(dctx, dst, dstSize, cSrc,
	cSrcSize, workSpace, wkspSize);
	#elif defined(HUF_FORCE_DECOMPRESS_X2)
	(void)algoNb;
	assert(algoNb == 1);
	return HUF_decompress1X2_DCtx_wksp(dctx, dst, dstSize, cSrc,
	cSrcSize, workSpace, wkspSize);
	#else
	return algoNb ? HUF_decompress1X2_DCtx_wksp(dctx, dst, dstSize, cSrc,
	cSrcSize, workSpace, wkspSize):
	HUF_decompress1X1_DCtx_wksp(dctx, dst, dstSize, cSrc,
	cSrcSize, workSpace, wkspSize);
	#endif
	}
	}

	size_t HUF_decompress1X_DCtx(HUF_DTable* dctx, void* dst, size_t dstSize,
	const void* cSrc, size_t cSrcSize)
	{
	U32 workSpace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32];
	return HUF_decompress1X_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize,
	workSpace, sizeof(workSpace));
	}


	size_t HUF_decompress1X_usingDTable_bmi2(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, int bmi2)
	{
	DTableDesc const dtd = HUF_getDTableDesc(DTable);
	#if defined(HUF_FORCE_DECOMPRESS_X1)
	(void)dtd;
	assert(dtd.tableType == 0);
	return HUF_decompress1X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, bmi2);
	#elif defined(HUF_FORCE_DECOMPRESS_X2)
	(void)dtd;
	assert(dtd.tableType == 1);
	return HUF_decompress1X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, bmi2);
	#else
	return dtd.tableType ? HUF_decompress1X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, bmi2) :
	HUF_decompress1X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, bmi2);
	#endif
	}

	#ifndef HUF_FORCE_DECOMPRESS_X2
	size_t HUF_decompress1X1_DCtx_wksp_bmi2(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int bmi2)
	{
	const BYTE* ip = (const BYTE*) cSrc;

	size_t const hSize = HUF_readDTableX1_wksp(dctx, cSrc, cSrcSize, workSpace, wkspSize);
	if (HUF_isError(hSize)) return hSize;
	if (hSize >= cSrcSize) return ERROR(srcSize_wrong);
	ip += hSize; cSrcSize -= hSize;

	return HUF_decompress1X1_usingDTable_internal(dst, dstSize, ip, cSrcSize, dctx, bmi2);
	}
	#endif

	size_t HUF_decompress4X_usingDTable_bmi2(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, int bmi2)
	{
	DTableDesc const dtd = HUF_getDTableDesc(DTable);
	#if defined(HUF_FORCE_DECOMPRESS_X1)
	(void)dtd;
	assert(dtd.tableType == 0);
	return HUF_decompress4X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, bmi2);
	#elif defined(HUF_FORCE_DECOMPRESS_X2)
	(void)dtd;
	assert(dtd.tableType == 1);
	return HUF_decompress4X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, bmi2);
	#else
	return dtd.tableType ? HUF_decompress4X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, bmi2) :
	HUF_decompress4X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, bmi2);
	#endif
	}

	size_t HUF_decompress4X_hufOnly_wksp_bmi2(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int bmi2)
	{
	/* validation checks */
	if (dstSize == 0) return ERROR(dstSize_tooSmall);
	if (cSrcSize == 0) return ERROR(corruption_detected);

	{ U32 const algoNb = HUF_selectDecoder(dstSize, cSrcSize);
	#if defined(HUF_FORCE_DECOMPRESS_X1)
	(void)algoNb;
	assert(algoNb == 0);
	return HUF_decompress4X1_DCtx_wksp_bmi2(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, bmi2);
	#elif defined(HUF_FORCE_DECOMPRESS_X2)
	(void)algoNb;
	assert(algoNb == 1);
	return HUF_decompress4X2_DCtx_wksp_bmi2(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, bmi2);
	#else
	return algoNb ? HUF_decompress4X2_DCtx_wksp_bmi2(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, bmi2) :
	HUF_decompress4X1_DCtx_wksp_bmi2(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, bmi2);
	#endif
	}
	}
	/** ended inlining decompress/huf_decompress.c **/
	/** start inlining decompress/zstd_ddict.c **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	/* zstd_ddict.c :
	* concentrates all logic that needs to know the internals of ZSTD_DDict object */

	/-******************************************************
	* Dependencies
	*********************************************************/
	#include <string.h> /* memcpy, memmove, memset */
	/** skipping file: ../common/cpu.h **/
	/** skipping file: ../common/mem.h **/
	#define FSE_STATIC_LINKING_ONLY
	/** skipping file: ../common/fse.h **/
	#define HUF_STATIC_LINKING_ONLY
	/** skipping file: ../common/huf.h **/
	/** start inlining zstd_decompress_internal.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/


	/* zstd_decompress_internal:
	* objects and definitions shared within lib/decompress modules */

	#ifndef ZSTD_DECOMPRESS_INTERNAL_H
	#define ZSTD_DECOMPRESS_INTERNAL_H


	/-******************************************************
	* Dependencies
	*********************************************************/
	/** skipping file: ../common/mem.h **/
	/** skipping file: ../common/zstd_internal.h **/



	/-******************************************************
	* Constants
	*********************************************************/
	static const U32 LL_base[MaxLL+1] = {
	0, 1, 2, 3, 4, 5, 6, 7,
	8, 9, 10, 11, 12, 13, 14, 15,
	16, 18, 20, 22, 24, 28, 32, 40,
	48, 64, 0x80, 0x100, 0x200, 0x400, 0x800, 0x1000,
	0x2000, 0x4000, 0x8000, 0x10000 };

	static const U32 OF_base[MaxOff+1] = {
	0, 1, 1, 5, 0xD, 0x1D, 0x3D, 0x7D,
	0xFD, 0x1FD, 0x3FD, 0x7FD, 0xFFD, 0x1FFD, 0x3FFD, 0x7FFD,
	0xFFFD, 0x1FFFD, 0x3FFFD, 0x7FFFD, 0xFFFFD, 0x1FFFFD, 0x3FFFFD, 0x7FFFFD,
	0xFFFFFD, 0x1FFFFFD, 0x3FFFFFD, 0x7FFFFFD, 0xFFFFFFD, 0x1FFFFFFD, 0x3FFFFFFD, 0x7FFFFFFD };

	static const U32 OF_bits[MaxOff+1] = {
	0, 1, 2, 3, 4, 5, 6, 7,
	8, 9, 10, 11, 12, 13, 14, 15,
	16, 17, 18, 19, 20, 21, 22, 23,
	24, 25, 26, 27, 28, 29, 30, 31 };

	static const U32 ML_base[MaxML+1] = {
	3, 4, 5, 6, 7, 8, 9, 10,
	11, 12, 13, 14, 15, 16, 17, 18,
	19, 20, 21, 22, 23, 24, 25, 26,
	27, 28, 29, 30, 31, 32, 33, 34,
	35, 37, 39, 41, 43, 47, 51, 59,
	67, 83, 99, 0x83, 0x103, 0x203, 0x403, 0x803,
	0x1003, 0x2003, 0x4003, 0x8003, 0x10003 };


	/-******************************************************
	* Decompression types
	*********************************************************/
	typedef struct {
	U32 fastMode;
	U32 tableLog;
	} ZSTD_seqSymbol_header;

	typedef struct {
	U16 nextState;
	BYTE nbAdditionalBits;
	BYTE nbBits;
	U32 baseValue;
	} ZSTD_seqSymbol;

	#define SEQSYMBOL_TABLE_SIZE(log) (1 + (1 << (log)))

	typedef struct {
	ZSTD_seqSymbol LLTable[SEQSYMBOL_TABLE_SIZE(LLFSELog)]; /* Note : Space reserved for FSE Tables */
	ZSTD_seqSymbol OFTable[SEQSYMBOL_TABLE_SIZE(OffFSELog)]; /* is also used as temporary workspace while building hufTable during DDict creation */
	ZSTD_seqSymbol MLTable[SEQSYMBOL_TABLE_SIZE(MLFSELog)]; /* and therefore must be at least HUF_DECOMPRESS_WORKSPACE_SIZE large */
	HUF_DTable hufTable[HUF_DTABLE_SIZE(HufLog)]; /* can accommodate HUF_decompress4X */
	U32 rep[ZSTD_REP_NUM];
	} ZSTD_entropyDTables_t;

	typedef enum { ZSTDds_getFrameHeaderSize, ZSTDds_decodeFrameHeader,
	ZSTDds_decodeBlockHeader, ZSTDds_decompressBlock,
	ZSTDds_decompressLastBlock, ZSTDds_checkChecksum,
	ZSTDds_decodeSkippableHeader, ZSTDds_skipFrame } ZSTD_dStage;

	typedef enum { zdss_init=0, zdss_loadHeader,
	zdss_read, zdss_load, zdss_flush } ZSTD_dStreamStage;

	typedef enum {
	ZSTD_use_indefinitely = -1, /* Use the dictionary indefinitely */
	ZSTD_dont_use = 0, /* Do not use the dictionary (if one exists free it) */
	ZSTD_use_once = 1 /* Use the dictionary once and set to ZSTD_dont_use */
	} ZSTD_dictUses_e;

	typedef enum {
	ZSTD_obm_buffered = 0, /* Buffer the output */
	ZSTD_obm_stable = 1 /* ZSTD_outBuffer is stable */
	} ZSTD_outBufferMode_e;

	struct ZSTD_DCtx_s
	{
	const ZSTD_seqSymbol* LLTptr;
	const ZSTD_seqSymbol* MLTptr;
	const ZSTD_seqSymbol* OFTptr;
	const HUF_DTable* HUFptr;
	ZSTD_entropyDTables_t entropy;
	U32 workspace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32]; /* space needed when building huffman tables */
	const void* previousDstEnd; /* detect continuity */
	const void* prefixStart; /* start of current segment */
	const void* virtualStart; /* virtual start of previous segment if it was just before current one */
	const void* dictEnd; /* end of previous segment */
	size_t expected;
	ZSTD_frameHeader fParams;
	U64 decodedSize;
	blockType_e bType; /* used in ZSTD_decompressContinue(), store blockType between block header decoding and block decompression stages */
	ZSTD_dStage stage;
	U32 litEntropy;
	U32 fseEntropy;
	XXH64_state_t xxhState;
	size_t headerSize;
	ZSTD_format_e format;
	const BYTE* litPtr;
	ZSTD_customMem customMem;
	size_t litSize;
	size_t rleSize;
	size_t staticSize;
	int bmi2; /* == 1 if the CPU supports BMI2 and 0 otherwise. CPU support is determined dynamically once per context lifetime. */

	/* dictionary */
	ZSTD_DDict* ddictLocal;
	const ZSTD_DDict* ddict; /* set by ZSTD_initDStream_usingDDict(), or ZSTD_DCtx_refDDict() */
	U32 dictID;
	int ddictIsCold; /* if == 1 : dictionary is "new" for working context, and presumed "cold" (not in cpu cache) */
	ZSTD_dictUses_e dictUses;

	/* streaming */
	ZSTD_dStreamStage streamStage;
	char* inBuff;
	size_t inBuffSize;
	size_t inPos;
	size_t maxWindowSize;
	char* outBuff;
	size_t outBuffSize;
	size_t outStart;
	size_t outEnd;
	size_t lhSize;
	void* legacyContext;
	U32 previousLegacyVersion;
	U32 legacyVersion;
	U32 hostageByte;
	int noForwardProgress;
	ZSTD_outBufferMode_e outBufferMode;
	ZSTD_outBuffer expectedOutBuffer;

	/* workspace */
	BYTE litBuffer[ZSTD_BLOCKSIZE_MAX + WILDCOPY_OVERLENGTH];
	BYTE headerBuffer[ZSTD_FRAMEHEADERSIZE_MAX];

	size_t oversizedDuration;

	#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
	void const* dictContentBeginForFuzzing;
	void const* dictContentEndForFuzzing;
	#endif
	}; /* typedef'd to ZSTD_DCtx within "zstd.h" */


	/-******************************************************
	* Shared internal functions
	*********************************************************/

	/*! ZSTD_loadDEntropy() :
	* dict : must point at beginning of a valid zstd dictionary.
	* @return : size of dictionary header (size of magic number + dict ID + entropy tables) */
	size_t ZSTD_loadDEntropy(ZSTD_entropyDTables_t* entropy,
	const void* const dict, size_t const dictSize);

	/*! ZSTD_checkContinuity() :
	* check if next `dst` follows previous position, where decompression ended.
	* If yes, do nothing (continue on current segment).
	* If not, classify previous segment as "external dictionary", and start a new segment.
	* This function cannot fail. */
	void ZSTD_checkContinuity(ZSTD_DCtx* dctx, const void* dst);


	#endif /* ZSTD_DECOMPRESS_INTERNAL_H */
	/** ended inlining zstd_decompress_internal.h **/
	/** start inlining zstd_ddict.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/


	#ifndef ZSTD_DDICT_H
	#define ZSTD_DDICT_H

	/-******************************************************
	* Dependencies
	*********************************************************/
	#include <stddef.h> /* size_t */
	/** skipping file: ../zstd.h **/


	/-******************************************************
	* Interface
	*********************************************************/

	/* note: several prototypes are already published in `zstd.h` :
	* ZSTD_createDDict()
	* ZSTD_createDDict_byReference()
	* ZSTD_createDDict_advanced()
	* ZSTD_freeDDict()
	* ZSTD_initStaticDDict()
	* ZSTD_sizeof_DDict()
	* ZSTD_estimateDDictSize()
	* ZSTD_getDictID_fromDict()
	*/

	const void* ZSTD_DDict_dictContent(const ZSTD_DDict* ddict);
	size_t ZSTD_DDict_dictSize(const ZSTD_DDict* ddict);

	void ZSTD_copyDDictParameters(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict);



	#endif /* ZSTD_DDICT_H */
	/** ended inlining zstd_ddict.h **/

	#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1)
	/** start inlining ../legacy/zstd_legacy.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTD_LEGACY_H
	#define ZSTD_LEGACY_H

	#if defined (__cplusplus)
	extern "C" {
	#endif

	/* *************************************
	* Includes
	***************************************/
	/** skipping file: ../common/mem.h **/
	/** skipping file: ../common/error_private.h **/
	/** skipping file: ../common/zstd_internal.h **/

	#if !defined (ZSTD_LEGACY_SUPPORT) \|\| (ZSTD_LEGACY_SUPPORT == 0)
	# undef ZSTD_LEGACY_SUPPORT
	# define ZSTD_LEGACY_SUPPORT 8
	#endif

	#if (ZSTD_LEGACY_SUPPORT <= 1)
	/** start inlining zstd_v01.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTD_V01_H_28739879432
	#define ZSTD_V01_H_28739879432

	#if defined (__cplusplus)
	extern "C" {
	#endif

	/* *************************************
	* Includes
	***************************************/
	#include <stddef.h> /* size_t */


	/* *************************************
	* Simple one-step function
	***************************************/
	/**
	ZSTDv01_decompress() : decompress ZSTD frames compliant with v0.1.x format
	compressedSize : is the exact source size
	maxOriginalSize : is the size of the 'dst' buffer, which must be already allocated.
	It must be equal or larger than originalSize, otherwise decompression will fail.
	return : the number of bytes decompressed into destination buffer (originalSize)
	or an errorCode if it fails (which can be tested using ZSTDv01_isError())
	*/
	size_t ZSTDv01_decompress( void* dst, size_t maxOriginalSize,
	const void* src, size_t compressedSize);

	/**
	ZSTDv01_findFrameSizeInfoLegacy() : get the source length and decompressed bound of a ZSTD frame compliant with v0.1.x format
	srcSize : The size of the 'src' buffer, at least as large as the frame pointed to by 'src'
	cSize (output parameter) : the number of bytes that would be read to decompress this frame
	or an error code if it fails (which can be tested using ZSTDv01_isError())
	dBound (output parameter) : an upper-bound for the decompressed size of the data in the frame
	or ZSTD_CONTENTSIZE_ERROR if an error occurs

	note : assumes `cSize` and `dBound` are _not_ NULL.
	*/
	void ZSTDv01_findFrameSizeInfoLegacy(const void *src, size_t srcSize,
	size_t* cSize, unsigned long long* dBound);

	/**
	ZSTDv01_isError() : tells if the result of ZSTDv01_decompress() is an error
	*/
	unsigned ZSTDv01_isError(size_t code);


	/* *************************************
	* Advanced functions
	***************************************/
	typedef struct ZSTDv01_Dctx_s ZSTDv01_Dctx;
	ZSTDv01_Dctx* ZSTDv01_createDCtx(void);
	size_t ZSTDv01_freeDCtx(ZSTDv01_Dctx* dctx);

	size_t ZSTDv01_decompressDCtx(void* ctx,
	void* dst, size_t maxOriginalSize,
	const void* src, size_t compressedSize);

	/* *************************************
	* Streaming functions
	***************************************/
	size_t ZSTDv01_resetDCtx(ZSTDv01_Dctx* dctx);

	size_t ZSTDv01_nextSrcSizeToDecompress(ZSTDv01_Dctx* dctx);
	size_t ZSTDv01_decompressContinue(ZSTDv01_Dctx* dctx, void* dst, size_t maxDstSize, const void* src, size_t srcSize);
	/**
	Use above functions alternatively.
	ZSTD_nextSrcSizeToDecompress() tells how much bytes to provide as 'srcSize' to ZSTD_decompressContinue().
	ZSTD_decompressContinue() will use previous data blocks to improve compression if they are located prior to current block.
	Result is the number of bytes regenerated within 'dst'.
	It can be zero, which is not an error; it just means ZSTD_decompressContinue() has decoded some header.
	*/

	/* *************************************
	* Prefix - version detection
	***************************************/
	#define ZSTDv01_magicNumber 0xFD2FB51E /* Big Endian version */
	#define ZSTDv01_magicNumberLE 0x1EB52FFD /* Little Endian version */


	#if defined (__cplusplus)
	}
	#endif

	#endif /* ZSTD_V01_H_28739879432 */
	/** ended inlining zstd_v01.h **/
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 2)
	/** start inlining zstd_v02.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTD_V02_H_4174539423
	#define ZSTD_V02_H_4174539423

	#if defined (__cplusplus)
	extern "C" {
	#endif

	/* *************************************
	* Includes
	***************************************/
	#include <stddef.h> /* size_t */


	/* *************************************
	* Simple one-step function
	***************************************/
	/**
	ZSTDv02_decompress() : decompress ZSTD frames compliant with v0.2.x format
	compressedSize : is the exact source size
	maxOriginalSize : is the size of the 'dst' buffer, which must be already allocated.
	It must be equal or larger than originalSize, otherwise decompression will fail.
	return : the number of bytes decompressed into destination buffer (originalSize)
	or an errorCode if it fails (which can be tested using ZSTDv01_isError())
	*/
	size_t ZSTDv02_decompress( void* dst, size_t maxOriginalSize,
	const void* src, size_t compressedSize);

	/**
	ZSTDv02_findFrameSizeInfoLegacy() : get the source length and decompressed bound of a ZSTD frame compliant with v0.2.x format
	srcSize : The size of the 'src' buffer, at least as large as the frame pointed to by 'src'
	cSize (output parameter) : the number of bytes that would be read to decompress this frame
	or an error code if it fails (which can be tested using ZSTDv01_isError())
	dBound (output parameter) : an upper-bound for the decompressed size of the data in the frame
	or ZSTD_CONTENTSIZE_ERROR if an error occurs

	note : assumes `cSize` and `dBound` are _not_ NULL.
	*/
	void ZSTDv02_findFrameSizeInfoLegacy(const void *src, size_t srcSize,
	size_t* cSize, unsigned long long* dBound);

	/**
	ZSTDv02_isError() : tells if the result of ZSTDv02_decompress() is an error
	*/
	unsigned ZSTDv02_isError(size_t code);


	/* *************************************
	* Advanced functions
	***************************************/
	typedef struct ZSTDv02_Dctx_s ZSTDv02_Dctx;
	ZSTDv02_Dctx* ZSTDv02_createDCtx(void);
	size_t ZSTDv02_freeDCtx(ZSTDv02_Dctx* dctx);

	size_t ZSTDv02_decompressDCtx(void* ctx,
	void* dst, size_t maxOriginalSize,
	const void* src, size_t compressedSize);

	/* *************************************
	* Streaming functions
	***************************************/
	size_t ZSTDv02_resetDCtx(ZSTDv02_Dctx* dctx);

	size_t ZSTDv02_nextSrcSizeToDecompress(ZSTDv02_Dctx* dctx);
	size_t ZSTDv02_decompressContinue(ZSTDv02_Dctx* dctx, void* dst, size_t maxDstSize, const void* src, size_t srcSize);
	/**
	Use above functions alternatively.
	ZSTD_nextSrcSizeToDecompress() tells how much bytes to provide as 'srcSize' to ZSTD_decompressContinue().
	ZSTD_decompressContinue() will use previous data blocks to improve compression if they are located prior to current block.
	Result is the number of bytes regenerated within 'dst'.
	It can be zero, which is not an error; it just means ZSTD_decompressContinue() has decoded some header.
	*/

	/* *************************************
	* Prefix - version detection
	***************************************/
	#define ZSTDv02_magicNumber 0xFD2FB522 /* v0.2 */


	#if defined (__cplusplus)
	}
	#endif

	#endif /* ZSTD_V02_H_4174539423 */
	/** ended inlining zstd_v02.h **/
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 3)
	/** start inlining zstd_v03.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTD_V03_H_298734209782
	#define ZSTD_V03_H_298734209782

	#if defined (__cplusplus)
	extern "C" {
	#endif

	/* *************************************
	* Includes
	***************************************/
	#include <stddef.h> /* size_t */


	/* *************************************
	* Simple one-step function
	***************************************/
	/**
	ZSTDv03_decompress() : decompress ZSTD frames compliant with v0.3.x format
	compressedSize : is the exact source size
	maxOriginalSize : is the size of the 'dst' buffer, which must be already allocated.
	It must be equal or larger than originalSize, otherwise decompression will fail.
	return : the number of bytes decompressed into destination buffer (originalSize)
	or an errorCode if it fails (which can be tested using ZSTDv01_isError())
	*/
	size_t ZSTDv03_decompress( void* dst, size_t maxOriginalSize,
	const void* src, size_t compressedSize);

	/**
	ZSTDv03_findFrameSizeInfoLegacy() : get the source length and decompressed bound of a ZSTD frame compliant with v0.3.x format
	srcSize : The size of the 'src' buffer, at least as large as the frame pointed to by 'src'
	cSize (output parameter) : the number of bytes that would be read to decompress this frame
	or an error code if it fails (which can be tested using ZSTDv01_isError())
	dBound (output parameter) : an upper-bound for the decompressed size of the data in the frame
	or ZSTD_CONTENTSIZE_ERROR if an error occurs

	note : assumes `cSize` and `dBound` are _not_ NULL.
	*/
	void ZSTDv03_findFrameSizeInfoLegacy(const void *src, size_t srcSize,
	size_t* cSize, unsigned long long* dBound);

	/**
	ZSTDv03_isError() : tells if the result of ZSTDv03_decompress() is an error
	*/
	unsigned ZSTDv03_isError(size_t code);


	/* *************************************
	* Advanced functions
	***************************************/
	typedef struct ZSTDv03_Dctx_s ZSTDv03_Dctx;
	ZSTDv03_Dctx* ZSTDv03_createDCtx(void);
	size_t ZSTDv03_freeDCtx(ZSTDv03_Dctx* dctx);

	size_t ZSTDv03_decompressDCtx(void* ctx,
	void* dst, size_t maxOriginalSize,
	const void* src, size_t compressedSize);

	/* *************************************
	* Streaming functions
	***************************************/
	size_t ZSTDv03_resetDCtx(ZSTDv03_Dctx* dctx);

	size_t ZSTDv03_nextSrcSizeToDecompress(ZSTDv03_Dctx* dctx);
	size_t ZSTDv03_decompressContinue(ZSTDv03_Dctx* dctx, void* dst, size_t maxDstSize, const void* src, size_t srcSize);
	/**
	Use above functions alternatively.
	ZSTD_nextSrcSizeToDecompress() tells how much bytes to provide as 'srcSize' to ZSTD_decompressContinue().
	ZSTD_decompressContinue() will use previous data blocks to improve compression if they are located prior to current block.
	Result is the number of bytes regenerated within 'dst'.
	It can be zero, which is not an error; it just means ZSTD_decompressContinue() has decoded some header.
	*/

	/* *************************************
	* Prefix - version detection
	***************************************/
	#define ZSTDv03_magicNumber 0xFD2FB523 /* v0.3 */


	#if defined (__cplusplus)
	}
	#endif

	#endif /* ZSTD_V03_H_298734209782 */
	/** ended inlining zstd_v03.h **/
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 4)
	/** start inlining zstd_v04.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTD_V04_H_91868324769238
	#define ZSTD_V04_H_91868324769238

	#if defined (__cplusplus)
	extern "C" {
	#endif

	/* *************************************
	* Includes
	***************************************/
	#include <stddef.h> /* size_t */


	/* *************************************
	* Simple one-step function
	***************************************/
	/**
	ZSTDv04_decompress() : decompress ZSTD frames compliant with v0.4.x format
	compressedSize : is the exact source size
	maxOriginalSize : is the size of the 'dst' buffer, which must be already allocated.
	It must be equal or larger than originalSize, otherwise decompression will fail.
	return : the number of bytes decompressed into destination buffer (originalSize)
	or an errorCode if it fails (which can be tested using ZSTDv01_isError())
	*/
	size_t ZSTDv04_decompress( void* dst, size_t maxOriginalSize,
	const void* src, size_t compressedSize);

	/**
	ZSTDv04_findFrameSizeInfoLegacy() : get the source length and decompressed bound of a ZSTD frame compliant with v0.4.x format
	srcSize : The size of the 'src' buffer, at least as large as the frame pointed to by 'src'
	cSize (output parameter) : the number of bytes that would be read to decompress this frame
	or an error code if it fails (which can be tested using ZSTDv01_isError())
	dBound (output parameter) : an upper-bound for the decompressed size of the data in the frame
	or ZSTD_CONTENTSIZE_ERROR if an error occurs

	note : assumes `cSize` and `dBound` are _not_ NULL.
	*/
	void ZSTDv04_findFrameSizeInfoLegacy(const void *src, size_t srcSize,
	size_t* cSize, unsigned long long* dBound);

	/**
	ZSTDv04_isError() : tells if the result of ZSTDv04_decompress() is an error
	*/
	unsigned ZSTDv04_isError(size_t code);


	/* *************************************
	* Advanced functions
	***************************************/
	typedef struct ZSTDv04_Dctx_s ZSTDv04_Dctx;
	ZSTDv04_Dctx* ZSTDv04_createDCtx(void);
	size_t ZSTDv04_freeDCtx(ZSTDv04_Dctx* dctx);

	size_t ZSTDv04_decompressDCtx(ZSTDv04_Dctx* dctx,
	void* dst, size_t maxOriginalSize,
	const void* src, size_t compressedSize);


	/* *************************************
	* Direct Streaming
	***************************************/
	size_t ZSTDv04_resetDCtx(ZSTDv04_Dctx* dctx);

	size_t ZSTDv04_nextSrcSizeToDecompress(ZSTDv04_Dctx* dctx);
	size_t ZSTDv04_decompressContinue(ZSTDv04_Dctx* dctx, void* dst, size_t maxDstSize, const void* src, size_t srcSize);
	/**
	Use above functions alternatively.
	ZSTD_nextSrcSizeToDecompress() tells how much bytes to provide as 'srcSize' to ZSTD_decompressContinue().
	ZSTD_decompressContinue() will use previous data blocks to improve compression if they are located prior to current block.
	Result is the number of bytes regenerated within 'dst'.
	It can be zero, which is not an error; it just means ZSTD_decompressContinue() has decoded some header.
	*/


	/* *************************************
	* Buffered Streaming
	***************************************/
	typedef struct ZBUFFv04_DCtx_s ZBUFFv04_DCtx;
	ZBUFFv04_DCtx* ZBUFFv04_createDCtx(void);
	size_t ZBUFFv04_freeDCtx(ZBUFFv04_DCtx* dctx);

	size_t ZBUFFv04_decompressInit(ZBUFFv04_DCtx* dctx);
	size_t ZBUFFv04_decompressWithDictionary(ZBUFFv04_DCtx* dctx, const void* dict, size_t dictSize);

	size_t ZBUFFv04_decompressContinue(ZBUFFv04_DCtx* dctx, void* dst, size_t* maxDstSizePtr, const void* src, size_t* srcSizePtr);

	/ **********************************************
	* Streaming decompression
	*
	* A ZBUFF_DCtx object is required to track streaming operation.
	* Use ZBUFF_createDCtx() and ZBUFF_freeDCtx() to create/release resources.
	* Use ZBUFF_decompressInit() to start a new decompression operation.
	* ZBUFF_DCtx objects can be reused multiple times.
	*
	* Optionally, a reference to a static dictionary can be set, using ZBUFF_decompressWithDictionary()
	* It must be the same content as the one set during compression phase.
	* Dictionary content must remain accessible during the decompression process.
	*
	* Use ZBUFF_decompressContinue() repetitively to consume your input.
	* srcSizePtr and maxDstSizePtr can be any size.
	* The function will report how many bytes were read or written by modifying srcSizePtr and maxDstSizePtr.
	* Note that it may not consume the entire input, in which case it's up to the caller to present remaining input again.
	* The content of dst will be overwritten (up to *maxDstSizePtr) at each function call, so save its content if it matters or change dst.
	* @return : a hint to preferred nb of bytes to use as input for next function call (it's only a hint, to improve latency)
	* or 0 when a frame is completely decoded
	* or an error code, which can be tested using ZBUFF_isError().
	*
	* Hint : recommended buffer sizes (not compulsory) : ZBUFF_recommendedDInSize / ZBUFF_recommendedDOutSize
	* output : ZBUFF_recommendedDOutSize==128 KB block size is the internal unit, it ensures it's always possible to write a full block when it's decoded.
	* input : ZBUFF_recommendedDInSize==128Kb+3; just follow indications from ZBUFF_decompressContinue() to minimize latency. It should always be <= 128 KB + 3 .
	* **************************************************/
	unsigned ZBUFFv04_isError(size_t errorCode);
	const char* ZBUFFv04_getErrorName(size_t errorCode);


	/** The below functions provide recommended buffer sizes for Compression or Decompression operations.
	* These sizes are not compulsory, they just tend to offer better latency */
	size_t ZBUFFv04_recommendedDInSize(void);
	size_t ZBUFFv04_recommendedDOutSize(void);


	/* *************************************
	* Prefix - version detection
	***************************************/
	#define ZSTDv04_magicNumber 0xFD2FB524 /* v0.4 */


	#if defined (__cplusplus)
	}
	#endif

	#endif /* ZSTD_V04_H_91868324769238 */
	/** ended inlining zstd_v04.h **/
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 5)
	/** start inlining zstd_v05.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTDv05_H
	#define ZSTDv05_H

	#if defined (__cplusplus)
	extern "C" {
	#endif

	/-************************************
	* Dependencies
	***************************************/
	#include <stddef.h> /* size_t */
	/** skipping file: ../common/mem.h **/


	/* *************************************
	* Simple functions
	***************************************/
	/*! ZSTDv05_decompress() :
	`compressedSize` : is the _exact_ size of the compressed blob, otherwise decompression will fail.
	`dstCapacity` must be large enough, equal or larger than originalSize.
	@return : the number of bytes decompressed into `dst` (<= `dstCapacity`),
	or an errorCode if it fails (which can be tested using ZSTDv05_isError()) */
	size_t ZSTDv05_decompress( void* dst, size_t dstCapacity,
	const void* src, size_t compressedSize);

	/**
	ZSTDv05_findFrameSizeInfoLegacy() : get the source length and decompressed bound of a ZSTD frame compliant with v0.5.x format
	srcSize : The size of the 'src' buffer, at least as large as the frame pointed to by 'src'
	cSize (output parameter) : the number of bytes that would be read to decompress this frame
	or an error code if it fails (which can be tested using ZSTDv01_isError())
	dBound (output parameter) : an upper-bound for the decompressed size of the data in the frame
	or ZSTD_CONTENTSIZE_ERROR if an error occurs

	note : assumes `cSize` and `dBound` are _not_ NULL.
	*/
	void ZSTDv05_findFrameSizeInfoLegacy(const void *src, size_t srcSize,
	size_t* cSize, unsigned long long* dBound);

	/* *************************************
	* Helper functions
	***************************************/
	/* Error Management */
	unsigned ZSTDv05_isError(size_t code); /!< tells if a `size_t` function result is an error code /
	const char* ZSTDv05_getErrorName(size_t code); /!< provides readable string for an error code /


	/* *************************************
	* Explicit memory management
	***************************************/
	/** Decompression context */
	typedef struct ZSTDv05_DCtx_s ZSTDv05_DCtx;
	ZSTDv05_DCtx* ZSTDv05_createDCtx(void);
	size_t ZSTDv05_freeDCtx(ZSTDv05_DCtx* dctx); /!< @return : errorCode /

	/** ZSTDv05_decompressDCtx() :
	* Same as ZSTDv05_decompress(), but requires an already allocated ZSTDv05_DCtx (see ZSTDv05_createDCtx()) */
	size_t ZSTDv05_decompressDCtx(ZSTDv05_DCtx* ctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);


	/-**********************
	* Simple Dictionary API
	*************************/
	/*! ZSTDv05_decompress_usingDict() :
	* Decompression using a pre-defined Dictionary content (see dictBuilder).
	* Dictionary must be identical to the one used during compression, otherwise regenerated data will be corrupted.
	* Note : dict can be NULL, in which case, it's equivalent to ZSTDv05_decompressDCtx() */
	size_t ZSTDv05_decompress_usingDict(ZSTDv05_DCtx* dctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const void* dict,size_t dictSize);

	/-***********************
	* Advanced Streaming API
	***************************/
	typedef enum { ZSTDv05_fast, ZSTDv05_greedy, ZSTDv05_lazy, ZSTDv05_lazy2, ZSTDv05_btlazy2, ZSTDv05_opt, ZSTDv05_btopt } ZSTDv05_strategy;
	typedef struct {
	U64 srcSize;
	U32 windowLog; /* the only useful information to retrieve */
	U32 contentLog; U32 hashLog; U32 searchLog; U32 searchLength; U32 targetLength; ZSTDv05_strategy strategy;
	} ZSTDv05_parameters;
	size_t ZSTDv05_getFrameParams(ZSTDv05_parameters* params, const void* src, size_t srcSize);

	size_t ZSTDv05_decompressBegin_usingDict(ZSTDv05_DCtx* dctx, const void* dict, size_t dictSize);
	void ZSTDv05_copyDCtx(ZSTDv05_DCtx* dstDCtx, const ZSTDv05_DCtx* srcDCtx);
	size_t ZSTDv05_nextSrcSizeToDecompress(ZSTDv05_DCtx* dctx);
	size_t ZSTDv05_decompressContinue(ZSTDv05_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);


	/-**********************
	* ZBUFF API
	*************************/
	typedef struct ZBUFFv05_DCtx_s ZBUFFv05_DCtx;
	ZBUFFv05_DCtx* ZBUFFv05_createDCtx(void);
	size_t ZBUFFv05_freeDCtx(ZBUFFv05_DCtx* dctx);

	size_t ZBUFFv05_decompressInit(ZBUFFv05_DCtx* dctx);
	size_t ZBUFFv05_decompressInitDictionary(ZBUFFv05_DCtx* dctx, const void* dict, size_t dictSize);

	size_t ZBUFFv05_decompressContinue(ZBUFFv05_DCtx* dctx,
	void* dst, size_t* dstCapacityPtr,
	const void* src, size_t* srcSizePtr);

	/-**************************************************************************
	* Streaming decompression
	*
	* A ZBUFFv05_DCtx object is required to track streaming operations.
	* Use ZBUFFv05_createDCtx() and ZBUFFv05_freeDCtx() to create/release resources.
	* Use ZBUFFv05_decompressInit() to start a new decompression operation,
	* or ZBUFFv05_decompressInitDictionary() if decompression requires a dictionary.
	* Note that ZBUFFv05_DCtx objects can be reused multiple times.
	*
	* Use ZBUFFv05_decompressContinue() repetitively to consume your input.
	* srcSizePtr and dstCapacityPtr can be any size.
	* The function will report how many bytes were read or written by modifying srcSizePtr and dstCapacityPtr.
	* Note that it may not consume the entire input, in which case it's up to the caller to present remaining input again.
	* The content of @dst will be overwritten (up to *dstCapacityPtr) at each function call, so save its content if it matters or change @dst.
	* @return : a hint to preferred nb of bytes to use as input for next function call (it's only a hint, to help latency)
	* or 0 when a frame is completely decoded
	* or an error code, which can be tested using ZBUFFv05_isError().
	*
	* Hint : recommended buffer sizes (not compulsory) : ZBUFFv05_recommendedDInSize() / ZBUFFv05_recommendedDOutSize()
	* output : ZBUFFv05_recommendedDOutSize==128 KB block size is the internal unit, it ensures it's always possible to write a full block when decoded.
	* input : ZBUFFv05_recommendedDInSize==128Kb+3; just follow indications from ZBUFFv05_decompressContinue() to minimize latency. It should always be <= 128 KB + 3 .
	* *******************************************************************************/


	/* *************************************
	* Tool functions
	***************************************/
	unsigned ZBUFFv05_isError(size_t errorCode);
	const char* ZBUFFv05_getErrorName(size_t errorCode);

	/** Functions below provide recommended buffer sizes for Compression or Decompression operations.
	* These sizes are just hints, and tend to offer better latency */
	size_t ZBUFFv05_recommendedDInSize(void);
	size_t ZBUFFv05_recommendedDOutSize(void);



	/-************************************
	* Constants
	***************************************/
	#define ZSTDv05_MAGICNUMBER 0xFD2FB525 /* v0.5 */




	#if defined (__cplusplus)
	}
	#endif

	#endif /* ZSTDv0505_H */
	/** ended inlining zstd_v05.h **/
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 6)
	/** start inlining zstd_v06.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTDv06_H
	#define ZSTDv06_H

	#if defined (__cplusplus)
	extern "C" {
	#endif

	/====== Dependency ======/
	#include <stddef.h> /* size_t */


	/====== Export for Windows ======/
	/*!
	* ZSTDv06_DLL_EXPORT :
	* Enable exporting of functions when building a Windows DLL
	*/
	#if defined(_WIN32) && defined(ZSTDv06_DLL_EXPORT) && (ZSTDv06_DLL_EXPORT==1)
	# define ZSTDLIBv06_API __declspec(dllexport)
	#else
	# define ZSTDLIBv06_API
	#endif


	/* *************************************
	* Simple functions
	***************************************/
	/*! ZSTDv06_decompress() :
	`compressedSize` : is the _exact_ size of the compressed blob, otherwise decompression will fail.
	`dstCapacity` must be large enough, equal or larger than originalSize.
	@return : the number of bytes decompressed into `dst` (<= `dstCapacity`),
	or an errorCode if it fails (which can be tested using ZSTDv06_isError()) */
	ZSTDLIBv06_API size_t ZSTDv06_decompress( void* dst, size_t dstCapacity,
	const void* src, size_t compressedSize);

	/**
	ZSTDv06_findFrameSizeInfoLegacy() : get the source length and decompressed bound of a ZSTD frame compliant with v0.6.x format
	srcSize : The size of the 'src' buffer, at least as large as the frame pointed to by 'src'
	cSize (output parameter) : the number of bytes that would be read to decompress this frame
	or an error code if it fails (which can be tested using ZSTDv01_isError())
	dBound (output parameter) : an upper-bound for the decompressed size of the data in the frame
	or ZSTD_CONTENTSIZE_ERROR if an error occurs

	note : assumes `cSize` and `dBound` are _not_ NULL.
	*/
	void ZSTDv06_findFrameSizeInfoLegacy(const void *src, size_t srcSize,
	size_t* cSize, unsigned long long* dBound);

	/* *************************************
	* Helper functions
	***************************************/
	ZSTDLIBv06_API size_t ZSTDv06_compressBound(size_t srcSize); /!< maximum compressed size (worst case scenario) /

	/* Error Management */
	ZSTDLIBv06_API unsigned ZSTDv06_isError(size_t code); /!< tells if a `size_t` function result is an error code /
	ZSTDLIBv06_API const char* ZSTDv06_getErrorName(size_t code); /!< provides readable string for an error code /


	/* *************************************
	* Explicit memory management
	***************************************/
	/** Decompression context */
	typedef struct ZSTDv06_DCtx_s ZSTDv06_DCtx;
	ZSTDLIBv06_API ZSTDv06_DCtx* ZSTDv06_createDCtx(void);
	ZSTDLIBv06_API size_t ZSTDv06_freeDCtx(ZSTDv06_DCtx* dctx); /!< @return : errorCode /

	/** ZSTDv06_decompressDCtx() :
	* Same as ZSTDv06_decompress(), but requires an already allocated ZSTDv06_DCtx (see ZSTDv06_createDCtx()) */
	ZSTDLIBv06_API size_t ZSTDv06_decompressDCtx(ZSTDv06_DCtx* ctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);


	/-**********************
	* Dictionary API
	*************************/
	/*! ZSTDv06_decompress_usingDict() :
	* Decompression using a pre-defined Dictionary content (see dictBuilder).
	* Dictionary must be identical to the one used during compression, otherwise regenerated data will be corrupted.
	* Note : dict can be NULL, in which case, it's equivalent to ZSTDv06_decompressDCtx() */
	ZSTDLIBv06_API size_t ZSTDv06_decompress_usingDict(ZSTDv06_DCtx* dctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const void* dict,size_t dictSize);


	/-***********************
	* Advanced Streaming API
	***************************/
	struct ZSTDv06_frameParams_s { unsigned long long frameContentSize; unsigned windowLog; };
	typedef struct ZSTDv06_frameParams_s ZSTDv06_frameParams;

	ZSTDLIBv06_API size_t ZSTDv06_getFrameParams(ZSTDv06_frameParams* fparamsPtr, const void* src, size_t srcSize); /*< doesn't consume input /
	ZSTDLIBv06_API size_t ZSTDv06_decompressBegin_usingDict(ZSTDv06_DCtx* dctx, const void* dict, size_t dictSize);
	ZSTDLIBv06_API void ZSTDv06_copyDCtx(ZSTDv06_DCtx* dctx, const ZSTDv06_DCtx* preparedDCtx);

	ZSTDLIBv06_API size_t ZSTDv06_nextSrcSizeToDecompress(ZSTDv06_DCtx* dctx);
	ZSTDLIBv06_API size_t ZSTDv06_decompressContinue(ZSTDv06_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);



	/* *************************************
	* ZBUFF API
	***************************************/

	typedef struct ZBUFFv06_DCtx_s ZBUFFv06_DCtx;
	ZSTDLIBv06_API ZBUFFv06_DCtx* ZBUFFv06_createDCtx(void);
	ZSTDLIBv06_API size_t ZBUFFv06_freeDCtx(ZBUFFv06_DCtx* dctx);

	ZSTDLIBv06_API size_t ZBUFFv06_decompressInit(ZBUFFv06_DCtx* dctx);
	ZSTDLIBv06_API size_t ZBUFFv06_decompressInitDictionary(ZBUFFv06_DCtx* dctx, const void* dict, size_t dictSize);

	ZSTDLIBv06_API size_t ZBUFFv06_decompressContinue(ZBUFFv06_DCtx* dctx,
	void* dst, size_t* dstCapacityPtr,
	const void* src, size_t* srcSizePtr);

	/-**************************************************************************
	* Streaming decompression howto
	*
	* A ZBUFFv06_DCtx object is required to track streaming operations.
	* Use ZBUFFv06_createDCtx() and ZBUFFv06_freeDCtx() to create/release resources.
	* Use ZBUFFv06_decompressInit() to start a new decompression operation,
	* or ZBUFFv06_decompressInitDictionary() if decompression requires a dictionary.
	* Note that ZBUFFv06_DCtx objects can be re-init multiple times.
	*
	* Use ZBUFFv06_decompressContinue() repetitively to consume your input.
	* srcSizePtr and dstCapacityPtr can be any size.
	* The function will report how many bytes were read or written by modifying srcSizePtr and dstCapacityPtr.
	* Note that it may not consume the entire input, in which case it's up to the caller to present remaining input again.
	* The content of `dst` will be overwritten (up to *dstCapacityPtr) at each function call, so save its content if it matters, or change `dst`.
	* @return : a hint to preferred nb of bytes to use as input for next function call (it's only a hint, to help latency),
	* or 0 when a frame is completely decoded,
	* or an error code, which can be tested using ZBUFFv06_isError().
	*
	* Hint : recommended buffer sizes (not compulsory) : ZBUFFv06_recommendedDInSize() and ZBUFFv06_recommendedDOutSize()
	* output : ZBUFFv06_recommendedDOutSize== 128 KB block size is the internal unit, it ensures it's always possible to write a full block when decoded.
	* input : ZBUFFv06_recommendedDInSize == 128KB + 3;
	* just follow indications from ZBUFFv06_decompressContinue() to minimize latency. It should always be <= 128 KB + 3 .
	* *******************************************************************************/


	/* *************************************
	* Tool functions
	***************************************/
	ZSTDLIBv06_API unsigned ZBUFFv06_isError(size_t errorCode);
	ZSTDLIBv06_API const char* ZBUFFv06_getErrorName(size_t errorCode);

	/** Functions below provide recommended buffer sizes for Compression or Decompression operations.
	* These sizes are just hints, they tend to offer better latency */
	ZSTDLIBv06_API size_t ZBUFFv06_recommendedDInSize(void);
	ZSTDLIBv06_API size_t ZBUFFv06_recommendedDOutSize(void);


	/-************************************
	* Constants
	***************************************/
	#define ZSTDv06_MAGICNUMBER 0xFD2FB526 /* v0.6 */



	#if defined (__cplusplus)
	}
	#endif

	#endif /* ZSTDv06_BUFFERED_H */
	/** ended inlining zstd_v06.h **/
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 7)
	/** start inlining zstd_v07.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	#ifndef ZSTDv07_H_235446
	#define ZSTDv07_H_235446

	#if defined (__cplusplus)
	extern "C" {
	#endif

	/====== Dependency ======/
	#include <stddef.h> /* size_t */


	/====== Export for Windows ======/
	/*!
	* ZSTDv07_DLL_EXPORT :
	* Enable exporting of functions when building a Windows DLL
	*/
	#if defined(_WIN32) && defined(ZSTDv07_DLL_EXPORT) && (ZSTDv07_DLL_EXPORT==1)
	# define ZSTDLIBv07_API __declspec(dllexport)
	#else
	# define ZSTDLIBv07_API
	#endif


	/* *************************************
	* Simple API
	***************************************/
	/*! ZSTDv07_getDecompressedSize() :
	* @return : decompressed size if known, 0 otherwise.
	note 1 : if `0`, follow up with ZSTDv07_getFrameParams() to know precise failure cause.
	note 2 : decompressed size could be wrong or intentionally modified !
	always ensure results fit within application's authorized limits */
	unsigned long long ZSTDv07_getDecompressedSize(const void* src, size_t srcSize);

	/*! ZSTDv07_decompress() :
	`compressedSize` : must be _exact_ size of compressed input, otherwise decompression will fail.
	`dstCapacity` must be equal or larger than originalSize.
	@return : the number of bytes decompressed into `dst` (<= `dstCapacity`),
	or an errorCode if it fails (which can be tested using ZSTDv07_isError()) */
	ZSTDLIBv07_API size_t ZSTDv07_decompress( void* dst, size_t dstCapacity,
	const void* src, size_t compressedSize);

	/**
	ZSTDv07_findFrameSizeInfoLegacy() : get the source length and decompressed bound of a ZSTD frame compliant with v0.7.x format
	srcSize : The size of the 'src' buffer, at least as large as the frame pointed to by 'src'
	cSize (output parameter) : the number of bytes that would be read to decompress this frame
	or an error code if it fails (which can be tested using ZSTDv01_isError())
	dBound (output parameter) : an upper-bound for the decompressed size of the data in the frame
	or ZSTD_CONTENTSIZE_ERROR if an error occurs

	note : assumes `cSize` and `dBound` are _not_ NULL.
	*/
	void ZSTDv07_findFrameSizeInfoLegacy(const void *src, size_t srcSize,
	size_t* cSize, unsigned long long* dBound);

	/====== Helper functions ======/
	ZSTDLIBv07_API unsigned ZSTDv07_isError(size_t code); /!< tells if a `size_t` function result is an error code /
	ZSTDLIBv07_API const char* ZSTDv07_getErrorName(size_t code); /!< provides readable string from an error code /


	/-************************************
	* Explicit memory management
	***************************************/
	/** Decompression context */
	typedef struct ZSTDv07_DCtx_s ZSTDv07_DCtx;
	ZSTDLIBv07_API ZSTDv07_DCtx* ZSTDv07_createDCtx(void);
	ZSTDLIBv07_API size_t ZSTDv07_freeDCtx(ZSTDv07_DCtx* dctx); /!< @return : errorCode /

	/** ZSTDv07_decompressDCtx() :
	* Same as ZSTDv07_decompress(), requires an allocated ZSTDv07_DCtx (see ZSTDv07_createDCtx()) */
	ZSTDLIBv07_API size_t ZSTDv07_decompressDCtx(ZSTDv07_DCtx* ctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);


	/-***********************
	* Simple dictionary API
	***************************/
	/*! ZSTDv07_decompress_usingDict() :
	* Decompression using a pre-defined Dictionary content (see dictBuilder).
	* Dictionary must be identical to the one used during compression.
	* Note : This function load the dictionary, resulting in a significant startup time */
	ZSTDLIBv07_API size_t ZSTDv07_decompress_usingDict(ZSTDv07_DCtx* dctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const void* dict,size_t dictSize);


	/-*************************
	* Advanced Dictionary API
	****************************/
	/*! ZSTDv07_createDDict() :
	* Create a digested dictionary, ready to start decompression operation without startup delay.
	* `dict` can be released after creation */
	typedef struct ZSTDv07_DDict_s ZSTDv07_DDict;
	ZSTDLIBv07_API ZSTDv07_DDict* ZSTDv07_createDDict(const void* dict, size_t dictSize);
	ZSTDLIBv07_API size_t ZSTDv07_freeDDict(ZSTDv07_DDict* ddict);

	/*! ZSTDv07_decompress_usingDDict() :
	* Decompression using a pre-digested Dictionary
	* Faster startup than ZSTDv07_decompress_usingDict(), recommended when same dictionary is used multiple times. */
	ZSTDLIBv07_API size_t ZSTDv07_decompress_usingDDict(ZSTDv07_DCtx* dctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const ZSTDv07_DDict* ddict);

	typedef struct {
	unsigned long long frameContentSize;
	unsigned windowSize;
	unsigned dictID;
	unsigned checksumFlag;
	} ZSTDv07_frameParams;

	ZSTDLIBv07_API size_t ZSTDv07_getFrameParams(ZSTDv07_frameParams* fparamsPtr, const void* src, size_t srcSize); /*< doesn't consume input /




	/* *************************************
	* Streaming functions
	***************************************/
	typedef struct ZBUFFv07_DCtx_s ZBUFFv07_DCtx;
	ZSTDLIBv07_API ZBUFFv07_DCtx* ZBUFFv07_createDCtx(void);
	ZSTDLIBv07_API size_t ZBUFFv07_freeDCtx(ZBUFFv07_DCtx* dctx);

	ZSTDLIBv07_API size_t ZBUFFv07_decompressInit(ZBUFFv07_DCtx* dctx);
	ZSTDLIBv07_API size_t ZBUFFv07_decompressInitDictionary(ZBUFFv07_DCtx* dctx, const void* dict, size_t dictSize);

	ZSTDLIBv07_API size_t ZBUFFv07_decompressContinue(ZBUFFv07_DCtx* dctx,
	void* dst, size_t* dstCapacityPtr,
	const void* src, size_t* srcSizePtr);

	/-**************************************************************************
	* Streaming decompression howto
	*
	* A ZBUFFv07_DCtx object is required to track streaming operations.
	* Use ZBUFFv07_createDCtx() and ZBUFFv07_freeDCtx() to create/release resources.
	* Use ZBUFFv07_decompressInit() to start a new decompression operation,
	* or ZBUFFv07_decompressInitDictionary() if decompression requires a dictionary.
	* Note that ZBUFFv07_DCtx objects can be re-init multiple times.
	*
	* Use ZBUFFv07_decompressContinue() repetitively to consume your input.
	* srcSizePtr and dstCapacityPtr can be any size.
	* The function will report how many bytes were read or written by modifying srcSizePtr and dstCapacityPtr.
	* Note that it may not consume the entire input, in which case it's up to the caller to present remaining input again.
	* The content of `dst` will be overwritten (up to *dstCapacityPtr) at each function call, so save its content if it matters, or change `dst`.
	* @return : a hint to preferred nb of bytes to use as input for next function call (it's only a hint, to help latency),
	* or 0 when a frame is completely decoded,
	* or an error code, which can be tested using ZBUFFv07_isError().
	*
	* Hint : recommended buffer sizes (not compulsory) : ZBUFFv07_recommendedDInSize() and ZBUFFv07_recommendedDOutSize()
	* output : ZBUFFv07_recommendedDOutSize== 128 KB block size is the internal unit, it ensures it's always possible to write a full block when decoded.
	* input : ZBUFFv07_recommendedDInSize == 128KB + 3;
	* just follow indications from ZBUFFv07_decompressContinue() to minimize latency. It should always be <= 128 KB + 3 .
	* *******************************************************************************/


	/* *************************************
	* Tool functions
	***************************************/
	ZSTDLIBv07_API unsigned ZBUFFv07_isError(size_t errorCode);
	ZSTDLIBv07_API const char* ZBUFFv07_getErrorName(size_t errorCode);

	/** Functions below provide recommended buffer sizes for Compression or Decompression operations.
	* These sizes are just hints, they tend to offer better latency */
	ZSTDLIBv07_API size_t ZBUFFv07_recommendedDInSize(void);
	ZSTDLIBv07_API size_t ZBUFFv07_recommendedDOutSize(void);


	/-************************************
	* Constants
	***************************************/
	#define ZSTDv07_MAGICNUMBER 0xFD2FB527 /* v0.7 */


	#if defined (__cplusplus)
	}
	#endif

	#endif /* ZSTDv07_H_235446 */
	/** ended inlining zstd_v07.h **/
	#endif

	/** ZSTD_isLegacy() :
	@return : > 0 if supported by legacy decoder. 0 otherwise.
	return value is the version.
	*/
	MEM_STATIC unsigned ZSTD_isLegacy(const void* src, size_t srcSize)
	{
	U32 magicNumberLE;
	if (srcSize<4) return 0;
	magicNumberLE = MEM_readLE32(src);
	switch(magicNumberLE)
	{
	#if (ZSTD_LEGACY_SUPPORT <= 1)
	case ZSTDv01_magicNumberLE:return 1;
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 2)
	case ZSTDv02_magicNumber : return 2;
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 3)
	case ZSTDv03_magicNumber : return 3;
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 4)
	case ZSTDv04_magicNumber : return 4;
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 5)
	case ZSTDv05_MAGICNUMBER : return 5;
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 6)
	case ZSTDv06_MAGICNUMBER : return 6;
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 7)
	case ZSTDv07_MAGICNUMBER : return 7;
	#endif
	default : return 0;
	}
	}


	MEM_STATIC unsigned long long ZSTD_getDecompressedSize_legacy(const void* src, size_t srcSize)
	{
	U32 const version = ZSTD_isLegacy(src, srcSize);
	if (version < 5) return 0; /* no decompressed size in frame header, or not a legacy format */
	#if (ZSTD_LEGACY_SUPPORT <= 5)
	if (version==5) {
	ZSTDv05_parameters fParams;
	size_t const frResult = ZSTDv05_getFrameParams(&fParams, src, srcSize);
	if (frResult != 0) return 0;
	return fParams.srcSize;
	}
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 6)
	if (version==6) {
	ZSTDv06_frameParams fParams;
	size_t const frResult = ZSTDv06_getFrameParams(&fParams, src, srcSize);
	if (frResult != 0) return 0;
	return fParams.frameContentSize;
	}
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 7)
	if (version==7) {
	ZSTDv07_frameParams fParams;
	size_t const frResult = ZSTDv07_getFrameParams(&fParams, src, srcSize);
	if (frResult != 0) return 0;
	return fParams.frameContentSize;
	}
	#endif
	return 0; /* should not be possible */
	}


	MEM_STATIC size_t ZSTD_decompressLegacy(
	void* dst, size_t dstCapacity,
	const void* src, size_t compressedSize,
	const void* dict,size_t dictSize)
	{
	U32 const version = ZSTD_isLegacy(src, compressedSize);
	(void)dst; (void)dstCapacity; (void)dict; (void)dictSize; /* unused when ZSTD_LEGACY_SUPPORT >= 8 */
	switch(version)
	{
	#if (ZSTD_LEGACY_SUPPORT <= 1)
	case 1 :
	return ZSTDv01_decompress(dst, dstCapacity, src, compressedSize);
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 2)
	case 2 :
	return ZSTDv02_decompress(dst, dstCapacity, src, compressedSize);
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 3)
	case 3 :
	return ZSTDv03_decompress(dst, dstCapacity, src, compressedSize);
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 4)
	case 4 :
	return ZSTDv04_decompress(dst, dstCapacity, src, compressedSize);
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 5)
	case 5 :
	{ size_t result;
	ZSTDv05_DCtx* const zd = ZSTDv05_createDCtx();
	if (zd==NULL) return ERROR(memory_allocation);
	result = ZSTDv05_decompress_usingDict(zd, dst, dstCapacity, src, compressedSize, dict, dictSize);
	ZSTDv05_freeDCtx(zd);
	return result;
	}
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 6)
	case 6 :
	{ size_t result;
	ZSTDv06_DCtx* const zd = ZSTDv06_createDCtx();
	if (zd==NULL) return ERROR(memory_allocation);
	result = ZSTDv06_decompress_usingDict(zd, dst, dstCapacity, src, compressedSize, dict, dictSize);
	ZSTDv06_freeDCtx(zd);
	return result;
	}
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 7)
	case 7 :
	{ size_t result;
	ZSTDv07_DCtx* const zd = ZSTDv07_createDCtx();
	if (zd==NULL) return ERROR(memory_allocation);
	result = ZSTDv07_decompress_usingDict(zd, dst, dstCapacity, src, compressedSize, dict, dictSize);
	ZSTDv07_freeDCtx(zd);
	return result;
	}
	#endif
	default :
	return ERROR(prefix_unknown);
	}
	}

	MEM_STATIC ZSTD_frameSizeInfo ZSTD_findFrameSizeInfoLegacy(const void *src, size_t srcSize)
	{
	ZSTD_frameSizeInfo frameSizeInfo;
	U32 const version = ZSTD_isLegacy(src, srcSize);
	switch(version)
	{
	#if (ZSTD_LEGACY_SUPPORT <= 1)
	case 1 :
	ZSTDv01_findFrameSizeInfoLegacy(src, srcSize,
	&frameSizeInfo.compressedSize,
	&frameSizeInfo.decompressedBound);
	break;
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 2)
	case 2 :
	ZSTDv02_findFrameSizeInfoLegacy(src, srcSize,
	&frameSizeInfo.compressedSize,
	&frameSizeInfo.decompressedBound);
	break;
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 3)
	case 3 :
	ZSTDv03_findFrameSizeInfoLegacy(src, srcSize,
	&frameSizeInfo.compressedSize,
	&frameSizeInfo.decompressedBound);
	break;
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 4)
	case 4 :
	ZSTDv04_findFrameSizeInfoLegacy(src, srcSize,
	&frameSizeInfo.compressedSize,
	&frameSizeInfo.decompressedBound);
	break;
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 5)
	case 5 :
	ZSTDv05_findFrameSizeInfoLegacy(src, srcSize,
	&frameSizeInfo.compressedSize,
	&frameSizeInfo.decompressedBound);
	break;
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 6)
	case 6 :
	ZSTDv06_findFrameSizeInfoLegacy(src, srcSize,
	&frameSizeInfo.compressedSize,
	&frameSizeInfo.decompressedBound);
	break;
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 7)
	case 7 :
	ZSTDv07_findFrameSizeInfoLegacy(src, srcSize,
	&frameSizeInfo.compressedSize,
	&frameSizeInfo.decompressedBound);
	break;
	#endif
	default :
	frameSizeInfo.compressedSize = ERROR(prefix_unknown);
	frameSizeInfo.decompressedBound = ZSTD_CONTENTSIZE_ERROR;
	break;
	}
	if (!ZSTD_isError(frameSizeInfo.compressedSize) && frameSizeInfo.compressedSize > srcSize) {
	frameSizeInfo.compressedSize = ERROR(srcSize_wrong);
	frameSizeInfo.decompressedBound = ZSTD_CONTENTSIZE_ERROR;
	}
	return frameSizeInfo;
	}

	MEM_STATIC size_t ZSTD_findFrameCompressedSizeLegacy(const void *src, size_t srcSize)
	{
	ZSTD_frameSizeInfo frameSizeInfo = ZSTD_findFrameSizeInfoLegacy(src, srcSize);
	return frameSizeInfo.compressedSize;
	}

	MEM_STATIC size_t ZSTD_freeLegacyStreamContext(void* legacyContext, U32 version)
	{
	switch(version)
	{
	default :
	case 1 :
	case 2 :
	case 3 :
	(void)legacyContext;
	return ERROR(version_unsupported);
	#if (ZSTD_LEGACY_SUPPORT <= 4)
	case 4 : return ZBUFFv04_freeDCtx((ZBUFFv04_DCtx*)legacyContext);
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 5)
	case 5 : return ZBUFFv05_freeDCtx((ZBUFFv05_DCtx*)legacyContext);
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 6)
	case 6 : return ZBUFFv06_freeDCtx((ZBUFFv06_DCtx*)legacyContext);
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 7)
	case 7 : return ZBUFFv07_freeDCtx((ZBUFFv07_DCtx*)legacyContext);
	#endif
	}
	}


	MEM_STATIC size_t ZSTD_initLegacyStream(void** legacyContext, U32 prevVersion, U32 newVersion,
	const void* dict, size_t dictSize)
	{
	DEBUGLOG(5, "ZSTD_initLegacyStream for v0.%u", newVersion);
	if (prevVersion != newVersion) ZSTD_freeLegacyStreamContext(*legacyContext, prevVersion);
	switch(newVersion)
	{
	default :
	case 1 :
	case 2 :
	case 3 :
	(void)dict; (void)dictSize;
	return 0;
	#if (ZSTD_LEGACY_SUPPORT <= 4)
	case 4 :
	{
	ZBUFFv04_DCtx* dctx = (prevVersion != newVersion) ? ZBUFFv04_createDCtx() : (ZBUFFv04_DCtx)legacyContext;
	if (dctx==NULL) return ERROR(memory_allocation);
	ZBUFFv04_decompressInit(dctx);
	ZBUFFv04_decompressWithDictionary(dctx, dict, dictSize);
	*legacyContext = dctx;
	return 0;
	}
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 5)
	case 5 :
	{
	ZBUFFv05_DCtx* dctx = (prevVersion != newVersion) ? ZBUFFv05_createDCtx() : (ZBUFFv05_DCtx)legacyContext;
	if (dctx==NULL) return ERROR(memory_allocation);
	ZBUFFv05_decompressInitDictionary(dctx, dict, dictSize);
	*legacyContext = dctx;
	return 0;
	}
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 6)
	case 6 :
	{
	ZBUFFv06_DCtx* dctx = (prevVersion != newVersion) ? ZBUFFv06_createDCtx() : (ZBUFFv06_DCtx)legacyContext;
	if (dctx==NULL) return ERROR(memory_allocation);
	ZBUFFv06_decompressInitDictionary(dctx, dict, dictSize);
	*legacyContext = dctx;
	return 0;
	}
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 7)
	case 7 :
	{
	ZBUFFv07_DCtx* dctx = (prevVersion != newVersion) ? ZBUFFv07_createDCtx() : (ZBUFFv07_DCtx)legacyContext;
	if (dctx==NULL) return ERROR(memory_allocation);
	ZBUFFv07_decompressInitDictionary(dctx, dict, dictSize);
	*legacyContext = dctx;
	return 0;
	}
	#endif
	}
	}



	MEM_STATIC size_t ZSTD_decompressLegacyStream(void* legacyContext, U32 version,
	ZSTD_outBuffer* output, ZSTD_inBuffer* input)
	{
	DEBUGLOG(5, "ZSTD_decompressLegacyStream for v0.%u", version);
	switch(version)
	{
	default :
	case 1 :
	case 2 :
	case 3 :
	(void)legacyContext; (void)output; (void)input;
	return ERROR(version_unsupported);
	#if (ZSTD_LEGACY_SUPPORT <= 4)
	case 4 :
	{
	ZBUFFv04_DCtx* dctx = (ZBUFFv04_DCtx*) legacyContext;
	const void* src = (const char*)input->src + input->pos;
	size_t readSize = input->size - input->pos;
	void* dst = (char*)output->dst + output->pos;
	size_t decodedSize = output->size - output->pos;
	size_t const hintSize = ZBUFFv04_decompressContinue(dctx, dst, &decodedSize, src, &readSize);
	output->pos += decodedSize;
	input->pos += readSize;
	return hintSize;
	}
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 5)
	case 5 :
	{
	ZBUFFv05_DCtx* dctx = (ZBUFFv05_DCtx*) legacyContext;
	const void* src = (const char*)input->src + input->pos;
	size_t readSize = input->size - input->pos;
	void* dst = (char*)output->dst + output->pos;
	size_t decodedSize = output->size - output->pos;
	size_t const hintSize = ZBUFFv05_decompressContinue(dctx, dst, &decodedSize, src, &readSize);
	output->pos += decodedSize;
	input->pos += readSize;
	return hintSize;
	}
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 6)
	case 6 :
	{
	ZBUFFv06_DCtx* dctx = (ZBUFFv06_DCtx*) legacyContext;
	const void* src = (const char*)input->src + input->pos;
	size_t readSize = input->size - input->pos;
	void* dst = (char*)output->dst + output->pos;
	size_t decodedSize = output->size - output->pos;
	size_t const hintSize = ZBUFFv06_decompressContinue(dctx, dst, &decodedSize, src, &readSize);
	output->pos += decodedSize;
	input->pos += readSize;
	return hintSize;
	}
	#endif
	#if (ZSTD_LEGACY_SUPPORT <= 7)
	case 7 :
	{
	ZBUFFv07_DCtx* dctx = (ZBUFFv07_DCtx*) legacyContext;
	const void* src = (const char*)input->src + input->pos;
	size_t readSize = input->size - input->pos;
	void* dst = (char*)output->dst + output->pos;
	size_t decodedSize = output->size - output->pos;
	size_t const hintSize = ZBUFFv07_decompressContinue(dctx, dst, &decodedSize, src, &readSize);
	output->pos += decodedSize;
	input->pos += readSize;
	return hintSize;
	}
	#endif
	}
	}


	#if defined (__cplusplus)
	}
	#endif

	#endif /* ZSTD_LEGACY_H */
	/** ended inlining ../legacy/zstd_legacy.h **/
	#endif



	/-******************************************************
	* Types
	*********************************************************/
	struct ZSTD_DDict_s {
	void* dictBuffer;
	const void* dictContent;
	size_t dictSize;
	ZSTD_entropyDTables_t entropy;
	U32 dictID;
	U32 entropyPresent;
	ZSTD_customMem cMem;
	}; /* typedef'd to ZSTD_DDict within "zstd.h" */

	const void* ZSTD_DDict_dictContent(const ZSTD_DDict* ddict)
	{
	assert(ddict != NULL);
	return ddict->dictContent;
	}

	size_t ZSTD_DDict_dictSize(const ZSTD_DDict* ddict)
	{
	assert(ddict != NULL);
	return ddict->dictSize;
	}

	void ZSTD_copyDDictParameters(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict)
	{
	DEBUGLOG(4, "ZSTD_copyDDictParameters");
	assert(dctx != NULL);
	assert(ddict != NULL);
	dctx->dictID = ddict->dictID;
	dctx->prefixStart = ddict->dictContent;
	dctx->virtualStart = ddict->dictContent;
	dctx->dictEnd = (const BYTE*)ddict->dictContent + ddict->dictSize;
	dctx->previousDstEnd = dctx->dictEnd;
	#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
	dctx->dictContentBeginForFuzzing = dctx->prefixStart;
	dctx->dictContentEndForFuzzing = dctx->previousDstEnd;
	#endif
	if (ddict->entropyPresent) {
	dctx->litEntropy = 1;
	dctx->fseEntropy = 1;
	dctx->LLTptr = ddict->entropy.LLTable;
	dctx->MLTptr = ddict->entropy.MLTable;
	dctx->OFTptr = ddict->entropy.OFTable;
	dctx->HUFptr = ddict->entropy.hufTable;
	dctx->entropy.rep[0] = ddict->entropy.rep[0];
	dctx->entropy.rep[1] = ddict->entropy.rep[1];
	dctx->entropy.rep[2] = ddict->entropy.rep[2];
	} else {
	dctx->litEntropy = 0;
	dctx->fseEntropy = 0;
	}
	}


	static size_t
	ZSTD_loadEntropy_intoDDict(ZSTD_DDict* ddict,
	ZSTD_dictContentType_e dictContentType)
	{
	ddict->dictID = 0;
	ddict->entropyPresent = 0;
	if (dictContentType == ZSTD_dct_rawContent) return 0;

	if (ddict->dictSize < 8) {
	if (dictContentType == ZSTD_dct_fullDict)
	return ERROR(dictionary_corrupted); /* only accept specified dictionaries */
	return 0; /* pure content mode */
	}
	{ U32 const magic = MEM_readLE32(ddict->dictContent);
	if (magic != ZSTD_MAGIC_DICTIONARY) {
	if (dictContentType == ZSTD_dct_fullDict)
	return ERROR(dictionary_corrupted); /* only accept specified dictionaries */
	return 0; /* pure content mode */
	}
	}
	ddict->dictID = MEM_readLE32((const char*)ddict->dictContent + ZSTD_FRAMEIDSIZE);

	/* load entropy tables */
	RETURN_ERROR_IF(ZSTD_isError(ZSTD_loadDEntropy(
	&ddict->entropy, ddict->dictContent, ddict->dictSize)),
	dictionary_corrupted, "");
	ddict->entropyPresent = 1;
	return 0;
	}


	static size_t ZSTD_initDDict_internal(ZSTD_DDict* ddict,
	const void* dict, size_t dictSize,
	ZSTD_dictLoadMethod_e dictLoadMethod,
	ZSTD_dictContentType_e dictContentType)
	{
	if ((dictLoadMethod == ZSTD_dlm_byRef) \|\| (!dict) \|\| (!dictSize)) {
	ddict->dictBuffer = NULL;
	ddict->dictContent = dict;
	if (!dict) dictSize = 0;
	} else {
	void* const internalBuffer = ZSTD_malloc(dictSize, ddict->cMem);
	ddict->dictBuffer = internalBuffer;
	ddict->dictContent = internalBuffer;
	if (!internalBuffer) return ERROR(memory_allocation);
	memcpy(internalBuffer, dict, dictSize);
	}
	ddict->dictSize = dictSize;
	ddict->entropy.hufTable[0] = (HUF_DTable)((HufLog)0x1000001); / cover both little and big endian */

	/* parse dictionary content */
	FORWARD_IF_ERROR( ZSTD_loadEntropy_intoDDict(ddict, dictContentType) , "");

	return 0;
	}

	ZSTD_DDict* ZSTD_createDDict_advanced(const void* dict, size_t dictSize,
	ZSTD_dictLoadMethod_e dictLoadMethod,
	ZSTD_dictContentType_e dictContentType,
	ZSTD_customMem customMem)
	{
	if (!customMem.customAlloc ^ !customMem.customFree) return NULL;

	{ ZSTD_DDict* const ddict = (ZSTD_DDict*) ZSTD_malloc(sizeof(ZSTD_DDict), customMem);
	if (ddict == NULL) return NULL;
	ddict->cMem = customMem;
	{ size_t const initResult = ZSTD_initDDict_internal(ddict,
	dict, dictSize,
	dictLoadMethod, dictContentType);
	if (ZSTD_isError(initResult)) {
	ZSTD_freeDDict(ddict);
	return NULL;
	} }
	return ddict;
	}
	}

	/*! ZSTD_createDDict() :
	* Create a digested dictionary, to start decompression without startup delay.
	* `dict` content is copied inside DDict.
	* Consequently, `dict` can be released after `ZSTD_DDict` creation */
	ZSTD_DDict* ZSTD_createDDict(const void* dict, size_t dictSize)
	{
	ZSTD_customMem const allocator = { NULL, NULL, NULL };
	return ZSTD_createDDict_advanced(dict, dictSize, ZSTD_dlm_byCopy, ZSTD_dct_auto, allocator);
	}

	/*! ZSTD_createDDict_byReference() :
	* Create a digested dictionary, to start decompression without startup delay.
	* Dictionary content is simply referenced, it will be accessed during decompression.
	* Warning : dictBuffer must outlive DDict (DDict must be freed before dictBuffer) */
	ZSTD_DDict* ZSTD_createDDict_byReference(const void* dictBuffer, size_t dictSize)
	{
	ZSTD_customMem const allocator = { NULL, NULL, NULL };
	return ZSTD_createDDict_advanced(dictBuffer, dictSize, ZSTD_dlm_byRef, ZSTD_dct_auto, allocator);
	}


	const ZSTD_DDict* ZSTD_initStaticDDict(
	void* sBuffer, size_t sBufferSize,
	const void* dict, size_t dictSize,
	ZSTD_dictLoadMethod_e dictLoadMethod,
	ZSTD_dictContentType_e dictContentType)
	{
	size_t const neededSpace = sizeof(ZSTD_DDict)
	+ (dictLoadMethod == ZSTD_dlm_byRef ? 0 : dictSize);
	ZSTD_DDict* const ddict = (ZSTD_DDict*)sBuffer;
	assert(sBuffer != NULL);
	assert(dict != NULL);
	if ((size_t)sBuffer & 7) return NULL; /* 8-aligned */
	if (sBufferSize < neededSpace) return NULL;
	if (dictLoadMethod == ZSTD_dlm_byCopy) {
	memcpy(ddict+1, dict, dictSize); /* local copy */
	dict = ddict+1;
	}
	if (ZSTD_isError( ZSTD_initDDict_internal(ddict,
	dict, dictSize,
	ZSTD_dlm_byRef, dictContentType) ))
	return NULL;
	return ddict;
	}


	size_t ZSTD_freeDDict(ZSTD_DDict* ddict)
	{
	if (ddict==NULL) return 0; /* support free on NULL */
	{ ZSTD_customMem const cMem = ddict->cMem;
	ZSTD_free(ddict->dictBuffer, cMem);
	ZSTD_free(ddict, cMem);
	return 0;
	}
	}

	/*! ZSTD_estimateDDictSize() :
	* Estimate amount of memory that will be needed to create a dictionary for decompression.
	* Note : dictionary created by reference using ZSTD_dlm_byRef are smaller */
	size_t ZSTD_estimateDDictSize(size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod)
	{
	return sizeof(ZSTD_DDict) + (dictLoadMethod == ZSTD_dlm_byRef ? 0 : dictSize);
	}

	size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict)
	{
	if (ddict==NULL) return 0; /* support sizeof on NULL */
	return sizeof(*ddict) + (ddict->dictBuffer ? ddict->dictSize : 0) ;
	}

	/*! ZSTD_getDictID_fromDDict() :
	* Provides the dictID of the dictionary loaded into `ddict`.
	* If @return == 0, the dictionary is not conformant to Zstandard specification, or empty.
	* Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
	unsigned ZSTD_getDictID_fromDDict(const ZSTD_DDict* ddict)
	{
	if (ddict==NULL) return 0;
	return ZSTD_getDictID_fromDict(ddict->dictContent, ddict->dictSize);
	}
	/** ended inlining decompress/zstd_ddict.c **/
	/** start inlining decompress/zstd_decompress.c **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/


	/* ***************************************************************
	* Tuning parameters
	*****************************************************************/
	/*!
	* HEAPMODE :
	* Select how default decompression function ZSTD_decompress() allocates its context,
	* on stack (0), or into heap (1, default; requires malloc()).
	* Note that functions with explicit context such as ZSTD_decompressDCtx() are unaffected.
	*/
	#ifndef ZSTD_HEAPMODE
	# define ZSTD_HEAPMODE 1
	#endif

	/*!
	* LEGACY_SUPPORT :
	* if set to 1+, ZSTD_decompress() can decode older formats (v0.1+)
	*/
	#ifndef ZSTD_LEGACY_SUPPORT
	# define ZSTD_LEGACY_SUPPORT 0
	#endif

	/*!
	* MAXWINDOWSIZE_DEFAULT :
	* maximum window size accepted by DStream __by default__.
	* Frames requiring more memory will be rejected.
	* It's possible to set a different limit using ZSTD_DCtx_setMaxWindowSize().
	*/
	#ifndef ZSTD_MAXWINDOWSIZE_DEFAULT
	# define ZSTD_MAXWINDOWSIZE_DEFAULT (((U32)1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT) + 1)
	#endif

	/*!
	* NO_FORWARD_PROGRESS_MAX :
	* maximum allowed nb of calls to ZSTD_decompressStream()
	* without any forward progress
	* (defined as: no byte read from input, and no byte flushed to output)
	* before triggering an error.
	*/
	#ifndef ZSTD_NO_FORWARD_PROGRESS_MAX
	# define ZSTD_NO_FORWARD_PROGRESS_MAX 16
	#endif


	/-******************************************************
	* Dependencies
	*********************************************************/
	#include <string.h> /* memcpy, memmove, memset */
	/** skipping file: ../common/cpu.h **/
	/** skipping file: ../common/mem.h **/
	#define FSE_STATIC_LINKING_ONLY
	/** skipping file: ../common/fse.h **/
	#define HUF_STATIC_LINKING_ONLY
	/** skipping file: ../common/huf.h **/
	/** skipping file: ../common/zstd_internal.h **/
	/** skipping file: zstd_decompress_internal.h **/
	/** skipping file: zstd_ddict.h **/
	/** start inlining zstd_decompress_block.h **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/


	#ifndef ZSTD_DEC_BLOCK_H
	#define ZSTD_DEC_BLOCK_H

	/-******************************************************
	* Dependencies
	*********************************************************/
	#include <stddef.h> /* size_t */
	/** skipping file: ../zstd.h **/
	/** skipping file: ../common/zstd_internal.h **/
	/** skipping file: zstd_decompress_internal.h **/


	/* === Prototypes === */

	/* note: prototypes already published within `zstd.h` :
	* ZSTD_decompressBlock()
	*/

	/* note: prototypes already published within `zstd_internal.h` :
	* ZSTD_getcBlockSize()
	* ZSTD_decodeSeqHeaders()
	*/


	/* ZSTD_decompressBlock_internal() :
	* decompress block, starting at `src`,
	* into destination buffer `dst`.
	* @return : decompressed block size,
	* or an error code (which can be tested using ZSTD_isError())
	*/
	size_t ZSTD_decompressBlock_internal(ZSTD_DCtx* dctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize, const int frame);

	/* ZSTD_buildFSETable() :
	* generate FSE decoding table for one symbol (ll, ml or off)
	* this function must be called with valid parameters only
	* (dt is large enough, normalizedCounter distribution total is a power of 2, max is within range, etc.)
	* in which case it cannot fail.
	* Internal use only.
	*/
	void ZSTD_buildFSETable(ZSTD_seqSymbol* dt,
	const short* normalizedCounter, unsigned maxSymbolValue,
	const U32* baseValue, const U32* nbAdditionalBits,
	unsigned tableLog);


	#endif /* ZSTD_DEC_BLOCK_H */
	/** ended inlining zstd_decompress_block.h **/

	#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1)
	/** skipping file: ../legacy/zstd_legacy.h **/
	#endif


	/-************************************************************
	* Context management
	***************************************************************/
	size_t ZSTD_sizeof_DCtx (const ZSTD_DCtx* dctx)
	{
	if (dctx==NULL) return 0; /* support sizeof NULL */
	return sizeof(*dctx)
	+ ZSTD_sizeof_DDict(dctx->ddictLocal)
	+ dctx->inBuffSize + dctx->outBuffSize;
	}

	size_t ZSTD_estimateDCtxSize(void) { return sizeof(ZSTD_DCtx); }


	static size_t ZSTD_startingInputLength(ZSTD_format_e format)
	{
	size_t const startingInputLength = ZSTD_FRAMEHEADERSIZE_PREFIX(format);
	/* only supports formats ZSTD_f_zstd1 and ZSTD_f_zstd1_magicless */
	assert( (format == ZSTD_f_zstd1) \|\| (format == ZSTD_f_zstd1_magicless) );
	return startingInputLength;
	}

	static void ZSTD_initDCtx_internal(ZSTD_DCtx* dctx)
	{
	dctx->format = ZSTD_f_zstd1; /* ZSTD_decompressBegin() invokes ZSTD_startingInputLength() with argument dctx->format */
	dctx->staticSize = 0;
	dctx->maxWindowSize = ZSTD_MAXWINDOWSIZE_DEFAULT;
	dctx->ddict = NULL;
	dctx->ddictLocal = NULL;
	dctx->dictEnd = NULL;
	dctx->ddictIsCold = 0;
	dctx->dictUses = ZSTD_dont_use;
	dctx->inBuff = NULL;
	dctx->inBuffSize = 0;
	dctx->outBuffSize = 0;
	dctx->streamStage = zdss_init;
	dctx->legacyContext = NULL;
	dctx->previousLegacyVersion = 0;
	dctx->noForwardProgress = 0;
	dctx->oversizedDuration = 0;
	dctx->bmi2 = ZSTD_cpuid_bmi2(ZSTD_cpuid());
	dctx->outBufferMode = ZSTD_obm_buffered;
	#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
	dctx->dictContentEndForFuzzing = NULL;
	#endif
	}

	ZSTD_DCtx* ZSTD_initStaticDCtx(void *workspace, size_t workspaceSize)
	{
	ZSTD_DCtx* const dctx = (ZSTD_DCtx*) workspace;

	if ((size_t)workspace & 7) return NULL; /* 8-aligned */
	if (workspaceSize < sizeof(ZSTD_DCtx)) return NULL; /* minimum size */

	ZSTD_initDCtx_internal(dctx);
	dctx->staticSize = workspaceSize;
	dctx->inBuff = (char*)(dctx+1);
	return dctx;
	}

	ZSTD_DCtx* ZSTD_createDCtx_advanced(ZSTD_customMem customMem)
	{
	if (!customMem.customAlloc ^ !customMem.customFree) return NULL;

	{ ZSTD_DCtx* const dctx = (ZSTD_DCtx)ZSTD_malloc(sizeof(dctx), customMem);
	if (!dctx) return NULL;
	dctx->customMem = customMem;
	ZSTD_initDCtx_internal(dctx);
	return dctx;
	}
	}

	ZSTD_DCtx* ZSTD_createDCtx(void)
	{
	DEBUGLOG(3, "ZSTD_createDCtx");
	return ZSTD_createDCtx_advanced(ZSTD_defaultCMem);
	}

	static void ZSTD_clearDict(ZSTD_DCtx* dctx)
	{
	ZSTD_freeDDict(dctx->ddictLocal);
	dctx->ddictLocal = NULL;
	dctx->ddict = NULL;
	dctx->dictUses = ZSTD_dont_use;
	}

	size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx)
	{
	if (dctx==NULL) return 0; /* support free on NULL */
	RETURN_ERROR_IF(dctx->staticSize, memory_allocation, "not compatible with static DCtx");
	{ ZSTD_customMem const cMem = dctx->customMem;
	ZSTD_clearDict(dctx);
	ZSTD_free(dctx->inBuff, cMem);
	dctx->inBuff = NULL;
	#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT >= 1)
	if (dctx->legacyContext)
	ZSTD_freeLegacyStreamContext(dctx->legacyContext, dctx->previousLegacyVersion);
	#endif
	ZSTD_free(dctx, cMem);
	return 0;
	}
	}

	/* no longer useful */
	void ZSTD_copyDCtx(ZSTD_DCtx* dstDCtx, const ZSTD_DCtx* srcDCtx)
	{
	size_t const toCopy = (size_t)((char)(&dstDCtx->inBuff) - (char)dstDCtx);
	memcpy(dstDCtx, srcDCtx, toCopy); /* no need to copy workspace */
	}


	/-************************************************************
	* Frame header decoding
	***************************************************************/

	/*! ZSTD_isFrame() :
	* Tells if the content of `buffer` starts with a valid Frame Identifier.
	* Note : Frame Identifier is 4 bytes. If `size < 4`, @return will always be 0.
	* Note 2 : Legacy Frame Identifiers are considered valid only if Legacy Support is enabled.
	* Note 3 : Skippable Frame Identifiers are considered valid. */
	unsigned ZSTD_isFrame(const void* buffer, size_t size)
	{
	if (size < ZSTD_FRAMEIDSIZE) return 0;
	{ U32 const magic = MEM_readLE32(buffer);
	if (magic == ZSTD_MAGICNUMBER) return 1;
	if ((magic & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) return 1;
	}
	#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT >= 1)
	if (ZSTD_isLegacy(buffer, size)) return 1;
	#endif
	return 0;
	}

	/** ZSTD_frameHeaderSize_internal() :
	* srcSize must be large enough to reach header size fields.
	* note : only works for formats ZSTD_f_zstd1 and ZSTD_f_zstd1_magicless.
	* @return : size of the Frame Header
	* or an error code, which can be tested with ZSTD_isError() */
	static size_t ZSTD_frameHeaderSize_internal(const void* src, size_t srcSize, ZSTD_format_e format)
	{
	size_t const minInputSize = ZSTD_startingInputLength(format);
	RETURN_ERROR_IF(srcSize < minInputSize, srcSize_wrong, "");

	{ BYTE const fhd = ((const BYTE*)src)[minInputSize-1];
	U32 const dictID= fhd & 3;
	U32 const singleSegment = (fhd >> 5) & 1;
	U32 const fcsId = fhd >> 6;
	return minInputSize + !singleSegment
	+ ZSTD_did_fieldSize[dictID] + ZSTD_fcs_fieldSize[fcsId]
	+ (singleSegment && !fcsId);
	}
	}

	/** ZSTD_frameHeaderSize() :
	* srcSize must be >= ZSTD_frameHeaderSize_prefix.
	* @return : size of the Frame Header,
	* or an error code (if srcSize is too small) */
	size_t ZSTD_frameHeaderSize(const void* src, size_t srcSize)
	{
	return ZSTD_frameHeaderSize_internal(src, srcSize, ZSTD_f_zstd1);
	}


	/** ZSTD_getFrameHeader_advanced() :
	* decode Frame Header, or require larger `srcSize`.
	* note : only works for formats ZSTD_f_zstd1 and ZSTD_f_zstd1_magicless
	* @return : 0, `zfhPtr` is correctly filled,
	* >0, `srcSize` is too small, value is wanted `srcSize` amount,
	* or an error code, which can be tested using ZSTD_isError() */
	size_t ZSTD_getFrameHeader_advanced(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize, ZSTD_format_e format)
	{
	const BYTE* ip = (const BYTE*)src;
	size_t const minInputSize = ZSTD_startingInputLength(format);

	memset(zfhPtr, 0, sizeof(zfhPtr)); / not strictly necessary, but static analyzer do not understand that zfhPtr is only going to be read only if return value is zero, since they are 2 different signals */
	if (srcSize < minInputSize) return minInputSize;
	RETURN_ERROR_IF(src==NULL, GENERIC, "invalid parameter");

	if ( (format != ZSTD_f_zstd1_magicless)
	&& (MEM_readLE32(src) != ZSTD_MAGICNUMBER) ) {
	if ((MEM_readLE32(src) & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) {
	/* skippable frame */
	if (srcSize < ZSTD_SKIPPABLEHEADERSIZE)
	return ZSTD_SKIPPABLEHEADERSIZE; /* magic number + frame length */
	memset(zfhPtr, 0, sizeof(*zfhPtr));
	zfhPtr->frameContentSize = MEM_readLE32((const char *)src + ZSTD_FRAMEIDSIZE);
	zfhPtr->frameType = ZSTD_skippableFrame;
	return 0;
	}
	RETURN_ERROR(prefix_unknown, "");
	}

	/* ensure there is enough `srcSize` to fully read/decode frame header */
	{ size_t const fhsize = ZSTD_frameHeaderSize_internal(src, srcSize, format);
	if (srcSize < fhsize) return fhsize;
	zfhPtr->headerSize = (U32)fhsize;
	}

	{ BYTE const fhdByte = ip[minInputSize-1];
	size_t pos = minInputSize;
	U32 const dictIDSizeCode = fhdByte&3;
	U32 const checksumFlag = (fhdByte>>2)&1;
	U32 const singleSegment = (fhdByte>>5)&1;
	U32 const fcsID = fhdByte>>6;
	U64 windowSize = 0;
	U32 dictID = 0;
	U64 frameContentSize = ZSTD_CONTENTSIZE_UNKNOWN;
	RETURN_ERROR_IF((fhdByte & 0x08) != 0, frameParameter_unsupported,
	"reserved bits, must be zero");

	if (!singleSegment) {
	BYTE const wlByte = ip[pos++];
	U32 const windowLog = (wlByte >> 3) + ZSTD_WINDOWLOG_ABSOLUTEMIN;
	RETURN_ERROR_IF(windowLog > ZSTD_WINDOWLOG_MAX, frameParameter_windowTooLarge, "");
	windowSize = (1ULL << windowLog);
	windowSize += (windowSize >> 3) * (wlByte&7);
	}
	switch(dictIDSizeCode)
	{
	default: assert(0); /* impossible */
	case 0 : break;
	case 1 : dictID = ip[pos]; pos++; break;
	case 2 : dictID = MEM_readLE16(ip+pos); pos+=2; break;
	case 3 : dictID = MEM_readLE32(ip+pos); pos+=4; break;
	}
	switch(fcsID)
	{
	default: assert(0); /* impossible */
	case 0 : if (singleSegment) frameContentSize = ip[pos]; break;
	case 1 : frameContentSize = MEM_readLE16(ip+pos)+256; break;
	case 2 : frameContentSize = MEM_readLE32(ip+pos); break;
	case 3 : frameContentSize = MEM_readLE64(ip+pos); break;
	}
	if (singleSegment) windowSize = frameContentSize;

	zfhPtr->frameType = ZSTD_frame;
	zfhPtr->frameContentSize = frameContentSize;
	zfhPtr->windowSize = windowSize;
	zfhPtr->blockSizeMax = (unsigned) MIN(windowSize, ZSTD_BLOCKSIZE_MAX);
	zfhPtr->dictID = dictID;
	zfhPtr->checksumFlag = checksumFlag;
	}
	return 0;
	}

	/** ZSTD_getFrameHeader() :
	* decode Frame Header, or require larger `srcSize`.
	* note : this function does not consume input, it only reads it.
	* @return : 0, `zfhPtr` is correctly filled,
	* >0, `srcSize` is too small, value is wanted `srcSize` amount,
	* or an error code, which can be tested using ZSTD_isError() */
	size_t ZSTD_getFrameHeader(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize)
	{
	return ZSTD_getFrameHeader_advanced(zfhPtr, src, srcSize, ZSTD_f_zstd1);
	}


	/** ZSTD_getFrameContentSize() :
	* compatible with legacy mode
	* @return : decompressed size of the single frame pointed to be `src` if known, otherwise
	* - ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined
	* - ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too small) */
	unsigned long long ZSTD_getFrameContentSize(const void *src, size_t srcSize)
	{
	#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT >= 1)
	if (ZSTD_isLegacy(src, srcSize)) {
	unsigned long long const ret = ZSTD_getDecompressedSize_legacy(src, srcSize);
	return ret == 0 ? ZSTD_CONTENTSIZE_UNKNOWN : ret;
	}
	#endif
	{ ZSTD_frameHeader zfh;
	if (ZSTD_getFrameHeader(&zfh, src, srcSize) != 0)
	return ZSTD_CONTENTSIZE_ERROR;
	if (zfh.frameType == ZSTD_skippableFrame) {
	return 0;
	} else {
	return zfh.frameContentSize;
	} }
	}

	static size_t readSkippableFrameSize(void const* src, size_t srcSize)
	{
	size_t const skippableHeaderSize = ZSTD_SKIPPABLEHEADERSIZE;
	U32 sizeU32;

	RETURN_ERROR_IF(srcSize < ZSTD_SKIPPABLEHEADERSIZE, srcSize_wrong, "");

	sizeU32 = MEM_readLE32((BYTE const*)src + ZSTD_FRAMEIDSIZE);
	RETURN_ERROR_IF((U32)(sizeU32 + ZSTD_SKIPPABLEHEADERSIZE) < sizeU32,
	frameParameter_unsupported, "");
	{
	size_t const skippableSize = skippableHeaderSize + sizeU32;
	RETURN_ERROR_IF(skippableSize > srcSize, srcSize_wrong, "");
	return skippableSize;
	}
	}

	/** ZSTD_findDecompressedSize() :
	* compatible with legacy mode
	* `srcSize` must be the exact length of some number of ZSTD compressed and/or
	* skippable frames
	* @return : decompressed size of the frames contained */
	unsigned long long ZSTD_findDecompressedSize(const void* src, size_t srcSize)
	{
	unsigned long long totalDstSize = 0;

	while (srcSize >= ZSTD_startingInputLength(ZSTD_f_zstd1)) {
	U32 const magicNumber = MEM_readLE32(src);

	if ((magicNumber & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) {
	size_t const skippableSize = readSkippableFrameSize(src, srcSize);
	if (ZSTD_isError(skippableSize)) {
	return ZSTD_CONTENTSIZE_ERROR;
	}
	assert(skippableSize <= srcSize);

	src = (const BYTE *)src + skippableSize;
	srcSize -= skippableSize;
	continue;
	}

	{ unsigned long long const ret = ZSTD_getFrameContentSize(src, srcSize);
	if (ret >= ZSTD_CONTENTSIZE_ERROR) return ret;

	/* check for overflow */
	if (totalDstSize + ret < totalDstSize) return ZSTD_CONTENTSIZE_ERROR;
	totalDstSize += ret;
	}
	{ size_t const frameSrcSize = ZSTD_findFrameCompressedSize(src, srcSize);
	if (ZSTD_isError(frameSrcSize)) {
	return ZSTD_CONTENTSIZE_ERROR;
	}

	src = (const BYTE *)src + frameSrcSize;
	srcSize -= frameSrcSize;
	}
	} /* while (srcSize >= ZSTD_frameHeaderSize_prefix) */

	if (srcSize) return ZSTD_CONTENTSIZE_ERROR;

	return totalDstSize;
	}

	/** ZSTD_getDecompressedSize() :
	* compatible with legacy mode
	* @return : decompressed size if known, 0 otherwise
	note : 0 can mean any of the following :
	- frame content is empty
	- decompressed size field is not present in frame header
	- frame header unknown / not supported
	- frame header not complete (`srcSize` too small) */
	unsigned long long ZSTD_getDecompressedSize(const void* src, size_t srcSize)
	{
	unsigned long long const ret = ZSTD_getFrameContentSize(src, srcSize);
	ZSTD_STATIC_ASSERT(ZSTD_CONTENTSIZE_ERROR < ZSTD_CONTENTSIZE_UNKNOWN);
	return (ret >= ZSTD_CONTENTSIZE_ERROR) ? 0 : ret;
	}


	/** ZSTD_decodeFrameHeader() :
	* `headerSize` must be the size provided by ZSTD_frameHeaderSize().
	* @return : 0 if success, or an error code, which can be tested using ZSTD_isError() */
	static size_t ZSTD_decodeFrameHeader(ZSTD_DCtx* dctx, const void* src, size_t headerSize)
	{
	size_t const result = ZSTD_getFrameHeader_advanced(&(dctx->fParams), src, headerSize, dctx->format);
	if (ZSTD_isError(result)) return result; /* invalid header */
	RETURN_ERROR_IF(result>0, srcSize_wrong, "headerSize too small");
	#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
	/* Skip the dictID check in fuzzing mode, because it makes the search
	* harder.
	*/
	RETURN_ERROR_IF(dctx->fParams.dictID && (dctx->dictID != dctx->fParams.dictID),
	dictionary_wrong, "");
	#endif
	if (dctx->fParams.checksumFlag) XXH64_reset(&dctx->xxhState, 0);
	return 0;
	}

	static ZSTD_frameSizeInfo ZSTD_errorFrameSizeInfo(size_t ret)
	{
	ZSTD_frameSizeInfo frameSizeInfo;
	frameSizeInfo.compressedSize = ret;
	frameSizeInfo.decompressedBound = ZSTD_CONTENTSIZE_ERROR;
	return frameSizeInfo;
	}

	static ZSTD_frameSizeInfo ZSTD_findFrameSizeInfo(const void* src, size_t srcSize)
	{
	ZSTD_frameSizeInfo frameSizeInfo;
	memset(&frameSizeInfo, 0, sizeof(ZSTD_frameSizeInfo));

	#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT >= 1)
	if (ZSTD_isLegacy(src, srcSize))
	return ZSTD_findFrameSizeInfoLegacy(src, srcSize);
	#endif

	if ((srcSize >= ZSTD_SKIPPABLEHEADERSIZE)
	&& (MEM_readLE32(src) & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) {
	frameSizeInfo.compressedSize = readSkippableFrameSize(src, srcSize);
	assert(ZSTD_isError(frameSizeInfo.compressedSize) \|\|
	frameSizeInfo.compressedSize <= srcSize);
	return frameSizeInfo;
	} else {
	const BYTE* ip = (const BYTE*)src;
	const BYTE* const ipstart = ip;
	size_t remainingSize = srcSize;
	size_t nbBlocks = 0;
	ZSTD_frameHeader zfh;

	/* Extract Frame Header */
	{ size_t const ret = ZSTD_getFrameHeader(&zfh, src, srcSize);
	if (ZSTD_isError(ret))
	return ZSTD_errorFrameSizeInfo(ret);
	if (ret > 0)
	return ZSTD_errorFrameSizeInfo(ERROR(srcSize_wrong));
	}

	ip += zfh.headerSize;
	remainingSize -= zfh.headerSize;

	/* Iterate over each block */
	while (1) {
	blockProperties_t blockProperties;
	size_t const cBlockSize = ZSTD_getcBlockSize(ip, remainingSize, &blockProperties);
	if (ZSTD_isError(cBlockSize))
	return ZSTD_errorFrameSizeInfo(cBlockSize);

	if (ZSTD_blockHeaderSize + cBlockSize > remainingSize)
	return ZSTD_errorFrameSizeInfo(ERROR(srcSize_wrong));

	ip += ZSTD_blockHeaderSize + cBlockSize;
	remainingSize -= ZSTD_blockHeaderSize + cBlockSize;
	nbBlocks++;

	if (blockProperties.lastBlock) break;
	}

	/* Final frame content checksum */
	if (zfh.checksumFlag) {
	if (remainingSize < 4)
	return ZSTD_errorFrameSizeInfo(ERROR(srcSize_wrong));
	ip += 4;
	}

	frameSizeInfo.compressedSize = ip - ipstart;
	frameSizeInfo.decompressedBound = (zfh.frameContentSize != ZSTD_CONTENTSIZE_UNKNOWN)
	? zfh.frameContentSize
	: nbBlocks * zfh.blockSizeMax;
	return frameSizeInfo;
	}
	}

	/** ZSTD_findFrameCompressedSize() :
	* compatible with legacy mode
	* `src` must point to the start of a ZSTD frame, ZSTD legacy frame, or skippable frame
	* `srcSize` must be at least as large as the frame contained
	* @return : the compressed size of the frame starting at `src` */
	size_t ZSTD_findFrameCompressedSize(const void *src, size_t srcSize)
	{
	ZSTD_frameSizeInfo const frameSizeInfo = ZSTD_findFrameSizeInfo(src, srcSize);
	return frameSizeInfo.compressedSize;
	}

	/** ZSTD_decompressBound() :
	* compatible with legacy mode
	* `src` must point to the start of a ZSTD frame or a skippeable frame
	* `srcSize` must be at least as large as the frame contained
	* @return : the maximum decompressed size of the compressed source
	*/
	unsigned long long ZSTD_decompressBound(const void* src, size_t srcSize)
	{
	unsigned long long bound = 0;
	/* Iterate over each frame */
	while (srcSize > 0) {
	ZSTD_frameSizeInfo const frameSizeInfo = ZSTD_findFrameSizeInfo(src, srcSize);
	size_t const compressedSize = frameSizeInfo.compressedSize;
	unsigned long long const decompressedBound = frameSizeInfo.decompressedBound;
	if (ZSTD_isError(compressedSize) \|\| decompressedBound == ZSTD_CONTENTSIZE_ERROR)
	return ZSTD_CONTENTSIZE_ERROR;
	assert(srcSize >= compressedSize);
	src = (const BYTE*)src + compressedSize;
	srcSize -= compressedSize;
	bound += decompressedBound;
	}
	return bound;
	}


	/-************************************************************
	* Frame decoding
	***************************************************************/

	/** ZSTD_insertBlock() :
	* insert `src` block into `dctx` history. Useful to track uncompressed blocks. */
	size_t ZSTD_insertBlock(ZSTD_DCtx* dctx, const void* blockStart, size_t blockSize)
	{
	DEBUGLOG(5, "ZSTD_insertBlock: %u bytes", (unsigned)blockSize);
	ZSTD_checkContinuity(dctx, blockStart);
	dctx->previousDstEnd = (const char*)blockStart + blockSize;
	return blockSize;
	}


	static size_t ZSTD_copyRawBlock(void* dst, size_t dstCapacity,
	const void* src, size_t srcSize)
	{
	DEBUGLOG(5, "ZSTD_copyRawBlock");
	if (dst == NULL) {
	if (srcSize == 0) return 0;
	RETURN_ERROR(dstBuffer_null, "");
	}
	RETURN_ERROR_IF(srcSize > dstCapacity, dstSize_tooSmall, "");
	memcpy(dst, src, srcSize);
	return srcSize;
	}

	static size_t ZSTD_setRleBlock(void* dst, size_t dstCapacity,
	BYTE b,
	size_t regenSize)
	{
	if (dst == NULL) {
	if (regenSize == 0) return 0;
	RETURN_ERROR(dstBuffer_null, "");
	}
	RETURN_ERROR_IF(regenSize > dstCapacity, dstSize_tooSmall, "");
	memset(dst, b, regenSize);
	return regenSize;
	}


	/*! ZSTD_decompressFrame() :
	* @dctx must be properly initialized
	* will update srcPtr and srcSizePtr,
	* to make srcPtr progress by one frame. /
	static size_t ZSTD_decompressFrame(ZSTD_DCtx* dctx,
	void* dst, size_t dstCapacity,
	const void** srcPtr, size_t *srcSizePtr)
	{
	const BYTE* ip = (const BYTE)(srcPtr);
	BYTE* const ostart = (BYTE* const)dst;
	BYTE* const oend = dstCapacity != 0 ? ostart + dstCapacity : ostart;
	BYTE* op = ostart;
	size_t remainingSrcSize = *srcSizePtr;

	DEBUGLOG(4, "ZSTD_decompressFrame (srcSize:%i)", (int)*srcSizePtr);

	/* check */
	RETURN_ERROR_IF(
	remainingSrcSize < ZSTD_FRAMEHEADERSIZE_MIN(dctx->format)+ZSTD_blockHeaderSize,
	srcSize_wrong, "");

	/* Frame Header */
	{ size_t const frameHeaderSize = ZSTD_frameHeaderSize_internal(
	ip, ZSTD_FRAMEHEADERSIZE_PREFIX(dctx->format), dctx->format);
	if (ZSTD_isError(frameHeaderSize)) return frameHeaderSize;
	RETURN_ERROR_IF(remainingSrcSize < frameHeaderSize+ZSTD_blockHeaderSize,
	srcSize_wrong, "");
	FORWARD_IF_ERROR( ZSTD_decodeFrameHeader(dctx, ip, frameHeaderSize) , "");
	ip += frameHeaderSize; remainingSrcSize -= frameHeaderSize;
	}

	/* Loop on each block */
	while (1) {
	size_t decodedSize;
	blockProperties_t blockProperties;
	size_t const cBlockSize = ZSTD_getcBlockSize(ip, remainingSrcSize, &blockProperties);
	if (ZSTD_isError(cBlockSize)) return cBlockSize;

	ip += ZSTD_blockHeaderSize;
	remainingSrcSize -= ZSTD_blockHeaderSize;
	RETURN_ERROR_IF(cBlockSize > remainingSrcSize, srcSize_wrong, "");

	switch(blockProperties.blockType)
	{
	case bt_compressed:
	decodedSize = ZSTD_decompressBlock_internal(dctx, op, oend-op, ip, cBlockSize, /* frame */ 1);
	break;
	case bt_raw :
	decodedSize = ZSTD_copyRawBlock(op, oend-op, ip, cBlockSize);
	break;
	case bt_rle :
	decodedSize = ZSTD_setRleBlock(op, oend-op, *ip, blockProperties.origSize);
	break;
	case bt_reserved :
	default:
	RETURN_ERROR(corruption_detected, "invalid block type");
	}

	if (ZSTD_isError(decodedSize)) return decodedSize;
	if (dctx->fParams.checksumFlag)
	XXH64_update(&dctx->xxhState, op, decodedSize);
	if (decodedSize != 0)
	op += decodedSize;
	assert(ip != NULL);
	ip += cBlockSize;
	remainingSrcSize -= cBlockSize;
	if (blockProperties.lastBlock) break;
	}

	if (dctx->fParams.frameContentSize != ZSTD_CONTENTSIZE_UNKNOWN) {
	RETURN_ERROR_IF((U64)(op-ostart) != dctx->fParams.frameContentSize,
	corruption_detected, "");
	}
	if (dctx->fParams.checksumFlag) { /* Frame content checksum verification */
	U32 const checkCalc = (U32)XXH64_digest(&dctx->xxhState);
	U32 checkRead;
	RETURN_ERROR_IF(remainingSrcSize<4, checksum_wrong, "");
	checkRead = MEM_readLE32(ip);
	RETURN_ERROR_IF(checkRead != checkCalc, checksum_wrong, "");
	ip += 4;
	remainingSrcSize -= 4;
	}

	/* Allow caller to get size read */
	*srcPtr = ip;
	*srcSizePtr = remainingSrcSize;
	return op-ostart;
	}

	static size_t ZSTD_decompressMultiFrame(ZSTD_DCtx* dctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const void* dict, size_t dictSize,
	const ZSTD_DDict* ddict)
	{
	void* const dststart = dst;
	int moreThan1Frame = 0;

	DEBUGLOG(5, "ZSTD_decompressMultiFrame");
	assert(dict==NULL \|\| ddict==NULL); /* either dict or ddict set, not both */

	if (ddict) {
	dict = ZSTD_DDict_dictContent(ddict);
	dictSize = ZSTD_DDict_dictSize(ddict);
	}

	while (srcSize >= ZSTD_startingInputLength(dctx->format)) {

	#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT >= 1)
	if (ZSTD_isLegacy(src, srcSize)) {
	size_t decodedSize;
	size_t const frameSize = ZSTD_findFrameCompressedSizeLegacy(src, srcSize);
	if (ZSTD_isError(frameSize)) return frameSize;
	RETURN_ERROR_IF(dctx->staticSize, memory_allocation,
	"legacy support is not compatible with static dctx");

	decodedSize = ZSTD_decompressLegacy(dst, dstCapacity, src, frameSize, dict, dictSize);
	if (ZSTD_isError(decodedSize)) return decodedSize;

	assert(decodedSize <=- dstCapacity);
	dst = (BYTE*)dst + decodedSize;
	dstCapacity -= decodedSize;

	src = (const BYTE*)src + frameSize;
	srcSize -= frameSize;

	continue;
	}
	#endif

	{ U32 const magicNumber = MEM_readLE32(src);
	DEBUGLOG(4, "reading magic number %08X (expecting %08X)",
	(unsigned)magicNumber, ZSTD_MAGICNUMBER);
	if ((magicNumber & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) {
	size_t const skippableSize = readSkippableFrameSize(src, srcSize);
	FORWARD_IF_ERROR(skippableSize, "readSkippableFrameSize failed");
	assert(skippableSize <= srcSize);

	src = (const BYTE *)src + skippableSize;
	srcSize -= skippableSize;
	continue;
	} }

	if (ddict) {
	/* we were called from ZSTD_decompress_usingDDict */
	FORWARD_IF_ERROR(ZSTD_decompressBegin_usingDDict(dctx, ddict), "");
	} else {
	/* this will initialize correctly with no dict if dict == NULL, so
	* use this in all cases but ddict */
	FORWARD_IF_ERROR(ZSTD_decompressBegin_usingDict(dctx, dict, dictSize), "");
	}
	ZSTD_checkContinuity(dctx, dst);

	{ const size_t res = ZSTD_decompressFrame(dctx, dst, dstCapacity,
	&src, &srcSize);
	RETURN_ERROR_IF(
	(ZSTD_getErrorCode(res) == ZSTD_error_prefix_unknown)
	&& (moreThan1Frame==1),
	srcSize_wrong,
	"at least one frame successfully completed, but following "
	"bytes are garbage: it's more likely to be a srcSize error, "
	"specifying more bytes than compressed size of frame(s). This "
	"error message replaces ERROR(prefix_unknown), which would be "
	"confusing, as the first header is actually correct. Note that "
	"one could be unlucky, it might be a corruption error instead, "
	"happening right at the place where we expect zstd magic "
	"bytes. But this is _much_ less likely than a srcSize field "
	"error.");
	if (ZSTD_isError(res)) return res;
	assert(res <= dstCapacity);
	if (res != 0)
	dst = (BYTE*)dst + res;
	dstCapacity -= res;
	}
	moreThan1Frame = 1;
	} /* while (srcSize >= ZSTD_frameHeaderSize_prefix) */

	RETURN_ERROR_IF(srcSize, srcSize_wrong, "input not entirely consumed");

	return (BYTE)dst - (BYTE)dststart;
	}

	size_t ZSTD_decompress_usingDict(ZSTD_DCtx* dctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const void* dict, size_t dictSize)
	{
	return ZSTD_decompressMultiFrame(dctx, dst, dstCapacity, src, srcSize, dict, dictSize, NULL);
	}


	static ZSTD_DDict const* ZSTD_getDDict(ZSTD_DCtx* dctx)
	{
	switch (dctx->dictUses) {
	default:
	assert(0 /* Impossible */);
	/* fall-through */
	case ZSTD_dont_use:
	ZSTD_clearDict(dctx);
	return NULL;
	case ZSTD_use_indefinitely:
	return dctx->ddict;
	case ZSTD_use_once:
	dctx->dictUses = ZSTD_dont_use;
	return dctx->ddict;
	}
	}

	size_t ZSTD_decompressDCtx(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize)
	{
	return ZSTD_decompress_usingDDict(dctx, dst, dstCapacity, src, srcSize, ZSTD_getDDict(dctx));
	}


	size_t ZSTD_decompress(void* dst, size_t dstCapacity, const void* src, size_t srcSize)
	{
	#if defined(ZSTD_HEAPMODE) && (ZSTD_HEAPMODE>=1)
	size_t regenSize;
	ZSTD_DCtx* const dctx = ZSTD_createDCtx();
	RETURN_ERROR_IF(dctx==NULL, memory_allocation, "NULL pointer!");
	regenSize = ZSTD_decompressDCtx(dctx, dst, dstCapacity, src, srcSize);
	ZSTD_freeDCtx(dctx);
	return regenSize;
	#else /* stack mode */
	ZSTD_DCtx dctx;
	ZSTD_initDCtx_internal(&dctx);
	return ZSTD_decompressDCtx(&dctx, dst, dstCapacity, src, srcSize);
	#endif
	}


	/-*************************************
	* Advanced Streaming Decompression API
	* Bufferless and synchronous
	****************************************/
	size_t ZSTD_nextSrcSizeToDecompress(ZSTD_DCtx* dctx) { return dctx->expected; }

	/**
	* Similar to ZSTD_nextSrcSizeToDecompress(), but when when a block input can be streamed,
	* we allow taking a partial block as the input. Currently only raw uncompressed blocks can
	* be streamed.
	*
	* For blocks that can be streamed, this allows us to reduce the latency until we produce
	* output, and avoid copying the input.
	*
	* @param inputSize - The total amount of input that the caller currently has.
	*/
	static size_t ZSTD_nextSrcSizeToDecompressWithInputSize(ZSTD_DCtx* dctx, size_t inputSize) {
	if (!(dctx->stage == ZSTDds_decompressBlock \|\| dctx->stage == ZSTDds_decompressLastBlock))
	return dctx->expected;
	if (dctx->bType != bt_raw)
	return dctx->expected;
	return MIN(MAX(inputSize, 1), dctx->expected);
	}

	ZSTD_nextInputType_e ZSTD_nextInputType(ZSTD_DCtx* dctx) {
	switch(dctx->stage)
	{
	default: /* should not happen */
	assert(0);
	case ZSTDds_getFrameHeaderSize:
	case ZSTDds_decodeFrameHeader:
	return ZSTDnit_frameHeader;
	case ZSTDds_decodeBlockHeader:
	return ZSTDnit_blockHeader;
	case ZSTDds_decompressBlock:
	return ZSTDnit_block;
	case ZSTDds_decompressLastBlock:
	return ZSTDnit_lastBlock;
	case ZSTDds_checkChecksum:
	return ZSTDnit_checksum;
	case ZSTDds_decodeSkippableHeader:
	case ZSTDds_skipFrame:
	return ZSTDnit_skippableFrame;
	}
	}

	static int ZSTD_isSkipFrame(ZSTD_DCtx* dctx) { return dctx->stage == ZSTDds_skipFrame; }

	/** ZSTD_decompressContinue() :
	* srcSize : must be the exact nb of bytes expected (see ZSTD_nextSrcSizeToDecompress())
	* @return : nb of bytes generated into `dst` (necessarily <= `dstCapacity)
	* or an error code, which can be tested using ZSTD_isError() */
	size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize)
	{
	DEBUGLOG(5, "ZSTD_decompressContinue (srcSize:%u)", (unsigned)srcSize);
	/* Sanity check */
	RETURN_ERROR_IF(srcSize != ZSTD_nextSrcSizeToDecompressWithInputSize(dctx, srcSize), srcSize_wrong, "not allowed");
	if (dstCapacity) ZSTD_checkContinuity(dctx, dst);

	switch (dctx->stage)
	{
	case ZSTDds_getFrameHeaderSize :
	assert(src != NULL);
	if (dctx->format == ZSTD_f_zstd1) { /* allows header */
	assert(srcSize >= ZSTD_FRAMEIDSIZE); /* to read skippable magic number */
	if ((MEM_readLE32(src) & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) { /* skippable frame */
	memcpy(dctx->headerBuffer, src, srcSize);
	dctx->expected = ZSTD_SKIPPABLEHEADERSIZE - srcSize; /* remaining to load to get full skippable frame header */
	dctx->stage = ZSTDds_decodeSkippableHeader;
	return 0;
	} }
	dctx->headerSize = ZSTD_frameHeaderSize_internal(src, srcSize, dctx->format);
	if (ZSTD_isError(dctx->headerSize)) return dctx->headerSize;
	memcpy(dctx->headerBuffer, src, srcSize);
	dctx->expected = dctx->headerSize - srcSize;
	dctx->stage = ZSTDds_decodeFrameHeader;
	return 0;

	case ZSTDds_decodeFrameHeader:
	assert(src != NULL);
	memcpy(dctx->headerBuffer + (dctx->headerSize - srcSize), src, srcSize);
	FORWARD_IF_ERROR(ZSTD_decodeFrameHeader(dctx, dctx->headerBuffer, dctx->headerSize), "");
	dctx->expected = ZSTD_blockHeaderSize;
	dctx->stage = ZSTDds_decodeBlockHeader;
	return 0;

	case ZSTDds_decodeBlockHeader:
	{ blockProperties_t bp;
	size_t const cBlockSize = ZSTD_getcBlockSize(src, ZSTD_blockHeaderSize, &bp);
	if (ZSTD_isError(cBlockSize)) return cBlockSize;
	RETURN_ERROR_IF(cBlockSize > dctx->fParams.blockSizeMax, corruption_detected, "Block Size Exceeds Maximum");
	dctx->expected = cBlockSize;
	dctx->bType = bp.blockType;
	dctx->rleSize = bp.origSize;
	if (cBlockSize) {
	dctx->stage = bp.lastBlock ? ZSTDds_decompressLastBlock : ZSTDds_decompressBlock;
	return 0;
	}
	/* empty block */
	if (bp.lastBlock) {
	if (dctx->fParams.checksumFlag) {
	dctx->expected = 4;
	dctx->stage = ZSTDds_checkChecksum;
	} else {
	dctx->expected = 0; /* end of frame */
	dctx->stage = ZSTDds_getFrameHeaderSize;
	}
	} else {
	dctx->expected = ZSTD_blockHeaderSize; /* jump to next header */
	dctx->stage = ZSTDds_decodeBlockHeader;
	}
	return 0;
	}

	case ZSTDds_decompressLastBlock:
	case ZSTDds_decompressBlock:
	DEBUGLOG(5, "ZSTD_decompressContinue: case ZSTDds_decompressBlock");
	{ size_t rSize;
	switch(dctx->bType)
	{
	case bt_compressed:
	DEBUGLOG(5, "ZSTD_decompressContinue: case bt_compressed");
	rSize = ZSTD_decompressBlock_internal(dctx, dst, dstCapacity, src, srcSize, /* frame */ 1);
	dctx->expected = 0; /* Streaming not supported */
	break;
	case bt_raw :
	assert(srcSize <= dctx->expected);
	rSize = ZSTD_copyRawBlock(dst, dstCapacity, src, srcSize);
	FORWARD_IF_ERROR(rSize, "ZSTD_copyRawBlock failed");
	assert(rSize == srcSize);
	dctx->expected -= rSize;
	break;
	case bt_rle :
	rSize = ZSTD_setRleBlock(dst, dstCapacity, (const BYTE)src, dctx->rleSize);
	dctx->expected = 0; /* Streaming not supported */
	break;
	case bt_reserved : /* should never happen */
	default:
	RETURN_ERROR(corruption_detected, "invalid block type");
	}
	FORWARD_IF_ERROR(rSize, "");
	RETURN_ERROR_IF(rSize > dctx->fParams.blockSizeMax, corruption_detected, "Decompressed Block Size Exceeds Maximum");
	DEBUGLOG(5, "ZSTD_decompressContinue: decoded size from block : %u", (unsigned)rSize);
	dctx->decodedSize += rSize;
	if (dctx->fParams.checksumFlag) XXH64_update(&dctx->xxhState, dst, rSize);
	dctx->previousDstEnd = (char*)dst + rSize;

	/* Stay on the same stage until we are finished streaming the block. */
	if (dctx->expected > 0) {
	return rSize;
	}

	if (dctx->stage == ZSTDds_decompressLastBlock) { /* end of frame */
	DEBUGLOG(4, "ZSTD_decompressContinue: decoded size from frame : %u", (unsigned)dctx->decodedSize);
	RETURN_ERROR_IF(
	dctx->fParams.frameContentSize != ZSTD_CONTENTSIZE_UNKNOWN
	&& dctx->decodedSize != dctx->fParams.frameContentSize,
	corruption_detected, "");
	if (dctx->fParams.checksumFlag) { /* another round for frame checksum */
	dctx->expected = 4;
	dctx->stage = ZSTDds_checkChecksum;
	} else {
	dctx->expected = 0; /* ends here */
	dctx->stage = ZSTDds_getFrameHeaderSize;
	}
	} else {
	dctx->stage = ZSTDds_decodeBlockHeader;
	dctx->expected = ZSTD_blockHeaderSize;
	}
	return rSize;
	}

	case ZSTDds_checkChecksum:
	assert(srcSize == 4); /* guaranteed by dctx->expected */
	{ U32 const h32 = (U32)XXH64_digest(&dctx->xxhState);
	U32 const check32 = MEM_readLE32(src);
	DEBUGLOG(4, "ZSTD_decompressContinue: checksum : calculated %08X :: %08X read", (unsigned)h32, (unsigned)check32);
	RETURN_ERROR_IF(check32 != h32, checksum_wrong, "");
	dctx->expected = 0;
	dctx->stage = ZSTDds_getFrameHeaderSize;
	return 0;
	}

	case ZSTDds_decodeSkippableHeader:
	assert(src != NULL);
	assert(srcSize <= ZSTD_SKIPPABLEHEADERSIZE);
	memcpy(dctx->headerBuffer + (ZSTD_SKIPPABLEHEADERSIZE - srcSize), src, srcSize); /* complete skippable header */
	dctx->expected = MEM_readLE32(dctx->headerBuffer + ZSTD_FRAMEIDSIZE); /* note : dctx->expected can grow seriously large, beyond local buffer size */
	dctx->stage = ZSTDds_skipFrame;
	return 0;

	case ZSTDds_skipFrame:
	dctx->expected = 0;
	dctx->stage = ZSTDds_getFrameHeaderSize;
	return 0;

	default:
	assert(0); /* impossible */
	RETURN_ERROR(GENERIC, "impossible to reach"); /* some compiler require default to do something */
	}
	}


	static size_t ZSTD_refDictContent(ZSTD_DCtx* dctx, const void* dict, size_t dictSize)
	{
	dctx->dictEnd = dctx->previousDstEnd;
	dctx->virtualStart = (const char)dict - ((const char)(dctx->previousDstEnd) - (const char*)(dctx->prefixStart));
	dctx->prefixStart = dict;
	dctx->previousDstEnd = (const char*)dict + dictSize;
	#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
	dctx->dictContentBeginForFuzzing = dctx->prefixStart;
	dctx->dictContentEndForFuzzing = dctx->previousDstEnd;
	#endif
	return 0;
	}

	/*! ZSTD_loadDEntropy() :
	* dict : must point at beginning of a valid zstd dictionary.
	* @return : size of entropy tables read */
	size_t
	ZSTD_loadDEntropy(ZSTD_entropyDTables_t* entropy,
	const void* const dict, size_t const dictSize)
	{
	const BYTE* dictPtr = (const BYTE*)dict;
	const BYTE* const dictEnd = dictPtr + dictSize;

	RETURN_ERROR_IF(dictSize <= 8, dictionary_corrupted, "dict is too small");
	assert(MEM_readLE32(dict) == ZSTD_MAGIC_DICTIONARY); /* dict must be valid */
	dictPtr += 8; /* skip header = magic + dictID */

	ZSTD_STATIC_ASSERT(offsetof(ZSTD_entropyDTables_t, OFTable) == offsetof(ZSTD_entropyDTables_t, LLTable) + sizeof(entropy->LLTable));
	ZSTD_STATIC_ASSERT(offsetof(ZSTD_entropyDTables_t, MLTable) == offsetof(ZSTD_entropyDTables_t, OFTable) + sizeof(entropy->OFTable));
	ZSTD_STATIC_ASSERT(sizeof(entropy->LLTable) + sizeof(entropy->OFTable) + sizeof(entropy->MLTable) >= HUF_DECOMPRESS_WORKSPACE_SIZE);
	{ void* const workspace = &entropy->LLTable; /* use fse tables as temporary workspace; implies fse tables are grouped together */
	size_t const workspaceSize = sizeof(entropy->LLTable) + sizeof(entropy->OFTable) + sizeof(entropy->MLTable);
	#ifdef HUF_FORCE_DECOMPRESS_X1
	/* in minimal huffman, we always use X1 variants */
	size_t const hSize = HUF_readDTableX1_wksp(entropy->hufTable,
	dictPtr, dictEnd - dictPtr,
	workspace, workspaceSize);
	#else
	size_t const hSize = HUF_readDTableX2_wksp(entropy->hufTable,
	dictPtr, dictEnd - dictPtr,
	workspace, workspaceSize);
	#endif
	RETURN_ERROR_IF(HUF_isError(hSize), dictionary_corrupted, "");
	dictPtr += hSize;
	}

	{ short offcodeNCount[MaxOff+1];
	unsigned offcodeMaxValue = MaxOff, offcodeLog;
	size_t const offcodeHeaderSize = FSE_readNCount(offcodeNCount, &offcodeMaxValue, &offcodeLog, dictPtr, dictEnd-dictPtr);
	RETURN_ERROR_IF(FSE_isError(offcodeHeaderSize), dictionary_corrupted, "");
	RETURN_ERROR_IF(offcodeMaxValue > MaxOff, dictionary_corrupted, "");
	RETURN_ERROR_IF(offcodeLog > OffFSELog, dictionary_corrupted, "");
	ZSTD_buildFSETable( entropy->OFTable,
	offcodeNCount, offcodeMaxValue,
	OF_base, OF_bits,
	offcodeLog);
	dictPtr += offcodeHeaderSize;
	}

	{ short matchlengthNCount[MaxML+1];
	unsigned matchlengthMaxValue = MaxML, matchlengthLog;
	size_t const matchlengthHeaderSize = FSE_readNCount(matchlengthNCount, &matchlengthMaxValue, &matchlengthLog, dictPtr, dictEnd-dictPtr);
	RETURN_ERROR_IF(FSE_isError(matchlengthHeaderSize), dictionary_corrupted, "");
	RETURN_ERROR_IF(matchlengthMaxValue > MaxML, dictionary_corrupted, "");
	RETURN_ERROR_IF(matchlengthLog > MLFSELog, dictionary_corrupted, "");
	ZSTD_buildFSETable( entropy->MLTable,
	matchlengthNCount, matchlengthMaxValue,
	ML_base, ML_bits,
	matchlengthLog);
	dictPtr += matchlengthHeaderSize;
	}

	{ short litlengthNCount[MaxLL+1];
	unsigned litlengthMaxValue = MaxLL, litlengthLog;
	size_t const litlengthHeaderSize = FSE_readNCount(litlengthNCount, &litlengthMaxValue, &litlengthLog, dictPtr, dictEnd-dictPtr);
	RETURN_ERROR_IF(FSE_isError(litlengthHeaderSize), dictionary_corrupted, "");
	RETURN_ERROR_IF(litlengthMaxValue > MaxLL, dictionary_corrupted, "");
	RETURN_ERROR_IF(litlengthLog > LLFSELog, dictionary_corrupted, "");
	ZSTD_buildFSETable( entropy->LLTable,
	litlengthNCount, litlengthMaxValue,
	LL_base, LL_bits,
	litlengthLog);
	dictPtr += litlengthHeaderSize;
	}

	RETURN_ERROR_IF(dictPtr+12 > dictEnd, dictionary_corrupted, "");
	{ int i;
	size_t const dictContentSize = (size_t)(dictEnd - (dictPtr+12));
	for (i=0; i<3; i++) {
	U32 const rep = MEM_readLE32(dictPtr); dictPtr += 4;
	RETURN_ERROR_IF(rep==0 \|\| rep > dictContentSize,
	dictionary_corrupted, "");
	entropy->rep[i] = rep;
	} }

	return dictPtr - (const BYTE*)dict;
	}

	static size_t ZSTD_decompress_insertDictionary(ZSTD_DCtx* dctx, const void* dict, size_t dictSize)
	{
	if (dictSize < 8) return ZSTD_refDictContent(dctx, dict, dictSize);
	{ U32 const magic = MEM_readLE32(dict);
	if (magic != ZSTD_MAGIC_DICTIONARY) {
	return ZSTD_refDictContent(dctx, dict, dictSize); /* pure content mode */
	} }
	dctx->dictID = MEM_readLE32((const char*)dict + ZSTD_FRAMEIDSIZE);

	/* load entropy tables */
	{ size_t const eSize = ZSTD_loadDEntropy(&dctx->entropy, dict, dictSize);
	RETURN_ERROR_IF(ZSTD_isError(eSize), dictionary_corrupted, "");
	dict = (const char*)dict + eSize;
	dictSize -= eSize;
	}
	dctx->litEntropy = dctx->fseEntropy = 1;

	/* reference dictionary content */
	return ZSTD_refDictContent(dctx, dict, dictSize);
	}

	size_t ZSTD_decompressBegin(ZSTD_DCtx* dctx)
	{
	assert(dctx != NULL);
	dctx->expected = ZSTD_startingInputLength(dctx->format); /* dctx->format must be properly set */
	dctx->stage = ZSTDds_getFrameHeaderSize;
	dctx->decodedSize = 0;
	dctx->previousDstEnd = NULL;
	dctx->prefixStart = NULL;
	dctx->virtualStart = NULL;
	dctx->dictEnd = NULL;
	dctx->entropy.hufTable[0] = (HUF_DTable)((HufLog)0x1000001); / cover both little and big endian */
	dctx->litEntropy = dctx->fseEntropy = 0;
	dctx->dictID = 0;
	dctx->bType = bt_reserved;
	ZSTD_STATIC_ASSERT(sizeof(dctx->entropy.rep) == sizeof(repStartValue));
	memcpy(dctx->entropy.rep, repStartValue, sizeof(repStartValue)); /* initial repcodes */
	dctx->LLTptr = dctx->entropy.LLTable;
	dctx->MLTptr = dctx->entropy.MLTable;
	dctx->OFTptr = dctx->entropy.OFTable;
	dctx->HUFptr = dctx->entropy.hufTable;
	return 0;
	}

	size_t ZSTD_decompressBegin_usingDict(ZSTD_DCtx* dctx, const void* dict, size_t dictSize)
	{
	FORWARD_IF_ERROR( ZSTD_decompressBegin(dctx) , "");
	if (dict && dictSize)
	RETURN_ERROR_IF(
	ZSTD_isError(ZSTD_decompress_insertDictionary(dctx, dict, dictSize)),
	dictionary_corrupted, "");
	return 0;
	}


	/* ====== ZSTD_DDict ====== */

	size_t ZSTD_decompressBegin_usingDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict)
	{
	DEBUGLOG(4, "ZSTD_decompressBegin_usingDDict");
	assert(dctx != NULL);
	if (ddict) {
	const char* const dictStart = (const char*)ZSTD_DDict_dictContent(ddict);
	size_t const dictSize = ZSTD_DDict_dictSize(ddict);
	const void* const dictEnd = dictStart + dictSize;
	dctx->ddictIsCold = (dctx->dictEnd != dictEnd);
	DEBUGLOG(4, "DDict is %s",
	dctx->ddictIsCold ? "~cold~" : "hot!");
	}
	FORWARD_IF_ERROR( ZSTD_decompressBegin(dctx) , "");
	if (ddict) { /* NULL ddict is equivalent to no dictionary */
	ZSTD_copyDDictParameters(dctx, ddict);
	}
	return 0;
	}

	/*! ZSTD_getDictID_fromDict() :
	* Provides the dictID stored within dictionary.
	* if @return == 0, the dictionary is not conformant with Zstandard specification.
	* It can still be loaded, but as a content-only dictionary. */
	unsigned ZSTD_getDictID_fromDict(const void* dict, size_t dictSize)
	{
	if (dictSize < 8) return 0;
	if (MEM_readLE32(dict) != ZSTD_MAGIC_DICTIONARY) return 0;
	return MEM_readLE32((const char*)dict + ZSTD_FRAMEIDSIZE);
	}

	/*! ZSTD_getDictID_fromFrame() :
	* Provides the dictID required to decompress frame stored within `src`.
	* If @return == 0, the dictID could not be decoded.
	* This could for one of the following reasons :
	* - The frame does not require a dictionary (most common case).
	* - The frame was built with dictID intentionally removed.
	* Needed dictionary is a hidden information.
	* Note : this use case also happens when using a non-conformant dictionary.
	* - `srcSize` is too small, and as a result, frame header could not be decoded.
	* Note : possible if `srcSize < ZSTD_FRAMEHEADERSIZE_MAX`.
	* - This is not a Zstandard frame.
	* When identifying the exact failure cause, it's possible to use
	* ZSTD_getFrameHeader(), which will provide a more precise error code. */
	unsigned ZSTD_getDictID_fromFrame(const void* src, size_t srcSize)
	{
	ZSTD_frameHeader zfp = { 0, 0, 0, ZSTD_frame, 0, 0, 0 };
	size_t const hError = ZSTD_getFrameHeader(&zfp, src, srcSize);
	if (ZSTD_isError(hError)) return 0;
	return zfp.dictID;
	}


	/*! ZSTD_decompress_usingDDict() :
	* Decompression using a pre-digested Dictionary
	* Use dictionary without significant overhead. */
	size_t ZSTD_decompress_usingDDict(ZSTD_DCtx* dctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize,
	const ZSTD_DDict* ddict)
	{
	/* pass content and size in case legacy frames are encountered */
	return ZSTD_decompressMultiFrame(dctx, dst, dstCapacity, src, srcSize,
	NULL, 0,
	ddict);
	}


	/*=====================================
	* Streaming decompression
	====================================/

	ZSTD_DStream* ZSTD_createDStream(void)
	{
	DEBUGLOG(3, "ZSTD_createDStream");
	return ZSTD_createDStream_advanced(ZSTD_defaultCMem);
	}

	ZSTD_DStream* ZSTD_initStaticDStream(void *workspace, size_t workspaceSize)
	{
	return ZSTD_initStaticDCtx(workspace, workspaceSize);
	}

	ZSTD_DStream* ZSTD_createDStream_advanced(ZSTD_customMem customMem)
	{
	return ZSTD_createDCtx_advanced(customMem);
	}

	size_t ZSTD_freeDStream(ZSTD_DStream* zds)
	{
	return ZSTD_freeDCtx(zds);
	}


	/* * Initialization * */

	size_t ZSTD_DStreamInSize(void) { return ZSTD_BLOCKSIZE_MAX + ZSTD_blockHeaderSize; }
	size_t ZSTD_DStreamOutSize(void) { return ZSTD_BLOCKSIZE_MAX; }

	size_t ZSTD_DCtx_loadDictionary_advanced(ZSTD_DCtx* dctx,
	const void* dict, size_t dictSize,
	ZSTD_dictLoadMethod_e dictLoadMethod,
	ZSTD_dictContentType_e dictContentType)
	{
	RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, "");
	ZSTD_clearDict(dctx);
	if (dict && dictSize != 0) {
	dctx->ddictLocal = ZSTD_createDDict_advanced(dict, dictSize, dictLoadMethod, dictContentType, dctx->customMem);
	RETURN_ERROR_IF(dctx->ddictLocal == NULL, memory_allocation, "NULL pointer!");
	dctx->ddict = dctx->ddictLocal;
	dctx->dictUses = ZSTD_use_indefinitely;
	}
	return 0;
	}

	size_t ZSTD_DCtx_loadDictionary_byReference(ZSTD_DCtx* dctx, const void* dict, size_t dictSize)
	{
	return ZSTD_DCtx_loadDictionary_advanced(dctx, dict, dictSize, ZSTD_dlm_byRef, ZSTD_dct_auto);
	}

	size_t ZSTD_DCtx_loadDictionary(ZSTD_DCtx* dctx, const void* dict, size_t dictSize)
	{
	return ZSTD_DCtx_loadDictionary_advanced(dctx, dict, dictSize, ZSTD_dlm_byCopy, ZSTD_dct_auto);
	}

	size_t ZSTD_DCtx_refPrefix_advanced(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType)
	{
	FORWARD_IF_ERROR(ZSTD_DCtx_loadDictionary_advanced(dctx, prefix, prefixSize, ZSTD_dlm_byRef, dictContentType), "");
	dctx->dictUses = ZSTD_use_once;
	return 0;
	}

	size_t ZSTD_DCtx_refPrefix(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize)
	{
	return ZSTD_DCtx_refPrefix_advanced(dctx, prefix, prefixSize, ZSTD_dct_rawContent);
	}


	/* ZSTD_initDStream_usingDict() :
	* return : expected size, aka ZSTD_startingInputLength().
	* this function cannot fail */
	size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize)
	{
	DEBUGLOG(4, "ZSTD_initDStream_usingDict");
	FORWARD_IF_ERROR( ZSTD_DCtx_reset(zds, ZSTD_reset_session_only) , "");
	FORWARD_IF_ERROR( ZSTD_DCtx_loadDictionary(zds, dict, dictSize) , "");
	return ZSTD_startingInputLength(zds->format);
	}

	/* note : this variant can't fail */
	size_t ZSTD_initDStream(ZSTD_DStream* zds)
	{
	DEBUGLOG(4, "ZSTD_initDStream");
	return ZSTD_initDStream_usingDDict(zds, NULL);
	}

	/* ZSTD_initDStream_usingDDict() :
	* ddict will just be referenced, and must outlive decompression session
	* this function cannot fail */
	size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* dctx, const ZSTD_DDict* ddict)
	{
	FORWARD_IF_ERROR( ZSTD_DCtx_reset(dctx, ZSTD_reset_session_only) , "");
	FORWARD_IF_ERROR( ZSTD_DCtx_refDDict(dctx, ddict) , "");
	return ZSTD_startingInputLength(dctx->format);
	}

	/* ZSTD_resetDStream() :
	* return : expected size, aka ZSTD_startingInputLength().
	* this function cannot fail */
	size_t ZSTD_resetDStream(ZSTD_DStream* dctx)
	{
	FORWARD_IF_ERROR(ZSTD_DCtx_reset(dctx, ZSTD_reset_session_only), "");
	return ZSTD_startingInputLength(dctx->format);
	}


	size_t ZSTD_DCtx_refDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict)
	{
	RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, "");
	ZSTD_clearDict(dctx);
	if (ddict) {
	dctx->ddict = ddict;
	dctx->dictUses = ZSTD_use_indefinitely;
	}
	return 0;
	}

	/* ZSTD_DCtx_setMaxWindowSize() :
	* note : no direct equivalence in ZSTD_DCtx_setParameter,
	* since this version sets windowSize, and the other sets windowLog */
	size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize)
	{
	ZSTD_bounds const bounds = ZSTD_dParam_getBounds(ZSTD_d_windowLogMax);
	size_t const min = (size_t)1 << bounds.lowerBound;
	size_t const max = (size_t)1 << bounds.upperBound;
	RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, "");
	RETURN_ERROR_IF(maxWindowSize < min, parameter_outOfBound, "");
	RETURN_ERROR_IF(maxWindowSize > max, parameter_outOfBound, "");
	dctx->maxWindowSize = maxWindowSize;
	return 0;
	}

	size_t ZSTD_DCtx_setFormat(ZSTD_DCtx* dctx, ZSTD_format_e format)
	{
	return ZSTD_DCtx_setParameter(dctx, ZSTD_d_format, format);
	}

	ZSTD_bounds ZSTD_dParam_getBounds(ZSTD_dParameter dParam)
	{
	ZSTD_bounds bounds = { 0, 0, 0 };
	switch(dParam) {
	case ZSTD_d_windowLogMax:
	bounds.lowerBound = ZSTD_WINDOWLOG_ABSOLUTEMIN;
	bounds.upperBound = ZSTD_WINDOWLOG_MAX;
	return bounds;
	case ZSTD_d_format:
	bounds.lowerBound = (int)ZSTD_f_zstd1;
	bounds.upperBound = (int)ZSTD_f_zstd1_magicless;
	ZSTD_STATIC_ASSERT(ZSTD_f_zstd1 < ZSTD_f_zstd1_magicless);
	return bounds;
	case ZSTD_d_stableOutBuffer:
	bounds.lowerBound = (int)ZSTD_obm_buffered;
	bounds.upperBound = (int)ZSTD_obm_stable;
	return bounds;
	default:;
	}
	bounds.error = ERROR(parameter_unsupported);
	return bounds;
	}

	/* ZSTD_dParam_withinBounds:
	* @return 1 if value is within dParam bounds,
	* 0 otherwise */
	static int ZSTD_dParam_withinBounds(ZSTD_dParameter dParam, int value)
	{
	ZSTD_bounds const bounds = ZSTD_dParam_getBounds(dParam);
	if (ZSTD_isError(bounds.error)) return 0;
	if (value < bounds.lowerBound) return 0;
	if (value > bounds.upperBound) return 0;
	return 1;
	}

	#define CHECK_DBOUNDS(p,v) { \
	RETURN_ERROR_IF(!ZSTD_dParam_withinBounds(p, v), parameter_outOfBound, ""); \
	}

	size_t ZSTD_DCtx_setParameter(ZSTD_DCtx* dctx, ZSTD_dParameter dParam, int value)
	{
	RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, "");
	switch(dParam) {
	case ZSTD_d_windowLogMax:
	if (value == 0) value = ZSTD_WINDOWLOG_LIMIT_DEFAULT;
	CHECK_DBOUNDS(ZSTD_d_windowLogMax, value);
	dctx->maxWindowSize = ((size_t)1) << value;
	return 0;
	case ZSTD_d_format:
	CHECK_DBOUNDS(ZSTD_d_format, value);
	dctx->format = (ZSTD_format_e)value;
	return 0;
	case ZSTD_d_stableOutBuffer:
	CHECK_DBOUNDS(ZSTD_d_stableOutBuffer, value);
	dctx->outBufferMode = (ZSTD_outBufferMode_e)value;
	return 0;
	default:;
	}
	RETURN_ERROR(parameter_unsupported, "");
	}

	size_t ZSTD_DCtx_reset(ZSTD_DCtx* dctx, ZSTD_ResetDirective reset)
	{
	if ( (reset == ZSTD_reset_session_only)
	\|\| (reset == ZSTD_reset_session_and_parameters) ) {
	dctx->streamStage = zdss_init;
	dctx->noForwardProgress = 0;
	}
	if ( (reset == ZSTD_reset_parameters)
	\|\| (reset == ZSTD_reset_session_and_parameters) ) {
	RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, "");
	ZSTD_clearDict(dctx);
	dctx->format = ZSTD_f_zstd1;
	dctx->maxWindowSize = ZSTD_MAXWINDOWSIZE_DEFAULT;
	}
	return 0;
	}


	size_t ZSTD_sizeof_DStream(const ZSTD_DStream* dctx)
	{
	return ZSTD_sizeof_DCtx(dctx);
	}

	size_t ZSTD_decodingBufferSize_min(unsigned long long windowSize, unsigned long long frameContentSize)
	{
	size_t const blockSize = (size_t) MIN(windowSize, ZSTD_BLOCKSIZE_MAX);
	unsigned long long const neededRBSize = windowSize + blockSize + (WILDCOPY_OVERLENGTH * 2);
	unsigned long long const neededSize = MIN(frameContentSize, neededRBSize);
	size_t const minRBSize = (size_t) neededSize;
	RETURN_ERROR_IF((unsigned long long)minRBSize != neededSize,
	frameParameter_windowTooLarge, "");
	return minRBSize;
	}

	size_t ZSTD_estimateDStreamSize(size_t windowSize)
	{
	size_t const blockSize = MIN(windowSize, ZSTD_BLOCKSIZE_MAX);
	size_t const inBuffSize = blockSize; /* no block can be larger */
	size_t const outBuffSize = ZSTD_decodingBufferSize_min(windowSize, ZSTD_CONTENTSIZE_UNKNOWN);
	return ZSTD_estimateDCtxSize() + inBuffSize + outBuffSize;
	}

	size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize)
	{
	U32 const windowSizeMax = 1U << ZSTD_WINDOWLOG_MAX; /* note : should be user-selectable, but requires an additional parameter (or a dctx) */
	ZSTD_frameHeader zfh;
	size_t const err = ZSTD_getFrameHeader(&zfh, src, srcSize);
	if (ZSTD_isError(err)) return err;
	RETURN_ERROR_IF(err>0, srcSize_wrong, "");
	RETURN_ERROR_IF(zfh.windowSize > windowSizeMax,
	frameParameter_windowTooLarge, "");
	return ZSTD_estimateDStreamSize((size_t)zfh.windowSize);
	}


	/* *** Decompression *** */

	static int ZSTD_DCtx_isOverflow(ZSTD_DStream* zds, size_t const neededInBuffSize, size_t const neededOutBuffSize)
	{
	return (zds->inBuffSize + zds->outBuffSize) >= (neededInBuffSize + neededOutBuffSize) * ZSTD_WORKSPACETOOLARGE_FACTOR;
	}

	static void ZSTD_DCtx_updateOversizedDuration(ZSTD_DStream* zds, size_t const neededInBuffSize, size_t const neededOutBuffSize)
	{
	if (ZSTD_DCtx_isOverflow(zds, neededInBuffSize, neededOutBuffSize))
	zds->oversizedDuration++;
	else
	zds->oversizedDuration = 0;
	}

	static int ZSTD_DCtx_isOversizedTooLong(ZSTD_DStream* zds)
	{
	return zds->oversizedDuration >= ZSTD_WORKSPACETOOLARGE_MAXDURATION;
	}

	/* Checks that the output buffer hasn't changed if ZSTD_obm_stable is used. */
	static size_t ZSTD_checkOutBuffer(ZSTD_DStream const* zds, ZSTD_outBuffer const* output)
	{
	ZSTD_outBuffer const expect = zds->expectedOutBuffer;
	/* No requirement when ZSTD_obm_stable is not enabled. */
	if (zds->outBufferMode != ZSTD_obm_stable)
	return 0;
	/* Any buffer is allowed in zdss_init, this must be the same for every other call until
	* the context is reset.
	*/
	if (zds->streamStage == zdss_init)
	return 0;
	/* The buffer must match our expectation exactly. */
	if (expect.dst == output->dst && expect.pos == output->pos && expect.size == output->size)
	return 0;
	RETURN_ERROR(dstBuffer_wrong, "ZSTD_obm_stable enabled but output differs!");
	}

	/* Calls ZSTD_decompressContinue() with the right parameters for ZSTD_decompressStream()
	* and updates the stage and the output buffer state. This call is extracted so it can be
	* used both when reading directly from the ZSTD_inBuffer, and in buffered input mode.
	* NOTE: You must break after calling this function since the streamStage is modified.
	*/
	static size_t ZSTD_decompressContinueStream(
	ZSTD_DStream* zds, char** op, char* oend,
	void const* src, size_t srcSize) {
	int const isSkipFrame = ZSTD_isSkipFrame(zds);
	if (zds->outBufferMode == ZSTD_obm_buffered) {
	size_t const dstSize = isSkipFrame ? 0 : zds->outBuffSize - zds->outStart;
	size_t const decodedSize = ZSTD_decompressContinue(zds,
	zds->outBuff + zds->outStart, dstSize, src, srcSize);
	FORWARD_IF_ERROR(decodedSize, "");
	if (!decodedSize && !isSkipFrame) {
	zds->streamStage = zdss_read;
	} else {
	zds->outEnd = zds->outStart + decodedSize;
	zds->streamStage = zdss_flush;
	}
	} else {
	/* Write directly into the output buffer */
	size_t const dstSize = isSkipFrame ? 0 : oend - *op;
	size_t const decodedSize = ZSTD_decompressContinue(zds, *op, dstSize, src, srcSize);
	FORWARD_IF_ERROR(decodedSize, "");
	*op += decodedSize;
	/* Flushing is not needed. */
	zds->streamStage = zdss_read;
	assert(*op <= oend);
	assert(zds->outBufferMode == ZSTD_obm_stable);
	}
	return 0;
	}

	size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inBuffer* input)
	{
	const char* const src = (const char*)input->src;
	const char* const istart = input->pos != 0 ? src + input->pos : src;
	const char* const iend = input->size != 0 ? src + input->size : src;
	const char* ip = istart;
	char* const dst = (char*)output->dst;
	char* const ostart = output->pos != 0 ? dst + output->pos : dst;
	char* const oend = output->size != 0 ? dst + output->size : dst;
	char* op = ostart;
	U32 someMoreWork = 1;

	DEBUGLOG(5, "ZSTD_decompressStream");
	RETURN_ERROR_IF(
	input->pos > input->size,
	srcSize_wrong,
	"forbidden. in: pos: %u vs size: %u",
	(U32)input->pos, (U32)input->size);
	RETURN_ERROR_IF(
	output->pos > output->size,
	dstSize_tooSmall,
	"forbidden. out: pos: %u vs size: %u",
	(U32)output->pos, (U32)output->size);
	DEBUGLOG(5, "input size : %u", (U32)(input->size - input->pos));
	FORWARD_IF_ERROR(ZSTD_checkOutBuffer(zds, output), "");

	while (someMoreWork) {
	switch(zds->streamStage)
	{
	case zdss_init :
	DEBUGLOG(5, "stage zdss_init => transparent reset ");
	zds->streamStage = zdss_loadHeader;
	zds->lhSize = zds->inPos = zds->outStart = zds->outEnd = 0;
	zds->legacyVersion = 0;
	zds->hostageByte = 0;
	zds->expectedOutBuffer = *output;
	/* fall-through */

	case zdss_loadHeader :
	DEBUGLOG(5, "stage zdss_loadHeader (srcSize : %u)", (U32)(iend - ip));
	#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1)
	if (zds->legacyVersion) {
	RETURN_ERROR_IF(zds->staticSize, memory_allocation,
	"legacy support is incompatible with static dctx");
	{ size_t const hint = ZSTD_decompressLegacyStream(zds->legacyContext, zds->legacyVersion, output, input);
	if (hint==0) zds->streamStage = zdss_init;
	return hint;
	} }
	#endif
	{ size_t const hSize = ZSTD_getFrameHeader_advanced(&zds->fParams, zds->headerBuffer, zds->lhSize, zds->format);
	DEBUGLOG(5, "header size : %u", (U32)hSize);
	if (ZSTD_isError(hSize)) {
	#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1)
	U32 const legacyVersion = ZSTD_isLegacy(istart, iend-istart);
	if (legacyVersion) {
	ZSTD_DDict const* const ddict = ZSTD_getDDict(zds);
	const void* const dict = ddict ? ZSTD_DDict_dictContent(ddict) : NULL;
	size_t const dictSize = ddict ? ZSTD_DDict_dictSize(ddict) : 0;
	DEBUGLOG(5, "ZSTD_decompressStream: detected legacy version v0.%u", legacyVersion);
	RETURN_ERROR_IF(zds->staticSize, memory_allocation,
	"legacy support is incompatible with static dctx");
	FORWARD_IF_ERROR(ZSTD_initLegacyStream(&zds->legacyContext,
	zds->previousLegacyVersion, legacyVersion,
	dict, dictSize), "");
	zds->legacyVersion = zds->previousLegacyVersion = legacyVersion;
	{ size_t const hint = ZSTD_decompressLegacyStream(zds->legacyContext, legacyVersion, output, input);
	if (hint==0) zds->streamStage = zdss_init; /* or stay in stage zdss_loadHeader */
	return hint;
	} }
	#endif
	return hSize; /* error */
	}
	if (hSize != 0) { /* need more input */
	size_t const toLoad = hSize - zds->lhSize; /* if hSize!=0, hSize > zds->lhSize */
	size_t const remainingInput = (size_t)(iend-ip);
	assert(iend >= ip);
	if (toLoad > remainingInput) { /* not enough input to load full header */
	if (remainingInput > 0) {
	memcpy(zds->headerBuffer + zds->lhSize, ip, remainingInput);
	zds->lhSize += remainingInput;
	}
	input->pos = input->size;
	return (MAX((size_t)ZSTD_FRAMEHEADERSIZE_MIN(zds->format), hSize) - zds->lhSize) + ZSTD_blockHeaderSize; /* remaining header bytes + next block header */
	}
	assert(ip != NULL);
	memcpy(zds->headerBuffer + zds->lhSize, ip, toLoad); zds->lhSize = hSize; ip += toLoad;
	break;
	} }

	/* check for single-pass mode opportunity */
	if (zds->fParams.frameContentSize != ZSTD_CONTENTSIZE_UNKNOWN
	&& zds->fParams.frameType != ZSTD_skippableFrame
	&& (U64)(size_t)(oend-op) >= zds->fParams.frameContentSize) {
	size_t const cSize = ZSTD_findFrameCompressedSize(istart, iend-istart);
	if (cSize <= (size_t)(iend-istart)) {
	/* shortcut : using single-pass mode */
	size_t const decompressedSize = ZSTD_decompress_usingDDict(zds, op, oend-op, istart, cSize, ZSTD_getDDict(zds));
	if (ZSTD_isError(decompressedSize)) return decompressedSize;
	DEBUGLOG(4, "shortcut to single-pass ZSTD_decompress_usingDDict()")
	ip = istart + cSize;
	op += decompressedSize;
	zds->expected = 0;
	zds->streamStage = zdss_init;
	someMoreWork = 0;
	break;
	} }

	/* Check output buffer is large enough for ZSTD_odm_stable. */
	if (zds->outBufferMode == ZSTD_obm_stable
	&& zds->fParams.frameType != ZSTD_skippableFrame
	&& zds->fParams.frameContentSize != ZSTD_CONTENTSIZE_UNKNOWN
	&& (U64)(size_t)(oend-op) < zds->fParams.frameContentSize) {
	RETURN_ERROR(dstSize_tooSmall, "ZSTD_obm_stable passed but ZSTD_outBuffer is too small");
	}

	/* Consume header (see ZSTDds_decodeFrameHeader) */
	DEBUGLOG(4, "Consume header");
	FORWARD_IF_ERROR(ZSTD_decompressBegin_usingDDict(zds, ZSTD_getDDict(zds)), "");

	if ((MEM_readLE32(zds->headerBuffer) & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) { /* skippable frame */
	zds->expected = MEM_readLE32(zds->headerBuffer + ZSTD_FRAMEIDSIZE);
	zds->stage = ZSTDds_skipFrame;
	} else {
	FORWARD_IF_ERROR(ZSTD_decodeFrameHeader(zds, zds->headerBuffer, zds->lhSize), "");
	zds->expected = ZSTD_blockHeaderSize;
	zds->stage = ZSTDds_decodeBlockHeader;
	}

	/* control buffer memory usage */
	DEBUGLOG(4, "Control max memory usage (%u KB <= max %u KB)",
	(U32)(zds->fParams.windowSize >>10),
	(U32)(zds->maxWindowSize >> 10) );
	zds->fParams.windowSize = MAX(zds->fParams.windowSize, 1U << ZSTD_WINDOWLOG_ABSOLUTEMIN);
	RETURN_ERROR_IF(zds->fParams.windowSize > zds->maxWindowSize,
	frameParameter_windowTooLarge, "");

	/* Adapt buffer sizes to frame header instructions */
	{ size_t const neededInBuffSize = MAX(zds->fParams.blockSizeMax, 4 /* frame checksum */);
	size_t const neededOutBuffSize = zds->outBufferMode == ZSTD_obm_buffered
	? ZSTD_decodingBufferSize_min(zds->fParams.windowSize, zds->fParams.frameContentSize)
	: 0;

	ZSTD_DCtx_updateOversizedDuration(zds, neededInBuffSize, neededOutBuffSize);

	{ int const tooSmall = (zds->inBuffSize < neededInBuffSize) \|\| (zds->outBuffSize < neededOutBuffSize);
	int const tooLarge = ZSTD_DCtx_isOversizedTooLong(zds);

	if (tooSmall \|\| tooLarge) {
	size_t const bufferSize = neededInBuffSize + neededOutBuffSize;
	DEBUGLOG(4, "inBuff : from %u to %u",
	(U32)zds->inBuffSize, (U32)neededInBuffSize);
	DEBUGLOG(4, "outBuff : from %u to %u",
	(U32)zds->outBuffSize, (U32)neededOutBuffSize);
	if (zds->staticSize) { /* static DCtx */
	DEBUGLOG(4, "staticSize : %u", (U32)zds->staticSize);
	assert(zds->staticSize >= sizeof(ZSTD_DCtx)); /* controlled at init */
	RETURN_ERROR_IF(
	bufferSize > zds->staticSize - sizeof(ZSTD_DCtx),
	memory_allocation, "");
	} else {
	ZSTD_free(zds->inBuff, zds->customMem);
	zds->inBuffSize = 0;
	zds->outBuffSize = 0;
	zds->inBuff = (char*)ZSTD_malloc(bufferSize, zds->customMem);
	RETURN_ERROR_IF(zds->inBuff == NULL, memory_allocation, "");
	}
	zds->inBuffSize = neededInBuffSize;
	zds->outBuff = zds->inBuff + zds->inBuffSize;
	zds->outBuffSize = neededOutBuffSize;
	} } }
	zds->streamStage = zdss_read;
	/* fall-through */

	case zdss_read:
	DEBUGLOG(5, "stage zdss_read");
	{ size_t const neededInSize = ZSTD_nextSrcSizeToDecompressWithInputSize(zds, iend - ip);
	DEBUGLOG(5, "neededInSize = %u", (U32)neededInSize);
	if (neededInSize==0) { /* end of frame */
	zds->streamStage = zdss_init;
	someMoreWork = 0;
	break;
	}
	if ((size_t)(iend-ip) >= neededInSize) { /* decode directly from src */
	FORWARD_IF_ERROR(ZSTD_decompressContinueStream(zds, &op, oend, ip, neededInSize), "");
	ip += neededInSize;
	/* Function modifies the stage so we must break */
	break;
	} }
	if (ip==iend) { someMoreWork = 0; break; } /* no more input */
	zds->streamStage = zdss_load;
	/* fall-through */

	case zdss_load:
	{ size_t const neededInSize = ZSTD_nextSrcSizeToDecompress(zds);
	size_t const toLoad = neededInSize - zds->inPos;
	int const isSkipFrame = ZSTD_isSkipFrame(zds);
	size_t loadedSize;
	/* At this point we shouldn't be decompressing a block that we can stream. */
	assert(neededInSize == ZSTD_nextSrcSizeToDecompressWithInputSize(zds, iend - ip));
	if (isSkipFrame) {
	loadedSize = MIN(toLoad, (size_t)(iend-ip));
	} else {
	RETURN_ERROR_IF(toLoad > zds->inBuffSize - zds->inPos,
	corruption_detected,
	"should never happen");
	loadedSize = ZSTD_limitCopy(zds->inBuff + zds->inPos, toLoad, ip, iend-ip);
	}
	ip += loadedSize;
	zds->inPos += loadedSize;
	if (loadedSize < toLoad) { someMoreWork = 0; break; } /* not enough input, wait for more */

	/* decode loaded input */
	zds->inPos = 0; /* input is consumed */
	FORWARD_IF_ERROR(ZSTD_decompressContinueStream(zds, &op, oend, zds->inBuff, neededInSize), "");
	/* Function modifies the stage so we must break */
	break;
	}
	case zdss_flush:
	{ size_t const toFlushSize = zds->outEnd - zds->outStart;
	size_t const flushedSize = ZSTD_limitCopy(op, oend-op, zds->outBuff + zds->outStart, toFlushSize);
	op += flushedSize;
	zds->outStart += flushedSize;
	if (flushedSize == toFlushSize) { /* flush completed */
	zds->streamStage = zdss_read;
	if ( (zds->outBuffSize < zds->fParams.frameContentSize)
	&& (zds->outStart + zds->fParams.blockSizeMax > zds->outBuffSize) ) {
	DEBUGLOG(5, "restart filling outBuff from beginning (left:%i, needed:%u)",
	(int)(zds->outBuffSize - zds->outStart),
	(U32)zds->fParams.blockSizeMax);
	zds->outStart = zds->outEnd = 0;
	}
	break;
	} }
	/* cannot complete flush */
	someMoreWork = 0;
	break;

	default:
	assert(0); /* impossible */
	RETURN_ERROR(GENERIC, "impossible to reach"); /* some compiler require default to do something */
	} }

	/* result */
	input->pos = (size_t)(ip - (const char*)(input->src));
	output->pos = (size_t)(op - (char*)(output->dst));

	/* Update the expected output buffer for ZSTD_obm_stable. */
	zds->expectedOutBuffer = *output;

	if ((ip==istart) && (op==ostart)) { /* no forward progress */
	zds->noForwardProgress ++;
	if (zds->noForwardProgress >= ZSTD_NO_FORWARD_PROGRESS_MAX) {
	RETURN_ERROR_IF(op==oend, dstSize_tooSmall, "");
	RETURN_ERROR_IF(ip==iend, srcSize_wrong, "");
	assert(0);
	}
	} else {
	zds->noForwardProgress = 0;
	}
	{ size_t nextSrcSizeHint = ZSTD_nextSrcSizeToDecompress(zds);
	if (!nextSrcSizeHint) { /* frame fully decoded */
	if (zds->outEnd == zds->outStart) { /* output fully flushed */
	if (zds->hostageByte) {
	if (input->pos >= input->size) {
	/* can't release hostage (not present) */
	zds->streamStage = zdss_read;
	return 1;
	}
	input->pos++; /* release hostage */
	} /* zds->hostageByte */
	return 0;
	} /* zds->outEnd == zds->outStart */
	if (!zds->hostageByte) { /* output not fully flushed; keep last byte as hostage; will be released when all output is flushed */
	input->pos--; /* note : pos > 0, otherwise, impossible to finish reading last block */
	zds->hostageByte=1;
	}
	return 1;
	} /* nextSrcSizeHint==0 */
	nextSrcSizeHint += ZSTD_blockHeaderSize * (ZSTD_nextInputType(zds) == ZSTDnit_block); /* preload header of next block */
	assert(zds->inPos <= nextSrcSizeHint);
	nextSrcSizeHint -= zds->inPos; /* part already loaded*/
	return nextSrcSizeHint;
	}
	}

	size_t ZSTD_decompressStream_simpleArgs (
	ZSTD_DCtx* dctx,
	void* dst, size_t dstCapacity, size_t* dstPos,
	const void* src, size_t srcSize, size_t* srcPos)
	{
	ZSTD_outBuffer output = { dst, dstCapacity, *dstPos };
	ZSTD_inBuffer input = { src, srcSize, *srcPos };
	/* ZSTD_compress_generic() will check validity of dstPos and srcPos */
	size_t const cErr = ZSTD_decompressStream(dctx, &output, &input);
	*dstPos = output.pos;
	*srcPos = input.pos;
	return cErr;
	}
	/** ended inlining decompress/zstd_decompress.c **/
	/** start inlining decompress/zstd_decompress_block.c **/
	/*
	* Copyright (c) 2016-2020, Yann Collet, Facebook, Inc.
	* All rights reserved.
	*
	* This source code is licensed under both the BSD-style license (found in the
	* LICENSE file in the root directory of this source tree) and the GPLv2 (found
	* in the COPYING file in the root directory of this source tree).
	* You may select, at your option, one of the above-listed licenses.
	*/

	/* zstd_decompress_block :
	* this module takes care of decompressing _compressed_ block */

	/-******************************************************
	* Dependencies
	*********************************************************/
	#include <string.h> /* memcpy, memmove, memset */
	/** skipping file: ../common/compiler.h **/
	/** skipping file: ../common/cpu.h **/
	/** skipping file: ../common/mem.h **/
	#define FSE_STATIC_LINKING_ONLY
	/** skipping file: ../common/fse.h **/
	#define HUF_STATIC_LINKING_ONLY
	/** skipping file: ../common/huf.h **/
	/** skipping file: ../common/zstd_internal.h **/
	/** skipping file: zstd_decompress_internal.h **/
	/** skipping file: zstd_ddict.h **/
	/** skipping file: zstd_decompress_block.h **/

	/_******************************************************
	* Macros
	**********************************************************/

	/* These two optional macros force the use one way or another of the two
	* ZSTD_decompressSequences implementations. You can't force in both directions
	* at the same time.
	*/
	#if defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT) && \
	defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG)
	#error "Cannot force the use of the short and the long ZSTD_decompressSequences variants!"
	#endif


	/_******************************************************
	* Memory operations
	**********************************************************/
	static void ZSTD_copy4(void* dst, const void* src) { memcpy(dst, src, 4); }


	/-************************************************************
	* Block decoding
	***************************************************************/

	/*! ZSTD_getcBlockSize() :
	* Provides the size of compressed block from block header `src` */
	size_t ZSTD_getcBlockSize(const void* src, size_t srcSize,
	blockProperties_t* bpPtr)
	{
	RETURN_ERROR_IF(srcSize < ZSTD_blockHeaderSize, srcSize_wrong, "");

	{ U32 const cBlockHeader = MEM_readLE24(src);
	U32 const cSize = cBlockHeader >> 3;
	bpPtr->lastBlock = cBlockHeader & 1;
	bpPtr->blockType = (blockType_e)((cBlockHeader >> 1) & 3);
	bpPtr->origSize = cSize; /* only useful for RLE */
	if (bpPtr->blockType == bt_rle) return 1;
	RETURN_ERROR_IF(bpPtr->blockType == bt_reserved, corruption_detected, "");
	return cSize;
	}
	}


	/* Hidden declaration for fullbench */
	size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
	const void* src, size_t srcSize);
	/*! ZSTD_decodeLiteralsBlock() :
	* @return : nb of bytes read from src (< srcSize )
	* note : symbol not declared but exposed for fullbench */
	size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
	const void* src, size_t srcSize) /* note : srcSize < BLOCKSIZE */
	{
	DEBUGLOG(5, "ZSTD_decodeLiteralsBlock");
	RETURN_ERROR_IF(srcSize < MIN_CBLOCK_SIZE, corruption_detected, "");

	{ const BYTE* const istart = (const BYTE*) src;
	symbolEncodingType_e const litEncType = (symbolEncodingType_e)(istart[0] & 3);

	switch(litEncType)
	{
	case set_repeat:
	DEBUGLOG(5, "set_repeat flag : re-using stats from previous compressed literals block");
	RETURN_ERROR_IF(dctx->litEntropy==0, dictionary_corrupted, "");
	/* fall-through */

	case set_compressed:
	RETURN_ERROR_IF(srcSize < 5, corruption_detected, "srcSize >= MIN_CBLOCK_SIZE == 3; here we need up to 5 for case 3");
	{ size_t lhSize, litSize, litCSize;
	U32 singleStream=0;
	U32 const lhlCode = (istart[0] >> 2) & 3;
	U32 const lhc = MEM_readLE32(istart);
	size_t hufSuccess;
	switch(lhlCode)
	{
	case 0: case 1: default: /* note : default is impossible, since lhlCode into [0..3] */
	/* 2 - 2 - 10 - 10 */
	singleStream = !lhlCode;
	lhSize = 3;
	litSize = (lhc >> 4) & 0x3FF;
	litCSize = (lhc >> 14) & 0x3FF;
	break;
	case 2:
	/* 2 - 2 - 14 - 14 */
	lhSize = 4;
	litSize = (lhc >> 4) & 0x3FFF;
	litCSize = lhc >> 18;
	break;
	case 3:
	/* 2 - 2 - 18 - 18 */
	lhSize = 5;
	litSize = (lhc >> 4) & 0x3FFFF;
	litCSize = (lhc >> 22) + ((size_t)istart[4] << 10);
	break;
	}
	RETURN_ERROR_IF(litSize > ZSTD_BLOCKSIZE_MAX, corruption_detected, "");
	RETURN_ERROR_IF(litCSize + lhSize > srcSize, corruption_detected, "");

	/* prefetch huffman table if cold */
	if (dctx->ddictIsCold && (litSize > 768 /* heuristic */)) {
	PREFETCH_AREA(dctx->HUFptr, sizeof(dctx->entropy.hufTable));
	}

	if (litEncType==set_repeat) {
	if (singleStream) {
	hufSuccess = HUF_decompress1X_usingDTable_bmi2(
	dctx->litBuffer, litSize, istart+lhSize, litCSize,
	dctx->HUFptr, dctx->bmi2);
	} else {
	hufSuccess = HUF_decompress4X_usingDTable_bmi2(
	dctx->litBuffer, litSize, istart+lhSize, litCSize,
	dctx->HUFptr, dctx->bmi2);
	}
	} else {
	if (singleStream) {
	#if defined(HUF_FORCE_DECOMPRESS_X2)
	hufSuccess = HUF_decompress1X_DCtx_wksp(
	dctx->entropy.hufTable, dctx->litBuffer, litSize,
	istart+lhSize, litCSize, dctx->workspace,
	sizeof(dctx->workspace));
	#else
	hufSuccess = HUF_decompress1X1_DCtx_wksp_bmi2(
	dctx->entropy.hufTable, dctx->litBuffer, litSize,
	istart+lhSize, litCSize, dctx->workspace,
	sizeof(dctx->workspace), dctx->bmi2);
	#endif
	} else {
	hufSuccess = HUF_decompress4X_hufOnly_wksp_bmi2(
	dctx->entropy.hufTable, dctx->litBuffer, litSize,
	istart+lhSize, litCSize, dctx->workspace,
	sizeof(dctx->workspace), dctx->bmi2);
	}
	}

	RETURN_ERROR_IF(HUF_isError(hufSuccess), corruption_detected, "");

	dctx->litPtr = dctx->litBuffer;
	dctx->litSize = litSize;
	dctx->litEntropy = 1;
	if (litEncType==set_compressed) dctx->HUFptr = dctx->entropy.hufTable;
	memset(dctx->litBuffer + dctx->litSize, 0, WILDCOPY_OVERLENGTH);
	return litCSize + lhSize;
	}

	case set_basic:
	{ size_t litSize, lhSize;
	U32 const lhlCode = ((istart[0]) >> 2) & 3;
	switch(lhlCode)
	{
	case 0: case 2: default: /* note : default is impossible, since lhlCode into [0..3] */
	lhSize = 1;
	litSize = istart[0] >> 3;
	break;
	case 1:
	lhSize = 2;
	litSize = MEM_readLE16(istart) >> 4;
	break;
	case 3:
	lhSize = 3;
	litSize = MEM_readLE24(istart) >> 4;
	break;
	}

	if (lhSize+litSize+WILDCOPY_OVERLENGTH > srcSize) { /* risk reading beyond src buffer with wildcopy */
	RETURN_ERROR_IF(litSize+lhSize > srcSize, corruption_detected, "");
	memcpy(dctx->litBuffer, istart+lhSize, litSize);
	dctx->litPtr = dctx->litBuffer;
	dctx->litSize = litSize;
	memset(dctx->litBuffer + dctx->litSize, 0, WILDCOPY_OVERLENGTH);
	return lhSize+litSize;
	}
	/* direct reference into compressed stream */
	dctx->litPtr = istart+lhSize;
	dctx->litSize = litSize;
	return lhSize+litSize;
	}

	case set_rle:
	{ U32 const lhlCode = ((istart[0]) >> 2) & 3;
	size_t litSize, lhSize;
	switch(lhlCode)
	{
	case 0: case 2: default: /* note : default is impossible, since lhlCode into [0..3] */
	lhSize = 1;
	litSize = istart[0] >> 3;
	break;
	case 1:
	lhSize = 2;
	litSize = MEM_readLE16(istart) >> 4;
	break;
	case 3:
	lhSize = 3;
	litSize = MEM_readLE24(istart) >> 4;
	RETURN_ERROR_IF(srcSize<4, corruption_detected, "srcSize >= MIN_CBLOCK_SIZE == 3; here we need lhSize+1 = 4");
	break;
	}
	RETURN_ERROR_IF(litSize > ZSTD_BLOCKSIZE_MAX, corruption_detected, "");
	memset(dctx->litBuffer, istart[lhSize], litSize + WILDCOPY_OVERLENGTH);
	dctx->litPtr = dctx->litBuffer;
	dctx->litSize = litSize;
	return lhSize+1;
	}
	default:
	RETURN_ERROR(corruption_detected, "impossible");
	}
	}
	}

	/* Default FSE distribution tables.
	* These are pre-calculated FSE decoding tables using default distributions as defined in specification :
	* https://github.com/facebook/zstd/blob/master/doc/zstd_compression_format.md#default-distributions
	* They were generated programmatically with following method :
	* - start from default distributions, present in /lib/common/zstd_internal.h
	* - generate tables normally, using ZSTD_buildFSETable()
	* - printout the content of tables
	* - pretify output, report below, test with fuzzer to ensure it's correct */

	/* Default FSE distribution table for Literal Lengths */
	static const ZSTD_seqSymbol LL_defaultDTable[(1<<LL_DEFAULTNORMLOG)+1] = {
	{ 1, 1, 1, LL_DEFAULTNORMLOG}, /* header : fastMode, tableLog */
	/* nextState, nbAddBits, nbBits, baseVal */
	{ 0, 0, 4, 0}, { 16, 0, 4, 0},
	{ 32, 0, 5, 1}, { 0, 0, 5, 3},
	{ 0, 0, 5, 4}, { 0, 0, 5, 6},
	{ 0, 0, 5, 7}, { 0, 0, 5, 9},
	{ 0, 0, 5, 10}, { 0, 0, 5, 12},
	{ 0, 0, 6, 14}, { 0, 1, 5, 16},
	{ 0, 1, 5, 20}, { 0, 1, 5, 22},
	{ 0, 2, 5, 28}, { 0, 3, 5, 32},
	{ 0, 4, 5, 48}, { 32, 6, 5, 64},
	{ 0, 7, 5, 128}, { 0, 8, 6, 256},
	{ 0, 10, 6, 1024}, { 0, 12, 6, 4096},
	{ 32, 0, 4, 0}, { 0, 0, 4, 1},
	{ 0, 0, 5, 2}, { 32, 0, 5, 4},
	{ 0, 0, 5, 5}, { 32, 0, 5, 7},
	{ 0, 0, 5, 8}, { 32, 0, 5, 10},
	{ 0, 0, 5, 11}, { 0, 0, 6, 13},
	{ 32, 1, 5, 16}, { 0, 1, 5, 18},
	{ 32, 1, 5, 22}, { 0, 2, 5, 24},
	{ 32, 3, 5, 32}, { 0, 3, 5, 40},
	{ 0, 6, 4, 64}, { 16, 6, 4, 64},
	{ 32, 7, 5, 128}, { 0, 9, 6, 512},
	{ 0, 11, 6, 2048}, { 48, 0, 4, 0},
	{ 16, 0, 4, 1}, { 32, 0, 5, 2},
	{ 32, 0, 5, 3}, { 32, 0, 5, 5},
	{ 32, 0, 5, 6}, { 32, 0, 5, 8},
	{ 32, 0, 5, 9}, { 32, 0, 5, 11},
	{ 32, 0, 5, 12}, { 0, 0, 6, 15},
	{ 32, 1, 5, 18}, { 32, 1, 5, 20},
	{ 32, 2, 5, 24}, { 32, 2, 5, 28},
	{ 32, 3, 5, 40}, { 32, 4, 5, 48},
	{ 0, 16, 6,65536}, { 0, 15, 6,32768},
	{ 0, 14, 6,16384}, { 0, 13, 6, 8192},
	}; /* LL_defaultDTable */

	/* Default FSE distribution table for Offset Codes */
	static const ZSTD_seqSymbol OF_defaultDTable[(1<<OF_DEFAULTNORMLOG)+1] = {
	{ 1, 1, 1, OF_DEFAULTNORMLOG}, /* header : fastMode, tableLog */
	/* nextState, nbAddBits, nbBits, baseVal */
	{ 0, 0, 5, 0}, { 0, 6, 4, 61},
	{ 0, 9, 5, 509}, { 0, 15, 5,32765},
	{ 0, 21, 5,2097149}, { 0, 3, 5, 5},
	{ 0, 7, 4, 125}, { 0, 12, 5, 4093},
	{ 0, 18, 5,262141}, { 0, 23, 5,8388605},
	{ 0, 5, 5, 29}, { 0, 8, 4, 253},
	{ 0, 14, 5,16381}, { 0, 20, 5,1048573},
	{ 0, 2, 5, 1}, { 16, 7, 4, 125},
	{ 0, 11, 5, 2045}, { 0, 17, 5,131069},
	{ 0, 22, 5,4194301}, { 0, 4, 5, 13},
	{ 16, 8, 4, 253}, { 0, 13, 5, 8189},
	{ 0, 19, 5,524285}, { 0, 1, 5, 1},
	{ 16, 6, 4, 61}, { 0, 10, 5, 1021},
	{ 0, 16, 5,65533}, { 0, 28, 5,268435453},
	{ 0, 27, 5,134217725}, { 0, 26, 5,67108861},
	{ 0, 25, 5,33554429}, { 0, 24, 5,16777213},
	}; /* OF_defaultDTable */


	/* Default FSE distribution table for Match Lengths */
	static const ZSTD_seqSymbol ML_defaultDTable[(1<<ML_DEFAULTNORMLOG)+1] = {
	{ 1, 1, 1, ML_DEFAULTNORMLOG}, /* header : fastMode, tableLog */
	/* nextState, nbAddBits, nbBits, baseVal */
	{ 0, 0, 6, 3}, { 0, 0, 4, 4},
	{ 32, 0, 5, 5}, { 0, 0, 5, 6},
	{ 0, 0, 5, 8}, { 0, 0, 5, 9},
	{ 0, 0, 5, 11}, { 0, 0, 6, 13},
	{ 0, 0, 6, 16}, { 0, 0, 6, 19},
	{ 0, 0, 6, 22}, { 0, 0, 6, 25},
	{ 0, 0, 6, 28}, { 0, 0, 6, 31},
	{ 0, 0, 6, 34}, { 0, 1, 6, 37},
	{ 0, 1, 6, 41}, { 0, 2, 6, 47},
	{ 0, 3, 6, 59}, { 0, 4, 6, 83},
	{ 0, 7, 6, 131}, { 0, 9, 6, 515},
	{ 16, 0, 4, 4}, { 0, 0, 4, 5},
	{ 32, 0, 5, 6}, { 0, 0, 5, 7},
	{ 32, 0, 5, 9}, { 0, 0, 5, 10},
	{ 0, 0, 6, 12}, { 0, 0, 6, 15},
	{ 0, 0, 6, 18}, { 0, 0, 6, 21},
	{ 0, 0, 6, 24}, { 0, 0, 6, 27},
	{ 0, 0, 6, 30}, { 0, 0, 6, 33},
	{ 0, 1, 6, 35}, { 0, 1, 6, 39},
	{ 0, 2, 6, 43}, { 0, 3, 6, 51},
	{ 0, 4, 6, 67}, { 0, 5, 6, 99},
	{ 0, 8, 6, 259}, { 32, 0, 4, 4},
	{ 48, 0, 4, 4}, { 16, 0, 4, 5},
	{ 32, 0, 5, 7}, { 32, 0, 5, 8},
	{ 32, 0, 5, 10}, { 32, 0, 5, 11},
	{ 0, 0, 6, 14}, { 0, 0, 6, 17},
	{ 0, 0, 6, 20}, { 0, 0, 6, 23},
	{ 0, 0, 6, 26}, { 0, 0, 6, 29},
	{ 0, 0, 6, 32}, { 0, 16, 6,65539},
	{ 0, 15, 6,32771}, { 0, 14, 6,16387},
	{ 0, 13, 6, 8195}, { 0, 12, 6, 4099},
	{ 0, 11, 6, 2051}, { 0, 10, 6, 1027},
	}; /* ML_defaultDTable */


	static void ZSTD_buildSeqTable_rle(ZSTD_seqSymbol* dt, U32 baseValue, U32 nbAddBits)
	{
	void* ptr = dt;
	ZSTD_seqSymbol_header* const DTableH = (ZSTD_seqSymbol_header*)ptr;
	ZSTD_seqSymbol* const cell = dt + 1;

	DTableH->tableLog = 0;
	DTableH->fastMode = 0;

	cell->nbBits = 0;
	cell->nextState = 0;
	assert(nbAddBits < 255);
	cell->nbAdditionalBits = (BYTE)nbAddBits;
	cell->baseValue = baseValue;
	}


	/* ZSTD_buildFSETable() :
	* generate FSE decoding table for one symbol (ll, ml or off)
	* cannot fail if input is valid =>
	* all inputs are presumed validated at this stage */
	void
	ZSTD_buildFSETable(ZSTD_seqSymbol* dt,
	const short* normalizedCounter, unsigned maxSymbolValue,
	const U32* baseValue, const U32* nbAdditionalBits,
	unsigned tableLog)
	{
	ZSTD_seqSymbol* const tableDecode = dt+1;
	U16 symbolNext[MaxSeq+1];

	U32 const maxSV1 = maxSymbolValue + 1;
	U32 const tableSize = 1 << tableLog;
	U32 highThreshold = tableSize-1;

	/* Sanity Checks */
	assert(maxSymbolValue <= MaxSeq);
	assert(tableLog <= MaxFSELog);

	/* Init, lay down lowprob symbols */
	{ ZSTD_seqSymbol_header DTableH;
	DTableH.tableLog = tableLog;
	DTableH.fastMode = 1;
	{ S16 const largeLimit= (S16)(1 << (tableLog-1));
	U32 s;
	for (s=0; s<maxSV1; s++) {
	if (normalizedCounter[s]==-1) {
	tableDecode[highThreshold--].baseValue = s;
	symbolNext[s] = 1;
	} else {
	if (normalizedCounter[s] >= largeLimit) DTableH.fastMode=0;
	assert(normalizedCounter[s]>=0);
	symbolNext[s] = (U16)normalizedCounter[s];
	} } }
	memcpy(dt, &DTableH, sizeof(DTableH));
	}

	/* Spread symbols */
	{ U32 const tableMask = tableSize-1;
	U32 const step = FSE_TABLESTEP(tableSize);
	U32 s, position = 0;
	for (s=0; s<maxSV1; s++) {
	int i;
	for (i=0; i<normalizedCounter[s]; i++) {
	tableDecode[position].baseValue = s;
	position = (position + step) & tableMask;
	while (position > highThreshold) position = (position + step) & tableMask; /* lowprob area */
	} }
	assert(position == 0); /* position must reach all cells once, otherwise normalizedCounter is incorrect */
	}

	/* Build Decoding table */
	{ U32 u;
	for (u=0; u<tableSize; u++) {
	U32 const symbol = tableDecode[u].baseValue;
	U32 const nextState = symbolNext[symbol]++;
	tableDecode[u].nbBits = (BYTE) (tableLog - BIT_highbit32(nextState) );
	tableDecode[u].nextState = (U16) ( (nextState << tableDecode[u].nbBits) - tableSize);
	assert(nbAdditionalBits[symbol] < 255);
	tableDecode[u].nbAdditionalBits = (BYTE)nbAdditionalBits[symbol];
	tableDecode[u].baseValue = baseValue[symbol];
	} }
	}


	/*! ZSTD_buildSeqTable() :
	* @return : nb bytes read from src,
	* or an error code if it fails */
	static size_t ZSTD_buildSeqTable(ZSTD_seqSymbol* DTableSpace, const ZSTD_seqSymbol** DTablePtr,
	symbolEncodingType_e type, unsigned max, U32 maxLog,
	const void* src, size_t srcSize,
	const U32* baseValue, const U32* nbAdditionalBits,
	const ZSTD_seqSymbol* defaultTable, U32 flagRepeatTable,
	int ddictIsCold, int nbSeq)
	{
	switch(type)
	{
	case set_rle :
	RETURN_ERROR_IF(!srcSize, srcSize_wrong, "");
	RETURN_ERROR_IF(((const BYTE)src) > max, corruption_detected, "");
	{ U32 const symbol = (const BYTE)src;
	U32 const baseline = baseValue[symbol];
	U32 const nbBits = nbAdditionalBits[symbol];
	ZSTD_buildSeqTable_rle(DTableSpace, baseline, nbBits);
	}
	*DTablePtr = DTableSpace;
	return 1;
	case set_basic :
	*DTablePtr = defaultTable;
	return 0;
	case set_repeat:
	RETURN_ERROR_IF(!flagRepeatTable, corruption_detected, "");
	/* prefetch FSE table if used */
	if (ddictIsCold && (nbSeq > 24 /* heuristic */)) {
	const void* const pStart = *DTablePtr;
	size_t const pSize = sizeof(ZSTD_seqSymbol) * (SEQSYMBOL_TABLE_SIZE(maxLog));
	PREFETCH_AREA(pStart, pSize);
	}
	return 0;
	case set_compressed :
	{ unsigned tableLog;
	S16 norm[MaxSeq+1];
	size_t const headerSize = FSE_readNCount(norm, &max, &tableLog, src, srcSize);
	RETURN_ERROR_IF(FSE_isError(headerSize), corruption_detected, "");
	RETURN_ERROR_IF(tableLog > maxLog, corruption_detected, "");
	ZSTD_buildFSETable(DTableSpace, norm, max, baseValue, nbAdditionalBits, tableLog);
	*DTablePtr = DTableSpace;
	return headerSize;
	}
	default :
	assert(0);
	RETURN_ERROR(GENERIC, "impossible");
	}
	}

	size_t ZSTD_decodeSeqHeaders(ZSTD_DCtx* dctx, int* nbSeqPtr,
	const void* src, size_t srcSize)
	{
	const BYTE* const istart = (const BYTE* const)src;
	const BYTE* const iend = istart + srcSize;
	const BYTE* ip = istart;
	int nbSeq;
	DEBUGLOG(5, "ZSTD_decodeSeqHeaders");

	/* check */
	RETURN_ERROR_IF(srcSize < MIN_SEQUENCES_SIZE, srcSize_wrong, "");

	/* SeqHead */
	nbSeq = *ip++;
	if (!nbSeq) {
	*nbSeqPtr=0;
	RETURN_ERROR_IF(srcSize != 1, srcSize_wrong, "");
	return 1;
	}
	if (nbSeq > 0x7F) {
	if (nbSeq == 0xFF) {
	RETURN_ERROR_IF(ip+2 > iend, srcSize_wrong, "");
	nbSeq = MEM_readLE16(ip) + LONGNBSEQ, ip+=2;
	} else {
	RETURN_ERROR_IF(ip >= iend, srcSize_wrong, "");
	nbSeq = ((nbSeq-0x80)<<8) + *ip++;
	}
	}
	*nbSeqPtr = nbSeq;

	/* FSE table descriptors */
	RETURN_ERROR_IF(ip+1 > iend, srcSize_wrong, ""); /* minimum possible size: 1 byte for symbol encoding types */
	{ symbolEncodingType_e const LLtype = (symbolEncodingType_e)(*ip >> 6);
	symbolEncodingType_e const OFtype = (symbolEncodingType_e)((*ip >> 4) & 3);
	symbolEncodingType_e const MLtype = (symbolEncodingType_e)((*ip >> 2) & 3);
	ip++;

	/* Build DTables */
	{ size_t const llhSize = ZSTD_buildSeqTable(dctx->entropy.LLTable, &dctx->LLTptr,
	LLtype, MaxLL, LLFSELog,
	ip, iend-ip,
	LL_base, LL_bits,
	LL_defaultDTable, dctx->fseEntropy,
	dctx->ddictIsCold, nbSeq);
	RETURN_ERROR_IF(ZSTD_isError(llhSize), corruption_detected, "ZSTD_buildSeqTable failed");
	ip += llhSize;
	}

	{ size_t const ofhSize = ZSTD_buildSeqTable(dctx->entropy.OFTable, &dctx->OFTptr,
	OFtype, MaxOff, OffFSELog,
	ip, iend-ip,
	OF_base, OF_bits,
	OF_defaultDTable, dctx->fseEntropy,
	dctx->ddictIsCold, nbSeq);
	RETURN_ERROR_IF(ZSTD_isError(ofhSize), corruption_detected, "ZSTD_buildSeqTable failed");
	ip += ofhSize;
	}

	{ size_t const mlhSize = ZSTD_buildSeqTable(dctx->entropy.MLTable, &dctx->MLTptr,
	MLtype, MaxML, MLFSELog,
	ip, iend-ip,
	ML_base, ML_bits,
	ML_defaultDTable, dctx->fseEntropy,
	dctx->ddictIsCold, nbSeq);
	RETURN_ERROR_IF(ZSTD_isError(mlhSize), corruption_detected, "ZSTD_buildSeqTable failed");
	ip += mlhSize;
	}
	}

	return ip-istart;
	}


	typedef struct {
	size_t litLength;
	size_t matchLength;
	size_t offset;
	const BYTE* match;
	} seq_t;

	typedef struct {
	size_t state;
	const ZSTD_seqSymbol* table;
	} ZSTD_fseState;

	typedef struct {
	BIT_DStream_t DStream;
	ZSTD_fseState stateLL;
	ZSTD_fseState stateOffb;
	ZSTD_fseState stateML;
	size_t prevOffset[ZSTD_REP_NUM];
	const BYTE* prefixStart;
	const BYTE* dictEnd;
	size_t pos;
	} seqState_t;

	/*! ZSTD_overlapCopy8() :
	* Copies 8 bytes from ip to op and updates op and ip where ip <= op.
	* If the offset is < 8 then the offset is spread to at least 8 bytes.
	*
	* Precondition: ip <= op
	* Postcondition: op - op >= 8
	*/
	HINT_INLINE void ZSTD_overlapCopy8(BYTE op, BYTE const ip, size_t offset) {
	assert(ip <= op);
	if (offset < 8) {
	/* close range match, overlap */
	static const U32 dec32table[] = { 0, 1, 2, 1, 4, 4, 4, 4 }; /* added */
	static const int dec64table[] = { 8, 8, 8, 7, 8, 9,10,11 }; /* subtracted */
	int const sub2 = dec64table[offset];
	(op)[0] = (ip)[0];
	(op)[1] = (ip)[1];
	(op)[2] = (ip)[2];
	(op)[3] = (ip)[3];
	*ip += dec32table[offset];
	ZSTD_copy4(op+4, ip);
	*ip -= sub2;
	} else {
	ZSTD_copy8(op, ip);
	}
	*ip += 8;
	*op += 8;
	assert(op - ip >= 8);
	}

	/*! ZSTD_safecopy() :
	* Specialized version of memcpy() that is allowed to READ up to WILDCOPY_OVERLENGTH past the input buffer
	* and write up to 16 bytes past oend_w (op >= oend_w is allowed).
	* This function is only called in the uncommon case where the sequence is near the end of the block. It
	* should be fast for a single long sequence, but can be slow for several short sequences.
	*
	* @param ovtype controls the overlap detection
	* - ZSTD_no_overlap: The source and destination are guaranteed to be at least WILDCOPY_VECLEN bytes apart.
	* - ZSTD_overlap_src_before_dst: The src and dst may overlap and may be any distance apart.
	* The src buffer must be before the dst buffer.
	*/
	static void ZSTD_safecopy(BYTE* op, BYTE* const oend_w, BYTE const* ip, ptrdiff_t length, ZSTD_overlap_e ovtype) {
	ptrdiff_t const diff = op - ip;
	BYTE* const oend = op + length;

	assert((ovtype == ZSTD_no_overlap && (diff <= -8 \|\| diff >= 8 \|\| op >= oend_w)) \|\|
	(ovtype == ZSTD_overlap_src_before_dst && diff >= 0));

	if (length < 8) {
	/* Handle short lengths. */
	while (op < oend) op++ = ip++;
	return;
	}
	if (ovtype == ZSTD_overlap_src_before_dst) {
	/* Copy 8 bytes and ensure the offset >= 8 when there can be overlap. */
	assert(length >= 8);
	ZSTD_overlapCopy8(&op, &ip, diff);
	assert(op - ip >= 8);
	assert(op <= oend);
	}

	if (oend <= oend_w) {
	/* No risk of overwrite. */
	ZSTD_wildcopy(op, ip, length, ovtype);
	return;
	}
	if (op <= oend_w) {
	/* Wildcopy until we get close to the end. */
	assert(oend > oend_w);
	ZSTD_wildcopy(op, ip, oend_w - op, ovtype);
	ip += oend_w - op;
	op = oend_w;
	}
	/* Handle the leftovers. */
	while (op < oend) op++ = ip++;
	}

	/* ZSTD_execSequenceEnd():
	* This version handles cases that are near the end of the output buffer. It requires
	* more careful checks to make sure there is no overflow. By separating out these hard
	* and unlikely cases, we can speed up the common cases.
	*
	* NOTE: This function needs to be fast for a single long sequence, but doesn't need
	* to be optimized for many small sequences, since those fall into ZSTD_execSequence().
	*/
	FORCE_NOINLINE
	size_t ZSTD_execSequenceEnd(BYTE* op,
	BYTE* const oend, seq_t sequence,
	const BYTE** litPtr, const BYTE* const litLimit,
	const BYTE* const prefixStart, const BYTE* const virtualStart, const BYTE* const dictEnd)
	{
	BYTE* const oLitEnd = op + sequence.litLength;
	size_t const sequenceLength = sequence.litLength + sequence.matchLength;
	const BYTE* const iLitEnd = *litPtr + sequence.litLength;
	const BYTE* match = oLitEnd - sequence.offset;
	BYTE* const oend_w = oend - WILDCOPY_OVERLENGTH;

	/* bounds checks : careful of address space overflow in 32-bit mode */
	RETURN_ERROR_IF(sequenceLength > (size_t)(oend - op), dstSize_tooSmall, "last match must fit within dstBuffer");
	RETURN_ERROR_IF(sequence.litLength > (size_t)(litLimit - *litPtr), corruption_detected, "try to read beyond literal buffer");
	assert(op < op + sequenceLength);
	assert(oLitEnd < op + sequenceLength);

	/* copy literals */
	ZSTD_safecopy(op, oend_w, *litPtr, sequence.litLength, ZSTD_no_overlap);
	op = oLitEnd;
	*litPtr = iLitEnd;

	/* copy Match */
	if (sequence.offset > (size_t)(oLitEnd - prefixStart)) {
	/* offset beyond prefix */
	RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - virtualStart), corruption_detected, "");
	match = dictEnd - (prefixStart-match);
	if (match + sequence.matchLength <= dictEnd) {
	memmove(oLitEnd, match, sequence.matchLength);
	return sequenceLength;
	}
	/* span extDict & currentPrefixSegment */
	{ size_t const length1 = dictEnd - match;
	memmove(oLitEnd, match, length1);
	op = oLitEnd + length1;
	sequence.matchLength -= length1;
	match = prefixStart;
	} }
	ZSTD_safecopy(op, oend_w, match, sequence.matchLength, ZSTD_overlap_src_before_dst);
	return sequenceLength;
	}

	HINT_INLINE
	size_t ZSTD_execSequence(BYTE* op,
	BYTE* const oend, seq_t sequence,
	const BYTE** litPtr, const BYTE* const litLimit,
	const BYTE* const prefixStart, const BYTE* const virtualStart, const BYTE* const dictEnd)
	{
	BYTE* const oLitEnd = op + sequence.litLength;
	size_t const sequenceLength = sequence.litLength + sequence.matchLength;
	BYTE* const oMatchEnd = op + sequenceLength; /* risk : address space overflow (32-bits) */
	BYTE* const oend_w = oend - WILDCOPY_OVERLENGTH; /* risk : address space underflow on oend=NULL */
	const BYTE* const iLitEnd = *litPtr + sequence.litLength;
	const BYTE* match = oLitEnd - sequence.offset;

	assert(op != NULL /* Precondition */);
	assert(oend_w < oend /* No underflow */);
	/* Handle edge cases in a slow path:
	* - Read beyond end of literals
	* - Match end is within WILDCOPY_OVERLIMIT of oend
	* - 32-bit mode and the match length overflows
	*/
	if (UNLIKELY(
	iLitEnd > litLimit \|\|
	oMatchEnd > oend_w \|\|
	(MEM_32bits() && (size_t)(oend - op) < sequenceLength + WILDCOPY_OVERLENGTH)))
	return ZSTD_execSequenceEnd(op, oend, sequence, litPtr, litLimit, prefixStart, virtualStart, dictEnd);

	/* Assumptions (everything else goes into ZSTD_execSequenceEnd()) */
	assert(op <= oLitEnd /* No overflow */);
	assert(oLitEnd < oMatchEnd /* Non-zero match & no overflow */);
	assert(oMatchEnd <= oend /* No underflow */);
	assert(iLitEnd <= litLimit /* Literal length is in bounds */);
	assert(oLitEnd <= oend_w /* Can wildcopy literals */);
	assert(oMatchEnd <= oend_w /* Can wildcopy matches */);

	/* Copy Literals:
	* Split out litLength <= 16 since it is nearly always true. +1.6% on gcc-9.
	* We likely don't need the full 32-byte wildcopy.
	*/
	assert(WILDCOPY_OVERLENGTH >= 16);
	ZSTD_copy16(op, (*litPtr));
	if (UNLIKELY(sequence.litLength > 16)) {
	ZSTD_wildcopy(op+16, (*litPtr)+16, sequence.litLength-16, ZSTD_no_overlap);
	}
	op = oLitEnd;
	litPtr = iLitEnd; / update for next sequence */

	/* Copy Match */
	if (sequence.offset > (size_t)(oLitEnd - prefixStart)) {
	/* offset beyond prefix -> go into extDict */
	RETURN_ERROR_IF(UNLIKELY(sequence.offset > (size_t)(oLitEnd - virtualStart)), corruption_detected, "");
	match = dictEnd + (match - prefixStart);
	if (match + sequence.matchLength <= dictEnd) {
	memmove(oLitEnd, match, sequence.matchLength);
	return sequenceLength;
	}
	/* span extDict & currentPrefixSegment */
	{ size_t const length1 = dictEnd - match;
	memmove(oLitEnd, match, length1);
	op = oLitEnd + length1;
	sequence.matchLength -= length1;
	match = prefixStart;
	} }
	/* Match within prefix of 1 or more bytes */
	assert(op <= oMatchEnd);
	assert(oMatchEnd <= oend_w);
	assert(match >= prefixStart);
	assert(sequence.matchLength >= 1);

	/* Nearly all offsets are >= WILDCOPY_VECLEN bytes, which means we can use wildcopy
	* without overlap checking.
	*/
	if (LIKELY(sequence.offset >= WILDCOPY_VECLEN)) {
	/* We bet on a full wildcopy for matches, since we expect matches to be
	* longer than literals (in general). In silesia, ~10% of matches are longer
	* than 16 bytes.
	*/
	ZSTD_wildcopy(op, match, (ptrdiff_t)sequence.matchLength, ZSTD_no_overlap);
	return sequenceLength;
	}
	assert(sequence.offset < WILDCOPY_VECLEN);

	/* Copy 8 bytes and spread the offset to be >= 8. */
	ZSTD_overlapCopy8(&op, &match, sequence.offset);

	/* If the match length is > 8 bytes, then continue with the wildcopy. */
	if (sequence.matchLength > 8) {
	assert(op < oMatchEnd);
	ZSTD_wildcopy(op, match, (ptrdiff_t)sequence.matchLength-8, ZSTD_overlap_src_before_dst);
	}
	return sequenceLength;
	}

	static void
	ZSTD_initFseState(ZSTD_fseState* DStatePtr, BIT_DStream_t* bitD, const ZSTD_seqSymbol* dt)
	{
	const void* ptr = dt;
	const ZSTD_seqSymbol_header* const DTableH = (const ZSTD_seqSymbol_header*)ptr;
	DStatePtr->state = BIT_readBits(bitD, DTableH->tableLog);
	DEBUGLOG(6, "ZSTD_initFseState : val=%u using %u bits",
	(U32)DStatePtr->state, DTableH->tableLog);
	BIT_reloadDStream(bitD);
	DStatePtr->table = dt + 1;
	}

	FORCE_INLINE_TEMPLATE void
	ZSTD_updateFseState(ZSTD_fseState* DStatePtr, BIT_DStream_t* bitD)
	{
	ZSTD_seqSymbol const DInfo = DStatePtr->table[DStatePtr->state];
	U32 const nbBits = DInfo.nbBits;
	size_t const lowBits = BIT_readBits(bitD, nbBits);
	DStatePtr->state = DInfo.nextState + lowBits;
	}

	FORCE_INLINE_TEMPLATE void
	ZSTD_updateFseStateWithDInfo(ZSTD_fseState* DStatePtr, BIT_DStream_t* bitD, ZSTD_seqSymbol const DInfo)
	{
	U32 const nbBits = DInfo.nbBits;
	size_t const lowBits = BIT_readBits(bitD, nbBits);
	DStatePtr->state = DInfo.nextState + lowBits;
	}

	/* We need to add at most (ZSTD_WINDOWLOG_MAX_32 - 1) bits to read the maximum
	* offset bits. But we can only read at most (STREAM_ACCUMULATOR_MIN_32 - 1)
	* bits before reloading. This value is the maximum number of bytes we read
	* after reloading when we are decoding long offsets.
	*/
	#define LONG_OFFSETS_MAX_EXTRA_BITS_32 \
	(ZSTD_WINDOWLOG_MAX_32 > STREAM_ACCUMULATOR_MIN_32 \
	? ZSTD_WINDOWLOG_MAX_32 - STREAM_ACCUMULATOR_MIN_32 \
	: 0)

	typedef enum { ZSTD_lo_isRegularOffset, ZSTD_lo_isLongOffset=1 } ZSTD_longOffset_e;
	typedef enum { ZSTD_p_noPrefetch=0, ZSTD_p_prefetch=1 } ZSTD_prefetch_e;

	FORCE_INLINE_TEMPLATE seq_t
	ZSTD_decodeSequence(seqState_t* seqState, const ZSTD_longOffset_e longOffsets, const ZSTD_prefetch_e prefetch)
	{
	seq_t seq;
	ZSTD_seqSymbol const llDInfo = seqState->stateLL.table[seqState->stateLL.state];
	ZSTD_seqSymbol const mlDInfo = seqState->stateML.table[seqState->stateML.state];
	ZSTD_seqSymbol const ofDInfo = seqState->stateOffb.table[seqState->stateOffb.state];
	U32 const llBase = llDInfo.baseValue;
	U32 const mlBase = mlDInfo.baseValue;
	U32 const ofBase = ofDInfo.baseValue;
	BYTE const llBits = llDInfo.nbAdditionalBits;
	BYTE const mlBits = mlDInfo.nbAdditionalBits;
	BYTE const ofBits = ofDInfo.nbAdditionalBits;
	BYTE const totalBits = llBits+mlBits+ofBits;

	/* sequence */
	{ size_t offset;
	if (ofBits > 1) {
	ZSTD_STATIC_ASSERT(ZSTD_lo_isLongOffset == 1);
	ZSTD_STATIC_ASSERT(LONG_OFFSETS_MAX_EXTRA_BITS_32 == 5);
	assert(ofBits <= MaxOff);
	if (MEM_32bits() && longOffsets && (ofBits >= STREAM_ACCUMULATOR_MIN_32)) {
	U32 const extraBits = ofBits - MIN(ofBits, 32 - seqState->DStream.bitsConsumed);
	offset = ofBase + (BIT_readBitsFast(&seqState->DStream, ofBits - extraBits) << extraBits);
	BIT_reloadDStream(&seqState->DStream);
	if (extraBits) offset += BIT_readBitsFast(&seqState->DStream, extraBits);
	assert(extraBits <= LONG_OFFSETS_MAX_EXTRA_BITS_32); /* to avoid another reload */
	} else {
	offset = ofBase + BIT_readBitsFast(&seqState->DStream, ofBits/>0/); /* <= (ZSTD_WINDOWLOG_MAX-1) bits */
	if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream);
	}
	seqState->prevOffset[2] = seqState->prevOffset[1];
	seqState->prevOffset[1] = seqState->prevOffset[0];
	seqState->prevOffset[0] = offset;
	} else {
	U32 const ll0 = (llBase == 0);
	if (LIKELY((ofBits == 0))) {
	if (LIKELY(!ll0))
	offset = seqState->prevOffset[0];
	else {
	offset = seqState->prevOffset[1];
	seqState->prevOffset[1] = seqState->prevOffset[0];
	seqState->prevOffset[0] = offset;
	}
	} else {
	offset = ofBase + ll0 + BIT_readBitsFast(&seqState->DStream, 1);
	{ size_t temp = (offset==3) ? seqState->prevOffset[0] - 1 : seqState->prevOffset[offset];
	temp += !temp; /* 0 is not valid; input is corrupted; force offset to 1 */
	if (offset != 1) seqState->prevOffset[2] = seqState->prevOffset[1];
	seqState->prevOffset[1] = seqState->prevOffset[0];
	seqState->prevOffset[0] = offset = temp;
	} } }
	seq.offset = offset;
	}

	seq.matchLength = mlBase;
	if (mlBits > 0)
	seq.matchLength += BIT_readBitsFast(&seqState->DStream, mlBits/>0/);

	if (MEM_32bits() && (mlBits+llBits >= STREAM_ACCUMULATOR_MIN_32-LONG_OFFSETS_MAX_EXTRA_BITS_32))
	BIT_reloadDStream(&seqState->DStream);
	if (MEM_64bits() && UNLIKELY(totalBits >= STREAM_ACCUMULATOR_MIN_64-(LLFSELog+MLFSELog+OffFSELog)))
	BIT_reloadDStream(&seqState->DStream);
	/* Ensure there are enough bits to read the rest of data in 64-bit mode. */
	ZSTD_STATIC_ASSERT(16+LLFSELog+MLFSELog+OffFSELog < STREAM_ACCUMULATOR_MIN_64);

	seq.litLength = llBase;
	if (llBits > 0)
	seq.litLength += BIT_readBitsFast(&seqState->DStream, llBits/>0/);

	if (MEM_32bits())
	BIT_reloadDStream(&seqState->DStream);

	DEBUGLOG(6, "seq: litL=%u, matchL=%u, offset=%u",
	(U32)seq.litLength, (U32)seq.matchLength, (U32)seq.offset);

	if (prefetch == ZSTD_p_prefetch) {
	size_t const pos = seqState->pos + seq.litLength;
	const BYTE* const matchBase = (seq.offset > pos) ? seqState->dictEnd : seqState->prefixStart;
	seq.match = matchBase + pos - seq.offset; /* note : this operation can overflow when seq.offset is really too large, which can only happen when input is corrupted.
	* No consequence though : no memory access will occur, offset is only used for prefetching */
	seqState->pos = pos + seq.matchLength;
	}

	/* ANS state update
	* gcc-9.0.0 does 2.5% worse with ZSTD_updateFseStateWithDInfo().
	* clang-9.2.0 does 7% worse with ZSTD_updateFseState().
	* Naturally it seems like ZSTD_updateFseStateWithDInfo() should be the
	* better option, so it is the default for other compilers. But, if you
	* measure that it is worse, please put up a pull request.
	*/
	{
	#if defined(__GNUC__) && !defined(__clang__)
	const int kUseUpdateFseState = 1;
	#else
	const int kUseUpdateFseState = 0;
	#endif
	if (kUseUpdateFseState) {
	ZSTD_updateFseState(&seqState->stateLL, &seqState->DStream); /* <= 9 bits */
	ZSTD_updateFseState(&seqState->stateML, &seqState->DStream); /* <= 9 bits */
	if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream); /* <= 18 bits */
	ZSTD_updateFseState(&seqState->stateOffb, &seqState->DStream); /* <= 8 bits */
	} else {
	ZSTD_updateFseStateWithDInfo(&seqState->stateLL, &seqState->DStream, llDInfo); /* <= 9 bits */
	ZSTD_updateFseStateWithDInfo(&seqState->stateML, &seqState->DStream, mlDInfo); /* <= 9 bits */
	if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream); /* <= 18 bits */
	ZSTD_updateFseStateWithDInfo(&seqState->stateOffb, &seqState->DStream, ofDInfo); /* <= 8 bits */
	}
	}

	return seq;
	}

	#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
	static int ZSTD_dictionaryIsActive(ZSTD_DCtx const* dctx, BYTE const* prefixStart, BYTE const* oLitEnd)
	{
	size_t const windowSize = dctx->fParams.windowSize;
	/* No dictionary used. */
	if (dctx->dictContentEndForFuzzing == NULL) return 0;
	/* Dictionary is our prefix. */
	if (prefixStart == dctx->dictContentBeginForFuzzing) return 1;
	/* Dictionary is not our ext-dict. */
	if (dctx->dictEnd != dctx->dictContentEndForFuzzing) return 0;
	/* Dictionary is not within our window size. */
	if ((size_t)(oLitEnd - prefixStart) >= windowSize) return 0;
	/* Dictionary is active. */
	return 1;
	}

	MEM_STATIC void ZSTD_assertValidSequence(
	ZSTD_DCtx const* dctx,
	BYTE const* op, BYTE const* oend,
	seq_t const seq,
	BYTE const* prefixStart, BYTE const* virtualStart)
	{
	size_t const windowSize = dctx->fParams.windowSize;
	size_t const sequenceSize = seq.litLength + seq.matchLength;
	BYTE const* const oLitEnd = op + seq.litLength;
	DEBUGLOG(6, "Checking sequence: litL=%u matchL=%u offset=%u",
	(U32)seq.litLength, (U32)seq.matchLength, (U32)seq.offset);
	assert(op <= oend);
	assert((size_t)(oend - op) >= sequenceSize);
	assert(sequenceSize <= ZSTD_BLOCKSIZE_MAX);
	if (ZSTD_dictionaryIsActive(dctx, prefixStart, oLitEnd)) {
	size_t const dictSize = (size_t)((char const)dctx->dictContentEndForFuzzing - (char const)dctx->dictContentBeginForFuzzing);
	/* Offset must be within the dictionary. */
	assert(seq.offset <= (size_t)(oLitEnd - virtualStart));
	assert(seq.offset <= windowSize + dictSize);
	} else {
	/* Offset must be within our window. */
	assert(seq.offset <= windowSize);
	}
	}
	#endif

	#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG
	FORCE_INLINE_TEMPLATE size_t
	DONT_VECTORIZE
	ZSTD_decompressSequences_body( ZSTD_DCtx* dctx,
	void* dst, size_t maxDstSize,
	const void* seqStart, size_t seqSize, int nbSeq,
	const ZSTD_longOffset_e isLongOffset,
	const int frame)
	{
	const BYTE* ip = (const BYTE*)seqStart;
	const BYTE* const iend = ip + seqSize;
	BYTE* const ostart = (BYTE* const)dst;
	BYTE* const oend = ostart + maxDstSize;
	BYTE* op = ostart;
	const BYTE* litPtr = dctx->litPtr;
	const BYTE* const litEnd = litPtr + dctx->litSize;
	const BYTE* const prefixStart = (const BYTE*) (dctx->prefixStart);
	const BYTE* const vBase = (const BYTE*) (dctx->virtualStart);
	const BYTE* const dictEnd = (const BYTE*) (dctx->dictEnd);
	DEBUGLOG(5, "ZSTD_decompressSequences_body");
	(void)frame;

	/* Regen sequences */
	if (nbSeq) {
	seqState_t seqState;
	size_t error = 0;
	dctx->fseEntropy = 1;
	{ U32 i; for (i=0; i<ZSTD_REP_NUM; i++) seqState.prevOffset[i] = dctx->entropy.rep[i]; }
	RETURN_ERROR_IF(
	ERR_isError(BIT_initDStream(&seqState.DStream, ip, iend-ip)),
	corruption_detected, "");
	ZSTD_initFseState(&seqState.stateLL, &seqState.DStream, dctx->LLTptr);
	ZSTD_initFseState(&seqState.stateOffb, &seqState.DStream, dctx->OFTptr);
	ZSTD_initFseState(&seqState.stateML, &seqState.DStream, dctx->MLTptr);
	assert(dst != NULL);

	ZSTD_STATIC_ASSERT(
	BIT_DStream_unfinished < BIT_DStream_completed &&
	BIT_DStream_endOfBuffer < BIT_DStream_completed &&
	BIT_DStream_completed < BIT_DStream_overflow);

	#if defined(__GNUC__) && defined(__x86_64__)
	/* Align the decompression loop to 32 + 16 bytes.
	*
	* zstd compiled with gcc-9 on an Intel i9-9900k shows 10% decompression
	* speed swings based on the alignment of the decompression loop. This
	* performance swing is caused by parts of the decompression loop falling
	* out of the DSB. The entire decompression loop should fit in the DSB,
	* when it can't we get much worse performance. You can measure if you've
	* hit the good case or the bad case with this perf command for some
	* compressed file test.zst:
	*
	* perf stat -e cycles -e instructions -e idq.all_dsb_cycles_any_uops \
	* -e idq.all_mite_cycles_any_uops -- ./zstd -tq test.zst
	*
	* If you see most cycles served out of the MITE you've hit the bad case.
	* If you see most cycles served out of the DSB you've hit the good case.
	* If it is pretty even then you may be in an okay case.
	*
	* I've been able to reproduce this issue on the following CPUs:
	* - Kabylake: Macbook Pro (15-inch, 2019) 2.4 GHz Intel Core i9
	* Use Instruments->Counters to get DSB/MITE cycles.
	* I never got performance swings, but I was able to
	* go from the good case of mostly DSB to half of the
	* cycles served from MITE.
	* - Coffeelake: Intel i9-9900k
	*
	* I haven't been able to reproduce the instability or DSB misses on any
	* of the following CPUS:
	* - Haswell
	* - Broadwell: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GH
	* - Skylake
	*
	* If you are seeing performance stability this script can help test.
	* It tests on 4 commits in zstd where I saw performance change.
	*
	* https://gist.github.com/terrelln/9889fc06a423fd5ca6e99351564473f4
	*/
	__asm__(".p2align 5");
	__asm__("nop");
	__asm__(".p2align 4");
	#endif
	for ( ; ; ) {
	seq_t const sequence = ZSTD_decodeSequence(&seqState, isLongOffset, ZSTD_p_noPrefetch);
	size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequence, &litPtr, litEnd, prefixStart, vBase, dictEnd);
	#if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE)
	assert(!ZSTD_isError(oneSeqSize));
	if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequence, prefixStart, vBase);
	#endif
	DEBUGLOG(6, "regenerated sequence size : %u", (U32)oneSeqSize);
	BIT_reloadDStream(&(seqState.DStream));
	/* gcc and clang both don't like early returns in this loop.
	* gcc doesn't like early breaks either.
	* Instead save an error and report it at the end.
	* When there is an error, don't increment op, so we don't
	* overwrite.
	*/
	if (UNLIKELY(ZSTD_isError(oneSeqSize))) error = oneSeqSize;
	else op += oneSeqSize;
	if (UNLIKELY(!--nbSeq)) break;
	}

	/* check if reached exact end */
	DEBUGLOG(5, "ZSTD_decompressSequences_body: after decode loop, remaining nbSeq : %i", nbSeq);
	if (ZSTD_isError(error)) return error;
	RETURN_ERROR_IF(nbSeq, corruption_detected, "");
	RETURN_ERROR_IF(BIT_reloadDStream(&seqState.DStream) < BIT_DStream_completed, corruption_detected, "");
	/* save reps for next block */
	{ U32 i; for (i=0; i<ZSTD_REP_NUM; i++) dctx->entropy.rep[i] = (U32)(seqState.prevOffset[i]); }
	}

	/* last literal segment */
	{ size_t const lastLLSize = litEnd - litPtr;
	RETURN_ERROR_IF(lastLLSize > (size_t)(oend-op), dstSize_tooSmall, "");
	if (op != NULL) {
	memcpy(op, litPtr, lastLLSize);
	op += lastLLSize;
	}
	}

	return op-ostart;
	}

	static size_t
	ZSTD_decompressSequences_default(ZSTD_DCtx* dctx,
	void* dst, size_t maxDstSize,
	const void* seqStart, size_t seqSize, int nbSeq,
	const ZSTD_longOffset_e isLongOffset,
	const int frame)
	{
	return ZSTD_decompressSequences_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
	}
	#endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG */

	#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT
	FORCE_INLINE_TEMPLATE size_t
	ZSTD_decompressSequencesLong_body(
	ZSTD_DCtx* dctx,
	void* dst, size_t maxDstSize,
	const void* seqStart, size_t seqSize, int nbSeq,
	const ZSTD_longOffset_e isLongOffset,
	const int frame)
	{
	const BYTE* ip = (const BYTE*)seqStart;
	const BYTE* const iend = ip + seqSize;
	BYTE* const ostart = (BYTE* const)dst;
	BYTE* const oend = ostart + maxDstSize;
	BYTE* op = ostart;
	const BYTE* litPtr = dctx->litPtr;
	const BYTE* const litEnd = litPtr + dctx->litSize;
	const BYTE* const prefixStart = (const BYTE*) (dctx->prefixStart);
	const BYTE* const dictStart = (const BYTE*) (dctx->virtualStart);
	const BYTE* const dictEnd = (const BYTE*) (dctx->dictEnd);
	(void)frame;

	/* Regen sequences */
	if (nbSeq) {
	#define STORED_SEQS 4
	#define STORED_SEQS_MASK (STORED_SEQS-1)
	#define ADVANCED_SEQS 4
	seq_t sequences[STORED_SEQS];
	int const seqAdvance = MIN(nbSeq, ADVANCED_SEQS);
	seqState_t seqState;
	int seqNb;
	dctx->fseEntropy = 1;
	{ int i; for (i=0; i<ZSTD_REP_NUM; i++) seqState.prevOffset[i] = dctx->entropy.rep[i]; }
	seqState.prefixStart = prefixStart;
	seqState.pos = (size_t)(op-prefixStart);
	seqState.dictEnd = dictEnd;
	assert(dst != NULL);
	assert(iend >= ip);
	RETURN_ERROR_IF(
	ERR_isError(BIT_initDStream(&seqState.DStream, ip, iend-ip)),
	corruption_detected, "");
	ZSTD_initFseState(&seqState.stateLL, &seqState.DStream, dctx->LLTptr);
	ZSTD_initFseState(&seqState.stateOffb, &seqState.DStream, dctx->OFTptr);
	ZSTD_initFseState(&seqState.stateML, &seqState.DStream, dctx->MLTptr);

	/* prepare in advance */
	for (seqNb=0; (BIT_reloadDStream(&seqState.DStream) <= BIT_DStream_completed) && (seqNb<seqAdvance); seqNb++) {
	sequences[seqNb] = ZSTD_decodeSequence(&seqState, isLongOffset, ZSTD_p_prefetch);
	PREFETCH_L1(sequences[seqNb].match); PREFETCH_L1(sequences[seqNb].match + sequences[seqNb].matchLength - 1); /* note : it's safe to invoke PREFETCH() on any memory address, including invalid ones */
	}
	RETURN_ERROR_IF(seqNb<seqAdvance, corruption_detected, "");

	/* decode and decompress */
	for ( ; (BIT_reloadDStream(&(seqState.DStream)) <= BIT_DStream_completed) && (seqNb<nbSeq) ; seqNb++) {
	seq_t const sequence = ZSTD_decodeSequence(&seqState, isLongOffset, ZSTD_p_prefetch);
	size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequences[(seqNb-ADVANCED_SEQS) & STORED_SEQS_MASK], &litPtr, litEnd, prefixStart, dictStart, dictEnd);
	#if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE)
	assert(!ZSTD_isError(oneSeqSize));
	if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequences[(seqNb-ADVANCED_SEQS) & STORED_SEQS_MASK], prefixStart, dictStart);
	#endif
	if (ZSTD_isError(oneSeqSize)) return oneSeqSize;
	PREFETCH_L1(sequence.match); PREFETCH_L1(sequence.match + sequence.matchLength - 1); /* note : it's safe to invoke PREFETCH() on any memory address, including invalid ones */
	sequences[seqNb & STORED_SEQS_MASK] = sequence;
	op += oneSeqSize;
	}
	RETURN_ERROR_IF(seqNb<nbSeq, corruption_detected, "");

	/* finish queue */
	seqNb -= seqAdvance;
	for ( ; seqNb<nbSeq ; seqNb++) {
	size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequences[seqNb&STORED_SEQS_MASK], &litPtr, litEnd, prefixStart, dictStart, dictEnd);
	#if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE)
	assert(!ZSTD_isError(oneSeqSize));
	if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequences[seqNb&STORED_SEQS_MASK], prefixStart, dictStart);
	#endif
	if (ZSTD_isError(oneSeqSize)) return oneSeqSize;
	op += oneSeqSize;
	}

	/* save reps for next block */
	{ U32 i; for (i=0; i<ZSTD_REP_NUM; i++) dctx->entropy.rep[i] = (U32)(seqState.prevOffset[i]); }
	}

	/* last literal segment */
	{ size_t const lastLLSize = litEnd - litPtr;
	RETURN_ERROR_IF(lastLLSize > (size_t)(oend-op), dstSize_tooSmall, "");
	if (op != NULL) {
	memcpy(op, litPtr, lastLLSize);
	op += lastLLSize;
	}
	}

	return op-ostart;
	}

	static size_t
	ZSTD_decompressSequencesLong_default(ZSTD_DCtx* dctx,
	void* dst, size_t maxDstSize,
	const void* seqStart, size_t seqSize, int nbSeq,
	const ZSTD_longOffset_e isLongOffset,
	const int frame)
	{
	return ZSTD_decompressSequencesLong_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
	}
	#endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT */



	#if DYNAMIC_BMI2

	#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG
	static TARGET_ATTRIBUTE("bmi2") size_t
	DONT_VECTORIZE
	ZSTD_decompressSequences_bmi2(ZSTD_DCtx* dctx,
	void* dst, size_t maxDstSize,
	const void* seqStart, size_t seqSize, int nbSeq,
	const ZSTD_longOffset_e isLongOffset,
	const int frame)
	{
	return ZSTD_decompressSequences_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
	}
	#endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG */

	#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT
	static TARGET_ATTRIBUTE("bmi2") size_t
	ZSTD_decompressSequencesLong_bmi2(ZSTD_DCtx* dctx,
	void* dst, size_t maxDstSize,
	const void* seqStart, size_t seqSize, int nbSeq,
	const ZSTD_longOffset_e isLongOffset,
	const int frame)
	{
	return ZSTD_decompressSequencesLong_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
	}
	#endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT */

	#endif /* DYNAMIC_BMI2 */

	typedef size_t (*ZSTD_decompressSequences_t)(
	ZSTD_DCtx* dctx,
	void* dst, size_t maxDstSize,
	const void* seqStart, size_t seqSize, int nbSeq,
	const ZSTD_longOffset_e isLongOffset,
	const int frame);

	#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG
	static size_t
	ZSTD_decompressSequences(ZSTD_DCtx* dctx, void* dst, size_t maxDstSize,
	const void* seqStart, size_t seqSize, int nbSeq,
	const ZSTD_longOffset_e isLongOffset,
	const int frame)
	{
	DEBUGLOG(5, "ZSTD_decompressSequences");
	#if DYNAMIC_BMI2
	if (dctx->bmi2) {
	return ZSTD_decompressSequences_bmi2(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
	}
	#endif
	return ZSTD_decompressSequences_default(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
	}
	#endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG */


	#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT
	/* ZSTD_decompressSequencesLong() :
	* decompression function triggered when a minimum share of offsets is considered "long",
	* aka out of cache.
	* note : "long" definition seems overloaded here, sometimes meaning "wider than bitstream register", and sometimes meaning "farther than memory cache distance".
	* This function will try to mitigate main memory latency through the use of prefetching */
	static size_t
	ZSTD_decompressSequencesLong(ZSTD_DCtx* dctx,
	void* dst, size_t maxDstSize,
	const void* seqStart, size_t seqSize, int nbSeq,
	const ZSTD_longOffset_e isLongOffset,
	const int frame)
	{
	DEBUGLOG(5, "ZSTD_decompressSequencesLong");
	#if DYNAMIC_BMI2
	if (dctx->bmi2) {
	return ZSTD_decompressSequencesLong_bmi2(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
	}
	#endif
	return ZSTD_decompressSequencesLong_default(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
	}
	#endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT */



	#if !defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT) && \
	!defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG)
	/* ZSTD_getLongOffsetsShare() :
	* condition : offTable must be valid
	* @return : "share" of long offsets (arbitrarily defined as > (1<<23))
	* compared to maximum possible of (1<<OffFSELog) */
	static unsigned
	ZSTD_getLongOffsetsShare(const ZSTD_seqSymbol* offTable)
	{
	const void* ptr = offTable;
	U32 const tableLog = ((const ZSTD_seqSymbol_header*)ptr)[0].tableLog;
	const ZSTD_seqSymbol* table = offTable + 1;
	U32 const max = 1 << tableLog;
	U32 u, total = 0;
	DEBUGLOG(5, "ZSTD_getLongOffsetsShare: (tableLog=%u)", tableLog);

	assert(max <= (1 << OffFSELog)); /* max not too large */
	for (u=0; u<max; u++) {
	if (table[u].nbAdditionalBits > 22) total += 1;
	}

	assert(tableLog <= OffFSELog);
	total <<= (OffFSELog - tableLog); /* scale to OffFSELog */

	return total;
	}
	#endif

	size_t
	ZSTD_decompressBlock_internal(ZSTD_DCtx* dctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize, const int frame)
	{ /* blockType == blockCompressed */
	const BYTE* ip = (const BYTE*)src;
	/* isLongOffset must be true if there are long offsets.
	* Offsets are long if they are larger than 2^STREAM_ACCUMULATOR_MIN.
	* We don't expect that to be the case in 64-bit mode.
	* In block mode, window size is not known, so we have to be conservative.
	* (note: but it could be evaluated from current-lowLimit)
	*/
	ZSTD_longOffset_e const isLongOffset = (ZSTD_longOffset_e)(MEM_32bits() && (!frame \|\| (dctx->fParams.windowSize > (1ULL << STREAM_ACCUMULATOR_MIN))));
	DEBUGLOG(5, "ZSTD_decompressBlock_internal (size : %u)", (U32)srcSize);

	RETURN_ERROR_IF(srcSize >= ZSTD_BLOCKSIZE_MAX, srcSize_wrong, "");

	/* Decode literals section */
	{ size_t const litCSize = ZSTD_decodeLiteralsBlock(dctx, src, srcSize);
	DEBUGLOG(5, "ZSTD_decodeLiteralsBlock : %u", (U32)litCSize);
	if (ZSTD_isError(litCSize)) return litCSize;
	ip += litCSize;
	srcSize -= litCSize;
	}

	/* Build Decoding Tables */
	{
	/* These macros control at build-time which decompressor implementation
	* we use. If neither is defined, we do some inspection and dispatch at
	* runtime.
	*/
	#if !defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT) && \
	!defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG)
	int usePrefetchDecoder = dctx->ddictIsCold;
	#endif
	int nbSeq;
	size_t const seqHSize = ZSTD_decodeSeqHeaders(dctx, &nbSeq, ip, srcSize);
	if (ZSTD_isError(seqHSize)) return seqHSize;
	ip += seqHSize;
	srcSize -= seqHSize;

	RETURN_ERROR_IF(dst == NULL && nbSeq > 0, dstSize_tooSmall, "NULL not handled");

	#if !defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT) && \
	!defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG)
	if ( !usePrefetchDecoder
	&& (!frame \|\| (dctx->fParams.windowSize > (1<<24)))
	&& (nbSeq>ADVANCED_SEQS) ) { /* could probably use a larger nbSeq limit */
	U32 const shareLongOffsets = ZSTD_getLongOffsetsShare(dctx->OFTptr);
	U32 const minShare = MEM_64bits() ? 7 : 20; /* heuristic values, correspond to 2.73% and 7.81% */
	usePrefetchDecoder = (shareLongOffsets >= minShare);
	}
	#endif

	dctx->ddictIsCold = 0;

	#if !defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT) && \
	!defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG)
	if (usePrefetchDecoder)
	#endif
	#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT
	return ZSTD_decompressSequencesLong(dctx, dst, dstCapacity, ip, srcSize, nbSeq, isLongOffset, frame);
	#endif

	#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG
	/* else */
	return ZSTD_decompressSequences(dctx, dst, dstCapacity, ip, srcSize, nbSeq, isLongOffset, frame);
	#endif
	}
	}


	void ZSTD_checkContinuity(ZSTD_DCtx* dctx, const void* dst)
	{
	if (dst != dctx->previousDstEnd) { /* not contiguous */
	dctx->dictEnd = dctx->previousDstEnd;
	dctx->virtualStart = (const char)dst - ((const char)(dctx->previousDstEnd) - (const char*)(dctx->prefixStart));
	dctx->prefixStart = dst;
	dctx->previousDstEnd = dst;
	}
	}


	size_t ZSTD_decompressBlock(ZSTD_DCtx* dctx,
	void* dst, size_t dstCapacity,
	const void* src, size_t srcSize)
	{
	size_t dSize;
	ZSTD_checkContinuity(dctx, dst);
	dSize = ZSTD_decompressBlock_internal(dctx, dst, dstCapacity, src, srcSize, /* frame */ 0);
	dctx->previousDstEnd = (char*)dst + dSize;
	return dSize;
	}
	/** ended inlining decompress/zstd_decompress_block.c **/
	Index: vendor-sys/openzfs/dist/rpm/generic/zfs-dkms.spec.in
	===================================================================
	--- vendor-sys/openzfs/dist/rpm/generic/zfs-dkms.spec.in (revision 366773)
	+++ vendor-sys/openzfs/dist/rpm/generic/zfs-dkms.spec.in (revision 366774)
	@@ -1,102 +1,102 @@
	%{?!packager: %define packager Brian Behlendorf <behlendorf1@llnl.gov>}

	%if ! 0%{?rhel}%{?fedora}%{?mageia}%{?suse_version}
	%define not_rpm 1
	%endif

	# Exclude input files from mangling
	%global __brp_mangle_shebangs_exclude_from ^/usr/src/.*$

	%define module @PACKAGE@
	%define mkconf scripts/dkms.mkconf

	Name: %{module}-dkms

	Version: @VERSION@
	Release: @RELEASE@%{?dist}
	Summary: Kernel module(s) (dkms)

	Group: System Environment/Kernel
	License: @ZFS_META_LICENSE@
	-URL: https://zfsonlinux.org/
	+URL: https://github.com/openzfs/zfs
	Source0: %{module}-%{version}.tar.gz
	BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
	BuildArch: noarch

	Requires: dkms >= 2.2.0.3
	Requires: gcc, make, perl, diffutils
	%if 0%{?rhel}%{?fedora}%{?mageia}%{?suse_version}
	Requires: kernel-devel
	Obsoletes: spl-dkms
	%endif
	Provides: %{module}-kmod = %{version}
	AutoReqProv: no

	%description
	This package contains the dkms ZFS kernel modules.

	%prep
	%setup -q -n %{module}-%{version}

	%build
	%{mkconf} -n %{module} -v %{version} -f dkms.conf

	%install
	if [ "$RPM_BUILD_ROOT" != "/" ]; then
	rm -rf $RPM_BUILD_ROOT
	fi
	mkdir -p $RPM_BUILD_ROOT/usr/src/
	cp -rf ${RPM_BUILD_DIR}/%{module}-%{version} $RPM_BUILD_ROOT/usr/src/

	%clean
	if [ "$RPM_BUILD_ROOT" != "/" ]; then
	rm -rf $RPM_BUILD_ROOT
	fi

	%files
	%defattr(-,root,root)
	/usr/src/%{module}-%{version}

	%post
	for POSTINST in /usr/lib/dkms/common.postinst; do
	if [ -f $POSTINST ]; then
	$POSTINST %{module} %{version}
	exit $?
	fi
	echo "WARNING: $POSTINST does not exist."
	done
	echo -e "ERROR: DKMS version is too old and %{module} was not"
	echo -e "built with legacy DKMS support."
	echo -e "You must either rebuild %{module} with legacy postinst"
	echo -e "support or upgrade DKMS to a more current version."
	exit 1

	%preun
	# Are we doing an upgrade?
	if [ "$1" = "1" -o "$1" = "upgrade" ] ; then
	# Yes we are. Are we upgrading to a new ZFS version?
	NEWEST_VER=$(dkms status zfs \| sed 's/,//g' \| sort -r -V \| awk '/installed/{print $2; exit}')
	if [ "$NEWEST_VER" != "%{version}" ] ; then
	# Yes, it's a new ZFS version. We'll uninstall the old module
	# later on in this script.
	true
	else
	# No, it's probably an upgrade of the same ZFS version
	# to a new distro (zfs-dkms-0.7.12.fc28->zfs-dkms-0.7.12.fc29).
	# Don't remove our modules, since the rebuild for the new
	# distro will automatically delete the old modules.
	exit 0
	fi
	fi

	# If we're here then we're doing an uninstall (not upgrade).
	CONFIG_H="/var/lib/dkms/%{module}/%{version}///%{module}_config.h"
	SPEC_META_ALIAS="@PACKAGE@-@VERSION@-@RELEASE@"
	DKMS_META_ALIAS=`cat $CONFIG_H 2>/dev/null \|
	awk -F'"' '/META_ALIAS\s+"/ { print $2; exit 0 }'`
	if [ "$SPEC_META_ALIAS" = "$DKMS_META_ALIAS" ]; then
	echo -e
	echo -e "Uninstall of %{module} module ($SPEC_META_ALIAS) beginning:"
	dkms remove -m %{module} -v %{version} --all %{!?not_rpm:--rpm_safe_upgrade}
	fi
	exit 0
	Index: vendor-sys/openzfs/dist/rpm/generic/zfs-kmod.spec.in
	===================================================================
	--- vendor-sys/openzfs/dist/rpm/generic/zfs-kmod.spec.in (revision 366773)
	+++ vendor-sys/openzfs/dist/rpm/generic/zfs-kmod.spec.in (revision 366774)
	@@ -1,164 +1,164 @@
	%define module @PACKAGE@

	%if !%{defined ksrc}
	%if 0%{?rhel}%{?fedora}
	%define ksrc ${kernel_version##*___}
	%else
	%define ksrc "$( \
	if [ -e "/usr/src/linux-${kernel_version%%___*}" ]; then \
	echo "/usr/src/linux-${kernel_version%%___*}"; \
	elif [ -e "/lib/modules/${kernel_version%%___*}/source" ]; then \
	echo "/lib/modules/${kernel_version%%___*}/source"; \
	else \
	echo "/lib/modules/${kernel_version%%___*}/build"; \
	fi)"
	%endif
	%endif

	%if !%{defined kobj}
	%if 0%{?rhel}%{?fedora}
	%define kobj ${kernel_version##*___}
	%else
	%define kobj "$( \
	if [ -e "/usr/src/linux-${kernel_version%%___*}" ]; then \
	echo "/usr/src/linux-${kernel_version%%___*}"; \
	else \
	echo "/lib/modules/${kernel_version%%___*}/build"; \
	fi)"
	%endif
	%endif

	#define repo rpmfusion
	#define repo chaos

	# (un)define the next line to either build for the newest or all current kernels
	%define buildforkernels newest
	#define buildforkernels current
	#define buildforkernels akmod

	%bcond_with debug
	%bcond_with debuginfo


	Name: %{module}-kmod

	Version: @VERSION@
	Release: @RELEASE@%{?dist}
	Summary: Kernel module(s)

	Group: System Environment/Kernel
	License: @ZFS_META_LICENSE@
	-URL: https://zfsonlinux.org/
	+URL: https://github.com/openzfs/zfs
	Source0: %{module}-%{version}.tar.gz
	Source10: kmodtool
	BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id} -u -n)
	%if 0%{?rhel}%{?fedora}
	BuildRequires: gcc, make
	BuildRequires: elfutils-libelf-devel
	%endif

	# The developments headers will conflict with the dkms packages.
	Conflicts: %{module}-dkms

	%if %{defined repo}

	# Building for a repository use the proper build-sysbuild package
	# to determine which kernel-devel packages should be installed.
	BuildRequires: %{_bindir}/kmodtool
	%{!?kernels:BuildRequires: buildsys-build-%{repo}-kerneldevpkgs-%{?buildforkernels:%{buildforkernels}}%{!?buildforkernels:current}-%{_target_cpu}}

	%else

	# Building local packages attempt to to use the installed kernel.
	%{?rhel:BuildRequires: kernel-devel}
	%{?fedora:BuildRequires: kernel-devel}
	%{?suse_version:BuildRequires: kernel-source}

	%if !%{defined kernels} && !%{defined build_src_rpm}
	%if 0%{?rhel}%{?fedora}%{?suse_version}
	%define kernels %(ls -1 /usr/src/kernels)
	%else
	%define kernels %(ls -1 /lib/modules)
	%endif
	%endif
	%endif

	# LDFLAGS are not sanitized by arch/*/Makefile for these architectures.
	%ifarch ppc ppc64 ppc64le aarch64
	%global __global_ldflags %{nil}
	%endif

	%if 0%{?fedora} >= 17
	%define prefix /usr
	%endif

	# Kmodtool does its magic here. A patched version of kmodtool is shipped
	# with the source rpm until kmod development packages are supported upstream.
	# https://bugzilla.rpmfusion.org/show_bug.cgi?id=2714
	%{expand:%(bash %{SOURCE10} --target %{_target_cpu} %{?repo:--repo %{?repo}} --kmodname %{name} %{?buildforkernels:--%{buildforkernels}} --devel %{?prefix:--prefix "%{?prefix}"} %{?kernels:--for-kernels "%{?kernels}"} %{?kernelbuildroot:--buildroot "%{?kernelbuildroot}"} --obsolete-name spl --obsolete-version 0.8 2>/dev/null) }


	%description
	This package contains the ZFS kernel modules.

	%prep
	# Error out if there was something wrong with kmodtool.
	%{?kmodtool_check}

	# Print kmodtool output for debugging purposes:
	bash %{SOURCE10} --target %{_target_cpu} %{?repo:--repo %{?repo}} --kmodname %{name} %{?buildforkernels:--%{buildforkernels}} --devel %{?prefix:--prefix "%{?prefix}"} %{?kernels:--for-kernels "%{?kernels}"} %{?kernelbuildroot:--buildroot "%{?kernelbuildroot}"} --obsolete-name spl --obsolete-version 0.8 2>/dev/null

	%if %{with debug}
	%define debug --enable-debug
	%else
	%define debug --disable-debug
	%endif

	%if %{with debuginfo}
	%define debuginfo --enable-debuginfo
	%else
	%define debuginfo --disable-debuginfo
	%endif

	# Leverage VPATH from configure to avoid making multiple copies.
	%define _configure ../%{module}-%{version}/configure

	%setup -q -c -T -a 0

	for kernel_version in %{?kernel_versions}; do
	%{__mkdir} _kmod_build_${kernel_version%%___*}
	done

	%build
	for kernel_version in %{?kernel_versions}; do
	cd _kmod_build_${kernel_version%%___*}
	%configure \
	--with-config=kernel \
	--with-linux=%{ksrc} \
	--with-linux-obj=%{kobj} \
	%{debug} \
	%{debuginfo}
	make %{?_smp_mflags}
	cd ..
	done


	%install
	rm -rf ${RPM_BUILD_ROOT}

	# Relies on the kernel 'modules_install' make target.
	for kernel_version in %{?kernel_versions}; do
	cd _kmod_build_${kernel_version%%___*}
	make install \
	DESTDIR=${RPM_BUILD_ROOT} \
	%{?prefix:INSTALL_MOD_PATH=%{?prefix}} \
	INSTALL_MOD_DIR=%{kmodinstdir_postfix}
	cd ..
	done
	# find-debuginfo.sh only considers executables
	chmod u+x ${RPM_BUILD_ROOT}%{kmodinstdir_prefix}//extra///
	%{?akmod_install}


	%clean
	rm -rf $RPM_BUILD_ROOT
	Index: vendor-sys/openzfs/dist/rpm/generic/zfs.spec.in
	===================================================================
	--- vendor-sys/openzfs/dist/rpm/generic/zfs.spec.in (revision 366773)
	+++ vendor-sys/openzfs/dist/rpm/generic/zfs.spec.in (revision 366774)
	@@ -1,524 +1,524 @@
	%global _sbindir /sbin
	%global _libdir /%{_lib}

	# Set the default udev directory based on distribution.
	%if %{undefined _udevdir}
	%if 0%{?fedora} >= 17 \|\| 0%{?rhel} >= 7 \|\| 0%{?centos} >= 7
	%global _udevdir %{_prefix}/lib/udev
	%else
	%global _udevdir /lib/udev
	%endif
	%endif

	# Set the default udevrule directory based on distribution.
	%if %{undefined _udevruledir}
	%if 0%{?fedora} >= 17 \|\| 0%{?rhel} >= 7 \|\| 0%{?centos} >= 7
	%global _udevruledir %{_prefix}/lib/udev/rules.d
	%else
	%global _udevruledir /lib/udev/rules.d
	%endif
	%endif

	# Set the default dracut directory based on distribution.
	%if %{undefined _dracutdir}
	%if 0%{?fedora} >= 17 \|\| 0%{?rhel} >= 7 \|\| 0%{?centos} >= 7
	%global _dracutdir %{_prefix}/lib/dracut
	%else
	%global _dracutdir %{_prefix}/share/dracut
	%endif
	%endif

	%if %{undefined _initconfdir}
	%global _initconfdir /etc/sysconfig
	%endif

	%if %{undefined _unitdir}
	%global _unitdir %{_prefix}/lib/systemd/system
	%endif

	%if %{undefined _presetdir}
	%global _presetdir %{_prefix}/lib/systemd/system-preset
	%endif

	%if %{undefined _modulesloaddir}
	%global _modulesloaddir %{_prefix}/lib/modules-load.d
	%endif

	%if %{undefined _systemdgeneratordir}
	%global _systemdgeneratordir %{_prefix}/lib/systemd/system-generators
	%endif

	%if %{undefined _pkgconfigdir}
	%global _pkgconfigdir %{_prefix}/%{_lib}/pkgconfig
	%endif

	%bcond_with debug
	%bcond_with debuginfo
	%bcond_with asan
	%bcond_with systemd
	%bcond_with pam

	# Generic enable switch for systemd
	%if %{with systemd}
	%define _systemd 1
	%endif

	# RHEL >= 7 comes with systemd
	%if 0%{?rhel} >= 7
	%define _systemd 1
	%endif

	# Fedora >= 15 comes with systemd, but only >= 18 has
	# the proper macros
	%if 0%{?fedora} >= 18
	%define _systemd 1
	%endif

	# opensuse >= 12.1 comes with systemd, but only >= 13.1
	# has the proper macros
	%if 0%{?suse_version} >= 1310
	%define _systemd 1
	%endif

	# When not specified default to distribution provided version. This
	# is normally Python 3, but for RHEL <= 7 only Python 2 is provided.
	%if %{undefined __use_python}
	%if 0%{?rhel} && 0%{?rhel} <= 7
	%define __python /usr/bin/python2
	%define __python_pkg_version 2
	%define __python_cffi_pkg python-cffi
	%define __python_setuptools_pkg python-setuptools
	%else
	%define __python /usr/bin/python3
	%define __python_pkg_version 3
	%define __python_cffi_pkg python3-cffi
	%define __python_setuptools_pkg python3-setuptools
	%endif
	%else
	%define __python %{__use_python}
	%define __python_pkg_version %{__use_python_pkg_version}
	%define __python_cffi_pkg python%{__python_pkg_version}-cffi
	%define __python_setuptools_pkg python%{__python_pkg_version}-setuptools
	%endif
	%define __python_sitelib %(%{__python} -Esc "from distutils.sysconfig import get_python_lib; print(get_python_lib())")

	# By default python-pyzfs is enabled, with the exception of
	# RHEL 6 which by default uses Python 2.6 which is too old.
	%if 0%{?rhel} == 6
	%bcond_with pyzfs
	%else
	%bcond_without pyzfs
	%endif

	Name: @PACKAGE@
	Version: @VERSION@
	Release: @RELEASE@%{?dist}
	Summary: Commands to control the kernel modules and libraries

	Group: System Environment/Kernel
	License: @ZFS_META_LICENSE@
	-URL: https://zfsonlinux.org/
	+URL: https://github.com/openzfs/zfs
	Source0: %{name}-%{version}.tar.gz
	BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
	Requires: libzpool2 = %{version}
	Requires: libnvpair1 = %{version}
	Requires: libuutil1 = %{version}
	Requires: libzfs2 = %{version}
	Requires: %{name}-kmod = %{version}
	Provides: %{name}-kmod-common = %{version}
	Obsoletes: spl

	# zfs-fuse provides the same commands and man pages that ZoL does. Renaming
	# those on either side would conflict with all available documentation.
	Conflicts: zfs-fuse

	%if 0%{?rhel}%{?fedora}%{?suse_version}
	BuildRequires: gcc, make
	BuildRequires: zlib-devel
	BuildRequires: libuuid-devel
	BuildRequires: libblkid-devel
	BuildRequires: libudev-devel
	BuildRequires: libattr-devel
	BuildRequires: openssl-devel
	%if 0%{?fedora} >= 28 \|\| 0%{?rhel} >= 8 \|\| 0%{?centos} >= 8
	BuildRequires: libtirpc-devel
	%endif
	Requires: openssl
	%if 0%{?_systemd}
	BuildRequires: systemd
	%endif
	%endif

	%if 0%{?_systemd}
	Requires(post): systemd
	Requires(preun): systemd
	Requires(postun): systemd
	%endif

	# The zpool iostat/status -c scripts call some utilities like lsblk and iostat
	Requires: util-linux
	Requires: sysstat

	%description
	This package contains the core ZFS command line utilities.

	%package -n libzpool2
	Summary: Native ZFS pool library for Linux
	Group: System Environment/Kernel

	%description -n libzpool2
	This package contains the zpool library, which provides support
	for managing zpools

	%post -n libzpool2 -p /sbin/ldconfig
	%postun -n libzpool2 -p /sbin/ldconfig

	%package -n libnvpair1
	Summary: Solaris name-value library for Linux
	Group: System Environment/Kernel

	%description -n libnvpair1
	This package contains routines for packing and unpacking name-value
	pairs. This functionality is used to portably transport data across
	process boundaries, between kernel and user space, and can be used
	to write self describing data structures on disk.

	%post -n libnvpair1 -p /sbin/ldconfig
	%postun -n libnvpair1 -p /sbin/ldconfig

	%package -n libuutil1
	Summary: Solaris userland utility library for Linux
	Group: System Environment/Kernel

	%description -n libuutil1
	-This library provides a variety of compatibility functions for ZFS on Linux:
	+This library provides a variety of compatibility functions for OpenZFS:
	* libspl: The Solaris Porting Layer userland library, which provides APIs
	that make it possible to run Solaris user code in a Linux environment
	with relatively minimal modification.
	* libavl: The Adelson-Velskii Landis balanced binary tree manipulation
	library.
	* libefi: The Extensible Firmware Interface library for GUID disk
	partitioning.
	* libshare: NFS, SMB, and iSCSI service integration for ZFS.

	%post -n libuutil1 -p /sbin/ldconfig
	%postun -n libuutil1 -p /sbin/ldconfig

	%package -n libzfs2
	Summary: Native ZFS filesystem library for Linux
	Group: System Environment/Kernel

	%description -n libzfs2
	This package provides support for managing ZFS filesystems

	%post -n libzfs2 -p /sbin/ldconfig
	%postun -n libzfs2 -p /sbin/ldconfig

	%package -n libzfs2-devel
	Summary: Development headers
	Group: System Environment/Kernel
	Requires: libzfs2 = %{version}
	Requires: libzpool2 = %{version}
	Requires: libnvpair1 = %{version}
	Requires: libuutil1 = %{version}
	Provides: libzpool2-devel
	Provides: libnvpair1-devel
	Provides: libuutil1-devel
	Obsoletes: zfs-devel

	%description -n libzfs2-devel
	This package contains the header files needed for building additional
	applications against the ZFS libraries.

	%package test
	Summary: Test infrastructure
	Group: System Environment/Kernel
	Requires: %{name}%{?_isa} = %{version}-%{release}
	Requires: parted
	Requires: lsscsi
	Requires: mdadm
	Requires: bc
	Requires: ksh
	Requires: fio
	Requires: acl
	Requires: sudo
	Requires: sysstat
	Requires: libaio
	Requires: python%{__python_pkg_version}
	%if 0%{?rhel}%{?fedora}%{?suse_version}
	BuildRequires: libaio-devel
	%endif
	AutoReqProv: no

	%description test
	This package contains test infrastructure and support scripts for
	validating the file system.

	%package dracut
	Summary: Dracut module
	Group: System Environment/Kernel
	BuildArch: noarch
	Requires: %{name} >= %{version}
	Requires: dracut
	Requires: /usr/bin/awk
	Requires: grep

	%description dracut
	This package contains a dracut module used to construct an initramfs
	image which is ZFS aware.

	%if %{with pyzfs}
	%package -n python%{__python_pkg_version}-pyzfs
	Summary: Python %{python_version} wrapper for libzfs_core
	Group: Development/Languages/Python
	License: Apache-2.0
	BuildArch: noarch
	Requires: libzfs2 = %{version}
	Requires: libnvpair1 = %{version}
	Requires: libffi
	Requires: python%{__python_pkg_version}
	Requires: %{__python_cffi_pkg}
	%if 0%{?rhel}%{?fedora}%{?suse_version}
	BuildRequires: python%{__python_pkg_version}-devel
	BuildRequires: %{__python_cffi_pkg}
	BuildRequires: %{__python_setuptools_pkg}
	BuildRequires: libffi-devel
	%endif

	%description -n python%{__python_pkg_version}-pyzfs
	This package provides a python wrapper for the libzfs_core C library.
	%endif

	%if 0%{?_initramfs}
	%package initramfs
	Summary: Initramfs module
	Group: System Environment/Kernel
	Requires: %{name}%{?_isa} = %{version}-%{release}
	Requires: %{name} = %{version}-%{release}
	Requires: initramfs-tools

	%description initramfs
	This package contains a initramfs module used to construct an initramfs
	image which is ZFS aware.
	%endif

	%prep
	%if %{with debug}
	%define debug --enable-debug
	%else
	%define debug --disable-debug
	%endif

	%if %{with debuginfo}
	%define debuginfo --enable-debuginfo
	%else
	%define debuginfo --disable-debuginfo
	%endif

	%if %{with asan}
	%define asan --enable-asan
	%else
	%define asan --disable-asan
	%endif

	%if 0%{?_systemd}
	%define systemd --enable-systemd --with-systemdunitdir=%{_unitdir} --with-systemdpresetdir=%{_presetdir} --with-systemdmodulesloaddir=%{_modulesloaddir} --with-systemdgeneratordir=%{_systemdgeneratordir} --disable-sysvinit
	%define systemd_svcs zfs-import-cache.service zfs-import-scan.service zfs-mount.service zfs-share.service zfs-zed.service zfs.target zfs-import.target zfs-volume-wait.service zfs-volumes.target
	%else
	%define systemd --enable-sysvinit --disable-systemd
	%endif

	%if %{with pyzfs}
	%define pyzfs --enable-pyzfs
	%else
	%define pyzfs --disable-pyzfs
	%endif

	%if %{with pam}
	%define pam --enable-pam
	%else
	%define pam --disable-pam
	%endif

	%setup -q

	%build
	%configure \
	--with-config=user \
	--with-udevdir=%{_udevdir} \
	--with-udevruledir=%{_udevruledir} \
	--with-dracutdir=%{_dracutdir} \
	--with-pamconfigsdir=%{_datadir}/pam-configs \
	--with-pammoduledir=%{_libdir}/security \
	--with-python=%{__python} \
	--with-pkgconfigdir=%{_pkgconfigdir} \
	--disable-static \
	%{debug} \
	%{debuginfo} \
	%{asan} \
	%{systemd} \
	%{pam} \
	%{pyzfs}
	make %{?_smp_mflags}

	%install
	%{__rm} -rf $RPM_BUILD_ROOT
	make install DESTDIR=%{?buildroot}
	find %{?buildroot}%{_libdir} -name '*.la' -exec rm -f {} \;
	%if 0%{!?__brp_mangle_shebangs:1}
	find %{?buildroot}%{_bindir} \
	$ -name arc_summary -or -name arcstat -or -name dbufstat $ \
	-exec %{__sed} -i 's\|^#!.*\|#!%{__python}\|' {} \;
	find %{?buildroot}%{_datadir} \
	$ -name test-runner.py -or -name zts-report.py $ \
	-exec %{__sed} -i 's\|^#!.*\|#!%{__python}\|' {} \;
	%endif

	%post
	%if 0%{?_systemd}
	%if 0%{?systemd_post:1}
	%systemd_post %{systemd_svcs}
	%else
	if [ "$1" = "1" -o "$1" = "install" ] ; then
	# Initial installation
	systemctl preset %{systemd_svcs} >/dev/null \|\| true
	fi
	%endif
	%else
	if [ -x /sbin/chkconfig ]; then
	/sbin/chkconfig --add zfs-import
	/sbin/chkconfig --add zfs-mount
	/sbin/chkconfig --add zfs-share
	/sbin/chkconfig --add zfs-zed
	fi
	%endif
	exit 0

	# On RHEL/CentOS 7 the static nodes aren't refreshed by default after
	# installing a package. This is the default behavior for Fedora.
	%posttrans
	%if 0%{?rhel} == 7 \|\| 0%{?centos} == 7
	systemctl restart kmod-static-nodes
	systemctl restart systemd-tmpfiles-setup-dev
	udevadm trigger
	%endif

	%preun
	%if 0%{?_systemd}
	%if 0%{?systemd_preun:1}
	%systemd_preun %{systemd_svcs}
	%else
	if [ "$1" = "0" -o "$1" = "remove" ] ; then
	# Package removal, not upgrade
	systemctl --no-reload disable %{systemd_svcs} >/dev/null \|\| true
	systemctl stop %{systemd_svcs} >/dev/null \|\| true
	fi
	%endif
	%else
	if [ "$1" = "0" -o "$1" = "remove" ] && [ -x /sbin/chkconfig ]; then
	/sbin/chkconfig --del zfs-import
	/sbin/chkconfig --del zfs-mount
	/sbin/chkconfig --del zfs-share
	/sbin/chkconfig --del zfs-zed
	fi
	%endif
	exit 0

	%postun
	%if 0%{?_systemd}
	%if 0%{?systemd_postun:1}
	%systemd_postun %{systemd_svcs}
	%else
	systemctl --system daemon-reload >/dev/null \|\| true
	%endif
	%endif

	%files
	# Core utilities
	%{_sbindir}/*
	%{_bindir}/raidz_test
	%{_bindir}/zgenhostid
	%{_bindir}/zvol_wait
	# Optional Python 2/3 scripts
	%{_bindir}/arc_summary
	%{_bindir}/arcstat
	%{_bindir}/dbufstat
	# Man pages
	%{_mandir}/man1/*
	%{_mandir}/man5/*
	%{_mandir}/man8/*
	# Configuration files and scripts
	%{_libexecdir}/%{name}
	%{_udevdir}/vdev_id
	%{_udevdir}/zvol_id
	%{_udevdir}/rules.d/*
	%if ! 0%{?_systemd} \|\| 0%{?_initramfs}
	# Files needed for sysvinit and initramfs-tools
	%{_sysconfdir}/%{name}/zfs-functions
	%config(noreplace) %{_initconfdir}/zfs
	%else
	%exclude %{_sysconfdir}/%{name}/zfs-functions
	%exclude %{_initconfdir}/zfs
	%endif
	%if 0%{?_systemd}
	%{_unitdir}/*
	%{_presetdir}/*
	%{_modulesloaddir}/*
	%{_systemdgeneratordir}/*
	%else
	%config(noreplace) %{_sysconfdir}/init.d/*
	%endif
	%config(noreplace) %{_sysconfdir}/%{name}/zed.d/*
	%config(noreplace) %{_sysconfdir}/%{name}/zpool.d/*
	%config(noreplace) %{_sysconfdir}/%{name}/vdev_id.conf.*.example
	%attr(440, root, root) %config(noreplace) %{_sysconfdir}/sudoers.d/*
	%if %{with pam}
	%{_libdir}/security/*
	%{_datadir}/pam-configs/*
	%endif

	%files -n libzpool2
	%{_libdir}/libzpool.so.*

	%files -n libnvpair1
	%{_libdir}/libnvpair.so.*

	%files -n libuutil1
	%{_libdir}/libuutil.so.*

	%files -n libzfs2
	%{_libdir}/libzfs.so.

	%files -n libzfs2-devel
	%{_pkgconfigdir}/libzfs.pc
	%{_pkgconfigdir}/libzfsbootenv.pc
	%{_pkgconfigdir}/libzfs_core.pc
	%{_libdir}/*.so
	%{_includedir}/*
	%doc AUTHORS COPYRIGHT LICENSE NOTICE README.md

	%files test
	%{_datadir}/%{name}

	%files dracut
	%doc contrib/dracut/README.dracut.markdown
	%{_dracutdir}/modules.d/*

	%if %{with pyzfs}
	%files -n python%{__python_pkg_version}-pyzfs
	%doc contrib/pyzfs/README
	%doc contrib/pyzfs/LICENSE
	%defattr(-,root,root,-)
	%{__python_sitelib}/libzfs_core/*
	%{__python_sitelib}/pyzfs*
	%endif

	%if 0%{?_initramfs}
	%files initramfs
	%doc contrib/initramfs/README.initramfs.markdown
	/usr/share/initramfs-tools/*
	%else
	# Since we're not building the initramfs package,
	# ignore those files.
	%exclude /usr/share/initramfs-tools
	%endif
	Index: vendor-sys/openzfs/dist/rpm/redhat/zfs-kmod.spec.in
	===================================================================
	--- vendor-sys/openzfs/dist/rpm/redhat/zfs-kmod.spec.in (revision 366773)
	+++ vendor-sys/openzfs/dist/rpm/redhat/zfs-kmod.spec.in (revision 366774)
	@@ -1,88 +1,88 @@
	%bcond_with debug
	%bcond_with debuginfo

	Name: @PACKAGE@-kmod
	Version: @VERSION@
	Release: @RELEASE@%{?dist}

	Summary: Kernel module(s)
	Group: System Environment/Kernel
	License: @ZFS_META_LICENSE@
	-URL: https://zfsonlinux.org/
	+URL: https://github.com/openzfs/zfs
	BuildRequires: %kernel_module_package_buildreqs
	Source0: @PACKAGE@-%{version}.tar.gz
	BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)

	# Additional dependency information for the kmod sub-package must be specified
	# by generating a preamble text file which kmodtool can append to the spec file.
	%(/bin/echo -e "\
	Requires: @PACKAGE@ = %{version}\n\
	Conflicts: @PACKAGE@-dkms\n\n" > %{_sourcedir}/kmod-preamble\n\
	Obsoletes: spl-kmod)

	# LDFLAGS are not sanitized by arch/*/Makefile for these architectures.
	%ifarch ppc ppc64 ppc64le aarch64
	%global __global_ldflags %{nil}
	%endif

	%description
	This package contains the ZFS kernel modules.

	%define kmod_name @PACKAGE@

	%kernel_module_package -n %{kmod_name} -p %{_sourcedir}/kmod-preamble

	%define ksrc %{_usrsrc}/kernels/%{kverrel}
	%define kobj %{ksrc}

	%package -n kmod-%{kmod_name}-devel
	Summary: ZFS kernel module(s) devel common
	Group: System Environment/Kernel
	Provides: kmod-spl-devel = %{version}

	%description -n kmod-%{kmod_name}-devel
	This package provides the header files and objects to build kernel modules.

	%prep
	if ! [ -d "%{ksrc}" ]; then
	echo "Kernel build directory isn't set properly, cannot continue"
	exit 1
	fi

	%if %{with debug}
	%define debug --enable-debug
	%else
	%define debug --disable-debug
	%endif

	%if %{with debuginfo}
	%define debuginfo --enable-debuginfo
	%else
	%define debuginfo --disable-debuginfo
	%endif

	%setup -n %{kmod_name}-%{version}
	%build
	%configure \
	--with-config=kernel \
	--with-linux=%{ksrc} \
	--with-linux-obj=%{kobj} \
	%{debug} \
	%{debuginfo}
	make %{?_smp_mflags}

	%install
	make install \
	DESTDIR=${RPM_BUILD_ROOT} \
	INSTALL_MOD_DIR=extra/%{kmod_name}
	%{__rm} -f %{buildroot}/lib/modules/%{kverrel}/modules.*

	# find-debuginfo.sh only considers executables
	%{__chmod} u+x %{buildroot}/lib/modules/%{kverrel}/extra///*

	%clean
	rm -rf $RPM_BUILD_ROOT

	%files -n kmod-%{kmod_name}-devel
	%{_usrsrc}/%{kmod_name}-%{version}
	%{_usrsrc}/spl-%{version}
	Index: vendor-sys/openzfs/dist/scripts/zimport.sh
	===================================================================
	--- vendor-sys/openzfs/dist/scripts/zimport.sh (revision 366773)
	+++ vendor-sys/openzfs/dist/scripts/zimport.sh (revision 366774)
	@@ -1,517 +1,517 @@
	#!/usr/bin/env bash
	#
	# Verify that an assortment of known good reference pools can be imported
	# using different versions of the ZoL code.
	#
	# By default references pools for the major ZFS implementation will be
	# checked against the most recent ZoL tags and the master development branch.
	# Alternate tags or branches may be verified with the '-s <src-tag> option.
	# Passing the keyword "installed" will instruct the script to test whatever
	# version is installed.
	#
	# Preferentially a reference pool is used for all tests. However, if one
	# does not exist and the pool-tag matches one of the src-tags then a new
	# reference pool will be created using binaries from that source build.
	# This is particularly useful when you need to test your changes before
	# opening a pull request. The keyword 'all' can be used as short hand
	# refer to all available reference pools.
	#
	# New reference pools may be added by placing a bzip2 compressed tarball
	# of the pool in the scripts/zfs-images directory and then passing
	# the -p <pool-tag> option. To increase the test coverage reference pools
	# should be collected for all the major ZFS implementations. Having these
	# pools easily available is also helpful to the developers.
	#
	# Care should be taken to run these tests with a kernel supported by all
	# the listed tags. Otherwise build failure will cause false positives.
	#
	#
	# EXAMPLES:
	#
	# The following example will verify the zfs-0.6.2 tag, the master branch,
	# and the installed zfs version can correctly import the listed pools.
	# Note there is no reference pool available for master and installed but
	# because binaries are available one is automatically constructed. The
	# working directory is also preserved between runs (-k) preventing the
	# need to rebuild from source for multiple runs.
	#
	# zimport.sh -k -f /var/tmp/zimport \
	# -s "zfs-0.6.2 master installed" \
	# -p "zevo-1.1.1 zol-0.6.2 zol-0.6.2-173 master installed"
	#
	-# --------------------- ZFS on Linux Source Versions --------------
	+# ------------------------ OpenZFS Source Versions ----------------
	# zfs-0.6.2 master 0.6.2-175_g36eb554
	# -----------------------------------------------------------------
	# Clone ZFS Local Local Skip
	# Build ZFS Pass Pass Skip
	# -----------------------------------------------------------------
	# zevo-1.1.1 Pass Pass Pass
	# zol-0.6.2 Pass Pass Pass
	# zol-0.6.2-173 Fail Pass Pass
	# master Pass Pass Pass
	# installed Pass Pass Pass
	#

	BASE_DIR=$(dirname "$0")
	SCRIPT_COMMON=common.sh
	if [ -f "${BASE_DIR}/${SCRIPT_COMMON}" ]; then
	. "${BASE_DIR}/${SCRIPT_COMMON}"
	else
	echo "Missing helper script ${SCRIPT_COMMON}" && exit 1
	fi

	PROG=zimport.sh
	SRC_TAGS="zfs-0.6.5.11 master"
	POOL_TAGS="all master"
	POOL_CREATE_OPTIONS=
	TEST_DIR=$(mktemp -u -d -p /var/tmp zimport.XXXXXXXX)
	KEEP="no"
	VERBOSE="no"
	COLOR="yes"
	-REPO="https://github.com/zfsonlinux"
	+REPO="https://github.com/openzfs"
	IMAGES_DIR="$SCRIPTDIR/zfs-images/"
	-IMAGES_TAR="https://github.com/zfsonlinux/zfs-images/tarball/master"
	+IMAGES_TAR="https://github.com/openzfs/zfs-images/tarball/master"
	ERROR=0

	CONFIG_LOG="configure.log"
	CONFIG_OPTIONS=${CONFIG_OPTIONS:-""}
	MAKE_LOG="make.log"
	MAKE_OPTIONS=${MAKE_OPTIONS:-"-s -j$(nproc)"}

	COLOR_GREEN="\033[0;32m"
	COLOR_RED="\033[0;31m"
	COLOR_BROWN="\033[0;33m"
	COLOR_RESET="\033[0m"

	usage() {
	cat << EOF
	USAGE:
	zimport.sh [hvl] [-r repo] [-s src-tag] [-i pool-dir] [-p pool-tag]
	[-f path] [-o options]

	DESCRIPTION:
	ZPOOL import verification tests

	OPTIONS:
	-h Show this message
	-v Verbose
	-c No color
	-k Keep temporary directory
	-r <repo> Source repository ($REPO)
	-s <src-tag>... Verify ZoL versions with the listed tags
	-i <pool-dir> Pool image directory
	-p <pool-tag>... Verify pools created with the listed tags
	-f <path> Temporary directory to use
	-o <options> Additional options to pass to 'zpool create'

	EOF
	}

	while getopts 'hvckr:s:i:p:f:o:?' OPTION; do
	case $OPTION in
	h)
	usage
	exit 1
	;;
	v)
	VERBOSE="yes"
	;;
	c)
	COLOR="no"
	;;
	k)
	KEEP="yes"
	;;
	r)
	REPO="$OPTARG"
	;;
	s)
	SRC_TAGS="$OPTARG"
	;;
	i)
	IMAGES_DIR="$OPTARG"
	;;
	p)
	POOL_TAGS="$OPTARG"
	;;
	f)
	TEST_DIR="$OPTARG"
	;;
	o)
	POOL_CREATE_OPTIONS="$OPTARG"
	;;
	?)
	usage
	exit 1
	;;
	esac
	done

	#
	# Verify the module start is not loaded
	#
	if lsmod \| grep zfs >/dev/null; then
	echo "ZFS modules must be unloaded"
	exit 1
	fi

	#
	# Create a random directory tree of files and sub-directories to
	# to act as a copy source for the various regression tests.
	#
	populate() {
	local ROOT=$1
	local MAX_DIR_SIZE=$2
	local MAX_FILE_SIZE=$3

	# shellcheck disable=SC2086
	mkdir -p $ROOT/{a,b,c,d,e,f,g}/{h,i}
	DIRS=$(find "$ROOT")

	for DIR in $DIRS; do
	COUNT=$((RANDOM % MAX_DIR_SIZE))

	# shellcheck disable=SC2034
	for i in $(seq $COUNT); do
	FILE=$(mktemp -p "$DIR")
	SIZE=$((RANDOM % MAX_FILE_SIZE))
	dd if=/dev/urandom of="$FILE" bs=1k \
	count="$SIZE" &>/dev/null
	done
	done

	return 0
	}

	SRC_DIR=$(mktemp -d -p /var/tmp/ zfs.src.XXXXXXXX)
	trap 'rm -Rf "$SRC_DIR"' INT TERM EXIT
	populate "$SRC_DIR" 10 100

	SRC_DIR="$TEST_DIR/src"
	SRC_DIR_ZFS="$SRC_DIR/zfs"

	if [ "$COLOR" = "no" ]; then
	COLOR_GREEN=""
	COLOR_BROWN=""
	COLOR_RED=""
	COLOR_RESET=""
	fi

	pass_nonewline() {
	echo -n -e "${COLOR_GREEN}Pass${COLOR_RESET}\t\t"
	}

	skip_nonewline() {
	echo -n -e "${COLOR_BROWN}Skip${COLOR_RESET}\t\t"
	}

	fail_nonewline() {
	echo -n -e "${COLOR_RED}Fail${COLOR_RESET}\t\t"
	}

	#
	# Log a failure message, cleanup, and return an error.
	#
	fail() {
	echo -e "$PROG: $1" >&2
	$ZFS_SH -u >/dev/null 2>&1
	exit 1
	}

	#
	# Set several helper variables which are derived from a source tag.
	#
	# ZFS_TAG - The passed zfs-x.y.z tag
	# ZFS_DIR - The zfs directory name
	# ZFS_URL - The zfs github URL to fetch the tarball
	#
	src_set_vars() {
	local TAG=$1

	ZFS_TAG="$TAG"
	ZFS_DIR="$SRC_DIR_ZFS/$ZFS_TAG"
	ZFS_URL="$REPO/zfs/tarball/$ZFS_TAG"

	if [ "$TAG" = "installed" ]; then
	ZPOOL_CMD=$(command -v zpool)
	ZFS_CMD=$(command -v zfs)
	ZFS_SH="/usr/share/zfs/zfs.sh"
	else
	ZPOOL_CMD="./cmd/zpool/zpool"
	ZFS_CMD="./cmd/zfs/zfs"
	ZFS_SH="./scripts/zfs.sh"
	fi
	}

	#
	# Set several helper variables which are derived from a pool name such
	# as zol-0.6.x, zevo-1.1.1, etc. These refer to example pools from various
	# ZFS implementations which are used to verify compatibility.
	#
	# POOL_TAG - The example pools name in scripts/zfs-images/.
	# POOL_BZIP - The full path to the example bzip2 compressed pool.
	# POOL_DIR - The top level test path for this pool.
	# POOL_DIR_PRISTINE - The directory containing a pristine version of the pool.
	# POOL_DIR_COPY - The directory containing a working copy of the pool.
	# POOL_DIR_SRC - Location of a source build if it exists for this pool.
	#
	pool_set_vars() {
	local TAG=$1

	POOL_TAG=$TAG
	POOL_BZIP=$IMAGES_DIR/$POOL_TAG.tar.bz2
	POOL_DIR=$TEST_DIR/pools/$POOL_TAG
	POOL_DIR_PRISTINE=$POOL_DIR/pristine
	POOL_DIR_COPY=$POOL_DIR/copy
	POOL_DIR_SRC="$SRC_DIR_ZFS/${POOL_TAG//zol/zfs}"
	}

	#
	# Construct a non-trivial pool given a specific version of the source. More
	# interesting pools provide better test coverage so this function should
	# extended as needed to create more realistic pools.
	#
	pool_create() {
	pool_set_vars "$1"
	src_set_vars "$1"

	if [ "$POOL_TAG" != "installed" ]; then
	cd "$POOL_DIR_SRC" \|\| fail "Failed 'cd $POOL_DIR_SRC'"
	fi

	$ZFS_SH zfs="spa_config_path=$POOL_DIR_PRISTINE" \|\| \
	fail "Failed to load kmods"

	# Create a file vdev RAIDZ pool.
	truncate -s 1G \
	"$POOL_DIR_PRISTINE/vdev1" "$POOL_DIR_PRISTINE/vdev2" \
	"$POOL_DIR_PRISTINE/vdev3" "$POOL_DIR_PRISTINE/vdev4" \|\| \
	fail "Failed 'truncate -s 1G ...'"
	# shellcheck disable=SC2086
	$ZPOOL_CMD create $POOL_CREATE_OPTIONS "$POOL_TAG" raidz \
	"$POOL_DIR_PRISTINE/vdev1" "$POOL_DIR_PRISTINE/vdev2" \
	"$POOL_DIR_PRISTINE/vdev3" "$POOL_DIR_PRISTINE/vdev4" \|\| \
	fail "Failed '$ZPOOL_CMD create $POOL_CREATE_OPTIONS $POOL_TAG ...'"

	# Create a pool/fs filesystem with some random contents.
	$ZFS_CMD create "$POOL_TAG/fs" \|\| \
	fail "Failed '$ZFS_CMD create $POOL_TAG/fs'"
	populate "/$POOL_TAG/fs/" 10 100

	# Snapshot that filesystem, clone it, remove the files/dirs,
	# replace them with new files/dirs.
	$ZFS_CMD snap "$POOL_TAG/fs@snap" \|\| \
	fail "Failed '$ZFS_CMD snap $POOL_TAG/fs@snap'"
	$ZFS_CMD clone "$POOL_TAG/fs@snap" "$POOL_TAG/clone" \|\| \
	fail "Failed '$ZFS_CMD clone $POOL_TAG/fs@snap $POOL_TAG/clone'"
	# shellcheck disable=SC2086
	rm -Rf /$POOL_TAG/clone/*
	populate "/$POOL_TAG/clone/" 10 100

	# Scrub the pool, delay slightly, then export it. It is now
	# somewhat interesting for testing purposes.
	$ZPOOL_CMD scrub "$POOL_TAG" \|\| \
	fail "Failed '$ZPOOL_CMD scrub $POOL_TAG'"
	sleep 10
	$ZPOOL_CMD export "$POOL_TAG" \|\| \
	fail "Failed '$ZPOOL_CMD export $POOL_TAG'"

	$ZFS_SH -u \|\| fail "Failed to unload kmods"
	}

	# If the zfs-images directory doesn't exist fetch a copy from Github then
	# cache it in the $TEST_DIR and update $IMAGES_DIR.
	if [ ! -d "$IMAGES_DIR" ]; then
	IMAGES_DIR="$TEST_DIR/zfs-images"
	mkdir -p "$IMAGES_DIR"
	curl -sL "$IMAGES_TAR" \| \
	tar -xz -C "$IMAGES_DIR" --strip-components=1 \|\| \
	fail "Failed to download pool images"
	fi

	# Given the available images in the zfs-images directory substitute the
	# list of available images for the reserved keyword 'all'.
	for TAG in $POOL_TAGS; do

	if [ "$TAG" = "all" ]; then
	# shellcheck disable=SC2010
	ALL_TAGS=$(ls "$IMAGES_DIR" \| grep "tar.bz2" \| \
	sed 's/.tar.bz2//' \| tr '\n' ' ')
	NEW_TAGS="$NEW_TAGS $ALL_TAGS"
	else
	NEW_TAGS="$NEW_TAGS $TAG"
	fi
	done
	POOL_TAGS="$NEW_TAGS"

	if [ "$VERBOSE" = "yes" ]; then
	echo "---------------------------- Options ----------------------------"
	echo "VERBOSE=$VERBOSE"
	echo "KEEP=$KEEP"
	echo "REPO=$REPO"
	echo "SRC_TAGS=$SRC_TAGS"
	echo "POOL_TAGS=$POOL_TAGS"
	echo "PATH=$TEST_DIR"
	echo "POOL_CREATE_OPTIONS=$POOL_CREATE_OPTIONS"
	echo
	fi

	if [ ! -d "$TEST_DIR" ]; then
	mkdir -p "$TEST_DIR"
	fi

	if [ ! -d "$SRC_DIR" ]; then
	mkdir -p "$SRC_DIR"
	fi

	# Print a header for all tags which are being tested.
	-echo "--------------------- ZFS on Linux Source Versions --------------"
	+echo "------------------------ OpenZFS Source Versions ----------------"
	printf "%-16s" " "
	for TAG in $SRC_TAGS; do
	src_set_vars "$TAG"

	if [ "$TAG" = "installed" ]; then
	ZFS_VERSION=$(modinfo zfs \| awk '/version:/ { print $2; exit }')
	if [ -n "$ZFS_VERSION" ]; then
	printf "%-16s" "$ZFS_VERSION"
	else
	fail "ZFS is not installed"
	fi
	else
	printf "%-16s" "$TAG"
	fi
	done
	echo -e "\n-----------------------------------------------------------------"

	#
	# Attempt to generate the tarball from your local git repository, if that
	# fails then attempt to download the tarball from Github.
	#
	printf "%-16s" "Clone ZFS"
	for TAG in $SRC_TAGS; do
	src_set_vars "$TAG"

	if [ -d "$ZFS_DIR" ]; then
	skip_nonewline
	elif [ "$ZFS_TAG" = "installed" ]; then
	skip_nonewline
	else
	cd "$SRC_DIR" \|\| fail "Failed 'cd $SRC_DIR'"

	if [ ! -d "$SRC_DIR_ZFS" ]; then
	mkdir -p "$SRC_DIR_ZFS"
	fi

	git archive --format=tar --prefix="$ZFS_TAG/ $ZFS_TAG" \
	-o "$SRC_DIR_ZFS/$ZFS_TAG.tar" &>/dev/null \|\| \
	rm "$SRC_DIR_ZFS/$ZFS_TAG.tar"
	if [ -s "$SRC_DIR_ZFS/$ZFS_TAG.tar" ]; then
	tar -xf "$SRC_DIR_ZFS/$ZFS_TAG.tar" -C "$SRC_DIR_ZFS"
	rm "$SRC_DIR_ZFS/$ZFS_TAG.tar"
	echo -n -e "${COLOR_GREEN}Local${COLOR_RESET}\t\t"
	else
	mkdir -p "$ZFS_DIR" \|\| fail "Failed to create $ZFS_DIR"
	curl -sL "$ZFS_URL" \| tar -xz -C "$ZFS_DIR" \
	--strip-components=1 \|\| \
	fail "Failed to download $ZFS_URL"
	echo -n -e "${COLOR_GREEN}Remote${COLOR_RESET}\t\t"
	fi
	fi
	done
	printf "\n"

	# Build the listed tags
	printf "%-16s" "Build ZFS"
	for TAG in $SRC_TAGS; do
	src_set_vars "$TAG"

	if [ -f "$ZFS_DIR/module/zfs/zfs.ko" ]; then
	skip_nonewline
	elif [ "$ZFS_TAG" = "installed" ]; then
	skip_nonewline
	else
	cd "$ZFS_DIR" \|\| fail "Failed 'cd $ZFS_DIR'"
	make distclean &>/dev/null
	./autogen.sh >>"$CONFIG_LOG" 2>&1 \|\| \
	fail "Failed ZFS 'autogen.sh'"
	# shellcheck disable=SC2086
	./configure $CONFIG_OPTIONS >>"$CONFIG_LOG" 2>&1 \|\| \
	fail "Failed ZFS 'configure $CONFIG_OPTIONS'"
	# shellcheck disable=SC2086
	make $MAKE_OPTIONS >>"$MAKE_LOG" 2>&1 \|\| \
	fail "Failed ZFS 'make $MAKE_OPTIONS'"
	pass_nonewline
	fi
	done
	printf "\n"
	echo "-----------------------------------------------------------------"

	# Either create a new pool using 'zpool create', or alternately restore an
	# existing pool from another ZFS implementation for compatibility testing.
	for TAG in $POOL_TAGS; do
	pool_set_vars "$TAG"
	SKIP=0

	printf "%-16s" "$POOL_TAG"
	rm -Rf "$POOL_DIR"
	mkdir -p "$POOL_DIR_PRISTINE"

	# Use the existing compressed image if available.
	if [ -f "$POOL_BZIP" ]; then
	tar -xjf "$POOL_BZIP" -C "$POOL_DIR_PRISTINE" \
	--strip-components=1 \|\| \
	fail "Failed 'tar -xjf $POOL_BZIP"
	# Use the installed version to create the pool.
	elif [ "$TAG" = "installed" ]; then
	pool_create "$TAG"
	# A source build is available to create the pool.
	elif [ -d "$POOL_DIR_SRC" ]; then
	pool_create "$TAG"
	else
	SKIP=1
	fi

	# Verify 'zpool import' works for all listed source versions.
	for SRC_TAG in $SRC_TAGS; do

	if [ $SKIP -eq 1 ]; then
	skip_nonewline
	continue
	fi

	src_set_vars "$SRC_TAG"
	if [ "$SRC_TAG" != "installed" ]; then
	cd "$ZFS_DIR" \|\| fail "Failed 'cd $ZFS_DIR'"
	fi
	$ZFS_SH zfs="spa_config_path=$POOL_DIR_COPY"

	cp -a --sparse=always "$POOL_DIR_PRISTINE" \
	"$POOL_DIR_COPY" \|\| \
	fail "Failed to copy $POOL_DIR_PRISTINE to $POOL_DIR_COPY"
	POOL_NAME=$($ZPOOL_CMD import -d "$POOL_DIR_COPY" \| \
	awk '/pool:/ { print $2; exit 0 }')

	$ZPOOL_CMD import -N -d "$POOL_DIR_COPY" \
	"$POOL_NAME" &>/dev/null
	# shellcheck disable=SC2181
	if [ $? -ne 0 ]; then
	fail_nonewline
	ERROR=1
	else
	$ZPOOL_CMD export "$POOL_NAME" \|\| \
	fail "Failed to export pool"
	pass_nonewline
	fi

	rm -Rf "$POOL_DIR_COPY"

	$ZFS_SH -u \|\| fail "Failed to unload kmods"
	done
	printf "\n"
	done

	if [ "$KEEP" = "no" ]; then
	rm -Rf "$TEST_DIR"
	fi

	exit $ERROR
	Index: vendor-sys/openzfs/dist/tests/runfiles/common.run
	===================================================================
	--- vendor-sys/openzfs/dist/tests/runfiles/common.run (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/runfiles/common.run (revision 366774)
	@@ -1,900 +1,905 @@
	#
	# This file and its contents are supplied under the terms of the
	# Common Development and Distribution License ("CDDL"), version 1.0.
	# You may only use this file in accordance with the terms of version
	# 1.0 of the CDDL.
	#
	# A full copy of the text of the CDDL should have accompanied this
	# source. A copy of the CDDL is also available via the Internet at
	# http://www.illumos.org/license/CDDL.
	#

	[DEFAULT]
	pre = setup
	quiet = False
	pre_user = root
	user = root
	timeout = 600
	post_user = root
	post = cleanup
	failsafe_user = root
	failsafe = callbacks/zfs_failsafe
	outputdir = /var/tmp/test_results
	tags = ['functional']

	[tests/functional/alloc_class]
	tests = ['alloc_class_001_pos', 'alloc_class_002_neg', 'alloc_class_003_pos',
	'alloc_class_004_pos', 'alloc_class_005_pos', 'alloc_class_006_pos',
	'alloc_class_007_pos', 'alloc_class_008_pos', 'alloc_class_009_pos',
	'alloc_class_010_pos', 'alloc_class_011_neg', 'alloc_class_012_pos',
	'alloc_class_013_pos']
	tags = ['functional', 'alloc_class']

	+[tests/functional/arc]
	+tests = ['dbufstats_001_pos', 'dbufstats_002_pos', 'dbufstats_003_pos',
	+ 'arcstats_runtime_tuning']
	+tags = ['functional', 'arc']
	+
	[tests/functional/atime]
	tests = ['atime_001_pos', 'atime_002_neg', 'root_atime_off', 'root_atime_on']
	tags = ['functional', 'atime']

	[tests/functional/bootfs]
	tests = ['bootfs_001_pos', 'bootfs_002_neg', 'bootfs_003_pos',
	'bootfs_004_neg', 'bootfs_005_neg', 'bootfs_006_pos', 'bootfs_007_pos',
	'bootfs_008_pos']
	tags = ['functional', 'bootfs']

	[tests/functional/btree]
	tests = ['btree_positive', 'btree_negative']
	tags = ['functional', 'btree']
	pre =
	post =

	[tests/functional/cache]
	tests = ['cache_001_pos', 'cache_002_pos', 'cache_003_pos', 'cache_004_neg',
	'cache_005_neg', 'cache_006_pos', 'cache_007_neg', 'cache_008_neg',
	'cache_009_pos', 'cache_010_pos', 'cache_011_pos', 'cache_012_pos']
	tags = ['functional', 'cache']

	[tests/functional/cachefile]
	tests = ['cachefile_001_pos', 'cachefile_002_pos', 'cachefile_003_pos',
	'cachefile_004_pos']
	tags = ['functional', 'cachefile']

	[tests/functional/casenorm]
	tests = ['case_all_values', 'norm_all_values', 'mixed_create_failure',
	'sensitive_none_lookup', 'sensitive_none_delete',
	'sensitive_formd_lookup', 'sensitive_formd_delete',
	'insensitive_none_lookup', 'insensitive_none_delete',
	'insensitive_formd_lookup', 'insensitive_formd_delete',
	'mixed_none_lookup', 'mixed_none_lookup_ci', 'mixed_none_delete',
	'mixed_formd_lookup', 'mixed_formd_lookup_ci', 'mixed_formd_delete']
	tags = ['functional', 'casenorm']

	[tests/functional/channel_program/lua_core]
	tests = ['tst.args_to_lua', 'tst.divide_by_zero', 'tst.exists',
	'tst.integer_illegal', 'tst.integer_overflow', 'tst.language_functions_neg',
	'tst.language_functions_pos', 'tst.large_prog', 'tst.libraries',
	'tst.memory_limit', 'tst.nested_neg', 'tst.nested_pos', 'tst.nvlist_to_lua',
	'tst.recursive_neg', 'tst.recursive_pos', 'tst.return_large',
	'tst.return_nvlist_neg', 'tst.return_nvlist_pos',
	'tst.return_recursive_table', 'tst.stack_gsub', 'tst.timeout']
	tags = ['functional', 'channel_program', 'lua_core']

	[tests/functional/channel_program/synctask_core]
	tests = ['tst.destroy_fs', 'tst.destroy_snap', 'tst.get_count_and_limit',
	'tst.get_index_props', 'tst.get_mountpoint', 'tst.get_neg',
	'tst.get_number_props', 'tst.get_string_props', 'tst.get_type',
	'tst.get_userquota', 'tst.get_written', 'tst.inherit', 'tst.list_bookmarks',
	'tst.list_children', 'tst.list_clones', 'tst.list_holds',
	'tst.list_snapshots', 'tst.list_system_props',
	'tst.list_user_props', 'tst.parse_args_neg','tst.promote_conflict',
	'tst.promote_multiple', 'tst.promote_simple', 'tst.rollback_mult',
	'tst.rollback_one', 'tst.set_props', 'tst.snapshot_destroy', 'tst.snapshot_neg',
	'tst.snapshot_recursive', 'tst.snapshot_simple',
	'tst.bookmark.create', 'tst.bookmark.copy',
	'tst.terminate_by_signal'
	]
	tags = ['functional', 'channel_program', 'synctask_core']

	[tests/functional/checksum]
	tests = ['run_sha2_test', 'run_skein_test', 'filetest_001_pos']
	tags = ['functional', 'checksum']

	[tests/functional/clean_mirror]
	tests = [ 'clean_mirror_001_pos', 'clean_mirror_002_pos',
	'clean_mirror_003_pos', 'clean_mirror_004_pos']
	tags = ['functional', 'clean_mirror']

	[tests/functional/cli_root/zdb]
	tests = ['zdb_002_pos', 'zdb_003_pos', 'zdb_004_pos', 'zdb_005_pos',
	'zdb_006_pos', 'zdb_args_neg', 'zdb_args_pos',
	'zdb_block_size_histogram', 'zdb_checksum', 'zdb_decompress',
	'zdb_display_block', 'zdb_object_range_neg', 'zdb_object_range_pos',
	'zdb_objset_id', 'zdb_decompress_zstd']
	pre =
	post =
	tags = ['functional', 'cli_root', 'zdb']

	[tests/functional/cli_root/zfs]
	tests = ['zfs_001_neg', 'zfs_002_pos']
	tags = ['functional', 'cli_root', 'zfs']

	[tests/functional/cli_root/zfs_bookmark]
	tests = ['zfs_bookmark_cliargs']
	tags = ['functional', 'cli_root', 'zfs_bookmark']

	[tests/functional/cli_root/zfs_change-key]
	tests = ['zfs_change-key', 'zfs_change-key_child', 'zfs_change-key_format',
	'zfs_change-key_inherit', 'zfs_change-key_load', 'zfs_change-key_location',
	'zfs_change-key_pbkdf2iters', 'zfs_change-key_clones']
	tags = ['functional', 'cli_root', 'zfs_change-key']

	[tests/functional/cli_root/zfs_clone]
	tests = ['zfs_clone_001_neg', 'zfs_clone_002_pos', 'zfs_clone_003_pos',
	'zfs_clone_004_pos', 'zfs_clone_005_pos', 'zfs_clone_006_pos',
	'zfs_clone_007_pos', 'zfs_clone_008_neg', 'zfs_clone_009_neg',
	'zfs_clone_010_pos', 'zfs_clone_encrypted', 'zfs_clone_deeply_nested']
	tags = ['functional', 'cli_root', 'zfs_clone']

	[tests/functional/cli_root/zfs_copies]
	tests = ['zfs_copies_001_pos', 'zfs_copies_002_pos', 'zfs_copies_003_pos',
	'zfs_copies_004_neg', 'zfs_copies_005_neg', 'zfs_copies_006_pos']
	tags = ['functional', 'cli_root', 'zfs_copies']

	[tests/functional/cli_root/zfs_create]
	tests = ['zfs_create_001_pos', 'zfs_create_002_pos', 'zfs_create_003_pos',
	'zfs_create_004_pos', 'zfs_create_005_pos', 'zfs_create_006_pos',
	'zfs_create_007_pos', 'zfs_create_008_neg', 'zfs_create_009_neg',
	'zfs_create_010_neg', 'zfs_create_011_pos', 'zfs_create_012_pos',
	'zfs_create_013_pos', 'zfs_create_014_pos', 'zfs_create_encrypted',
	'zfs_create_crypt_combos', 'zfs_create_dryrun', 'zfs_create_verbose']
	tags = ['functional', 'cli_root', 'zfs_create']

	[tests/functional/cli_root/zfs_destroy]
	tests = ['zfs_clone_livelist_condense_and_disable',
	'zfs_clone_livelist_condense_races', 'zfs_destroy_001_pos',
	'zfs_destroy_002_pos', 'zfs_destroy_003_pos',
	'zfs_destroy_004_pos', 'zfs_destroy_005_neg', 'zfs_destroy_006_neg',
	'zfs_destroy_007_neg', 'zfs_destroy_008_pos', 'zfs_destroy_009_pos',
	'zfs_destroy_010_pos', 'zfs_destroy_011_pos', 'zfs_destroy_012_pos',
	'zfs_destroy_013_neg', 'zfs_destroy_014_pos', 'zfs_destroy_015_pos',
	'zfs_destroy_016_pos', 'zfs_destroy_clone_livelist',
	'zfs_destroy_dev_removal', 'zfs_destroy_dev_removal_condense']
	tags = ['functional', 'cli_root', 'zfs_destroy']

	[tests/functional/cli_root/zfs_diff]
	tests = ['zfs_diff_changes', 'zfs_diff_cliargs', 'zfs_diff_timestamp',
	'zfs_diff_types', 'zfs_diff_encrypted']
	tags = ['functional', 'cli_root', 'zfs_diff']

	[tests/functional/cli_root/zfs_get]
	tests = ['zfs_get_001_pos', 'zfs_get_002_pos', 'zfs_get_003_pos',
	'zfs_get_004_pos', 'zfs_get_005_neg', 'zfs_get_006_neg', 'zfs_get_007_neg',
	'zfs_get_008_pos', 'zfs_get_009_pos', 'zfs_get_010_neg']
	tags = ['functional', 'cli_root', 'zfs_get']

	[tests/functional/cli_root/zfs_ids_to_path]
	tests = ['zfs_ids_to_path_001_pos']
	tags = ['functional', 'cli_root', 'zfs_ids_to_path']

	[tests/functional/cli_root/zfs_inherit]
	tests = ['zfs_inherit_001_neg', 'zfs_inherit_002_neg', 'zfs_inherit_003_pos',
	'zfs_inherit_mountpoint']
	tags = ['functional', 'cli_root', 'zfs_inherit']

	[tests/functional/cli_root/zfs_load-key]
	tests = ['zfs_load-key', 'zfs_load-key_all', 'zfs_load-key_file',
	'zfs_load-key_location', 'zfs_load-key_noop', 'zfs_load-key_recursive']
	tags = ['functional', 'cli_root', 'zfs_load-key']

	[tests/functional/cli_root/zfs_mount]
	tests = ['zfs_mount_001_pos', 'zfs_mount_002_pos', 'zfs_mount_003_pos',
	'zfs_mount_004_pos', 'zfs_mount_005_pos', 'zfs_mount_007_pos',
	'zfs_mount_009_neg', 'zfs_mount_010_neg', 'zfs_mount_011_neg',
	'zfs_mount_012_pos', 'zfs_mount_all_001_pos', 'zfs_mount_encrypted',
	'zfs_mount_remount', 'zfs_mount_all_fail', 'zfs_mount_all_mountpoints',
	'zfs_mount_test_race']
	tags = ['functional', 'cli_root', 'zfs_mount']

	[tests/functional/cli_root/zfs_program]
	tests = ['zfs_program_json']
	tags = ['functional', 'cli_root', 'zfs_program']

	[tests/functional/cli_root/zfs_promote]
	tests = ['zfs_promote_001_pos', 'zfs_promote_002_pos', 'zfs_promote_003_pos',
	'zfs_promote_004_pos', 'zfs_promote_005_pos', 'zfs_promote_006_neg',
	'zfs_promote_007_neg', 'zfs_promote_008_pos', 'zfs_promote_encryptionroot']
	tags = ['functional', 'cli_root', 'zfs_promote']

	[tests/functional/cli_root/zfs_property]
	tests = ['zfs_written_property_001_pos']
	tags = ['functional', 'cli_root', 'zfs_property']

	[tests/functional/cli_root/zfs_receive]
	tests = ['zfs_receive_001_pos', 'zfs_receive_002_pos', 'zfs_receive_003_pos',
	'zfs_receive_004_neg', 'zfs_receive_005_neg', 'zfs_receive_006_pos',
	'zfs_receive_007_neg', 'zfs_receive_008_pos', 'zfs_receive_009_neg',
	'zfs_receive_010_pos', 'zfs_receive_011_pos', 'zfs_receive_012_pos',
	'zfs_receive_013_pos', 'zfs_receive_014_pos', 'zfs_receive_015_pos',
	'zfs_receive_016_pos', 'receive-o-x_props_override',
	'zfs_receive_from_encrypted', 'zfs_receive_to_encrypted',
	'zfs_receive_raw', 'zfs_receive_raw_incremental', 'zfs_receive_-e',
	'zfs_receive_raw_-d', 'zfs_receive_from_zstd', 'zfs_receive_new_props']
	tags = ['functional', 'cli_root', 'zfs_receive']

	[tests/functional/cli_root/zfs_rename]
	tests = ['zfs_rename_001_pos', 'zfs_rename_002_pos', 'zfs_rename_003_pos',
	'zfs_rename_004_neg', 'zfs_rename_005_neg', 'zfs_rename_006_pos',
	'zfs_rename_007_pos', 'zfs_rename_008_pos', 'zfs_rename_009_neg',
	'zfs_rename_010_neg', 'zfs_rename_011_pos', 'zfs_rename_012_neg',
	'zfs_rename_013_pos', 'zfs_rename_014_neg', 'zfs_rename_encrypted_child',
	'zfs_rename_to_encrypted', 'zfs_rename_mountpoint', 'zfs_rename_nounmount']
	tags = ['functional', 'cli_root', 'zfs_rename']

	[tests/functional/cli_root/zfs_reservation]
	tests = ['zfs_reservation_001_pos', 'zfs_reservation_002_pos']
	tags = ['functional', 'cli_root', 'zfs_reservation']

	[tests/functional/cli_root/zfs_rollback]
	tests = ['zfs_rollback_001_pos', 'zfs_rollback_002_pos',
	'zfs_rollback_003_neg', 'zfs_rollback_004_neg']
	tags = ['functional', 'cli_root', 'zfs_rollback']

	[tests/functional/cli_root/zfs_send]
	tests = ['zfs_send_001_pos', 'zfs_send_002_pos', 'zfs_send_003_pos',
	'zfs_send_004_neg', 'zfs_send_005_pos', 'zfs_send_006_pos',
	'zfs_send_007_pos', 'zfs_send_encrypted', 'zfs_send_raw',
	'zfs_send_sparse', 'zfs_send-b']
	tags = ['functional', 'cli_root', 'zfs_send']

	[tests/functional/cli_root/zfs_set]
	tests = ['cache_001_pos', 'cache_002_neg', 'canmount_001_pos',
	'canmount_002_pos', 'canmount_003_pos', 'canmount_004_pos',
	'checksum_001_pos', 'compression_001_pos', 'mountpoint_001_pos',
	'mountpoint_002_pos', 'reservation_001_neg', 'user_property_002_pos',
	'share_mount_001_neg', 'snapdir_001_pos', 'onoffs_001_pos',
	'user_property_001_pos', 'user_property_003_neg', 'readonly_001_pos',
	'user_property_004_pos', 'version_001_neg', 'zfs_set_001_neg',
	'zfs_set_002_neg', 'zfs_set_003_neg', 'property_alias_001_pos',
	'mountpoint_003_pos', 'ro_props_001_pos', 'zfs_set_keylocation',
	'zfs_set_feature_activation']
	tags = ['functional', 'cli_root', 'zfs_set']

	[tests/functional/cli_root/zfs_share]
	tests = ['zfs_share_001_pos', 'zfs_share_002_pos', 'zfs_share_003_pos',
	'zfs_share_004_pos', 'zfs_share_006_pos', 'zfs_share_008_neg',
	'zfs_share_010_neg', 'zfs_share_011_pos', 'zfs_share_concurrent_shares']
	tags = ['functional', 'cli_root', 'zfs_share']

	[tests/functional/cli_root/zfs_snapshot]
	tests = ['zfs_snapshot_001_neg', 'zfs_snapshot_002_neg',
	'zfs_snapshot_003_neg', 'zfs_snapshot_004_neg', 'zfs_snapshot_005_neg',
	'zfs_snapshot_006_pos', 'zfs_snapshot_007_neg', 'zfs_snapshot_008_neg',
	'zfs_snapshot_009_pos']
	tags = ['functional', 'cli_root', 'zfs_snapshot']

	[tests/functional/cli_root/zfs_unload-key]
	tests = ['zfs_unload-key', 'zfs_unload-key_all', 'zfs_unload-key_recursive']
	tags = ['functional', 'cli_root', 'zfs_unload-key']

	[tests/functional/cli_root/zfs_unmount]
	tests = ['zfs_unmount_001_pos', 'zfs_unmount_002_pos', 'zfs_unmount_003_pos',
	'zfs_unmount_004_pos', 'zfs_unmount_005_pos', 'zfs_unmount_006_pos',
	'zfs_unmount_007_neg', 'zfs_unmount_008_neg', 'zfs_unmount_009_pos',
	'zfs_unmount_all_001_pos', 'zfs_unmount_nested', 'zfs_unmount_unload_keys']
	tags = ['functional', 'cli_root', 'zfs_unmount']

	[tests/functional/cli_root/zfs_unshare]
	tests = ['zfs_unshare_001_pos', 'zfs_unshare_002_pos', 'zfs_unshare_003_pos',
	'zfs_unshare_004_neg', 'zfs_unshare_005_neg', 'zfs_unshare_006_pos',
	'zfs_unshare_007_pos']
	tags = ['functional', 'cli_root', 'zfs_unshare']

	[tests/functional/cli_root/zfs_upgrade]
	tests = ['zfs_upgrade_001_pos', 'zfs_upgrade_002_pos', 'zfs_upgrade_003_pos',
	'zfs_upgrade_004_pos', 'zfs_upgrade_005_pos', 'zfs_upgrade_006_neg',
	'zfs_upgrade_007_neg']
	tags = ['functional', 'cli_root', 'zfs_upgrade']

	[tests/functional/cli_root/zfs_wait]
	tests = ['zfs_wait_deleteq']
	tags = ['functional', 'cli_root', 'zfs_wait']

	[tests/functional/cli_root/zpool]
	tests = ['zpool_001_neg', 'zpool_002_pos', 'zpool_003_pos', 'zpool_colors']
	tags = ['functional', 'cli_root', 'zpool']

	[tests/functional/cli_root/zpool_add]
	tests = ['zpool_add_001_pos', 'zpool_add_002_pos', 'zpool_add_003_pos',
	'zpool_add_004_pos', 'zpool_add_006_pos', 'zpool_add_007_neg',
	'zpool_add_008_neg', 'zpool_add_009_neg', 'zpool_add_010_pos',
	'add-o_ashift', 'add_prop_ashift']
	tags = ['functional', 'cli_root', 'zpool_add']

	[tests/functional/cli_root/zpool_attach]
	tests = ['zpool_attach_001_neg', 'attach-o_ashift']
	tags = ['functional', 'cli_root', 'zpool_attach']

	[tests/functional/cli_root/zpool_clear]
	tests = ['zpool_clear_001_pos', 'zpool_clear_002_neg', 'zpool_clear_003_neg',
	'zpool_clear_readonly']
	tags = ['functional', 'cli_root', 'zpool_clear']

	[tests/functional/cli_root/zpool_create]
	tests = ['zpool_create_001_pos', 'zpool_create_002_pos',
	'zpool_create_003_pos', 'zpool_create_004_pos', 'zpool_create_005_pos',
	'zpool_create_006_pos', 'zpool_create_007_neg', 'zpool_create_008_pos',
	'zpool_create_009_neg', 'zpool_create_010_neg', 'zpool_create_011_neg',
	'zpool_create_012_neg', 'zpool_create_014_neg', 'zpool_create_015_neg',
	'zpool_create_017_neg', 'zpool_create_018_pos', 'zpool_create_019_pos',
	'zpool_create_020_pos', 'zpool_create_021_pos', 'zpool_create_022_pos',
	'zpool_create_023_neg', 'zpool_create_024_pos',
	'zpool_create_encrypted', 'zpool_create_crypt_combos',
	'zpool_create_features_001_pos', 'zpool_create_features_002_pos',
	'zpool_create_features_003_pos', 'zpool_create_features_004_neg',
	'zpool_create_features_005_pos',
	'create-o_ashift', 'zpool_create_tempname']
	tags = ['functional', 'cli_root', 'zpool_create']

	[tests/functional/cli_root/zpool_destroy]
	tests = ['zpool_destroy_001_pos', 'zpool_destroy_002_pos',
	'zpool_destroy_003_neg']
	pre =
	post =
	tags = ['functional', 'cli_root', 'zpool_destroy']

	[tests/functional/cli_root/zpool_detach]
	tests = ['zpool_detach_001_neg']
	tags = ['functional', 'cli_root', 'zpool_detach']

	[tests/functional/cli_root/zpool_events]
	tests = ['zpool_events_clear', 'zpool_events_cliargs', 'zpool_events_follow',
	'zpool_events_poolname', 'zpool_events_errors', 'zpool_events_duplicates']
	tags = ['functional', 'cli_root', 'zpool_events']

	[tests/functional/cli_root/zpool_export]
	tests = ['zpool_export_001_pos', 'zpool_export_002_pos',
	'zpool_export_003_neg', 'zpool_export_004_pos']
	tags = ['functional', 'cli_root', 'zpool_export']

	[tests/functional/cli_root/zpool_get]
	tests = ['zpool_get_001_pos', 'zpool_get_002_pos', 'zpool_get_003_pos',
	'zpool_get_004_neg', 'zpool_get_005_pos']
	tags = ['functional', 'cli_root', 'zpool_get']

	[tests/functional/cli_root/zpool_history]
	tests = ['zpool_history_001_neg', 'zpool_history_002_pos']
	tags = ['functional', 'cli_root', 'zpool_history']

	[tests/functional/cli_root/zpool_import]
	tests = ['zpool_import_001_pos', 'zpool_import_002_pos',
	'zpool_import_003_pos', 'zpool_import_004_pos', 'zpool_import_005_pos',
	'zpool_import_006_pos', 'zpool_import_007_pos', 'zpool_import_008_pos',
	'zpool_import_009_neg', 'zpool_import_010_pos', 'zpool_import_011_neg',
	'zpool_import_012_pos', 'zpool_import_013_neg', 'zpool_import_014_pos',
	'zpool_import_015_pos',
	'zpool_import_features_001_pos', 'zpool_import_features_002_neg',
	'zpool_import_features_003_pos', 'zpool_import_missing_001_pos',
	'zpool_import_missing_002_pos', 'zpool_import_missing_003_pos',
	'zpool_import_rename_001_pos', 'zpool_import_all_001_pos',
	'zpool_import_encrypted', 'zpool_import_encrypted_load',
	'zpool_import_errata3', 'zpool_import_errata4',
	'import_cachefile_device_added',
	'import_cachefile_device_removed',
	'import_cachefile_device_replaced',
	'import_cachefile_mirror_attached',
	'import_cachefile_mirror_detached',
	'import_cachefile_shared_device',
	'import_devices_missing',
	'import_paths_changed',
	'import_rewind_config_changed',
	'import_rewind_device_replaced']
	tags = ['functional', 'cli_root', 'zpool_import']

	[tests/functional/cli_root/zpool_labelclear]
	tests = ['zpool_labelclear_active', 'zpool_labelclear_exported',
	'zpool_labelclear_removed', 'zpool_labelclear_valid']
	pre =
	post =
	tags = ['functional', 'cli_root', 'zpool_labelclear']

	[tests/functional/cli_root/zpool_initialize]
	tests = ['zpool_initialize_attach_detach_add_remove',
	'zpool_initialize_import_export',
	'zpool_initialize_offline_export_import_online',
	'zpool_initialize_online_offline',
	'zpool_initialize_split',
	'zpool_initialize_start_and_cancel_neg',
	'zpool_initialize_start_and_cancel_pos',
	'zpool_initialize_suspend_resume',
	'zpool_initialize_unsupported_vdevs',
	'zpool_initialize_verify_checksums',
	'zpool_initialize_verify_initialized']
	pre =
	tags = ['functional', 'cli_root', 'zpool_initialize']

	[tests/functional/cli_root/zpool_offline]
	tests = ['zpool_offline_001_pos', 'zpool_offline_002_neg',
	'zpool_offline_003_pos']
	tags = ['functional', 'cli_root', 'zpool_offline']

	[tests/functional/cli_root/zpool_online]
	tests = ['zpool_online_001_pos', 'zpool_online_002_neg']
	tags = ['functional', 'cli_root', 'zpool_online']

	[tests/functional/cli_root/zpool_remove]
	tests = ['zpool_remove_001_neg', 'zpool_remove_002_pos',
	'zpool_remove_003_pos']
	tags = ['functional', 'cli_root', 'zpool_remove']

	[tests/functional/cli_root/zpool_replace]
	tests = ['zpool_replace_001_neg', 'replace-o_ashift', 'replace_prop_ashift']
	tags = ['functional', 'cli_root', 'zpool_replace']

	[tests/functional/cli_root/zpool_resilver]
	tests = ['zpool_resilver_bad_args', 'zpool_resilver_restart']
	tags = ['functional', 'cli_root', 'zpool_resilver']

	[tests/functional/cli_root/zpool_scrub]
	tests = ['zpool_scrub_001_neg', 'zpool_scrub_002_pos', 'zpool_scrub_003_pos',
	'zpool_scrub_004_pos', 'zpool_scrub_005_pos',
	'zpool_scrub_encrypted_unloaded', 'zpool_scrub_print_repairing',
	'zpool_scrub_offline_device', 'zpool_scrub_multiple_copies']
	tags = ['functional', 'cli_root', 'zpool_scrub']

	[tests/functional/cli_root/zpool_set]
	tests = ['zpool_set_001_pos', 'zpool_set_002_neg', 'zpool_set_003_neg',
	'zpool_set_ashift', 'zpool_set_features']
	tags = ['functional', 'cli_root', 'zpool_set']

	[tests/functional/cli_root/zpool_split]
	tests = ['zpool_split_cliargs', 'zpool_split_devices',
	'zpool_split_encryption', 'zpool_split_props', 'zpool_split_vdevs',
	'zpool_split_resilver', 'zpool_split_indirect']
	tags = ['functional', 'cli_root', 'zpool_split']

	[tests/functional/cli_root/zpool_status]
	tests = ['zpool_status_001_pos', 'zpool_status_002_pos']
	tags = ['functional', 'cli_root', 'zpool_status']

	[tests/functional/cli_root/zpool_sync]
	tests = ['zpool_sync_001_pos', 'zpool_sync_002_neg']
	tags = ['functional', 'cli_root', 'zpool_sync']

	[tests/functional/cli_root/zpool_trim]
	tests = ['zpool_trim_attach_detach_add_remove',
	'zpool_trim_import_export', 'zpool_trim_multiple', 'zpool_trim_neg',
	'zpool_trim_offline_export_import_online', 'zpool_trim_online_offline',
	'zpool_trim_partial', 'zpool_trim_rate', 'zpool_trim_rate_neg',
	'zpool_trim_secure', 'zpool_trim_split', 'zpool_trim_start_and_cancel_neg',
	'zpool_trim_start_and_cancel_pos', 'zpool_trim_suspend_resume',
	'zpool_trim_unsupported_vdevs', 'zpool_trim_verify_checksums',
	'zpool_trim_verify_trimmed']
	tags = ['functional', 'zpool_trim']

	[tests/functional/cli_root/zpool_upgrade]
	tests = ['zpool_upgrade_001_pos', 'zpool_upgrade_002_pos',
	'zpool_upgrade_003_pos', 'zpool_upgrade_004_pos',
	'zpool_upgrade_005_neg', 'zpool_upgrade_006_neg',
	'zpool_upgrade_007_pos', 'zpool_upgrade_008_pos',
	'zpool_upgrade_009_neg']
	tags = ['functional', 'cli_root', 'zpool_upgrade']

	[tests/functional/cli_root/zpool_wait]
	tests = ['zpool_wait_discard', 'zpool_wait_freeing',
	'zpool_wait_initialize_basic', 'zpool_wait_initialize_cancel',
	'zpool_wait_initialize_flag', 'zpool_wait_multiple',
	'zpool_wait_no_activity', 'zpool_wait_remove', 'zpool_wait_remove_cancel',
	'zpool_wait_trim_basic', 'zpool_wait_trim_cancel', 'zpool_wait_trim_flag',
	'zpool_wait_usage']
	tags = ['functional', 'cli_root', 'zpool_wait']

	[tests/functional/cli_root/zpool_wait/scan]
	tests = ['zpool_wait_replace_cancel', 'zpool_wait_rebuild',
	'zpool_wait_resilver', 'zpool_wait_scrub_cancel',
	'zpool_wait_replace', 'zpool_wait_scrub_basic', 'zpool_wait_scrub_flag']
	tags = ['functional', 'cli_root', 'zpool_wait']

	[tests/functional/cli_user/misc]
	tests = ['zdb_001_neg', 'zfs_001_neg', 'zfs_allow_001_neg',
	'zfs_clone_001_neg', 'zfs_create_001_neg', 'zfs_destroy_001_neg',
	'zfs_get_001_neg', 'zfs_inherit_001_neg', 'zfs_mount_001_neg',
	'zfs_promote_001_neg', 'zfs_receive_001_neg', 'zfs_rename_001_neg',
	'zfs_rollback_001_neg', 'zfs_send_001_neg', 'zfs_set_001_neg',
	'zfs_share_001_neg', 'zfs_snapshot_001_neg', 'zfs_unallow_001_neg',
	'zfs_unmount_001_neg', 'zfs_unshare_001_neg', 'zfs_upgrade_001_neg',
	'zpool_001_neg', 'zpool_add_001_neg', 'zpool_attach_001_neg',
	'zpool_clear_001_neg', 'zpool_create_001_neg', 'zpool_destroy_001_neg',
	'zpool_detach_001_neg', 'zpool_export_001_neg', 'zpool_get_001_neg',
	'zpool_history_001_neg', 'zpool_import_001_neg', 'zpool_import_002_neg',
	'zpool_offline_001_neg', 'zpool_online_001_neg', 'zpool_remove_001_neg',
	'zpool_replace_001_neg', 'zpool_scrub_001_neg', 'zpool_set_001_neg',
	'zpool_status_001_neg', 'zpool_upgrade_001_neg', 'arcstat_001_pos',
	'arc_summary_001_pos', 'arc_summary_002_neg', 'zpool_wait_privilege']
	user =
	tags = ['functional', 'cli_user', 'misc']

	[tests/functional/cli_user/zfs_list]
	tests = ['zfs_list_001_pos', 'zfs_list_002_pos', 'zfs_list_003_pos',
	'zfs_list_004_neg', 'zfs_list_007_pos', 'zfs_list_008_neg']
	user =
	tags = ['functional', 'cli_user', 'zfs_list']

	[tests/functional/cli_user/zpool_iostat]
	tests = ['zpool_iostat_001_neg', 'zpool_iostat_002_pos',
	'zpool_iostat_003_neg', 'zpool_iostat_004_pos',
	'zpool_iostat_005_pos', 'zpool_iostat_-c_disable',
	'zpool_iostat_-c_homedir', 'zpool_iostat_-c_searchpath']
	user =
	tags = ['functional', 'cli_user', 'zpool_iostat']

	[tests/functional/cli_user/zpool_list]
	tests = ['zpool_list_001_pos', 'zpool_list_002_neg']
	user =
	tags = ['functional', 'cli_user', 'zpool_list']

	[tests/functional/cli_user/zpool_status]
	tests = ['zpool_status_003_pos', 'zpool_status_-c_disable',
	'zpool_status_-c_homedir', 'zpool_status_-c_searchpath']
	user =
	tags = ['functional', 'cli_user', 'zpool_status']

	[tests/functional/compression]
	tests = ['compress_001_pos', 'compress_002_pos', 'compress_003_pos',
	'l2arc_compressed_arc', 'l2arc_compressed_arc_disabled',
	'l2arc_encrypted', 'l2arc_encrypted_no_compressed_arc']
	tags = ['functional', 'compression']

	[tests/functional/cp_files]
	tests = ['cp_files_001_pos']
	tags = ['functional', 'cp_files']

	[tests/functional/ctime]
	tests = ['ctime_001_pos' ]
	tags = ['functional', 'ctime']

	[tests/functional/delegate]
	tests = ['zfs_allow_001_pos', 'zfs_allow_002_pos', 'zfs_allow_003_pos',
	'zfs_allow_004_pos', 'zfs_allow_005_pos', 'zfs_allow_006_pos',
	'zfs_allow_007_pos', 'zfs_allow_008_pos', 'zfs_allow_009_neg',
	'zfs_allow_010_pos', 'zfs_allow_011_neg', 'zfs_allow_012_neg',
	'zfs_unallow_001_pos', 'zfs_unallow_002_pos', 'zfs_unallow_003_pos',
	'zfs_unallow_004_pos', 'zfs_unallow_005_pos', 'zfs_unallow_006_pos',
	'zfs_unallow_007_neg', 'zfs_unallow_008_neg']
	tags = ['functional', 'delegate']

	[tests/functional/exec]
	tests = ['exec_001_pos', 'exec_002_neg']
	tags = ['functional', 'exec']

	[tests/functional/features/async_destroy]
	tests = ['async_destroy_001_pos']
	tags = ['functional', 'features', 'async_destroy']

	[tests/functional/features/large_dnode]
	tests = ['large_dnode_001_pos', 'large_dnode_003_pos', 'large_dnode_004_neg',
	'large_dnode_005_pos', 'large_dnode_007_neg', 'large_dnode_009_pos']
	tags = ['functional', 'features', 'large_dnode']

	[tests/functional/grow]
	pre =
	post =
	tests = ['grow_pool_001_pos', 'grow_replicas_001_pos']
	tags = ['functional', 'grow']

	[tests/functional/history]
	tests = ['history_001_pos', 'history_002_pos', 'history_003_pos',
	'history_004_pos', 'history_005_neg', 'history_006_neg',
	'history_007_pos', 'history_008_pos', 'history_009_pos',
	'history_010_pos']
	tags = ['functional', 'history']

	[tests/functional/hkdf]
	tests = ['run_hkdf_test']
	tags = ['functional', 'hkdf']

	[tests/functional/inheritance]
	tests = ['inherit_001_pos']
	pre =
	tags = ['functional', 'inheritance']

	[tests/functional/io]
	tests = ['sync', 'psync', 'posixaio', 'mmap']
	tags = ['functional', 'io']

	[tests/functional/inuse]
	tests = ['inuse_004_pos', 'inuse_005_pos', 'inuse_008_pos', 'inuse_009_pos']
	post =
	tags = ['functional', 'inuse']

	[tests/functional/large_files]
	tests = ['large_files_001_pos', 'large_files_002_pos']
	tags = ['functional', 'large_files']

	[tests/functional/largest_pool]
	tests = ['largest_pool_001_pos']
	pre =
	post =
	tags = ['functional', 'largest_pool']

	[tests/functional/limits]
	tests = ['filesystem_count', 'filesystem_limit', 'snapshot_count',
	'snapshot_limit']
	tags = ['functional', 'limits']

	[tests/functional/link_count]
	tests = ['link_count_001', 'link_count_root_inode']
	tags = ['functional', 'link_count']

	[tests/functional/migration]
	tests = ['migration_001_pos', 'migration_002_pos', 'migration_003_pos',
	'migration_004_pos', 'migration_005_pos', 'migration_006_pos',
	'migration_007_pos', 'migration_008_pos', 'migration_009_pos',
	'migration_010_pos', 'migration_011_pos', 'migration_012_pos']
	tags = ['functional', 'migration']

	[tests/functional/mmap]
	tests = ['mmap_write_001_pos', 'mmap_read_001_pos']
	tags = ['functional', 'mmap']

	[tests/functional/mount]
	tests = ['umount_001', 'umountall_001']
	tags = ['functional', 'mount']

	[tests/functional/mv_files]
	tests = ['mv_files_001_pos', 'mv_files_002_pos', 'random_creation']
	tags = ['functional', 'mv_files']

	[tests/functional/nestedfs]
	tests = ['nestedfs_001_pos']
	tags = ['functional', 'nestedfs']

	[tests/functional/no_space]
	tests = ['enospc_001_pos', 'enospc_002_pos', 'enospc_003_pos',
	'enospc_df']
	tags = ['functional', 'no_space']

	[tests/functional/nopwrite]
	tests = ['nopwrite_copies', 'nopwrite_mtime', 'nopwrite_negative',
	'nopwrite_promoted_clone', 'nopwrite_recsize', 'nopwrite_sync',
	'nopwrite_varying_compression', 'nopwrite_volume']
	tags = ['functional', 'nopwrite']

	[tests/functional/online_offline]
	tests = ['online_offline_001_pos', 'online_offline_002_neg',
	'online_offline_003_neg']
	tags = ['functional', 'online_offline']

	[tests/functional/persist_l2arc]
	tests = ['persist_l2arc_001_pos', 'persist_l2arc_002_pos',
	'persist_l2arc_003_neg', 'persist_l2arc_004_pos', 'persist_l2arc_005_pos',
	'persist_l2arc_006_pos', 'persist_l2arc_007_pos', 'persist_l2arc_008_pos']
	tags = ['functional', 'persist_l2arc']

	[tests/functional/pool_checkpoint]
	tests = ['checkpoint_after_rewind', 'checkpoint_big_rewind',
	'checkpoint_capacity', 'checkpoint_conf_change', 'checkpoint_discard',
	'checkpoint_discard_busy', 'checkpoint_discard_many',
	'checkpoint_indirect', 'checkpoint_invalid', 'checkpoint_lun_expsz',
	'checkpoint_open', 'checkpoint_removal', 'checkpoint_rewind',
	'checkpoint_ro_rewind', 'checkpoint_sm_scale', 'checkpoint_twice',
	'checkpoint_vdev_add', 'checkpoint_zdb', 'checkpoint_zhack_feat']
	tags = ['functional', 'pool_checkpoint']
	timeout = 1800

	[tests/functional/pool_names]
	tests = ['pool_names_001_pos', 'pool_names_002_neg']
	pre =
	post =
	tags = ['functional', 'pool_names']

	[tests/functional/poolversion]
	tests = ['poolversion_001_pos', 'poolversion_002_pos']
	tags = ['functional', 'poolversion']

	[tests/functional/pyzfs]
	tests = ['pyzfs_unittest']
	pre =
	post =
	tags = ['functional', 'pyzfs']

	[tests/functional/quota]
	tests = ['quota_001_pos', 'quota_002_pos', 'quota_003_pos',
	'quota_004_pos', 'quota_005_pos', 'quota_006_neg']
	tags = ['functional', 'quota']

	[tests/functional/redacted_send]
	tests = ['redacted_compressed', 'redacted_contents', 'redacted_deleted',
	'redacted_disabled_feature', 'redacted_embedded', 'redacted_holes',
	'redacted_incrementals', 'redacted_largeblocks', 'redacted_many_clones',
	'redacted_mixed_recsize', 'redacted_mounts', 'redacted_negative',
	'redacted_origin', 'redacted_props', 'redacted_resume', 'redacted_size',
	'redacted_volume']
	tags = ['functional', 'redacted_send']

	[tests/functional/raidz]
	tests = ['raidz_001_neg', 'raidz_002_pos']
	tags = ['functional', 'raidz']

	[tests/functional/redundancy]
	tests = ['redundancy_001_pos', 'redundancy_002_pos', 'redundancy_003_pos',
	'redundancy_004_neg']
	tags = ['functional', 'redundancy']

	[tests/functional/refquota]
	tests = ['refquota_001_pos', 'refquota_002_pos', 'refquota_003_pos',
	'refquota_004_pos', 'refquota_005_pos', 'refquota_006_neg',
	'refquota_007_neg', 'refquota_008_neg']
	tags = ['functional', 'refquota']

	[tests/functional/refreserv]
	tests = ['refreserv_001_pos', 'refreserv_002_pos', 'refreserv_003_pos',
	'refreserv_004_pos', 'refreserv_005_pos', 'refreserv_multi_raidz',
	'refreserv_raidz']
	tags = ['functional', 'refreserv']

	[tests/functional/removal]
	pre =
	tests = ['removal_all_vdev', 'removal_cancel', 'removal_check_space',
	'removal_condense_export', 'removal_multiple_indirection',
	'removal_nopwrite', 'removal_remap_deadlists',
	'removal_resume_export', 'removal_sanity', 'removal_with_add',
	'removal_with_create_fs', 'removal_with_dedup',
	'removal_with_errors', 'removal_with_export',
	'removal_with_ganging', 'removal_with_faulted',
	'removal_with_remove', 'removal_with_scrub', 'removal_with_send',
	'removal_with_send_recv', 'removal_with_snapshot',
	'removal_with_write', 'removal_with_zdb', 'remove_expanded',
	'remove_mirror', 'remove_mirror_sanity', 'remove_raidz',
	'remove_indirect']
	tags = ['functional', 'removal']

	[tests/functional/rename_dirs]
	tests = ['rename_dirs_001_pos']
	tags = ['functional', 'rename_dirs']

	[tests/functional/replacement]
	tests = ['attach_import', 'attach_multiple', 'attach_rebuild',
	'attach_resilver', 'detach', 'rebuild_disabled_feature',
	'rebuild_multiple', 'rebuild_raidz', 'replace_import', 'replace_rebuild',
	'replace_resilver', 'resilver_restart_001', 'resilver_restart_002',
	'scrub_cancel']
	tags = ['functional', 'replacement']

	[tests/functional/reservation]
	tests = ['reservation_001_pos', 'reservation_002_pos', 'reservation_003_pos',
	'reservation_004_pos', 'reservation_005_pos', 'reservation_006_pos',
	'reservation_007_pos', 'reservation_008_pos', 'reservation_009_pos',
	'reservation_010_pos', 'reservation_011_pos', 'reservation_012_pos',
	'reservation_013_pos', 'reservation_014_pos', 'reservation_015_pos',
	'reservation_016_pos', 'reservation_017_pos', 'reservation_018_pos',
	'reservation_019_pos', 'reservation_020_pos', 'reservation_021_neg',
	'reservation_022_pos']
	tags = ['functional', 'reservation']

	[tests/functional/rootpool]
	tests = ['rootpool_002_neg', 'rootpool_003_neg', 'rootpool_007_pos']
	tags = ['functional', 'rootpool']

	[tests/functional/rsend]
	tests = ['recv_dedup', 'recv_dedup_encrypted_zvol', 'rsend_001_pos',
	'rsend_002_pos', 'rsend_003_pos', 'rsend_004_pos', 'rsend_005_pos',
	'rsend_006_pos', 'rsend_007_pos', 'rsend_008_pos', 'rsend_009_pos',
	'rsend_010_pos', 'rsend_011_pos', 'rsend_012_pos', 'rsend_013_pos',
	'rsend_014_pos', 'rsend_016_neg', 'rsend_019_pos', 'rsend_020_pos',
	'rsend_021_pos', 'rsend_022_pos', 'rsend_024_pos',
	'send-c_verify_ratio', 'send-c_verify_contents', 'send-c_props',
	'send-c_incremental', 'send-c_volume', 'send-c_zstreamdump',
	'send-c_lz4_disabled', 'send-c_recv_lz4_disabled',
	'send-c_mixed_compression', 'send-c_stream_size_estimate',
	'send-c_embedded_blocks', 'send-c_resume', 'send-cpL_varied_recsize',
	'send-c_recv_dedup', 'send-L_toggle', 'send_encrypted_hierarchy',
	'send_encrypted_props', 'send_encrypted_truncated_files',
	'send_freeobjects', 'send_realloc_files',
	'send_realloc_encrypted_files', 'send_spill_block', 'send_holds',
	'send_hole_birth', 'send_mixed_raw', 'send-wR_encrypted_zvol',
	'send_partial_dataset', 'send_invalid']
	tags = ['functional', 'rsend']

	[tests/functional/scrub_mirror]
	tests = ['scrub_mirror_001_pos', 'scrub_mirror_002_pos',
	'scrub_mirror_003_pos', 'scrub_mirror_004_pos']
	tags = ['functional', 'scrub_mirror']

	[tests/functional/slog]
	tests = ['slog_001_pos', 'slog_002_pos', 'slog_003_pos', 'slog_004_pos',
	'slog_005_pos', 'slog_006_pos', 'slog_007_pos', 'slog_008_neg',
	'slog_009_neg', 'slog_010_neg', 'slog_011_neg', 'slog_012_neg',
	'slog_013_pos', 'slog_014_pos', 'slog_015_neg', 'slog_replay_fs_001',
	'slog_replay_fs_002', 'slog_replay_volume']
	tags = ['functional', 'slog']

	[tests/functional/snapshot]
	tests = ['clone_001_pos', 'rollback_001_pos', 'rollback_002_pos',
	'rollback_003_pos', 'snapshot_001_pos', 'snapshot_002_pos',
	'snapshot_003_pos', 'snapshot_004_pos', 'snapshot_005_pos',
	'snapshot_006_pos', 'snapshot_007_pos', 'snapshot_008_pos',
	'snapshot_009_pos', 'snapshot_010_pos', 'snapshot_011_pos',
	'snapshot_012_pos', 'snapshot_013_pos', 'snapshot_014_pos',
	'snapshot_017_pos']
	tags = ['functional', 'snapshot']

	[tests/functional/snapused]
	tests = ['snapused_001_pos', 'snapused_002_pos', 'snapused_003_pos',
	'snapused_004_pos', 'snapused_005_pos']
	tags = ['functional', 'snapused']

	[tests/functional/sparse]
	tests = ['sparse_001_pos']
	tags = ['functional', 'sparse']

	[tests/functional/suid]
	tests = ['suid_write_to_suid', 'suid_write_to_sgid', 'suid_write_to_suid_sgid',
	'suid_write_to_none']
	tags = ['functional', 'suid']

	[tests/functional/threadsappend]
	tests = ['threadsappend_001_pos']
	tags = ['functional', 'threadsappend']

	[tests/functional/trim]
	tests = ['autotrim_integrity', 'autotrim_config', 'autotrim_trim_integrity',
	'trim_integrity', 'trim_config', 'trim_l2arc']
	tags = ['functional', 'trim']

	[tests/functional/truncate]
	tests = ['truncate_001_pos', 'truncate_002_pos', 'truncate_timestamps']
	tags = ['functional', 'truncate']

	[tests/functional/upgrade]
	tests = ['upgrade_userobj_001_pos', 'upgrade_readonly_pool']
	tags = ['functional', 'upgrade']

	[tests/functional/userquota]
	tests = [
	'userquota_001_pos', 'userquota_002_pos', 'userquota_003_pos',
	'userquota_004_pos', 'userquota_005_neg', 'userquota_006_pos',
	'userquota_007_pos', 'userquota_008_pos', 'userquota_009_pos',
	'userquota_010_pos', 'userquota_011_pos', 'userquota_012_neg',
	'userspace_001_pos', 'userspace_002_pos']
	tags = ['functional', 'userquota']

	[tests/functional/vdev_zaps]
	tests = ['vdev_zaps_001_pos', 'vdev_zaps_002_pos', 'vdev_zaps_003_pos',
	'vdev_zaps_004_pos', 'vdev_zaps_005_pos', 'vdev_zaps_006_pos',
	'vdev_zaps_007_pos']
	tags = ['functional', 'vdev_zaps']

	[tests/functional/write_dirs]
	tests = ['write_dirs_001_pos', 'write_dirs_002_pos']
	tags = ['functional', 'write_dirs']

	[tests/functional/xattr]
	tests = ['xattr_001_pos', 'xattr_002_neg', 'xattr_003_neg', 'xattr_004_pos',
	'xattr_005_pos', 'xattr_006_pos', 'xattr_007_neg',
	'xattr_011_pos', 'xattr_012_pos', 'xattr_013_pos']
	tags = ['functional', 'xattr']

	[tests/functional/zvol/zvol_ENOSPC]
	tests = ['zvol_ENOSPC_001_pos']
	tags = ['functional', 'zvol', 'zvol_ENOSPC']

	[tests/functional/zvol/zvol_cli]
	tests = ['zvol_cli_001_pos', 'zvol_cli_002_pos', 'zvol_cli_003_neg']
	tags = ['functional', 'zvol', 'zvol_cli']

	[tests/functional/zvol/zvol_misc]
	tests = ['zvol_misc_002_pos', 'zvol_misc_hierarchy', 'zvol_misc_rename_inuse',
	'zvol_misc_snapdev', 'zvol_misc_volmode', 'zvol_misc_zil']
	tags = ['functional', 'zvol', 'zvol_misc']

	[tests/functional/zvol/zvol_swap]
	tests = ['zvol_swap_001_pos', 'zvol_swap_002_pos', 'zvol_swap_004_pos']
	tags = ['functional', 'zvol', 'zvol_swap']

	[tests/functional/libzfs]
	tests = ['many_fds', 'libzfs_input']
	tags = ['functional', 'libzfs']

	[tests/functional/log_spacemap]
	tests = ['log_spacemap_import_logs']
	pre =
	post =
	tags = ['functional', 'log_spacemap']
	Index: vendor-sys/openzfs/dist/tests/runfiles/linux.run
	===================================================================
	--- vendor-sys/openzfs/dist/tests/runfiles/linux.run (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/runfiles/linux.run (revision 366774)
	@@ -1,179 +1,178 @@
	#
	# This file and its contents are supplied under the terms of the
	# Common Development and Distribution License ("CDDL"), version 1.0.
	# You may only use this file in accordance with the terms of version
	# 1.0 of the CDDL.
	#
	# A full copy of the text of the CDDL should have accompanied this
	# source. A copy of the CDDL is also available via the Internet at
	# http://www.illumos.org/license/CDDL.
	#

	[DEFAULT]
	pre = setup
	quiet = False
	pre_user = root
	user = root
	timeout = 600
	post_user = root
	post = cleanup
	failsafe_user = root
	failsafe = callbacks/zfs_failsafe
	outputdir = /var/tmp/test_results
	tags = ['functional']

	[tests/functional/acl/posix:Linux]
	-tests = ['posix_001_pos', 'posix_002_pos', 'posix_003_pos']
	+tests = ['posix_001_pos', 'posix_002_pos', 'posix_003_pos', 'posix_004_pos']
	tags = ['functional', 'acl', 'posix']

	-[tests/functional/arc:Linux]
	-tests = ['dbufstats_001_pos', 'dbufstats_002_pos', 'dbufstats_003_pos',
	- 'arcstats_runtime_tuning']
	-tags = ['functional', 'arc']
	+[tests/functional/acl/posix-sa:Linux]
	+tests = ['posix_001_pos', 'posix_002_pos', 'posix_003_pos', 'posix_004_pos']
	+tags = ['functional', 'acl', 'posix-sa']

	[tests/functional/atime:Linux]
	tests = ['atime_003_pos', 'root_relatime_on']
	tags = ['functional', 'atime']

	[tests/functional/chattr:Linux]
	tests = ['chattr_001_pos', 'chattr_002_neg']
	tags = ['functional', 'chattr']

	[tests/functional/checksum:Linux]
	tests = ['run_edonr_test']
	tags = ['functional', 'checksum']

	[tests/functional/cli_root/zfs:Linux]
	tests = ['zfs_003_neg']
	tags = ['functional', 'cli_root', 'zfs']

	[tests/functional/cli_root/zfs_mount:Linux]
	tests = ['zfs_mount_006_pos', 'zfs_mount_008_pos', 'zfs_multi_mount']
	tags = ['functional', 'cli_root', 'zfs_mount']

	[tests/functional/cli_root/zfs_share:Linux]
	tests = ['zfs_share_005_pos', 'zfs_share_007_neg', 'zfs_share_009_neg',
	'zfs_share_012_pos']
	tags = ['functional', 'cli_root', 'zfs_share']

	[tests/functional/cli_root/zfs_sysfs:Linux]
	tests = ['zfeature_set_unsupported', 'zfs_get_unsupported',
	'zfs_set_unsupported', 'zfs_sysfs_live', 'zpool_get_unsupported',
	'zpool_set_unsupported']
	tags = ['functional', 'cli_root', 'zfs_sysfs']

	[tests/functional/cli_root/zpool_add:Linux]
	tests = ['add_nested_replacing_spare']
	tags = ['functional', 'cli_root', 'zpool_add']

	[tests/functional/cli_root/zpool_expand:Linux]
	tests = ['zpool_expand_001_pos', 'zpool_expand_002_pos',
	'zpool_expand_003_neg', 'zpool_expand_004_pos', 'zpool_expand_005_pos']
	tags = ['functional', 'cli_root', 'zpool_expand']

	[tests/functional/cli_root/zpool_reopen:Linux]
	tests = ['zpool_reopen_001_pos', 'zpool_reopen_002_pos',
	'zpool_reopen_003_pos', 'zpool_reopen_004_pos', 'zpool_reopen_005_pos',
	'zpool_reopen_006_neg', 'zpool_reopen_007_pos']
	tags = ['functional', 'cli_root', 'zpool_reopen']

	[tests/functional/cli_root/zpool_split:Linux]
	tests = ['zpool_split_wholedisk']
	tags = ['functional', 'cli_root', 'zpool_split']

	[tests/functional/compression:Linux]
	tests = ['compress_004_pos']
	tags = ['functional', 'compression']

	[tests/functional/deadman:Linux]
	tests = ['deadman_sync', 'deadman_zio']
	pre =
	post =
	tags = ['functional', 'deadman']

	[tests/functional/devices:Linux]
	tests = ['devices_001_pos', 'devices_002_neg', 'devices_003_pos']
	tags = ['functional', 'devices']

	[tests/functional/events:Linux]
	tests = ['events_001_pos', 'events_002_pos', 'zed_rc_filter']
	tags = ['functional', 'events']

	[tests/functional/fallocate:Linux]
	tests = ['fallocate_prealloc', 'fallocate_punch-hole']
	tags = ['functional', 'fallocate']

	[tests/functional/fault:Linux]
	tests = ['auto_offline_001_pos', 'auto_online_001_pos', 'auto_replace_001_pos',
	'auto_spare_001_pos', 'auto_spare_002_pos', 'auto_spare_multiple',
	'auto_spare_ashift', 'auto_spare_shared', 'decrypt_fault',
	'decompress_fault', 'scrub_after_resilver', 'zpool_status_-s']
	tags = ['functional', 'fault']

	[tests/functional/features/large_dnode:Linux]
	tests = ['large_dnode_002_pos', 'large_dnode_006_pos', 'large_dnode_008_pos']
	tags = ['functional', 'features', 'large_dnode']

	[tests/functional/io:Linux]
	tests = ['libaio']
	tags = ['functional', 'io']

	[tests/functional/mmap:Linux]
	tests = ['mmap_libaio_001_pos']
	tags = ['functional', 'mmap']

	[tests/functional/mmp:Linux]
	tests = ['mmp_on_thread', 'mmp_on_uberblocks', 'mmp_on_off', 'mmp_interval',
	'mmp_active_import', 'mmp_inactive_import', 'mmp_exported_import',
	'mmp_write_uberblocks', 'mmp_reset_interval', 'multihost_history',
	'mmp_on_zdb', 'mmp_write_distribution', 'mmp_hostid']
	tags = ['functional', 'mmp']

	[tests/functional/mount:Linux]
	tests = ['umount_unlinked_drain']
	tags = ['functional', 'mount']

	[tests/functional/pam:Linux]
	tests = ['pam_basic', 'pam_nounmount']
	tags = ['functional', 'pam']

	[tests/functional/procfs:Linux]
	tests = ['procfs_list_basic', 'procfs_list_concurrent_readers',
	'procfs_list_stale_read', 'pool_state']
	tags = ['functional', 'procfs']

	[tests/functional/projectquota:Linux]
	tests = ['projectid_001_pos', 'projectid_002_pos', 'projectid_003_pos',
	'projectquota_001_pos', 'projectquota_002_pos', 'projectquota_003_pos',
	'projectquota_004_neg', 'projectquota_005_pos', 'projectquota_006_pos',
	'projectquota_007_pos', 'projectquota_008_pos', 'projectquota_009_pos',
	'projectspace_001_pos', 'projectspace_002_pos', 'projectspace_003_pos',
	'projectspace_004_pos',
	'projecttree_001_pos', 'projecttree_002_pos', 'projecttree_003_neg']
	tags = ['functional', 'projectquota']

	[tests/functional/rsend:Linux]
	tests = ['send_realloc_dnode_size', 'send_encrypted_files']
	tags = ['functional', 'rsend']

	[tests/functional/snapshot:Linux]
	tests = ['snapshot_015_pos', 'snapshot_016_pos']
	tags = ['functional', 'snapshot']

	[tests/functional/tmpfile:Linux]
	tests = ['tmpfile_001_pos', 'tmpfile_002_pos', 'tmpfile_003_pos',
	'tmpfile_stat_mode']
	tags = ['functional', 'tmpfile']

	[tests/functional/upgrade:Linux]
	tests = ['upgrade_projectquota_001_pos']
	tags = ['functional', 'upgrade']

	[tests/functional/user_namespace:Linux]
	tests = ['user_namespace_001']
	tags = ['functional', 'user_namespace']

	[tests/functional/userquota:Linux]
	tests = ['groupspace_001_pos', 'groupspace_002_pos', 'groupspace_003_pos',
	'userquota_013_pos', 'userspace_003_pos']
	tags = ['functional', 'userquota']
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/include/libtest.shlib
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/include/libtest.shlib (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/include/libtest.shlib (revision 366774)
	@@ -1,4194 +1,4212 @@
	#
	# CDDL HEADER START
	#
	# The contents of this file are subject to the terms of the
	# Common Development and Distribution License (the "License").
	# You may not use this file except in compliance with the License.
	#
	# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	# or http://www.opensolaris.org/os/licensing.
	# See the License for the specific language governing permissions
	# and limitations under the License.
	#
	# When distributing Covered Code, include this CDDL HEADER in each
	# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	# If applicable, add the following below this CDDL HEADER, with the
	# fields enclosed by brackets "[]" replaced with your own identifying
	# information: Portions Copyright [yyyy] [name of copyright owner]
	#
	# CDDL HEADER END
	#

	#
	# Copyright (c) 2009, Sun Microsystems Inc. All rights reserved.
	# Copyright (c) 2012, 2020, Delphix. All rights reserved.
	# Copyright (c) 2017, Tim Chase. All rights reserved.
	# Copyright (c) 2017, Nexenta Systems Inc. All rights reserved.
	# Copyright (c) 2017, Lawrence Livermore National Security LLC.
	# Copyright (c) 2017, Datto Inc. All rights reserved.
	# Copyright (c) 2017, Open-E Inc. All rights reserved.
	# Use is subject to license terms.
	#

	. ${STF_TOOLS}/include/logapi.shlib
	. ${STF_SUITE}/include/math.shlib
	. ${STF_SUITE}/include/blkdev.shlib

	. ${STF_SUITE}/include/tunables.cfg

	#
	# Apply constrained path when available. This is required since the
	# PATH may have been modified by sudo's secure_path behavior.
	#
	if [ -n "$STF_PATH" ]; then
	PATH="$STF_PATH"
	fi

	#
	# Generic dot version comparison function
	#
	# Returns success when version $1 is greater than or equal to $2.
	#
	function compare_version_gte
	{
	if [[ "$(printf "$1\n$2" \| sort -V \| tail -n1)" == "$1" ]]; then
	return 0
	else
	return 1
	fi
	}

	# Linux kernel version comparison function
	#
	# $1 Linux version ("4.10", "2.6.32") or blank for installed Linux version
	#
	# Used for comparison: if [ $(linux_version) -ge $(linux_version "2.6.32") ]
	#
	function linux_version
	{
	typeset ver="$1"

	[[ -z "$ver" ]] && ver=$(uname -r \| grep -Eo "^[0-9]+\.[0-9]+\.[0-9]+")

	typeset version=$(echo $ver \| cut -d '.' -f 1)
	typeset major=$(echo $ver \| cut -d '.' -f 2)
	typeset minor=$(echo $ver \| cut -d '.' -f 3)

	[[ -z "$version" ]] && version=0
	[[ -z "$major" ]] && major=0
	[[ -z "$minor" ]] && minor=0

	echo $((version * 10000 + major * 100 + minor))
	}

	# Determine if this is a Linux test system
	#
	# Return 0 if platform Linux, 1 if otherwise

	function is_linux
	{
	if [[ $(uname -o) == "GNU/Linux" ]]; then
	return 0
	else
	return 1
	fi
	}

	# Determine if this is an illumos test system
	#
	# Return 0 if platform illumos, 1 if otherwise
	function is_illumos
	{
	if [[ $(uname -o) == "illumos" ]]; then
	return 0
	else
	return 1
	fi
	}

	# Determine if this is a FreeBSD test system
	#
	# Return 0 if platform FreeBSD, 1 if otherwise

	function is_freebsd
	{
	if [[ $(uname -o) == "FreeBSD" ]]; then
	return 0
	else
	return 1
	fi
	}

	# Determine if this is a DilOS test system
	#
	# Return 0 if platform DilOS, 1 if otherwise

	function is_dilos
	{
	typeset ID=""
	[[ -f /etc/os-release ]] && . /etc/os-release
	if [[ $ID == "dilos" ]]; then
	return 0
	else
	return 1
	fi
	}

	# Determine if this is a 32-bit system
	#
	# Return 0 if platform is 32-bit, 1 if otherwise

	function is_32bit
	{
	if [[ $(getconf LONG_BIT) == "32" ]]; then
	return 0
	else
	return 1
	fi
	}

	# Determine if kmemleak is enabled
	#
	# Return 0 if kmemleak is enabled, 1 if otherwise

	function is_kmemleak
	{
	if is_linux && [[ -e /sys/kernel/debug/kmemleak ]]; then
	return 0
	else
	return 1
	fi
	}

	# Determine whether a dataset is mounted
	#
	# $1 dataset name
	# $2 filesystem type; optional - defaulted to zfs
	#
	# Return 0 if dataset is mounted; 1 if unmounted; 2 on error

	function ismounted
	{
	typeset fstype=$2
	[[ -z $fstype ]] && fstype=zfs
	typeset out dir name ret

	case $fstype in
	zfs)
	if [[ "$1" == "/"* ]] ; then
	for out in $(zfs mount \| awk '{print $2}'); do
	[[ $1 == $out ]] && return 0
	done
	else
	for out in $(zfs mount \| awk '{print $1}'); do
	[[ $1 == $out ]] && return 0
	done
	fi
	;;
	ufs\|nfs)
	if is_freebsd; then
	mount -pt $fstype \| while read dev dir _t _flags; do
	[[ "$1" == "$dev" \|\| "$1" == "$dir" ]] && return 0
	done
	else
	out=$(df -F $fstype $1 2>/dev/null)
	ret=$?
	(($ret != 0)) && return $ret

	dir=${out%%\(*}
	dir=${dir%% *}
	name=${out##*\(}
	name=${name%%\)*}
	name=${name%% *}

	[[ "$1" == "$dir" \|\| "$1" == "$name" ]] && return 0
	fi
	;;
	ext*)
	out=$(df -t $fstype $1 2>/dev/null)
	return $?
	;;
	zvol)
	if [[ -L "$ZVOL_DEVDIR/$1" ]]; then
	link=$(readlink -f $ZVOL_DEVDIR/$1)
	[[ -n "$link" ]] && \
	mount \| grep -q "^$link" && \
	return 0
	fi
	;;
	esac

	return 1
	}

	# Return 0 if a dataset is mounted; 1 otherwise
	#
	# $1 dataset name
	# $2 filesystem type; optional - defaulted to zfs

	function mounted
	{
	ismounted $1 $2
	(($? == 0)) && return 0
	return 1
	}

	# Return 0 if a dataset is unmounted; 1 otherwise
	#
	# $1 dataset name
	# $2 filesystem type; optional - defaulted to zfs

	function unmounted
	{
	ismounted $1 $2
	(($? == 1)) && return 0
	return 1
	}

	# split line on ","
	#
	# $1 - line to split

	function splitline
	{
	echo $1 \| sed "s/,/ /g"
	}

	function default_setup
	{
	default_setup_noexit "$@"

	log_pass
	}

	function default_setup_no_mountpoint
	{
	default_setup_noexit "$1" "$2" "$3" "yes"

	log_pass
	}

	#
	# Given a list of disks, setup storage pools and datasets.
	#
	function default_setup_noexit
	{
	typeset disklist=$1
	typeset container=$2
	typeset volume=$3
	typeset no_mountpoint=$4
	log_note begin default_setup_noexit

	if is_global_zone; then
	if poolexists $TESTPOOL ; then
	destroy_pool $TESTPOOL
	fi
	[[ -d /$TESTPOOL ]] && rm -rf /$TESTPOOL
	log_must zpool create -f $TESTPOOL $disklist
	else
	reexport_pool
	fi

	rm -rf $TESTDIR \|\| log_unresolved Could not remove $TESTDIR
	mkdir -p $TESTDIR \|\| log_unresolved Could not create $TESTDIR

	log_must zfs create $TESTPOOL/$TESTFS
	if [[ -z $no_mountpoint ]]; then
	log_must zfs set mountpoint=$TESTDIR $TESTPOOL/$TESTFS
	fi

	if [[ -n $container ]]; then
	rm -rf $TESTDIR1 \|\| \
	log_unresolved Could not remove $TESTDIR1
	mkdir -p $TESTDIR1 \|\| \
	log_unresolved Could not create $TESTDIR1

	log_must zfs create $TESTPOOL/$TESTCTR
	log_must zfs set canmount=off $TESTPOOL/$TESTCTR
	log_must zfs create $TESTPOOL/$TESTCTR/$TESTFS1
	if [[ -z $no_mountpoint ]]; then
	log_must zfs set mountpoint=$TESTDIR1 \
	$TESTPOOL/$TESTCTR/$TESTFS1
	fi
	fi

	if [[ -n $volume ]]; then
	if is_global_zone ; then
	log_must zfs create -V $VOLSIZE $TESTPOOL/$TESTVOL
	block_device_wait
	else
	log_must zfs create $TESTPOOL/$TESTVOL
	fi
	fi
	}

	#
	# Given a list of disks, setup a storage pool, file system and
	# a container.
	#
	function default_container_setup
	{
	typeset disklist=$1

	default_setup "$disklist" "true"
	}

	#
	# Given a list of disks, setup a storage pool,file system
	# and a volume.
	#
	function default_volume_setup
	{
	typeset disklist=$1

	default_setup "$disklist" "" "true"
	}

	#
	# Given a list of disks, setup a storage pool,file system,
	# a container and a volume.
	#
	function default_container_volume_setup
	{
	typeset disklist=$1

	default_setup "$disklist" "true" "true"
	}

	#
	# Create a snapshot on a filesystem or volume. Defaultly create a snapshot on
	# filesystem
	#
	# $1 Existing filesystem or volume name. Default, $TESTPOOL/$TESTFS
	# $2 snapshot name. Default, $TESTSNAP
	#
	function create_snapshot
	{
	typeset fs_vol=${1:-$TESTPOOL/$TESTFS}
	typeset snap=${2:-$TESTSNAP}

	[[ -z $fs_vol ]] && log_fail "Filesystem or volume's name is undefined."
	[[ -z $snap ]] && log_fail "Snapshot's name is undefined."

	if snapexists $fs_vol@$snap; then
	log_fail "$fs_vol@$snap already exists."
	fi
	datasetexists $fs_vol \|\| \
	log_fail "$fs_vol must exist."

	log_must zfs snapshot $fs_vol@$snap
	}

	#
	# Create a clone from a snapshot, default clone name is $TESTCLONE.
	#
	# $1 Existing snapshot, $TESTPOOL/$TESTFS@$TESTSNAP is default.
	# $2 Clone name, $TESTPOOL/$TESTCLONE is default.
	#
	function create_clone # snapshot clone
	{
	typeset snap=${1:-$TESTPOOL/$TESTFS@$TESTSNAP}
	typeset clone=${2:-$TESTPOOL/$TESTCLONE}

	[[ -z $snap ]] && \
	log_fail "Snapshot name is undefined."
	[[ -z $clone ]] && \
	log_fail "Clone name is undefined."

	log_must zfs clone $snap $clone
	}

	#
	# Create a bookmark of the given snapshot. Defaultly create a bookmark on
	# filesystem.
	#
	# $1 Existing filesystem or volume name. Default, $TESTFS
	# $2 Existing snapshot name. Default, $TESTSNAP
	# $3 bookmark name. Default, $TESTBKMARK
	#
	function create_bookmark
	{
	typeset fs_vol=${1:-$TESTFS}
	typeset snap=${2:-$TESTSNAP}
	typeset bkmark=${3:-$TESTBKMARK}

	[[ -z $fs_vol ]] && log_fail "Filesystem or volume's name is undefined."
	[[ -z $snap ]] && log_fail "Snapshot's name is undefined."
	[[ -z $bkmark ]] && log_fail "Bookmark's name is undefined."

	if bkmarkexists $fs_vol#$bkmark; then
	log_fail "$fs_vol#$bkmark already exists."
	fi
	datasetexists $fs_vol \|\| \
	log_fail "$fs_vol must exist."
	snapexists $fs_vol@$snap \|\| \
	log_fail "$fs_vol@$snap must exist."

	log_must zfs bookmark $fs_vol@$snap $fs_vol#$bkmark
	}

	#
	# Create a temporary clone result of an interrupted resumable 'zfs receive'
	# $1 Destination filesystem name. Must not exist, will be created as the result
	# of this function along with its %recv temporary clone
	# $2 Source filesystem name. Must not exist, will be created and destroyed
	#
	function create_recv_clone
	{
	typeset recvfs="$1"
	typeset sendfs="${2:-$TESTPOOL/create_recv_clone}"
	typeset snap="$sendfs@snap1"
	typeset incr="$sendfs@snap2"
	typeset mountpoint="$TESTDIR/create_recv_clone"
	typeset sendfile="$TESTDIR/create_recv_clone.zsnap"

	[[ -z $recvfs ]] && log_fail "Recv filesystem's name is undefined."

	datasetexists $recvfs && log_fail "Recv filesystem must not exist."
	datasetexists $sendfs && log_fail "Send filesystem must not exist."

	log_must zfs create -o mountpoint="$mountpoint" $sendfs
	log_must zfs snapshot $snap
	log_must eval "zfs send $snap \| zfs recv -u $recvfs"
	log_must mkfile 1m "$mountpoint/data"
	log_must zfs snapshot $incr
	log_must eval "zfs send -i $snap $incr \| dd bs=10K count=1 \
	iflag=fullblock > $sendfile"
	log_mustnot eval "zfs recv -su $recvfs < $sendfile"
	destroy_dataset "$sendfs" "-r"
	log_must rm -f "$sendfile"

	if [[ $(get_prop 'inconsistent' "$recvfs/%recv") -ne 1 ]]; then
	log_fail "Error creating temporary $recvfs/%recv clone"
	fi
	}

	function default_mirror_setup
	{
	default_mirror_setup_noexit $1 $2 $3

	log_pass
	}

	#
	# Given a pair of disks, set up a storage pool and dataset for the mirror
	# @parameters: $1 the primary side of the mirror
	# $2 the secondary side of the mirror
	# @uses: ZPOOL ZFS TESTPOOL TESTFS
	function default_mirror_setup_noexit
	{
	readonly func="default_mirror_setup_noexit"
	typeset primary=$1
	typeset secondary=$2

	[[ -z $primary ]] && \
	log_fail "$func: No parameters passed"
	[[ -z $secondary ]] && \
	log_fail "$func: No secondary partition passed"
	[[ -d /$TESTPOOL ]] && rm -rf /$TESTPOOL
	log_must zpool create -f $TESTPOOL mirror $@
	log_must zfs create $TESTPOOL/$TESTFS
	log_must zfs set mountpoint=$TESTDIR $TESTPOOL/$TESTFS
	}

	#
	# create a number of mirrors.
	# We create a number($1) of 2 way mirrors using the pairs of disks named
	# on the command line. These mirrors are not mounted
	# @parameters: $1 the number of mirrors to create
	# $... the devices to use to create the mirrors on
	# @uses: ZPOOL ZFS TESTPOOL
	function setup_mirrors
	{
	typeset -i nmirrors=$1

	shift
	while ((nmirrors > 0)); do
	log_must test -n "$1" -a -n "$2"
	[[ -d /$TESTPOOL$nmirrors ]] && rm -rf /$TESTPOOL$nmirrors
	log_must zpool create -f $TESTPOOL$nmirrors mirror $1 $2
	shift 2
	((nmirrors = nmirrors - 1))
	done
	}

	#
	# create a number of raidz pools.
	# We create a number($1) of 2 raidz pools using the pairs of disks named
	# on the command line. These pools are not mounted
	# @parameters: $1 the number of pools to create
	# $... the devices to use to create the pools on
	# @uses: ZPOOL ZFS TESTPOOL
	function setup_raidzs
	{
	typeset -i nraidzs=$1

	shift
	while ((nraidzs > 0)); do
	log_must test -n "$1" -a -n "$2"
	[[ -d /$TESTPOOL$nraidzs ]] && rm -rf /$TESTPOOL$nraidzs
	log_must zpool create -f $TESTPOOL$nraidzs raidz $1 $2
	shift 2
	((nraidzs = nraidzs - 1))
	done
	}

	#
	# Destroy the configured testpool mirrors.
	# the mirrors are of the form ${TESTPOOL}{number}
	# @uses: ZPOOL ZFS TESTPOOL
	function destroy_mirrors
	{
	default_cleanup_noexit

	log_pass
	}

	#
	# Given a minimum of two disks, set up a storage pool and dataset for the raid-z
	# $1 the list of disks
	#
	function default_raidz_setup
	{
	typeset disklist="$*"
	disks=(${disklist[*]})

	if [[ ${#disks[*]} -lt 2 ]]; then
	log_fail "A raid-z requires a minimum of two disks."
	fi

	[[ -d /$TESTPOOL ]] && rm -rf /$TESTPOOL
	log_must zpool create -f $TESTPOOL raidz $disklist
	log_must zfs create $TESTPOOL/$TESTFS
	log_must zfs set mountpoint=$TESTDIR $TESTPOOL/$TESTFS

	log_pass
	}

	#
	# Common function used to cleanup storage pools and datasets.
	#
	# Invoked at the start of the test suite to ensure the system
	# is in a known state, and also at the end of each set of
	# sub-tests to ensure errors from one set of tests doesn't
	# impact the execution of the next set.

	function default_cleanup
	{
	default_cleanup_noexit

	log_pass
	}

	#
	# Utility function used to list all available pool names.
	#
	# NOTE: $KEEP is a variable containing pool names, separated by a newline
	# character, that must be excluded from the returned list.
	#
	function get_all_pools
	{
	zpool list -H -o name \| grep -Fvx "$KEEP" \| grep -v "$NO_POOLS"
	}

	function default_cleanup_noexit
	{
	typeset pool=""
	#
	# Destroying the pool will also destroy any
	# filesystems it contains.
	#
	if is_global_zone; then
	zfs unmount -a > /dev/null 2>&1
	ALL_POOLS=$(get_all_pools)
	# Here, we loop through the pools we're allowed to
	# destroy, only destroying them if it's safe to do
	# so.
	while [ ! -z ${ALL_POOLS} ]
	do
	for pool in ${ALL_POOLS}
	do
	if safe_to_destroy_pool $pool ;
	then
	destroy_pool $pool
	fi
	done
	ALL_POOLS=$(get_all_pools)
	done

	zfs mount -a
	else
	typeset fs=""
	for fs in $(zfs list -H -o name \
	\| grep "^$ZONE_POOL/$ZONE_CTR[01234]/"); do
	destroy_dataset "$fs" "-Rf"
	done

	# Need cleanup here to avoid garbage dir left.
	for fs in $(zfs list -H -o name); do
	[[ $fs == /$ZONE_POOL ]] && continue
	[[ -d $fs ]] && log_must rm -rf $fs/*
	done

	#
	# Reset the $ZONE_POOL/$ZONE_CTR[01234] file systems property to
	# the default value
	#
	for fs in $(zfs list -H -o name); do
	if [[ $fs == $ZONE_POOL/$ZONE_CTR[01234] ]]; then
	log_must zfs set reservation=none $fs
	log_must zfs set recordsize=128K $fs
	log_must zfs set mountpoint=/$fs $fs
	typeset enc=""
	enc=$(get_prop encryption $fs)
	if [[ $? -ne 0 ]] \|\| [[ -z "$enc" ]] \|\| \
	[[ "$enc" == "off" ]]; then
	log_must zfs set checksum=on $fs
	fi
	log_must zfs set compression=off $fs
	log_must zfs set atime=on $fs
	log_must zfs set devices=off $fs
	log_must zfs set exec=on $fs
	log_must zfs set setuid=on $fs
	log_must zfs set readonly=off $fs
	log_must zfs set snapdir=hidden $fs
	log_must zfs set aclmode=groupmask $fs
	log_must zfs set aclinherit=secure $fs
	fi
	done
	fi

	[[ -d $TESTDIR ]] && \
	log_must rm -rf $TESTDIR

	disk1=${DISKS%% *}
	if is_mpath_device $disk1; then
	delete_partitions
	fi

	rm -f $TEST_BASE_DIR/{err,out}
	}


	#
	# Common function used to cleanup storage pools, file systems
	# and containers.
	#
	function default_container_cleanup
	{
	if ! is_global_zone; then
	reexport_pool
	fi

	ismounted $TESTPOOL/$TESTCTR/$TESTFS1
	[[ $? -eq 0 ]] && \
	log_must zfs unmount $TESTPOOL/$TESTCTR/$TESTFS1

	destroy_dataset "$TESTPOOL/$TESTCTR/$TESTFS1" "-R"
	destroy_dataset "$TESTPOOL/$TESTCTR" "-Rf"

	[[ -e $TESTDIR1 ]] && \
	log_must rm -rf $TESTDIR1 > /dev/null 2>&1

	default_cleanup
	}

	#
	# Common function used to cleanup snapshot of file system or volume. Default to
	# delete the file system's snapshot
	#
	# $1 snapshot name
	#
	function destroy_snapshot
	{
	typeset snap=${1:-$TESTPOOL/$TESTFS@$TESTSNAP}

	if ! snapexists $snap; then
	log_fail "'$snap' does not exist."
	fi

	#
	# For the sake of the value which come from 'get_prop' is not equal
	# to the really mountpoint when the snapshot is unmounted. So, firstly
	# check and make sure this snapshot's been mounted in current system.
	#
	typeset mtpt=""
	if ismounted $snap; then
	mtpt=$(get_prop mountpoint $snap)
	(($? != 0)) && \
	log_fail "get_prop mountpoint $snap failed."
	fi

	destroy_dataset "$snap"
	[[ $mtpt != "" && -d $mtpt ]] && \
	log_must rm -rf $mtpt
	}

	#
	# Common function used to cleanup clone.
	#
	# $1 clone name
	#
	function destroy_clone
	{
	typeset clone=${1:-$TESTPOOL/$TESTCLONE}

	if ! datasetexists $clone; then
	log_fail "'$clone' does not existed."
	fi

	# With the same reason in destroy_snapshot
	typeset mtpt=""
	if ismounted $clone; then
	mtpt=$(get_prop mountpoint $clone)
	(($? != 0)) && \
	log_fail "get_prop mountpoint $clone failed."
	fi

	destroy_dataset "$clone"
	[[ $mtpt != "" && -d $mtpt ]] && \
	log_must rm -rf $mtpt
	}

	#
	# Common function used to cleanup bookmark of file system or volume. Default
	# to delete the file system's bookmark.
	#
	# $1 bookmark name
	#
	function destroy_bookmark
	{
	typeset bkmark=${1:-$TESTPOOL/$TESTFS#$TESTBKMARK}

	if ! bkmarkexists $bkmark; then
	log_fail "'$bkmarkp' does not existed."
	fi

	destroy_dataset "$bkmark"
	}

	# Return 0 if a snapshot exists; $? otherwise
	#
	# $1 - snapshot name

	function snapexists
	{
	zfs list -H -t snapshot "$1" > /dev/null 2>&1
	return $?
	}

	#
	# Return 0 if a bookmark exists; $? otherwise
	#
	# $1 - bookmark name
	#
	function bkmarkexists
	{
	zfs list -H -t bookmark "$1" > /dev/null 2>&1
	return $?
	}

	#
	# Return 0 if a hold exists; $? otherwise
	#
	# $1 - hold tag
	# $2 - snapshot name
	#
	function holdexists
	{
	zfs holds "$2" \| awk '{ print $2 }' \| grep "$1" > /dev/null 2>&1
	return $?
	}

	#
	# Set a property to a certain value on a dataset.
	# Sets a property of the dataset to the value as passed in.
	# @param:
	# $1 dataset who's property is being set
	# $2 property to set
	# $3 value to set property to
	# @return:
	# 0 if the property could be set.
	# non-zero otherwise.
	# @use: ZFS
	#
	function dataset_setprop
	{
	typeset fn=dataset_setprop

	if (($# < 3)); then
	log_note "$fn: Insufficient parameters (need 3, had $#)"
	return 1
	fi
	typeset output=
	output=$(zfs set $2=$3 $1 2>&1)
	typeset rv=$?
	if ((rv != 0)); then
	log_note "Setting property on $1 failed."
	log_note "property $2=$3"
	log_note "Return Code: $rv"
	log_note "Output: $output"
	return $rv
	fi
	return 0
	}

	#
	# Assign suite defined dataset properties.
	# This function is used to apply the suite's defined default set of
	# properties to a dataset.
	# @parameters: $1 dataset to use
	# @uses: ZFS COMPRESSION_PROP CHECKSUM_PROP
	# @returns:
	# 0 if the dataset has been altered.
	# 1 if no pool name was passed in.
	# 2 if the dataset could not be found.
	# 3 if the dataset could not have it's properties set.
	#
	function dataset_set_defaultproperties
	{
	typeset dataset="$1"

	[[ -z $dataset ]] && return 1

	typeset confset=
	typeset -i found=0
	for confset in $(zfs list); do
	if [[ $dataset = $confset ]]; then
	found=1
	break
	fi
	done
	[[ $found -eq 0 ]] && return 2
	if [[ -n $COMPRESSION_PROP ]]; then
	dataset_setprop $dataset compression $COMPRESSION_PROP \|\| \
	return 3
	log_note "Compression set to '$COMPRESSION_PROP' on $dataset"
	fi
	if [[ -n $CHECKSUM_PROP ]]; then
	dataset_setprop $dataset checksum $CHECKSUM_PROP \|\| \
	return 3
	log_note "Checksum set to '$CHECKSUM_PROP' on $dataset"
	fi
	return 0
	}

	#
	# Check a numeric assertion
	# @parameter: $@ the assertion to check
	# @output: big loud notice if assertion failed
	# @use: log_fail
	#
	function assert
	{
	(($@)) \|\| log_fail "$@"
	}

	#
	# Function to format partition size of a disk
	# Given a disk cxtxdx reduces all partitions
	# to 0 size
	#
	function zero_partitions #<whole_disk_name>
	{
	typeset diskname=$1
	typeset i

	if is_freebsd; then
	gpart destroy -F $diskname
	elif is_linux; then
	DSK=$DEV_DSKDIR/$diskname
	DSK=$(echo $DSK \| sed -e "s\|//\|/\|g")
	log_must parted $DSK -s -- mklabel gpt
	blockdev --rereadpt $DSK 2>/dev/null
	block_device_wait
	else
	for i in 0 1 3 4 5 6 7
	do
	log_must set_partition $i "" 0mb $diskname
	done
	fi

	return 0
	}

	#
	# Given a slice, size and disk, this function
	# formats the slice to the specified size.
	# Size should be specified with units as per
	# the `format` command requirements eg. 100mb 3gb
	#
	# NOTE: This entire interface is problematic for the Linux parted utility
	# which requires the end of the partition to be specified. It would be
	# best to retire this interface and replace it with something more flexible.
	# At the moment a best effort is made.
	#
	# arguments: <slice_num> <slice_start> <size_plus_units> <whole_disk_name>
	function set_partition
	{
	typeset -i slicenum=$1
	typeset start=$2
	typeset size=$3
	typeset disk=${4#$DEV_DSKDIR/}
	disk=${disk#$DEV_RDSKDIR/}

	case "$(uname)" in
	Linux)
	if [[ -z $size \|\| -z $disk ]]; then
	log_fail "The size or disk name is unspecified."
	fi
	disk=$DEV_DSKDIR/$disk
	typeset size_mb=${size%%[mMgG]}

	size_mb=${size_mb%%[mMgG][bB]}
	if [[ ${size:1:1} == 'g' ]]; then
	((size_mb = size_mb * 1024))
	fi

	# Create GPT partition table when setting slice 0 or
	# when the device doesn't already contain a GPT label.
	parted $disk -s -- print 1 >/dev/null
	typeset ret_val=$?
	if [[ $slicenum -eq 0 \|\| $ret_val -ne 0 ]]; then
	parted $disk -s -- mklabel gpt
	if [[ $? -ne 0 ]]; then
	log_note "Failed to create GPT partition table on $disk"
	return 1
	fi
	fi

	# When no start is given align on the first cylinder.
	if [[ -z "$start" ]]; then
	start=1
	fi

	# Determine the cylinder size for the device and using
	# that calculate the end offset in cylinders.
	typeset -i cly_size_kb=0
	cly_size_kb=$(parted -m $disk -s -- \
	unit cyl print \| head -3 \| tail -1 \| \
	awk -F '[:k.]' '{print $4}')
	((end = (size_mb * 1024 / cly_size_kb) + start))

	parted $disk -s -- \
	mkpart part$slicenum ${start}cyl ${end}cyl
	typeset ret_val=$?
	if [[ $ret_val -ne 0 ]]; then
	log_note "Failed to create partition $slicenum on $disk"
	return 1
	fi

	blockdev --rereadpt $disk 2>/dev/null
	block_device_wait $disk
	;;
	FreeBSD)
	if [[ -z $size \|\| -z $disk ]]; then
	log_fail "The size or disk name is unspecified."
	fi
	disk=$DEV_DSKDIR/$disk

	if [[ $slicenum -eq 0 ]] \|\| ! gpart show $disk >/dev/null 2>&1; then
	gpart destroy -F $disk >/dev/null 2>&1
	gpart create -s GPT $disk
	if [[ $? -ne 0 ]]; then
	log_note "Failed to create GPT partition table on $disk"
	return 1
	fi
	fi

	typeset index=$((slicenum + 1))

	if [[ -n $start ]]; then
	start="-b $start"
	fi
	gpart add -t freebsd-zfs $start -s $size -i $index $disk
	if [[ $ret_val -ne 0 ]]; then
	log_note "Failed to create partition $slicenum on $disk"
	return 1
	fi

	block_device_wait $disk
	;;
	*)
	if [[ -z $slicenum \|\| -z $size \|\| -z $disk ]]; then
	log_fail "The slice, size or disk name is unspecified."
	fi

	typeset format_file=/var/tmp/format_in.$$

	echo "partition" >$format_file
	echo "$slicenum" >> $format_file
	echo "" >> $format_file
	echo "" >> $format_file
	echo "$start" >> $format_file
	echo "$size" >> $format_file
	echo "label" >> $format_file
	echo "" >> $format_file
	echo "q" >> $format_file
	echo "q" >> $format_file

	format -e -s -d $disk -f $format_file
	typeset ret_val=$?
	rm -f $format_file
	;;
	esac

	if [[ $ret_val -ne 0 ]]; then
	log_note "Unable to format $disk slice $slicenum to $size"
	return 1
	fi
	return 0
	}

	#
	# Delete all partitions on all disks - this is specifically for the use of multipath
	# devices which currently can only be used in the test suite as raw/un-partitioned
	# devices (ie a zpool cannot be created on a whole mpath device that has partitions)
	#
	function delete_partitions
	{
	typeset disk

	if [[ -z $DISKSARRAY ]]; then
	DISKSARRAY=$DISKS
	fi

	if is_linux; then
	typeset -i part
	for disk in $DISKSARRAY; do
	for (( part = 1; part < MAX_PARTITIONS; part++ )); do
	typeset partition=${disk}${SLICE_PREFIX}${part}
	parted $DEV_DSKDIR/$disk -s rm $part > /dev/null 2>&1
	if lsblk \| grep -qF ${partition}; then
	log_fail "Partition ${partition} not deleted"
	else
	log_note "Partition ${partition} deleted"
	fi
	done
	done
	elif is_freebsd; then
	for disk in $DISKSARRAY; do
	if gpart destroy -F $disk; then
	log_note "Partitions for ${disk} deleted"
	else
	log_fail "Partitions for ${disk} not deleted"
	fi
	done
	fi
	}

	#
	# Get the end cyl of the given slice
	#
	function get_endslice #<disk> <slice>
	{
	typeset disk=$1
	typeset slice=$2
	if [[ -z $disk \|\| -z $slice ]] ; then
	log_fail "The disk name or slice number is unspecified."
	fi

	case "$(uname)" in
	Linux)
	endcyl=$(parted -s $DEV_DSKDIR/$disk -- unit cyl print \| \
	grep "part${slice}" \| \
	awk '{print $3}' \| \
	sed 's,cyl,,')
	((endcyl = (endcyl + 1)))
	;;
	FreeBSD)
	disk=${disk#/dev/zvol/}
	disk=${disk%p*}
	slice=$((slice + 1))
	endcyl=$(gpart show $disk \| \
	awk -v slice=$slice '$3 == slice { print $1 + $2 }')
	;;
	*)
	disk=${disk#/dev/dsk/}
	disk=${disk#/dev/rdsk/}
	disk=${disk%s*}

	typeset -i ratio=0
	ratio=$(prtvtoc /dev/rdsk/${disk}s2 \| \
	grep "sectors\/cylinder" \| \
	awk '{print $2}')

	if ((ratio == 0)); then
	return
	fi

	typeset -i endcyl=$(prtvtoc -h /dev/rdsk/${disk}s2 \|
	nawk -v token="$slice" '{if ($1==token) print $6}')

	((endcyl = (endcyl + 1) / ratio))
	;;
	esac

	echo $endcyl
	}


	#
	# Given a size,disk and total slice number, this function formats the
	# disk slices from 0 to the total slice number with the same specified
	# size.
	#
	function partition_disk #<slice_size> <whole_disk_name> <total_slices>
	{
	typeset -i i=0
	typeset slice_size=$1
	typeset disk_name=$2
	typeset total_slices=$3
	typeset cyl

	zero_partitions $disk_name
	while ((i < $total_slices)); do
	if ! is_linux; then
	if ((i == 2)); then
	((i = i + 1))
	continue
	fi
	fi
	log_must set_partition $i "$cyl" $slice_size $disk_name
	cyl=$(get_endslice $disk_name $i)
	((i = i+1))
	done
	}

	#
	# This function continues to write to a filenum number of files into dirnum
	# number of directories until either file_write returns an error or the
	# maximum number of files per directory have been written.
	#
	# Usage:
	# fill_fs [destdir] [dirnum] [filenum] [bytes] [num_writes] [data]
	#
	# Return value: 0 on success
	# non 0 on error
	#
	# Where :
	# destdir: is the directory where everything is to be created under
	# dirnum: the maximum number of subdirectories to use, -1 no limit
	# filenum: the maximum number of files per subdirectory
	# bytes: number of bytes to write
	# num_writes: number of types to write out bytes
	# data: the data that will be written
	#
	# E.g.
	# fill_fs /testdir 20 25 1024 256 0
	#
	# Note: bytes * num_writes equals the size of the testfile
	#
	function fill_fs # destdir dirnum filenum bytes num_writes data
	{
	typeset destdir=${1:-$TESTDIR}
	typeset -i dirnum=${2:-50}
	typeset -i filenum=${3:-50}
	typeset -i bytes=${4:-8192}
	typeset -i num_writes=${5:-10240}
	typeset data=${6:-0}

	mkdir -p $destdir/{1..$dirnum}
	for f in $destdir/{1..$dirnum}/$TESTFILE{1..$filenum}; do
	file_write -o create -f $f -b $bytes -c $num_writes -d $data \
	\|\| return $?
	done
	return 0
	}

	#
	# Simple function to get the specified property. If unable to
	# get the property then exits.
	#
	# Note property is in 'parsable' format (-p)
	#
	function get_prop # property dataset
	{
	typeset prop_val
	typeset prop=$1
	typeset dataset=$2

	prop_val=$(zfs get -pH -o value $prop $dataset 2>/dev/null)
	if [[ $? -ne 0 ]]; then
	log_note "Unable to get $prop property for dataset " \
	"$dataset"
	return 1
	fi

	echo "$prop_val"
	return 0
	}

	#
	# Simple function to get the specified property of pool. If unable to
	# get the property then exits.
	#
	# Note property is in 'parsable' format (-p)
	#
	function get_pool_prop # property pool
	{
	typeset prop_val
	typeset prop=$1
	typeset pool=$2

	if poolexists $pool ; then
	prop_val=$(zpool get -pH $prop $pool 2>/dev/null \| tail -1 \| \
	awk '{print $3}')
	if [[ $? -ne 0 ]]; then
	log_note "Unable to get $prop property for pool " \
	"$pool"
	return 1
	fi
	else
	log_note "Pool $pool not exists."
	return 1
	fi

	echo "$prop_val"
	return 0
	}

	# Return 0 if a pool exists; $? otherwise
	#
	# $1 - pool name

	function poolexists
	{
	typeset pool=$1

	if [[ -z $pool ]]; then
	log_note "No pool name given."
	return 1
	fi

	zpool get name "$pool" > /dev/null 2>&1
	return $?
	}

	# Return 0 if all the specified datasets exist; $? otherwise
	#
	# $1-n dataset name
	function datasetexists
	{
	if (($# == 0)); then
	log_note "No dataset name given."
	return 1
	fi

	while (($# > 0)); do
	zfs get name $1 > /dev/null 2>&1 \|\| \
	return $?
	shift
	done

	return 0
	}

	# return 0 if none of the specified datasets exists, otherwise return 1.
	#
	# $1-n dataset name
	function datasetnonexists
	{
	if (($# == 0)); then
	log_note "No dataset name given."
	return 1
	fi

	while (($# > 0)); do
	zfs list -H -t filesystem,snapshot,volume $1 > /dev/null 2>&1 \
	&& return 1
	shift
	done

	return 0
	}

	function is_shared_freebsd
	{
	typeset fs=$1

	pgrep -q mountd && showmount -E \| grep -qx $fs
	}

	function is_shared_illumos
	{
	typeset fs=$1
	typeset mtpt

	for mtpt in `share \| awk '{print $2}'` ; do
	if [[ $mtpt == $fs ]] ; then
	return 0
	fi
	done

	typeset stat=$(svcs -H -o STA nfs/server:default)
	if [[ $stat != "ON" ]]; then
	log_note "Current nfs/server status: $stat"
	fi

	return 1
	}

	function is_shared_linux
	{
	typeset fs=$1
	typeset mtpt

	for mtpt in `share \| awk '{print $1}'` ; do
	if [[ $mtpt == $fs ]] ; then
	return 0
	fi
	done
	return 1
	}

	#
	# Given a mountpoint, or a dataset name, determine if it is shared via NFS.
	#
	# Returns 0 if shared, 1 otherwise.
	#
	function is_shared
	{
	typeset fs=$1
	typeset mtpt

	if [[ $fs != "/"* ]] ; then
	if datasetnonexists "$fs" ; then
	return 1
	else
	mtpt=$(get_prop mountpoint "$fs")
	case $mtpt in
	none\|legacy\|-) return 1
	;;
	*) fs=$mtpt
	;;
	esac
	fi
	fi

	case $(uname) in
	FreeBSD) is_shared_freebsd "$fs" ;;
	Linux) is_shared_linux "$fs" ;;
	*) is_shared_illumos "$fs" ;;
	esac
	}

	function is_exported_illumos
	{
	typeset fs=$1
	typeset mtpt

	for mtpt in `awk '{print $1}' /etc/dfs/sharetab` ; do
	if [[ $mtpt == $fs ]] ; then
	return 0
	fi
	done

	return 1
	}

	function is_exported_freebsd
	{
	typeset fs=$1
	typeset mtpt

	for mtpt in `awk '{print $1}' /etc/zfs/exports` ; do
	if [[ $mtpt == $fs ]] ; then
	return 0
	fi
	done

	return 1
	}

	function is_exported_linux
	{
	typeset fs=$1
	typeset mtpt

	for mtpt in `awk '{print $1}' /etc/exports.d/zfs.exports` ; do
	if [[ $mtpt == $fs ]] ; then
	return 0
	fi
	done

	return 1
	}

	#
	# Given a mountpoint, or a dataset name, determine if it is exported via
	# the os-specific NFS exports file.
	#
	# Returns 0 if exported, 1 otherwise.
	#
	function is_exported
	{
	typeset fs=$1
	typeset mtpt

	if [[ $fs != "/"* ]] ; then
	if datasetnonexists "$fs" ; then
	return 1
	else
	mtpt=$(get_prop mountpoint "$fs")
	case $mtpt in
	none\|legacy\|-) return 1
	;;
	*) fs=$mtpt
	;;
	esac
	fi
	fi

	case $(uname) in
	FreeBSD) is_exported_freebsd "$fs" ;;
	Linux) is_exported_linux "$fs" ;;
	*) is_exported_illumos "$fs" ;;
	esac
	}

	#
	# Given a dataset name determine if it is shared via SMB.
	#
	# Returns 0 if shared, 1 otherwise.
	#
	function is_shared_smb
	{
	typeset fs=$1
	typeset mtpt

	if datasetnonexists "$fs" ; then
	return 1
	else
	fs=$(echo $fs \| sed 's@/@_@g')
	fi

	if is_linux; then
	for mtpt in `net usershare list \| awk '{print $1}'` ; do
	if [[ $mtpt == $fs ]] ; then
	return 0
	fi
	done
	return 1
	else
	log_note "Currently unsupported by the test framework"
	return 1
	fi
	}

	#
	# Given a mountpoint, determine if it is not shared via NFS.
	#
	# Returns 0 if not shared, 1 otherwise.
	#
	function not_shared
	{
	typeset fs=$1

	is_shared $fs
	if (($? == 0)); then
	return 1
	fi

	return 0
	}

	#
	# Given a dataset determine if it is not shared via SMB.
	#
	# Returns 0 if not shared, 1 otherwise.
	#
	function not_shared_smb
	{
	typeset fs=$1

	is_shared_smb $fs
	if (($? == 0)); then
	return 1
	fi

	return 0
	}

	#
	# Helper function to unshare a mountpoint.
	#
	function unshare_fs #fs
	{
	typeset fs=$1

	is_shared $fs \|\| is_shared_smb $fs
	if (($? == 0)); then
	zfs unshare $fs \|\| log_fail "zfs unshare $fs failed"
	fi

	return 0
	}

	#
	# Helper function to share a NFS mountpoint.
	#
	function share_nfs #fs
	{
	typeset fs=$1

	if is_linux; then
	is_shared $fs
	if (($? != 0)); then
	log_must share "*:$fs"
	fi
	else
	is_shared $fs
	if (($? != 0)); then
	log_must share -F nfs $fs
	fi
	fi

	return 0
	}

	#
	# Helper function to unshare a NFS mountpoint.
	#
	function unshare_nfs #fs
	{
	typeset fs=$1

	if is_linux; then
	is_shared $fs
	if (($? == 0)); then
	log_must unshare -u "*:$fs"
	fi
	else
	is_shared $fs
	if (($? == 0)); then
	log_must unshare -F nfs $fs
	fi
	fi

	return 0
	}

	#
	# Helper function to show NFS shares.
	#
	function showshares_nfs
	{
	if is_linux; then
	share -v
	else
	share -F nfs
	fi

	return 0
	}

	#
	# Helper function to show SMB shares.
	#
	function showshares_smb
	{
	if is_linux; then
	net usershare list
	else
	share -F smb
	fi

	return 0
	}

	function check_nfs
	{
	if is_linux; then
	share -s
	elif is_freebsd; then
	showmount -e
	else
	log_unsupported "Unknown platform"
	fi

	if [[ $? -ne 0 ]]; then
	log_unsupported "The NFS utilities are not installed"
	fi
	}

	#
	# Check NFS server status and trigger it online.
	#
	function setup_nfs_server
	{
	# Cannot share directory in non-global zone.
	#
	if ! is_global_zone; then
	log_note "Cannot trigger NFS server by sharing in LZ."
	return
	fi

	if is_linux; then
	#
	# Re-synchronize /var/lib/nfs/etab with /etc/exports and
	# /etc/exports.d./* to provide a clean test environment.
	#
	log_must share -r

	log_note "NFS server must be started prior to running ZTS."
	return
	elif is_freebsd; then
	kill -s HUP $(cat /var/run/mountd.pid)

	log_note "NFS server must be started prior to running ZTS."
	return
	fi

	typeset nfs_fmri="svc:/network/nfs/server:default"
	if [[ $(svcs -Ho STA $nfs_fmri) != "ON" ]]; then
	#
	# Only really sharing operation can enable NFS server
	# to online permanently.
	#
	typeset dummy=/tmp/dummy

	if [[ -d $dummy ]]; then
	log_must rm -rf $dummy
	fi

	log_must mkdir $dummy
	log_must share $dummy

	#
	# Waiting for fmri's status to be the final status.
	# Otherwise, in transition, an asterisk (*) is appended for
	# instances, unshare will reverse status to 'DIS' again.
	#
	# Waiting for 1's at least.
	#
	log_must sleep 1
	timeout=10
	while [[ timeout -ne 0 && $(svcs -Ho STA $nfs_fmri) == '' ]]
	do
	log_must sleep 1

	((timeout -= 1))
	done

	log_must unshare $dummy
	log_must rm -rf $dummy
	fi

	log_note "Current NFS status: '$(svcs -Ho STA,FMRI $nfs_fmri)'"
	}

	#
	# To verify whether calling process is in global zone
	#
	# Return 0 if in global zone, 1 in non-global zone
	#
	function is_global_zone
	{
	if is_linux \|\| is_freebsd; then
	return 0
	else
	typeset cur_zone=$(zonename 2>/dev/null)
	if [[ $cur_zone != "global" ]]; then
	return 1
	fi
	return 0
	fi
	}

	#
	# Verify whether test is permitted to run from
	# global zone, local zone, or both
	#
	# $1 zone limit, could be "global", "local", or "both"(no limit)
	#
	# Return 0 if permitted, otherwise exit with log_unsupported
	#
	function verify_runnable # zone limit
	{
	typeset limit=$1

	[[ -z $limit ]] && return 0

	if is_global_zone ; then
	case $limit in
	global\|both)
	;;
	local) log_unsupported "Test is unable to run from "\
	"global zone."
	;;
	*) log_note "Warning: unknown limit $limit - " \
	"use both."
	;;
	esac
	else
	case $limit in
	local\|both)
	;;
	global) log_unsupported "Test is unable to run from "\
	"local zone."
	;;
	*) log_note "Warning: unknown limit $limit - " \
	"use both."
	;;
	esac

	reexport_pool
	fi

	return 0
	}

	# Return 0 if create successfully or the pool exists; $? otherwise
	# Note: In local zones, this function should return 0 silently.
	#
	# $1 - pool name
	# $2-n - [keyword] devs_list

	function create_pool #pool devs_list
	{
	typeset pool=${1%%/*}

	shift

	if [[ -z $pool ]]; then
	log_note "Missing pool name."
	return 1
	fi

	if poolexists $pool ; then
	destroy_pool $pool
	fi

	if is_global_zone ; then
	[[ -d /$pool ]] && rm -rf /$pool
	log_must zpool create -f $pool $@
	fi

	return 0
	}

	# Return 0 if destroy successfully or the pool exists; $? otherwise
	# Note: In local zones, this function should return 0 silently.
	#
	# $1 - pool name
	# Destroy pool with the given parameters.

	function destroy_pool #pool
	{
	typeset pool=${1%%/*}
	typeset mtpt

	if [[ -z $pool ]]; then
	log_note "No pool name given."
	return 1
	fi

	if is_global_zone ; then
	if poolexists "$pool" ; then
	mtpt=$(get_prop mountpoint "$pool")

	# At times, syseventd/udev activity can cause attempts
	# to destroy a pool to fail with EBUSY. We retry a few
	# times allowing failures before requiring the destroy
	# to succeed.
	log_must_busy zpool destroy -f $pool

	[[ -d $mtpt ]] && \
	log_must rm -rf $mtpt
	else
	log_note "Pool does not exist. ($pool)"
	return 1
	fi
	fi

	return 0
	}

	# Return 0 if created successfully; $? otherwise
	#
	# $1 - dataset name
	# $2-n - dataset options

	function create_dataset #dataset dataset_options
	{
	typeset dataset=$1

	shift

	if [[ -z $dataset ]]; then
	log_note "Missing dataset name."
	return 1
	fi

	if datasetexists $dataset ; then
	destroy_dataset $dataset
	fi

	log_must zfs create $@ $dataset

	return 0
	}

	# Return 0 if destroy successfully or the dataset exists; $? otherwise
	# Note: In local zones, this function should return 0 silently.
	#
	# $1 - dataset name
	# $2 - custom arguments for zfs destroy
	# Destroy dataset with the given parameters.

	function destroy_dataset #dataset #args
	{
	typeset dataset=$1
	typeset mtpt
	typeset args=${2:-""}

	if [[ -z $dataset ]]; then
	log_note "No dataset name given."
	return 1
	fi

	if is_global_zone ; then
	if datasetexists "$dataset" ; then
	mtpt=$(get_prop mountpoint "$dataset")
	log_must_busy zfs destroy $args $dataset

	[[ -d $mtpt ]] && \
	log_must rm -rf $mtpt
	else
	log_note "Dataset does not exist. ($dataset)"
	return 1
	fi
	fi

	return 0
	}

	#
	# Firstly, create a pool with 5 datasets. Then, create a single zone and
	# export the 5 datasets to it. In addition, we also add a ZFS filesystem
	# and a zvol device to the zone.
	#
	# $1 zone name
	# $2 zone root directory prefix
	# $3 zone ip
	#
	function zfs_zones_setup #zone_name zone_root zone_ip
	{
	typeset zone_name=${1:-$(hostname)-z}
	typeset zone_root=${2:-"/zone_root"}
	typeset zone_ip=${3:-"10.1.1.10"}
	typeset prefix_ctr=$ZONE_CTR
	typeset pool_name=$ZONE_POOL
	typeset -i cntctr=5
	typeset -i i=0

	# Create pool and 5 container within it
	#
	[[ -d /$pool_name ]] && rm -rf /$pool_name
	log_must zpool create -f $pool_name $DISKS
	while ((i < cntctr)); do
	log_must zfs create $pool_name/$prefix_ctr$i
	((i += 1))
	done

	# create a zvol
	log_must zfs create -V 1g $pool_name/zone_zvol
	block_device_wait

	#
	# If current system support slog, add slog device for pool
	#
	if verify_slog_support ; then
	typeset sdevs="$TEST_BASE_DIR/sdev1 $TEST_BASE_DIR/sdev2"
	log_must mkfile $MINVDEVSIZE $sdevs
	log_must zpool add $pool_name log mirror $sdevs
	fi

	# this isn't supported just yet.
	# Create a filesystem. In order to add this to
	# the zone, it must have it's mountpoint set to 'legacy'
	# log_must zfs create $pool_name/zfs_filesystem
	# log_must zfs set mountpoint=legacy $pool_name/zfs_filesystem

	[[ -d $zone_root ]] && \
	log_must rm -rf $zone_root/$zone_name
	[[ ! -d $zone_root ]] && \
	log_must mkdir -p -m 0700 $zone_root/$zone_name

	# Create zone configure file and configure the zone
	#
	typeset zone_conf=/tmp/zone_conf.$$
	echo "create" > $zone_conf
	echo "set zonepath=$zone_root/$zone_name" >> $zone_conf
	echo "set autoboot=true" >> $zone_conf
	i=0
	while ((i < cntctr)); do
	echo "add dataset" >> $zone_conf
	echo "set name=$pool_name/$prefix_ctr$i" >> \
	$zone_conf
	echo "end" >> $zone_conf
	((i += 1))
	done

	# add our zvol to the zone
	echo "add device" >> $zone_conf
	echo "set match=/dev/zvol/dsk/$pool_name/zone_zvol" >> $zone_conf
	echo "end" >> $zone_conf

	# add a corresponding zvol rdsk to the zone
	echo "add device" >> $zone_conf
	echo "set match=$ZVOL_RDEVDIR/$pool_name/zone_zvol" >> $zone_conf
	echo "end" >> $zone_conf

	# once it's supported, we'll add our filesystem to the zone
	# echo "add fs" >> $zone_conf
	# echo "set type=zfs" >> $zone_conf
	# echo "set special=$pool_name/zfs_filesystem" >> $zone_conf
	# echo "set dir=/export/zfs_filesystem" >> $zone_conf
	# echo "end" >> $zone_conf

	echo "verify" >> $zone_conf
	echo "commit" >> $zone_conf
	log_must zonecfg -z $zone_name -f $zone_conf
	log_must rm -f $zone_conf

	# Install the zone
	zoneadm -z $zone_name install
	if (($? == 0)); then
	log_note "SUCCESS: zoneadm -z $zone_name install"
	else
	log_fail "FAIL: zoneadm -z $zone_name install"
	fi

	# Install sysidcfg file
	#
	typeset sysidcfg=$zone_root/$zone_name/root/etc/sysidcfg
	echo "system_locale=C" > $sysidcfg
	echo "terminal=dtterm" >> $sysidcfg
	echo "network_interface=primary {" >> $sysidcfg
	echo "hostname=$zone_name" >> $sysidcfg
	echo "}" >> $sysidcfg
	echo "name_service=NONE" >> $sysidcfg
	echo "root_password=mo791xfZ/SFiw" >> $sysidcfg
	echo "security_policy=NONE" >> $sysidcfg
	echo "timezone=US/Eastern" >> $sysidcfg

	# Boot this zone
	log_must zoneadm -z $zone_name boot
	}

	#
	# Reexport TESTPOOL & TESTPOOL(1-4)
	#
	function reexport_pool
	{
	typeset -i cntctr=5
	typeset -i i=0

	while ((i < cntctr)); do
	if ((i == 0)); then
	TESTPOOL=$ZONE_POOL/$ZONE_CTR$i
	if ! ismounted $TESTPOOL; then
	log_must zfs mount $TESTPOOL
	fi
	else
	eval TESTPOOL$i=$ZONE_POOL/$ZONE_CTR$i
	if eval ! ismounted \$TESTPOOL$i; then
	log_must eval zfs mount \$TESTPOOL$i
	fi
	fi
	((i += 1))
	done
	}

	#
	# Verify a given disk or pool state
	#
	# Return 0 is pool/disk matches expected state, 1 otherwise
	#
	function check_state # pool disk state{online,offline,degraded}
	{
	typeset pool=$1
	typeset disk=${2#$DEV_DSKDIR/}
	typeset state=$3

	[[ -z $pool ]] \|\| [[ -z $state ]] \
	&& log_fail "Arguments invalid or missing"

	if [[ -z $disk ]]; then
	#check pool state only
	zpool get -H -o value health $pool \
	\| grep -i "$state" > /dev/null 2>&1
	else
	zpool status -v $pool \| grep "$disk" \
	\| grep -i "$state" > /dev/null 2>&1
	fi

	return $?
	}

	#
	# Get the mountpoint of snapshot
	# For the snapshot use <mp_filesystem>/.zfs/snapshot/<snap>
	# as its mountpoint
	#
	function snapshot_mountpoint
	{
	typeset dataset=${1:-$TESTPOOL/$TESTFS@$TESTSNAP}

	if [[ $dataset != @ ]]; then
	log_fail "Error name of snapshot '$dataset'."
	fi

	typeset fs=${dataset%@*}
	typeset snap=${dataset#*@}

	if [[ -z $fs \|\| -z $snap ]]; then
	log_fail "Error name of snapshot '$dataset'."
	fi

	echo $(get_prop mountpoint $fs)/.zfs/snapshot/$snap
	}

	#
	# Given a device and 'ashift' value verify it's correctly set on every label
	#
	function verify_ashift # device ashift
	{
	typeset device="$1"
	typeset ashift="$2"

	zdb -e -lll $device \| awk -v ashift=$ashift '/ashift: / {
	if (ashift != $2)
	exit 1;
	else
	count++;
	} END {
	if (count != 4)
	exit 1;
	else
	exit 0;
	}'

	return $?
	}

	#
	# Given a pool and file system, this function will verify the file system
	# using the zdb internal tool. Note that the pool is exported and imported
	# to ensure it has consistent state.
	#
	function verify_filesys # pool filesystem dir
	{
	typeset pool="$1"
	typeset filesys="$2"
	typeset zdbout="/tmp/zdbout.$$"

	shift
	shift
	typeset dirs=$@
	typeset search_path=""

	log_note "Calling zdb to verify filesystem '$filesys'"
	zfs unmount -a > /dev/null 2>&1
	log_must zpool export $pool

	if [[ -n $dirs ]] ; then
	for dir in $dirs ; do
	search_path="$search_path -d $dir"
	done
	fi

	log_must zpool import $search_path $pool

	zdb -cudi $filesys > $zdbout 2>&1
	if [[ $? != 0 ]]; then
	log_note "Output: zdb -cudi $filesys"
	cat $zdbout
	log_fail "zdb detected errors with: '$filesys'"
	fi

	log_must zfs mount -a
	log_must rm -rf $zdbout
	}

	#
	# Given a pool issue a scrub and verify that no checksum errors are reported.
	#
	function verify_pool
	{
	typeset pool=${1:-$TESTPOOL}

	log_must zpool scrub $pool
	log_must wait_scrubbed $pool

	typeset -i cksum=$(zpool status $pool \| awk '
	!NF { isvdev = 0 }
	isvdev { errors += $NF }
	/CKSUM$/ { isvdev = 1 }
	END { print errors }
	')
	if [[ $cksum != 0 ]]; then
	log_must zpool status -v
	log_fail "Unexpected CKSUM errors found on $pool ($cksum)"
	fi
	}

	#
	# Given a pool, and this function list all disks in the pool
	#
	function get_disklist # pool
	{
	typeset disklist=""

	disklist=$(zpool iostat -v $1 \| nawk '(NR >4) {print $1}' \| \
	grep -v "\-\-\-\-\-" \| \
	egrep -v -e "^(mirror\|raidz[1-3]\|spare\|log\|cache\|special\|dedup)$")

	echo $disklist
	}

	#
	# Given a pool, and this function list all disks in the pool with their full
	# path (like "/dev/sda" instead of "sda").
	#
	function get_disklist_fullpath # pool
	{
	args="-P $1"
	get_disklist $args
	}



	# /**
	# This function kills a given list of processes after a time period. We use
	# this in the stress tests instead of STF_TIMEOUT so that we can have processes
	# run for a fixed amount of time, yet still pass. Tests that hit STF_TIMEOUT
	# would be listed as FAIL, which we don't want : we're happy with stress tests
	# running for a certain amount of time, then finishing.
	#
	# @param $1 the time in seconds after which we should terminate these processes
	# @param $2..$n the processes we wish to terminate.
	# */
	function stress_timeout
	{
	typeset -i TIMEOUT=$1
	shift
	typeset cpids="$@"

	log_note "Waiting for child processes($cpids). " \
	"It could last dozens of minutes, please be patient ..."
	log_must sleep $TIMEOUT

	log_note "Killing child processes after ${TIMEOUT} stress timeout."
	typeset pid
	for pid in $cpids; do
	ps -p $pid > /dev/null 2>&1
	if (($? == 0)); then
	log_must kill -USR1 $pid
	fi
	done
	}

	#
	# Verify a given hotspare disk is inuse or avail
	#
	# Return 0 is pool/disk matches expected state, 1 otherwise
	#
	function check_hotspare_state # pool disk state{inuse,avail}
	{
	typeset pool=$1
	typeset disk=${2#$DEV_DSKDIR/}
	typeset state=$3

	cur_state=$(get_device_state $pool $disk "spares")

	if [[ $state != ${cur_state} ]]; then
	return 1
	fi
	return 0
	}

	#
	# Wait until a hotspare transitions to a given state or times out.
	#
	# Return 0 when pool/disk matches expected state, 1 on timeout.
	#
	function wait_hotspare_state # pool disk state timeout
	{
	typeset pool=$1
	typeset disk=${2#*$DEV_DSKDIR/}
	typeset state=$3
	typeset timeout=${4:-60}
	typeset -i i=0

	while [[ $i -lt $timeout ]]; do
	if check_hotspare_state $pool $disk $state; then
	return 0
	fi

	i=$((i+1))
	sleep 1
	done

	return 1
	}

	#
	# Verify a given slog disk is inuse or avail
	#
	# Return 0 is pool/disk matches expected state, 1 otherwise
	#
	function check_slog_state # pool disk state{online,offline,unavail}
	{
	typeset pool=$1
	typeset disk=${2#$DEV_DSKDIR/}
	typeset state=$3

	cur_state=$(get_device_state $pool $disk "logs")

	if [[ $state != ${cur_state} ]]; then
	return 1
	fi
	return 0
	}

	#
	# Verify a given vdev disk is inuse or avail
	#
	# Return 0 is pool/disk matches expected state, 1 otherwise
	#
	function check_vdev_state # pool disk state{online,offline,unavail}
	{
	typeset pool=$1
	typeset disk=${2#*$DEV_DSKDIR/}
	typeset state=$3

	cur_state=$(get_device_state $pool $disk)

	if [[ $state != ${cur_state} ]]; then
	return 1
	fi
	return 0
	}

	#
	# Wait until a vdev transitions to a given state or times out.
	#
	# Return 0 when pool/disk matches expected state, 1 on timeout.
	#
	function wait_vdev_state # pool disk state timeout
	{
	typeset pool=$1
	typeset disk=${2#*$DEV_DSKDIR/}
	typeset state=$3
	typeset timeout=${4:-60}
	typeset -i i=0

	while [[ $i -lt $timeout ]]; do
	if check_vdev_state $pool $disk $state; then
	return 0
	fi

	i=$((i+1))
	sleep 1
	done

	return 1
	}

	#
	# Check the output of 'zpool status -v <pool>',
	# and to see if the content of <token> contain the <keyword> specified.
	#
	# Return 0 is contain, 1 otherwise
	#
	function check_pool_status # pool token keyword <verbose>
	{
	typeset pool=$1
	typeset token=$2
	typeset keyword=$3
	typeset verbose=${4:-false}

	scan=$(zpool status -v "$pool" 2>/dev/null \| nawk -v token="$token:" '
	($1==token) {print $0}')
	if [[ $verbose == true ]]; then
	log_note $scan
	fi
	echo $scan \| egrep -i "$keyword" > /dev/null 2>&1

	return $?
	}

	#
	# The following functions are instance of check_pool_status()
	# is_pool_resilvering - to check if the pool resilver is in progress
	# is_pool_resilvered - to check if the pool resilver is completed
	# is_pool_scrubbing - to check if the pool scrub is in progress
	# is_pool_scrubbed - to check if the pool scrub is completed
	# is_pool_scrub_stopped - to check if the pool scrub is stopped
	# is_pool_scrub_paused - to check if the pool scrub has paused
	# is_pool_removing - to check if the pool removing is a vdev
	# is_pool_removed - to check if the pool remove is completed
	# is_pool_discarding - to check if the pool checkpoint is being discarded
	#
	function is_pool_resilvering #pool <verbose>
	{
	check_pool_status "$1" "scan" \
	"resilver[ ()0-9A-Za-z_-]* in progress since" $2
	return $?
	}

	function is_pool_resilvered #pool <verbose>
	{
	check_pool_status "$1" "scan" "resilvered " $2
	return $?
	}

	function is_pool_scrubbing #pool <verbose>
	{
	check_pool_status "$1" "scan" "scrub in progress since " $2
	return $?
	}

	function is_pool_scrubbed #pool <verbose>
	{
	check_pool_status "$1" "scan" "scrub repaired" $2
	return $?
	}

	function is_pool_scrub_stopped #pool <verbose>
	{
	check_pool_status "$1" "scan" "scrub canceled" $2
	return $?
	}

	function is_pool_scrub_paused #pool <verbose>
	{
	check_pool_status "$1" "scan" "scrub paused since " $2
	return $?
	}

	function is_pool_removing #pool
	{
	check_pool_status "$1" "remove" "in progress since "
	return $?
	}

	function is_pool_removed #pool
	{
	check_pool_status "$1" "remove" "completed on"
	return $?
	}

	function is_pool_discarding #pool
	{
	check_pool_status "$1" "checkpoint" "discarding"
	return $?
	}

	function wait_for_degraded
	{
	typeset pool=$1
	typeset timeout=${2:-30}
	typeset t0=$SECONDS

	while :; do
	[[ $(get_pool_prop health $pool) == "DEGRADED" ]] && break
	log_note "$pool is not yet degraded."
	sleep 1
	if ((SECONDS - t0 > $timeout)); then
	log_note "$pool not degraded after $timeout seconds."
	return 1
	fi
	done

	return 0
	}

	#
	# Use create_pool()/destroy_pool() to clean up the information in
	# in the given disk to avoid slice overlapping.
	#
	function cleanup_devices #vdevs
	{
	typeset pool="foopool$$"

	for vdev in $@; do
	zero_partitions $vdev
	done

	poolexists $pool && destroy_pool $pool
	create_pool $pool $@
	destroy_pool $pool

	return 0
	}

	#/**
	# A function to find and locate free disks on a system or from given
	# disks as the parameter. It works by locating disks that are in use
	# as swap devices and dump devices, and also disks listed in /etc/vfstab
	#
	# $@ given disks to find which are free, default is all disks in
	# the test system
	#
	# @return a string containing the list of available disks
	#*/
	function find_disks
	{
	# Trust provided list, no attempt is made to locate unused devices.
	if is_linux \|\| is_freebsd; then
	echo "$@"
	return
	fi


	sfi=/tmp/swaplist.$$
	dmpi=/tmp/dumpdev.$$
	max_finddisksnum=${MAX_FINDDISKSNUM:-6}

	swap -l > $sfi
	dumpadm > $dmpi 2>/dev/null

	# write an awk script that can process the output of format
	# to produce a list of disks we know about. Note that we have
	# to escape "$2" so that the shell doesn't interpret it while
	# we're creating the awk script.
	# -------------------
	cat > /tmp/find_disks.awk <<EOF
	#!/bin/nawk -f
	BEGIN { FS="."; }

	/^Specify disk/{
	searchdisks=0;
	}

	{
	if (searchdisks && \$2 !~ "^$"){
	split(\$2,arr," ");
	print arr[1];
	}
	}

	/^AVAILABLE DISK SELECTIONS:/{
	searchdisks=1;
	}
	EOF
	#---------------------

	chmod 755 /tmp/find_disks.awk
	disks=${@:-$(echo "" \| format -e 2>/dev/null \| /tmp/find_disks.awk)}
	rm /tmp/find_disks.awk

	unused=""
	for disk in $disks; do
	# Check for mounted
	grep "${disk}[sp]" /etc/mnttab >/dev/null
	(($? == 0)) && continue
	# Check for swap
	grep "${disk}[sp]" $sfi >/dev/null
	(($? == 0)) && continue
	# check for dump device
	grep "${disk}[sp]" $dmpi >/dev/null
	(($? == 0)) && continue
	# check to see if this disk hasn't been explicitly excluded
	# by a user-set environment variable
	echo "${ZFS_HOST_DEVICES_IGNORE}" \| grep "${disk}" > /dev/null
	(($? == 0)) && continue
	unused_candidates="$unused_candidates $disk"
	done
	rm $sfi
	rm $dmpi

	# now just check to see if those disks do actually exist
	# by looking for a device pointing to the first slice in
	# each case. limit the number to max_finddisksnum
	count=0
	for disk in $unused_candidates; do
	if is_disk_device $DEV_DSKDIR/${disk}s0 && \
	[ $count -lt $max_finddisksnum ]; then
	unused="$unused $disk"
	# do not impose limit if $@ is provided
	[[ -z $@ ]] && ((count = count + 1))
	fi
	done

	# finally, return our disk list
	echo $unused
	}

	function add_user_freebsd #<group_name> <user_name> <basedir>
	{
	typeset group=$1
	typeset user=$2
	typeset basedir=$3

	# Check to see if the user exists.
	if id $user > /dev/null 2>&1; then
	return 0
	fi

	# Assign 1000 as the base uid
	typeset -i uid=1000
	while true; do
	typeset -i ret
	pw useradd -u $uid -g $group -d $basedir/$user -m -n $user
	ret=$?
	case $ret in
	0) break ;;
	# The uid is not unique
	65) ((uid += 1)) ;;
	*) return 1 ;;
	esac
	if [[ $uid == 65000 ]]; then
	log_fail "No user id available under 65000 for $user"
	fi
	done

	# Silence MOTD
	touch $basedir/$user/.hushlogin

	return 0
	}

	#
	# Delete the specified user.
	#
	# $1 login name
	#
	function del_user_freebsd #<logname>
	{
	typeset user=$1

	if id $user > /dev/null 2>&1; then
	log_must pw userdel $user
	fi

	return 0
	}

	#
	# Select valid gid and create specified group.
	#
	# $1 group name
	#
	function add_group_freebsd #<group_name>
	{
	typeset group=$1

	# See if the group already exists.
	if pw groupshow $group >/dev/null 2>&1; then
	return 0
	fi

	# Assign 1000 as the base gid
	typeset -i gid=1000
	while true; do
	pw groupadd -g $gid -n $group > /dev/null 2>&1
	typeset -i ret=$?
	case $ret in
	0) return 0 ;;
	# The gid is not unique
	65) ((gid += 1)) ;;
	*) return 1 ;;
	esac
	if [[ $gid == 65000 ]]; then
	log_fail "No user id available under 65000 for $group"
	fi
	done
	}

	#
	# Delete the specified group.
	#
	# $1 group name
	#
	function del_group_freebsd #<group_name>
	{
	typeset group=$1

	pw groupdel -n $group > /dev/null 2>&1
	typeset -i ret=$?
	case $ret in
	# Group does not exist, or was deleted successfully.
	0\|6\|65) return 0 ;;
	# Name already exists as a group name
	9) log_must pw groupdel $group ;;
	*) return 1 ;;
	esac

	return 0
	}

	function add_user_illumos #<group_name> <user_name> <basedir>
	{
	typeset group=$1
	typeset user=$2
	typeset basedir=$3

	log_must useradd -g $group -d $basedir/$user -m $user

	return 0
	}

	function del_user_illumos #<user_name>
	{
	typeset user=$1

	if id $user > /dev/null 2>&1; then
	log_must_retry "currently used" 6 userdel $user
	fi

	return 0
	}

	function add_group_illumos #<group_name>
	{
	typeset group=$1

	typeset -i gid=100
	while true; do
	groupadd -g $gid $group > /dev/null 2>&1
	typeset -i ret=$?
	case $ret in
	0) return 0 ;;
	# The gid is not unique
	4) ((gid += 1)) ;;
	*) return 1 ;;
	esac
	done
	}

	function del_group_illumos #<group_name>
	{
	typeset group=$1

	groupmod -n $grp $grp > /dev/null 2>&1
	typeset -i ret=$?
	case $ret in
	# Group does not exist.
	6) return 0 ;;
	# Name already exists as a group name
	9) log_must groupdel $grp ;;
	*) return 1 ;;
	esac
	}

	function add_user_linux #<group_name> <user_name> <basedir>
	{
	typeset group=$1
	typeset user=$2
	typeset basedir=$3

	log_must useradd -g $group -d $basedir/$user -m $user

	# Add new users to the same group and the command line utils.
	# This allows them to be run out of the original users home
	# directory as long as it permissioned to be group readable.
	cmd_group=$(stat --format="%G" $(which zfs))
	log_must usermod -a -G $cmd_group $user

	return 0
	}

	function del_user_linux #<user_name>
	{
	typeset user=$1

	if id $user > /dev/null 2>&1; then
	log_must_retry "currently used" 6 userdel $user
	fi

	return 0
	}

	function add_group_linux #<group_name>
	{
	typeset group=$1

	# Assign 100 as the base gid, a larger value is selected for
	# Linux because for many distributions 1000 and under are reserved.
	while true; do
	groupadd $group > /dev/null 2>&1
	typeset -i ret=$?
	case $ret in
	0) return 0 ;;
	*) return 1 ;;
	esac
	done
	}

	function del_group_linux #<group_name>
	{
	typeset group=$1

	getent group $group > /dev/null 2>&1
	typeset -i ret=$?
	case $ret in
	# Group does not exist.
	2) return 0 ;;
	# Name already exists as a group name
	0) log_must groupdel $group ;;
	*) return 1 ;;
	esac

	return 0
	}

	#
	# Add specified user to specified group
	#
	# $1 group name
	# $2 user name
	# $3 base of the homedir (optional)
	#
	function add_user #<group_name> <user_name> <basedir>
	{
	typeset group=$1
	typeset user=$2
	typeset basedir=${3:-"/var/tmp"}

	if ((${#group} == 0 \|\| ${#user} == 0)); then
	log_fail "group name or user name are not defined."
	fi

	case $(uname) in
	FreeBSD)
	add_user_freebsd "$group" "$user" "$basedir"
	;;
	Linux)
	add_user_linux "$group" "$user" "$basedir"
	;;
	*)
	add_user_illumos "$group" "$user" "$basedir"
	;;
	esac

	echo "export PATH=\"$STF_PATH\"" >>$basedir/$user/.profile
	echo "export PATH=\"$STF_PATH\"" >>$basedir/$user/.bash_profile
	echo "export PATH=\"$STF_PATH\"" >>$basedir/$user/.login

	return 0
	}

	#
	# Delete the specified user.
	#
	# $1 login name
	# $2 base of the homedir (optional)
	#
	function del_user #<logname> <basedir>
	{
	typeset user=$1
	typeset basedir=${2:-"/var/tmp"}

	if ((${#user} == 0)); then
	log_fail "login name is necessary."
	fi

	case $(uname) in
	FreeBSD)
	del_user_freebsd "$user"
	;;
	Linux)
	del_user_linux "$user"
	;;
	*)
	del_user_illumos "$user"
	;;
	esac

	[[ -d $basedir/$user ]] && rm -fr $basedir/$user

	return 0
	}

	#
	# Select valid gid and create specified group.
	#
	# $1 group name
	#
	function add_group #<group_name>
	{
	typeset group=$1

	if ((${#group} == 0)); then
	log_fail "group name is necessary."
	fi

	case $(uname) in
	FreeBSD)
	add_group_freebsd "$group"
	;;
	Linux)
	add_group_linux "$group"
	;;
	*)
	add_group_illumos "$group"
	;;
	esac

	return 0
	}

	#
	# Delete the specified group.
	#
	# $1 group name
	#
	function del_group #<group_name>
	{
	typeset group=$1

	if ((${#group} == 0)); then
	log_fail "group name is necessary."
	fi

	case $(uname) in
	FreeBSD)
	del_group_freebsd "$group"
	;;
	Linux)
	del_group_linux "$group"
	;;
	*)
	del_group_illumos "$group"
	;;
	esac

	return 0
	}

	#
	# This function will return true if it's safe to destroy the pool passed
	# as argument 1. It checks for pools based on zvols and files, and also
	# files contained in a pool that may have a different mountpoint.
	#
	function safe_to_destroy_pool { # $1 the pool name

	typeset pool=""
	typeset DONT_DESTROY=""

	# We check that by deleting the $1 pool, we're not
	# going to pull the rug out from other pools. Do this
	# by looking at all other pools, ensuring that they
	# aren't built from files or zvols contained in this pool.

	for pool in $(zpool list -H -o name)
	do
	ALTMOUNTPOOL=""

	# this is a list of the top-level directories in each of the
	# files that make up the path to the files the pool is based on
	FILEPOOL=$(zpool status -v $pool \| grep /$1/ \| \
	awk '{print $1}')

	# this is a list of the zvols that make up the pool
	ZVOLPOOL=$(zpool status -v $pool \| grep "$ZVOL_DEVDIR/$1$" \
	\| awk '{print $1}')

	# also want to determine if it's a file-based pool using an
	# alternate mountpoint...
	POOL_FILE_DIRS=$(zpool status -v $pool \| \
	grep / \| awk '{print $1}' \| \
	awk -F/ '{print $2}' \| grep -v "dev")

	for pooldir in $POOL_FILE_DIRS
	do
	OUTPUT=$(zfs list -H -r -o mountpoint $1 \| \
	grep "${pooldir}$" \| awk '{print $1}')

	ALTMOUNTPOOL="${ALTMOUNTPOOL}${OUTPUT}"
	done


	if [ ! -z "$ZVOLPOOL" ]
	then
	DONT_DESTROY="true"
	log_note "Pool $pool is built from $ZVOLPOOL on $1"
	fi

	if [ ! -z "$FILEPOOL" ]
	then
	DONT_DESTROY="true"
	log_note "Pool $pool is built from $FILEPOOL on $1"
	fi

	if [ ! -z "$ALTMOUNTPOOL" ]
	then
	DONT_DESTROY="true"
	log_note "Pool $pool is built from $ALTMOUNTPOOL on $1"
	fi
	done

	if [ -z "${DONT_DESTROY}" ]
	then
	return 0
	else
	log_note "Warning: it is not safe to destroy $1!"
	return 1
	fi
	}

	#
	# Verify zfs operation with -p option work as expected
	# $1 operation, value could be create, clone or rename
	# $2 dataset type, value could be fs or vol
	# $3 dataset name
	# $4 new dataset name
	#
	function verify_opt_p_ops
	{
	typeset ops=$1
	typeset datatype=$2
	typeset dataset=$3
	typeset newdataset=$4

	if [[ $datatype != "fs" && $datatype != "vol" ]]; then
	log_fail "$datatype is not supported."
	fi

	# check parameters accordingly
	case $ops in
	create)
	newdataset=$dataset
	dataset=""
	if [[ $datatype == "vol" ]]; then
	ops="create -V $VOLSIZE"
	fi
	;;
	clone)
	if [[ -z $newdataset ]]; then
	log_fail "newdataset should not be empty" \
	"when ops is $ops."
	fi
	log_must datasetexists $dataset
	log_must snapexists $dataset
	;;
	rename)
	if [[ -z $newdataset ]]; then
	log_fail "newdataset should not be empty" \
	"when ops is $ops."
	fi
	log_must datasetexists $dataset
	;;
	*)
	log_fail "$ops is not supported."
	;;
	esac

	# make sure the upper level filesystem does not exist
	destroy_dataset "${newdataset%/*}" "-rRf"

	# without -p option, operation will fail
	log_mustnot zfs $ops $dataset $newdataset
	log_mustnot datasetexists $newdataset ${newdataset%/*}

	# with -p option, operation should succeed
	log_must zfs $ops -p $dataset $newdataset
	block_device_wait

	if ! datasetexists $newdataset ; then
	log_fail "-p option does not work for $ops"
	fi

	# when $ops is create or clone, redo the operation still return zero
	if [[ $ops != "rename" ]]; then
	log_must zfs $ops -p $dataset $newdataset
	fi

	return 0
	}

	#
	# Get configuration of pool
	# $1 pool name
	# $2 config name
	#
	function get_config
	{
	typeset pool=$1
	typeset config=$2
	typeset alt_root

	if ! poolexists "$pool" ; then
	return 1
	fi
	alt_root=$(zpool list -H $pool \| awk '{print $NF}')
	if [[ $alt_root == "-" ]]; then
	value=$(zdb -C $pool \| grep "$config:" \| awk -F: \
	'{print $2}')
	else
	value=$(zdb -e $pool \| grep "$config:" \| awk -F: \
	'{print $2}')
	fi
	if [[ -n $value ]] ; then
	value=${value#'}
	value=${value%'}
	fi
	echo $value

	return 0
	}

	#
	# Privated function. Random select one of items from arguments.
	#
	# $1 count
	# $2-n string
	#
	function _random_get
	{
	typeset cnt=$1
	shift

	typeset str="$@"
	typeset -i ind
	((ind = RANDOM % cnt + 1))

	typeset ret=$(echo "$str" \| cut -f $ind -d ' ')
	echo $ret
	}

	#
	# Random select one of item from arguments which include NONE string
	#
	function random_get_with_non
	{
	typeset -i cnt=$#
	((cnt =+ 1))

	_random_get "$cnt" "$@"
	}

	#
	# Random select one of item from arguments which doesn't include NONE string
	#
	function random_get
	{
	_random_get "$#" "$@"
	}

	#
	# Detect if the current system support slog
	#
	function verify_slog_support
	{
	typeset dir=$TEST_BASE_DIR/disk.$$
	typeset pool=foo.$$
	typeset vdev=$dir/a
	typeset sdev=$dir/b

	mkdir -p $dir
	mkfile $MINVDEVSIZE $vdev $sdev

	typeset -i ret=0
	if ! zpool create -n $pool $vdev log $sdev > /dev/null 2>&1; then
	ret=1
	fi
	rm -r $dir

	return $ret
	}

	#
	# The function will generate a dataset name with specific length
	# $1, the length of the name
	# $2, the base string to construct the name
	#
	function gen_dataset_name
	{
	typeset -i len=$1
	typeset basestr="$2"
	typeset -i baselen=${#basestr}
	typeset -i iter=0
	typeset l_name=""

	if ((len % baselen == 0)); then
	((iter = len / baselen))
	else
	((iter = len / baselen + 1))
	fi
	while ((iter > 0)); do
	l_name="${l_name}$basestr"

	((iter -= 1))
	done

	echo $l_name
	}

	#
	# Get cksum tuple of dataset
	# $1 dataset name
	#
	# sample zdb output:
	# Dataset data/test [ZPL], ID 355, cr_txg 2413856, 31.0K, 7 objects, rootbp
	# DVA[0]=<0:803046400:200> DVA[1]=<0:81199000:200> [L0 DMU objset] fletcher4
	# lzjb LE contiguous unique double size=800L/200P birth=2413856L/2413856P
	# fill=7 cksum=11ce125712:643a9c18ee2:125e25238fca0:254a3f74b59744
	function datasetcksum
	{
	typeset cksum
	sync
	cksum=$(zdb -vvv $1 \| grep "^Dataset $1 \[" \| grep "cksum" \
	\| awk -F= '{print $7}')
	echo $cksum
	}

	#
	# Get cksum of file
	# #1 file path
	#
	function checksum
	{
	typeset cksum
	cksum=$(cksum $1 \| awk '{print $1}')
	echo $cksum
	}

	#
	# Get the given disk/slice state from the specific field of the pool
	#
	function get_device_state #pool disk field("", "spares","logs")
	{
	typeset pool=$1
	typeset disk=${2#$DEV_DSKDIR/}
	typeset field=${3:-$pool}

	state=$(zpool status -v "$pool" 2>/dev/null \| \
	nawk -v device=$disk -v pool=$pool -v field=$field \
	'BEGIN {startconfig=0; startfield=0; }
	/config:/ {startconfig=1}
	(startconfig==1) && ($1==field) {startfield=1; next;}
	(startfield==1) && ($1==device) {print $2; exit;}
	(startfield==1) &&
	($1==field \|\| $1 ~ "^spares$" \|\| $1 ~ "^logs$") {startfield=0}')
	echo $state
	}


	#
	# print the given directory filesystem type
	#
	# $1 directory name
	#
	function get_fstype
	{
	typeset dir=$1

	if [[ -z $dir ]]; then
	log_fail "Usage: get_fstype <directory>"
	fi

	#
	# $ df -n /
	# / : ufs
	#
	df -n $dir \| awk '{print $3}'
	}

	#
	# Given a disk, label it to VTOC regardless what label was on the disk
	# $1 disk
	#
	function labelvtoc
	{
	typeset disk=$1
	if [[ -z $disk ]]; then
	log_fail "The disk name is unspecified."
	fi
	typeset label_file=/var/tmp/labelvtoc.$$
	typeset arch=$(uname -p)

	if is_linux \|\| is_freebsd; then
	log_note "Currently unsupported by the test framework"
	return 1
	fi

	if [[ $arch == "i386" ]]; then
	echo "label" > $label_file
	echo "0" >> $label_file
	echo "" >> $label_file
	echo "q" >> $label_file
	echo "q" >> $label_file

	fdisk -B $disk >/dev/null 2>&1
	# wait a while for fdisk finishes
	sleep 60
	elif [[ $arch == "sparc" ]]; then
	echo "label" > $label_file
	echo "0" >> $label_file
	echo "" >> $label_file
	echo "" >> $label_file
	echo "" >> $label_file
	echo "q" >> $label_file
	else
	log_fail "unknown arch type"
	fi

	format -e -s -d $disk -f $label_file
	typeset -i ret_val=$?
	rm -f $label_file
	#
	# wait the format to finish
	#
	sleep 60
	if ((ret_val != 0)); then
	log_fail "unable to label $disk as VTOC."
	fi

	return 0
	}

	#
	# check if the system was installed as zfsroot or not
	# return: 0 if zfsroot, non-zero if not
	#
	function is_zfsroot
	{
	df -n / \| grep zfs > /dev/null 2>&1
	return $?
	}

	#
	# get the root filesystem name if it's zfsroot system.
	#
	# return: root filesystem name
	function get_rootfs
	{
	typeset rootfs=""

	if is_freebsd; then
	rootfs=$(mount -p \| awk '$2 == "/" && $3 == "zfs" {print $1}')
	elif ! is_linux; then
	rootfs=$(awk '{if ($2 == "/" && $3 == "zfs") print $1}' \
	/etc/mnttab)
	fi
	if [[ -z "$rootfs" ]]; then
	log_fail "Can not get rootfs"
	fi
	zfs list $rootfs > /dev/null 2>&1
	if (($? == 0)); then
	echo $rootfs
	else
	log_fail "This is not a zfsroot system."
	fi
	}

	#
	# get the rootfs's pool name
	# return:
	# rootpool name
	#
	function get_rootpool
	{
	typeset rootfs=""
	typeset rootpool=""

	if is_freebsd; then
	rootfs=$(mount -p \| awk '$2 == "/" && $3 == "zfs" {print $1}')
	elif ! is_linux; then
	rootfs=$(awk '{if ($2 == "/" && $3 =="zfs") print $1}' \
	/etc/mnttab)
	fi
	if [[ -z "$rootfs" ]]; then
	log_fail "Can not get rootpool"
	fi
	zfs list $rootfs > /dev/null 2>&1
	if (($? == 0)); then
	echo ${rootfs%%/*}
	else
	log_fail "This is not a zfsroot system."
	fi
	}

	#
	# Get the word numbers from a string separated by white space
	#
	function get_word_count
	{
	echo $1 \| wc -w
	}

	#
	# To verify if the require numbers of disks is given
	#
	function verify_disk_count
	{
	typeset -i min=${2:-1}

	typeset -i count=$(get_word_count "$1")

	if ((count < min)); then
	log_untested "A minimum of $min disks is required to run." \
	" You specified $count disk(s)"
	fi
	}

	function ds_is_volume
	{
	typeset type=$(get_prop type $1)
	[[ $type = "volume" ]] && return 0
	return 1
	}

	function ds_is_filesystem
	{
	typeset type=$(get_prop type $1)
	[[ $type = "filesystem" ]] && return 0
	return 1
	}

	function ds_is_snapshot
	{
	typeset type=$(get_prop type $1)
	[[ $type = "snapshot" ]] && return 0
	return 1
	}

	#
	# Check if Trusted Extensions are installed and enabled
	#
	function is_te_enabled
	{
	svcs -H -o state labeld 2>/dev/null \| grep "enabled"
	if (($? != 0)); then
	return 1
	else
	return 0
	fi
	}

	# Utility function to determine if a system has multiple cpus.
	function is_mp
	{
	if is_linux; then
	(($(nproc) > 1))
	elif is_freebsd; then
	sysctl -n kern.smp.cpus
	else
	(($(psrinfo \| wc -l) > 1))
	fi

	return $?
	}

	function get_cpu_freq
	{
	if is_linux; then
	lscpu \| awk '/CPU MHz/ { print $3 }'
	elif is_freebsd; then
	sysctl -n hw.clockrate
	else
	psrinfo -v 0 \| awk '/processor operates at/ {print $6}'
	fi
	}

	# Run the given command as the user provided.
	function user_run
	{
	typeset user=$1
	shift

	log_note "user:$user $@"
	eval su - \$user -c \"$@\" > $TEST_BASE_DIR/out 2>$TEST_BASE_DIR/err
	}

	#
	# Check if the pool contains the specified vdevs
	#
	# $1 pool
	# $2..n <vdev> ...
	#
	# Return 0 if the vdevs are contained in the pool, 1 if any of the specified
	# vdevs is not in the pool, and 2 if pool name is missing.
	#
	function vdevs_in_pool
	{
	typeset pool=$1
	typeset vdev

	if [[ -z $pool ]]; then
	log_note "Missing pool name."
	return 2
	fi

	shift

	# We could use 'zpool list' to only get the vdevs of the pool but we
	# can't reference a mirror/raidz vdev using its ID (i.e mirror-0),
	# therefore we use the 'zpool status' output.
	typeset tmpfile=$(mktemp)
	zpool status -v "$pool" \| grep -A 1000 "config:" >$tmpfile
	for vdev in $@; do
	grep -w ${vdev##*/} $tmpfile >/dev/null 2>&1
	[[ $? -ne 0 ]] && return 1
	done

	rm -f $tmpfile

	return 0;
	}

	function get_max
	{
	typeset -l i max=$1
	shift

	for i in "$@"; do
	max=$((max > i ? max : i))
	done

	echo $max
	}

	function get_min
	{
	typeset -l i min=$1
	shift

	for i in "$@"; do
	min=$((min < i ? min : i))
	done

	echo $min
	}

	# Write data that can be compressed into a directory
	function write_compressible
	{
	typeset dir=$1
	typeset megs=$2
	typeset nfiles=${3:-1}
	typeset bs=${4:-1024k}
	typeset fname=${5:-file}

	[[ -d $dir ]] \|\| log_fail "No directory: $dir"

	# Under Linux fio is not currently used since its behavior can
	# differ significantly across versions. This includes missing
	# command line options and cases where the --buffer_compress_*
	# options fail to behave as expected.
	if is_linux; then
	typeset file_bytes=$(to_bytes $megs)
	typeset bs_bytes=4096
	typeset blocks=$(($file_bytes / $bs_bytes))

	for (( i = 0; i < $nfiles; i++ )); do
	truncate -s $file_bytes $dir/$fname.$i

	# Write every third block to get 66% compression.
	for (( j = 0; j < $blocks; j += 3 )); do
	dd if=/dev/urandom of=$dir/$fname.$i \
	seek=$j bs=$bs_bytes count=1 \
	conv=notrunc >/dev/null 2>&1
	done
	done
	else
	log_must eval "fio \
	--name=job \
	--fallocate=0 \
	--minimal \
	--randrepeat=0 \
	--buffer_compress_percentage=66 \
	--buffer_compress_chunk=4096 \
	--directory=$dir \
	--numjobs=$nfiles \
	--nrfiles=$nfiles \
	--rw=write \
	--bs=$bs \
	--filesize=$megs \
	--filename_format='$fname.\$jobnum' >/dev/null"
	fi
	}

	function get_objnum
	{
	typeset pathname=$1
	typeset objnum

	[[ -e $pathname ]] \|\| log_fail "No such file or directory: $pathname"
	if is_freebsd; then
	objnum=$(stat -f "%i" $pathname)
	else
	objnum=$(stat -c %i $pathname)
	fi
	echo $objnum
	}

	#
	# Sync data to the pool
	#
	# $1 pool name
	# $2 boolean to force uberblock (and config including zpool cache file) update
	#
	function sync_pool #pool <force>
	{
	typeset pool=${1:-$TESTPOOL}
	typeset force=${2:-false}

	if [[ $force == true ]]; then
	log_must zpool sync -f $pool
	else
	log_must zpool sync $pool
	fi

	return 0
	}

	#
	# Wait for zpool 'freeing' property drops to zero.
	#
	# $1 pool name
	#
	function wait_freeing #pool
	{
	typeset pool=${1:-$TESTPOOL}
	while true; do
	[[ "0" == "$(zpool list -Ho freeing $pool)" ]] && break
	log_must sleep 1
	done
	}

	#
	# Wait for every device replace operation to complete
	#
	# $1 pool name
	#
	function wait_replacing #pool
	{
	typeset pool=${1:-$TESTPOOL}
	while true; do
	[[ "" == "$(zpool status $pool \|
	awk '/replacing-[0-9]+/ {print $1}')" ]] && break
	log_must sleep 1
	done
	}

	#
	# Wait for a pool to be scrubbed
	#
	# $1 pool name
	# $2 number of seconds to wait (optional)
	#
	# Returns true when pool has been scrubbed, or false if there's a timeout or if
	# no scrub was done.
	#
	function wait_scrubbed
	{
	typeset pool=${1:-$TESTPOOL}
	while true ; do
	is_pool_scrubbed $pool && break
	sleep 1
	done
	}

	# Backup the zed.rc in our test directory so that we can edit it for our test.
	#
	# Returns: Backup file name. You will need to pass this to zed_rc_restore().
	function zed_rc_backup
	{
	zedrc_backup="$(mktemp)"
	cp $ZEDLET_DIR/zed.rc $zedrc_backup
	echo $zedrc_backup
	}

	function zed_rc_restore
	{
	mv $1 $ZEDLET_DIR/zed.rc
	}

	#
	# Setup custom environment for the ZED.
	#
	# $@ Optional list of zedlets to run under zed.
	function zed_setup
	{
	if ! is_linux; then
	log_unsupported "No zed on $(uname)"
	fi

	if [[ ! -d $ZEDLET_DIR ]]; then
	log_must mkdir $ZEDLET_DIR
	fi

	if [[ ! -e $VDEVID_CONF ]]; then
	log_must touch $VDEVID_CONF
	fi

	if [[ -e $VDEVID_CONF_ETC ]]; then
	log_fail "Must not have $VDEVID_CONF_ETC file present on system"
	fi
	EXTRA_ZEDLETS=$@

	# Create a symlink for /etc/zfs/vdev_id.conf file.
	log_must ln -s $VDEVID_CONF $VDEVID_CONF_ETC

	# Setup minimal ZED configuration. Individual test cases should
	# add additional ZEDLETs as needed for their specific test.
	log_must cp ${ZEDLET_ETC_DIR}/zed.rc $ZEDLET_DIR
	log_must cp ${ZEDLET_ETC_DIR}/zed-functions.sh $ZEDLET_DIR

	# Scripts must only be user writable.
	if [[ -n "$EXTRA_ZEDLETS" ]] ; then
	saved_umask=$(umask)
	log_must umask 0022
	for i in $EXTRA_ZEDLETS ; do
	log_must cp ${ZEDLET_LIBEXEC_DIR}/$i $ZEDLET_DIR
	done
	log_must umask $saved_umask
	fi

	# Customize the zed.rc file to enable the full debug log.
	log_must sed -i '/\#ZED_DEBUG_LOG=.*/d' $ZEDLET_DIR/zed.rc
	echo "ZED_DEBUG_LOG=$ZED_DEBUG_LOG" >>$ZEDLET_DIR/zed.rc

	}

	#
	# Cleanup custom ZED environment.
	#
	# $@ Optional list of zedlets to remove from our test zed.d directory.
	function zed_cleanup
	{
	if ! is_linux; then
	return
	fi
	EXTRA_ZEDLETS=$@

	log_must rm -f ${ZEDLET_DIR}/zed.rc
	log_must rm -f ${ZEDLET_DIR}/zed-functions.sh
	log_must rm -f ${ZEDLET_DIR}/all-syslog.sh
	log_must rm -f ${ZEDLET_DIR}/all-debug.sh
	log_must rm -f ${ZEDLET_DIR}/state

	if [[ -n "$EXTRA_ZEDLETS" ]] ; then
	for i in $EXTRA_ZEDLETS ; do
	log_must rm -f ${ZEDLET_DIR}/$i
	done
	fi
	log_must rm -f $ZED_LOG
	log_must rm -f $ZED_DEBUG_LOG
	log_must rm -f $VDEVID_CONF_ETC
	log_must rm -f $VDEVID_CONF
	rmdir $ZEDLET_DIR
	}

	#
	# Check if ZED is currently running, if not start ZED.
	#
	function zed_start
	{
	if ! is_linux; then
	return
	fi

	# ZEDLET_DIR=/var/tmp/zed
	if [[ ! -d $ZEDLET_DIR ]]; then
	log_must mkdir $ZEDLET_DIR
	fi

	# Verify the ZED is not already running.
	pgrep -x zed > /dev/null
	if (($? == 0)); then
	log_note "ZED already running"
	else
	log_note "Starting ZED"
	# run ZED in the background and redirect foreground logging
	# output to $ZED_LOG.
	log_must truncate -s 0 $ZED_DEBUG_LOG
	log_must eval "zed -vF -d $ZEDLET_DIR -p $ZEDLET_DIR/zed.pid -P $PATH" \
	"-s $ZEDLET_DIR/state 2>$ZED_LOG &"
	fi

	return 0
	}

	#
	# Kill ZED process
	#
	function zed_stop
	{
	if ! is_linux; then
	return
	fi

	log_note "Stopping ZED"
	if [[ -f ${ZEDLET_DIR}/zed.pid ]]; then
	zedpid=$(<${ZEDLET_DIR}/zed.pid)
	kill $zedpid
	while ps -p $zedpid > /dev/null; do
	sleep 1
	done
	rm -f ${ZEDLET_DIR}/zed.pid
	fi
	return 0
	}

	#
	# Drain all zevents
	#
	function zed_events_drain
	{
	while [ $(zpool events -H \| wc -l) -ne 0 ]; do
	sleep 1
	zpool events -c >/dev/null
	done
	}

	# Set a variable in zed.rc to something, un-commenting it in the process.
	#
	# $1 variable
	# $2 value
	function zed_rc_set
	{
	var="$1"
	val="$2"
	# Remove the line
	cmd="'/$var/d'"
	eval sed -i $cmd $ZEDLET_DIR/zed.rc

	# Add it at the end
	echo "$var=$val" >> $ZEDLET_DIR/zed.rc
	}


	#
	# Check is provided device is being active used as a swap device.
	#
	function is_swap_inuse
	{
	typeset device=$1

	if [[ -z $device ]] ; then
	log_note "No device specified."
	return 1
	fi

	if is_linux; then
	swapon -s \| grep -w $(readlink -f $device) > /dev/null 2>&1
	elif is_freebsd; then
	swapctl -l \| grep -w $device
	else
	swap -l \| grep -w $device > /dev/null 2>&1
	fi

	return $?
	}

	#
	# Setup a swap device using the provided device.
	#
	function swap_setup
	{
	typeset swapdev=$1

	if is_linux; then
	log_must eval "mkswap $swapdev > /dev/null 2>&1"
	log_must swapon $swapdev
	elif is_freebsd; then
	log_must swapctl -a $swapdev
	else
	log_must swap -a $swapdev
	fi

	return 0
	}

	#
	# Cleanup a swap device on the provided device.
	#
	function swap_cleanup
	{
	typeset swapdev=$1

	if is_swap_inuse $swapdev; then
	if is_linux; then
	log_must swapoff $swapdev
	elif is_freebsd; then
	log_must swapoff $swapdev
	else
	log_must swap -d $swapdev
	fi
	fi

	return 0
	}

	#
	# Set a global system tunable (64-bit value)
	#
	# $1 tunable name (use a NAME defined in tunables.cfg)
	# $2 tunable values
	#
	function set_tunable64
	{
	set_tunable_impl "$1" "$2" Z
	}

	#
	# Set a global system tunable (32-bit value)
	#
	# $1 tunable name (use a NAME defined in tunables.cfg)
	# $2 tunable values
	#
	function set_tunable32
	{
	set_tunable_impl "$1" "$2" W
	}

	function set_tunable_impl
	{
	typeset name="$1"
	typeset value="$2"
	typeset mdb_cmd="$3"
	typeset module="${4:-zfs}"

	eval "typeset tunable=\$$name"
	case "$tunable" in
	UNSUPPORTED)
	log_unsupported "Tunable '$name' is unsupported on $(uname)"
	;;
	"")
	log_fail "Tunable '$name' must be added to tunables.cfg"
	;;
	*)
	;;
	esac

	[[ -z "$value" ]] && return 1
	[[ -z "$mdb_cmd" ]] && return 1

	case "$(uname)" in
	Linux)
	typeset zfs_tunables="/sys/module/$module/parameters"
	[[ -w "$zfs_tunables/$tunable" ]] \|\| return 1
	cat >"$zfs_tunables/$tunable" <<<"$value"
	return $?
	;;
	FreeBSD)
	sysctl vfs.zfs.$tunable=$value
	return "$?"
	;;
	SunOS)
	[[ "$module" -eq "zfs" ]] \|\| return 1
	echo "${tunable}/${mdb_cmd}0t${value}" \| mdb -kw
	return $?
	;;
	esac
	}

	#
	# Get a global system tunable
	#
	# $1 tunable name (use a NAME defined in tunables.cfg)
	#
	function get_tunable
	{
	get_tunable_impl "$1"
	}

	function get_tunable_impl
	{
	typeset name="$1"
	typeset module="${2:-zfs}"

	eval "typeset tunable=\$$name"
	case "$tunable" in
	UNSUPPORTED)
	log_unsupported "Tunable '$name' is unsupported on $(uname)"
	;;
	"")
	log_fail "Tunable '$name' must be added to tunables.cfg"
	;;
	*)
	;;
	esac

	case "$(uname)" in
	Linux)
	typeset zfs_tunables="/sys/module/$module/parameters"
	[[ -f "$zfs_tunables/$tunable" ]] \|\| return 1
	cat $zfs_tunables/$tunable
	return $?
	;;
	FreeBSD)
	sysctl -n vfs.zfs.$tunable
	;;
	SunOS)
	[[ "$module" -eq "zfs" ]] \|\| return 1
	;;
	esac

	return 1
	}

	#
	# Prints the current time in seconds since UNIX Epoch.
	#
	function current_epoch
	{
	printf '%(%s)T'
	}

	#
	# Get decimal value of global uint32_t variable using mdb.
	#
	function mdb_get_uint32
	{
	typeset variable=$1
	typeset value

	value=$(mdb -k -e "$variable/X \| ::eval .=U")
	if [[ $? -ne 0 ]]; then
	log_fail "Failed to get value of '$variable' from mdb."
	return 1
	fi

	echo $value
	return 0
	}

	#
	# Set global uint32_t variable to a decimal value using mdb.
	#
	function mdb_set_uint32
	{
	typeset variable=$1
	typeset value=$2

	mdb -kw -e "$variable/W 0t$value" > /dev/null
	if [[ $? -ne 0 ]]; then
	echo "Failed to set '$variable' to '$value' in mdb."
	return 1
	fi

	return 0
	}

	#
	# Set global scalar integer variable to a hex value using mdb.
	# Note: Target should have CTF data loaded.
	#
	function mdb_ctf_set_int
	{
	typeset variable=$1
	typeset value=$2

	mdb -kw -e "$variable/z $value" > /dev/null
	if [[ $? -ne 0 ]]; then
	echo "Failed to set '$variable' to '$value' in mdb."
	return 1
	fi

	return 0
	}

	#
	# Compute MD5 digest for given file or stdin if no file given.
	# Note: file path must not contain spaces
	#
	function md5digest
	{
	typeset file=$1

	case $(uname) in
	FreeBSD)
	md5 -q $file
	;;
	*)
	md5sum -b $file \| awk '{ print $1 }'
	;;
	esac
	}

	#
	# Compute SHA256 digest for given file or stdin if no file given.
	# Note: file path must not contain spaces
	#
	function sha256digest
	{
	typeset file=$1

	case $(uname) in
	FreeBSD)
	sha256 -q $file
	;;
	*)
	sha256sum -b $file \| awk '{ print $1 }'
	;;
	esac
	}

	function new_fs #<args>
	{
	case $(uname) in
	FreeBSD)
	newfs "$@"
	;;
	*)
	echo y \| newfs -v "$@"
	;;
	esac
	}

	function stat_size #<path>
	{
	typeset path=$1

	case $(uname) in
	FreeBSD)
	stat -f %z "$path"
	;;
	*)
	stat -c %s "$path"
	;;
	esac
	}

	# Run a command as if it was being run in a TTY.
	#
	# Usage:
	#
	# faketty command
	#
	function faketty
	{
	if is_freebsd; then
	script -q /dev/null env "$@"
	else
	script --return --quiet -c "$*" /dev/null
	fi
	}

	#
	# Produce a random permutation of the integers in a given range (inclusive).
	#
	function range_shuffle # begin end
	{
	typeset -i begin=$1
	typeset -i end=$2

	seq ${begin} ${end} \| sort -R
	}

	#
	# Cross-platform xattr helpers
	#

	function get_xattr # name path
	{
	typeset name=$1
	typeset path=$2

	case $(uname) in
	FreeBSD)
	getextattr -qq user "${name}" "${path}"
	;;
	*)
	attr -qg "${name}" "${path}"
	;;
	esac
	}

	function set_xattr # name value path
	{
	typeset name=$1
	typeset value=$2
	typeset path=$3

	case $(uname) in
	FreeBSD)
	setextattr user "${name}" "${value}" "${path}"
	;;
	*)
	attr -qs "${name}" -V "${value}" "${path}"
	;;
	esac
	}

	function set_xattr_stdin # name value
	{
	typeset name=$1
	typeset path=$2

	case $(uname) in
	FreeBSD)
	setextattr -i user "${name}" "${path}"
	;;
	*)
	attr -qs "${name}" "${path}"
	;;
	esac
	}

	function rm_xattr # name path
	{
	typeset name=$1
	typeset path=$2

	case $(uname) in
	FreeBSD)
	rmextattr -q user "${name}" "${path}"
	;;
	*)
	attr -qr "${name}" "${path}"
	;;
	esac
	}

	function ls_xattr # path
	{
	typeset path=$1

	case $(uname) in
	FreeBSD)
	lsextattr -qq user "${path}"
	;;
	*)
	attr -ql "${path}"
	;;
	esac
	}

	+function kstat # stat flags?
	+{
	+ typeset stat=$1
	+ typeset flags=${2-"-n"}
	+
	+ case $(uname) in
	+ FreeBSD)
	+ sysctl $flags kstat.zfs.misc.$stat
	+ ;;
	+ Linux)
	+ typeset zfs_kstat="/proc/spl/kstat/zfs/$stat"
	+ [[ -f "$zfs_kstat" ]] \|\| return 1
	+ cat $zfs_kstat
	+ ;;
	+ *)
	+ false
	+ ;;
	+ esac
	+}
	+
	function get_arcstat # stat
	{
	typeset stat=$1

	case $(uname) in
	FreeBSD)
	- sysctl -n kstat.zfs.misc.arcstats.$stat
	+ kstat arcstats.$stat
	;;
	Linux)
	- typeset zfs_arcstats="/proc/spl/kstat/zfs/arcstats"
	- [[ -f "$zfs_arcstats" ]] \|\| return 1
	- grep $stat $zfs_arcstats \| awk '{print $3}'
	+ kstat arcstats \| awk "/$stat/ { print \$3 }"
	;;
	*)
	false
	;;
	esac
	}

	#
	# Given an array of pids, wait until all processes
	# have completed and check their return status.
	#
	function wait_for_children #children
	{
	rv=0
	children=("$@")
	for child in "${children[@]}"
	do
	child_exit=0
	wait ${child} \|\| child_exit=$?
	if [ $child_exit -ne 0 ]; then
	echo "child ${child} failed with ${child_exit}"
	rv=1
	fi
	done
	return $rv
	}
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/Makefile.am
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/Makefile.am (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/Makefile.am (revision 366774)
	@@ -1,6 +1,6 @@
	pkgdatadir = $(datadir)/@PACKAGE@/zfs-tests/tests/functional/acl
	dist_pkgdata_DATA = \
	acl.cfg \
	acl_common.kshlib

	-SUBDIRS = posix
	+SUBDIRS = posix posix-sa
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix/Makefile.am
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix/Makefile.am (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix/Makefile.am (revision 366774)
	@@ -1,7 +1,8 @@
	pkgdatadir = $(datadir)/@PACKAGE@/zfs-tests/tests/functional/acl/posix
	dist_pkgdata_SCRIPTS = \
	cleanup.ksh \
	setup.ksh \
	posix_001_pos.ksh \
	posix_002_pos.ksh \
	- posix_003_pos.ksh
	+ posix_003_pos.ksh \
	+ posix_004_pos.ksh
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix/posix_003_pos.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix/posix_003_pos.ksh (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix/posix_003_pos.ksh (revision 366774)
	@@ -1,60 +1,59 @@
	#!/bin/ksh -p
	#
	# CDDL HEADER START
	#
	# The contents of this file are subject to the terms of the
	# Common Development and Distribution License (the "License").
	# You may not use this file except in compliance with the License.
	#
	# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	# or http://www.opensolaris.org/os/licensing.
	# See the License for the specific language governing permissions
	# and limitations under the License.
	#
	# When distributing Covered Code, include this CDDL HEADER in each
	# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	# If applicable, add the following below this CDDL HEADER, with the
	# fields enclosed by brackets "[]" replaced with your own identifying
	# information: Portions Copyright [yyyy] [name of copyright owner]
	#
	# CDDL HEADER END
	#

	. $STF_SUITE/include/libtest.shlib

	#
	# DESCRIPTION:
	# Verify that ACLs survive remount.
	-# Regression test for https://github.com/zfsonlinux/zfs/issues/4520
	#
	# STRATEGY:
	# 1. Test presence of default and regular ACLs after remount
	# a. Can set and list ACL before remount
	# b. Can list ACL after remount
	#

	verify_runnable "both"
	log_assert "Verify regular and default POSIX ACLs survive remount"

	typeset acl_str1="^group:$ZFS_ACL_STAFF_GROUP:-wx$"
	typeset acl_str2="^default:group:$ZFS_ACL_STAFF_GROUP:-wx$"
	typeset ACLDIR="$TESTDIR/dir.1"

	log_note "Testing access to DIRECTORY"
	log_must mkdir $ACLDIR
	log_must setfacl -m g:$ZFS_ACL_STAFF_GROUP:wx $ACLDIR
	log_must setfacl -d -m g:$ZFS_ACL_STAFF_GROUP:wx $ACLDIR
	getfacl $ACLDIR 2> /dev/null \| egrep -q "$acl_str1"
	if [ "$?" -eq "0" ]; then
	getfacl $ACLDIR 2> /dev/null \| egrep -q "$acl_str2"
	fi

	if [ "$?" -eq "0" ]; then
	log_must zfs unmount $TESTPOOL/$TESTFS
	log_must zfs mount $TESTPOOL/$TESTFS
	log_must eval "getfacl $ACLDIR 2> /dev/null \| egrep -q \"$acl_str1\""
	log_must eval "getfacl $ACLDIR 2> /dev/null \| egrep -q \"$acl_str2\""
	log_pass "POSIX ACLs survive remount"
	else
	log_fail "Group '$ZFS_ACL_STAFF_GROUP' does not have 'rwx'"
	fi
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix/posix_004_pos.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix/posix_004_pos.ksh (nonexistent)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix/posix_004_pos.ksh (revision 366774)
	@@ -0,0 +1,49 @@
	+#!/bin/ksh -p
	+#
	+# CDDL HEADER START
	+#
	+# The contents of this file are subject to the terms of the
	+# Common Development and Distribution License (the "License").
	+# You may not use this file except in compliance with the License.
	+#
	+# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	+# or http://www.opensolaris.org/os/licensing.
	+# See the License for the specific language governing permissions
	+# and limitations under the License.
	+#
	+# When distributing Covered Code, include this CDDL HEADER in each
	+# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	+# If applicable, add the following below this CDDL HEADER, with the
	+# fields enclosed by brackets "[]" replaced with your own identifying
	+# information: Portions Copyright [yyyy] [name of copyright owner]
	+#
	+# CDDL HEADER END
	+#
	+
	+#
	+# Portions Copyright 2020 iXsystems, Inc.
	+#
	+
	+. $STF_SUITE/include/libtest.shlib
	+. $STF_SUITE/tests/functional/acl/acl_common.kshlib
	+
	+#
	+# DESCRIPTION:
	+# Verify chown works with POSIX ACLs.
	+# Regression test for https://github.com/openzfs/zfs/issues/10043
	+#
	+# STRATEGY:
	+# 1. Prepare an appropriate ACL on the test directory
	+# 2. Change the owner of the directory
	+#
	+
	+verify_runnable "both"
	+log_assert "Verify chown works with POSIX ACLs"
	+
	+log_must setfacl -d -m u:$ZFS_ACL_STAFF1:rwx $TESTDIR
	+log_must setfacl -b $TESTDIR
	+
	+log_must chown $ZFS_ACL_STAFF1 $TESTDIR
	+log_must chown 0 $TESTDIR
	+
	+log_pass "chown works with POSIX ACLs"
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix/setup.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix/setup.ksh (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix/setup.ksh (revision 366774)
	@@ -1,52 +1,51 @@
	#!/bin/ksh -p
	#
	# CDDL HEADER START
	#
	# The contents of this file are subject to the terms of the
	# Common Development and Distribution License (the "License").
	# You may not use this file except in compliance with the License.
	#
	# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	# or http://www.opensolaris.org/os/licensing.
	# See the License for the specific language governing permissions
	# and limitations under the License.
	#
	# When distributing Covered Code, include this CDDL HEADER in each
	# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	# If applicable, add the following below this CDDL HEADER, with the
	# fields enclosed by brackets "[]" replaced with your own identifying
	# information: Portions Copyright [yyyy] [name of copyright owner]
	#
	# CDDL HEADER END
	#

	#
	# Copyright 2007 Sun Microsystems, Inc. All rights reserved.
	# Use is subject to license terms.
	#

	#
	# Copyright (c) 2016 by Delphix. All rights reserved.
	#

	. $STF_SUITE/include/libtest.shlib
	. $STF_SUITE/tests/functional/acl/acl_common.kshlib

	log_must getfacl --version
	log_must setfacl --version

	cleanup_user_group

	# Create staff group and add user to it
	log_must add_group $ZFS_ACL_STAFF_GROUP
	log_must add_user $ZFS_ACL_STAFF_GROUP $ZFS_ACL_STAFF1

	DISK=${DISKS%% *}
	default_setup_noexit $DISK
	log_must chmod 777 $TESTDIR

	# Use POSIX ACLs on filesystem
	log_must zfs set acltype=posix $TESTPOOL/$TESTFS
	-log_must zfs set xattr=sa $TESTPOOL/$TESTFS

	log_pass
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/Makefile.am
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/Makefile.am (nonexistent)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/Makefile.am (revision 366774)
	@@ -0,0 +1,8 @@
	+pkgdatadir = $(datadir)/@PACKAGE@/zfs-tests/tests/functional/acl/posix-sa
	+dist_pkgdata_SCRIPTS = \
	+ cleanup.ksh \
	+ setup.ksh \
	+ posix_001_pos.ksh \
	+ posix_002_pos.ksh \
	+ posix_003_pos.ksh \
	+ posix_004_pos.ksh

	Property changes on: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/Makefile.am
	___________________________________________________________________
	Added: svn:eol-style
	## -0,0 +1 ##
	+native
	\ No newline at end of property
	Added: svn:keywords
	## -0,0 +1 ##
	+FreeBSD=%H
	\ No newline at end of property
	Added: svn:mime-type
	## -0,0 +1 ##
	+text/plain
	\ No newline at end of property
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/cleanup.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/cleanup.ksh (nonexistent)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/cleanup.ksh (revision 366774)
	@@ -0,0 +1,33 @@
	+#!/bin/ksh -p
	+#
	+# CDDL HEADER START
	+#
	+# The contents of this file are subject to the terms of the
	+# Common Development and Distribution License (the "License").
	+# You may not use this file except in compliance with the License.
	+#
	+# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	+# or http://www.opensolaris.org/os/licensing.
	+# See the License for the specific language governing permissions
	+# and limitations under the License.
	+#
	+# When distributing Covered Code, include this CDDL HEADER in each
	+# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	+# If applicable, add the following below this CDDL HEADER, with the
	+# fields enclosed by brackets "[]" replaced with your own identifying
	+# information: Portions Copyright [yyyy] [name of copyright owner]
	+#
	+# CDDL HEADER END
	+#
	+
	+#
	+# Copyright 2007 Sun Microsystems, Inc. All rights reserved.
	+# Use is subject to license terms.
	+#
	+
	+. $STF_SUITE/include/libtest.shlib
	+. $STF_SUITE/tests/functional/acl/acl_common.kshlib
	+
	+cleanup_user_group
	+
	+default_cleanup
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/posix_001_pos.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/posix_001_pos.ksh (nonexistent)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/posix_001_pos.ksh (revision 366774)
	@@ -0,0 +1 @@
	+../posix/posix_001_pos.ksh
	\ No newline at end of file
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/posix_002_pos.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/posix_002_pos.ksh (nonexistent)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/posix_002_pos.ksh (revision 366774)
	@@ -0,0 +1 @@
	+../posix/posix_002_pos.ksh
	\ No newline at end of file
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/posix_003_pos.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/posix_003_pos.ksh (nonexistent)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/posix_003_pos.ksh (revision 366774)
	@@ -0,0 +1 @@
	+../posix/posix_003_pos.ksh
	\ No newline at end of file
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/posix_004_pos.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/posix_004_pos.ksh (nonexistent)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/posix_004_pos.ksh (revision 366774)
	@@ -0,0 +1 @@
	+../posix/posix_004_pos.ksh
	\ No newline at end of file
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/setup.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/setup.ksh (nonexistent)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/acl/posix-sa/setup.ksh (revision 366774)
	@@ -0,0 +1,52 @@
	+#!/bin/ksh -p
	+#
	+# CDDL HEADER START
	+#
	+# The contents of this file are subject to the terms of the
	+# Common Development and Distribution License (the "License").
	+# You may not use this file except in compliance with the License.
	+#
	+# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	+# or http://www.opensolaris.org/os/licensing.
	+# See the License for the specific language governing permissions
	+# and limitations under the License.
	+#
	+# When distributing Covered Code, include this CDDL HEADER in each
	+# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	+# If applicable, add the following below this CDDL HEADER, with the
	+# fields enclosed by brackets "[]" replaced with your own identifying
	+# information: Portions Copyright [yyyy] [name of copyright owner]
	+#
	+# CDDL HEADER END
	+#
	+
	+#
	+# Copyright 2007 Sun Microsystems, Inc. All rights reserved.
	+# Use is subject to license terms.
	+#
	+
	+#
	+# Copyright (c) 2016 by Delphix. All rights reserved.
	+#
	+
	+. $STF_SUITE/include/libtest.shlib
	+. $STF_SUITE/tests/functional/acl/acl_common.kshlib
	+
	+log_must getfacl --version
	+log_must setfacl --version
	+
	+cleanup_user_group
	+
	+# Create staff group and add user to it
	+log_must add_group $ZFS_ACL_STAFF_GROUP
	+log_must add_user $ZFS_ACL_STAFF_GROUP $ZFS_ACL_STAFF1
	+
	+DISK=${DISKS%% *}
	+default_setup_noexit $DISK
	+log_must chmod 777 $TESTDIR
	+
	+# Use POSIX ACLs on filesystem
	+log_must zfs set acltype=posix $TESTPOOL/$TESTFS
	+log_must zfs set xattr=sa $TESTPOOL/$TESTFS
	+
	+log_pass
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/arc/dbufstats_001_pos.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/arc/dbufstats_001_pos.ksh (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/arc/dbufstats_001_pos.ksh (revision 366774)
	@@ -1,84 +1,90 @@
	#!/bin/ksh -p
	#
	# CDDL HEADER START
	#
	# The contents of this file are subject to the terms of the
	# Common Development and Distribution License (the "License").
	# You may not use this file except in compliance with the License.
	#
	# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	# or http://www.opensolaris.org/os/licensing.
	# See the License for the specific language governing permissions
	# and limitations under the License.
	#
	# When distributing Covered Code, include this CDDL HEADER in each
	# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	# If applicable, add the following below this CDDL HEADER, with the
	# fields enclosed by brackets "[]" replaced with your own identifying
	# information: Portions Copyright [yyyy] [name of copyright owner]
	#
	# CDDL HEADER END
	#

	#
	# Copyright (c) 2017, Lawrence Livermore National Security, LLC.
	#

	. $STF_SUITE/include/libtest.shlib
	. $STF_SUITE/include/math.shlib

	#
	# DESCRIPTION:
	# Ensure stats presented in /proc/spl/kstat/zfs/dbufstats are correct
	# based on /proc/spl/kstat/zfs/dbufs.
	#
	# STRATEGY:
	# 1. Generate a file with random data in it
	# 2. Store output from dbufs kstat
	# 3. Store output from dbufstats kstat
	# 4. Compare stats presented in dbufstats with stat generated using
	# dbufstat and the dbufs kstat output
	#

	DBUFSTATS_FILE=$(mktemp $TEST_BASE_DIR/dbufstats.out.XXXXXX)
	DBUFS_FILE=$(mktemp $TEST_BASE_DIR/dbufs.out.XXXXXX)

	function cleanup
	{
	log_must rm -f $TESTDIR/file $DBUFS_FILE $DBUFSTATS_FILE
	}

	function testdbufstat # stat_name dbufstat_filter
	{
	name=$1
	filter=""

	[[ -n "$2" ]] && filter="-F $2"

	- from_dbufstat=$(grep -w "$name" "$DBUFSTATS_FILE" \| awk '{ print $3 }')
	+ if is_linux; then
	+ from_dbufstat=$(grep -w "$name" "$DBUFSTATS_FILE" \|
	+ awk '{ print $3 }')
	+ else
	+ from_dbufstat=$(awk "/dbufstats\.$name:/ { print \$2 }" \
	+ "$DBUFSTATS_FILE")
	+ fi
	from_dbufs=$(dbufstat -bxn -i "$DBUFS_FILE" "$filter" \| wc -l)

	within_tolerance $from_dbufstat $from_dbufs 15 \
	\|\| log_fail "Stat $name exceeded tolerance"
	}

	verify_runnable "both"

	log_assert "dbufstats produces correct statistics"

	log_onexit cleanup

	log_must file_write -o create -f "$TESTDIR/file" -b 1048576 -c 20 -d R
	log_must zpool sync

	-log_must eval "cat /proc/spl/kstat/zfs/dbufs > $DBUFS_FILE"
	-log_must eval "cat /proc/spl/kstat/zfs/dbufstats > $DBUFSTATS_FILE"
	+log_must eval "kstat dbufs > $DBUFS_FILE"
	+log_must eval "kstat dbufstats '' > $DBUFSTATS_FILE"

	for level in {0..11}; do
	testdbufstat "cache_level_$level" "dbc=1,level=$level"
	done

	testdbufstat "cache_count" "dbc=1"
	testdbufstat "hash_elements" ""

	log_pass "dbufstats produces correct statistics passed"
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/arc/dbufstats_002_pos.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/arc/dbufstats_002_pos.ksh (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/arc/dbufstats_002_pos.ksh (revision 366774)
	@@ -1,80 +1,80 @@
	#!/bin/ksh -p
	#
	# CDDL HEADER START
	#
	# The contents of this file are subject to the terms of the
	# Common Development and Distribution License (the "License").
	# You may not use this file except in compliance with the License.
	#
	# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	# or http://www.opensolaris.org/os/licensing.
	# See the License for the specific language governing permissions
	# and limitations under the License.
	#
	# When distributing Covered Code, include this CDDL HEADER in each
	# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	# If applicable, add the following below this CDDL HEADER, with the
	# fields enclosed by brackets "[]" replaced with your own identifying
	# information: Portions Copyright [yyyy] [name of copyright owner]
	#
	# CDDL HEADER END
	#

	#
	# Copyright (c) 2017, Lawrence Livermore National Security, LLC.
	#

	. $STF_SUITE/include/libtest.shlib
	. $STF_SUITE/include/math.shlib

	#
	# DESCRIPTION:
	# Ensure that dbufs move from mru to mfu as expected.
	#
	# STRATEGY:
	# 1. Set dbuf cache size to a small size (10M for this test)
	# 2. Generate a file with random data (small enough to fit in cache)
	# 3. zpool sync to remove dbufs from anon list in ARC
	# 4. Obtain the object ID using linux stat command
	# 5. Ensure that all dbufs are on the mru list in the ARC
	# 6. Generate another random file large enough to flush dbuf cache
	# 7. cat the first generated file
	# 8. Ensure that at least some dbufs moved to the mfu list in the ARC
	#

	DBUFS_FILE=$(mktemp $TEST_BASE_DIR/dbufs.out.XXXXXX)

	function cleanup
	{
	log_must rm -f $TESTDIR/file $TESTDIR/file2 $DBUFS_FILE
	}

	verify_runnable "both"

	log_assert "dbufs move from mru to mfu list"

	log_onexit cleanup

	log_must file_write -o create -f "$TESTDIR/file" -b 1048576 -c 1 -d R
	log_must zpool sync

	-objid=$(stat --format="%i" "$TESTDIR/file")
	+objid=$(get_objnum "$TESTDIR/file")
	log_note "Object ID for $TESTDIR/file is $objid"

	-log_must eval "cat /proc/spl/kstat/zfs/dbufs > $DBUFS_FILE"
	+log_must eval "kstat dbufs > $DBUFS_FILE"
	dbuf=$(dbufstat -bxn -i "$DBUFS_FILE" -F "object=$objid" \| wc -l)
	mru=$(dbufstat -bxn -i "$DBUFS_FILE" -F "object=$objid,list=1" \| wc -l)
	mfu=$(dbufstat -bxn -i "$DBUFS_FILE" -F "object=$objid,list=3" \| wc -l)
	log_note "dbuf count is $dbuf, mru count is $mru, mfu count is $mfu"
	verify_ne "0" "$mru" "mru count"
	verify_eq "0" "$mfu" "mfu count"

	log_must eval "cat $TESTDIR/file > /dev/null"
	-log_must eval "cat /proc/spl/kstat/zfs/dbufs > $DBUFS_FILE"
	+log_must eval "kstat dbufs > $DBUFS_FILE"
	dbuf=$(dbufstat -bxn -i "$DBUFS_FILE" -F "object=$objid" \| wc -l)
	mru=$(dbufstat -bxn -i "$DBUFS_FILE" -F "object=$objid,list=1" \| wc -l)
	mfu=$(dbufstat -bxn -i "$DBUFS_FILE" -F "object=$objid,list=3" \| wc -l)
	log_note "dbuf count is $dbuf, mru count is $mru, mfu count is $mfu"
	verify_ne "0" "$mfu" "mfu count"

	log_pass "dbufs move from mru to mfu list passed"
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/channel_program/synctask_core/tst.get_number_props.zcp
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/channel_program/synctask_core/tst.get_number_props.zcp (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/channel_program/synctask_core/tst.get_number_props.zcp (revision 366774)
	@@ -1,101 +1,101 @@
	--
	-- This file and its contents are supplied under the terms of the
	-- Common Development and Distribution License ("CDDL"), version 1.0.
	-- You may only use this file in accordance with the terms of version
	-- 1.0 of the CDDL.
	--
	-- A full copy of the text of the CDDL should have accompanied this
	-- source. A copy of the CDDL is also available via the Internet at
	-- http://www.illumos.org/license/CDDL.
	--

	--
	-- Copyright (c) 2016 by Delphix. All rights reserved.
	--

	arg = ...
	fs = arg["argv"][1]
	snap = arg["argv"][2]
	vol = arg["argv"][3]

	props = {}

	-- The values indicate whether or not a property should be returned,
	-- not the value of the property. A better approach might be to compare
	-- their values to the output of 'zfs get <prop>'

	-- prop filesystem snapshot volume
	props['used'] = {{true, nil}, {true, nil}, {true, nil}}
	props['available'] = {{true, nil}, {nil, nil}, {true, nil}}
	props['referenced'] = {{true, nil}, {true, nil}, {true, nil}}
	props['compressratio'] = {{true, nil}, {true, nil}, {true, nil}}
	props['refcompressratio'] = {{true, nil}, {true, nil}, {true, nil}}
	props['volblocksize'] = {{nil, nil}, {nil, nil}, {true, nil}}
	props['usedbysnapshots'] = {{true, nil}, {nil, nil}, {true, nil}}
	props['usedbydataset'] = {{true, nil}, {nil, nil}, {true, nil}}
	props['usedbychildren'] = {{true, nil}, {nil, nil}, {true, nil}}
	props['usedbyrefreservation'] = {{true, nil}, {nil, nil}, {true, nil}}
	props['userrefs'] = {{nil, nil}, {true, nil}, {nil, nil}}
	props['written'] = {{true, nil}, {true, nil}, {true, nil}}
	props['logicalused'] = {{true, nil}, {nil, nil}, {true, nil}}
	props['logicalreferenced'] = {{true, nil}, {true, nil}, {true, nil}}
	props['quota'] = {{true, 'default'}, {nil, nil}, {nil, nil}}
	props['reservation'] = {{true, 'default'}, {nil, nil}, {true, 'default'}}
	--- Note that zfsonlinux allows volsize for snapshot which differs from openzfs
	+-- Note that OpenZFS allows volsize for snapshot
	-- props['volsize'] = {{nil, nil}, {nil, nil}, {true, vol}}
	props['refquota'] = {{true, 'default'}, {nil, nil}, {nil, nil}}
	props['refreservation'] = {{true, 'default'}, {nil, nil}, {true, vol}}
	props['filesystem_limit'] = {{true, fs}, {nil, nil}, {nil, nil}}
	props['snapshot_limit'] = {{true, fs}, {nil, nil}, {true, vol}}
	props['filesystem_count'] = {{true, nil}, {nil, nil}, {nil, nil}}
	props['snapshot_count'] = {{true, nil}, {nil, nil}, {true, nil}}
	props['recordsize'] = {{true, 'default'}, {nil, nil}, {nil, nil}}
	props['creation'] = {{true, nil}, {true, nil}, {true, nil}}
	-- hidden props
	props['createtxg'] = {{true, nil}, {true, nil}, {true, nil}}
	props['numclones'] = {{nil, nil}, {true, nil}, {nil, nil}}
	props['guid'] = {{true, nil}, {true, nil}, {true, nil}}
	props['useraccounting'] = {{true, nil}, {true, nil}, {true, nil}}
	props['unique'] = {{true, nil}, {true, nil}, {true, nil}}
	props['objsetid'] = {{true, nil}, {true, nil}, {true, nil}}
	props['inconsistent'] = {{true, nil}, {true, nil}, {true, nil}}


	fs_fails = {}
	snap_fails = {}
	vol_fails = {}

	function match(n, ans, src, expected)
	if ((expected[n][1] == nil) and (ans ~= nil)) then
	return false
	end

	if ((expected[n][1] == true) and (ans == nil)) then
	return false
	end

	if (expected[n][2] ~= src) then
	return false
	end

	return true
	end

	for prop, expected in pairs(props) do
	ans, src = zfs.get_prop(fs, prop)
	if not (match(1, ans, src, expected)) then
	fs_fails[prop] = {ans, src}
	end

	ans, src = zfs.get_prop(snap, prop)
	if not (match(2, ans, src, expected)) then
	snap_fails[prop] = {ans, src}
	end

	ans, src = zfs.get_prop(vol, prop)
	if not (match(3, ans, src, expected)) then
	vol_fails[prop] = {ans, src}
	end
	end

	return {fs_fails, snap_fails, vol_fails}
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/cli_root/zfs_mount/zfs_mount_test_race.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/cli_root/zfs_mount/zfs_mount_test_race.ksh (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/cli_root/zfs_mount/zfs_mount_test_race.ksh (revision 366774)
	@@ -1,116 +1,116 @@
	#!/bin/ksh

	#
	# This file and its contents are supplied under the terms of the
	# Common Development and Distribution License ("CDDL"), version 1.0.
	# You may only use this file in accordance with the terms of version
	# 1.0 of the CDDL.
	#
	# A full copy of the text of the CDDL should have accompanied this
	# source. A copy of the CDDL is also available via the Internet at
	# http://www.illumos.org/license/CDDL.
	#

	#
	# Copyright (c) 2019 by Tomohiro Kusumi. All rights reserved.
	#

	. $STF_SUITE/include/libtest.shlib
	. $STF_SUITE/tests/functional/cli_root/zfs_mount/zfs_mount.cfg

	#
	# DESCRIPTION:
	# Verify parallel mount ordering is consistent.
	#
	# There was a bug in initial thread dispatching algorithm which put threads
	# under race condition which resulted in undefined mount order. The purpose
	# of this test is to verify `zfs unmount -a` succeeds (not `zfs mount -a`
	# succeeds, it always does) after `zfs mount -a`, which could fail if threads
	-# race. See github.com/zfsonlinux/zfs/issues/{8450,8833,8878} for details.
	+# race. See github.com/openzfs/zfs/issues/{8450,8833,8878} for details.
	#
	# STRATEGY:
	# 1. Create pools and filesystems.
	# 2. Set same mount point for >1 datasets.
	# 3. Unmount all datasets.
	# 4. Mount all datasets.
	# 5. Unmount all datasets (verify this succeeds).
	#

	verify_runnable "both"

	TMPDIR=${TMPDIR:-$TEST_BASE_DIR}
	MNTPT=$TMPDIR/zfs_mount_test_race_mntpt
	DISK1="$TMPDIR/zfs_mount_test_race_disk1"
	DISK2="$TMPDIR/zfs_mount_test_race_disk2"

	TESTPOOL1=zfs_mount_test_race_tp1
	TESTPOOL2=zfs_mount_test_race_tp2

	export __ZFS_POOL_RESTRICT="$TESTPOOL1 $TESTPOOL2"
	log_must zfs $unmountall
	unset __ZFS_POOL_RESTRICT

	function cleanup
	{
	zpool destroy $TESTPOOL1
	zpool destroy $TESTPOOL2
	rm -rf $MNTPT
	rm -rf /$TESTPOOL1
	rm -rf /$TESTPOOL2
	rm -f $DISK1
	rm -f $DISK2
	export __ZFS_POOL_RESTRICT="$TESTPOOL1 $TESTPOOL2"
	log_must zfs $mountall
	unset __ZFS_POOL_RESTRICT
	}
	log_onexit cleanup

	log_note "Verify parallel mount ordering is consistent"

	log_must truncate -s $MINVDEVSIZE $DISK1
	log_must truncate -s $MINVDEVSIZE $DISK2

	log_must zpool create -f $TESTPOOL1 $DISK1
	log_must zpool create -f $TESTPOOL2 $DISK2

	log_must zfs create $TESTPOOL1/$TESTFS1
	log_must zfs create $TESTPOOL2/$TESTFS2

	log_must zfs set mountpoint=none $TESTPOOL1
	log_must zfs set mountpoint=$MNTPT $TESTPOOL1/$TESTFS1

	# Note that unmount can fail (due to race condition on `zfs mount -a`) with or
	# without `canmount=off`. The race has nothing to do with canmount property,
	# but turn it off for convenience of mount layout used in this test case.
	log_must zfs set canmount=off $TESTPOOL2
	log_must zfs set mountpoint=$MNTPT $TESTPOOL2

	# At this point, layout of datasets in two pools will look like below.
	# Previously, on next `zfs mount -a`, pthreads assigned to TESTFS1 and TESTFS2
	# could race, and TESTFS2 usually (actually always) won in ZoL. Note that the
	# problem is how two or more threads could initially be assigned to the same
	# top level directory, not this specific layout. This layout is just an example
	# that can reproduce race, and is also the layout reported in #8833.
	#
	# NAME MOUNTED MOUNTPOINT
	# ----------------------------------------------
	# /$TESTPOOL1 no none
	# /$TESTPOOL1/$TESTFS1 yes $MNTPT
	# /$TESTPOOL2 no $MNTPT
	# /$TESTPOOL2/$TESTFS2 yes $MNTPT/$TESTFS2

	# Apparently two datasets must be mounted.
	log_must ismounted $TESTPOOL1/$TESTFS1
	log_must ismounted $TESTPOOL2/$TESTFS2
	# This unmount always succeeds, because potential race hasn't happened yet.
	log_must zfs unmount -a
	# This mount always succeeds, whether threads are under race condition or not.
	log_must zfs mount -a

	# Verify datasets are mounted (TESTFS2 fails if the race broke mount order).
	log_must ismounted $TESTPOOL1/$TESTFS1
	log_must ismounted $TESTPOOL2/$TESTFS2
	# Verify unmount succeeds (fails if the race broke mount order).
	log_must zfs unmount -a

	log_pass "Verify parallel mount ordering is consistent passed"
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/cli_root/zfs_send/zfs_send_sparse.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/cli_root/zfs_send/zfs_send_sparse.ksh (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/cli_root/zfs_send/zfs_send_sparse.ksh (revision 366774)
	@@ -1,83 +1,83 @@
	#!/bin/ksh -p
	#
	# This file and its contents are supplied under the terms of the
	# Common Development and Distribution License ("CDDL"), version 1.0.
	# You may only use this file in accordance with the terms of version
	# 1.0 of the CDDL.
	#
	# A full copy of the text of the CDDL should have accompanied this
	# source. A copy of the CDDL is also available via the Internet at
	# http://www.illumos.org/license/CDDL.
	#

	#
	# Copyright 2017, loli10K <ezomori.nozomu@gmail.com>. All rights reserved.
	#

	. $STF_SUITE/include/libtest.shlib

	#
	# DESCRIPTION:
	# 'zfs send' should be able to send (big) sparse files correctly.
	#
	# STRATEGY:
	# 1. Create sparse files of various size
	# 2. Snapshot and send these sparse files
	# 3. Verify these files are received correctly and we don't trigger any issue
	-# like the one described in https://github.com/zfsonlinux/zfs/pull/6760
	+# like the one described in https://github.com/openzfs/zfs/pull/6760
	#

	verify_runnable "both"

	function cleanup
	{
	datasetexists $SENDFS && log_must zfs destroy -r $SENDFS
	datasetexists $RECVFS && log_must zfs destroy -r $RECVFS
	}

	#
	# Write 1 random byte at $offset of "source" file in $sendfs dataset
	# Snapshot and send $sendfs dataset to $recvfs
	# Compare the received file with its source
	#
	function write_compare_files # <sendfs> <recvfs> <offset>
	{
	typeset sendfs="$1"
	typeset recvfs="$2"
	typeset offset="$3"

	# create source filesystem
	log_must zfs create $sendfs
	# write sparse file
	sendfile="$(get_prop mountpoint $sendfs)/data.bin"
	log_must dd if=/dev/urandom of=$sendfile bs=1 count=1 seek=$offset
	# send/receive the file
	log_must zfs snapshot $sendfs@snap
	log_must eval "zfs send $sendfs@snap \| zfs receive $recvfs"
	# compare sparse files
	recvfile="$(get_prop mountpoint $recvfs)/data.bin"
	log_must cmp $sendfile $recvfile $offset $offset
	sendsz=$(stat_size $sendfile)
	recvsz=$(stat_size $recvfile)
	if [[ $sendsz -ne $recvsz ]]; then
	log_fail "$sendfile ($sendsz) and $recvfile ($recvsz) differ."
	fi
	# cleanup
	log_must zfs destroy -r $sendfs
	log_must zfs destroy -r $recvfs
	}

	log_assert "'zfs send' should be able to send (big) sparse files correctly."
	log_onexit cleanup

	SENDFS="$TESTPOOL/sendfs"
	RECVFS="$TESTPOOL/recvfs"
	OFF_T_MAX="$(echo '2 ^ 40 * 8 - 1' \| bc)"

	for i in {1..60}; do
	offset=$(echo "2 ^ $i" \| bc)
	[[ is_32bit ]] && [[ $offset -ge $OFF_T_MAX ]] && continue;
	write_compare_files $SENDFS $RECVFS $offset
	done

	log_pass "'zfs send' sends (big) sparse files correctly."
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/cli_user/misc/misc.cfg
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/cli_user/misc/misc.cfg (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/cli_user/misc/misc.cfg (revision 366774)
	@@ -1,123 +1,123 @@
	#
	# CDDL HEADER START
	#
	# The contents of this file are subject to the terms of the
	# Common Development and Distribution License (the "License").
	# You may not use this file except in compliance with the License.
	#
	# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	# or http://www.opensolaris.org/os/licensing.
	# See the License for the specific language governing permissions
	# and limitations under the License.
	#
	# When distributing Covered Code, include this CDDL HEADER in each
	# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	# If applicable, add the following below this CDDL HEADER, with the
	# fields enclosed by brackets "[]" replaced with your own identifying
	# information: Portions Copyright [yyyy] [name of copyright owner]
	#
	# CDDL HEADER END
	#

	#
	# Copyright 2007 Sun Microsystems, Inc. All rights reserved.
	# Use is subject to license terms.
	#

	#
	# Copyright (c) 2013 by Delphix. All rights reserved.
	#

	if is_linux; then
	# these are the set of setable ZFS properties
	PROP_NAMES="\
	acltype atime \
	checksum compression devices \
	exec mountpoint quota readonly \
	recordsize reservation setuid \
	snapdir"

	# these are a set of values we apply, for use when testing the
	# zfs get/set subcommands - ordered as per the list above so we
	# can iterate over both sets in an array
	PROP_VALS="\
	posix on \
	fletcher2 on on \
	on legacy none on \
	128K none on \
	visible"

	# these are an alternate set of property values
	PROP_ALTVALS="\
	- off off \
	+ nfsv4 off \
	fletcher4 lzjb off \
	off /tmp/zfstest 100M off \
	512 10m off \
	hidden"
	elif is_freebsd; then
	PROP_NAMES="\
	acltype atime \
	checksum compression devices \
	exec mountpoint quota readonly \
	recordsize reservation setuid \
	snapdir"

	# these are a set of values we apply, for use when testing the
	# zfs get/set subcommands - ordered as per the list above so we
	# can iterate over both sets in an array
	PROP_VALS="\
	posix on \
	fletcher2 on on \
	on legacy none on \
	128K none on \
	visible"

	# these are an alternate set of property values
	PROP_ALTVALS="\
	- off off \
	+ nfsv4 off \
	fletcher4 lzjb off \
	off /tmp/zfstest 100M off \
	512 10m off \
	hidden"

	else
	# these are the set of setable ZFS properties
	PROP_NAMES="\
	aclinherit aclmode atime \
	checksum compression devices \
	exec mountpoint quota readonly \
	recordsize reservation setuid sharenfs \
	snapdir"

	# these are a set of values we apply, for use when testing the
	# zfs get/set subcommands - ordered as per the list above so we
	# can iterate over both sets in an array
	PROP_VALS="\
	passthrough discard on \
	fletcher2 on on \
	on legacy none on \
	128K none on on \
	visible"

	# these are an alternate set of property values
	PROP_ALTVALS="\
	passthrough-x groupmask off \
	fletcher4 lzjb off \
	off /tmp/zfstest 100M off \
	512 10m off off \
	hidden"
	fi

	# additional properties to worry about: canmount copies xattr zoned version

	POOL_PROPS="\
	failmode autoreplace"

	POOL_VALS="\
	continue on"

	POOL_ALTVALS="\
	panic off"

	export TESTSNAP=testsnap-misc
	export TESTCLCT=testclct-misc
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/history/history_002_pos.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/history/history_002_pos.ksh (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/history/history_002_pos.ksh (revision 366774)
	@@ -1,198 +1,199 @@
	#!/bin/ksh -p
	#
	# CDDL HEADER START
	#
	# The contents of this file are subject to the terms of the
	# Common Development and Distribution License (the "License").
	# You may not use this file except in compliance with the License.
	#
	# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	# or http://www.opensolaris.org/os/licensing.
	# See the License for the specific language governing permissions
	# and limitations under the License.
	#
	# When distributing Covered Code, include this CDDL HEADER in each
	# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	# If applicable, add the following below this CDDL HEADER, with the
	# fields enclosed by brackets "[]" replaced with your own identifying
	# information: Portions Copyright [yyyy] [name of copyright owner]
	#
	# CDDL HEADER END
	#

	#
	# Copyright 2009 Sun Microsystems, Inc. All rights reserved.
	# Use is subject to license terms.

	#
	# Copyright (c) 2013, 2016 by Delphix. All rights reserved.
	#

	. $STF_SUITE/tests/functional/history/history_common.kshlib

	#
	# DESCRIPTION:
	# Create a scenario to verify the following zfs subcommands are logged.
	# create, destroy, clone, rename, snapshot, rollback, set, inherit,
	# receive, promote, hold and release.
	#
	# STRATEGY:
	# 1. Verify that all the zfs commands listed (barring send) produce an
	# entry in the pool history.
	#

	verify_runnable "global"

	function cleanup
	{

	[[ -f $tmpfile ]] && rm -f $tmpfile
	[[ -f $tmpfile2 ]] && rm -f $tmpfile2
	for dataset in $fs $newfs $fsclone $vol $newvol $volclone; do
	datasetexists $dataset && zfs destroy -Rf $dataset
	done
	rm -rf /history.$$
	}

	log_assert "Verify zfs sub-commands which modify state are logged."
	log_onexit cleanup

	fs=$TESTPOOL/$TESTFS1; newfs=$TESTPOOL/newfs; fsclone=$TESTPOOL/clone
	vol=$TESTPOOL/$TESTVOL ; newvol=$TESTPOOL/newvol; volclone=$TESTPOOL/volclone
	fssnap=$fs@fssnap; fssnap2=$fs@fssnap2
	volsnap=$vol@volsnap; volsnap2=$vol@volsnap2
	tmpfile=$TEST_BASE_DIR/tmpfile.$$ ; tmpfile2=$TEST_BASE_DIR/tmpfile2.$$

	if is_linux; then
	# property value property value
	#
	props=(
	quota 64M recordsize 512
	reservation 32M reservation none
	mountpoint /history.$$ mountpoint legacy
	mountpoint none compression lz4
	compression on compression off
	compression lzjb acltype off
	- acltype posix xattr sa
	+ acltype posix acltype nfsv4
	atime on atime off
	devices on devices off
	exec on exec off
	setuid on setuid off
	readonly on readonly off
	zoned on zoned off
	snapdir hidden snapdir visible
	aclinherit discard aclinherit noallow
	aclinherit secure aclinherit passthrough
	canmount off canmount on
	- xattr on xattr off
	compression gzip compression gzip-$((RANDOM%9 + 1))
	compression zstd compression zstd-$((RANDOM%9 + 1))
	compression zstd-fast copies $((RANDOM%3 + 1))
	- compression zstd-fast-$((RANDOM%9 + 1))
	+ compression zstd-fast-$((RANDOM%9 + 1)) xattr sa
	+ xattr on xattr off
	)
	elif is_freebsd; then
	# property value property value
	#
	props=(
	quota 64M recordsize 512
	reservation 32M reservation none
	mountpoint /history.$$ mountpoint legacy
	mountpoint none sharenfs on
	sharenfs off
	compression on compression off
	compression lzjb aclmode discard
	aclmode groupmask aclmode passthrough
	atime on atime off
	devices on devices off
	exec on exec off
	setuid on setuid off
	readonly on readonly off
	jailed on jailed off
	snapdir hidden snapdir visible
	aclinherit discard aclinherit noallow
	aclinherit secure aclinherit passthrough
	canmount off canmount on
	compression gzip compression gzip-$((RANDOM%9 + 1))
	compression zstd compression zstd-$((RANDOM%9 + 1))
	compression zstd-fast copies $((RANDOM%3 + 1))
	- compression zstd-fast-$((RANDOM%9 + 1))
	+ compression zstd-fast-$((RANDOM%9 + 1)) acltype off
	+ acltype posix acltype nfsv4
	)
	else
	# property value property value
	#
	props=(
	quota 64M recordsize 512
	reservation 32M reservation none
	mountpoint /history.$$ mountpoint legacy
	mountpoint none sharenfs on
	sharenfs off
	compression on compression off
	compression lzjb aclmode discard
	aclmode groupmask aclmode passthrough
	atime on atime off
	devices on devices off
	exec on exec off
	setuid on setuid off
	readonly on readonly off
	zoned on zoned off
	snapdir hidden snapdir visible
	aclinherit discard aclinherit noallow
	aclinherit secure aclinherit passthrough
	canmount off canmount on
	xattr on xattr off
	compression gzip compression gzip-$((RANDOM%9 + 1))
	copies $((RANDOM%3 + 1))
	)
	fi

	run_and_verify "zfs create $fs"
	# Set all the property for filesystem
	typeset -i i=0
	while ((i < ${#props[@]})) ; do
	run_and_verify "zfs set ${props[$i]}=${props[((i+1))]} $fs"

	# quota, reservation, canmount can not be inherited.
	#
	if [[ ${props[$i]} != "quota" && ${props[$i]} != "reservation" && \
	${props[$i]} != "canmount" ]];
	then
	run_and_verify "zfs inherit ${props[$i]} $fs"
	fi

	((i += 2))
	done

	run_and_verify "zfs create -V 64M $vol"
	run_and_verify "zfs set volsize=32M $vol"
	run_and_verify "zfs snapshot $fssnap"
	run_and_verify "zfs hold tag $fssnap"
	run_and_verify "zfs release tag $fssnap"
	run_and_verify "zfs snapshot $volsnap"
	run_and_verify "zfs snapshot $fssnap2"
	run_and_verify "zfs snapshot $volsnap2"

	# Send isn't logged...
	log_must eval "zfs send -i $fssnap $fssnap2 > $tmpfile"
	log_must eval "zfs send -i $volsnap $volsnap2 > $tmpfile2"
	# Verify that's true
	zpool history $TESTPOOL \| grep 'zfs send' >/dev/null 2>&1 && \
	log_fail "'zfs send' found in history of \"$TESTPOOL\""

	run_and_verify "zfs destroy $fssnap2"
	run_and_verify "zfs destroy $volsnap2"
	run_and_verify "zfs receive $fs < $tmpfile"
	run_and_verify "zfs receive $vol < $tmpfile2"
	run_and_verify "zfs rollback -r $fssnap"
	run_and_verify "zfs rollback -r $volsnap"
	run_and_verify "zfs clone $fssnap $fsclone"
	run_and_verify "zfs clone $volsnap $volclone"
	run_and_verify "zfs rename $fs $newfs"
	run_and_verify "zfs rename $vol $newvol"
	run_and_verify "zfs promote $fsclone"
	run_and_verify "zfs promote $volclone"
	run_and_verify "zfs destroy $newfs"
	run_and_verify "zfs destroy $newvol"
	run_and_verify "zfs destroy -rf $fsclone"
	run_and_verify "zfs destroy -rf $volclone"

	log_pass "zfs sub-commands which modify state are logged passed."
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/nopwrite/nopwrite_recsize.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/nopwrite/nopwrite_recsize.ksh (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/nopwrite/nopwrite_recsize.ksh (revision 366774)
	@@ -1,57 +1,57 @@
	#!/bin/ksh

	#
	# This file and its contents are supplied under the terms of the
	# Common Development and Distribution License ("CDDL"), version 1.0.
	# You may only use this file in accordance with the terms of version
	# 1.0 of the CDDL.
	#
	# A full copy of the text of the CDDL should have accompanied this
	# source. A copy of the CDDL is also available via the Internet at
	# http://www.illumos.org/license/CDDL.
	#

	#
	# Copyright (c) 2012, 2016 by Delphix. All rights reserved.
	#

	. $STF_SUITE/include/libtest.shlib
	. $STF_SUITE/tests/functional/nopwrite/nopwrite.shlib

	#
	# Description:
	# Verify that nopwrite works regardless of recsize property setting.
	#
	# Strategy:
	# 1. Create an origin fs that's suitable to make nopwrite clones.
	# 2. For each possible recsize, create a clone that inherits the compress and
	# checksum, and verify overwriting the origin file consumes no new space.
	#

	verify_runnable "global"
	origin="$TESTPOOL/$TESTFS"
	log_onexit cleanup

	function cleanup
	{
	datasetexists $origin && log_must zfs destroy -R $origin
	log_must zfs create -o mountpoint=$TESTDIR $origin
	}

	log_assert "nopwrite updates file metadata correctly"

	log_must zfs set compress=on $origin
	log_must zfs set checksum=sha256 $origin
	dd if=/dev/urandom of=$TESTDIR/file bs=1024k count=$MEGS conv=notrunc \
	>/dev/null 2>&1 \|\| log_fail "dd into $TESTDIR/file failed."
	zfs snapshot $origin@a \|\| log_fail "zfs snap failed"
	log_must zfs clone $origin@a $origin/clone

	for rs in 512 1024 2048 4096 8192 16384 32768 65536 131072 ; do
	log_must zfs set recsize=$rs $origin/clone
	dd if=/$TESTDIR/file of=/$TESTDIR/clone/file bs=1024k count=$MEGS \
	- conv=notrunc > $TEST_BASE_DIR/null 2>&1 \|\| log_fail "dd failed."
	+ conv=notrunc >/dev/null 2>&1 \|\| log_fail "dd failed."
	log_must verify_nopwrite $origin $origin@a $origin/clone
	done

	log_pass "nopwrite updates file metadata correctly"
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/redacted_send/redacted_negative.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/redacted_send/redacted_negative.ksh (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/redacted_send/redacted_negative.ksh (revision 366774)
	@@ -1,92 +1,94 @@
	#!/bin/ksh

	#
	# This file and its contents are supplied under the terms of the
	# Common Development and Distribution License ("CDDL"), version 1.0.
	# You may only use this file in accordance with the terms of version
	# 1.0 of the CDDL.
	#
	# A full copy of the text of the CDDL should have accompanied this
	# source. A copy of the CDDL is also available via the Internet at
	# http://www.illumos.org/license/CDDL.
	#

	#
	# Copyright (c) 2018 by Delphix. All rights reserved.
	#

	. $STF_SUITE/tests/functional/redacted_send/redacted.kshlib

	#
	# Description:
	# Test that redacted send correctly detects invalid arguments.
	#

	typeset sendfs="$POOL2/sendfs"
	typeset recvfs="$POOL2/recvfs"
	typeset clone1="$POOL2/clone1"
	typeset clone2="$POOL2/clone2"
	typeset clone3="$POOL2/clone3"
	typeset clone3="$POOL2/clone4"
	typeset tmpdir="$(get_prop mountpoint $POOL)/tmp"
	typeset stream=$(mktemp $tmpdir/stream.XXXX)

	log_onexit redacted_cleanup $sendfs $recvfs $clone3

	log_must zfs create $sendfs
	log_must zfs snapshot $sendfs@snap1
	log_must zfs snapshot $sendfs@snap2
	log_must zfs snapshot $sendfs@snap3
	log_must zfs clone $sendfs@snap2 $clone1
	log_must zfs snapshot $clone1@snap
	log_must zfs bookmark $clone1@snap $clone1#book
	log_must zfs clone $sendfs@snap2 $clone2
	log_must zfs snapshot $clone2@snap

	# Incompatible flags
	log_must zfs redact $sendfs@snap2 book $clone1@snap
	log_mustnot eval "zfs send -R --redact book $sendfs@snap2 >/dev/null"

	typeset arg
	for arg in "$sendfs" "$clone1#book"; do
	log_mustnot eval "zfs send --redact book $arg >/dev/null"
	done

	# Bad redaction list arguments
	log_mustnot zfs redact $sendfs@snap1
	log_mustnot zfs redact $sendfs@snap1 book
	log_mustnot zfs redact $sendfs#book1 book4 $clone1
	+log_mustnot zfs redact $sendfs@snap1 book snap2 snap3
	+log_mustnot zfs redact $sendfs@snap1 book @snap2 @snap3
	log_mustnot eval "zfs send --redact $sendfs#book $sendfs@snap >/dev/null"

	# Redaction snapshots not a descendant of tosnap
	log_mustnot zfs redact $sendfs@snap2 book $sendfs@snap2
	log_must zfs redact $sendfs@snap2 book2 $clone1@snap $clone2@snap
	log_must eval "zfs send --redact book2 $sendfs@snap2 >$stream"
	log_must zfs redact $sendfs@snap2 book3 $clone1@snap $clone2@snap
	log_must eval "zfs send -i $sendfs@snap1 --redact book3 $sendfs@snap2 \
	>/dev/null"
	log_mustnot zfs redact $sendfs@snap3 $sendfs@snap3 $clone1@snap

	# Full redacted sends of redacted datasets are not allowed.
	log_must eval "zfs recv $recvfs <$stream"
	log_must zfs snapshot $recvfs@snap
	log_must zfs clone $recvfs@snap $clone3
	log_must zfs snapshot $clone3@snap
	log_mustnot zfs redact $recvfs@snap book5 $clone3@snap

	# Nor may a redacted dataset appear in the redaction list.
	log_mustnot zfs redact testpool2/recvfs@snap2 book7 testpool2/recvfs@snap

	# Non-redaction bookmark cannot be sent and produces invalid argument error
	log_must zfs bookmark "$sendfs@snap1" "$sendfs#book8"
	log_must eval "zfs send --redact book8 -i $sendfs@snap1 $sendfs@snap2 2>&1 \| head -n 100 \| grep 'not a redaction bookmark'"

	# Error messages for common usage errors
	log_mustnot_expect "not contain '#'" zfs redact $sendfs@snap1 \#book $sendfs@snap2
	log_mustnot_expect "not contain '#'" zfs redact $sendfs@snap1 $sendfs#book $sendfs@snap2
	log_mustnot_expect "full dataset names" zfs redact $sendfs@snap1 book @snap2
	log_mustnot_expect "full dataset names" zfs redact $sendfs@snap1 book @snap2
	log_mustnot_expect "full dataset names" zfs redact $sendfs@snap1 \#book @snap2
	log_mustnot_expect "descendent of snapshot" zfs redact $sendfs@snap2 book $sendfs@snap1

	log_pass "Verify that redacted send correctly detects invalid arguments."
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/rsend/rsend.kshlib
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/rsend/rsend.kshlib (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/rsend/rsend.kshlib (revision 366774)
	@@ -1,850 +1,857 @@
	#
	# CDDL HEADER START
	#
	# The contents of this file are subject to the terms of the
	# Common Development and Distribution License (the "License").
	# You may not use this file except in compliance with the License.
	#
	# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	# or http://www.opensolaris.org/os/licensing.
	# See the License for the specific language governing permissions
	# and limitations under the License.
	#
	# When distributing Covered Code, include this CDDL HEADER in each
	# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	# If applicable, add the following below this CDDL HEADER, with the
	# fields enclosed by brackets "[]" replaced with your own identifying
	# information: Portions Copyright [yyyy] [name of copyright owner]
	#
	# CDDL HEADER END
	#

	#
	# Copyright 2009 Sun Microsystems, Inc. All rights reserved.
	# Use is subject to license terms.
	#

	#
	# Copyright (c) 2013, 2018 by Delphix. All rights reserved.
	# Copyright (c) 2020 by Datto Inc. All rights reserved.
	#

	. $STF_SUITE/include/libtest.shlib
	. $STF_SUITE/include/math.shlib
	. $STF_SUITE/tests/functional/cli_root/zfs_set/zfs_set_common.kshlib
	. $STF_SUITE/tests/functional/rsend/rsend.cfg

	#
	# Set up test model which includes various datasets
	#
	# @final
	# @snapB
	# @init
	# \|
	# ______ pclone
	# \| /
	# \|@psnap
	# \|\| @final
	# \|\|@final @final @snapC
	# \|\|@snapC @snapC @snapB
	# \|\|@snapA @snapB @snapA
	# \|\|@init @init @init
	# \|\|\| \| \|
	# $pool -------- $FS ------- fs1 ------- fs2
	# \ \\_____ \ \|
	# vol vol \____ \ @fsnap
	# \| \| \ \ \
	# @init @vsnap \| ------------ fclone
	# @snapA @init \ \| \|
	# @final @snapB \ \| @init
	# @snapC vclone @snapA
	# @final \| @final
	# @init
	# @snapC
	# @final
	#
	# $1 pool name
	#
	function setup_test_model
	{
	typeset pool=$1

	log_must zfs create -p $pool/$FS/fs1/fs2

	log_must zfs snapshot $pool@psnap
	log_must zfs clone $pool@psnap $pool/pclone

	if is_global_zone ; then
	log_must zfs create -V 16M $pool/vol
	log_must zfs create -V 16M $pool/$FS/vol
	block_device_wait

	log_must zfs snapshot $pool/$FS/vol@vsnap
	log_must zfs clone $pool/$FS/vol@vsnap $pool/$FS/vclone
	block_device_wait
	fi

	log_must snapshot_tree $pool/$FS/fs1/fs2@fsnap
	log_must zfs clone $pool/$FS/fs1/fs2@fsnap $pool/$FS/fs1/fclone
	log_must zfs snapshot -r $pool@init

	log_must snapshot_tree $pool@snapA
	log_must snapshot_tree $pool@snapC
	log_must snapshot_tree $pool/pclone@snapB
	log_must snapshot_tree $pool/$FS@snapB
	log_must snapshot_tree $pool/$FS@snapC
	log_must snapshot_tree $pool/$FS/fs1@snapA
	log_must snapshot_tree $pool/$FS/fs1@snapB
	log_must snapshot_tree $pool/$FS/fs1@snapC
	log_must snapshot_tree $pool/$FS/fs1/fclone@snapA

	if is_global_zone ; then
	log_must zfs snapshot $pool/vol@snapA
	log_must zfs snapshot $pool/$FS/vol@snapB
	log_must zfs snapshot $pool/$FS/vol@snapC
	log_must zfs snapshot $pool/$FS/vclone@snapC
	fi

	log_must zfs snapshot -r $pool@final

	return 0
	}

	#
	# Cleanup the BACKDIR and given pool content and all the sub datasets
	#
	# $1 pool name
	#
	function cleanup_pool
	{
	typeset pool=$1
	log_must rm -rf $BACKDIR/*

	if is_global_zone ; then
	log_must_busy zfs destroy -Rf $pool
	else
	typeset list=$(zfs list -H -r -t all -o name $pool)
	for ds in $list ; do
	if [[ $ds != $pool ]] ; then
	if datasetexists $ds ; then
	log_must_busy zfs destroy -Rf $ds
	fi
	fi
	done
	fi

	typeset mntpnt=$(get_prop mountpoint $pool)
	if ! ismounted $pool ; then
	# Make sure mountpoint directory is empty
	if [[ -d $mntpnt ]]; then
	log_must rm -rf $mntpnt/*
	fi

	log_must zfs mount $pool
	fi
	if [[ -d $mntpnt ]]; then
	rm -rf $mntpnt/*
	fi

	return 0
	}

	function cleanup_pools
	{
	cleanup_pool $POOL2
	destroy_pool $POOL3
	}

	function cmp_md5s {
	typeset file1=$1
	typeset file2=$2

	typeset sum1=$(md5digest $file1)
	typeset sum2=$(md5digest $file2)
	test "$sum1" = "$sum2"
	}

	#
	# Detect if the given two filesystems have same sub-datasets
	#
	# $1 source filesystem
	# $2 destination filesystem
	#
	function cmp_ds_subs
	{
	typeset src_fs=$1
	typeset dst_fs=$2

	zfs list -r -H -t all -o name $src_fs > $BACKDIR/src1
	zfs list -r -H -t all -o name $dst_fs > $BACKDIR/dst1

	eval sed -e 's:^$src_fs:PREFIX:g' < $BACKDIR/src1 > $BACKDIR/src
	eval sed -e 's:^$dst_fs:PREFIX:g' < $BACKDIR/dst1 > $BACKDIR/dst

	diff $BACKDIR/src $BACKDIR/dst
	typeset -i ret=$?

	rm -f $BACKDIR/src $BACKDIR/dst $BACKDIR/src1 $BACKDIR/dst1

	return $ret
	}

	#
	# Compare all the directories and files in two filesystems
	#
	# $1 source filesystem
	# $2 destination filesystem
	#
	function cmp_ds_cont
	{
	typeset src_fs=$1
	typeset dst_fs=$2

	typeset srcdir dstdir
	srcdir=$(get_prop mountpoint $src_fs)
	dstdir=$(get_prop mountpoint $dst_fs)

	diff -r $srcdir $dstdir > /dev/null 2>&1
	return $?
	}

	#
	# Compare the given two dataset properties
	#
	# $1 dataset 1
	# $2 dataset 2
	#
	function cmp_ds_prop
	{
	typeset dtst1=$1
	typeset dtst2=$2
	typeset -a props=("type" "origin" "volblocksize" "acltype" "dnodesize" \
	"atime" "canmount" "checksum" "compression" "copies" "devices" \
	"exec" "quota" "readonly" "recordsize" "reservation" "setuid" \
	"snapdir" "version" "volsize" "xattr" "mountpoint");
	if is_freebsd; then
	props+=("jailed")
	else
	props+=("zoned")
	fi

	for prop in $props;
	do
	zfs get -H -o property,value,source $prop $dtst1 >> \
	$BACKDIR/dtst1
	zfs get -H -o property,value,source $prop $dtst2 >> \
	$BACKDIR/dtst2
	done

	eval sed -e 's:$dtst1:PREFIX:g' < $BACKDIR/dtst1 > $BACKDIR/dtst1
	eval sed -e 's:$dtst2:PREFIX:g' < $BACKDIR/dtst2 > $BACKDIR/dtst2

	diff $BACKDIR/dtst1 $BACKDIR/dtst2
	typeset -i ret=$?

	rm -f $BACKDIR/dtst1 $BACKDIR/dtst2

	return $ret

	}

	#
	# Random create directories and files
	#
	# $1 directory
	#
	function random_tree
	{
	typeset dir=$1

	if [[ -d $dir ]]; then
	rm -rf $dir
	fi
	mkdir -p $dir
	typeset -i ret=$?

	typeset -i nl nd nf
	((nl = RANDOM % 6 + 1))
	((nd = RANDOM % 3 ))
	((nf = RANDOM % 5 ))
	mktree -b $dir -l $nl -d $nd -f $nf
	((ret \|= $?))

	return $ret
	}

	#
	# Put data in filesystem and take snapshot
	#
	# $1 snapshot name
	#
	function snapshot_tree
	{
	typeset snap=$1
	typeset ds=${snap%%@*}
	typeset type=$(get_prop "type" $ds)

	typeset -i ret=0
	if [[ $type == "filesystem" ]]; then
	typeset mntpnt=$(get_prop mountpoint $ds)
	((ret \|= $?))

	if ((ret == 0)) ; then
	eval random_tree $mntpnt/${snap##$ds}
	((ret \|= $?))
	fi
	fi

	if ((ret == 0)) ; then
	zfs snapshot $snap
	((ret \|= $?))
	fi

	return $ret
	}

	#
	# Destroy the given snapshot and stuff
	#
	# $1 snapshot
	#
	function destroy_tree
	{
	typeset -i ret=0
	typeset snap
	for snap in "$@" ; do
	log_must_busy zfs destroy $snap

	typeset ds=${snap%%@*}
	typeset type=$(get_prop "type" $ds)
	if [[ $type == "filesystem" ]]; then
	typeset mntpnt=$(get_prop mountpoint $ds)
	if [[ -n $mntpnt ]]; then
	rm -rf $mntpnt/$snap
	fi
	fi
	done

	return 0
	}

	#
	# Get all the sub-datasets of give dataset with specific suffix
	#
	# $1 Given dataset
	# $2 Suffix
	#
	function getds_with_suffix
	{
	typeset ds=$1
	typeset suffix=$2

	typeset list=$(zfs list -r -H -t all -o name $ds \| grep "$suffix$")

	echo $list
	}

	#
	# Output inherited properties which is edited for file system
	#
	function fs_inherit_prop
	{
	typeset fs_prop
	if is_global_zone ; then
	fs_prop=$(zfs inherit 2>&1 \| \
	awk '$2=="YES" && $3=="YES" {print $1}')
	if ! is_te_enabled ; then
	fs_prop=$(echo $fs_prop \| grep -v "mlslabel")
	fi
	else
	fs_prop=$(zfs inherit 2>&1 \| \
	awk '$2=="YES" && $3=="YES" {print $1}'\|
	egrep -v "devices\|mlslabel\|sharenfs\|sharesmb\|zoned")
	fi

	echo $fs_prop
	}

	#
	# Output inherited properties for volume
	#
	function vol_inherit_prop
	{
	echo "checksum readonly"
	}

	#
	# Get the destination dataset to compare
	#
	function get_dst_ds
	{
	typeset srcfs=$1
	typeset dstfs=$2

	#
	# If the srcfs is not pool
	#
	if ! zpool list $srcfs > /dev/null 2>&1 ; then
	eval dstfs="$dstfs/${srcfs#*/}"
	fi

	echo $dstfs
	}

	#
	# Make test files
	#
	# $1 Number of files to create
	# $2 Maximum file size
	# $3 File ID offset
	# $4 File system to create the files on
	#
	function mk_files
	{
	nfiles=$1
	maxsize=$2
	file_id_offset=$3
	fs=$4
	bs=512

	for ((i=0; i<$nfiles; i=i+1)); do
	file_name="/$fs/file-$maxsize-$((i+$file_id_offset))"
	file_size=$((($RANDOM * $RANDOM % ($maxsize - 1)) + 1))

	#
	# Create an interesting mix of files which contain both
	# data blocks and holes for more realistic test coverage.
	# Half the files are created as sparse then partially filled,
	# the other half is dense then a hole is punched in the file.
	#
	if [ $((RANDOM % 2)) -eq 0 ]; then
	truncate -s $file_size $file_name \|\| \
	log_fail "Failed to create $file_name"
	dd if=/dev/urandom of=$file_name \
	bs=$bs count=$(($file_size / 2 / $bs)) \
	seek=$(($RANDOM % (($file_size / 2 / $bs) + 1))) \
	conv=notrunc >/dev/null 2>&1 \|\| \
	log_fail "Failed to create $file_name"
	else
	dd if=/dev/urandom of=$file_name \
	bs=$file_size count=1 >/dev/null 2>&1 \|\| \
	log_fail "Failed to create $file_name"
	dd if=/dev/zero of=$file_name \
	bs=$bs count=$(($file_size / 2 / $bs)) \
	seek=$(($RANDOM % (($file_size / 2 / $bs) + 1))) \
	conv=notrunc >/dev/null 2>&1 \|\| \
	log_fail "Failed to create $file_name"
	fi
	done
	echo Created $nfiles files of random sizes up to $maxsize bytes
	}

	#
	# Remove test files
	#
	# $1 Number of files to remove
	# $2 Maximum file size
	# $3 File ID offset
	# $4 File system to remove the files from
	#
	function rm_files
	{
	nfiles=$1
	maxsize=$2
	file_id_offset=$3
	fs=$4

	for ((i=0; i<$nfiles; i=i+1)); do
	rm -f /$fs/file-$maxsize-$((i+$file_id_offset))
	done
	echo Removed $nfiles files of random sizes up to $maxsize bytes
	}

	#
	# Simulate a random set of operations which could be reasonably expected
	# to occur on an average filesystem.
	#
	# $1 Number of files to modify
	# $2 Maximum file size
	# $3 File system to modify the file on
	# $4 Enabled xattrs (optional)
	#
	function churn_files
	{
	nfiles=$1
	maxsize=$2
	fs=$3
	xattrs=${4:-1}

	#
	# Remove roughly half of the files in order to make it more
	# likely that a dnode will be reallocated.
	#
	for ((i=0; i<$nfiles; i=i+1)); do
	file_name="/$fs/file-$i"

	if [[ -e $file_name ]]; then
	if [ $((RANDOM % 2)) -eq 0 ]; then
	rm $file_name \|\| \
	log_fail "Failed to remove $file_name"
	fi
	fi
	done

	#
	# Remount the filesystem to simulate normal usage. This resets
	# the last allocated object id allowing for new objects to be
	# reallocated in the locations of previously freed objects.
	#
	log_must zfs unmount $fs
	log_must zfs mount $fs

	for i in {0..$nfiles}; do
	file_name="/$fs/file-$i"
	file_size=$((($RANDOM * $RANDOM % ($maxsize - 1)) + 1))

	#
	# When the file exists modify it in one of five ways to
	# simulate normal usage:
	# - (20%) Remove and set and extended attribute on the file
	# - (20%) Overwrite the existing file
	# - (20%) Truncate the existing file to a random length
	# - (20%) Truncate the existing file to zero length
	# - (20%) Remove the file
	#
	# Otherwise create the missing file. 20% of the created
	# files will be small and use embedded block pointers, the
	# remainder with have random sizes up to the maximum size.
	# Three extended attributes are attached to all of the files.
	#
	if [[ -e $file_name ]]; then
	value=$((RANDOM % 5))
	if [ $value -eq 0 -a $xattrs -ne 0 ]; then
	attrname="testattr$((RANDOM % 3))"
	attrlen="$(((RANDOM % 1000) + 1))"
	attrvalue="$(random_string VALID_NAME_CHAR \
	$attrlen)"
	rm_xattr $attrname $file_name \|\| \
	log_fail "Failed to remove $attrname"
	set_xattr $attrname "$attrvalue" $file_name \|\| \
	log_fail "Failed to set $attrname"
	elif [ $value -eq 1 ]; then
	dd if=/dev/urandom of=$file_name \
	bs=$file_size count=1 >/dev/null 2>&1 \|\| \
	log_fail "Failed to overwrite $file_name"
	elif [ $value -eq 2 ]; then
	truncate -s $file_size $file_name \|\| \
	log_fail "Failed to truncate $file_name"
	elif [ $value -eq 3 ]; then
	truncate -s 0 $file_name \|\| \
	log_fail "Failed to truncate $file_name"
	else
	rm $file_name \|\| \
	log_fail "Failed to remove $file_name"
	fi
	else
	if [ $((RANDOM % 5)) -eq 0 ]; then
	file_size=$((($RANDOM % 64) + 1))
	fi

	dd if=/dev/urandom of=$file_name \
	bs=$file_size count=1 >/dev/null 2>&1 \|\| \
	log_fail "Failed to create $file_name"

	if [ $xattrs -ne 0 ]; then
	for j in {0..2}; do
	attrname="testattr$j"
	attrlen="$(((RANDOM % 1000) + 1))"
	attrvalue="$(random_string \
	VALID_NAME_CHAR $attrlen)"
	set_xattr $attrname \
	"$attrvalue" $file_name \|\| \
	log_fail "Failed to set $attrname"
	done
	fi
	fi
	done

	return 0
	}

	#
	# Mess up a send file's contents
	#
	# $1 The send file path
	#
	function mess_send_file
	{
	file=$1

	filesize=$(stat_size $file)

	offset=$(($RANDOM * $RANDOM % $filesize))

	# The random offset might truncate the send stream to be
	# smaller than the DRR_BEGIN record. If this happens, then
	# the receiving system won't have enough info to create the
	# partial dataset at all. We use zstreamdump to check for
	# this and retry in this case.
	nr_begins=$(head -c $offset $file \| zstreamdump \| \
	grep DRR_BEGIN \| awk '{ print $5 }')
	while [ "$nr_begins" -eq 0 ]; do
	offset=$(($RANDOM * $RANDOM % $filesize))
	nr_begins=$(head -c $offset $file \| zstreamdump \| \
	grep DRR_BEGIN \| awk '{ print $5 }')
	done

	if (($RANDOM % 7 <= 1)); then
	#
	# We corrupt 2 bytes to minimize the chance that we
	# write the same value that's already there.
	#
	log_must eval "dd if=/dev/urandom of=$file conv=notrunc " \
	"bs=1 count=2 seek=$offset >/dev/null 2>&1"
	else
	log_must truncate -s $offset $file
	fi
	}

	#
	# Diff the send/receive filesystems
	#
	# $1 The sent filesystem
	# $2 The received filesystem
	#
	function file_check
	{
	sendfs=$1
	recvfs=$2

	if [[ -d /$recvfs/.zfs/snapshot/a && -d \
	/$sendfs/.zfs/snapshot/a ]]; then
	diff -r /$recvfs/.zfs/snapshot/a /$sendfs/.zfs/snapshot/a
	[[ $? -eq 0 ]] \|\| log_fail "Differences found in snap a"
	fi
	if [[ -d /$recvfs/.zfs/snapshot/b && -d \
	/$sendfs/.zfs/snapshot/b ]]; then
	diff -r /$recvfs/.zfs/snapshot/b /$sendfs/.zfs/snapshot/b
	[[ $? -eq 0 ]] \|\| log_fail "Differences found in snap b"
	fi
	}

	#
	# Resume test helper
	#
	# $1 The ZFS send command
	# $2 The filesystem where the streams are sent
	# $3 The receive filesystem
	+# $4 Test dry-run (optional)
	#
	function resume_test
	{
	- sendcmd=$1
	- streamfs=$2
	- recvfs=$3
	+ typeset sendcmd=$1
	+ typeset streamfs=$2
	+ typeset recvfs=$3
	+ typeset dryrun=${4:-1}

	stream_num=1
	log_must eval "$sendcmd >/$streamfs/$stream_num"

	for ((i=0; i<2; i=i+1)); do
	mess_send_file /$streamfs/$stream_num
	log_mustnot zfs recv -suv $recvfs </$streamfs/$stream_num
	stream_num=$((stream_num+1))

	token=$(zfs get -Hp -o value receive_resume_token $recvfs)
	+
	+ # Do a dry-run
	+ [ $dryrun -ne 0 ] && \
	+ log_must eval "zfs send -nvt $token > /dev/null"
	+
	log_must eval "zfs send -t $token >/$streamfs/$stream_num"
	[[ -f /$streamfs/$stream_num ]] \|\| \
	log_fail "NO FILE /$streamfs/$stream_num"
	done
	log_must zfs recv -suv $recvfs </$streamfs/$stream_num
	}

	function get_resume_token
	{
	sendcmd=$1
	streamfs=$2
	recvfs=$3

	log_must eval "$sendcmd > /$streamfs/1"
	mess_send_file /$streamfs/1
	log_mustnot zfs recv -suv $recvfs < /$streamfs/1 2>&1
	token=$(zfs get -Hp -o value receive_resume_token $recvfs)
	echo "$token" > /$streamfs/resume_token

	return 0
	}

	#
	# Setup filesystems for the resumable send/receive tests
	#
	# $1 The "send" filesystem
	# $2 The "recv" filesystem
	#
	function test_fs_setup
	{
	typeset sendfs=$1
	typeset recvfs=$2
	typeset streamfs=$3
	typeset sendpool=${sendfs%%/*}
	typeset recvpool=${recvfs%%/*}

	datasetexists $sendfs && log_must_busy zfs destroy -r $sendpool
	datasetexists $recvfs && log_must_busy zfs destroy -r $recvpool
	datasetexists $streamfs && log_must_busy zfs destroy -r $streamfs

	if datasetexists $sendfs \|\| zfs create -o compress=lz4 $sendfs; then
	mk_files 1000 256 0 $sendfs &
	mk_files 1000 131072 0 $sendfs &
	mk_files 100 1048576 0 $sendfs &
	mk_files 10 10485760 0 $sendfs &
	mk_files 1 104857600 0 $sendfs &
	log_must wait
	log_must zfs snapshot $sendfs@a

	rm_files 200 256 0 $sendfs &
	rm_files 200 131072 0 $sendfs &
	rm_files 20 1048576 0 $sendfs &
	rm_files 2 10485760 0 $sendfs &
	log_must wait

	mk_files 400 256 0 $sendfs &
	mk_files 400 131072 0 $sendfs &
	mk_files 40 1048576 0 $sendfs &
	mk_files 4 10485760 0 $sendfs &
	log_must wait

	log_must zfs snapshot $sendfs@b
	log_must eval "zfs send -v $sendfs@a >/$sendpool/initial.zsend"
	log_must eval "zfs send -v -i @a $sendfs@b " \
	">/$sendpool/incremental.zsend"
	fi

	log_must zfs create -o compress=lz4 $streamfs
	}

	#
	# Check to see if the specified features are set in a send stream.
	# The values for these features are found in include/sys/zfs_ioctl.h
	#
	# $1 The stream file
	# $2-$n The flags expected in the stream
	#
	function stream_has_features
	{
	typeset file=$1
	shift

	[[ -f $file ]] \|\| log_fail "Couldn't find file: $file"
	typeset flags=$(cat $file \| zstreamdump \| \
	awk '/features =/ {features = $3} END {print features}')
	typeset -A feature
	feature[dedup]="1"
	feature[dedupprops]="2"
	feature[sa_spill]="4"
	feature[embed_data]="10000"
	feature[lz4]="20000"
	feature[mooch_byteswap]="40000"
	feature[large_blocks]="80000"
	feature[resuming]="100000"
	feature[redacted]="200000"
	feature[compressed]="400000"

	typeset flag known derived=0
	for flag in "$@"; do
	known=${feature[$flag]}
	[[ -z $known ]] && log_fail "Unknown feature: $flag"

	derived=$(printf "%x" $((0x${flags} & 0x${feature[$flag]})))
	[[ $derived = $known ]] \|\| return 1
	done

	return 0
	}

	#
	# Given a send stream, verify that the size of the stream matches what's
	# expected based on the source or target dataset. If the stream is an
	# incremental stream, subtract the size of the source snapshot before
	# comparing. This function does not currently handle incremental streams
	# that remove data.
	#
	# $1 The zstreamdump output file
	# $2 The dataset to compare against
	# This can be a source of a send or recv target (fs, not snapshot)
	# $3 The percentage below which verification is deemed a failure
	# $4 The source snapshot of an incremental send
	#

	function verify_stream_size
	{
	typeset stream=$1
	typeset ds=$2
	typeset percent=${3:-90}
	typeset inc_src=$4

	[[ -f $stream ]] \|\| log_fail "No such file: $stream"
	datasetexists $ds \|\| log_fail "No such dataset: $ds"

	typeset stream_size=$(cat $stream \| zstreamdump \| sed -n \
	's/ Total payload size = $.$ (0x.)/\1/p')

	typeset inc_size=0
	if [[ -n $inc_src ]]; then
	inc_size=$(get_prop lrefer $inc_src)
	if stream_has_features $stream compressed; then
	inc_size=$(get_prop refer $inc_src)
	fi
	fi

	if stream_has_features $stream compressed; then
	ds_size=$(get_prop refer $ds)
	else
	ds_size=$(get_prop lrefer $ds)
	fi
	ds_size=$((ds_size - inc_size))

	within_percent $stream_size $ds_size $percent \|\| log_fail \
	"$stream_size $ds_size differed by too much"
	}

	# Cleanup function for tests involving resumable send
	function resume_cleanup
	{
	typeset sendfs=$1
	typeset streamfs=$2
	typeset sendpool=${sendfs%%/*}

	datasetexists $sendfs && log_must_busy zfs destroy -r $sendfs
	datasetexists $streamfs && log_must_busy zfs destroy -r $streamfs
	cleanup_pool $POOL2
	rm -f /$sendpool/initial.zsend /$sendpool/incremental.zsend
	}

	# Randomly set the property to one of the enumerated values.
	function rand_set_prop
	{
	typeset dtst=$1
	typeset prop=$2
	shift 2
	typeset value=$(random_get $@)

	log_must eval "zfs set $prop='$value' $dtst"
	}

	# Generate a recursive checksum of a filesystem which includes the file
	# contents and any associated extended attributes.
	function recursive_cksum
	{
	case "$(uname)" in
	FreeBSD)
	find $1 -type f -exec sh -c 'sha256 -q {}; lsextattr -q \
	system {} \| sha256 -q; lsextattr -q user {} \| sha256 -q' \
	\; \| sort \| sha256 -q
	;;
	*)
	find $1 -type f -exec sh -c 'sha256sum {}; getfattr \
	--absolute-names --only-values -d {} \| sha256sum' \; \| \
	sort -k 2 \| awk '{ print $1 }' \| sha256sum \| \
	awk '{ print $1 }'
	;;
	esac
	}
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/rsend/rsend_008_pos.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/rsend/rsend_008_pos.ksh (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/rsend/rsend_008_pos.ksh (revision 366774)
	@@ -1,131 +1,131 @@
	#!/bin/ksh -p
	#
	# CDDL HEADER START
	#
	# The contents of this file are subject to the terms of the
	# Common Development and Distribution License (the "License").
	# You may not use this file except in compliance with the License.
	#
	# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	# or http://www.opensolaris.org/os/licensing.
	# See the License for the specific language governing permissions
	# and limitations under the License.
	#
	# When distributing Covered Code, include this CDDL HEADER in each
	# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	# If applicable, add the following below this CDDL HEADER, with the
	# fields enclosed by brackets "[]" replaced with your own identifying
	# information: Portions Copyright [yyyy] [name of copyright owner]
	#
	# CDDL HEADER END
	#

	#
	# Copyright 2009 Sun Microsystems, Inc. All rights reserved.
	# Use is subject to license terms.
	#

	#
	# Copyright (c) 2013, 2016 by Delphix. All rights reserved.
	#

	. $STF_SUITE/tests/functional/rsend/rsend.kshlib

	#
	# DESCRIPTION:
	# Changes made by 'zfs promote' can be properly received.
	#
	# STRATEGY:
	# 1. Separately promote pool clone, filesystem clone and volume clone.
	# 2. Recursively backup all the POOL and restore in POOL2
	# 3. Verify all the datasets and properties were properly received.
	#

	verify_runnable "both"

	-# See issue: https://github.com/zfsonlinux/zfs/issues/6066
	+# See issue: https://github.com/openzfs/zfs/issues/6066
	log_unsupported "Occasionally hangs"

	# Origin Clone
	#
	set -A dtst "$POOL" "$POOL/pclone" \
	"$POOL/$FS/fs1/fs2" "$POOL/$FS/fs1/fclone"
	if is_global_zone ; then
	typeset -i n=${#dtst[@]}
	dtst[((n))]="$POOL/$FS/vol"; dtst[((n+1))]="$POOL/$FS/vclone"
	fi

	function cleanup
	{
	typeset origin
	typeset -i i=0
	while ((i < ${#dtst[@]})); do
	origin=$(get_prop origin ${dtst[$i]})

	if [[ $origin != "-" ]]; then
	log_must zfs promote ${dtst[$i]}
	fi

	((i += 2))
	done

	origin=$(get_prop origin $POOL2)
	if [[ $origin != "-" ]]; then
	log_must zfs promote $POOL2
	fi
	log_must cleanup_pool $POOL2
	}

	log_assert "Changes made by 'zfs promote' can be properly received."
	log_onexit cleanup

	typeset -i i=0
	while ((i < ${#dtst[@]})); do
	log_must zfs promote ${dtst[((i+1))]}

	((i += 2))
	done

	#
	# Verify zfs send -R should succeed
	#
	log_must eval "zfs send -R $POOL@final > $BACKDIR/pool-final-R"
	log_must eval "zfs receive -d -F $POOL2 < $BACKDIR/pool-final-R"

	dstds=$(get_dst_ds $POOL $POOL2)
	#
	# Define all the POOL/POOL2 datasets pair
	#
	set -A pair "$POOL" "$dstds" \
	"$POOL/$FS" "$dstds/$FS" \
	"$POOL/$FS/fs1" "$dstds/$FS/fs1" \
	"$POOL/$FS/fs1/fs2" "$dstds/$FS/fs1/fs2" \
	"$POOL/pclone" "$dstds/pclone" \
	"$POOL/$FS/fs1/fclone" "$dstds/$FS/fs1/fclone"

	if is_global_zone ; then
	typeset -i n=${#pair[@]}
	pair[((n))]="$POOL/vol"; pair[((n+1))]="$dstds/vol"
	pair[((n+2))]="$POOL/$FS/vol" pair[((n+3))]="$dstds/$FS/vol"
	fi

	#
	# Verify all the sub-datasets can be properly received.
	#
	log_must cmp_ds_subs $POOL $dstds
	typeset -i i=0
	while ((i < ${#pair[@]})); do
	log_must cmp_ds_cont ${pair[$i]} ${pair[((i+1))]}
	log_must cmp_ds_prop ${pair[$i]} ${pair[((i+1))]}

	((i += 2))
	done

	# Verify the original filesystem can be promoted
	log_must zfs promote $dstds
	if is_global_zone ; then
	log_must zfs promote $dstds/$FS/vol
	fi
	log_must zfs promote $dstds/$FS/fs1/fs2

	log_pass "Changes made by 'zfs promote' can be properly received."
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/rsend/rsend_012_pos.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/rsend/rsend_012_pos.ksh (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/rsend/rsend_012_pos.ksh (revision 366774)
	@@ -1,186 +1,186 @@
	#!/bin/ksh -p
	#
	# CDDL HEADER START
	#
	# The contents of this file are subject to the terms of the
	# Common Development and Distribution License (the "License").
	# You may not use this file except in compliance with the License.
	#
	# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
	# or http://www.opensolaris.org/os/licensing.
	# See the License for the specific language governing permissions
	# and limitations under the License.
	#
	# When distributing Covered Code, include this CDDL HEADER in each
	# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
	# If applicable, add the following below this CDDL HEADER, with the
	# fields enclosed by brackets "[]" replaced with your own identifying
	# information: Portions Copyright [yyyy] [name of copyright owner]
	#
	# CDDL HEADER END
	#

	#
	# Copyright (c) 2009, Sun Microsystems Inc. All rights reserved.
	# Copyright (c) 2013, 2016, Delphix. All rights reserved.
	# Use is subject to license terms.
	#

	. $STF_SUITE/include/properties.shlib
	. $STF_SUITE/tests/functional/rsend/rsend.kshlib

	#
	# DESCRIPTION:
	# zfs send -R will backup all the filesystem properties correctly.
	#
	# STRATEGY:
	# 1. Setting properties for all the filesystem and volumes randomly
	# 2. Backup all the data from POOL by send -R
	# 3. Restore all the data in POOL2
	# 4. Verify all the properties in the two pools are the same
	#

	verify_runnable "global"

	function edited_prop
	{
	typeset behaviour=$1
	typeset ds=$2
	typeset backfile=$TESTDIR/edited_prop_$ds

	case $behaviour in
	"get")
	typeset props=$(zfs inherit 2>&1 \| \
	awk '$2=="YES" {print $1}' \| \
	egrep -v "^vol\|\.\.\.$")
	for item in $props ; do
	if [[ $item == "mlslabel" ]] && \
	! is_te_enabled ; then
	continue
	fi
	zfs get -H -o property,value $item $ds >> \
	$backfile
	if (($? != 0)); then
	log_fail "zfs get -H -o property,value"\
	"$item $ds > $backfile"
	fi
	done
	;;
	"set")
	if [[ ! -f $backfile ]] ; then
	log_fail "$ds need backup properties firstly."
	fi

	typeset prop value
	while read prop value ; do
	eval zfs set $prop='$value' $ds
	if (($? != 0)); then
	log_fail "zfs set $prop=$value $ds"
	fi
	done < $backfile
	;;
	*)
	log_fail "Unrecognized behaviour: $behaviour"
	esac
	}

	function cleanup
	{
	log_must cleanup_pool $POOL
	log_must cleanup_pool $POOL2

	log_must edited_prop "set" $POOL
	log_must edited_prop "set" $POOL2

	typeset prop
	for prop in $(fs_inherit_prop) ; do
	log_must zfs inherit $prop $POOL
	log_must zfs inherit $prop $POOL2
	done

	log_must setup_test_model $POOL

	if [[ -d $TESTDIR ]]; then
	log_must rm -rf $TESTDIR/*
	fi
	}

	log_assert "Verify zfs send -R will backup all the filesystem properties " \
	"correctly."
	log_onexit cleanup

	log_must edited_prop "get" $POOL
	log_must edited_prop "get" $POOL2

	for fs in "$POOL" "$POOL/pclone" "$POOL/$FS" "$POOL/$FS/fs1" \
	"$POOL/$FS/fs1/fs2" "$POOL/$FS/fs1/fclone" ; do
	rand_set_prop $fs aclinherit "discard" "noallow" "secure" "passthrough"
	rand_set_prop $fs checksum "on" "off" "fletcher2" "fletcher4" "sha256"
	- rand_set_prop $fs acltype "off" "posix" "noacl" "posixacl"
	+ rand_set_prop $fs acltype "off" "posix" "nfsv4" "noacl" "posixacl"
	rand_set_prop $fs atime "on" "off"
	rand_set_prop $fs checksum "on" "off" "fletcher2" "fletcher4" "sha256"
	rand_set_prop $fs compression "${compress_prop_vals[@]}"
	rand_set_prop $fs copies "1" "2" "3"
	rand_set_prop $fs devices "on" "off"
	rand_set_prop $fs exec "on" "off"
	rand_set_prop $fs quota "512M" "1024M"
	rand_set_prop $fs recordsize "512" "2K" "8K" "32K" "128K"
	rand_set_prop $fs dnodesize "legacy" "auto" "1k" "2k" "4k" "8k" "16k"
	rand_set_prop $fs setuid "on" "off"
	rand_set_prop $fs snapdir "hidden" "visible"
	if ! is_freebsd; then
	rand_set_prop $fs xattr "on" "off"
	fi
	rand_set_prop $fs user:prop "aaa" "bbb" "23421" "()-+?"
	done

	for vol in "$POOL/vol" "$POOL/$FS/vol" ; do
	rand_set_prop $vol checksum "on" "off" "fletcher2" "fletcher4" "sha256"
	rand_set_prop $vol compression "${compress_prop_vals[@]}"
	rand_set_prop $vol readonly "on" "off"
	rand_set_prop $vol copies "1" "2" "3"
	rand_set_prop $vol user:prop "aaa" "bbb" "23421" "()-+?"
	done


	# Verify inherited property can be received
	rand_set_prop $POOL redundant_metadata "all" "most"
	rand_set_prop $POOL sync "standard" "always" "disabled"

	#
	# Duplicate POOL2 for testing
	#
	log_must eval "zfs send -R $POOL@final > $BACKDIR/pool-final-R"
	log_must eval "zfs receive -d -F $POOL2 < $BACKDIR/pool-final-R"

	#
	# Define all the POOL/POOL2 datasets pair
	#
	set -A pair "$POOL" "$POOL2" \
	"$POOL/$FS" "$POOL2/$FS" \
	"$POOL/$FS/fs1" "$POOL2/$FS/fs1" \
	"$POOL/$FS/fs1/fs2" "$POOL2/$FS/fs1/fs2" \
	"$POOL/pclone" "$POOL2/pclone" \
	"$POOL/$FS/fs1/fclone" "$POOL2/$FS/fs1/fclone" \
	"$POOL/vol" "$POOL2/vol" \
	"$POOL/$FS/vol" "$POOL2/$FS/vol"

	typeset -i i=0
	while ((i < ${#pair[@]})); do
	log_must cmp_ds_prop ${pair[$i]} ${pair[((i+1))]}

	((i += 2))
	done


	zpool upgrade -v \| grep "Snapshot properties" > /dev/null 2>&1
	if (( $? == 0 )) ; then
	i=0
	while ((i < ${#pair[@]})); do
	log_must cmp_ds_prop ${pair[$i]}@final ${pair[((i+1))]}@final
	((i += 2))
	done
	fi

	log_pass "Verify zfs send -R will backup all the filesystem properties " \
	"correctly."
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/rsend/rsend_024_pos.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/rsend/rsend_024_pos.ksh (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/rsend/rsend_024_pos.ksh (revision 366774)
	@@ -1,53 +1,53 @@
	#!/bin/ksh

	#
	# This file and its contents are supplied under the terms of the
	# Common Development and Distribution License ("CDDL"), version 1.0.
	# You may only use this file in accordance with the terms of version
	# 1.0 of the CDDL.
	#
	# A full copy of the text of the CDDL should have accompanied this
	# source. A copy of the CDDL is also available via the Internet at
	# http://www.illumos.org/license/CDDL.
	#

	#
	# Copyright (c) 2014, 2016 by Delphix. All rights reserved.
	#

	. $STF_SUITE/include/libtest.shlib
	. $STF_SUITE/tests/functional/rsend/rsend.kshlib

	#
	# Description:
	# Verify resumability of a full ZFS send/receive with the source filesystem
	# unmounted.
	#
	# Strategy:
	# 1. Destroy the filesystem for the receive
	# 2. Unmount the source filesystem
	# 3. Start a full ZFS send, redirect output to a file
	# 4. Mess up the contents of the stream state file on disk
	# 5. Try ZFS receive, which should fail with a checksum mismatch error
	# 6. ZFS send to the stream state file again using the receive_resume_token
	# 7. ZFS receive and verify the receive completes successfully
	#

	verify_runnable "both"

	log_assert "Verify resumability of a full ZFS send/receive with the source " \
	"filesystem unmounted"

	sendfs=$POOL/sendfs
	recvfs=$POOL2/recvfs
	streamfs=$POOL/stream

	log_onexit resume_cleanup $sendfs $streamfs

	test_fs_setup $sendfs $recvfs $streamfs
	log_must zfs unmount -f $sendfs
	-resume_test "zfs send $sendfs" $streamfs $recvfs
	+resume_test "zfs send $sendfs" $streamfs $recvfs 0
	file_check $sendfs $recvfs

	log_pass "Verify resumability of a full ZFS send/receive with the source " \
	"filesystem unmounted"
	Index: vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/rsend/send_freeobjects.ksh
	===================================================================
	--- vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/rsend/send_freeobjects.ksh (revision 366773)
	+++ vendor-sys/openzfs/dist/tests/zfs-tests/tests/functional/rsend/send_freeobjects.ksh (revision 366774)
	@@ -1,81 +1,81 @@
	#!/bin/ksh

	#
	# This file and its contents are supplied under the terms of the
	# Common Development and Distribution License ("CDDL"), version 1.0.
	# You may only use this file in accordance with the terms of version
	# 1.0 of the CDDL.
	#
	# A full copy of the text of the CDDL should have accompanied this
	# source. A copy of the CDDL is also available via the Internet at
	# http://www.illumos.org/license/CDDL.
	#

	#
	# Copyright (c) 2017 by Lawrence Livermore National Security, LLC.
	#

	. $STF_SUITE/include/libtest.shlib
	. $STF_SUITE/tests/functional/rsend/rsend.kshlib

	#
	# Description:
	# Verify FREEOBJECTS record frees sequential objects (See
	-# https://github.com/zfsonlinux/zfs/issues/6694)
	+# https://github.com/openzfs/zfs/issues/6694)
	#
	# Strategy:
	# 1. Create three files with sequential object numbers, f1 f2 and f3
	# 2. Delete f2
	# 3. Take snapshot A
	# 4. Delete f3
	# 5. Take snapshot B
	# 6. Receive a full send of A
	# 7. Receive an incremental send of B
	# 8. Fail test if f3 exists on received snapshot B
	#

	verify_runnable "both"

	log_assert "Verify FREEOBJECTS record frees sequential objects"

	sendds=sendfo
	recvds=recvfo
	f1=/$POOL/$sendds/f1
	f2=/$POOL/$sendds/f2
	f3=/$POOL/$sendds/f3

	#
	# We need to set xattr=sa and dnodesize=legacy to guarantee sequential
	# object numbers for this test. Otherwise, if we used directory-based
	# xattrs, SELinux extended attributes might consume intervening object
	# numbers.
	#
	log_must zfs create -o xattr=sa -o dnodesize=legacy $POOL/$sendds

	tries=100
	for ((i=0; i<$tries; i++)); do
	touch $f1 $f2 $f3
	o1=$(ls -li $f1 \| awk '{print $1}')
	o2=$(ls -li $f2 \| awk '{print $1}')
	o3=$(ls -li $f3 \| awk '{print $1}')

	if [[ $o2 -ne $(( $o1 + 1 )) ]] \|\| [[ $o3 -ne $(( $o2 + 1 )) ]]; then
	rm -f $f1 $f2 $f3
	else
	break
	fi
	done

	if [[ $i -eq $tries ]]; then
	log_fail "Failed to create three sequential objects"
	fi

	log_must rm $f2
	log_must zfs snap $POOL/$sendds@A
	log_must rm $f3
	log_must zfs snap $POOL/$sendds@B
	log_must eval "zfs send $POOL/$sendds@A \| zfs recv $POOL/$recvds"
	log_must eval "zfs send -i $POOL/$sendds@A $POOL/$sendds@B \|" \
	"zfs recv $POOL/$recvds"
	log_mustnot zdb $POOL/$recvds@B $o3
	log_pass "Verify FREEOBJECTS record frees sequential objects"

File Metadata

Mime Type: application/octet-stream
Expires: Sat, Oct 18, 7:49 AM (1 d, 23 h)
Storage Engine: chunks
Storage Format: Chunks
Storage Handle: 6jQxRRNnBrlo
Default Alt Text: (5 MB)

No OneTemporaryActions

View Options

File Metadata

Event Timeline

No OneTemporary
Actions