Add delete throttle functionality from upstream openzfs

[jpaetzel]

Summary

ZFS I/O is done in txg groups. If you are deleting a large amount of data txg groups will be immediately filled with deletes, starving out other write I/O. This is especially easy to see if you run a local delete on an NFS fileserver. NFS writes will stall until the delete is complete.

[jpaetzel]

The default value for this sysctl (30) is taken from upstream, but prior to this change the default is essentially 0 (this feature is disabled)

While this does change the behavior of ZFS it's mostly a positive thing. In my testing setting the sysctl to 60 essentially preserves the default behavior of ZFS if you are doing local I/O only. The upstream value of 30 slows down local deletes by ~ 15% if the delete I/O is the only load on the system.

Also, Nexenta has been running with this change for a year, but they don't deal so much with local filesystem I/O.

[avg]

Josh, it seems that this change is mostly a merge of an upstream, illumos, change.
So, please make sure to import the upstream change (illumos commit ff5177ee8bf9a355131ce2cc61ae2da6a5a6fdd6) into the svn vendor area vendor-sys/illumos and merge the change from there.
Also, please follow the loosely defined convention on how commit messages for illumos imports should look like. There are examples in commit logs.

[jpaetzel]

I've done the vendor import but keep getting tree conflicts when trying to merge it to HEAD

[avg]

In D9008#191275, @jpaetzel wrote:

I've done the vendor import but keep getting tree conflicts when trying to merge it to HEAD

There are two possibilities. Either the conflicts are caused by a FreeBSD change and in that case you have to resolve them. Or the conflicts are because of earlier upstream changes that we have not imported yet, in that case it's best to import them.

[avg]

Hmm, tree conflicts...
This is a command that I use to merge from the illumos system vendor area:

svn merge -c XXXXXX ^/vendor-sys/illumos/dist sys/cddl/contrib/opensolaris/