Use BIO_VLISTS to aggregate ZFS I/O

[asomers]

Summary

This revision is a WIP. I need to add some defines for additional
architectures like arm64, and I haven't finished testing yet. But I'm starting
the revision so I can collect feedback.

Instead of copying data, use BIO_VLISTs to aggregate ZFS I/O.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio.h:
Mark zio's that represent aggregated I/O with the ZIO_FLAG_AGGREGATED
flag. A ZIO_FLAG_AGGREGATED zio has io_data pointing to an array of
bus_dma_segment_t of length io_vcount.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_file.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c:
Handle ZIO_FLAG_AGGREGATED zios.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:
Use the new aggregating zio type. Additionally, use FreeBSD's
zero_region and bitbucket_region to avoid allocations and copying for
optional/no-data regions.

Add an always mapped, write only, VM region for discarding data. This
"bitbucket_region" is intended to be used when aggregated read I/O to a device
spans a region that is not desired, but is included in the I/O to avoid the
extra iop that would otherwise be needed to exclude it from the request.

sys/amd64/include/vmparam.h:
sys/arm/include/vmparam.h:
sys/i386/include/vmparam.h:
sys/ia64/include/vmparam.h:
sys/mips/include/vmparam.h:
sys/powerpc/include/vmparam.h:
sys/sparc64/include/vmparam.h:
Make the bitbucket region the same size as the zero region on all
architectures.

sys/sys/systm.h:
Add the bitbucket_region to FreeBSD's KPI.

sys/vm/vm_extern.h:
sys/vm/vm_init.c:
sys/vm/vm_kern.c:
Initialize the bitbucket region during VM startup.

Test Plan

ZFS test suite.

[asomers]

@gibbs, can you suggest any other testing I should do for this revision? Did you do any benchmarking for it?

[gibbs]

When I did this work originally, I had hoped Spectra would find time to finish memory descriptors and embed those in BIOs instead. The BIO_VLIST thing was an expedient "hack".

[asomers]

In D7207#149844, @gibbs wrote:

When I did this work originally, I had hoped Spectra would find time to finish memory descriptors and embed those in BIOs instead. The BIO_VLIST thing was an expedient "hack".

So are you saying that you would not approve this change? Time issues aside, I don't even know enough about the context of the change to complete it with memory descriptors.

[asomers]

This revision requires substantial rework as a result of r320156 (MFV r318946: 8021 ARC buf data scatter-ization). I don't have the time to do it, so I'm going to abandon the review.