This revision aims to address a few BIO ordering bugs:
BIO_FLUSH requests are dispatched directly to the mirrors from
g_mirror_start() rather than being queued for processing by the gmirror
worker thread. This means that a BIO_FLUSH may "jump ahead" of a
preceding BIO_WRITE, which is undesirable. Address this by handling
BIO_FLUSH requests the same as BIO_{READ,WRITE,DELETE}. We take care to
avoid breaking mirrors if a BIO_FLUSH comes back with EOPNOTSUPP.
When a BIO_WRITE or BIO_DELETE request overlaps with an offset that's
actively being synchronized (i.e., we're currently reading at that
offset in order to copy its contents to a synchronizing mirror, or we're
currently writing to that offset as part of a block copy), the request
is placed in the regular_delayed queue. Once the colliding
synchronization BIO completes, BIOs from the delayed queue are put back
on the head of the work queue. However, this allows writes to be
reordered when synchronizing. Fix this by freezing all I/O requests
(including reads) when a collision occurs.
The regular_delayed queue may reorder reads and writes to the same
offset, since reads never went into the regular_delayed queue. Fix this
the same as above: once a BIO_WRITE or BIO_DELETE collides with a
synchronization BIO, all subsequent regular I/Os are placed in the
regular_delayed queue until that initial collision is over, at which
point all BIOs in regular_delayed are moved back to the main I/O queue.