Fix some I/O ordering issues in gmirror.
- BIO_FLUSH requests were dispatched to the disks directly from g_mirror_start() rather than going through the mirror's I/O request queue, so they could have been reordered with preceding writes. Address this by processing such requests from the queue, avoiding direct dispatch.
- Handling for collisions with synchronization requests was too fine-grained and could cause reordering of writes. In particular, BIO_ORDERED was not being honoured. Address this by effectively freezing the request queue any time a collision with a synchronization request occurs. The queue is unfrozen once the collision with the first frozen request is over.
- The above-mentioned collision handling allowed reads to jump ahead of writes to the same offset. Address this by freezing all request types when a collision occurs, not just BIO_WRITEs and BIO_DELETEs.
Also add some more fail points for use in testing error handling.
Reviewed by: imp
MFC after: 3 weeks
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D13559