nvme: Use ISR exclusion to make reset safer

[imp]

Summary

Before we send a hardware reset to the controller, mark all the admin
and I/O queues as being 'in the interrupt handler' so that we know all
interrupt handlers have excited. Once the hardware reset is complete,
unblock the interrupt hanlders so that we can bring the card back
up. This replaces a 100ms sleep that we use for this purpose today,
potentially making reset a little faster as well as safer.

Sponsored by: Netflix

Test Plan

Note: This assumes that this atomic is the right way to go.
Drew suggested maybe a full mutex, in which case this would change to that idiom.

[imp]

sys/dev/nvme/nvme_ctrlr.c

1234

the last bit could be a separate commit, or I could drop it.

[imp]

Add a timeout for waiting for the ISR to exit of 100ms. And fail the driver /
controller if it is stuck. I've not seen this, but I got to wondering what would
happen if...

[imp]

rebase

[imp]

Replaced by https://reviews.freebsd.org/D41452