When we suspend, we need to properly shutdown the NVME controller. The
controller may go into D3 state (or may have the power removed), and
to properly flush the metadata to non-volatile RAM, we must complete a
normal shutdown. This consists of deleting the I/O queues and setting
the shutodown bit. We have to do some extra stuff to make sure we
reset the software state of the queues as well.
On resume, we have to reset the card twice, for reasons described in
the attach funcion. Once we've done that, we can restart the card. If
any of this fails, we'll fail the NVMe card, just like we do when a
reset fails.
Pass if we're in resetting or not to nvme_ctrlr_start. We use that on
the resume path, even though we're not really resetting (we set
is_resetting out of paranoia, in case something else tries to initiate
a reset during resume, we block it). However, other than the taskqueue
interlock, we only use is_resetting inside of nvme_ctrlr_start, so
it's best to just pass it in since we know why we're calling it.
Export a number of use ctrlr functions. Also, rename a
nvme_ctrlr_destory_qpairs to nvme_ctrlr_delete_qpairs to avoid
confusion with all the other destroy functions.
nvme_ctrlr_delete_qpairs just removes the queues in hardware, while
the other _destroy_ functions tear down data structures.
Finally, split parts of the hardware reset function up so that I can
do part of the reset in suspsend. Split out the software disabling
of the qpairs into nvme_ctrlr_disable_qpairs.