bhyve: fix NVMe emulation update of SQHD

[chuck]

Summary

The SQHD field of a Completion Queue entry indicates the current
Submission Queue head pointer value. The head pointer represents the
next entry to be consumed and is updated after consuming the current
entry.

In the Admin queue processing, the current code updates the head pointer
after reporting the value to the host via the SQHD. This gives the
impression that the Controller is perpetually one command behind in its
processing of the Admin SQ. And while this doesn't appear to bother some
initiators, it is wrong.

Fix is to update the SQ head pointer prior to writing the SQHD value in
the completion.

Reported by khng300

[imp]

This seems legit, but I've not done a deep dive...

[jhb]

usr.sbin/bhyve/pci_nvme.c

1261 ↗

(On Diff #69559)

Writing cdw0 seems unrelated to this change? Also, the acquire memory barrier seems odd. Normally for writing to a descriptor entry what you want is a release barrier on the write to set the word in the descriptor entry that the host reads to determine if an entry is valid.

[chuck]

usr.sbin/bhyve/pci_nvme.c

1261 ↗

(On Diff #69559)

1. Yes, the cdw0 change is unrelated but it's a bug I noticed while fixing the OP's issue
2. Agreed that the acquire is odd, but it's original code. I have some WIP that may remove this

[jhb]

I guess at least add a note in the commit log explaining why you are setting cdw0 (e.g. While here, set the cdw0 field for foo case which was missing).

The barrier in its new location (halfway through the update) makes no real sense. It probably just wants to be atomic_load_short() which I think didn't exist when this was first written.