bhyve: add snapshot support to the NVMe controller

[wanpengqian_gmail.com]

Summary

A guest with an NVMe disk could not be checkpointed or suspended:

Failed to snapshot nvme@pci.0.4.0; ret=45

Save and restore the controller's registers, its identify data and log
pages, the features, the queues and the asynchronous events: the
requests that wait for an event and the events that wait for a request.

Pausing the device waits for the I/O requests of the controller, not
only for those of the block device: a Dataset Management command with
several ranges issues the block request for its next range when the
previous one completes. So no I/O is in flight in a snapshot.

A namespace backed by RAM (ram=) is refused, its data is not saved.

Signed-off-by: Wanpeng Qian <[e-mail]>
Sponsored by: keelos.dev

Test Plan

main (f958aa7e7) with WITH_BHYVE_SNAPSHOT, FreeBSD 16.0-CURRENT guest with two vCPUs.

Before: bhyvectl --checkpoint of a guest with -s 4,nvme,... fails with "Failed to snapshot nvme@pci.0.4.0; ret=45".

After:

1. NVMe data disk on a slow device (gnop(8), 20 ms on every read and write, so that requests are in flight when the device is paused). In the guest a program writes 1 MB of a pattern at random offsets of /dev/nda0, reads it back and compares, all the time. 10 checkpoints (bhyvectl --checkpoint), then bhyvectl --suspend and bhyve -r: the program keeps running in the restored guest (120 rounds before the first checkpoint, 490 after the tenth, 660 at the suspend, 1090 half a minute after the restore) and finds no mismatch. The SMART counters go on from their values (nvmecontrol logpage -p 2).

2. The guest's root disk on the NVMe controller (-s 2,nvme,guest.raw). In the guest a loop copies a 200 MB file of random data on the root file system and compares the SHA-256 of the copy. 5 checkpoints, a suspend and a restore: 13 rounds before the suspend, 45 a minute after the restore, no mismatch.

3. Asynchronous events: FreeBSD's nvme(4) keeps Asynchronous Event Requests outstanding, so they are in the snapshot. After the restore of 2, truncate -s 13g guest.raw on the host:

nvme0: async event occurred (type 0x2, info 0x00, page 0x04)
GEOM_PART: nda0 was automatically resized.

The restored request was completed with the namespace change notice and the guest sees the new size.

4. A guest started with the UEFI firmware (edk2-bhyve). The firmware moves the controller's 64-bit BAR 0 to 0x800000000, and a restore needs D60235 to register it there: without it the restored guest has lost the controller ("Emulating access to non-existent address to 0x800001014"), with it test 1 passes for this guest as well (160 rounds before the suspend, 670 after the restore). Guests started with bhyveload (tests 1 to 3) keep bhyve's addresses and do not need it.

Also built without WITH_BHYVE_SNAPSHOT.

An earlier version of this code (it polled for the pending I/O instead of using a condition variable) has been in use on FreeBSD 14.5 since the end of September 2026, where guests with NVMe disks are suspended and restored each time the host loads a new kernel.