bhyve: restore the BARs at the addresses the guest gave them

[wanpengqian_gmail.com]

Summary

A restore set the addresses in the BARs and the command register to
those of the snapshot, but left the regions registered where
pci_emul_alloc_bar() had put them when bhyve started. That is only
right for a guest that kept bhyve's addresses. The UEFI firmware moves
64-bit BARs above 4 GB, and after a restore every access to such a BAR
went nowhere:

Emulating access to non-existent address to 0x800001014

Before the devices are restored, unregister the regions of all of them;
when a device is restored, register its regions at the restored
addresses, if the restored command register enables them. All devices
go first because a restored BAR may be where another device's BAR is
before the restore.

Signed-off-by: Wanpeng Qian <[e-mail]>
Sponsored by: keelos.dev

Test Plan

main (f958aa7e7) with WITH_BHYVE_SNAPSHOT.

The NVMe controller has the 64-bit BAR that shows it, so this was tested together with D60234 (snapshot support for the NVMe controller). FreeBSD 16.0-CURRENT guest started with the UEFI firmware (edk2-bhyve), -s 4,nvme,...; in the guest:

# pciconf -lb nvme0
    bar   [10] = type Memory, range 64, base 0x800000000, size 16384, enabled

A program in the guest writes 1 MB of a pattern at random offsets of /dev/nda0, reads it back and compares, all the time. bhyvectl --suspend, then bhyve -r.

Before (D60234 alone): the restored guest has lost its NVMe controller. bhyve prints

Emulating access to non-existent address to 0x800001014
Emulating access to non-existent address to 0x800001010
Emulating access to non-existent address to 0x80000001c

and the guest

nvme0: Waiting for reset to complete

until it gives up.

After: the program keeps running in the restored guest (160 rounds before the suspend, 510 and 670 after the restore), no mismatch; the network (e1000) and the AHCI root disk work as before.

Guests started with bhyveload, whose BARs stay where bhyve put them, are restored as before (same test, AHCI root disk and NVMe data disk, and with the root disk on the NVMe controller).

On FreeBSD 14.5 this change has also been needed for a guest that had exchanged the addresses of two devices' BARs, which is why all devices are unregistered before the first one is restored.

[bnovkov]

LGTM (one minor suggestion inline), I'll wait for @jhb 's comments before accepting.

usr.sbin/bhyve/pci_emul.c

2501

[wanpengqian_gmail.com]

Rename pci_bars_registration() to pci_restore_register_bars() as suggested (bnovkov).

[wanpengqian_gmail.com]

Thanks, renamed to pci_restore_register_bars(). Builds with and without BHYVE_SNAPSHOT.