vmm: [snapshot] MTRR should be saved/restored

[gusev.vitaliy_gmail.com]

Summary

Ubuntu 22.04 (Linux 5.15.0-84-generic) gets panic (hangs) if do:

    
1. Snapshot VM while it is in GRUB menu
2. Resume
3. Boot Linux.

Linux dmesg:

    
total RAM covered: 1M
gran_size: 64K  chunk_size: 64K         num_reg: 4      lose cover RAM: 92K
...
mtrr_cleanup: can not find optimal value
please specify mtrr_gran_size/mtrr_chunk_size
WARNING: BIOS bug: CPU MTRRs don't cover all of memory, losing 510MB of RAM.
WARNING: CPU: 0 PID: 0 at arch/x86/kernel/cpu/mtrr/cleanup.c:978
                            mtrr_trim_uncached_memory+0x426/0x469
...
BUG: unable to handle page fault for address: ffff9d0047e15ff8

Adding MTTR increases kernel dump size for 256 bytes per vCPU:

Sponsored by: vStack

Git commit:

https://github.com/gusev-vitaliy/freebsd/commit/d3cc0ec8b4b890e5a2b82773f6b92d97265719e8

Test Plan

Steps:

1. Boot VM with Ubuntu and stay in GRUB menu, 1 vcpu (suspend/resume more than 1 vcpu is still broken, it is separate issue).

~# bhyve -c 1 -m 1024M -H -A -s 0:0,hostbridge -s 31,lpc   -l bootrom,BHYVE_UEFI.fd     -s 4:0,virtio-net,tap0 -s 7,fbuf,tcp=0.0.0.0:5900,w=1024,h=768    -s 5,ahci-cd,/disk/iso/ubuntu-22.04.1-live-server-amd64.iso   ubuntu22

2. Suspend VM

~# bhyvectl --suspend=a10 --vm=ubuntu22

3. Resume VM

bhyve  -c 1 -m 1024M -H -A -s 0:0,hostbridge -s 31,lpc   -l bootrom,BHYVE_UEFI.fd     -s 4:0,virtio-net,tap0 -s 7,fbuf,tcp=0.0.0.0:5900,w=1024,h=768    -s 5,ahci-cd,/disk/iso/ubuntu-22.04.1-live-server-amd64.iso  -r a10 ubuntu22

4. Enter in GRUB menu to Boot Linux

---

Result for non patched: VM hangs, with 100%vCPU usage.

Result for patched: VM boots and all works, including NIC, etc.

---

Output of the 'cat /proc/mtrr' command in VM before suspend and after restore
should be the same, like this:

root@ub22:~# cat /proc/mtrr
reg00: base=0x080000000 ( 2048MB), size= 2048MB, count=1: uncachable
reg01: base=0x040000000 ( 1024MB), size= 1024MB, count=1: uncachable
reg02: base=0x020000000 (  512MB), size=  512MB, count=1: uncachable

Without patch, cat /proc/mtrr produces no output after resume.

Tested only on Intel CPU.