Saving a checkpoint reads all guest memory, which faults in every page
that the guest never touched or that was discarded, for example by
virtio-balloon. Restoring writes every page of the image. Either way,
the VM ends up backed by host memory for its full size until the guest
frees pages again; a balloon cannot undo this because it does not track
ballooned pages. On a 2 GiB guest with 1.2 GiB ballooned, a live
checkpoint grew host residency to the full 2048 MiB, and a restored VM
started at 2048 MiB.
Use VM_GET_POPULATED_PAGES for both. When saving, write zeroes for
unpopulated pages without reading them. Guest memory is not modified,
so this is safe while devices without a pause method, such as e1000,
write to guest memory during the checkpoint. When restoring, read the
image through a bounce buffer and skip zero pages that are not
populated, since they already read as zeroes. Copy the others,
including pages that bhyve wrote before the restore, such as firmware
tables. If the kernel cannot answer, access every page as before.
In the same test, residency stayed at 748 MiB throughout a live
checkpoint, which took 2.4 s instead of 2.9 s, and a restored VM started
at 639 MiB.
The image format is unchanged, so checkpoints remain compatible in both
directions. Guest memory is now always accessed in chunks, and a
truncated image fails the restore instead of looping.
Sponsored by: The FreeBSD Foundation