Page MenuHomeFreeBSD

Live Migration feature for bhyve
Needs ReviewPublic

Authored by on Jun 30 2021, 10:31 AM.
Referenced Files
F81263281: D30954.diff
Sat, Apr 13, 1:03 PM
Unknown Object (File)
Fri, Mar 22, 8:13 AM
Unknown Object (File)
Thu, Mar 21, 11:51 PM
Unknown Object (File)
Mar 6 2024, 10:54 PM
Unknown Object (File)
Mar 6 2024, 10:18 PM
Unknown Object (File)
Mar 6 2024, 5:12 PM
Unknown Object (File)
Dec 20 2023, 6:08 AM
Unknown Object (File)
Dec 11 2023, 12:25 AM


Group Reviewers

This patch presents an implementation of the live migration feature for bhyve. The feature is a wrapper over the warm migration feature ( and sends the VM's state (memory, kernel structures, emulated devices) from a source host to a destination host over the network.

This feature sends the guest's memory while the virtual machine is still running. To do this, it uses migration rounds. In the first round, we send all the guest's memory. In the next rounds, we only send the guest's memory pages that were modified since the last round (because the guest is still running, its memory can be modified anytime and we need to update the state). Currently, we use 4 migration rounds (this may be modified in future). When the defined number of rounds are completed, the guest is stopped and the remaining memory pages alongside to the guest's kernel structures' state and emulated devices' state are copied to the destination. Then, the guest is resumed at destination.

For detecting the pages that were modified between rounds, we use a flag (VPO_VMM_DIRTY) for each vm_page. This flag is set each time vm_page->dirty field is updated and reset only when the page is migrated.

The features works only for IPv4 guests that use wired memory (-S option for bhyve). Moreover, the migration procedure works only between identical workstations.

More information regarding the implementation can be found here:

The usage of the feature is detailed here: Virtual Machine Migration using bhyve.

The patch was applied over

Test Plan

Diff Detail

rG FreeBSD src repository
Lint Skipped
Tests Skipped

Event Timeline

This patch doesn't look incremental on top of D28270 but includes (almost) all of it again. The only obvious difference I can see is that one includes BHYVE_SNAPSHOT in amd64's GENERIC, but this doesn't. I don't know if there are any security concerns with the current kernel code that warrant it being off by default, or if it's just off by default in the kernel because it's off by default in userspace.


Put in the MK_BHYVE_SNAPSHOT-guarded block below?


Should these be made mutually exclusive?


The pause and resume are shared by both restore and migration. Can that be shared, so you have:

generic preparation

if (restore) {
  restore code

if (migrate) {
  migrate code

generic resume

Might be better to have a variable that means "any kind of restore"? Especially if you follow my suggestion above, you'll need two more uses of this construct otherwise.


This doesn't give user-friendly error messages yet is being used in places where the errors are caused by the user's input. You're also inconsistent, sometimes using perror instead.


Indentation for members is inconsistent


This seems unnecessary. You're using size_t in these so it's architecture-dependent, at which point the ABI is already forced to match (and that's implied anyway by the fact you're migrating VMs unless you're doing strange things...).




Why not use MAXHOSTNAMELEN? Which is 256 not 255...


MB is defined inside C files, don't use it here