This patch presents an implementation of the live migration feature for bhyve. The feature is a wrapper over the warm migration feature (https://reviews.freebsd.org/D28270) and sends the VM's state (memory, kernel structures, emulated devices) from a source host to a destination host over the network.
This feature sends the guest's memory while the virtual machine is still running. To do this, it uses migration rounds. In the first round, we send all the guest's memory. In the next rounds, we only send the guest's memory pages that were modified since the last round (because the guest is still running, its memory can be modified anytime and we need to update the state). Currently, we use 4 migration rounds (this may be modified in future). When the defined number of rounds are completed, the guest is stopped and the remaining memory pages alongside to the guest's kernel structures' state and emulated devices' state are copied to the destination. Then, the guest is resumed at destination.
For detecting the pages that were modified between rounds, we use a flag (VPO_VMM_DIRTY) for each vm_page. This flag is set each time vm_page->dirty field is updated and reset only when the page is migrated.
The features works only for IPv4 guests that use wired memory (-S option for bhyve). Moreover, the migration procedure works only between identical workstations.
More information regarding the implementation can be found here:
The usage of the feature is detailed here: Virtual Machine Migration using bhyve.
The patch was applied over https://github.com/FreeBSD-UPB/freebsd-src/commit/d26ef5c7ac830812f07a02787f25fed5d6f8609e.