Live Migration feature for bhyve
Needs ReviewPublic
Actions

Authored by elenamihailescu22_gmail.com on Jun 30 2021, 10:31 AM.

Details

Reviewers

Group Reviewers

Summary

This patch presents an implementation of the live migration feature for bhyve. The feature is a wrapper over the warm migration feature (https://reviews.freebsd.org/D28270) and sends the VM's state (memory, kernel structures, emulated devices) from a source host to a destination host over the network.

This feature sends the guest's memory while the virtual machine is still running. To do this, it uses migration rounds. In the first round, we send all the guest's memory. In the next rounds, we only send the guest's memory pages that were modified since the last round (because the guest is still running, its memory can be modified anytime and we need to update the state). Currently, we use 4 migration rounds (this may be modified in future). When the defined number of rounds are completed, the guest is stopped and the remaining memory pages alongside to the guest's kernel structures' state and emulated devices' state are copied to the destination. Then, the guest is resumed at destination.

For detecting the pages that were modified between rounds, we use a flag (VPO_VMM_DIRTY) for each vm_page. This flag is set each time vm_page->dirty field is updated and reset only when the page is migrated.

The features works only for IPv4 guests that use wired memory (-S option for bhyve). Moreover, the migration procedure works only between identical workstations.

More information regarding the implementation can be found here:

The usage of the feature is detailed here: Virtual Machine Migration using bhyve.

The patch was applied over https://github.com/FreeBSD-UPB/freebsd-src/commit/d26ef5c7ac830812f07a02787f25fed5d6f8609e.

Test Plan

Diff Detail

Repository

rG FreeBSD src repository

Lint

Lint Skipped

Unit

Tests Skipped

Event Timeline

elenamihailescu22_gmail.com created this revision.Jun 30 2021, 10:31 AM

Herald added a reviewer: bhyve. · View Herald TranscriptJun 30 2021, 10:31 AM

Herald added subscribers: bcran, jonathan, rgrimes. · View Herald Transcript

elenamihailescu22_gmail.com requested review of this revision.Jun 30 2021, 10:31 AM

afedorov added a subscriber: afedorov.Jun 30 2021, 2:17 PM

kalle.carlbark_freebsd_kcbark.net added a subscriber: kalle.carlbark_freebsd_kcbark.net.Jul 3 2021, 2:12 PM

This patch doesn't look incremental on top of D28270 but includes (almost) all of it again. The only obvious difference I can see is that one includes BHYVE_SNAPSHOT in amd64's GENERIC, but this doesn't. I don't know if there are any security concerns with the current kernel code that warrant it being off by default, or if it's just off by default in the kernel because it's off by default in userspace.

usr.sbin/bhyve/Makefile
37	Put in the MK_BHYVE_SNAPSHOT-guarded block below?
usr.sbin/bhyve/bhyverun.c
1284–1289	Should these be made mutually exclusive?
1508	The pause and resume are shared by both restore and migration. Can that be shared, so you have: generic preparation if (restore) { restore code } if (migrate) { migrate code } generic resume
1579	Might be better to have a variable that means "any kind of restore"? Especially if you follow my suggestion above, you'll need two more uses of this construct otherwise.
usr.sbin/bhyve/migration.c
96	This doesn't give user-friendly error messages yet is being used in places where the errors are caused by the user's input. You're also inconsistent, sometimes using perror instead.
usr.sbin/bhyve/migration.h
57–82	Indentation for members is inconsistent
71	This seems unnecessary. You're using size_t in these so it's architecture-dependent, at which point the ABI is already forced to match (and that's implied anyway by the fact you're migrating VMs unless you're doing strange things...).
usr.sbin/bhyve/snapshot.c
35	Don't
usr.sbin/bhyve/snapshot.h
47	Why not use MAXHOSTNAMELEN? Which is 256 not 255...
52	MB is defined inside C files, don't use it here