Page MenuHomeFreeBSD

bhyve: snapshot capsicum support
Needs ReviewPublic

Authored by gusev.vitaliy_gmail.com on Jun 10 2022, 11:50 PM.

Details

Reviewers
jhb
markj
Group Reviewers
bhyve
Summary

vm destroy context needs more efforts, so ignore that error
until VM_DESTROY ioctl is implemented.

Snapshots are placed to created directory described either in
"BHYVE_SNAPDIR" environment variable if set, or in the current
directory by default.

Sponsored by vStack.


This is similar as D34547, but has only needed things.

Test Plan

Start VM, suspend and then resume.

Start VM, do checkpoint several times. Verify no errors occurred.

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

I'd rather see the snapshot functionality write it's data to a single file. A file descriptor for said file would be passed in from the nvlist via an IPC command.

In other words, bhyvectl would open the snapshot file and pass that file descriptor over the socket to bhyve.

In D35454#804127, @rew wrote:

I'd rather see the snapshot functionality write it's data to a single file. A file descriptor for said file would be passed in from the nvlist via an IPC command.

In other words, bhyvectl would open the snapshot file and pass that file descriptor over the socket to bhyve.

Good point. I have one more proposal. Instead of allowing bhyve to do all things at once (dump memory, kernel pages, etc), I would suggest to introduce several small commands that bhyve can perform:

  • Pause vCPUs
  • Dump VM memory to the passed fd.
  • Dump kernel&device states and config to the passed fd.
  • Unpause vCPUs

In other words, bhyvectl will send several commands to get VM memory file, kernel&device&conf file and 'bhyve' will handle it.

That approach will help in implementing Live Migration, that will imply addition commands:

  • Dump changed VM pages to a passed fd
  • Apply changed VM pages from passed fd.

@rew What do you think about idea implementing "bricks/handlers" in bhyve and full logic in bhyvectl ?

not sold on it

My perspective about save/restore is that the limitations/file format issues described in the original commit message, 483d953a86a2507355f8287c5107dc827a0ff516, need to be resolved/discussed.

Until those issues are addressed, the save/restore feature (and any feature that builds off of it) will not make it into main.

While capsicum support is useful towards obtaining that goal, I'm not convinced that save/restore needs multiple fd's to do what it needs - and I'd prefer to not let that design bleed into other userland tools (bhyvectl).

In D35454#806691, @rew wrote:

not sold on it

My perspective about save/restore is that the limitations/file format issues described in the original commit message, 483d953a86a2507355f8287c5107dc827a0ff516, need to be resolved/discussed.

Until those issues are addressed, the save/restore feature (and any feature that builds off of it) will not make it into main.

So I see the two major issues there :

  1. versioning
  2. file format.

While it is under discussing how to save binary data, I would create review that saves entire VM config to a .meta file with config's format, and all fields from .meta are saved in config's format. Example:

kern_structs.device.vioapic.size=388
kern_structs.device.vioapic.file_offset=7844

etc.

This will allow to resume VM just provide this command: bhyve -r $snapshot_name
Also it will protect against different command lines in suspended VM and resumed.

While capsicum support is useful towards obtaining that goal, I'm not convinced that save/restore needs multiple fd's to do what it needs - and I'd prefer to not let that design bleed into other userland tools (bhyvectl).

@rew bhyvectl doesn't need pass multiple fd's. It can pass one fd at time or pass bitmask of commands. If use one fd for all steps (save all things - ram, kernel, config to one file) - it will require completly rework saving format.

If use one fd for all steps (save all things - ram, kernel, config to one file) - it will require completly rework saving format.

Yes, I'm suggesting that the saving/file format will need an overhaul before the save/restore feature can be turned on by default.

From my point of view, what we have in the tree right now is a working prototype which needs to be iterated on. And given that the file format is likely to change, the usage/requirement of these fd's might also change - which is why I'm not convinced that these fd's are worth messing with right now.

Let me make it clear though...this is my current line of thinking and I'm not requesting or blocking any changes here.