Page MenuHomeFreeBSD

bhyve: multiple devices support for suspend/checkpoint/resume
Needs ReviewPublic

Authored by gusev.vitaliy_gmail.com on Jun 24 2022, 7:57 PM.

Details

Reviewers
jhb
markj
rew
Group Reviewers
bhyve
Summary

This is another implementation of https://reviews.freebsd.org/D26387 but with easier way.

Idea is: using unique name as identifier. For PCI devices it would be, for example, virtio-blk-pci-$bus-$slot-$func
Changes:

  • Use unique name to identify device.
  • Add simple register_snapshot_dev() function, for any type of device.
  • Changed some function names and defs related to "user devices" and renamed to "devices"
  • Renamed "struct" section to "kern_structs"
  • Removed pci_find_slotted_dev function
  • Added printing error number if pause/snapshot/resume fail

For example: "devices" section in $dumpfile.meta would be:

"devices": [
    {
      "device": "lpc-pci-0-31-0",
      "size": 520,
      "file_offset": 80418
    },
    {
      "device": "virtio-net-pci-0-4-0",
      "size": 66180,
      "file_offset": 80938
    },
    {
      "device": "virtio-blk-pci-0-3-0",
      "size": 8819,
      "file_offset": 147118
    },
    {
      "device": "hostbridge-pci-0-0-0",
      "size": 436,
      "file_offset": 155937
    },
    {
      "device": "atkbdc",
      "size": 141,
      "file_offset": 156373
    }
  ]

Sponsored by vStack.

Test Plan

Current patch has been working well for more than year.

Command line for example (VM has two virtio-blk disks):

bhyve -c sockets=1,cores=2 -m 2048M -H -A -P -S -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 3:0,virtio-blk,/dev/zvol/datavm/vms/win10h2/disk0,sectorsize=4096 -s 4:0,virtio-net,tap0 -s 3:1,virtio-blk,disk2,sectorsize=4096 -s 7,fbuf,tcp=10.2.0.8:5900,w=1024,h=768 -s 30,xhci,tablet -s 31:0,lpc win10h2

With added addition debug, messages during suspend are:

pci_pause: not implemented for: lpc-pci-0-31-0
pci_pause: not implemented for: xhci-pci-0-30-0
pci_pause: not implemented for: fbuf-pci-0-7-0
pci_pause: not implemented for: hostbridge-pci-0-0-0
[2048.000MiB / 2048.000MiB] |#########################################################################################################################|
Doing snapshot for: lpc-pci-0-31-0 
Doing snapshot for: xhci-pci-0-30-0 
Doing snapshot for: fbuf-pci-0-7-0 
Doing snapshot for: virtio-net-pci-0-4-0 
Doing snapshot for: virtio-blk-pci-0-3-1 
Doing snapshot for: virtio-blk-pci-0-3-0 
Doing snapshot for: hostbridge-pci-0-0-0 
Doing snapshot for: atkbdc

Messages during resume are:

fbuf frame buffer base: 0x888200000 [sz 16777216]
Pausing devices...
pci_pause: not implemented for: lpc-pci-0-31-0
pci_pause: not implemented for: xhci-pci-0-30-0
pci_pause: not implemented for: fbuf-pci-0-7-0
pci_pause: not implemented for: hostbridge-pci-0-0-0
Restoring vm mem...
[2048.000MiB / 2048.000MiB] |#########################################################################################################################|
Restoring devices...
Restoring kernel data...
Resuming devices...
pci_resume: not implemented for: lpc-pci-0-31-0
pci_resume: not implemented for: xhci-pci-0-30-0
pci_resume: not implemented for: fbuf-pci-0-7-0
pci_resume: not implemented for: hostbridge-pci-0-0-0
spinning up vcpu no 1...

Suspend of devices in .meta file looks like this:

{
  "kern_structs": [
    {
      "debug_name": "vhpet",
      "device": 5,
      "size": 284,
      "file_offset": 0
    },
    {
      "debug_name": "vm",
      "device": 2,
      "size": 11776,
      "file_offset": 284
    },
    {
      "debug_name": "vmx",
      "device": 0,
      "size": 18440,
      "file_offset": 12060
    },
    {
      "debug_name": "vioapic",
      "device": 1,
      "size": 388,
      "file_offset": 30500
    },
    {
      "debug_name": "vlapic",
      "device": 3,
      "size": 267328,
      "file_offset": 30888
    },
    {
      "debug_name": "vmcx",
      "device": 6,
      "size": 24064,
      "file_offset": 298216
    },
    {
      "debug_name": "vatpit",
      "device": 8,
      "size": 172,
      "file_offset": 322280
    },
    {
      "debug_name": "vatpic",
      "device": 7,
      "size": 118,
      "file_offset": 322452
    },
    {
      "debug_name": "vpmtmr",
      "device": 9,
      "size": 4,
      "file_offset": 322570
    },
    {
      "debug_name": "vrtc",
      "device": 10,
      "size": 140,
      "file_offset": 322574
    }
  ],
  "basic metadata": {
    "ncpus": 2,
    "vmname": "win10h2",
    "memsize": 2147483648,
    "memflags": 2
  },
  "devices": [
    {
      "device": "lpc-pci-0-31-0",
      "size": 600,
      "file_offset": 322714
    },
    {
      "device": "xhci-pci-0-30-0",
      "size": 2057,
      "file_offset": 323314
    },
    {
      "device": "fbuf-pci-0-7-0",
      "size": 16777648,
      "file_offset": 325371
    },
    {
      "device": "virtio-net-pci-0-4-0",
      "size": 66176,
      "file_offset": 17103019
    },
    {
      "device": "virtio-blk-pci-0-3-1",
      "size": 8815,
      "file_offset": 17169195
    },
    {
      "device": "virtio-blk-pci-0-3-0",
      "size": 8815,
      "file_offset": 17178010
    },
    {
      "device": "hostbridge-pci-0-0-0",
      "size": 432,
      "file_offset": 17186825
    },
    {
      "device": "atkbdc",
      "size": 141,
      "file_offset": 17187257
    }
  ]
}

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Unit Tests Skipped