In D40105#921197, @jhb wrote:

Can you provide more description of what this change is doing? From what I can tell it uses err directly to abort quicker if it fails to restore an in-kernel structure, it inlines vm_restore_kern_struct into the loop in vm_restore_kern_structs. It renames vm_snapshot_kern_structs which is somewhat gratuitous IMO as it now no longer matches the other function names like vm_snapshot_user_devs. But I think the big change is not using lookup_struct and instead using meta->dev_name instead of meta->dev_req as the main key for the JSON for a given kernel struct? I think this means you can avoid lookup_struct because now instead of a flat array of all kernel structures they are separate objects with unique names? Can you expand on that more perhaps maybe with some examples of before/after JSON snippets? I would also suggest perhaps only doing the JSON change in this commit and not mixing in the other changes that I think obscure the real change you are making (e.g. the function rename, or inlining the function).

Can you provide more description of what this change is doing? From what I can tell it uses err directly to abort quicker if it fails to restore an in-kernel structure, it inlines vm_restore_kern_struct into the loop in vm_restore_kern_structs. It renames vm_snapshot_kern_structs which is somewhat gratuitous IMO as it now no longer matches the other function names like vm_snapshot_user_devs. But I think the big change is not using lookup_struct and instead using meta->dev_name instead of meta->dev_req as the main key for the JSON for a given kernel struct? I think this means you can avoid lookup_struct because now instead of a flat array of all kernel structures they are separate objects with unique names? Can you expand on that more perhaps maybe with some examples of before/after JSON snippets? I would also suggest perhaps only doing the JSON change in this commit and not mixing in the other changes that I think obscure the real change you are making (e.g. the function rename, or inlining the function).

• split into multiple commits

@jhb @markj Any comments?

Addressed @corvink's comments and added Casper support.

Updated the patches to a newer version of FreeBSD.
Addressed @corvink's and @imp's comments.

Fixed note from @corvink

In D40108#913868, @corvink wrote:

Why don't we call pci_snapshot_pci_dev unconditionally even if pe_snapshot is NULL?

Why don't we call pci_snapshot_pci_dev unconditionally even if pe_snapshot is NULL?

One general note: Please try to explain why a change is required in your commit message. "Related to xy" is not enough. It should explain why it is related to xy and what's the advantage of the patch.

Thanks

@rew Could you look again ?

In D40108#913426, @gusev.vitaliy_gmail.com wrote:

At least it saves pdi.pi_cfgdata. Do you think it is waste doing and .pi_cfgdata can not be changed by guest OS or be changed between bhyve versions ?

Fixed notes from @rew.

In D40108#913426, @gusev.vitaliy_gmail.com wrote:

In D40108#913423, @rew wrote:

not sure that devices should define a snapshot handler when not supported by the device.

At least it saves pdi.pi_cfgdata.

small nits - other than that, looks good

Close as whole patch series D38886 is committed.

In D40108#913423, @rew wrote:

not sure that devices should define a snapshot handler when not supported by the device.

not sure that devices should define a snapshot handler when not supported by the device.

@corvink I've split this patch with several patches. Note, according to @rew comments about adding and usage "register_snapshot_dev", I replaced that code with pci_next() function to walk all available PCI devices and save/restore them. 'atkbdc` device is saved/restored separately in this case.

• don't emulate writes of ASLS. It's not required.

This isn't required any. Neither by GVT-d nor by TPM emulation.

• rebase and split into smaller commits

Perhaps adjust the commit message to say something like "Failing to preserve pir_desc can result in pending interrupts being lost on resume leading to a hung VM" or some such to clarify that the hang is due to missing interrupts.

In D35447#910663, @corvink wrote:

Patch looks good but it doesn't apply any more. Could you send an updated version please?

Rebased on main.

In D35826#910200, @bojan.novkovic_fer.hr wrote:

I'll update the patch and start chopping it up into smaller pieces, probably starting with breakpoint and single-stepping support for SVM.

Patch looks good but it doesn't apply any more. Could you send an updated version please?

In D35826#909488, @corvink wrote:

This is a quite large patch. Could it be split into multiple ones to simplify reviewing and merging?

@corvink Could you look at this ? While Patrick's changes requires (in comment) a lot of modifying generic code, this simple fix could solve issue related for suspend/resume.

This is a quite large patch. Could it be split into multiple ones to simplify reviewing and merging?

We're preparing an update for the patches to a current version of FreeBSD. We'll address @corvink's and @imp's comments there. Thank you for those!

In D35590#908783, @corvink wrote:

@gusev.vitaliy_gmail.com As said, please split this commit. You can also send me a link to a personal git repo where I can pull the changes from.

Just looked at copyright / legal stuff. My comments to migration.h also apply to migration.c.
Corvink had most of the stuff, but I thought I'd add the detail or two that was missing

@gusev.vitaliy_gmail.com As said, please split this commit. You can also send me a link to a personal git repo where I can pull the changes from.

@corvink Can it be committed or I should improve/correct it ?