In D38858#885508, @rew wrote:

! In D38858#884878, @gusev.vitaliy_gmail.com wrote:
Did you mean about format of the image files or something another? If format, I am going to resolve it with coming nvlist changes.

you plan on converting the snapshot file format to an nvlist?

Moved casper dependency to lib9p.

Added revert commit 966026246e62769f3bcd8247a47fe0f4f0433aba

Fixed @markj notes.

@markj @corvink Please re-accept.

@markj @corvink Please re-accept.

Fix note, use init_checkpoint_thread(ctx) != 0

Add empty line after #endif

Corrected according with @rew notes.

In D38858#884892, @rew wrote:

In D38858#884878, @gusev.vitaliy_gmail.com wrote:

Did you mean about format of the image files or something another? If format, I am going to resolve it with coming nvlist changes.

Have you sought out jhb's advice on this yet? I know he has expressed his opinion on it in the past.

In D38858#884827, @rew wrote:

I still don't understand why you haven't addressed my comments that I raised in the original review. Do you feel it's not worth addressing or is that I'm out of line?

@corvink As you suggested I split this patch to the several reviews:

Rebase on to review https://reviews.freebsd.org/D38872

Correct 'fd' check:

@rew Could you take a look?

Update according to https://reviews.freebsd.org/D38836

Updated to the latest main branch. Open directory at bhyvectl side and pass fd to bhyve so that it can use openat() under capsicum.

up?

@jhb Could you look at this review ?

Could anyone commit it ?

Could anyone commit it ?

Could anyone commit it ?

spinup_vcpu all guest-cpus with suspended state and later resume it. Resume only BSP in case of startup.

@jhb Updated diff accordingly with your note.

Removed "vmmops_snapshot" and related code.

@jhb Current things are good enough to proceed?

@jhb @corvink Please look at updated diff. Is it good enough and perform what you suggested?

@jhb Could you review this patch, please? If possible it should be committed, since w/o patch, suspend/resume doesn't work with multi devices.

In D38476#875774, @markj wrote:

The commit hash in the review title doesn't match anything in the src repo.

In D38474#875991, @jhb wrote:

A more concise description for the log message might be "actually restore vcpu->tsc_offset".

@jhb Could you commit it?

@markj Could you commit it?

@jhb up?

In D36119#820585, @markj wrote:

In D36119#820583, @gusev.vitaliy_gmail.com wrote:

I don't see any changes in Makefile or rated. What warnings are you fixing?

In this review, a couple of warnings about uninitialized variables I found by removing the WARNS?=2 line in the Makefile and recompiling pci_nvme.c. D36118 contains fixes for most of the other warnings in that file. Everything else I did so far is not in phabricator yet.

I don't see any changes in Makefile or rated. What warnings are you fixing?

what if enable extra warning by default? It would help avoid adding new bugs.

In D29262#817674, @mgrooms_shrew.net wrote:

In D29262#817628, @gusev.vitaliy_gmail.com wrote:

In D29262#815770, @ionut.mihalache1506_gmail.com wrote:

What is the real purpose for this review? If debugging, I think this is not right way by adding huge amount of new code. I would suggest to remove all json things, remove all SNAPSHOT_VAR_OR_LEAVE and its variations and move basis to nvlist.

Using nvlist is more elegant and simpler, brings ability to extract (dump) all information for debugging, and gives to snapshot/resume dealing with optional parameters, removing some parameters, adding new parameters without breaking "resume" with previously created snapshots.

I've been funding work in collaboration with Mihai Carabas at UPB since 2016. The goal was to attract more developers to the bhyve project and implement suspend/resume with live migration. After the initial patch landed thanks to JHB, we spoke to project leaders about what needed to be done to move the suspend/resume code out from under #ifdef so it could become a generally useful feature. Mihai and I prioritized which projects should receive scholarship funding based on that feedback and engaged students that were willing to further these goals. One of the blocking items was to improve the state file format. This patch was the result of a student project to work on that item. Sadly, there has been little to no review or feedback. As a result, these considerable efforts are wasted, students eventually get frustrated and loose interest in working on FreeBSD projects.

We had a few discussions related to what file format to use. One of the options discussed was to use a format similar to the one used for the basic VM config file. Maybe Mihai can recall more specific details. I just remember that using json was the result of a consensus. If there was a new discussion to reach a new consensus, we'd be thrilled. That would at least be progress.

In D29262#815770, @ionut.mihalache1506_gmail.com wrote:

@jhb up?

In D35328#811976, @andy_omniosce.org wrote:

This change looks good to me. Is there anything stopping this being merged?
(I'm doing another sync to illumos at the end of the month and I'd like to include this if possible)

@markj Could you approve again and commit?

Corrected include order according to Mark's comment.

In D35447#809008, @jhb wrote:

Ok. Long term Patrick is more correct and we should not be storing this state but instead only trying to store "architectural" state like the APIC page, not the PIR, but I understand why for now we might need to do this.

@jhb ping?