In D34718#925619, @gusev.vitaliy_gmail.com wrote:

In D34718#925608, @elenamihailescu22_gmail.com wrote:

We consider that the migration part is a must-have feature for a mature hypervisor and we really want to have it for bhyve. The end goal is to have a live migration feature for bhyve. It means inspecting the guest memory pages to check for changes between rounds. It also modifies a dirty bit. The inspection (and changes) should be done from within the bhyve process. I'm not sure how this part should be tackled in your suggestion. However, please keep in mind that this approach was discussed with the bhyve's maintainers 4 years ago. We asked for advice and came up with this idea of having a custom dirty bit for migration.

I don't see any problem with that. I will open review with possibility to get diff pages.

So instead of having a lot code in bhyve, I would suggest to have portion of primitives that can help to implement warm, live migration outside of bhyve process.

It's pretty simple: The most constructive way to help a review process is to provide feedback on the proposed code changes.

In D34718#924935, @gusev.vitaliy_gmail.com wrote:

Before proceed this work to commit, could you provide high level design for this approach ?

In D34717#925255, @rew wrote:

In D34717#925210, @afedorov wrote:

@corvink , I think this commit is too early. Because we haven't finished supporting snapshots, but this commit already fixes the user interface for live migration, which is not finished at all.

Sorry for the late comment, but maybe this commit should be rolled back?

This was more of a drive-by commit than anything else - it contains dead code, duplicated code, and code that's been commented out.

In D29262#817628, @gusev.vitaliy_gmail.com wrote:

In D29262#815770, @ionut.mihalache1506_gmail.com wrote:

What is the real purpose for this review? If debugging, I think this is not right way by adding huge amount of new code. I would suggest to remove all json things, remove all SNAPSHOT_VAR_OR_LEAVE and its variations and move basis to nvlist.

Using nvlist is more elegant and simpler, brings ability to extract (dump) all information for debugging, and gives to snapshot/resume dealing with optional parameters, removing some parameters, adding new parameters without breaking "resume" with previously created snapshots.

Do these get bhyve closer to what you believe to be required?

Do you intend to try and commit this to head before the freeze? It functionally looks the same as D801 which was also accepted by adrian and gnn, but with slightly different syntax.

In D801#139427, @kristof wrote:

In D801#138630, @mgrooms_shrew.net wrote:

I noticed that code slush begins May 27th, which is two days from now. Can I help test anything else?

I think this patch isn't going to be ready before the code slush.
I'm honestly not happy with the approach taken here to set and use the PCP bits.

I noticed that code slush begins May 27th, which is two days from now. Can I help test anything else?

I was able to get some light testing done with this patch applied. Things appear to be working well. NOTE: You'r missing a closing bracket in sys/netpfil/pf/pf.c around line 6078.

The pf_ieee8021q_setpcp is being called twice in the ipv6 code path. I submitted a patch to fix this in pfSense so it should apply here as well ...

There's also a OpenBSD VMware PVSCSI driver. If there are licensing concerns with this driver, perhaps the other could be ported instead ...

Any chance this will be reviewed and committed for the 11 release? Seems like it would be a pretty useful addition.

Could this please make it into the tree for the 11 release? Also, is there a review in progress for the pf related changes yet?

Anything I can do to help as far as testing goes?

Just wanted to offer some feedback: I've applied this patch to a 10.2-RELEASE host and have been running it for a week now with zero issues. I also have a patch extracted from pfSense that allows pf to match on or set the vlan pcp values. Both were required to build a FreeBSD pf firewall that works well with Google Fiber internet service. With these applied, I'm able to push/pull the full gigabit up/down using a tiny single core ESXi VM. Without the patches I'm only able to get about 10 megabit up.