warm migration is not being worked on actively at the moment - at least to my knowledge. This set of patches were created by a handful of students and stakeholders in academics, unfortunately they seem to have moved on for the moment.

In D34718#969767, @gusev.vitaliy_gmail.com wrote:

In D34718#969176, @freebsd_ny-central.org wrote:

That's all. Kernel crashes with panic: general protection fault. Who wants to fix that ?)

I've read through the various open reviews and your suggestions for a new file format on the mailing list. Would a more robust file format fix that? I.e. adding a checksum or signing the file with a cryptographic hash?

It was pointing that instead of adding new things to "snapshot" code, it would be better to make it stable first.

I think steps should be:

1. Discuss file format for snapshot (one file). I suggest to use nvlist internals.
2. Implement that file format in bhyve and get rid of current approach with three files for snapshot.
3. Add useful debug and testing functions to bhyve to analyse VM snapshots. Again, using nvlist can make it easy.
4. Discuss migrations: warm/live.
5. Add primitives to implement warm/live migration.
6. Implement warm migration.
7. Implement live migration.

I've recently joined the Enterprise Working Group (https://wiki.freebsd.org/EnterpriseWorkingGroup) and I'm supporting Greg Wallace as project manager for the bhyve+jails management work stream. Migration was raised multiple times as critical point and requirement for enterprise stakeholders.

Might it be helpful to arrange a zoom or Teams call to discuss this in more detail?

Yes, it is good idea. We arrange a call for that. I can use both Zoom or Teams.

There's weekly calls on jails and bhyve that could be a good forum to move things forward or alternatively,

we can look into arranging a call specifically for snapshot/migration. Usually, on the weekly calls, there's a

good mix of developers and admins/users to have a good set of sparring partners. If a more dev-focused call

is required, I'd be more than willing to look into setting that up.

Let me know if there's interest!

Thank you for your efforts! We can start discussion at the bhyve Call on Thursday.

From a style perspective, the use of __result_use_check is inconsistent, in one file it appears at the end, in the other it appears right after the return type. I think some of the lines are also over 80 colums.

In D39620#1008905, @sean_rogue-research.com wrote:

stable/13 has this patch
releng/13.2 doesn't have this patch (yet).

I'm not very familiar with FreeBSD's branching system... I see FreeBSD 13.3-RELEASE was released today, is this bug fix included?

stable/13 has this patch
releng/13.2 doesn't have this patch (yet).

Thanks to all of you for moving this along so swiftly! I've already started working on the next iteration. Will post that soon.

Thanks again!

Combining Ed's and Rod's feedback on processors: removing controversial paragraph on Intel architecture

I asked some doc committers on IRC to have a look.

• Updated paragraph on CPU architecture to reference Nehalem instead.
• Added --usage reference for vmstart.sh
• Fixed disconnect key sequence for cu

Its getting much better, probably with pushing a commit soon.

Included the relevant feedback points into this latest update. Thanks for the great inputs. Any further inputs/feedback welcome!

And finally condensed the processor feature constraints into a simpler and shorter statement.

Included additional updates after Rod's feedback. Thanks for the inputs!

Sorry, just realized I missed the -U99999; included this now.

Updates after initial feedback - also fixed version of ISO files in the command line listings.

Oh, one other comment - when you regenerate the new patch please use -U999999 to get full context - Phabricator will then allow the reader to expand the blocks in between what's been changed.

Just some nits.

Thanks for the great feedback. I'll update the patch accordingly.

Thanks. These are nice updates. I added a few mostly nit-picky comments and added a few other reviewers.

Move vm_snapshot_buf_cmp prototype under #ifndef _KERNEL.

Looks good to me, aside from the comment.

removed vm_snapshot_buf_cmp() in the kernel code

Removed vm_snapshot_buf_cmp() function in the kernel code.

In D43218#985321, @rew wrote:

In D43218#985292, @markj wrote:

This seems fine as a stopgap. If you mail me git patches for this and the MTRR review, I'll apply them.

I went ahead and took care of these, hopefully you don't mind.

In D43218#985292, @markj wrote:

This seems fine as a stopgap. If you mail me git patches for this and the MTRR review, I'll apply them.

This seems fine as a stopgap. If you mail me git patches for this and the MTRR review, I'll apply them.

In D43218#985023, @rew wrote:

as a side thought, wondering if FB_SIZE and SNAPSHOT_BUFFER_SIZE should be computed on the fly at some point in the future.

This looks good - thanks for the fix.

rG7de582874eb9d08f3f87d11ed9e2b9ce8306db79

Address @jhb 's comments.

Note, there is still a hole here I think which is that the RFLAGS pushed as part of an exception frame when an exception or interrupt occurs can "leak" the shadow TF into the guest. That is, if you have enabled RFLAGS.TF to request a step and the guest takes an interrupt, the saved copy of RFLAGS on the stack will have TF set. Masking hardware interrupts during stepping prevents a good bit of this, but if the code hits an exception it will still leak the shadow TF. That isn't fatal, per se, but might be something to think about fixing at some point. I think the fix is you have to hook all exceptions and rewrite the saved RFLAGS on the stack similar to what happens now for PUSHF/POPF. You'd also need to deal with IRET as well so you can rewrite the saved RFLAGS on the stack before letting IRET continue.

This generally looks fine to me modulo some nits. I think we should probably remove checking for the VM_CAP_MASK_HWINTR capability though.

It could be, but I would expect that Linux 6.x kernel just changed behaviour a little and therefore race caused that issue is rarely seen.

In D39620#978525, @sean_rogue-research.com wrote:

Assuming the bug I see is indeed this bug, I reproduce in all 3 of my linux VMs, running Ubuntu 20.04 LTS and 22.04 LTS, which are all linux kernel 5.x. Looks like there is no LTS Ubuntu with kernel 6.x until upcoming 24.04. But 23.10 uses kernel 6.5, which I could try. Is kernel 6.x expected to fix something?

stable/13 has this patch

In D39620#978503, @sean_rogue-research.com wrote:

...
I'm not very familiar with phabricator, so I'm unsure: is this fixed in FreeBSD 13.2 or only 14.0?

Hi all,

Address @jhb 's comments.

In D34718#969176, @freebsd_ny-central.org wrote:

That's all. Kernel crashes with panic: general protection fault. Who wants to fix that ?)

I've read through the various open reviews and your suggestions for a new file format on the mailing list. Would a more robust file format fix that? I.e. adding a checksum or signing the file with a cryptographic hash?

In D34718#926530, @gusev.vitaliy_gmail.com wrote:

In D34718#925884, @gusev.vitaliy_gmail.com wrote:

I suppose that fixing existing issues and make the code stable is more important for now.

And if somebody feels that snapshot code is stable enough for adding new features, I would say it is not!

You can simply crash host with the following steps:

1. Suspend VM
2. Run "dd if=/dev/zero bs=1K count=1 conv=notrunc of=${snapshot}.kern"
3. Resume VM

That's all. Kernel crashes with panic: general protection fault. Who wants to fix that ?)

Address @corvink 's comments.

Address @corvink 's comments.

Address @corvink's comments, unrelated exception reflection change was split into its own patch (D42405), updated commit message to be more clear.

Maybe add some more information to the commit message. Something like "This commit adds a new capability to enable software breakpoints. When enabled, vmexits are passed to userspace where they should be handled. ..."