Regenerated diff with full context (-U999999) as requested

When uploading a raw patch via the web UI, please be sure to add -U999999 per the guidance here: https://wiki.freebsd.org/Phabricator#Create_a_Revision_via_Web_Interface

update synopsis as well. Here is the new output for /bin/sh /usr/share/examples/bhyve/vmrun.sh -h:

Usage: vmrun.sh [-aAEhiTuvw] [-9 <9p share>=<path>[,<opts>]]
                [-c <CPUs>] [-C <console>] [-d <disk file>]
                [-e <name=value>] [-f <path of firmware>] [-F <size>]
                [-G [w][address:]port] [-H <directory>]
                [-I <location of installation iso>] [-l <loader>]
                [-L <VNC IP for UEFI framebuffer>]
                [-m <memsize>] [-n <network adapter emulation type>]
                [-p <pcidev|bus/slot/func>]
                [-P <port>] [-t <tapdev>] <vmname>

Works fine for me on arm64. There is one last issue with the usage message, once it's fixed I can land this.

fix unititialized i=... (I had that on my local copy and somehow lost it).
add usage for -9. I'm open to better suggestions, though.

adding 9: to opts for arm64 and re-creating diff with right options.

When uploading diffs, please include extra context by adding -U 999999 to the diff command options: https://wiki.freebsd.org/Phabricator#Create_a_Revision_via_Web_Interface

warm migration is not being worked on actively at the moment - at least to my knowledge. This set of patches were created by a handful of students and stakeholders in academics, unfortunately they seem to have moved on for the moment.

In D34718#969767, @gusev.vitaliy_gmail.com wrote:

In D34718#969176, @freebsd_ny-central.org wrote:

That's all. Kernel crashes with panic: general protection fault. Who wants to fix that ?)

I've read through the various open reviews and your suggestions for a new file format on the mailing list. Would a more robust file format fix that? I.e. adding a checksum or signing the file with a cryptographic hash?

It was pointing that instead of adding new things to "snapshot" code, it would be better to make it stable first.

I think steps should be:

1. Discuss file format for snapshot (one file). I suggest to use nvlist internals.
2. Implement that file format in bhyve and get rid of current approach with three files for snapshot.
3. Add useful debug and testing functions to bhyve to analyse VM snapshots. Again, using nvlist can make it easy.
4. Discuss migrations: warm/live.
5. Add primitives to implement warm/live migration.
6. Implement warm migration.
7. Implement live migration.

I've recently joined the Enterprise Working Group (https://wiki.freebsd.org/EnterpriseWorkingGroup) and I'm supporting Greg Wallace as project manager for the bhyve+jails management work stream. Migration was raised multiple times as critical point and requirement for enterprise stakeholders.

Might it be helpful to arrange a zoom or Teams call to discuss this in more detail?

Yes, it is good idea. We arrange a call for that. I can use both Zoom or Teams.

There's weekly calls on jails and bhyve that could be a good forum to move things forward or alternatively,

we can look into arranging a call specifically for snapshot/migration. Usually, on the weekly calls, there's a

good mix of developers and admins/users to have a good set of sparring partners. If a more dev-focused call

is required, I'd be more than willing to look into setting that up.

Let me know if there's interest!

Thank you for your efforts! We can start discussion at the bhyve Call on Thursday.

From a style perspective, the use of __result_use_check is inconsistent, in one file it appears at the end, in the other it appears right after the return type. I think some of the lines are also over 80 colums.

In D39620#1008905, @sean_rogue-research.com wrote:

stable/13 has this patch
releng/13.2 doesn't have this patch (yet).

I'm not very familiar with FreeBSD's branching system... I see FreeBSD 13.3-RELEASE was released today, is this bug fix included?

stable/13 has this patch
releng/13.2 doesn't have this patch (yet).

Thanks to all of you for moving this along so swiftly! I've already started working on the next iteration. Will post that soon.

Thanks again!

Combining Ed's and Rod's feedback on processors: removing controversial paragraph on Intel architecture

I asked some doc committers on IRC to have a look.

• Updated paragraph on CPU architecture to reference Nehalem instead.
• Added --usage reference for vmstart.sh
• Fixed disconnect key sequence for cu

Its getting much better, probably with pushing a commit soon.

Included the relevant feedback points into this latest update. Thanks for the great inputs. Any further inputs/feedback welcome!

And finally condensed the processor feature constraints into a simpler and shorter statement.

Included additional updates after Rod's feedback. Thanks for the inputs!