In D41868#954259, @corvink wrote:

If you don't configure a TPM device for your VM, the TPM driver refuses to start and the boot wents on properly.

If you don't configure a TPM device for your VM, the TPM driver refuses to start and the boot wents on properly.

Would that break something on FreeBSD < 14 ?

• bump portrevision

In D41778#952345, @markj wrote:

The bhyve manual page needs to be updated as well.

Would -o acpi_tables=false still work to restore the old behaviour, if one wanted?

• update man page

The bhyve manual page needs to be updated as well.

• bump portrevision

You need to bump PORTREVISION in edk2/Makefile otherwise package will not be upgraded.

Tested here and it fixes the issue at hand, LGTM.

In D32961#944105, @jhb wrote:

I think to address Mark's concern you would just need to have a tpm_init with the new code currently added to lpc_init. I would perhaps suggest that we need a init_devices function called from main that would call init_pci and then this new tpm_init. Eventually we might want to have a linker set or C++ constructors or something for device models to register init routines, but for now just adding a static call to tpm_init would be ok.

• create a init_devices and init_tpm function

I think to address Mark's concern you would just need to have a tpm_init with the new code currently added to lpc_init. I would perhaps suggest that we need a init_devices function called from main that would call init_pci and then this new tpm_init. Eventually we might want to have a linker set or C++ constructors or something for device models to register init routines, but for now just adding a static call to tpm_init would be ok.

• mention supported tpm types in man page

@jhb @markj Any additional comments?

• improve man page

Originally the debug_cpus mask was only used for vCPUs stopped while the debug stub was active and in a stop, so it didn't seem worth adding to bhyvectl at the time. Over time it has been reused for other things (snapshots and now idle APs).

In D34717#926534, @mihaiburcea15_gmail.com wrote:

In D34717#925881, @corvink wrote:

I do agree, that this patch series does things in the wrong order. It would be better to add the cmdline parameter and especially the manpage entry as last step. So, I'm going to revert this commit.

Sounds fair. I can move the parameters and manpage stuff to the last parts. Would you prefer splitting the warm and live migration parameters in different parts as well?

In D34720#926533, @mihaiburcea15_gmail.com wrote:

In D34720#924367, @corvink wrote:

There are some patches in flight for using a nvlist for snapshot/restore. Wouldn't it be better to use it for migration too?
So, the sender could send a nvlist which contains information about all sections like memory, kern_structs etc. After that, both (sender and reciever) can send/recv all sections mentioned in the nvlist.

Sounds good. Could you point me to those patches, please?

In D34717#925881, @corvink wrote:

I do agree, that this patch series does things in the wrong order. It would be better to add the cmdline parameter and especially the manpage entry as last step. So, I'm going to revert this commit.

In D34720#924367, @corvink wrote:

There are some patches in flight for using a nvlist for snapshot/restore. Wouldn't it be better to use it for migration too?
So, the sender could send a nvlist which contains information about all sections like memory, kern_structs etc. After that, both (sender and reciever) can send/recv all sections mentioned in the nvlist.

In D34718#925884, @gusev.vitaliy_gmail.com wrote:

I suppose that fixing existing issues and make the code stable is more important for now.

In D34718#925873, @elenamihailescu22_gmail.com wrote:

Do you already have a migration feature/tool implemented (bmigrate)? If so, please let us know if you plan to make it available in the future.

In D34718#925207, @afedorov wrote:

I think we should bring support for snapshots first. And enable BHYVE_SNAPSHOT by default.

To do this, it remains to agree on the snapshot format: https://lists.freebsd.org/archives/freebsd-virtualization/2023-May/001295.html

Would be really nice to receive some feedback before it's committed ...
Just a short comment would be enough.

In D34718#925838, @gusev.vitaliy_gmail.com wrote:

In D34718#925665, @mgrooms_shrew.net wrote:

In D34718#925619, @gusev.vitaliy_gmail.com wrote:

So instead of having a lot code in bhyve, I would suggest to have portion of primitives that can help to implement warm, live migration outside of bhyve process.

Wait. You're casually suggesting that years of work be discarded in favor of an alternate approach. Please be more specific about what you perceive to be the problem with the method taken in this patch series. "I have another idea" isn't a very good reason to go in a completely different direction.

Suggestion was about to help you to move to the right direction. I understand that this patch series ate time to implement, but it should be not just reason to integrate it into bhyve. Do you agree? Instead, I would talk about better implementation with more robust approach.

I suppose the following reasons to discuss about this patch-series:

1. Snapshot/resume code was totally thrown and "implementing team" forget about the code since it was integrated in 2020. Yes, it was a good job, but it was incomplete and has a lot of issues and nobody wants to fix them.
2. Who will support integrated code ? This patch-series has a lot code and it potentially has bugs. Will it be the same story just integrate something and forget about code, i.e. do not participate in bug fixing?
3. Instead of adding a lot of code to bhyve, it is better to place code outside from bhyve source, to make bhyve pretty small and secure.
4. Warm/Live migrations can have different steps and requirements, hooks, etc. For example, it can be done with storage migration (Storage vMotion) or without, with additional tunning VM like vCPU slowdown or without. And better place for implementing those is another program that will handle all needed requirements and functionalities.

Correct me, if I see something wrong.

In D34718#925665, @mgrooms_shrew.net wrote:

In D34718#925619, @gusev.vitaliy_gmail.com wrote:

So instead of having a lot code in bhyve, I would suggest to have portion of primitives that can help to implement warm, live migration outside of bhyve process.

Wait. You're casually suggesting that years of work be discarded in favor of an alternate approach. Please be more specific about what you perceive to be the problem with the method taken in this patch series. "I have another idea" isn't a very good reason to go in a completely different direction.

In D34718#925619, @gusev.vitaliy_gmail.com wrote:

In D34718#925608, @elenamihailescu22_gmail.com wrote:

We consider that the migration part is a must-have feature for a mature hypervisor and we really want to have it for bhyve. The end goal is to have a live migration feature for bhyve. It means inspecting the guest memory pages to check for changes between rounds. It also modifies a dirty bit. The inspection (and changes) should be done from within the bhyve process. I'm not sure how this part should be tackled in your suggestion. However, please keep in mind that this approach was discussed with the bhyve's maintainers 4 years ago. We asked for advice and came up with this idea of having a custom dirty bit for migration.

I don't see any problem with that. I will open review with possibility to get diff pages.

So instead of having a lot code in bhyve, I would suggest to have portion of primitives that can help to implement warm, live migration outside of bhyve process.

In D34718#925619, @gusev.vitaliy_gmail.com wrote:

In D34718#925608, @elenamihailescu22_gmail.com wrote:

We consider that the migration part is a must-have feature for a mature hypervisor and we really want to have it for bhyve. The end goal is to have a live migration feature for bhyve. It means inspecting the guest memory pages to check for changes between rounds. It also modifies a dirty bit. The inspection (and changes) should be done from within the bhyve process. I'm not sure how this part should be tackled in your suggestion. However, please keep in mind that this approach was discussed with the bhyve's maintainers 4 years ago. We asked for advice and came up with this idea of having a custom dirty bit for migration.

I don't see any problem with that. I will open review with possibility to get diff pages.

So instead of having a lot code in bhyve, I would suggest to have portion of primitives that can help to implement warm, live migration outside of bhyve process.

It's pretty simple: The most constructive way to help a review process is to provide feedback on the proposed code changes.

In D34718#925608, @elenamihailescu22_gmail.com wrote:

We consider that the migration part is a must-have feature for a mature hypervisor and we really want to have it for bhyve. The end goal is to have a live migration feature for bhyve. It means inspecting the guest memory pages to check for changes between rounds. It also modifies a dirty bit. The inspection (and changes) should be done from within the bhyve process. I'm not sure how this part should be tackled in your suggestion. However, please keep in mind that this approach was discussed with the bhyve's maintainers 4 years ago. We asked for advice and came up with this idea of having a custom dirty bit for migration.

In D34718#925599, @gusev.vitaliy_gmail.com wrote:

In D34718#925598, @darius.mihaim_gmail.com wrote:

The *live* migration must be part of bhyve, as the migration process must transfer the virtual machine's memory from one host to the other while the virtual machine runs on the original host.

Of course, some changes to the migration code are unavoidable in case the snapshot process changes in some particular ways, but they should remain minimal in my expectations.

I believe that migration code can be outside of bhyve process even with warm or live migration. Really, suppose to have bmigrate process written in C, shell, python (high level language):

a. bhyve dumps diff-pages to a file1.
b. bmigrate sends data to host2 or just places on shared storage
c. bmigrate calls bhyve to suspend without memory.
d. bmigrate sends suspended data to host2. and restore memory 1-st iteration on host2.
e. bmigrate calls vmm to dump diff-pages to file2.
f. bmigrate sends diff-pages to host2 and restore 2-st iteration.
g. bmigrate restore process on host2.

This is rough idea, but it should work w/o adding a lot of code to bhyve.

The same is for warm migration. Why this code should be in bhyve? All states, cpus, etc. could be tracked outside of bhyve.

In D34718#925599, @gusev.vitaliy_gmail.com wrote:

In D34718#925598, @darius.mihaim_gmail.com wrote:

The *live* migration must be part of bhyve, as the migration process must transfer the virtual machine's memory from one host to the other while the virtual machine runs on the original host.

Of course, some changes to the migration code are unavoidable in case the snapshot process changes in some particular ways, but they should remain minimal in my expectations.

I believe that migration code can be outside of bhyve process even with warm or live migration. Really, suppose to have bmigrate process written in C, shell, python (high level language):

a. bhyve dumps diff-pages to a file1.
b. bmigrate sends data to host2 or just places on shared storage
c. bmigrate calls bhyve to suspend without memory.
d. bmigrate sends suspended data to host2. and restore memory 1-st iteration on host2.
e. bmigrate calls vmm to dump diff-pages to file2.
f. bmigrate sends diff-pages to host2 and restore 2-st iteration.
g. bmigrate restore process on host2.

This is rough idea, but it should work w/o adding a lot of code to bhyve.

The same is for warm migration. Why this code should be in bhyve? All states, cpus, etc. could be tracked outside of bhyve.

In D34718#925598, @darius.mihaim_gmail.com wrote:

The *live* migration must be part of bhyve, as the migration process must transfer the virtual machine's memory from one host to the other while the virtual machine runs on the original host.

Of course, some changes to the migration code are unavoidable in case the snapshot process changes in some particular ways, but they should remain minimal in my expectations.

In D34718#925593, @gusev.vitaliy_gmail.com wrote:

In D34718#925583, @darius.mihaim_gmail.com wrote:

In D34718#925207, @afedorov wrote:

I think we should bring support for snapshots first. And enable BHYVE_SNAPSHOT by default.

To do this, it remains to agree on the snapshot format: https://lists.freebsd.org/archives/freebsd-virtualization/2023-May/001295.html

I'm not sure whether to tie the transfer of the memory dump to the TCP.

Or add an opportunity similar to 'zfs send' to be independent of transport. For example, share a memory image using SSH or NFS.

Not sure how the transfer would be *tied* to the TCP protocol just by having the implementation use TCP by default. Many tools like curl, socat and browsers have figured out that you can use URIs to specify protocols and even authentication when connecting to a service. The migration parameters can be extended to use ssh://, file:/// or even the basic - for piping the binary data to standard output.

I think asking this question more than one year after this review has been open, and more than two after the original review (https://reviews.freebsd.org/D28270) was created is simply meant to delay accepting the changes. I understand that having a good and secure code base is a must, but this change simply asks for "why don't we over engineer this part which has very little to do with the actual virtual machine migration process?", without actually providing a good case for "this is not up to the BSD standards".

I had review (D35590) that was opened about a year. This is fix for current snapshot implementation. We are talking about two things:

1. Why this feature cannot be done outside of bhyve?
2. Who will do fixing if this migration review(s) becomes committed?

I suppose, both questions should answered before committing those changes. Because even without this migration additions, current snapshot implementation still has a lot of bugs and, I believe, they should be fixed first before add new functionality for snapshot/resume.

In D34717#925263, @mgrooms_shrew.net wrote:

I can't tell you how disappointing it is to see comments like this after a review sits in public review for months. You could have easily voiced any of these concerns during the time leading up to the commit instead of withholding them until after it was made. What exactly is your goal hear?

In D34718#925583, @darius.mihaim_gmail.com wrote:

In D34718#925207, @afedorov wrote:

I think we should bring support for snapshots first. And enable BHYVE_SNAPSHOT by default.

To do this, it remains to agree on the snapshot format: https://lists.freebsd.org/archives/freebsd-virtualization/2023-May/001295.html

I'm not sure whether to tie the transfer of the memory dump to the TCP.

Or add an opportunity similar to 'zfs send' to be independent of transport. For example, share a memory image using SSH or NFS.

Not sure how the transfer would be *tied* to the TCP protocol just by having the implementation use TCP by default. Many tools like curl, socat and browsers have figured out that you can use URIs to specify protocols and even authentication when connecting to a service. The migration parameters can be extended to use ssh://, file:/// or even the basic - for piping the binary data to standard output.

I think asking this question more than one year after this review has been open, and more than two after the original review (https://reviews.freebsd.org/D28270) was created is simply meant to delay accepting the changes. I understand that having a good and secure code base is a must, but this change simply asks for "why don't we over engineer this part which has very little to do with the actual virtual machine migration process?", without actually providing a good case for "this is not up to the BSD standards".

In D34718#925207, @afedorov wrote:

I think we should bring support for snapshots first. And enable BHYVE_SNAPSHOT by default.

To do this, it remains to agree on the snapshot format: https://lists.freebsd.org/archives/freebsd-virtualization/2023-May/001295.html

I'm not sure whether to tie the transfer of the memory dump to the TCP.

Or add an opportunity similar to 'zfs send' to be independent of transport. For example, share a memory image using SSH or NFS.

@elenamihailescu22_gmail.com , I really respect your work and what you have done.

In D34718#924935, @gusev.vitaliy_gmail.com wrote:

Before proceed this work to commit, could you provide high level design for this approach ?

In D34717#925210, @afedorov wrote:

@corvink , I think this commit is too early. Because we haven't finished supporting snapshots, but this commit already fixes the user interface for live migration, which is not finished at all.

Sorry for the late comment, but maybe this commit should be rolled back?

In D34718#924935, @gusev.vitaliy_gmail.com wrote:

Before proceed this work to commit, could you provide high level design for this approach ?

In D34717#925255, @rew wrote:

In D34717#925210, @afedorov wrote:

@corvink , I think this commit is too early. Because we haven't finished supporting snapshots, but this commit already fixes the user interface for live migration, which is not finished at all.

Sorry for the late comment, but maybe this commit should be rolled back?

This was more of a drive-by commit than anything else - it contains dead code, duplicated code, and code that's been commented out.

In D34717#925210, @afedorov wrote:

@corvink , I think this commit is too early. Because we haven't finished supporting snapshots, but this commit already fixes the user interface for live migration, which is not finished at all.

Sorry for the late comment, but maybe this commit should be rolled back?

In D34718#925207, @afedorov wrote:

I think we should bring support for snapshots first. And enable BHYVE_SNAPSHOT by default.

@corvink , I think this commit is too early. Because we haven't finished supporting snapshots, but this commit already fixes the user interface for live migration, which is not finished at all.

I think we should bring support for snapshots first. And enable BHYVE_SNAPSHOT by default.

The same for this review. Is it really required to place this code into bhyve or it can be added to upper layer, for example to bhyvectl?

Before proceed this work to commit, could you provide high level design for this approach ?

There are some patches in flight for using a nvlist for snapshot/restore. Wouldn't it be better to use it for migration too?
So, the sender could send a nvlist which contains information about all sections like memory, kern_structs etc. After that, both (sender and reciever) can send/recv all sections mentioned in the nvlist.

One additional thought: What do you think about sending e.g. a nvlist for the spec check? A nvlist would be more flexible for future addition.

Closed by D40109 381ef27d7b7f9d9130b6b308d1d598d093d7cfba

In D40105#923164, @jhb wrote:

Can you provide some before/after JSON snippets?

Can you provide some before/after JSON snippets?