bhyveload: note that newer FreeBSD is explicitly not supported

[kevans]

Summary

If it works, that's fine, but we can't support being able to load
arbitrarily newer FreeBSD versions because of how this is designed. We
don't really want to advocate for using a newer userboot with older
bhyveload in the general case to be safe, so just call it out as
unsupported.

PR: 274689, 273099

[michaelo]

Please MFC this.

[michaelo]

@kevans Anything holding your off applying it?

[kevans]

In D59388#1368872, @michaelo wrote:

@kevans Anything holding your off applying it?

I was going to see if bhyve folks wanted to object. Techncally we could write up how to do it because we have a versioning scheme for the protocol, but my concerns are:

1. Nobody tests these combinations, so no one will really catch if we did break it somehow, and
2. We need a lot more words to describe securely grabbing a proper userboot, because we can't just say "grab it from your guest image if you trust that it hasn't been modified by the guest"

Though, as I write the last point, the damage that could be done is pretty limited these days. bhyveload will be sandboxed before the custom loader gets fdlopen'd. If we *do* want to suggest that, we just want to advocate for not overwriting the host /boot/userboot.so with it (i.e., always stash it somewhere and use bhyveload -l so that we don't need to hold a /boot dirfd to limit its resources to whatever guest code would normally have accessible)

[markj]

Just to make sure I understand: we are saying that one can't expect to load and boot a new FreeBSD kernel with an old bhyveload+userboot.so? Or is it old bhyveload and newer userboot.so? I see that one reason for this is that userboot.so might not recognize some ZFS features used in a newer VM image. Are there other reasons?

P.S., I plan to move bhyveload's functionality into bhyve(8) itself.

[kevans]

In D59388#1369907, @markj wrote:

Just to make sure I understand: we are saying that one can't expect to load and boot a new FreeBSD kernel with an old bhyveload+userboot.so? Or is it old bhyveload and newer userboot.so? I see that one reason for this is that userboot.so might not recognize some ZFS features used in a newer VM image. Are there other reasons?

Old bhyveload + old userboot.so, purely because the version of stand/ the latter builds against.

P.S., I plan to move bhyveload's functionality into bhyve(8) itself.

Oh? Presumably the current userboot protocol would be maintained, or do you have another idea in mind there?

[markj]

In D59388#1370015, @kevans wrote:

In D59388#1369907, @markj wrote:

Just to make sure I understand: we are saying that one can't expect to load and boot a new FreeBSD kernel with an old bhyveload+userboot.so? Or is it old bhyveload and newer userboot.so? I see that one reason for this is that userboot.so might not recognize some ZFS features used in a newer VM image. Are there other reasons?

Old bhyveload + old userboot.so, purely because the version of stand/ the latter builds against.

Ok, that makes sense.

P.S., I plan to move bhyveload's functionality into bhyve(8) itself.

Oh? Presumably the current userboot protocol would be maintained, or do you have another idea in mind there?

Unprivileged bhyve doesn't work with bhyveload today, as that needs the VM to be created and executed by a single process. I'd still use userboot, I just want to have all of the functionality in one executable. I also want to be able to boot a kernel directly without a bootloader. I think we'll want to support different "flavours" of vCPU initialization in order to support these different usecases.

[kevans]

In D59388#1370038, @markj wrote:

Oh? Presumably the current userboot protocol would be maintained, or do you have another idea in mind there?

Unprivileged bhyve doesn't work with bhyveload today, as that needs the VM to be created and executed by a single process. I'd still use userboot, I just want to have all of the functionality in one executable. I also want to be able to boot a kernel directly without a bootloader. I think we'll want to support different "flavours" of vCPU initialization in order to support these different usecases.

Ahh, right. ISTR that Rob Norris had a prototype that added Linux kernel booting to bhyve, or something to that effect. I wonder if he has anything salvageable here? I think he was abstracting it away so that it could be made to work with something like this.

[markj]

In D59388#1370113, @kevans wrote:

In D59388#1370038, @markj wrote:

Oh? Presumably the current userboot protocol would be maintained, or do you have another idea in mind there?

Unprivileged bhyve doesn't work with bhyveload today, as that needs the VM to be created and executed by a single process. I'd still use userboot, I just want to have all of the functionality in one executable. I also want to be able to boot a kernel directly without a bootloader. I think we'll want to support different "flavours" of vCPU initialization in order to support these different usecases.

Ahh, right. ISTR that Rob Norris had a prototype that added Linux kernel booting to bhyve, or something to that effect. I wonder if he has anything salvageable here? I think he was abstracting it away so that it could be made to work with something like this.

Yeah, he had something along these lines. I'm imagining an option -o bootflavour=<linux,freebsd-loader,freebsd-kernel,bootrom> which selects how the BSP is initialized before the guest is launched. "freebsd-loader" would let you run userboot, "freebsd-kernel" would let you boot a kernel directly, "bootrom" would provide the current behaviour, etc..

[jhb]

FWIW, I run head VMs on my stable desktop all the time using bhyveload, but they use UFS. :) I think the caveat in the manpage is fine though (and agree it's not something we support).

usr.sbin/bhyveload/bhyveload.8

182