bhyve: Allow an arm64 kernel as the boot ROM

[dch]

Summary

• on arm64, bhyve copies the boot ROM to guest memory and starts it
• it passes the FDT address in x0
• that is the Linux arm64 boot protocol, which FreeBSD arm64 kernels accept through LINUX_BOOT_ABI
• so bhyve can boot a kernel.bin directly, with no U-Boot and no loader
• fdt.bootargs gives that kernel its environment
• this failed because the FDT goes 1MB into guest memory, inside any kernel
• read image_size from the Image header, which includes the bss
• put the FDT after the kernel when the image is larger than 1MB
• small boot ROMs such as U-Boot keep the FDT where it was
• fail cleanly when the image does not fit in guest memory
• bhyve.8, bhyve_config.5: document fdt.bootargs, and booting a kernel directly
• tested: an arm64 kernel with p9fs, virtio_console and virtio_random built in boots to init over a p9fs root in 0.19s
• U-Boot and loader.efi take 2.8s
• U-Boot still boots as before

[andrew]

We should remove FDT_BASE, but will need to update the u-boot port to use the correct fdt offset first.

I expect we could also replace FDT_SIZE with a dynamic value, but would need to check.

[markj]

JFYI I've been working towards something similar for amd64, though paused for now. There, bhyve needs to know which boot protocol to use, so I added a bootflavour option that one can specify on the command line; this lets one explicitly select from a small number of protocols, one of which would be "directly boot this amd64 kernel".

Here I guess that's not quite necessary, though I still don't love that we have to taste the bootrom image to see if it's a kernel. Do you think it'd be worth introducing that here?

[imp]

usr.sbin/bhyve/aarch64/bhyverun_machdep.c

98

This should be defined somehwere common and included from there.

296

"after its bss" is likely going to cause some grief.
One of the big issues that I hit on amd64 was that we have various 'slop' allowances for some calculations so that the pages we allocate early don't collide or run off the end of available memory. The latter isn't an issue here, but we need to make sure that the kernel starts these pages after the FDT that's loaded this way.
Nearly anything that's loaded here is going to be larger than 1MB, I'd think. I don't think we need to mention the kernel specifically in this comment: Anything larger than 1MB needs special treatment for the FDT.

306

what's a 2 * 1024 * 1024? Why this magic number?

usr.sbin/bhyve/bhyve.8

128

On arm64, bhyve can load the kernel can and start it emulating the u-boot booti protocol.
The kernel must have the drivers it needs to mount root compiled in.

is a shorter way to day this. Also, kldload should still work after boot. For your use case, you want them all compiled in, but that's not strictly necessary.

1316

"with its root filesystem embedded"

but the example then has ufs:/dev/vtb0p2 which isn't embedded, but attached via the virtio drivers. I think you want ufs:/dev/md0 instead.

usr.sbin/bhyve/bhyve_config.5

332

You should maybe cross reference the loader man pages here because it does more than just that. It also parses things like -h for serial, etc.

[imp]

In D61102#1401496, @markj wrote:

JFYI I've been working towards something similar for amd64, though paused for now. There, bhyve needs to know which boot protocol to use, so I added a bootflavour option that one can specify on the command line; this lets one explicitly select from a small number of protocols, one of which would be "directly boot this amd64 kernel".

Here I guess that's not quite necessary, though I still don't love that we have to taste the bootrom image to see if it's a kernel. Do you think it'd be worth introducing that here?

The other option, a better one imho, is to always boot with the same protocol. It's trivial to fake up the minimal metadata needed to do this and makes the kernel entry simpler. We can get away without doing that on arm64 because we have the linux hack (that I thought we wanted to move away from). PowerPc already has the 'fake the boot loader metadata' code needed. All the data needed fists in one page and allows for the possibility of passing in kernel env as well later if we need it. amd64 is going to need to get the memory layout too, at the very least, since that's not possible for the kernel to grab. If there's no loader modules, then it's simple. But then you have device enumeration issues that you'll need a solution to, and you can put the hints into the kenv blob you pass in w/o needing a bunch of special cases (and you know exactly what to pass in when you do this since bhyve knows all the devices).

Our boot test matrix is already horrific to test (like 150 total across all booting scenarios), and a new boot flavor fills me with dread. I don't think it's really necessary. FDT on arm64 is likely OK to continue supporting since many embedded boards don't have ACPI, but amd64 has a lot of deep-baked assumptions about ACPI that may be tricky to elide (the base address on amd64 for ACPI is passed in with a kenv value).

I'm happy to jump on a call to consult about design here.