bhyve - ARMv8 implementation

[andrei.cos.martin_gmail.com]

Summary

This patch implements the entire ARM virtualization.
The ARMv8 version supports only the processors with GICv3.
It supports virtio-block, virtio-net, virtio-console and virtio-rnd.
The virtual machine uses bvmconsole in order to give a shell.

Test Plan

Building ARMv8 world/host/guest. (On a FreeBSD virtual machine)
The scripts can be found in this utils repository: https://github.com/FreeBSD-UPB/bhyvearm64-utils
Change the paths in host.json, guest_kernel.json and disk.json accordingly.

./build.py --config config/arm64/host.json --build buildworld
./build.py --config config/arm64/guest_kernel.json --build buildkernel
./build.py --config config/arm64/host.json --build buildkernel
./build.py --config config/arm64/host.json --build installworld
./build.py --config config/arm64/host.json --build installkernel

ARM Foundation Platform emulator - Cortex A53
(On a FreeBSD virtual machine - utils repository)
./disk.py --config config/arm64/disk.json

(On Linux host)
Copy the generated disk.img from the FreeBSD virtual machine on the Linux host.
./run_image.sh
An XTerm terminal will show up. Inside it, the following script will start a virtual machine:
./virtio_run.sh

On ESPRESSOBin
After installworld and installkernel, in the filesystem's root directory, copy the virtio_run.sh script from the utils repository and the guest kernel.
Start the host FreeBSD and run the virtio_run.sh script. It will load the vmm module, load the guest kernel using bhyveload and start the virtual machine.

At this point the virtual machine supports only one vcpu.

---

The auxiliary files (custom kernel configs or device trees) have been moved in the utils repository, in the freebsd directory.

• 11 January 2021

Updated the codebase in order to apply de patch easily.
Changed the GENERIC kernel config and added it in the utils repository, at "freebsd/sys/arm64/conf/".
Added the "projects/bhyvearm64-v11Jan2021" tag for the utils and forked FreeBSD repositories.

• 26 April 2021

Added the "projects/bhyvearm64-v26Apr2021" tag for the utils and forked FreeBSD repositories.
At this point, in the utils repository, the 'files_host/bsd_run.sh' starts the virtual machine with a detached device tree
and communicates with it using MMIO UART.
In order to make it work, you need to manually compile the device tree, using
dtc <utils_repository>/freebsd/sys/dts/arm64armada-3720-espressobin-guest.dts > device-tree.dtb

[emaste]

sbin/shutdown/shutdown.c

346 ↗

(On Diff #78821)

?

[andrei.cos.martin_gmail.com]

sbin/shutdown/shutdown.c

346 ↗

(On Diff #78821)

The function, for some reason, stalls indefinitely when the guest calls it.
I am investigating the reason and revert it.

[andrei.cos.martin_gmail.com]

Removed the armv7 support.
Removed auxiliary files used for specific devices (kernel configs, device trees).

[me_jamesrm.com]

sys/arm64/vmm/arm64.c

173

Bug? IRGN0 is associated with VTCR_EL2 not TCR_EL2 *
Should this be removed?

• https://developer.arm.com/docs/ddi0595/h/aarch32-system-registers/vtcr#IRGN0_9

[andrei.cos.martin_gmail.com]

sys/arm64/vmm/arm64.c

173

The link you sent is for the aarch32. I believe we should look at:

https://developer.arm.com/docs/ddi0595/h/aarch64-system-registers/tcr_el2
https://developer.arm.com/docs/ddi0595/h/aarch64-system-registers/vtcr_el2

[me_jamesrm.com]

sys/arm64/vmm/arm64.c

173

Ah yes. My mistake. Thanks for the correct link.

[andrei.cos.martin_gmail.com]

Solved a kernel panic while destroying the virtual machine:

bhyvectl --vm=test --destroy

Added support for multi-vcpu guests. The "-c" parameter no longer
has an argument. If the parameter is specified, the guest will start with
the number of CPUs specified in the guest's device tree, otherwise 1 vCPU.

[emaste]

Is this work also available in a git repo?

sys/arm/arm/generic_timer.h

14–16

Are you fine with using the standard 2-clause license (i.e., the one in COPYRIGHT at the top of the tree)? I suspect this is just copy-pasted from an existing example.

sys/kern/kern_cons.c

176 ↗

(On Diff #83233)

changes in this file seem extraneous

[rew]

Is there a reason why uart (instead of bvmconsole) can't be used to get a shell for armv8?

[andrei.cos.martin_gmail.com]

In D26976#647375, @emaste wrote:

Is this work also available in a git repo?

Yes, at the following link:
https://github.com/FreeBSD-UPB/freebsd-src/tree/projects/bhyvearm64

In D26976#647684, @rew wrote:

Is there a reason why uart (instead of bvmconsole) can't be used to get a shell for armv8?

We are using bvmconsole is because, currently, there is no implementation for mmio uart (only for pci).
We have this issue in mind and we plan to address it in the near future.

[andrei.cos.martin_gmail.com]

Rebased with CURRENT
Removed consport and added MMIO UART.
Detached the device tree from the guest's kernel images. The compiled device tree can be sent as an argument to bhyveload.

[bcran]

I tested the code from https://github.com/FreeBSD-UPB/freebsd-src/tree/projects/bhyvearm64/ (as of 2021-05-29) on my SolidRun Honeycomb system and it fails to allocate resources in the gicv3 code when I run kldload vmm.

vgic0: <ARM Virtual Generic Interrupt Controller v3> numa-domain -24576 on gic0
vgic0: Could not setup maintenance interrupt

[grehan]

bcran that's just informational: it still works fine.

[andrei.cos.martin_gmail.com]

Added the "projects/bhyvearm64-v8Sep2021" tag for the utils and FreeBSD repositories.
Added the bootrom feature, which was tested with the FreeBSD and Debian for arm64 images.
There is a script in the utils reopsitory, called bootrom_run.sh, that starts a guest.
As bootrom, you can compile and use U-Boot:

https://github.com/FreeBSD-UPB/u-boot

The following instructions are used to generate the u-boot.bin file:

gmake bhyve_arm64_defconfig
gmake CROSS_COMPILE=aarch64-none-elf- -j2