Page MenuHomeFreeBSD

bhyve - ARMv8 implementation
Needs ReviewPublic

Authored by on Oct 27 2020, 4:30 PM.
Referenced Files
Unknown Object (File)
Sat, Jul 20, 1:30 PM
Unknown Object (File)
Wed, Jul 10, 3:17 PM
Unknown Object (File)
Mon, Jul 8, 4:31 AM
Unknown Object (File)
Sat, Jul 6, 3:35 AM
Unknown Object (File)
Sat, Jul 6, 12:30 AM
Unknown Object (File)
Fri, Jul 5, 9:18 PM
Unknown Object (File)
Fri, Jul 5, 8:48 PM
Unknown Object (File)
Fri, Jul 5, 8:31 PM


Group Reviewers

This patch implements the entire ARM virtualization.
The ARMv8 version supports only the processors with GICv3.
It supports virtio-block, virtio-net, virtio-console and virtio-rnd.
The virtual machine uses bvmconsole in order to give a shell.

Test Plan

Building ARMv8 world/host/guest. (On a FreeBSD virtual machine)
The scripts can be found in this utils repository:
Change the paths in host.json, guest_kernel.json and disk.json accordingly.

./ --config config/arm64/host.json --build buildworld
./ --config config/arm64/guest_kernel.json --build buildkernel
./ --config config/arm64/host.json --build buildkernel
./ --config config/arm64/host.json --build installworld
./ --config config/arm64/host.json --build installkernel

ARM Foundation Platform emulator - Cortex A53
(On a FreeBSD virtual machine - utils repository)
./ --config config/arm64/disk.json

(On Linux host)
Copy the generated disk.img from the FreeBSD virtual machine on the Linux host.
An XTerm terminal will show up. Inside it, the following script will start a virtual machine:

After installworld and installkernel, in the filesystem's root directory, copy the script from the utils repository and the guest kernel.
Start the host FreeBSD and run the script. It will load the vmm module, load the guest kernel using bhyveload and start the virtual machine.

At this point the virtual machine supports only one vcpu.

The auxiliary files (custom kernel configs or device trees) have been moved in the utils repository, in the freebsd directory.

  • 11 January 2021

Updated the codebase in order to apply de patch easily.
Changed the GENERIC kernel config and added it in the utils repository, at "freebsd/sys/arm64/conf/".
Added the "projects/bhyvearm64-v11Jan2021" tag for the utils and forked FreeBSD repositories.

  • 26 April 2021

Added the "projects/bhyvearm64-v26Apr2021" tag for the utils and forked FreeBSD repositories.
At this point, in the utils repository, the 'files_host/' starts the virtual machine with a detached device tree
and communicates with it using MMIO UART.
In order to make it work, you need to manually compile the device tree, using
dtc <utils_repository>/freebsd/sys/dts/arm64armada-3720-espressobin-guest.dts > device-tree.dtb

Diff Detail

Lint Skipped
Tests Skipped

Event Timeline

346 ↗(On Diff #78821)


346 ↗(On Diff #78821)

The function, for some reason, stalls indefinitely when the guest calls it.
I am investigating the reason and revert it. retitled this revision from bhyve - ARMv7/8 implementation to bhyve - ARMv8 implementation. edited the test plan for this revision. (Show Details)

Removed the armv7 support.
Removed auxiliary files used for specific devices (kernel configs, device trees). added inline comments.

Bug? IRGN0 is associated with VTCR_EL2 not TCR_EL2 *
Should this be removed?


Ah yes. My mistake. Thanks for the correct link.

Solved a kernel panic while destroying the virtual machine:

bhyvectl --vm=test --destroy

Added support for multi-vcpu guests. The "-c" parameter no longer
has an argument. If the parameter is specified, the guest will start with
the number of CPUs specified in the guest's device tree, otherwise 1 vCPU.

Is this work also available in a git repo?


Are you fine with using the standard 2-clause license (i.e., the one in COPYRIGHT at the top of the tree)? I suspect this is just copy-pasted from an existing example.

176 ↗(On Diff #83233)

changes in this file seem extraneous

Is there a reason why uart (instead of bvmconsole) can't be used to get a shell for armv8?

Is this work also available in a git repo?

Yes, at the following link:

In D26976#647684, @rew wrote:

Is there a reason why uart (instead of bvmconsole) can't be used to get a shell for armv8?

We are using bvmconsole is because, currently, there is no implementation for mmio uart (only for pci).
We have this issue in mind and we plan to address it in the near future. edited the test plan for this revision. (Show Details)

Rebased with CURRENT
Removed consport and added MMIO UART.
Detached the device tree from the guest's kernel images. The compiled device tree can be sent as an argument to bhyveload.

I tested the code from (as of 2021-05-29) on my SolidRun Honeycomb system and it fails to allocate resources in the gicv3 code when I run kldload vmm.

vgic0: <ARM Virtual Generic Interrupt Controller v3> numa-domain -24576 on gic0
vgic0: Could not setup maintenance interrupt

bcran that's just informational: it still works fine.

Added the "projects/bhyvearm64-v8Sep2021" tag for the utils and FreeBSD repositories.
Added the bootrom feature, which was tested with the FreeBSD and Debian for arm64 images.
There is a script in the utils reopsitory, called, that starts a guest.
As bootrom, you can compile and use U-Boot:

The following instructions are used to generate the u-boot.bin file:

gmake bhyve_arm64_defconfig
gmake CROSS_COMPILE=aarch64-none-elf- -j2