Page MenuHomeFreeBSD

Add support for gicv2m as a child of gicv3
ClosedPublic

Authored by andrew on Sep 1 2021, 9:48 AM.

Details

Summary

On some systems, e.g. Parallels set to host a Linux VM under an M1 Mac,
there is a GICv2m as a child of the GICv3. We previously assumed the
GICv2m was always a child of a GICv2. Fix this by adding the needed
support to the GICv3 driver.

PR: 258136
Reported by: trasz
Sponsored by: The FreeBSD Foundation

Diff Detail

Repository
R10 FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

andrew requested review of this revision.Sep 1 2021, 9:48 AM

Would be nice if someone could test this on ACPI system, without FDT (I don't have a FreeBSD build environment atm).

If someone is interested, here is how it works in Parallels v17:

  • System flag (VM Configuration -> Hardware -> Boot Order -> Advanced) "vm.efi.dtb=0" will hide FDT and will force ACPI. Likely this can be done also with BSD kernel boot options
  • In Parallels GICv3 is enabled if VM is configured (VM Configuration -> General) as Linux or Windows. The Os Type "Other" forces GICv2

I've tested this (applied on top of https://reviews.freebsd.org/D31767) and it fixes the problem, thank you :-)

Just in case, the difference in dmesg between "Other" and "Other Linux", is:

--- other.txt	2021-09-01 16:24:24.720243000 +0000
+++ linux.txt	2021-09-01 16:21:58.590526000 +0000
@@ -34,8 +34,7 @@ kbd0 at kbdmux0
 ofwbus0: <Open Firmware Device Tree>
 clk_fixed0: <Fixed clock> on ofwbus0
 psci0: <ARM Power State Co-ordination Interface Driver> on ofwbus0
-gic0: <ARM Generic Interrupt Controller> mem 0x2010000-0x201ffff,0x2000000-0x2002fff on ofwbus0
-gic0: pn 0x0, arch 0x2, rev 0x0, implementer 0x0 irqs 128
+gic0: <ARM Generic Interrupt Controller v3.0> mem 0x2010000-0x201ffff,0x2500000-0x2cfffff on ofwbus0
 gicv2m0: <ARM Generic Interrupt Controller MSI/MSIX> mem 0x2250000-0x2250fff on gic0
 generic_timer0: <ARMv8 Generic Timer> irq 0,1,2,3 on ofwbus0
 Timecounter "ARM MPCore Timecounter" frequency 24000000 Hz quality 1000
@@ -59,8 +58,9 @@ ahcich2: <AHCI channel> at channel 2 on ahci0
 ahcich3: <AHCI channel> at channel 3 on ahci0
 ahcich4: <AHCI channel> at channel 4 on ahci0
 ahcich5: <AHCI channel> at channel 5 on ahci0
-virtio_mmio0: <VirtIO MMIO adapter> mem 0x21c0000-0x21c3fff irq 10 on ofwbus0
-vtnet0: <VirtIO Networking Adapter> on virtio_mmio0
+virtio_mmio0: <VirtIO MMIO adapter> mem 0x21a0000-0x21a3fff irq 10 on ofwbus0
+virtio_mmio1: <VirtIO MMIO adapter> mem 0x21c0000-0x21c3fff irq 11 on ofwbus0
+vtnet0: <VirtIO Networking Adapter> on virtio_mmio1
 vtnet0: Ethernet address: 00:1c:42:aa:d3:02
 pcib0: <Generic PCI host controller> mem 0x2300000-0x23fffff on ofwbus0
 pci0: <OFW PCI bus> on pcib0
@@ -105,9 +105,9 @@ WARNING: WITNESS option enabled, expect reduced perfor
 ugen0.1: <Intel EHCI root HUB> at usbus0
 ugen1.1: <0x1033 XHCI root HUB> at usbus1
 uhub0 on usbus0
+uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
 uhub1 on usbus1
 uhub1: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
-uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
 ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
 ada0: <v3 F.79KQEG> ATA8-ACS SATA 2.x device
 ada0: Serial Number 2JM3AK5JVEDMB4BN7SZE
@@ -132,6 +132,7 @@ Root mount waiting for: usbus0
 Root mount waiting for: usbus0
 uhub0: 15 ports with 15 removable, self powered
 mountroot: waiting for device /dev/gpt/rootfs...
+vtballoon0: <VirtIO Balloon Adapter> on virtio_mmio0
 lo0: link state changed to UP
 vtnet0: link state changed to UP
 Invoking IPv6 network device address event may sleep with the following non-sleepable locks held:

With "Windows 11" it's almost like "Other Linux", except you also get:

+uhid2: <Pen Interface> on usbus1

Now, regarding "vm.efi.dtb=0": with "Other", things change like this (I've omitted changes to USB enumeration order, I guess that's random):

--- other.txt   2021-09-01 16:24:24.720243000 +0000
+++ other-nodtb.txt     2021-09-01 16:29:38.989275000 +0000
@@ -2,7 +2,6 @@
 GDB: no debug ports present
 KDB: debugger backends: ddb
 KDB: current backend: ddb
-WARNING: Cannot find freebsd,dts-version property, cannot check DTB compliance
 Copyright (c) 1992-2021 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
@@ -14,7 +13,7 @@ WARNING: WITNESS option enabled, expect reduced perfor
 VT(efifb): resolution 1024x768
 module firmware already present!
 real memory  = 8586985472 (8189 MB)
-avail memory = 8345776128 (7959 MB)
+avail memory = 8345804800 (7959 MB)
 Starting CPU 1 (1)
 Starting CPU 2 (2)
 Starting CPU 3 (3)
@@ -31,39 +30,22 @@ MAP 23f800000 mode 2 pages 592
 MAP 1e00000 mode 0 pages 78
 MAP 2120000 mode 0 pages 1
 kbd0 at kbdmux0
-ofwbus0: <Open Firmware Device Tree>
-clk_fixed0: <Fixed clock> on ofwbus0
-psci0: <ARM Power State Co-ordination Interface Driver> on ofwbus0
-gic0: <ARM Generic Interrupt Controller> mem 0x2010000-0x201ffff,0x2000000-0x2002fff on ofwbus0
+acpi0: <PRLS PRLS_OEM>
+acpi0: Power Button (fixed)
+acpi0: Sleep Button (fixed)
+acpi0: Could not update all GPEs: AE_NOT_CONFIGURED
+psci0: <ARM Power State Co-ordination Interface Driver> on acpi0
+gic0: <ARM Generic Interrupt Controller> iomem 0x2010000-0x2010fff,0x2000000-0x2000fff on acpi0
 gic0: pn 0x0, arch 0x2, rev 0x0, implementer 0x0 irqs 128
+gic0: frame: 0 2250000 0 0 0
 gicv2m0: <ARM Generic Interrupt Controller MSI/MSIX> mem 0x2250000-0x2250fff on gic0
-generic_timer0: <ARMv8 Generic Timer> irq 0,1,2,3 on ofwbus0
+generic_timer0: <ARM Generic Timer> irq 8,9,10 on acpi0
 Timecounter "ARM MPCore Timecounter" frequency 24000000 Hz quality 1000
 Event timer "ARM MPCore Eventtimer" frequency 24000000 Hz quality 1000
 efirtc0: <EFI Realtime Clock>
 efirtc0: registered as a time-of-day clock, resolution 1.000000s
-cpulist0: <Open Firmware CPU Group> on ofwbus0
-cpu0: <Open Firmware CPU> on cpulist0
-cpu1: <Open Firmware CPU> on cpulist0
-cpu2: <Open Firmware CPU> on cpulist0
-cpu3: <Open Firmware CPU> on cpulist0
-cpu4: <Open Firmware CPU> on cpulist0
-cpu5: <Open Firmware CPU> on cpulist0
-cpu6: <Open Firmware CPU> on cpulist0
-cpu7: <Open Firmware CPU> on cpulist0
-ahci0: <AHCI SATA controller> mem 0x2140000-0x2141fff irq 4 on ofwbus0
-ahci0: AHCI v1.10 with 6 3Gbps ports, Port Multiplier not supported
-ahcich0: <AHCI channel> at channel 0 on ahci0
-ahcich1: <AHCI channel> at channel 1 on ahci0
-ahcich2: <AHCI channel> at channel 2 on ahci0
-ahcich3: <AHCI channel> at channel 3 on ahci0
-ahcich4: <AHCI channel> at channel 4 on ahci0
-ahcich5: <AHCI channel> at channel 5 on ahci0
-virtio_mmio0: <VirtIO MMIO adapter> mem 0x21c0000-0x21c3fff irq 10 on ofwbus0
-vtnet0: <VirtIO Networking Adapter> on virtio_mmio0
-vtnet0: Ethernet address: 00:1c:42:aa:d3:02
-pcib0: <Generic PCI host controller> mem 0x2300000-0x23fffff on ofwbus0
-pci0: <OFW PCI bus> on pcib0
+pcib0: <Generic PCI host controller> on acpi0
+pci0: <PCI bus> on pcib0
 pci0: <multimedia, HDA> at device 1.0 (no driver attached)
 ehci0: <Intel 82801FB (ICH6) USB 2.0 controller> mem 0x10005000-0x100053ff at device 2.0 on pci0
 usbus0: EHCI version 1.0
@@ -71,6 +53,21 @@ usbus0 on ehci0
 xhci0: <NEC uPD720200 USB 3.0 controller> mem 0x10004000-0x10004fff at device 3.0 on pci0
 xhci0: 32 bytes context size, 32-bit DMA
 usbus1 on xhci0
+cpu0: <ACPI CPU> on acpi0
+acpi_button0: <Power Button> on acpi0
+acpi_acad0: <AC Adapter> on acpi0
+battery0: <ACPI Control Method Battery> on acpi0
+virtio_mmio0: <VirtIO MMIO adapter> iomem 0x21c0000-0x21c3fff irq 6 on acpi0
+vtnet0: <VirtIO Networking Adapter> on virtio_mmio0
+vtnet0: Ethernet address: 00:1c:42:aa:d3:02
+ahci0: <AHCI SATA controller> iomem 0x2140000-0x2141fff irq 7 on acpi0
+ahci0: AHCI v1.10 with 6 3Gbps ports, Port Multiplier not supported
+ahcich0: <AHCI channel> at channel 0 on ahci0
+ahcich1: <AHCI channel> at channel 1 on ahci0
+ahcich2: <AHCI channel> at channel 2 on ahci0
+ahcich3: <AHCI channel> at channel 3 on ahci0
+ahcich4: <AHCI channel> at channel 4 on ahci0
+ahcich5: <AHCI channel> at channel 5 on ahci0
 armv8crypto0: <AES-CBC,AES-XTS,AES-GCM>
 Timecounters tick every 1.000 msec
 CPU  0: ARM Unknown CPU r0p0 (midr: 410f0000) affinity:  0

When set to "Windows 11" the difference seems to be the same. Both configurations seem to work fine.

Does Parallels have a way to provide fine-grained control over guest devices? I.e., instead of "Other Linux" or such, a "Custom" setting that offers individual on/off controls? (Even if Parallels offers a "FreeBSD" setting, the appropriate devices to enable may change over time and with FreeBSD versions.)

Now, regarding "vm.efi.dtb=0": with "Other", things change like this (I've omitted changes to USB enumeration order, I guess that's random):

--- other.txt   2021-09-01 16:24:24.720243000 +0000
+++ other-nodtb.txt     2021-09-01 16:29:38.989275000 +0000
 gic0: pn 0x0, arch 0x2, rev 0x0, implementer 0x0 irqs 128
+gic0: frame: 0 2250000 0 0 0
 gicv2m0: <ARM Generic Interrupt Controller MSI/MSIX> mem 0x2250000-0x2250fff on gic0

When set to "Windows 11" the difference seems to be the same. Both configurations seem to work fine.

Looks like "gic0: pn 0x0, arch 0x2" means that GICv2 is in use. If all works correctly, then when "Windows 11" is set, there should be "arch 0x3" and line about gicv2m0. Also XHCI should be configured to use MSI interrupts.

I'd suggest also to recreate VM in Parallels v17. It is because in Parallels 17 some of the devices (like network adapters) are now located on PCI bus. But on earlier created VMs they are still on mmio (we had to implement that this way to not break networking configurations of existing VMs). Alternatively it is possible to edit the config.pvs file of the VM and change NetworkBusType setting to 0:

<Runtime>
...
<NetworkBusType>0</NetworkBusType>
</Runtime>

Does Parallels have a way to provide fine-grained control over guest devices? I.e., instead of "Other Linux" or such, a "Custom" setting that offers individual on/off controls? (Even if Parallels offers a "FreeBSD" setting, the appropriate devices to enable may change over time and with FreeBSD versions.)

There are "undocumented" system flags. They are undocumented mostly because nobody wants to take responsibility to support such flags in later versions. But most of them are not secret and if something specific is needed, I can reply with corresponding flag, if it exists. There are not so many flags. Tendency is always try to make an option in config file.
gic version can be modified with flag "devices.gic.version=3" (or 2 if needed)

Hm, interesting. With "<NetworkBusType>0</NetworkBusType>" and the VM type set to "Windows 11", it panics (with the patch above) after successfully probing vtnet0 on virtio_pci0:

panic: giv_v3_release_msix: Trying to release an unused MSI-X interrupt

Backtrace looks roughly like this:

gic_v3_release_msix()
intr_release_msix()
generic_pcie_fdt_release_msix()
pci_release_msi_method()
vtpci_free_interrupts()
vtpci_setup_interrupts()
vtnet_attach()
device_attach()
vtpci_legacy_probe_and_attach_child()
vtpci_legacy_attach()
device_attach()
device_probe_and_attach()
bus_generic_attach()
pci_attach()
device_attach()
device_probe_and_attach()
bus_generic_attach()
device_attach()
device_device_probe_and_attach()
bus_generic_new_pass()
bus_generic_new_pass()
bus_generic_new_pass()
bus_set_pass()
mi_startup()
virtdone()

If, however, I add "vm.efi.dtb=0", the panic is gone and vtnet0 probes correctly on virtio_pci0 instead of virtio_mmio1. For some reason it reports as "<VirtIO PCI (legacy) Network adapter>", not "<VirtIO Networking Adapter>".

This revision was not accepted when it landed; it landed in state Needs Review.Sep 14 2021, 10:43 AM
This revision was automatically updated to reflect the committed changes.