This lays initial groundwork for eventual enablement of the IOMMU
on powerpc64 systems.
Tested to operate without regressions on an RCS Blackbird system.
Details
Details
Diff Detail
Diff Detail
- Repository
- rG FreeBSD src repository
- Lint
Lint Skipped - Unit
Tests Skipped
Event Timeline
Comment Actions
@adrian Could you give this patch set a thorough test as well? It is not dependent on D54745, I mainly want some additional testing without this on top of D54745 and a merge before going to the next phase of IOMMU enablement. D54745 has some issues with DMA and the AHCI controller that I don't currently understand and will need further investigation.
Comment Actions
This boots fine in qemu-system-ppc64 pseries9 + power9, but in qemu-system-ppc64 pseries-8.0 + power8, it immediately panics on boot with
virtio_pci1: <VirtIO PCI (legacy) Block adapter> port 0x80-0xff mem 0x81030000-0x81030fff,0x210000000000-0x210000003fff irq 4610 at device 2.0 numa-domain 0 on pci0 vtblk0: <VirtIO Block Adapter> numa-domain 0 on virtio_pci1 vtblk0: 8192MB (16777216 512 byte sectors) xhci0: <NEC uPD720200 USB 3.0 controller> mem 0x81020000-0x81023fff irq 4609 at device 1.0 numa-domain 0 on pci0 xhci0: 32 bytes context size, 32-bit DMA xhci0: xECP capabilities <PROTO,PROTO> panic: Page insertion error: -4 (ptegidx: 0x3d780/0x20000, PTE 0x1d80364240391/0xa8b002e cpuid = 0 time = 1 KDB: stack backtrace: 0xc00800000000b8c0: at vpanic+0x1b0 0xc00800000000b970: at panic+0x44 0xc00800000000b9a0: at mphyp_pte_insert_locked+0x144 0xc00800000000ba50: at mphyp_pte_insert+0x9c 0xc00800000000baf0: at moea64_pvo_enter+0x100 0xc00800000000bb80: at moea64_enter_single+0x690 0xc00800000000bce0: at moea64_enter+0x30 0xc00800000000bd20: at kmem_alloc_contig_domainset+0x2a4 0xc00800000000beb0: at kmem_alloc_contig+0x40 0xc00800000000bee0: at bounce_bus_dmamem_alloc+0x1b0 0xc00800000000bf90: at usb_pc_alloc_mem+0xc0 0xc00800000000c000: at usb_bus_mem_alloc_all_cb+0x3c 0xc00800000000c030: at xhci_iterate_hw_softc+0x64 0xc00800000000c080: at usb_bus_mem_alloc_all+0x134 0xc00800000000c130: at xhci_init+0x7f4 0xc00800000000c250: at xhci_pci_attach+0x278 0xc00800000000c310: at device_attach+0x568 0xc00800000000c3f0: at bus_attach_children+0x88 0xc00800000000c430: at ofw_pcibus_attach+0x3bc 0xc00800000000c550: at device_attach+0x568 0xc00800000000c630: at bus_attach_children+0x88 0xc00800000000c670: at ofw_pcib_attach+0x54 0xc00800000000c6a0: at rtaspci_attach+0x19c 0xc00800000000c6f0: at device_attach+0x568 0xc00800000000c7d0: at bus_generic_new_pass+0x190 0xc00800000000c820: at bus_generic_new_pass+0x104 0xc00800000000c870: at bus_generic_new_pass+0x104 0xc00800000000c8c0: at root_bus_configure+0x6c 0xc00800000000c910: at configure+0x1c 0xc00800000000c940: at mi_startup+0x298 0xc00800000000ca50: at __start+0xc4 KDB: enter: panic [ thread pid 0 tid 100000 ] Stopped at kdb_enter+0x70: ori r0, r0, 0x0 db>
I can provide more info if you'd like as I'm setup with remote gdb / qemu for debugging exactly this kind of stuff.
Comment Actions
and from justin on irc
10:32 < jhibbits> I see the problem. ptegidx is much greater than moea64_pteg_count
Comment Actions
ok, tried with 8g in the VM instead of 2g, same issue.
Booting...
cas: selected hash MMU
Kernel entry at 0x101540 ...
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
mmu_phyp: Support for hugepages not found
---<<BOOT>>---
Copyright (c) 1992-2026 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 16.0-CURRENT #9 ahc_20260208_powerpc64-n283761-ce7292337159-dirty: Fri Feb 13 07:17:54 PST 2026
adrian@francine:/home/adrian/work/freebsd/head/freebsd-obj-ppc64/home/adrian/work/freebsd/head/freebsd-src-ppc 64/powerpc.powerpc64/sys/GENERIC64 powerpc
clang version 19.1.7
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: Running on a broken hypervisor that does not support mandatory H_CLEAR_MOD and H_CLEAR_REF hypercalls. Pe rformance will be suboptimal.
mmu_oea64: HW does not support large pages. Disabling superpages...
VT: init without driver.
cpu0: IBM POWER8 revision 2.0, 1000.00 MHz
cpu0: Features dc007182<PPC32,PPC64,ALTIVEC,FPU,MMU,SMT,ISNOOP,ARCH205,ARCH206,VSX,TRUELE>
cpu0: Features2 ae000000<ARCH207,DSCR,ISEL,TAR,VCRYPTO>
real memory = 8544370688 (8148 MB)
avail memory = 8221245440 (7840 MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.
random: entropy device external interface
kbd0 at kbdmux0
ofwbus0: <Open Firmware Device Tree> on nexus0
cpulist0: <Open Firmware CPU Group> on ofwbus0
cpu0: <Open Firmware CPU> on cpulist0
xicp0: <External Interrupt Presentation Controller> on ofwbus0
xicp0: Handling CPUs 0-3
vdevice0: <POWER Hypervisor Virtual Device Root> on ofwbus0
vscsi0: <POWER Hypervisor Virtual SCSI Bus> irq 16781571 on vdevice0
vscsi0: Queue depth 22 commands
uart0: <POWER Hypervisor Virtual Serial Port> irq 16781569 on vdevice0
pcib0: <RTAS Host-PCI bridge> on ofwbus0
pci0: <POWER Hypervisor PCI bus> on pcib0
virtio_pci0: <VirtIO PCI (legacy) Network adapter> port 0x100-0x11f mem 0x81040000-0x81040fff,0x210000010000-0x210000013fff irq 4611 at device 3.0 numa-domain 0 on pci0
vtnet0: <VirtIO Networking Adapter> numa-domain 0 on virtio_pci0
vtnet0: Ethernet address: 00:08:01:12:00:01
vtnet0: netmap queues/slots: TX 1/256, RX 1/128
000.000055 [ 452] vtnet_netmap_attach vtnet attached txq=1, txd=256 rxq=1, rxd=128
virtio_pci1: <VirtIO PCI (legacy) Block adapter> port 0x80-0xff mem 0x81030000-0x81030fff,0x210000000000-0x210000003fff irq 4610 at device 2.0 numa-domain 0 on pci0
vtblk0: <VirtIO Block Adapter> numa-domain 0 on virtio_pci1
vtblk0: 8192MB (16777216 512 byte sectors)
xhci0: <NEC uPD720200 USB 3.0 controller> mem 0x81020000-0x81023fff irq 4609 at device 1.0 numa-domain 0 on pci0
xhci0: 32 bytes context size, 32-bit DMA
xhci0: xECP capabilities <PROTO,PROTO>
panic: Page insertion error: -4 (ptegidx: 0x34c7a0/0x80000, PTE 0x1d800ecc40511/0xa4b402e
panic: Page insertion error: -4 (ptegidx: 0x34c7a0/0x80000, PTE 0x1d800ecc40511/0xa4b402e
cpuid = 0
time = 1
KDB: stack backtrace:
0xc00800000000b8c0: at vpanic+0x1b0
0xc00800000000b970: at panic+0x44
0xc00800000000b9a0: at mphyp_pte_insert_locked+0x144
0xc00800000000ba50: at mphyp_pte_insert+0x9c
0xc00800000000baf0: at moea64_pvo_enter+0x100
0xc00800000000bb80: at moea64_enter_single+0x690
0xc00800000000bce0: at moea64_enter+0x30
0xc00800000000bd20: at kmem_alloc_contig_domainset+0x2a4
0xc00800000000beb0: at kmem_alloc_contig+0x40
0xc00800000000bee0: at bounce_bus_dmamem_alloc+0x1b0
0xc00800000000bf90: at usb_pc_alloc_mem+0xc0
0xc00800000000c000: at usb_bus_mem_alloc_all_cb+0x3c
0xc00800000000c030: at xhci_iterate_hw_softc+0x64
0xc00800000000c080: at usb_bus_mem_alloc_all+0x134
0xc00800000000c130: at xhci_init+0x7f4
0xc00800000000c250: at xhci_pci_attach+0x278
0xc00800000000c310: at device_attach+0x568
0xc00800000000c3f0: at bus_attach_children+0x88
0xc00800000000c430: at ofw_pcibus_attach+0x3bc
0xc00800000000c550: at device_attach+0x568
0xc00800000000c630: at bus_attach_children+0x88
0xc00800000000c670: at ofw_pcib_attach+0x54
0xc00800000000c6a0: at rtaspci_attach+0x19c
0xc00800000000c6f0: at device_attach+0x568
0xc00800000000c7d0: at bus_generic_new_pass+0x190
0xc00800000000c820: at bus_generic_new_pass+0x104
0xc00800000000c870: at bus_generic_new_pass+0x104
0xc00800000000c8c0: at root_bus_configure+0x6c
0xc00800000000c910: at configure+0x1c
0xc00800000000c940: at mi_startup+0x298
0xc00800000000ca50: at __start+0xc4
KDB: enter: panic
[ thread pid 0 tid 100000 ]
Stopped at kdb_enter+0x70: ori r0, r0, 0x0
db>Comment Actions
pvo->pvo_pte.slot is initialized to a hash mod moea64_pteg_count, then updated by the return of the H_ENTER hypercall. There are two possibilities here:
- Some kind of weird buffer overflow that scribbled onto the PVO
- A bug in Qemu
A way to rule out Qemu's hypercall handling, you can try to boot FreeBSD in a KVM-accelerated qemu instance, and see if the problem still occurs. Also, throw some KASSERTs around to double-check that pvo_pte.slot is sane.
Comment Actions
I can't imagine why this would matter offhand, but a key difference is POWER8 is HPT, POWER9 is Radix. You've already ruled out the pSeries/PowerNV question by checking on POWER9/pSeries, and both platforms are supposed to be IODA2 compliant, so the MMU mode is about all I can think of as a major difference.
Comment Actions
ok i thought this was OK on power9 but it seems not:
Booting...
cas: selected radix MMU
Kernel entry at 0x101540 ...
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
---<<BOOT>>---
Copyright (c) 1992-2026 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 16.0-CURRENT #0 n283908-1a6bdd6266e3-dirty: Fri Feb 13 11:24:28 PST 2026
adrian@francine:/home/adrian/work/freebsd/head/freebsd-obj-ppc64/home/adrian/work/freebsd/head/freebsd-src-ppc64/powerpc.powerpc64/sys/GENERIC64 powerpc
clang version 19.1.7
WARNING: WITNESS option enabled, expect reduced performance.
VT: init without driver.
Preloaded elf kernel "/boot/kernel.testing/kernel" at 0xc000000001fde000.
cpu0: IBM POWER9 revision 2.2, 1000.00 MHz
cpu0: Features dc007182<PPC32,PPC64,ALTIVEC,FPU,MMU,SMT,ISNOOP,ARCH205,ARCH206,VSX,TRUELE>
cpu0: Features2 bee00000<ARCH207,DSCR,ISEL,TAR,VCRYPTO,ARCH300,IEEE128,DARN>
real memory = 2101919744 (2004 MB)
available KVA = 34359619583 (32767 MB)
Physical memory chunk(s):
0x000000000000f000 - 0x000000000000ffff, 4096 bytes (1 pages)
0x000000000002c000 - 0x0000000000037fff, 49152 bytes (12 pages)
0x000000000009d000 - 0x00000000000fffff, 405504 bytes (99 pages)
0x0000000001fe1000 - 0x0000000002bfffff, 12709888 bytes (3103 pages)
0x000000000345a000 - 0x0000000003ffffff, 12214272 bytes (2982 pages)
0x0000000005000000 - 0x0000000079619fff, 1952555008 bytes (476698 pages)
0x000000007c84c000 - 0x000000007d86bfff, 16908288 bytes (4128 pages)
0x000000007fdf1000 - 0x000000007fdfffff, 61440 bytes (15 pages)
0x000000007fe04000 - 0x000000007fffffff, 2080768 bytes (508 pages)
avail memory = 1983447040 (1891 MB)
random: registering fast source PowerISA DARN random number generator
random: fast provider: "PowerISA DARN random number generator"
random: no preloaded entropy cache
random: no platform bootloader entropy
arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.
VIMAGE (virtualized network stack) enabled
hostuuid: using 00000000-0000-0000-0000-000000000000
ULE: setup cpu 0
firmware: 't4fw_cfg' version 0: 4523 bytes loaded at 0xc00000000120afa8
firmware: 't4fw_cfg_uwire' version 0: 21433 bytes loaded at 0xc00000000120c153
firmware: 't4fw' version 0: 570880 bytes loaded at 0xc00000000121150c
firmware: 't5fw_cfg' version 0: 5902 bytes loaded at 0xc00000000129cb0c
firmware: 't5fw_cfg_uwire' version 0: 21859 bytes loaded at 0xc00000000129e21a
firmware: 't5fw' version 0: 678912 bytes loaded at 0xc0000000012a377d
firmware: 't6fw_cfg' version 0: 5960 bytes loaded at 0xc00000000134937d
firmware: 't6fw_cfg_uwire' version 0: 22693 bytes loaded at 0xc00000000134aac5
firmware: 't6fw' version 0: 730624 bytes loaded at 0xc00000000135036a
firmware: 't7fw_cfg' version 0: 24216 bytes loaded at 0xc00000000140296a
firmware: 't7fw_cfg_uwire' version 0: 24212 bytes loaded at 0xc000000001408802
random: entropy device external interface
wlan: <802.11 Link Layer>
firmware: 'isp_2400' version 1: 204396 bytes loaded at 0xc00000000141489c
firmware: 'isp_2500' version 1: 205736 bytes loaded at 0xc0000000014554f0
firmware: 'isp_2600' version 1: 290876 bytes loaded at 0xc000000001493fbc
firmware: 'isp_2700' version 1: 343552 bytes loaded at 0xc0000000014eb444
firmware: 'isp_2800' version 1: 380416 bytes loaded at 0xc000000001556444
[ath_hal] loaded
kbd0 at kbdmux0
mem: <memory>
null: <full device, null device, zero device>
openfirm: <Open Firmware control device>
sound_global_init: snd_unit=-1 snd_vchans_enable=1 latency=2 feeder_rate_min=1 feeder_rate_max=2016000 feeder_rate_round=25
tcp_log: tcp_log device
crypto: <crypto core>
ofwbus0: <Open Firmware Device Tree> on nexus0
cpulist0: <Open Firmware CPU Group> on ofwbus0
cpu0: <Open Firmware CPU> on cpulist0
cpu0: Nominal frequency 1000Mhz
xicp0: <External Interrupt Presentation Controller> on ofwbus0
xicp0: Handling CPUs 0-0
vdevice0: <POWER Hypervisor Virtual Device Root> on ofwbus0
vscsi0: <POWER Hypervisor Virtual SCSI Bus> irq 16781571 on vdevice0
vscsi0: no domain found
vscsi0: Mapping IOMMU domain 0x71000002
vscsi0: Queue depth 22 commands
uart0: <POWER Hypervisor Virtual Serial Port> irq 16781569 on vdevice0
uart0: no domain found
pcib0: <RTAS Host-PCI bridge> on ofwbus0
pci0: <POWER Hypervisor PCI bus> on pcib0
pci0: domain=0, physical bus=0
found-> vendor=0x1af4, dev=0x1000, revid=0x00
domain=0, bus=0, slot=3, func=0
class=02-00-00, hdrtype=0x00, mfdev=0
cmdreg=0x0100, statreg=0x0010, cachelnsz=0 (dwords)
lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
intpin=a, irq=0
MSI-X supports 4 messages in map 0x14
map[10]: type I/O Port, range 32, base 0x100, size 5, port disabled
map[14]: type Memory, range 32, base 0x81040000, size 12, memory disabled
map[20]: type Prefetchable Memory, range 64, base 0x210000010000, size 14, enabled
found-> vendor=0x1af4, dev=0x1001, revid=0x00
domain=0, bus=0, slot=2, func=0
class=01-00-00, hdrtype=0x00, mfdev=0
cmdreg=0x0100, statreg=0x0010, cachelnsz=0 (dwords)
lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
intpin=a, irq=0
MSI-X supports 2 messages in map 0x14
map[10]: type I/O Port, range 32, base 0x80, size 7, port disabled
map[14]: type Memory, range 32, base 0x81030000, size 12, memory disabled
map[20]: type Prefetchable Memory, range 64, base 0x210000000000, size 14, enabled
found-> vendor=0x1033, dev=0x0194, revid=0x03
domain=0, bus=0, slot=1, func=0
class=0c-03-30, hdrtype=0x00, mfdev=0
cmdreg=0x0106, statreg=0x0010, cachelnsz=0 (dwords)
lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
intpin=a, irq=0
MSI supports 16 messages, 64 bit
MSI-X supports 16 messages in map 0x10
map[10]: type Memory, range 64, base 0x81020000, size 14, enabled
found-> vendor=0x1234, dev=0x1111, revid=0x02
domain=0, bus=0, slot=0, func=0
class=03-00-00, hdrtype=0x00, mfdev=0
cmdreg=0x0106, statreg=0x0000, cachelnsz=0 (dwords)
lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
map[10]: type Prefetchable Memory, range 32, base 0x80000000, size 24, enabled
map[18]: type Memory, range 32, base 0x81000000, size 12, enabled
virtio_pci0: <VirtIO PCI (legacy) Network adapter> port 0x100-0x11f mem 0x81040000-0x81040fff,0x210000010000-0x210000013fff irq 4611 at device 3.0 numa-domain 0 on pci0
ofw_pci mapdev: start 200000000100, len 32
mmu_radix_mapdev_attr(0x200000000100, 4096, 0)
ofw_pci mapdev: start 200081040000, len 4096
mmu_radix_mapdev_attr(0x200081040000, 4096, 0)
vtnet0: <VirtIO Networking Adapter> numa-domain 0 on virtio_pci0
virtio_pci0: host features: 0x79bf8064 <RingEventIdx,RingIndirectDesc,AnyLayout,NotifyOnEmpty,CtrlMacAddr,GuestAnnounce,CtrlRxModeExtra,CtrlVLANFilter,CtrlRxMode,CtrlVq,Status,MrgRxBuf,TxGSO,MAC,CtrlRxOffloads>
virtio_pci0: negotiated features: 0x308f8064 <RingEventIdx,RingIndirectDesc,CtrlMacAddr,CtrlVLANFilter,CtrlRxMode,CtrlVq,Status,MrgRxBuf,TxGSO,MAC,CtrlRxOffloads>
vtnet0: bpf attached
vtnet0: Ethernet address: 00:08:01:12:00:01
virtio_pci0: using legacy interrupt
vtnet0: netmap queues/slots: TX 1/256, RX 1/128
000.000055 [ 452] vtnet_netmap_attach vtnet attached txq=1, txd=256 rxq=1, rxd=128
virtio_pci1: <VirtIO PCI (legacy) Block adapter> port 0x80-0xff mem 0x81030000-0x81030fff,0x210000000000-0x210000003fff irq 4610 at device 2.0 numa-domain 0 on pci0
ofw_pci mapdev: start 200000000080, len 128
mmu_radix_mapdev_attr(0x200000000080, 4096, 0)
ofw_pci mapdev: start 200081030000, len 4096
mmu_radix_mapdev_attr(0x200081030000, 4096, 0)
vtblk0: <VirtIO Block Adapter> numa-domain 0 on virtio_pci1
virtio_pci1: host features: 0x79006e54 <RingEventIdx,RingIndirectDesc,AnyLayout,NotifyOnEmpty,WriteZeros,Discard,ConfigWCE,Topology,FlushCmd,BlockSize,DiskGeometry,MaxNumSegs>
virtio_pci1: negotiated features: 0x10002e54 <RingIndirectDesc,Discard,ConfigWCE,Topology,FlushCmd,BlockSize,DiskGeometry,MaxNumSegs>
virtio_pci1: Mapping IOMMU domain 0x80000000
virtio_pci1: using legacy interrupt
vtblk0: 8192MB (16777216 512 byte sectors)
xhci0: <NEC uPD720200 USB 3.0 controller> mem 0x81020000-0x81023fff irq 4609 at device 1.0 numa-domain 0 on pci0
ofw_pci mapdev: start 200081020000, len 16384
mmu_radix_mapdev_attr(0x200081020000, 16384, 0)
xhci0: 32 bytes context size, 32-bit DMA
xhci0: xECP capabilities <PROTO,PROTO>
xhci0: Mapping IOMMU domain 0x80000000
xhci0: (New XHCI DeviceId=0x01941033)
usbus0 numa-domain 0 on xhci0
xhci0: usbpf: Attached
vgapci0: <VGA-compatible display> mem 0x80000000-0x80ffffff,0x81000000-0x81000fff at device 0.0 numa-domain 0 on pci0
vgapci0: Boot video device
rtas0: <Run-Time Abstraction Services> on ofwbus0
rtas0: registered as a time-of-day clock, resolution 0.002000s
ossl0: <OpenSSL crypto> on nexus0
crypto: assign ossl0 driver id 0, flags 0xe000000
crypto: assign cryptosoft0 driver id 1, flags 0x6000000
procfs registered
Timecounter "timebase" frequency 512000000 Hz quality 1000
Event timer "decrementer" frequency 512000000 Hz quality 1000
Timecounters tick every 1.000 msec
vlan: initialized, using hash tables with chaining
lo0: bpf attached
IPsec: Initialized Security Association Processing.
tcp_init: net.inet.tcp.tcbhashsize auto tuned to 16384
GEOM: new disk vtbd0
usbus0: 5.0Gbps Super Speed USB v3.0
ugen0.1: <(0x1033) XHCI root HUB> at usbus0
uhub0 numa-domain 0 on usbus0
uhub0: <(0x1033) XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
pci0: driver added
panic: VMEM failure: 12
cpuid = 0
time = 1
KDB: stack backtrace:
0xc008000011236590: at vpanic+0x1b0
0xc008000011236640: at panic+0x44
0xc008000011236670: at phyp_iommu_map+0x25c
0xc008000011236760: at bounce_bus_dmamap_complete+0xd4
0xc0080000112367e0: at bus_dmamap_load_mem+0x460
0xc0080000112368b0: at bus_dmamap_load_ccb+0xb0
0xc008000011236920: at vscsi_cam_action+0x420
0xc0080000112369c0: at xpt_run_devq+0x3d8
0xc008000011236ac0: at xpt_action_default+0x9f8
0xc008000011236b60: at scsi_action+0x30
0xc008000011236be0: at xpt_action+0x60
0xc008000011236c50: at probestart+0x7b0
0xc008000011236d50: at xpt_run_allocq+0x25c
0xc008000011236dd0: at probeschedule+0x2d4
0xc008000011237180: at proberegister+0x12c
0xc008000011237200: at cam_periph_alloc+0x790
0xc008000011237380: at scsi_scan_lun+0x284
0xc008000011237600: at xpt_action+0x60
0xc008000011237670: at scsi_scan_bus+0x750
0xc008000011237740: at xpt_scanner_thread+0x1ec
0xc008000011237810: at fork_exit+0xc4
0xc0080000112378b0: at fork_trampoline+0x18
0xc0080000112378e0: at -0x4
KDB: enter: panic
[ thread pid 4 tid 100054 ]
Stopped at kdb_enter+0x70: ori r0, r0, 0x0
db>this is with pseries + power9 qemu vm, 2g ram, single CPU core.
Comment Actions
OK, so we're probably dealing with a pSeries specific issue here then. IIRC the pSeries functionality requires some kind of pSeries-specific IOMMU setup calls to the hyypervisor to bring PCIe online, so it's possible that those were inadvertently bypassed or damaged somehow.
Comment Actions
Just tried the latest diff on power9/pseries qemu, it panics differently now
vdevice0: <POWER Hypervisor Virtual Device Root> on ofwbus0
vscsi0: <POWER Hypervisor Virtual SCSI Bus> irq 16781571 on vdevice0
panic: Assertion m != NULL failed at /home/adrian/work/freebsd/head/freebsd-src-ppc64/sys/powerpc/aim/mmu_radix.c:993
cpuid = 0
time = 1
KDB: stack backtrace:
0xc00800000000ba70: at vpanic+0x1b0
0xc00800000000bb20: at panic+0x44
0xc00800000000bb50: at pmap_nofault+0x3c8
0xc00800000000bc50: at trap_pfault+0xc0
0xc00800000000bd10: at trap+0x12c
0xc00800000000be40: at powerpc_interrupt+0x1cc
0xc00800000000bed0: kernel DSI read trap @ 0 by phyp_iommu_map+0xb4: srr1=0x8000000000001032
r1=0xc00800000000c180 cr=0x22200422 xer=0x20040000 ctr=0xc000000000e9efcc r2=0xc0000000018c1000 sr=0x80000 frame=0xc00800000000bf00
0xc00800000000c180: at 0xc00800000000c20c
0xc00800000000c270: at bounce_bus_dmamap_complete+0x100
0xc00800000000c2f0: at bus_dmamap_load+0x11c
0xc00800000000c370: at vscsi_attach+0x2f8
0xc00800000000c530: at device_attach+0x568
0xc00800000000c610: at bus_attach_children+0x88
0xc00800000000c650: at vdevice_attach+0x1e0
0xc00800000000c6f0: at device_attach+0x568
0xc00800000000c7d0: at bus_generic_new_pass+0x190
0xc00800000000c820: at bus_generic_new_pass+0x104
0xc00800000000c870: at bus_generic_new_pass+0x104
0xc00800000000c8c0: at root_bus_configure+0x6c
0xc00800000000c910: at configure+0x1c
0xc00800000000c940: at mi_startup+0x298
0xc00800000000ca50: at __start+0xc4
KDB: enter: panic
[ thread pid 0 tid 100000 ]
Stopped at kdb_enter+0x70: ori r0, r0, 0x0
db>And with gdb open:
(kgdb) frame 24
#24 0xc000000000e746a0 in IOMMU_MAP (iommu=0xc0000000052ca500, segs=0x0, nsegs=0xc0000000052cb440,
lowaddr=18446744073709551615, highaddr=18446744073709551615, alignment=4096, boundary=0, cookie=0xc0000000054a5020)
at ./iommu_if.h:34
34 rc = ((iommu_map_t *) _m)(iommu, segs, nsegs, lowaddr, highaddr, alignment, boundary, cookie);
(kgdb) print segs
$5 = (bus_dma_segment_t *) 0x0
(kgdb) print nsegs
$6 = (int *) 0xc0000000052cb440
(kgdb) print *nsegs
$7 = 1
(kgdb)and the code path:
(kgdb) frame 27
#27 bus_dmamap_load (dmat=0xc0000000052cb500, map=0xc0000000052cb400, buf=<optimized out>, buflen=<optimized out>,
callback=0xc000000000e9c99c <vscsi_crq_load_cb>, callback_arg=0xc000000005124a80, flags=<optimized out>)
at /home/adrian/work/freebsd/head/freebsd-src-ppc64/sys/kern/subr_bus_dma.c:355
355 segs = _bus_dmamap_complete(dmat, map, NULL, nsegs, error);
(kgdb) frame 26
#26 0xc0000000008f0e50 in _bus_dmamap_complete (dmat=0xc0000000052cb500, map=0xc0000000052cb400, segs=0x0,
nsegs=<optimized out>, error=0) at /home/adrian/work/freebsd/head/freebsd-src-ppc64/sys/powerpc/include/bus_dma.h:165
165 return (tc->impl->map_complete(dmat, map, segs, nsegs, error));
(kgdb) print map
$10 = (bus_dmamap_t) 0xc0000000052cb400
(kgdb) print segs
$11 = (bus_dma_segment_t *) 0x0
(kgdb) print *nsegs
value has been optimized out
(kgdb)*nsegs is 1, the seg array is NULL because of this code in kern/subr_bus_dma.c
segs = _bus_dmamap_complete(dmat, map, NULL, nsegs, error);
if (error)
(*callback)(callback_arg, segs, 0, error);
else
(*callback)(callback_arg, segs, nsegs, 0);So something's not handling that path correct? @jhibbits any idea what it's supposed to be doing here?
Comment Actions
Thanks for the check. I'm able to reproduce and am working on it. On initial glance, it looks like we're calling bounce_bus_dmamap_complete() on a tag that was never created with bounce_bus_dmamap_create(), and I need to find out if that's expected or something we need to trap / ignore.
Comment Actions
Fourth time's the charm? Tested to work on both pSeries/POWER9 and PowerNV/POWER9 -- hopefully POWER8 is similarly functional.
Comment Actions
ok, lemme try this latest one in qemu real quick, hold.
(I also started a port of JUST the busdma code from busdma_machdep.c into busdma_bounce.c, instead of porting the arm/amd code over, just to see what's the delta between a straight port of the two.)
Comment Actions
nope, still no dice
vgapci0: <VGA-compatible display> mem 0x80000000-0x80ffffff,0x81000000-0x81000fff at device 0.0 numa-domain 0 on pci0 vgapci0: Boot video device rtas0: <Run-Time Abstraction Services> on ofwbus0 rtas0: registered as a time-of-day clock, resolution 0.002000s ossl0: <OpenSSL crypto> on nexus0 Timecounter "timebase" frequency 512000000 Hz quality 1000 Event timer "decrementer" frequency 512000000 Hz quality 1000 Timecounters tick every 1.000 msec llan0: link state changed to UP usbus0: 5.0Gbps Super Speed USB v3.0 ugen0.1: <(0x1033) XHCI root HUB> at usbus0 uhub0 numa-domain 0 on usbus0 uhub0: <(0x1033) XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from cd9660:/dev/iso9660/BOOT [ro]... cd0 at vscsi0 bus 0 scbus0 target 0 lun 8200 cd0: <QEMU QEMU CD-ROM 2.5+> Removable CD-ROM SPC-3 SCSI device cd0: 150.000MB/s transfers cd0: 99MB (50930 2048 byte sectors) random: unblocking device. uhub0: 8 ports with 8 removable, self powered panic: Assertion bt != NULL failed at /home/adrian/work/freebsd/head/freebsd-src-ppc64/sys/kern/subr_vmem.c:1609 cpuid = 0 time = 2 KDB: stack backtrace: 0xc008000011291b90: at vpanic+0x1b0 0xc008000011291c40: at panic+0x44 0xc008000011291c70: at vmem_xfree+0x13c 0xc008000011291d00: at phyp_iommu_unmap+0x90 0xc008000011291db0: at bounce_bus_dmamap_unload+0xb0 0xc008000011291e60: at usb_pc_free_mem+0x5c 0xc008000011291e90: at usbd_transfer_unsetup_sub+0x12c 0xc008000011291ed0: at usbd_transfer_unsetup+0x280 0xc008000011291f80: at usbd_ctrl_transfer_setup+0x178 0xc008000011292050: at usbd_do_request_flags+0x28c 0xc008000011292180: at usbd_req_get_desc+0x150 0xc008000011292290: at usbd_req_get_string_desc+0x40 0xc008000011292310: at usb_get_langid+0xf4 0xc0080000112923a0: at usb_alloc_device+0x8ac 0xc0080000112924e0: at uhub_explore+0x98c 0xc0080000112926c0: at usb_bus_explore+0x160 0xc008000011292740: at usb_process+0x178 0xc008000011292810: at fork_exit+0xc4 0xc0080000112928b0: at fork_trampoline+0x18 0xc0080000112928e0: at -0x4 KDB: enter: panic [ thread pid 14 tid 100047 ] Stopped at kdb_enter+0x70: ori r0, r0, 0x0 db> db>
Lemme go finish my initial port and figure out why it panics. It's literally a straight port of the existing ppc code, so anything that isn't working is either something I didn't port right or it's something in the abstraction/common code that's wrong.
Comment Actions
Sigh. Can you share your qemu command line so we're testing using the same peripherals / CPU setup?
The real problem here is we're trying to glue the old idiosyncratic POWER IOMMU code for pSeries onto the common DMA core, since I really wanted to do this in two stages vs. rewriting the DMA code *and* the IOMMU code in one huge patch. From the backtrace, I bet we're unconditionally freeing something; it's not an issue in the DMA core per se since we are most certainly bypassing all the IOMMU logic that would normally be present (and is on e.g. x86).
Comment Actions
FWIW on my POWER9 qemu box, I get a good boot even with the USB controller turned on:
SLOF **********************************************************************
QEMU Starting
Build Date = Nov 6 2024 15:31:54
FW Version = git-ee03aec2c106a699
Press "s" to enter Open Firmware.
Populating /vdevice methods
Populating /vdevice/vty@71000000
Populating /vdevice/nvram@71000001
Populating /vdevice/l-lan@71000002
Populating /vdevice/v-scsi@71000003
SCSI: Looking for devices
8200000000000000 CD-ROM : "QEMU QEMU CD-ROM 2.5+"
Populating /pci@800000020000000
00 0000 (D) : 1033 0194 serial bus [ usb-xhci ]
00 0800 (D) : 1af4 1001 virtio [ block ]
No NVRAM common partition, re-initializing...
Scanning USB
XHCI: Initializing
Using default console: /vdevice/vty@71000000
Detected RAM kernel at 500000 (1dd6826 bytes)
Welcome to Open Firmware
Copyright (c) 2004, 2017 IBM Corporation All rights reserved.
This program and the accompanying materials are made available
under the terms of the BSD License available at
http://www.opensource.org/licenses/bsd-license.php
Booting from memory...
NOT FOUNDGDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
---<<BOOT>>---
Copyright (c) 1992-2025 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 16.0-CURRENT #219 main-n282748-4cd7be3e8186-dirty: Sat Feb 14 12:43:02 CST 2026
root@freebsd-ppc64le-dev:/usr/obj/usr/src/powerpc.powerpc64le/sys/GENERIC64LE powerpc
FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2)
WARNING: WITNESS option enabled, expect reduced performance.
VT: init without driver.
cpu0: IBM POWER9 revision 2.2, 2134.00 MHz
cpu0: Features dc007182<PPC32,PPC64,ALTIVEC,FPU,MMU,SMT,ISNOOP,ARCH205,ARCH206,VSX,TRUELE>
cpu0: Features2 bee00000<ARCH207,DSCR,ISEL,TAR,VCRYPTO,ARCH300,IEEE128,DARN>
real memory = 68651405312 (65471 MB)
avail memory = 66769346560 (63676 MB)
FreeBSD/SMP: Multiprocessor System Detected: 72 CPUs
random: registering fast source PowerISA DARN random number generator
random: fast provider: "PowerISA DARN random number generator"
arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.
random: entropy device external interface
kbd0 at kbdmux0
ofwbus0: <Open Firmware Device Tree> on nexus0
cpulist0: <Open Firmware CPU Group> on ofwbus0
cpu0: <Open Firmware CPU> on cpulist0
xicp0: <External Interrupt Presentation Controller> on ofwbus0
xicp0: Handling CPUs 0-71
rtas0: <Run-Time Abstraction Services> on ofwbus0
rtas0: registered as a time-of-day clock, resolution 0.002000s
pcib0: <RTAS Host-PCI bridge> on ofwbus0
pcib0: [RAJA DEBUG 090.0] In ofw_pcib_init()
pci0: <POWER Hypervisor PCI bus> on pcib0
xhci0: <NEC uPD720200 USB 3.0 controller> mem 0x80000000-0x80003fff irq 4608 at device 0.0 numa-domain 0 on pci0
xhci0: 32 bytes context size, 32-bit DMA
xhci0: xECP capabilities <PROTO,PROTO>
usbus0 numa-domain 0 on xhci0
virtio_pci0: <VirtIO PCI (legacy) Block adapter> port 0x80-0xff mem 0x80010000-0x80010fff,0x210000000000-0x210000003fff irq 4609 at device 1.0 numa-domain 0 on pci0
vtblk0: <VirtIO Block Adapter> numa-domain 0 on virtio_pci0
vtblk0: 65536MB (134217728 512 byte sectors)
vdevice0: <POWER Hypervisor Virtual Device Root> on ofwbus0
uart0: <POWER Hypervisor Virtual Serial Port> irq 16781569 on vdevice0
llan0: <POWER Hypervisor Virtual Ethernet> irq 16781571 on vdevice0
llan0: Ethernet address: 52:54:00:12:34:56
vscsi0: <POWER Hypervisor Virtual SCSI Bus> irq 16781572 on vdevice0
vscsi0: Queue depth 22 commands
ossl0: <OpenSSL crypto> on nexus0
Timecounter "timebase" frequency 512000000 Hz quality 1000
Event timer "decrementer" frequency 512000000 Hz quality 1000
Timecounters tick every 1.000 msec
llan0: link state changed to UP
usbus0: 5.0Gbps Super Speed USB v3.0
ugen0.1: <(0x1033) XHCI root HUB> at usbus0
uhub0 numa-domain 0 on usbus0
uhub0: <(0x1033) XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Launching APs: 56 19 21 59 5 53 65 37 32 67 11 44 10 39 30 66 68 14 4 64 46 54 20 71 36 52 13 70 38 6 7 2 3 60 33 49 22 43 15 18 16 9 69 63 34 29 23 48 35 58 31 57 61 40 50 45 25 27 42 17 8 1 26 28 12 24 41 55 62 47 51
random: unblocking device.
cd0 at vscsi0 bus 0 scbus0 target 0 lun 8200
cd0: <QEMU QEMU CD-ROM 2.5+> Removable CD-ROM SPC-3 SCSI device
cd0: 150.000MB/s transfers
cd0: Attempt to query device size failed: NOT READY, Medium not present
WARNING: WITNESS option enabled, expect reduced performance.
uhub0: 8 ports with 8 removable, self powered
Mounting local filesystems:.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/glib-bootstrap/lib /usr/local/gobject-introspection-bootstrap/lib /usr/local/lib/R/lib /usr/local/lib/compat/pkg /usr/local/lib/gcc13 /usr/local/lib/perl5/5.42/mach/CORE
Setting hostname: freebsd-current.
Setting up harvesting: PURE_DARN,RANDOMDEV,[CALLOUT],[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
Feeding entropy: .
lo0: link state changed to UP
Starting Network: lo0 llan0.
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
llan0: flags=1008802<BROADCAST,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=0
ether 52:54:00:12:34:56
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Starting devd.
Starting Network: llan0.
llan0: flags=1008802<BROADCAST,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=0
ether 52:54:00:12:34:56
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
route: message indicates error: File exists
add host 127.0.0.1: gateway lo0 fib 0: route already in table
route: message indicates error: File exists
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Clearing /tmp (X related).
Creating and/or trimming log files.
Updating motd:.
Updating /var/run/os-release done.
Starting syslogd.
Mounting late filesystems:.
Starting cron.
Performing sanity check on sshd configuration.
Starting sshd.
Starting background file system checks in 60 seconds.
Sat Feb 14 13:33:47 CST 2026
FreeBSD/powerpc (freebsd-current) (ttyu0)
login:Comment Actions
OK, looks like adding the "-usb -usbdevice mouse" option to qemu causes the panic Adrian saw.