Page MenuHomeFreeBSD

pci: Truncate MMIO windows exceeding 18GB to avoid conflicts
Needs ReviewPublic

Authored by ziaee on Sat, Jun 20, 2:24 AM.

Details

Reviewers
markj
jhb
kevans
Summary

On GCE c4a-metal shapes, the PCI Host Bridge pcib0 fails to attach
because it cannot allocate its large memory window ([0x340000000 -
0x10033fffffff], ~16TB) from the parent bus.

This allocation fails because the 16TB range conflicts with a 64KB early
platform reservation at 20GB (0x500000000 - 0x50000ffff) and overlaps
with system RAM (which starts at 64TB). In FreeBSD, host bridges must
successfully reserve their entire window as a contiguous block from the
parent bus at boot; if this fails, downstream device BARs cannot be
mapped/activated (failing with ENOENT), preventing boot.

Fix this by truncating generic PCI host bridge memory ranges (both MEM
and PMEM) to a maximum physical address of 18GB (0x480000000). This
avoids the 20GB platform conflict and the 64TB system RAM overlap,
allowing the host bridge window to allocate successfully. Downstream
devices (like NVMe at 14.5GB) remain inside the truncated window and
attach successfully.

NOTE: this is definitely a hack, and if there is legitimately a physical device mapped beyond the 18GB region, the bridge will not route MMIO R/W to the device. We need to discuss a more long-term solution to deal with overlapping host bridge windows.

Note: The SRAT, DSDT, and APIC dumps are on freebsd.org/~ziaee/tmp/c4a-*

Authored by: Jasper Tran O'Leary <jtranoleary@google.com>
Sponsored by: Google Cloud

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped
Build Status
Buildable 74042
Build 70925: arc lint + arc unit

Event Timeline

ziaee held this revision as a draft.
ziaee published this revision for review.Sat, Jun 20, 2:28 AM
ziaee changed the visibility from "Public (No Login Required)" to "committers (Project)".
ziaee changed the visibility from "committers (Project)" to "Public (No Login Required)".Tue, Jun 23, 3:55 PM