As a matter of curiosity, did something specific prompt having this from of PCI Configuration Space Access added? FreeBSD has been able to boot the RPi4B's for a long time, even via EDK2. So the test case does not seem to be a driving use case. Are there other known examples of this form of PCI Configuration Space Access needing-to-be/being-able-to-be used?

In D34697#808973, @imp wrote:

. . .
arm would have trouble here, but we no longer make compat libs for arm.
. . .

In D34697#808973, @imp wrote:

. . .
Also arm64 doesn't support -m32, . . .

A list of the FreeBSD servers is maintained at: https://github.com/bdrewery/pkg-status.freebsd.org/blob/master/servers.txt

Going to https://reviews.freebsd.org/D37419 and clicking on the link shows the log just fine for me. (I just tried it again.) ampere2.nyi.freebsd.org is a well known FreeBSD builder machine.

Looks to me like https://developer.arm.com/documentation/100336/0002/aba1429015078674 has the description of the GICD Distributor registers by offset from the Physical Base Address:

I did the 3 tests via booting an installed FreeBSD-14.0-CURRENT-arm-armv7-GENERICSD-20230302-005cca8361a4-261233.img.xz (from today) and they all worked fine.

I did the 3 tests via booting an installed FreeBSD-14.0-CURRENT-arm-armv7-GENERICSD-20230302-005cca8361a4-261233.img.xz (from today) and they all worked fine.

Note: The following was done with a 4 GiByte RPI4B in order to avoid having a 8 GiByte RPi4B used with the U-Boot 2023.01 that FreeBSD is using: 2023.01 fails to load FreeBSD's EFI loader for a 8 GiByte RPi4B, reporting a message that is a misnomer for the actual internal problem.

Retesting my 3 armv7 tests based on use of the updates https://reviews.freebsd.org/D38696?id=117780 and https://reviews.freebsd.org/D38698?id=117779 worked fine.

Retesting my 3 armv7 tests based on use of the updates https://reviews.freebsd.org/D38696?id=117780 and https://reviews.freebsd.org/D38698?id=117779 worked fine.

With the savectx blne -> bl change, D38696.diff, and D38698.diff all applied, all
the activities with all 3 of my small example programs for the armv7 floating
point related problems look to be working just fine: no KASSERT's ( simple_dbl.c
and dbl_and_ull_via_async.cpp activities) and no odd values showing up in a
thread ( dbl_and_ull_multithread.cpp runs for minutes at a time).

With the savectx blne -> bl change, D38696.diff, and D38698.diff all applied, all
the activities with all 3 of my small example programs for the armv7 floating
point related problems look to be working just fine: no KASSERT's ( simple_dbl.c
and dbl_and_ull_via_async.cpp activities) and no odd values showing up in a
thread ( dbl_and_ull_multithread.cpp runs for minutes at a time).

Do not take the following as indicating anything is necessarily wrong. It is more about my ignorance in the subjects involved.

On the lists, I've provided 3 small C/C++ programs that when used,
sometimes mixed with some use of gdb or lldb, get assert panics
from a debug kernel or corrupted floating point datanow. Look at the
new text in the messages indicated below to see the programs with
instructions in the comments for how to build and test:

For a failure bisected (or analogous) to have started with this change, see: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=268995

Looks good to me.

In D37462#860547, @karels wrote:

I found these over the weekend too, testing the snapshot images. Fixed by 4c8a257810a6.

Given a "Don't" -> "Do not" workaround after dd but before booting in order to try to see if the rest worked . . .

I tried the new main [so: 14] snapshot, dd'd to a USB3 SSD and booted:
snaphot: FreeBSD-14.0-CURRENT-arm64-aarch64-RPI-20221224-c89209c674f2-259842.img
so: FreeBSD 14.0-CURRENT #0 main-n259842-c89209c674f2: Sat Dec 24 05:52:28 UTC 2022
Result (from the serial console capture):

Starting file system checks:
/dev/ufs/rootfs: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ufs/rootfs: clean, 599506 free (242 frags, 74908 blocks, 0.0% fragmentation)
/etc/rc.d/growfs: 203: Syntax error: "(" unexpected (expecting "}")

FYI: After updating the kernel, my test of building the problematical devel/nasm commit via USE_TMPFS=all in poudriere-devel on/for amd64 main worked just fine. (No runaway process creation, unlike before.)

In D35962#856931, @imp wrote:

. . .

This divergence is quite old. We've been forever fixing problems with amd64 vs x86_64 in builds as well, and that dates back 15 years...

FYI:

# sysctl hw.machine
hw.machine: arm64

In D35962#856587, @pkubaj wrote:

Well, Poudriere should use uname -p. Using -p doesn't make sense. On powerpc*, it prints just powerpc on all 4 platforms and is totally misleading on 3 of them. On armv{6,7}, it's imprecise (do we deal with arm as in the removed ARMv5, armeb, armv6 or maybe armv7?), although not as incorrect as on powerpc*. On riscv64, it makes other software think it's actually riscv32, which FreeBSD doesn't even support. I have posted previously links here to software and ugly workarounds that I needed to use.

In D35962#856540, @imp wrote:

[Editied to correct inaccurate recall of old details.]

Does this change the status of the man page content for uname? :

The options are as follows:

In D37462#854293, @cperciva wrote:

Another question: Do you want to respect vm.swap_maxpages? Or just allocate the swap space and let the kernel potentially issue a warning about the device being too large and not being fully utilized?

In exploring testing a "C0T" RPi4B 8 GiByte (no odd DMA address limits) and retesting the old "B0T" one (has odd DMA address limits), I've learned that the maximum-address view is directly available via what ACPI's _DMA provides. I've updated the code and the comments for such. This is a recent change, not one that I've used for long at this point. (The prior coding has been in sustained use for a ZFS bectl context but was poorer based on what I now understand.)

". . . and the existing root is less than 40% of the disk" What if other partitions/slices than root's are taking up space as well? Do you need to use an available free space's size that is sufficiently large instead?

Is the script used with the likes of:

"other values indicate a swap size in sectors": so from media to media, with different sector sizes, the same figure means differing sizes?

What of the i386 failure example that https://lists.freebsd.org/archives/dev-commits-src-main/2022-October/010362.html reports:

ln -s efi ${DESTDIR}/boot/msdos

The update looks right to me.

When you divide and multiply by constants, the compiler will optimize this. Have you looked at the resulting assembly code using -O2 ?

Try to compile the new code and look at the assembly.

I made the same comments in D36857 . . .

Something to consider? :

FYI: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262554#c9 indicates that a home-grown variant is still in use in sys/geom/part/g_part.c . It disallowed /dev/ prefix usage in a gpart add for what that submittal was reporting.

I have updated to the system being based on (line split for readability):

# uname -apKU
FreeBSD amd64_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #30
main-n252475-e76c0108990b-dirty: Sat Jan 15 21:18:14 PST 2022
root@amd64_ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG 
amd64 amd64 1400047 1400047

so it has the committed material (and does not have my prior personal workaround). Based on also using:

env SH_DISABLE_VFORK= ASAN_OPTIONS=detect_container_overflow=0 chroot /usr/obj/DESTDIRs/main-amd64-xSAN-chroot/

for the (also updated) chroot area, I'm now getting:

===> Summary
Results read from /usr/home/root/.kyua/store/results.usr_tests.20220116-063311-263708.db
Test cases: 6840 total, 894 skipped, 29 expected failures, 142 broken, 303 failed
Total time: 3132.706s

with no fread.c reports. In my first runs, before I'd learned some about what to control,
there were over 1200 failures reported (not limited to fread.c ones).

The code looks good to me and is strongly related to what I've been recently using personally for chrooting
into a WITH_UBSAN= WITH_ASAN= world and running the kyua tests, avoiding the particular UBSAN stderr
outputs from the NULL+0 messing up 600+ test results.

Looks to me like limits.lowaddr is still one too large, and so identifying the start of the wrong page instead of the end of the correct page by the code's overall criteria. As I remember, I still use limits.lowaddr-1 where I originally indicated in order to have the RPi4B have huge copy-then-diff operations work. (Though I've not been directly testing that for some time. It has been almost a year since I added the "-1" notes.)

I no longer have access to any operational old-PowerMacs. (And never had access to any other example types of PowerPC/Power hardware.) As such, I'll no longer be involved in powerpc64 or 32-bit powerpc activities.

Thanks for the patch, even if FreeBSD did not adopt it.

FYI: G5 2-sockets/2-cores-each basic info follows.

I'll remind of the old issue that the pre-exisiting code's use of platform_smp_timebase_sync does
not follow a synchronize everything at once protocol everywhere, sometimes using the ap
argument being 1 without any matching ap being 0. This invalidated one of jhibbits patches in
the past. (I'm writing from vague memory here. So be careful from reading too much detail.)

In case it helps for some G4 dual socket contexts (captured while the machine happens
to be up and accessible). Turns out each of the cpus has a timebase-enable listed, but
they are numerically equal on the machine in question.

https://ci.freebsd.org/job/FreeBSD-main-i386-build/19172/console shows:

FYI: Both of the "PowerMac11,2 with 2 sockets, 2 cores each" have died of cooling system failures. (I can sometimes boot one, but it is problematical to do much with without overheating.) So, now the only G5 I now have access to for general testing is a "2 socket/1-core-each PowerMac7,2" . I still have access to the two PowerMac3,6 with "2 sockets, 1 core each" G4 examples. (But there are other major FreeBSD kernel issues for 32-bit PowerMacs, even for single socket ones: it gradually clears some user-process pages to zero.)

This made the only amd64 FreeBSD system that I have access to display a blank
screen in the loader. It is a ThreadRipper 1950X system.

For your side note: A comparison/contrast. Your linked material shows:

pcib0: <BCM2838-compatible PCI-express controller> mem 0x7d500000-0x7d50930f irq 70,71 on simplebus2
pcib0: hardware identifies as revision 0x304.
pci1: <PCI bus> on pcib0
pcib1: <PCI-PCI bridge> irq 81 at device 0.0 on pci1
pci2: <PCI bus> on pcib1
bcm_xhci0: <VL805 USB 3.0 controller (on the Raspberry Pi 4b)> irq 82 at device 0.0 on pci2
bcm_xhci0: 32 bytes context size, 64-bit DMA
usbus0 on bcm_xhci0
pci0: <PCI bus> on pcib0
pci0: failed to allocate bus number
device_attach: pci0 attach returned 6

But what I get from my boot (indirectly via a microsd card and use of modern released-firmware's .dtb) is:

pcib0: <BCM2838-compatible PCI-express controller> mem 0x7d500000-0x7d50930f irq 70,71 on simplebus2
pcib0: hardware identifies as revision 0x304.
pci0: <PCI bus> on pcib0
pcib1: <PCI-PCI bridge> irq 81 at device 0.0 on pci0
pci1: <PCI bus> on pcib1
bcm_xhci0: <VL805 USB 3.0 controller (on the Raspberry Pi 4b)> irq 82 at device 0.0 on pci1
bcm_xhci0: 32 bytes context size, 64-bit DMA
usbus0 on bcm_xhci0

So the first differences are:
Your context: "pci1: <PCI bus> on pcib0" (no pci0 shown at this stage)
My context has instead: "pci0: <PCI bus> on pcib0"

Ahh, gradual understanding on my part. Thanks.

In D26853#599113, @maciphone2_googlemail.com wrote:

In D26853#598984, @markmi_dsl-only.net wrote:

But the FreeBSD kernel does need to deal with the VL805-firmware issue in that context and does so without the .dtb file change.
So why would u-boot need the change when the kernel does not? (Again: out of my depth here.) Are you saying that the FreeBSD kernell is doing something wrong >currently and needs to be fixed once the .dtb adjustment is in place?

the DeviceTree is loaded before u-boot(and the kernel) , so in this case it can trigger the firmware-load.
It's an RPI4 hardware quirk on newer board-revisions.

In D26853#598842, @maciphone2_googlemail.com wrote:

In D26853#598685, @markmi_dsl-only.net wrote:

reset = "/soc/firmware/reset";

. . .

https://github.com/u-boot/u-boot/commit/f676eb217bdff3bd734a42c8f9bbc58c9100055c

on newer board revisions a dedicated chip which holded the VL805-firmware was removed,
so u-boot needs to be instructed to reset the controller to load the firmware from the VideoCore in a very early stage(before the OS itself
initializes the driver).that early stage is when bcm2711-rpi-4-b.dtb is loaded.

. . .

of course u-boot wouldn`t' need an early stage xhci-reset when it boots from an SD-card

Ignoring what appears to be general, systematic phandle and phy-handle numbering differences, the following appears to be what this has changed vs. a fairly modern firmware's .dtb file. I'll start with the completely new additions it made:

May be the following notes will be useful to someone with an appropriate background . . .

I've tested head -r363932 under uefi/ACPI v1.20 with the 3072 limit disabled and it failed the huge-file duplicate and diff/cmp test:

# cp -aRx /usr/obj/clang-armv7-on-aarch64.tar /usr/obj/clang-armv7-on-aarch64.alt_tar
# diff /usr/obj/clang-armv7-on-aarch64.tar /usr/obj/clang-armv7-on-aarch64.alt_tar
Binary files /usr/obj/clang-armv7-on-aarch64.tar and /usr/obj/clang-armv7-on-aarch64.alt_tar differ
# cmp -l /usr/obj/clang-armv7-on-aarch64.tar /usr/obj/clang-armv7-on-aarch64.alt_tar | head -30
2633269249   3   0
2633269251   3   0
2633269252  55   0
2633269253   6   0
2633269254  21   0
2633269255 227   0
2633269256   1   0
2633269257 135   0
2633269258 336   0
2633269259  22 140
2633269260   0 100
2633269261 346   0
2633269262 353   0
2633269265 227   0
2633269266   1 160
2633269267 170 140
2633269268 336 100
2633269269  22   0
2633269271 362   0
2633269272 353   0
2633269275 227   0
2633269276   1   0
2633269277 225   0
2633269278 336   0
2633269279  22   0
2633269281 376   0
2633269282 353   1
2633269285   0   1
2633269289   0 223
2633269290   0 321

For reference:

# ls -ldT /usr/obj/clang-armv7-on-aarch64*
-rw-r--r--  1 root  wheel  11570948096 Jul 18 18:32:37 2020 /usr/obj/clang-armv7-on-aarch64.alt_tar
-rw-r--r--  1 root  wheel  11570948096 Jul 18 18:32:37 2020 /usr/obj/clang-armv7-on-aarch64.tar

(So the over 10 GiByte original file is significantly larger than the 8 GiByte RAM, although I've had large but much smaller files fail as well.)

As I have just reported on the arm list, the checked-in patch fails the huge file duplicate and diff/cmp test. xhci
DMA is still a problem.

Given what I've seen about the 5.8 mainline Linux kernel status for the RPi4 and that Fedora just branched 33 from Rawhide (for a late Oct. release), I decided to try installing a Fedora 33 branch server and UEFI/ACPI v1.20 and doing a "yum update" on the booted result. (USB3 SSD based, no microsd card involved.) That ended up with a 5.8.7-300.fc33.aarch64 kernel and it sees all 8GiB of RAM. I've tried the huge file copy and diff type of test and it has not failed yet. (The files are each bigger than the RAM.) (I've not noticed other issues but that likely does not imply much.) For reference, the boot reported the following "Zone ranges":

[    0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x00000001ffffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x1fefec8c0-0x1ff002fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000003fffffff]
[    0.000000]   DMA32    [mem 0x0000000040000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000001ffffffff]
[    0.000000]   Device   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x00000000001fffff]
[    0.000000]   node   0: [mem 0x0000000000200000-0x00000000337d7fff]
[    0.000000]   node   0: [mem 0x00000000337d8000-0x000000003387ffff]
[    0.000000]   node   0: [mem 0x0000000033880000-0x000000003390ffff]
[    0.000000]   node   0: [mem 0x0000000033910000-0x000000003398ffff]
[    0.000000]   node   0: [mem 0x0000000033990000-0x00000000339affff]
[    0.000000]   node   0: [mem 0x00000000339b0000-0x0000000033a2ffff]
[    0.000000]   node   0: [mem 0x0000000033a30000-0x0000000036efffff]
[    0.000000]   node   0: [mem 0x0000000036f00000-0x00000000372dffff]
[    0.000000]   node   0: [mem 0x00000000372e0000-0x000000003b2fffff]
[    0.000000]   node   0: [mem 0x0000000040000000-0x00000000fbffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x00000001ffffffff]
[    0.000000] Zeroed struct page in unavailable ranges: 552 pages
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000001ffffffff]

What OpenBSD did for the size of the DMA area for XHCI looks to be unusual (relative to NetBSD and Linux). But the next-to-last "node 0:" line above does show not using the whole DMA32 range. (I do not now why they had it do that.)

FYI: Something that I do not see represented in the NetBSD debug information is that DMA channel 11 (of DMA4 engine type) is the only one explicitly noted in rpi_DATA_2711_1p0.pdf as "additionally able to access the PCIe interface".

In D26344#585901, @crowston_protonmail.com wrote:

. . .

The 0x3c000000 limit is taken from OpenBSD.

My understanding is that different devices have different DMA limits. XHCI has 3 GiB but others have 1 GiB. There is not one DMA controller enforcing one limit common to all contexts but multiple separate limits. Using the smallest limit for all devices may well be an alternative but that is not what the ACPI tables describe (from what I read on the lists).

This code leads to an OverDrive 1000 getting the following error.

I have removed my prior comments: they traced to a different issue that, when worked around, got rid of the behavior that I had reported. Sorry for the noise.

In D23376#522931, @jhibbits wrote:

. . .
You can look in Linux's arch/powerpc/platform/powermac/smp.c, and arch/powerpc/kernel/smp-tbsync.c for how Linux does it. Their generic tbsync handshake is similar to yours.

In D23376#522809, @jhibbits wrote:

. . .
I do intend to shortly (next few days) clean up the one additional platform_timebase_sync() user (cpu_sleep()) to not use it, and handle it directly, since the code in cpu_sleep() is actually mpc74xx-specific, it's not generic at all. So once that's made explicit, a proper rendezvous sync should be done in platform_powermac.c instead.
. . .

In D23376#522809, @jhibbits wrote:

What we really want to achieve is a better timebase sync mechanism. This can only be achieved by disabling the timebase, rendezvousing, and re-enabling it. Adding a drift into the generic code for a specific platform is not the correct solution.

In D23729#521426, @markj wrote:

. . .> Ah, that clarifies things. I was looking for a problem in r357548.

In D23729#521312, @markj wrote:

Can you explain why this makes a difference, at least for slbt_zone? I think I am missing something.