In D55132#1261784, @imp wrote:

I shared this with Chuck Silvers and he had some concerns I thought I understood, but that I'm having trouble articulating, so I'm adding him to the review rather than holding things up by puzzling through them and maybe getting it wrong..

For reference, this is the Petitboot commit that added the feature in originally:

Use system defined MAXPHYS value

In D55132#1260924, @jhibbits wrote:

On LP64 platforms (such as POWER9), MAXPHYS is 1MB (see sys/sys/_maxphys.h). But could you instead just make it 64 * maxphys (the tuned variable)?

In D55132#1260360, @ziaee wrote:

The man subsystem has a lot of lore, if you tag manpages when your patch changes a manual, we'll help you with it! It's easier for us if you use -U9999 so we can get the full context in phabricator.

Update manpage and fix tunable registration

Updated to move this to a tunable. From what I can see MAXPHYS is only 128k, and is not available to drivers as a constant. As a result, we actually end up only allocating 8MB by default; if I missed something just let me know and I'll update the MAXPHYS to match...

Understood. What would a sane value be here? 4GB is way too large (as an example, the allocation we actually get on a server that doesn't have a ton of other cards installed is "only" ~1.8GB), would 128MB or 512MB be workable?

Example boot, hotplug, and hot remove of disk device with this patch applied. Note that without this patch the driver won't load, since ~2GB of bounce buffers are allocated and the DMA template tag request fails.

Committed in 1364e7d0921b

In D55095#1259290, @adrian wrote:

oh that's a good catch!

What about regions, pregions, numa_pregions, etc? Are there any other shared variables that need to be unified here?

After overnight stress testing, a combination of D55095 and using the correct DMA tag seems to have completely resolved the AHCI instability I was seeing with this patchset applied.

Thanks for that!

Digging further, at least some of the problem here seems to be from our rather unique memory allocation on POWER9:

OK, thanks for testing. I spent most of the day already going over the codebase trying to figure out what might be going wrong, and all I've come up with thus far is that the 32-bit MMIO window setup and DMA configuration both make no sense. It shouldn't even be working at all on POWER9 using the stock code (without this patch).

! In D54745#1259084, @adrian wrote:
my POWER8 box does, but that may not be good enough?

@adrian Could you give this patch set a thorough test as well? It is not dependent on D54745, I mainly want some additional testing without this on top of D54745 and a merge before going to the next phase of IOMMU enablement. D54745 has some issues with DMA and the AHCI controller that I don't currently understand and will need further investigation.

When testing, does anyone else have access to a SATA controller that does DMA? I'm sporadically seeing the Blackbird's AHCI controller lock up but I don't know if this is a PE freeze, bad DMA, or something completely unrelated (flaky cabling?):

In D15818#1253712, @jhibbits wrote:

In D15818#1253597, @tpearson_raptorengineering.com wrote:

In D15818#1253535, @fuz wrote:

I have tested this patch in an adapted form on ppc64le FreeBSD 15.0 and it crashes early on boot. This is a Raptor Blackbird. @tpearson_raptorengineering.com, any idea?

Since we don't provide Open Firmware interfaces on PowerNV systems, that makes some sense. What we do provide (at least for certain video controllers) is a linear framebuffer available at the address listed in the device tree.

ofwfb isn't Open Firmware-specific, it just depends on the device tree providing the necessary properties, but those properties appear to no longer be provided, and even when launched from the local VGA interface, the "linux,stdout-path" still points to the serial console.

In D15818#1253535, @fuz wrote:

I have tested this patch in an adapted form on ppc64le FreeBSD 15.0 and it crashes early on boot. This is a Raptor Blackbird. @tpearson_raptorengineering.com, any idea?

In D54745#1253492, @adrian wrote:

ok, now that i have power8 hardware up and running, what should i be on the lookout for?
Just the same devinfo/dmesg resource assignment, devices found, etc?

I'm leaning toward @jrtc27's side of things here, but I would like to know more about the issues that were seen that prompted adding this code in the first place. Worst case is that the code being reverted "fixed" the original issue by simply modifying timing / avoiding a race condition, especially given that a.) we tend to have a lot more cores than the other two weak memory ordering architectures and b.) they don't have this kind of flushing behavior yet apparently function normally.

Merged as 201af1a14

In D53801#1229373, @pkubaj wrote:

One note from a ports maintainer perspective: rust currently uses FreeBSD 11 syscalls. Although those are new binaries, they need COMPAT_FREEBSD11, even for powerpc64le (which was introduced in 13.0). There is long-going plan to bump that to 12, but even then we'll still need COMPAT_FREEBSD12.
Another thing: dropping support for running old binaries (as in compiled long ago) is not an issue for me (and probably most users), but then ELFv1 support should probably also be dropped. We switched to ELFv2 in 13.0.

In D53801#1229366, @brooks wrote:

In D53801#1229342, @kib wrote:

After thinking about this code some more, I realized that it probably should be removed altogether. Basically, it is there to allow to run newer binaries (rtld/libsys) on older (pre-AT_ renumbering) kernels. This was done in ~2019, and arguably outlived even the limited usefulness it had at the time of the commit.

I'm not sure this analysis is entirely correct. __elfN(powerpc_copyout_auxargs) will use the old numbering if the binary is old (but not so old it doesn't have an OSREL embedded). What I'm not sure about is if this will happen with a new rtld and old binary or just with an old static binary.

That being said I'd like both bits to go away if possible. @tpearson_raptorengineering.com how important to you think old binary support is to the powerpc64 community?

On second read through, caught a couple of code issues that don't affect functionality. Once those are fixed this looks good.

Looks good to me on a POWER9 Blackbird / powerpc64le.

In D52400#1198743, @markj wrote:

I just went ahead and uploaded an extended diff, I hope that's ok: https://reviews.freebsd.org/D52490

Added aesni and armv8crypto alongside ossl

In D44274#1185321, @pkubaj wrote:

@shawn_anastas.io
FYI, OpenSSL 3.5 has been merged, so you should probably rebase this review. I hope it can finally be pushed before stable/15.

Rebase for 15.0

@jhibbits I'm wondering the same thing. On my side this seems good, any chance we can get it merged? Raptor is still carrying this as a downstream patch in its OPNsense builds. Thanks!