ok, this one boots fine in p8 pseries and p9 pseries, but p8 powernv it .. just hangs during boot and eventually ends up at OPAL.

In D55315#1266124, @tpearson_raptorengineering.com wrote:

Seeing the same problem with just D54745, D55313, and D55314. I suppose it's possible D54745 could shift things around enough to make the problem appear, but I'm not comfortable merging code into the DMA system when I already know we're violating platform constraints (without D54745 I know for a fact we're allocating bounce buffer in an illegal region and Bad Things via undefined behavior are likely -- just because we get away with it long enough to boot doesn't mean we're not going to get data corruption and lockups later on).

Seeing the same problem with just D54745, D55313, and D55314. I suppose it's possible D54745 could shift things around enough to make the problem appear, but I'm not comfortable merging code into the DMA system when I already know we're violating platform constraints (without D54745 I know for a fact we're allocating bounce buffer in an illegal region and Bad Things via undefined behavior are likely -- just because we get away with it long enough to boot doesn't mean we're not going to get data corruption and lockups later on).

That actually does make sense -- physmem is different on PowerNV vs. pSeries, especially because use of the IOMMU is effectively mandatory on pSeries. I'm not sure you're going to be able to reproduce easily on qemu short of setting up a PowerNV system to test with, which in turn requires a current skiboot build, etc. Additionally, the mpr driver is unique in the entire tree as far as I can tell, I'm not sure the DMA over-allocation issue will be reproducible without an LSI card attached.

ok, does this happen without this domain change? can you just try D55312, D55313, D55314?

All that said, I bet if the mpr DMA issue was fixed the problem introduced in the port would at least be masked. I'm unclear what the "correct" behavior should be when we run out of bounce buffers; with the stock kernel, it looks like ahci and usb are simply denied more buffers so they fall back to a DMA-free mode, whereas with the patch series applied they're allocated too small of a bounce region and just lock up.

This is why I was saying we should probably find the fault in my patchset, and fix that, along with whatever is going wrong with the DMA tag setup where the mpr driver is allocating too much bounce memory. There's something different in the DMA core vs. what the old powerpc DMA code is trying to do, and whatever it is is quite subtle.

Yeah, here's the boot from a stock kernel:

oh interesting. is this different from straight -head? If it is then i need to fix it. If it isn't then it's expected behaviour!

Unfortunately this fails to boot on POWER9/PowerNV with the following patch series:

refactor

refactor

refactor

refactor

migrate this to /just/ do rwlock migration, not the iteration stuff.

compile tested

rw_init_flags() -> rw_init()

OK, looks like adding the "-usb -usbdevice mouse" option to qemu causes the panic Adrian saw.

FWIW on my POWER9 qemu box, I get a good boot even with the USB controller turned on:

Sigh. Can you share your qemu command line so we're testing using the same peripherals / CPU setup?

nope, still no dice

ok, lemme try this latest one in qemu real quick, hold.

Fourth time's the charm? Tested to work on both pSeries/POWER9 and PowerNV/POWER9 -- hopefully POWER8 is similarly functional.

Thanks for the check. I'm able to reproduce and am working on it. On initial glance, it looks like we're calling bounce_bus_dmamap_complete() on a tag that was never created with bounce_bus_dmamap_create(), and I need to find out if that's expected or something we need to trap / ignore.

Just tried the latest diff on power9/pseries qemu, it panics differently now

Fix failure on PowerNV platforms introduced in last update

Now with 100% less pSeries IOMMU errors!

OK, so we're probably dealing with a pSeries specific issue here then. IIRC the pSeries functionality requires some kind of pSeries-specific IOMMU setup calls to the hyypervisor to bring PCIe online, so it's possible that those were inadvertently bypassed or damaged somehow.

ok i thought this was OK on power9 but it seems not:

I can't imagine why this would matter offhand, but a key difference is POWER8 is HPT, POWER9 is Radix. You've already ruled out the pSeries/PowerNV question by checking on POWER9/pSeries, and both platforms are supposed to be IODA2 compliant, so the MMU mode is about all I can think of as a major difference.

pvo->pvo_pte.slot is initialized to a hash mod moea64_pteg_count, then updated by the return of the H_ENTER hypercall. There are two possibilities here:

• Some kind of weird buffer overflow that scribbled onto the PVO
• A bug in Qemu

ok, tried with 8g in the VM instead of 2g, same issue.

and from justin on irc

This boots fine in qemu-system-ppc64 pseries9 + power9, but in qemu-system-ppc64 pseries-8.0 + power8, it immediately panics on boot with

oh crap i missed testing this; lemme test this on -HEAD today on the G5 and POWER8.