OK, sounds good, thanks for testing!

Just make the page table size really small. There's a spill counter in the statistics; when it gets small enough, you should see it moving. If spill-handling doesn't work, and you have spills, everything breaks fast.

Does this break PTE spills? This code path historically was involved in re-adding PVOs to the hashed page table on faults if they got spilled.

That's a fair point. I've used them to assess hash-table spill rates when performance is not great, but not for a long time. We could also just delete them.

Yeah, counter(9) seems like the actual solution here.

Looks good to me; do you know how much this actually helps?

Should we panic if freeze_timebase somehow doesn't get set? Or, alternately, fall back on the old lame thing?

Longer-term, it might be better to avoid a proliferation of special cases like this to set LOADER_MSDOS_SUPPORT for the U-boot loader and make loader flexible enough to find the kernel etc. in the root rather than /boot. But this is fine for now.

Aside from one inline note, this looks OK for non-SPE.

Can we make the check the presence of a PHB4 node in the device tree rather than an #ifdef? It might also be best to do this in platform_powernv.c, but that's a bit larger of a project.

In D16794#361563, @jhb wrote:

On closer examination, the loader doesn't actually modify the kernel or modules in place at all. Instead, this routine is used to apply relocations on copies of data. So for example, the code to parse module metadata copies individual records out of the kernel into a local variable on the stack, then asks the relocation to be applied to that local copy. This is a really good reason to honor the the 'data' and 'len' bounds as otherwise it allows writing to random crap on the stack. Also, we do explicitly ignore all relocations for the kernel itself, so it must be some relocation in tcp_rack.ko that wasn't handled before that is being handled now perhaps. I would have expected that to trigger a warning if so in the existing code though for an "unhandled relocation type".

In D16794#361523, @jhb wrote:

I believe since ppc is using Elf_Rela and that all of the relocations calculated absolute values to store at *where or *hwhere (vs using += or the like), the loader relocations just get overwritten by the kernel. I think when using Elf_Rel you might use the original value of *where as the addend in which case multiple passes of relocations is indeed destructive, but I don't think it is for Elf_Rela.

I just looked at the actual patch. The kernel sometimes (almost always on ppc64, less often on ppc32) relocates itself again post-loader to an address that loader does not know, so I think this does not solve the actual problem, or at least solves it in only a subset of cases. Processing these relocations in the kernel would solve the issue in all cases.

In D16794#357383, @jhb wrote:

I added Justin and Nathan (who I meant to include from the start).
I believe PPC kernels are relocatable. I think they use ET_DYN instead of ET_EXEC (and there is a pending PR to fix that issue in libkvm as libkvm doesn't work on it currently as it only expects ET_EXEC for a kernel). Justin and/or Nathan can confirm.

Looks OK to me. One question: Do we want the various instances of fdt_check_header() in our code to be spelled FDT_RO_PROBE() now?

Since I'm not familiar with the clock code, where is EXT_RESOURCES defined? Can we just use that code unconditionally?

Assuming you have tested this and it works, looks great.

Looks good to me.

Approved if you fix the #if 0 bit.

Don't you want strcmp() == 0?

It's mildly stupid, since the universe of pci* isn't big, but I think I would prefer a check for "pci" || "pciex" to just checking the first three characters.

In D15229#333064, @imp wrote:

In D15229#333057, @nwhitehorn wrote:

In D15229#333038, @imp wrote:

The final option would be to do a variation on the prior option, but institutionalize it in DEVICE_ATTACH_LATE. It would be called just after interrupts were enabled for deferred work like this. It would cover the vast majority of driver independencies by giving a 'last' point. At that point, you return an error if you still weren't ready, and we'd loop while the number of devices returning an error was declining. It could largely replace config_intrhook.

I think this would work reasonably well for the general case here; the other suggestions would solve the case of smu(4), but there are other, simpler, ways to do that.
That said, I'm not sure what the advantage of a DEVICE_ATTACH_LATE() is over the mechanism originally proposed here or something like it except that it, as a side effect, delays until after interrupts -- which might not even be wanted, for example if the device in question is an interrupt controller.

If the device is an interrupt controller, you need to put it on a different pass. There's no way around that. But this isn't an interrupt controller, and interrupt controllers are special beasts that already have special handling in many places. Having a delayed attach for them is likely completely unacceptable so none of the proposals is appropriate for them.

In D15229#333038, @imp wrote:

The final option would be to do a variation on the prior option, but institutionalize it in DEVICE_ATTACH_LATE. It would be called just after interrupts were enabled for deferred work like this. It would cover the vast majority of driver independencies by giving a 'last' point. At that point, you return an error if you still weren't ready, and we'd loop while the number of devices returning an error was declining. It could largely replace config_intrhook.

It's been another couple weeks. Any objections or approvals?

This looks good to me, though I haven't tested on P8.

On POWER9, I think the lock is unnecessary (the ISA spec doesn't mention it).

Any other thoughts? I'd like to get this in soon since it is blocking other things.

Fix style and spelling issues in man pages.

Probe for pending devices at the end of each pass and after modules loaded. Add documentation.

See the comment on line 467 about powernv_smp_ap_init(). Otherwise, looks good.

Approved if you also install it to EXC_HEA.

In D15482#326710, @breno.leitao_gmail.com wrote:

1. Is there a document describing this that we can reference?

I was not able to find any documentation. Looking at the web, I found a patch[1] that documents it quickly in Linux, but it seems it was never accepted. I will ask internally.

This looks good, even if the deice tree situation is a bit lamentable. Two questions:

1. Is there a document describing this that we can reference?
2. If this is just part of the PowerNV platform, maybe the code belongs in platform_powernv.c instead?

Thanks -- I'll tweak the implementation to handle KLDs properly and write a man page, then update this diff.

Any further comments? I'd really like to get this, or something like it, in.

We should do the same thing (check fdtbootcpu rather than PIR) on PowerNV too. Thanks for the work on this patch!

Looks good to me. I haven't tested it, of course -- is this working well for you?

This is an interesting problem that I need to think about -- thanks for identifying it! I think this particular patch is not the right one, so let's put a pin in this for a couple days.

The CPU is stored in the device tree in one of two ways:

1. As an ihandle "cpu" in /chosen
2. The "fdtbootcpu" integer property from /chosen

In D15229#321549, @jhb wrote:

(For future reference, a diff with context is nicer.)
(BTW, while reviewing this I found the device_register() "gem" which is a bit of a hack that snuck in with iflib and should be replaced with an iflib_if.m and a dedicated IFLIB_REGISTER rather than looking at magic numbers in the first 4 bytes of a pointer, etc. That's just ridiculous.)
I assume you're waiting on docs until the design is ok'd (which is sensible), but once that is settled DEVICE_ATTACH.9 probably needs extending to document this.

Simplify logic dealing with pending attachments.

We should fix that, too, though. This code assumes that we are booting on CPU 0 -- the same assumption could continue as a stopgap until we figure out a reliable way to evaluate the boot processor.

Is there a reason not to just adopt the PowerNV version of this code, which already does the right thing?

This looks great, with one nit about the way you determine whether VSX is available.

I think there is also a subtlety for uncaught signals if the previous trap was a kernel fault, since you then end up in the wrong branch of the kernel/user if in trap() and probably crash.

Sure, it will deliver the signal. But the value of MSR when control is returned to userland may be wrong (it may even have MSR[PR] unset and return in supervisor mode if the previous trap on this CPU was in the kernel!) and it will resume execution in the wrong place if the signal is caught and handled.

I'm looking at this again. Does this actually work? HEA interrupts set HSRR0/HSSR1, which we don't save, so I think this will jump back to some random other memory location (whatever happened to be in SRR0).

Use label arithmetic rather than instruction counting, which is much more robust.

Looks good. Thanks for doing this!

In D14499#306484, @breno.leitao_gmail.com wrote:

In D14499#306478, @nwhitehorn wrote:

In D14499#306477, @breno.leitao_gmail.com wrote:

Hi Nathan,
With the current patch, I was able to test it properly on my pseries VM, but I didn't use usefdt=1, mainly because I didn't know where to use it. Should I need to recompile the loader to use it, or, is it a parameter somewhere?

And it was working fine?

If I do not set the usefdt=1, it works fine. If I set it, it break in the early start.

OK set usefdt=1
OK boot
Booting...
Kernel entry at 0x102670 ...
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #102 66160e2fc87(nathan): Tue Mar  6 16:19:44 CET 2018
    root@free8:/usr/obj/root/kernel/freebsd/powerpc.powerpc64/sys/GENERIC64 powerpc
gcc version 4.2.1 20070831 patched [FreeBSD]
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: Running on a broken hypervisor that does not support mandatory H_CLEAR_MOD and H_CLEAR_REF hypercalls. Performance will be suboptimal.
[ thread pid 0 tid 0 ]
Stopped at      .trap+0x40:     stdu    r1, r1, 0xfe40
db> bt
Tracing pid 0 tid 0 td 0x14eb5a0
0xe000000000001190: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe000000000001440 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000001440: at 0x38000000901f0020
0xe0000000000014e0: at .trap_pfault+0xb0
0xe000000000001590: at .trap+0xef0

Minor cleanups. Fix Book-E.