Looks good to me.

Approved if you fix the #if 0 bit.

Don't you want strcmp() == 0?

It's mildly stupid, since the universe of pci* isn't big, but I think I would prefer a check for "pci" || "pciex" to just checking the first three characters.

In D15229#333064, @imp wrote:

In D15229#333057, @nwhitehorn wrote:

In D15229#333038, @imp wrote:

The final option would be to do a variation on the prior option, but institutionalize it in DEVICE_ATTACH_LATE. It would be called just after interrupts were enabled for deferred work like this. It would cover the vast majority of driver independencies by giving a 'last' point. At that point, you return an error if you still weren't ready, and we'd loop while the number of devices returning an error was declining. It could largely replace config_intrhook.

I think this would work reasonably well for the general case here; the other suggestions would solve the case of smu(4), but there are other, simpler, ways to do that.

That said, I'm not sure what the advantage of a DEVICE_ATTACH_LATE() is over the mechanism originally proposed here or something like it except that it, as a side effect, delays until after interrupts -- which might not even be wanted, for example if the device in question is an interrupt controller.

If the device is an interrupt controller, you need to put it on a different pass. There's no way around that. But this isn't an interrupt controller, and interrupt controllers are special beasts that already have special handling in many places. Having a delayed attach for them is likely completely unacceptable so none of the proposals is appropriate for them.

In D15229#333038, @imp wrote:

The final option would be to do a variation on the prior option, but institutionalize it in DEVICE_ATTACH_LATE. It would be called just after interrupts were enabled for deferred work like this. It would cover the vast majority of driver independencies by giving a 'last' point. At that point, you return an error if you still weren't ready, and we'd loop while the number of devices returning an error was declining. It could largely replace config_intrhook.

It's been another couple weeks. Any objections or approvals?

This looks good to me, though I haven't tested on P8.

On POWER9, I think the lock is unnecessary (the ISA spec doesn't mention it).

Any other thoughts? I'd like to get this in soon since it is blocking other things.

Fix style and spelling issues in man pages.

Probe for pending devices at the end of each pass and after modules loaded. Add documentation.

See the comment on line 467 about powernv_smp_ap_init(). Otherwise, looks good.

Approved if you also install it to EXC_HEA.

In D15482#326710, @breno.leitao_gmail.com wrote:

1. Is there a document describing this that we can reference?

I was not able to find any documentation. Looking at the web, I found a patch[1] that documents it quickly in Linux, but it seems it was never accepted. I will ask internally.

This looks good, even if the deice tree situation is a bit lamentable. Two questions:

1. Is there a document describing this that we can reference?
2. If this is just part of the PowerNV platform, maybe the code belongs in platform_powernv.c instead?

Thanks -- I'll tweak the implementation to handle KLDs properly and write a man page, then update this diff.

Any further comments? I'd really like to get this, or something like it, in.

We should do the same thing (check fdtbootcpu rather than PIR) on PowerNV too. Thanks for the work on this patch!

Looks good to me. I haven't tested it, of course -- is this working well for you?

This is an interesting problem that I need to think about -- thanks for identifying it! I think this particular patch is not the right one, so let's put a pin in this for a couple days.

The CPU is stored in the device tree in one of two ways:

1. As an ihandle "cpu" in /chosen
2. The "fdtbootcpu" integer property from /chosen

In D15229#321549, @jhb wrote:

(For future reference, a diff with context is nicer.)

(BTW, while reviewing this I found the device_register() "gem" which is a bit of a hack that snuck in with iflib and should be replaced with an iflib_if.m and a dedicated IFLIB_REGISTER rather than looking at magic numbers in the first 4 bytes of a pointer, etc. That's just ridiculous.)

I assume you're waiting on docs until the design is ok'd (which is sensible), but once that is settled DEVICE_ATTACH.9 probably needs extending to document this.

Simplify logic dealing with pending attachments.

We should fix that, too, though. This code assumes that we are booting on CPU 0 -- the same assumption could continue as a stopgap until we figure out a reliable way to evaluate the boot processor.

Is there a reason not to just adopt the PowerNV version of this code, which already does the right thing?

This looks great, with one nit about the way you determine whether VSX is available.

I think there is also a subtlety for uncaught signals if the previous trap was a kernel fault, since you then end up in the wrong branch of the kernel/user if in trap() and probably crash.

Sure, it will deliver the signal. But the value of MSR when control is returned to userland may be wrong (it may even have MSR[PR] unset and return in supervisor mode if the previous trap on this CPU was in the kernel!) and it will resume execution in the wrong place if the signal is caught and handled.

I'm looking at this again. Does this actually work? HEA interrupts set HSRR0/HSSR1, which we don't save, so I think this will jump back to some random other memory location (whatever happened to be in SRR0).

Use label arithmetic rather than instruction counting, which is much more robust.

Looks good. Thanks for doing this!

In D14499#306484, @breno.leitao_gmail.com wrote:

In D14499#306478, @nwhitehorn wrote:

In D14499#306477, @breno.leitao_gmail.com wrote:

Hi Nathan,

With the current patch, I was able to test it properly on my pseries VM, but I didn't use usefdt=1, mainly because I didn't know where to use it. Should I need to recompile the loader to use it, or, is it a parameter somewhere?

And it was working fine?

If I do not set the usefdt=1, it works fine. If I set it, it break in the early start.

OK set usefdt=1
OK boot
Booting...
Kernel entry at 0x102670 ...
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #102 66160e2fc87(nathan): Tue Mar  6 16:19:44 CET 2018
    root@free8:/usr/obj/root/kernel/freebsd/powerpc.powerpc64/sys/GENERIC64 powerpc
gcc version 4.2.1 20070831 patched [FreeBSD]
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: Running on a broken hypervisor that does not support mandatory H_CLEAR_MOD and H_CLEAR_REF hypercalls. Performance will be suboptimal.
[ thread pid 0 tid 0 ]
Stopped at      .trap+0x40:     stdu    r1, r1, 0xfe40
db> bt
Tracing pid 0 tid 0 td 0x14eb5a0
0xe000000000001190: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe000000000001440 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000001440: at 0x38000000901f0020
0xe0000000000014e0: at .trap_pfault+0xb0
0xe000000000001590: at .trap+0xef0

Minor cleanups. Fix Book-E.

In D14499#306477, @breno.leitao_gmail.com wrote:

Hi Nathan,

With the current patch, I was able to test it properly on my pseries VM, but I didn't use usefdt=1, mainly because I didn't know where to use it. Should I need to recompile the loader to use it, or, is it a parameter somewhere?

In D14499#306155, @breno.leitao_gmail.com wrote:

Nathan,

I tried you most recent patchset and it is still not working on pseries.

Fix fault when synchronizing icache.

• Fix graphics console on pSeries
• Apparent PS3 bugs fixed by other patches (this was only triggering latent problems)

Another thing to try with the Open Firmware crash is to set 'usefdt=1' in the loader. For me, either of these resolve the crash. Could you verify this on your hardware?

In D14538#305284, @pdk_semihalf.com wrote:

I have a question about your 1st worry. Is this a scenario in which we are doing memcpy in eg. interrupt filter code? If not, could you explain what are you worry about?

In D14499#304559, @breno.leitao_gmail.com wrote:

If the above analyzes is correct, It seems that a function ends up with a conditional branch, so, if the condition is not reached, then it moved to the function epilogue, which is 0x0.

(gdb) x/10i 0x000000007dbe02f0
    0x7dbe02f0:	std     r0,296(r1)
    0x7dbe02f4:	mfdar   r0
    0x7dbe02f8:	std     r0,304(r1)
    0x7dbe02fc:	mfdsisr r0
    0x7dbe0300:	std     r0,312(r1)
    0x7dbe0304:	bcl     20,4*cr7+so,0x7dbe0310
    0x7dbe0308:	.long 0x0
    0x7dbe030c:	.long 0x87eef8
    0x7dbe0310:	mflr    r2
    0x7dbe0314:	ld      r1,0(r2)

But this is still very confusing, since there just two places in the code we use mfdsisr, and it is always with %r31.

I am wondering if something caused the code to leave the kernel code, and now I am debugging somewhere else (OpenFirmware?).

I have a few main worries about this, mostly also in the inline comments:

1. I think this trashes userland vector registers, which is very bad.
2. It breaks booting on Apple hardware.
3. Is there really a performance benefit to this? The kernel tries pretty vigorously to avoid memcpy and no other architectures have vectorized in-kernel memcpy(), substantially to avoid the serious issues associated with concern 1, but also because it doesn't help much. I would imagine you would get much more benefit, with much less hassle, from optimizing the copy in libc, which would be 90% of the same code but with all the complex cases deleted.

Looks great to me.

cpu_subr64.S isn't really a great name, since it doesn't convey a ton of information. How about cpu_idle64.S? In any case, is there a reason it needs to be #included into locore64.S rather than built as an independent file?

In D14330#302660, @pdk_semihalf.com wrote:

In D14330#302615, @nwhitehorn wrote:

In D14330#302546, @pdk_semihalf.com wrote:

Could you tell me more about scenario in which we branch to rstcode from software? I was looking for it but I found nothing.

Line 99 of locore64.S.

Thanks. I will clear whole register, right?

I think this breaks 32-bit builds since ps_per_tick is too small there. I think it needs to be bumped to uint64_t.

In D14330#302546, @pdk_semihalf.com wrote:

Could you tell me more about scenario in which we branch to rstcode from software? I was looking for it but I found nothing.