Looks good. Thanks for doing this!

In D14499#306484, @breno.leitao_gmail.com wrote:

In D14499#306478, @nwhitehorn wrote:

In D14499#306477, @breno.leitao_gmail.com wrote:

Hi Nathan,

With the current patch, I was able to test it properly on my pseries VM, but I didn't use usefdt=1, mainly because I didn't know where to use it. Should I need to recompile the loader to use it, or, is it a parameter somewhere?

And it was working fine?

If I do not set the usefdt=1, it works fine. If I set it, it break in the early start.

OK set usefdt=1
OK boot
Booting...
Kernel entry at 0x102670 ...
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #102 66160e2fc87(nathan): Tue Mar  6 16:19:44 CET 2018
    root@free8:/usr/obj/root/kernel/freebsd/powerpc.powerpc64/sys/GENERIC64 powerpc
gcc version 4.2.1 20070831 patched [FreeBSD]
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: Running on a broken hypervisor that does not support mandatory H_CLEAR_MOD and H_CLEAR_REF hypercalls. Performance will be suboptimal.
[ thread pid 0 tid 0 ]
Stopped at      .trap+0x40:     stdu    r1, r1, 0xfe40
db> bt
Tracing pid 0 tid 0 td 0x14eb5a0
0xe000000000001190: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe000000000001440 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000001440: at 0x38000000901f0020
0xe0000000000014e0: at .trap_pfault+0xb0
0xe000000000001590: at .trap+0xef0

Minor cleanups. Fix Book-E.

In D14499#306477, @breno.leitao_gmail.com wrote:

Hi Nathan,

With the current patch, I was able to test it properly on my pseries VM, but I didn't use usefdt=1, mainly because I didn't know where to use it. Should I need to recompile the loader to use it, or, is it a parameter somewhere?

In D14499#306155, @breno.leitao_gmail.com wrote:

Nathan,

I tried you most recent patchset and it is still not working on pseries.

Fix fault when synchronizing icache.

• Fix graphics console on pSeries
• Apparent PS3 bugs fixed by other patches (this was only triggering latent problems)

Another thing to try with the Open Firmware crash is to set 'usefdt=1' in the loader. For me, either of these resolve the crash. Could you verify this on your hardware?

In D14538#305284, @pdk_semihalf.com wrote:

I have a question about your 1st worry. Is this a scenario in which we are doing memcpy in eg. interrupt filter code? If not, could you explain what are you worry about?

In D14499#304559, @breno.leitao_gmail.com wrote:

If the above analyzes is correct, It seems that a function ends up with a conditional branch, so, if the condition is not reached, then it moved to the function epilogue, which is 0x0.

(gdb) x/10i 0x000000007dbe02f0
   0x7dbe02f0:	std     r0,296(r1)
   0x7dbe02f4:	mfdar   r0
   0x7dbe02f8:	std     r0,304(r1)
   0x7dbe02fc:	mfdsisr r0
   0x7dbe0300:	std     r0,312(r1)
   0x7dbe0304:	bcl     20,4*cr7+so,0x7dbe0310
   0x7dbe0308:	.long 0x0
   0x7dbe030c:	.long 0x87eef8
   0x7dbe0310:	mflr    r2
   0x7dbe0314:	ld      r1,0(r2)

But this is still very confusing, since there just two places in the code we use mfdsisr, and it is always with %r31.

I am wondering if something caused the code to leave the kernel code, and now I am debugging somewhere else (OpenFirmware?).

I have a few main worries about this, mostly also in the inline comments:

1. I think this trashes userland vector registers, which is very bad.
2. It breaks booting on Apple hardware.
3. Is there really a performance benefit to this? The kernel tries pretty vigorously to avoid memcpy and no other architectures have vectorized in-kernel memcpy(), substantially to avoid the serious issues associated with concern 1, but also because it doesn't help much. I would imagine you would get much more benefit, with much less hassle, from optimizing the copy in libc, which would be 90% of the same code but with all the complex cases deleted.

Looks great to me.

cpu_subr64.S isn't really a great name, since it doesn't convey a ton of information. How about cpu_idle64.S? In any case, is there a reason it needs to be #included into locore64.S rather than built as an independent file?

In D14330#302660, @pdk_semihalf.com wrote:

In D14330#302615, @nwhitehorn wrote:

In D14330#302546, @pdk_semihalf.com wrote:

Could you tell me more about scenario in which we branch to rstcode from software? I was looking for it but I found nothing.

Line 99 of locore64.S.

Thanks. I will clear whole register, right?

I think this breaks 32-bit builds since ps_per_tick is too small there. I think it needs to be bumped to uint64_t.

In D14330#302546, @pdk_semihalf.com wrote:

Could you tell me more about scenario in which we branch to rstcode from software? I was looking for it but I found nothing.

In D14405#302166, @jhibbits wrote:

In D14405#302165, @nwhitehorn wrote:

Why would you want SLB on ppc32? It's only needed to support > 4 GB of VA space, but you can't have that by construction.

I wasn't thinking correctly... I was thinking about the >4GB PA, which that doesn't cover (can we do >4GB PA in 32-bit space on Book-S generically? I only know of the MPC7450-class means, but I don't think that's generic).

Why would you want SLB on ppc32? It's only needed to support > 4 GB of VA space, but you can't have that by construction.

Looks great to me (no testing, though).

There is a yield instruction ("yield"), but it is a bit of a different thing. Earlier versions of POWER also do implement instructions that do this without going through reset (MSR[POW], notably). As a result, I have somewhat mixed feelings implementing this in the platform layer. Is there a reason to do it there rather than cpu.c, where the code for MSR[POW] etc. lives?

Is cpu_idle() the right place for this? I'm not sure what the wake-up latency of this code is, but, if it involves restoring registers in software, it seems like it might be heavier-weight than intended for cpu_idle().

Yes, this is a WIP. Is there anything different about ARM[64] EFI boot vs. amd64? I don't have any such hardware to test on, but I would guess this code can basically be copied and pasted.

Use new general FAT code.

The framework in the install to support this ended up being a little different. The updated diff is attached.

There is intrinsically no difference between CHRP and PowerNV. Could you please update CHRP to match, including the BSP 0 code, (copy-and-paste is fine) and test it on your POWER8 system? I don't have access to any POWER8 hardware and the moment and the risk of breaking Power[K]VM is high.

Looks great!

I like the approach. Why doesn't it work if BSP is non-zero? Doesn't this code arrange for the BSP to always be zero?

This looks great.

This is conceptually unworkable, for two core reasons. First, the PIR register is unreliable when set, especially on VMs in the face of movement to another core. We've tried to remove all uses of PIR in code that does not know it is reliable (all generic code) for this reason. The real kicker, though, is that on some of our supported hardware (Apple G4 machines, for instance), PIR needs to be set by *us* and all CPUs boot with identical PIR. In the face of this, the proposed mechanism simply cannot work.

I think this is a great idea. Don't see any obvious problems with the implementation.

I think this is fine. Have you tested on loader-using hardware (Apple, IBM paravirtualized)? I have some vague memory this was needed on Apple hardware.

Nice catch. Thanks!

I agree with Warner's comments -- if there is any other way to do the endian conversions at all, we should.

Looks good to me. Thanks!

Works just fine on PowerPC.

It would be good if we could add some #ifdef here such that it still builds without both PSERIES and POWERNV in the kernel (especially the latter, as committing this will break all custom PSERIES kernels as-is). Other than that, it looks good to me.