Details

Reviewers

jhibbits
breno.leitao_gmail.com
andreast

Commits

rS330610: Move the powerpc64 direct map base address from zero to high memory. This

Summary

This change moves the ppc64 direct map base address up to high memory and fixes a -- suprisingly large -- amount of fallout from the change. We want to move it for three reasons:

It makes NULL and related addresses invalid pointers in the kernel, giving us a fighting chance of catching NULL-pointer bugs.
It needs to be here for POWER9 Radix translation to work.
If 64-bit Book-E grows a direct map, it also needs to be in high memory.

The new base address is chosen as the base of the fourth radix quadrant (the minimum kernel address in this translation mode) and because all supported CPUs ignore at least the first two bits of addresses in real mode, allowing direct-map addresses to be used in real-mode handlers. This is required by Linux and is part of the architecture standard starting in POWER ISA 3, so can be relied upon.

The kernel continues to execute from addresses below the (new) direct map with a 1:1 mapping. This is a consequence of the early bootstrap code, which will be addressed in a follow-up commit on systems that can handle it (all but Apple hardware when running with OF rather than FDT).

Because the amount of fallout from moving this was surprisingly large, testing on a diverse set of systems would be appreciated. So far, this is tested only on Apple hardware (working well), PS3 (likewise), and pSeries and PowerNV (in QEMU).

Diff Detail

Repository

rS FreeBSD src repository - subversion

Lint

Lint Skipped

Unit

Tests Skipped

Build Status

Buildable 15370

Event Timeline

nwhitehorn created this revision.Feb 24 2018, 10:21 PM

Herald added a subscriber: imp. · View Herald TranscriptFeb 24 2018, 10:21 PM

nwhitehorn updated this revision to Diff 39694.Feb 24 2018, 11:02 PM

nwhitehorn updated this revision to Diff 39695.Feb 24 2018, 11:24 PM

breno.leitao_gmail.com added a comment.Feb 26 2018, 5:01 PM

This comment was removed by breno.leitao_gmail.com.

sys/powerpc/aim/mmu_oea64.c
555	It seems that there is space and tabs mixed here, as '\b\t\t\t"
686	There is an space as the last char.
928	There is several spaces on this line

Once my machine stops, it just shows:

reeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #59 997dcd88057(nathan): Mon Feb 26 17:58:00 CET 2018

root@free8:/usr/obj/root/kernel/freebsd/powerpc.powerpc64/sys/GENERIC64 powerpc

gcc version 4.2.1 20070831 patched [FreeBSD]

WARNING: WITNESS option enabled, expect reduced performance.

I can it stops on the 0x480 'Instruction Segment' trap, which is looping forever, so, something is causing trap 0x480 all the time:

=> 0x480:	mtsprg  0,r0
   0x484:	mfctr   r0
   0x488:	mtsprg  2,r0
   0x48c:	mflr    r0
   0x490:	mtsprg  3,r0
   0x494:	ld      r0,12272(0)
   0x498:	mtctr   r0
   0x49c:	li      r0,1152
   0x4a0:	bctr
   0x4a4:	.long 0x0

This is the registers:

r0             0x7dbe0210	2109604368
r1             0x7e45f0f0	2118512880
r2             0x7dbffc00	2109733888
r3             0x3	3
r4             0x480	1152
r5             0x2e00	11776
r6             0xee	238
r7             0xf826c0	16262848
r8             0x5c0	1472
r9             0x2e80	11904
r10            0xffffffffffffff80	18446744073709551488
r11            0x80	128
r12            0x84000024	2214592548
r13            0x14e4420	21906464
r14            0x0	0
r15            0x0	0
r16            0x0	0
r17            0x0	0
r18            0x0	0
r19            0x0	0
r20            0x8000000000003000	9223372036854788096
r21            0x1	1
r22            0x0	0
r23            0x0	0
r24            0x2c429ac	46410156
r25            0x2c40df8	46403064
r26            0x2c3fa44	46398020
r27            0xc000000000000100	13835058055282163968
r28            0x16df8f0	23984368
r29            0x1190290	18416272
r30            0x8000000000001032	9223372036854779954
r31            0xe000000000008690	16140901064495892112
pc             0x48c	0x48c
msr            0x8000000000001000	9223372036854779904
cr             0x84000024	2214592548
lr             0x7dbe0308	0x7dbe0308
ctr            0x7dbe0210	2109604368
xer            0x0	0`

LR is pointing to 0x7dbe0308, which has 0x0 instruction

x/10i 0x7dbe0308

0x7dbe0308:	.long 0x0
0x7dbe030c:	.long 0x87eef8
0x7dbe0310:	mflr    r2
0x7dbe0314:	ld      r1,0(r2)
0x7dbe0318:	add     r1,r1,r2
0x7dbe031c:	li      r0,0
0x7dbe0320:	stdu    r0,-16(r1)
0x7dbe0324:	stdu    r1,-256(r1)
0x7dbe0328:	lis     r2,32191
0x7dbe032c:	ld      r0,26720(r2)

This is a code that I do not know where it is from, since I grepped my binaries to find these instructions and they are not there.

If the above analyzes is correct, It seems that a function ends up with a conditional branch, so, if the condition is not reached, then it moved to the function epilogue, which is 0x0.

(gdb) x/10i 0x000000007dbe02f0
   0x7dbe02f0:	std     r0,296(r1)
   0x7dbe02f4:	mfdar   r0
   0x7dbe02f8:	std     r0,304(r1)
   0x7dbe02fc:	mfdsisr r0
   0x7dbe0300:	std     r0,312(r1)
   0x7dbe0304:	bcl     20,4*cr7+so,0x7dbe0310
   0x7dbe0308:	.long 0x0
   0x7dbe030c:	.long 0x87eef8
   0x7dbe0310:	mflr    r2
   0x7dbe0314:	ld      r1,0(r2)

But this is still very confusing, since there just two places in the code we use mfdsisr, and it is always with %r31.

I am wondering if something caused the code to leave the kernel code, and now I am debugging somewhere else (OpenFirmware?).

In D14499#304559, @breno.leitao_gmail.com wrote:
If the above analyzes is correct, It seems that a function ends up with a conditional branch, so, if the condition is not reached, then it moved to the function epilogue, which is 0x0.
(gdb) x/10i 0x000000007dbe02f0
   0x7dbe02f0:	std     r0,296(r1)
   0x7dbe02f4:	mfdar   r0
   0x7dbe02f8:	std     r0,304(r1)
   0x7dbe02fc:	mfdsisr r0
   0x7dbe0300:	std     r0,312(r1)
   0x7dbe0304:	bcl     20,4*cr7+so,0x7dbe0310
   0x7dbe0308:	.long 0x0
   0x7dbe030c:	.long 0x87eef8
   0x7dbe0310:	mflr    r2
   0x7dbe0314:	ld      r1,0(r2)
But this is still very confusing, since there just two places in the code we use mfdsisr, and it is always with %r31.

I am wondering if something caused the code to leave the kernel code, and now I am debugging somewhere else (OpenFirmware?).

That sounds plausible. I think there is a possible failure mode here involving firmware calls. When we call into OF on pSeries, we disable interrupts and restore OF's trap handlers. But the MMU is still on, so we could, in principle, cross a segment boundary and get a DSE or ISE, which could be more common now. What happens if you disable trap handler save/restore in ofw_save_trap_vec()/ofw_restore_trap_vec()?

Another thing to try with the Open Firmware crash is to set 'usefdt=1' in the loader. For me, either of these resolve the crash. Could you verify this on your hardware?

What I think is going on is that we are hitting a failure mode that was always present with the trap/save restore code added in r258504 by andreast. If the kernel -- any thread -- takes a trap after OF's trap vectors are restored, Bad Things happen. We disable interrupts, which helps, but there is still the following corner case, which has always been with us but seems more likely now: The MMU is still on when we restore the OF vectors. If the thread takes an segment exception. the kernel blows up. That could happen for a number of reasons: we could touch KVA we haven't used in a while while setting up the OF call, for instance. More perniciously, the hypervisor is free to invalidate arbitrary SLB entries at will, so any thread could take an ISE/DSE at any time for no reason. This includes the other CPUs, while have their MMUs on for the duration.

There are three possible workarounds that come to mind:

We turn off the trap save/restore code. This will break POWER5 systems.
We move all threads into MMU-off mode in the rendezvous, spinning on some small block of assembler that uses no stack.
We make usefdt=1 the default on all non-Apple systems (it doesn't seem to work right on Apple hardware, for unclear reasons, but seems to be 100% reliable on IBM-ish systems).

I think my preferred approach would be 3, then, post-commit, try to widen the scope of usefdt to include Apple hardware and start retiring the giant pile of hacks that is ofwcall*. Opinions?

Breno, those addresses you posted look suspiciously like instructions:

0x7dbe0308: tdi 0, r3, -16771
0x7dbe0210: vaddecuq v0,v2,v23,v25

Fix graphics console on pSeries
Apparent PS3 bugs fixed by other patches (this was only triggering latent problems)

nwhitehorn edited the summary of this revision. (Show Details)Mar 4 2018, 3:30 PM

Thanks for being invited to the review.
Well, I'm looking forward to a solution. I do not care in a first priority about POWER5+, it is broken anyway, genius iflib. But I can at least test the bring-up.
A more important priority is the G5 support. I'd need one and I'm willing to help.
For POWER8/9, others have to jump in.

So, take this as a go from myside.

Nathan,

I tried you most recent patchset and it is still not working on pseries.

fatal kernel trap:

   exception       = 0x300 (data storage interrupt)
   virtual address = 0x0
   dsisr           = 0x40000000
   srr0            = 0x9bf298 (0x9bf298)
   srr1            = 0x8000000000001032
   lr              = 0x98dbe4 (0x98dbe4)
   curthread       = 0x14eb5a0
          pid = 0, comm = 

[ thread pid 0 tid 0 ]
Stopped at      .__syncicache+0x38:     dcbst   r0, r0
db> bt
Tracing pid 0 tid 0 td 0x14eb5a0
0xe0000000000086e0: at 0xe00000000000876c
0xe000000000008770: at .openfirmware+0x40
0xe000000000008800: at .ofw_real_finddevice+0x88
0xe0000000000088b0: at .OF_finddevice+0x90
0xe000000000008940: at .rtas_setup+0x40
0xe000000000008a10: at .mi_startup+0x11c
0xe000000000008aa0: at btext+0xb4

Looking at DSISR (0x40000000), it seems that the only bit enable represents the following thing:

Set to 1 if MSR DR =1 and the translation for an attempted access is not found in the Page Table;

Does it make sense?

Fix fault when synchronizing icache.

In D14499#306155, @breno.leitao_gmail.com wrote:

Nathan,

I tried you most recent patchset and it is still not working on pseries.

I think I fixed this. Could you also try with 'usefdt=1' in the loader?

Hi Nathan,

With the current patch, I was able to test it properly on my pseries VM, but I didn't use usefdt=1, mainly because I didn't know where to use it. Should I need to recompile the loader to use it, or, is it a parameter somewhere?

In D14499#306477, @breno.leitao_gmail.com wrote:

Hi Nathan,

With the current patch, I was able to test it properly on my pseries VM, but I didn't use usefdt=1, mainly because I didn't know where to use it. Should I need to recompile the loader to use it, or, is it a parameter somewhere?

And it was working fine?

usefdt is just a command-line argument to the loader. At the loader prompt, you can do:
OK set usefdt=1
OK boot

In D14499#306478, @nwhitehorn wrote:

In D14499#306477, @breno.leitao_gmail.com wrote:

Hi Nathan,

With the current patch, I was able to test it properly on my pseries VM, but I didn't use usefdt=1, mainly because I didn't know where to use it. Should I need to recompile the loader to use it, or, is it a parameter somewhere?

And it was working fine?

If I do not set the usefdt=1, it works fine. If I set it, it break in the early start.

OK set usefdt=1
OK boot
Booting...
Kernel entry at 0x102670 ...
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #102 66160e2fc87(nathan): Tue Mar  6 16:19:44 CET 2018
    root@free8:/usr/obj/root/kernel/freebsd/powerpc.powerpc64/sys/GENERIC64 powerpc
gcc version 4.2.1 20070831 patched [FreeBSD]
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: Running on a broken hypervisor that does not support mandatory H_CLEAR_MOD and H_CLEAR_REF hypercalls. Performance will be suboptimal.
[ thread pid 0 tid 0 ]
Stopped at      .trap+0x40:     stdu    r1, r1, 0xfe40
db> bt
Tracing pid 0 tid 0 td 0x14eb5a0
0xe000000000001190: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe000000000001440 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000001440: at 0x38000000901f0020
0xe0000000000014e0: at .trap_pfault+0xb0
0xe000000000001590: at .trap+0xef0
0xe000000000001750: at .powerpc_interrupt+0x1e0
0xe0000000000017f0: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe000000000001aa0 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000001aa0: at 0x801f001c901f0004
0xe000000000001b40: at .trap_pfault+0xb0
0xe000000000001bf0: at .trap+0xef0
0xe000000000001db0: at .powerpc_interrupt+0x1e0
0xe000000000001e50: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe000000000002100 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000002100: at 0x600019547f88fffc
0xe0000000000021a0: at .trap_pfault+0xb0
0xe000000000002250: at .trap+0xef0
0xe000000000002410: at .powerpc_interrupt+0x1e0
0xe0000000000024b0: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe000000000002760 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000002760: at 0x817f00e0819f00e0
0xe000000000002800: at .trap_pfault+0xb0
0xe0000000000028b0: at .trap+0xef0
0xe000000000002a70: at .powerpc_interrupt+0x1e0
0xe000000000002b10: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe000000000002dc0 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000002dc0: at 0x3d2001818129cb00
0xe000000000002e60: at .trap_pfault+0xb0
0xe000000000002f10: at .trap+0xef0
0xe0000000000030d0: at .powerpc_interrupt+0x1e0
0xe000000000003170: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe000000000003420 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000003420: at 0x809f006c389ffffc
0xe0000000000034c0: at .trap_pfault+0xb0
0xe000000000003570: at .trap+0xef0
0xe000000000003730: at .powerpc_interrupt+0x1e0
0xe0000000000037d0: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe000000000003a80 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000003a80: at 0x80090000901f001c
0xe000000000003b20: at .trap_pfault+0xb0
0xe000000000003bd0: at .trap+0xef0
0xe000000000003d90: at .powerpc_interrupt+0x1e0
0xe000000000003e30: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe0000000000040e0 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe0000000000040e0: at 0x813f00588808fffc
0xe000000000004180: at .trap_pfault+0xb0
0xe000000000004230: at .trap+0xef0
0xe0000000000043f0: at .powerpc_interrupt+0x1e0
0xe000000000004490: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe000000000004740 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000004740: at 0x2046726565425340
0xe0000000000047e0: at .trap_pfault+0xb0
0xe000000000004890: at .trap+0xef0
0xe000000000004a50: at .powerpc_interrupt+0x1e0
0xe000000000004af0: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe000000000004da0 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000004da0: at -0x4
0xe000000000004e40: at .trap_pfault+0xb0
0xe000000000004ef0: at .trap+0xef0
0xe0000000000050b0: at .powerpc_interrupt+0x1e0
0xe000000000005150: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe000000000005400 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000005400: at -0x4
0xe0000000000054a0: at .trap_pfault+0xb0
0xe000000000005550: at .trap+0xef0
0xe000000000005710: at .powerpc_interrupt+0x1e0
0xe0000000000057b0: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe000000000005a60 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000005a60: at -0x4
0xe000000000005b00: at .trap_pfault+0xb0
0xe000000000005bb0: at .trap+0xef0
0xe000000000005d70: at .powerpc_interrupt+0x1e0
0xe000000000005e10: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe0000000000060c0 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe0000000000060c0: at -0x4
0xe000000000006160: at .trap_pfault+0xb0
0xe000000000006210: at .trap+0xef0
0xe0000000000063d0: at .powerpc_interrupt+0x1e0
0xe000000000006470: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe000000000006720 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000006720: at -0x4
0xe0000000000067c0: at .trap_pfault+0xb0
0xe000000000006870: at .trap+0xef0
0xe000000000006a30: at .powerpc_interrupt+0x1e0
0xe000000000006ad0: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe000000000006d80 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000006d80: at -0x4
0xe000000000006e20: at .trap_pfault+0xb0
0xe000000000006ed0: at .trap+0xef0
0xe000000000007090: at .powerpc_interrupt+0x1e0
0xe000000000007130: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe0000000000073e0 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe0000000000073e0: at -0x4
0xe000000000007480: at .trap_pfault+0xb0
0xe000000000007530: at .trap+0xef0
0xe0000000000076f0: at .powerpc_interrupt+0x1e0
0xe000000000007790: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe000000000007a40 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000007a40: at 0xe000000000007acc
0xe000000000007ae0: at .trap_pfault+0xb0
0xe000000000007b90: at .trap+0xef0
0xe000000000007d50: at .powerpc_interrupt+0x1e0
0xe000000000007df0: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe0000000000080a0 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe0000000000080a0: at 0xe00000000000812c
0xe000000000008140: at .trap_pfault+0xb0
0xe0000000000081f0: at .trap+0xef0
0xe0000000000083b0: at .powerpc_interrupt+0x1e0
0xe000000000008450: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_kextract+0x28: srr1=0x8000000000001032
            r1=0xe000000000008700 cr=0x48000022 xer=0 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000008700: at 0xe0000000000087fc
0xe000000000008790: at .uma_dbg_getslab+0x38
0xe000000000008820: at .uma_dbg_alloc+0x4c
0xe0000000000088b0: at .uma_zalloc_arg+0x1a8
0xe000000000008980: at .malloc_init+0x70
0xe000000000008a10: at .mi_startup+0x11c
0xe000000000008aa0: at btext+0xb4

This is the regular boot (without usefdt=1).

Loading /boot/defaults/loader.conf
/boot/kernel/kernel data=0x129ebb8+0x4ccf88 syms=[0x8+0x16d370+0x8+0x1698ac]
/boot/entropy size=0x1000

Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...               
Kernel entry at 0x102670 ...
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #102 66160e2fc87(nathan): Tue Mar  6 16:19:44 CET 2018
    root@free8:/usr/obj/root/kernel/freebsd/powerpc.powerpc64/sys/GENERIC64 powerpc
gcc version 4.2.1 20070831 patched [FreeBSD]
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: Running on a broken hypervisor that does not support mandatory H_CLEAR_MOD and H_CLEAR_REF hypercalls. Performance will be suboptimal.
VT: init without driver.
cpu0: IBM POWER8E revision 2.1, 3425.00 MHz
....

Minor cleanups. Fix Book-E.

In D14499#306484, @breno.leitao_gmail.com wrote:

In D14499#306478, @nwhitehorn wrote:

In D14499#306477, @breno.leitao_gmail.com wrote:

Hi Nathan,

With the current patch, I was able to test it properly on my pseries VM, but I didn't use usefdt=1, mainly because I didn't know where to use it. Should I need to recompile the loader to use it, or, is it a parameter somewhere?

And it was working fine?

If I do not set the usefdt=1, it works fine. If I set it, it break in the early start.

OK set usefdt=1
OK boot
Booting...
Kernel entry at 0x102670 ...
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #102 66160e2fc87(nathan): Tue Mar  6 16:19:44 CET 2018
    root@free8:/usr/obj/root/kernel/freebsd/powerpc.powerpc64/sys/GENERIC64 powerpc
gcc version 4.2.1 20070831 patched [FreeBSD]
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: Running on a broken hypervisor that does not support mandatory H_CLEAR_MOD and H_CLEAR_REF hypercalls. Performance will be suboptimal.
[ thread pid 0 tid 0 ]
Stopped at      .trap+0x40:     stdu    r1, r1, 0xfe40
db> bt
Tracing pid 0 tid 0 td 0x14eb5a0
0xe000000000001190: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_decode_kernel_ptr+0x38: srr1=0x8000000000001032
            r1=0xe000000000001440 cr=0x20001032 xer=0x20000000 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000001440: at 0x38000000901f0020
0xe0000000000014e0: at .trap_pfault+0xb0
0xe000000000001590: at .trap+0xef0

[snip]

0xe0000000000083b0: at .powerpc_interrupt+0x1e0
0xe000000000008450: kernel DSI read trap @ 0xdeadc0dedeadc0de by .pmap_kextract+0x28: srr1=0x8000000000001032
r1=0xe000000000008700 cr=0x48000022 xer=0 ctr=0 r2=0x13a17b8 sr=0x40000000
0xe000000000008700: at 0xe0000000000087fc
0xe000000000008790: at .uma_dbg_getslab+0x38
0xe000000000008820: at .uma_dbg_alloc+0x4c
0xe0000000000088b0: at .uma_zalloc_arg+0x1a8
0xe000000000008980: at .malloc_init+0x70
0xe000000000008a10: at .mi_startup+0x11c
0xe000000000008aa0: at btext+0xb4

This is fascinating. Could you verify that this happens without this patch (I assume it is unrelated?) and give some more details on your setup that gave this error. I would like to try to get to the bottom of it.

This is fascinating. Could you verify that this happens without this patch (I assume it is unrelated?) and give some more details on your setup that gave this error. I would like to try to get to the bottom of it.

I tested it on a kernel without this patchset and the problem seems to happen, so, it is not related to this change.

OK set usefdt=1
OK boot
Booting...
Kernel entry at 0x102690 ...
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #104 258337e550f(fscr)-dirty: Tue Mar  6 18:51:01 CET 2018
    root@free8:/usr/obj/root/kernel/freebsd/powerpc.powerpc64/sys/GENERIC64 powerpc
gcc version 4.2.1 20070831 patched [FreeBSD]
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: Running on a broken hypervisor that does not support mandatory H_CLEAR_MOD and H_CLEAR_REF hypercalls. Performance will be suboptimal.
[ thread pid 0 tid 0 ]
Stopped at      +0x40:  stdu    r1, r1, 0xfe40

Boots fine on Book-E. After this lands, and I finish my other work, I'll give 64-bit Book-E a DMAP, too, which will reduce the vmparam.h diff even further. Actually, on closer inspection, we should be able to just move the kernel to 0xe00.... already anyway.

This revision is now accepted and ready to land.Mar 7 2018, 3:20 PM

This is working fine on pseries also. The problem I reported above was found to not be related to this revision.

Closed by commit rS330610: Move the powerpc64 direct map base address from zero to high memory. This (authored by nwhitehorn). · Explain WhyMar 7 2018, 5:08 PM

This revision was automatically updated to reflect the committed changes.

nwhitehorn mentioned this in rS330610: Move the powerpc64 direct map base address from zero to high memory. This.

Move powerpc64 direct map from 0 to 0xc000000000000000
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 39954

sys/dev/vt/hw/ofwfb/ofwfb.c

sys/powerpc/aim/aim_machdep.c

sys/powerpc/aim/mmu_oea64.c

sys/powerpc/aim/moea64_native.c

sys/powerpc/aim/slb.c

sys/powerpc/aim/trap_subr64.S

sys/powerpc/include/sr.h

sys/powerpc/include/vmparam.h

sys/powerpc/ofw/ofw_machdep.c

sys/powerpc/ofw/ofw_real.c

sys/powerpc/powerpc/bus_machdep.c

sys/powerpc/powerpc/genassym.c

sys/powerpc/powerpc/mem.c

sys/powerpc/powerpc/uma_machdep.c

sys/powerpc/ps3/platform_ps3.c

Move powerpc64 direct map from 0 to 0xc000000000000000ClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 39954

sys/dev/vt/hw/ofwfb/ofwfb.c

sys/powerpc/aim/aim_machdep.c

sys/powerpc/aim/mmu_oea64.c

sys/powerpc/aim/moea64_native.c

sys/powerpc/aim/slb.c

sys/powerpc/aim/trap_subr64.S

sys/powerpc/include/sr.h

sys/powerpc/include/vmparam.h

sys/powerpc/ofw/ofw_machdep.c

sys/powerpc/ofw/ofw_real.c

sys/powerpc/powerpc/bus_machdep.c

sys/powerpc/powerpc/genassym.c

sys/powerpc/powerpc/mem.c

sys/powerpc/powerpc/uma_machdep.c

sys/powerpc/ps3/platform_ps3.c

Move powerpc64 direct map from 0 to 0xc000000000000000
ClosedPublic
Actions

Revision Contents
Changeset List