Page MenuHomeFreeBSD

amd64 pmap: move DMAP out of kernel pml4 table for LA57.
Needs ReviewPublic

Authored by kib on Sat, Jun 28, 11:18 PM.
Tags
None
Referenced Files
F122500699: D51085.id157729.diff
Sat, Jul 5, 7:46 PM
Unknown Object (File)
Sat, Jul 5, 4:47 AM
Unknown Object (File)
Sat, Jul 5, 12:46 AM
Unknown Object (File)
Sat, Jul 5, 12:21 AM
Unknown Object (File)
Sat, Jul 5, 12:21 AM
Unknown Object (File)
Fri, Jul 4, 9:52 PM
Unknown Object (File)
Fri, Jul 4, 2:19 PM
Unknown Object (File)
Thu, Jul 3, 11:35 PM
Subscribers

Details

Reviewers
alc
markj
Summary
amd64: for LA57, connect DMAP directly to PML5, currently reserving 32 slots


amd64 pmap: for la57, copy whole kernel pml5 table to user pmap pm_pml5 table


amd64: add kva_layout


amd64 pmap: do not hardcode 512GB

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped

Event Timeline

kib requested review of this revision.Sat, Jun 28, 11:18 PM

This effectively allows to boot on >4Tb machines and use all memory, not clipping it at 4TB. I tried it on qemu with 8TB configured, the machine boot, but eventually qemu dies on my 64GB w/s due to OOM (really out of swap space).
The sysctl vm.vmtotal looks funny:

System wide totals computed every five seconds: (values in kilobytes)
===============================================
Processes:		(RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 1)
Virtual Memory:		(Total: 4295023404K Active: 4295023404K)
Real Memory:		(Total: 6192K Active: 6192K)
Shared Virtual Memory:	(Total: 2704K Active: 2704K)
Shared Real Memory:	(Total: 2224K Active: 2224K)
Free Memory:	8402748544K

and top memory usage line cannot display the free memory size.

One limit that is not yet in place but probably should be added, is to cup the memory by the max size of vm_page_array[] which is put into kpml4. (Previous variant of the paragraph stated that vm_page_array[] is indexed by ints but I retract that).

After this change is discussed, the next action would be to move the large map out of kernel pml4. When done, we can change the KVA layout for amd64 for real, allowing larger kmem if needed.

kib added reviewers: alc, markj.
kib added subscribers: emaste, pho.

I ran tests with D51085.id157729.diff for 18 hours without seeing any issues.

sys/amd64/amd64/pmap.c
484

Can these be const?

sys/amd64/include/pmap.h
561

Why not use vm_offset_t for these?

sys/amd64/include/vmparam.h
189

Why not keep these constants, defining them the same way as VM_{MIN,MAX}_KERNEL_ADDRESS? Then the minidump code doesn't need to change.

kib marked 3 inline comments as done.Wed, Jul 2, 2:19 PM
kib added inline comments.
sys/amd64/amd64/pmap.c
484

Right now it cannot. In la57 case, I copy kva_layout_la57 over kva_layout. I do not want to add an indirection through some pointer.

I can make kva_layout_la48 and kva_layout_la57 const, and copy either one over real kva_layout, but I do not think it buys much.

sys/amd64/include/vmparam.h
189

VM_MIN/MAX_KERNEL_ADDRESS symbols are required by MI code, so I provide them.
DMAP_MIN/MAX_ADDRESS are confined to the arch sources, I decided to remove them to make it more explicit the references to kva_layout (less magic).

kib marked 2 inline comments as done.

Use vm_offset_t for kva_layout members

markj added inline comments.
sys/amd64/amd64/pmap.c
497
506

It would be nice to update the comment in vmparam.h which describes the kernel memory map, to define the layout for LA57 kernels.

4389

It would be nice to initialize PTPs in a consistent style. I find loops easier to read there, as is done in pmap_pinit_pml4_pti().

4947

IMO it is more clearly written as a loop, like all of the other PMLtop initializations below.

sys/amd64/include/vmparam.h
189

I still find the old names clearer, if only because they're still there on other platforms, even if they are only used in MD code. But it is just a preference.

This revision is now accepted and ready to land.Thu, Jul 3, 1:32 PM
kib marked 4 inline comments as done.Thu, Jul 3, 2:17 PM
kib added inline comments.
sys/amd64/include/vmparam.h
189

I tried to convince myself, but lets wait some time more for the decision.

Add KVA map.
Use loops instead of memcpy/memset.

This revision now requires review to proceed.Thu, Jul 3, 2:17 PM
markj added inline comments.
sys/amd64/include/pmap.h
176
sys/amd64/include/vmparam.h
188

This line should be removed.

This revision is now accepted and ready to land.Thu, Jul 3, 3:04 PM
kib marked 3 inline comments as done.Thu, Jul 3, 6:14 PM
kib added inline comments.
sys/amd64/include/pmap.h
176

Also this sentence is not true anymore, the start of the direct map does not need to be aligned on its size, because new code uses '+' instead of '|'.

sys/amd64/include/vmparam.h
188

Rather, I keep it with s/direct map/unused/.

kib marked 2 inline comments as done.

Update comments.

This revision now requires review to proceed.Thu, Jul 3, 6:14 PM
sys/amd64/include/vmparam.h
189

I'm just curious as to why you switched from min and max to low and high. Admittedly, max has usually been a misnomer, because the ranges have not been inclusive of the max values.

sys/amd64/include/vmparam.h
189

I always imagine an address space as growing vertically and having the lowest addresses at bottom. This model leaked into the naming, I guess.