amd64 pmap: move DMAP out of kernel pml4 table for LA57.
Needs ReviewPublic
Actions

Authored by kib on Sat, Jun 28, 11:18 PM.

Details

Reviewers

alc
markj

Summary

amd64: for LA57, connect DMAP directly to PML5, currently reserving 32 slots


amd64 pmap: for la57, copy whole kernel pml5 table to user pmap pm_pml5 table


amd64: add kva_layout


amd64 pmap: do not hardcode 512GB

Diff Detail

Repository

rG FreeBSD src repository

Lint

Lint Skipped

Unit

Tests Skipped

Event Timeline

kib created this revision.Sat, Jun 28, 11:18 PM

Herald added a subscriber: imp. · View Herald TranscriptSat, Jun 28, 11:18 PM

kib requested review of this revision.Sat, Jun 28, 11:18 PM

This effectively allows to boot on >4Tb machines and use all memory, not clipping it at 4TB. I tried it on qemu with 8TB configured, the machine boot, but eventually qemu dies on my 64GB w/s due to OOM (really out of swap space).
The sysctl vm.vmtotal looks funny:

System wide totals computed every five seconds: (values in kilobytes)
===============================================
Processes:		(RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 1)
Virtual Memory:		(Total: 4295023404K Active: 4295023404K)
Real Memory:		(Total: 6192K Active: 6192K)
Shared Virtual Memory:	(Total: 2704K Active: 2704K)
Shared Real Memory:	(Total: 2224K Active: 2224K)
Free Memory:	8402748544K

and top memory usage line cannot display the free memory size.

One limit that is not yet in place but probably should be added, is to cup the memory by the max size of vm_page_array[] which is put into kpml4. (Previous variant of the paragraph stated that vm_page_array[] is indexed by ints but I retract that).

After this change is discussed, the next action would be to move the large map out of kernel pml4. When done, we can change the KVA layout for amd64 for real, allowing larger kmem if needed.

kib edited the summary of this revision. (Show Details)Sat, Jun 28, 11:21 PM

kib added reviewers: alc, markj.

kib added subscribers: emaste, pho.

I ran tests with D51085.id157729.diff for 18 hours without seeing any issues.

markj added inline comments.Wed, Jul 2, 1:41 PM

sys/amd64/amd64/pmap.c
484	Can these be `const`?
sys/amd64/include/pmap.h
561	Why not use vm_offset_t for these?
sys/amd64/include/vmparam.h
189	Why not keep these constants, defining them the same way as VM_{MIN,MAX}_KERNEL_ADDRESS? Then the minidump code doesn't need to change.

kib marked 3 inline comments as done.Wed, Jul 2, 2:19 PM

kib added inline comments.

sys/amd64/amd64/pmap.c
484	Right now it cannot. In la57 case, I copy kva_layout_la57 over kva_layout. I do not want to add an indirection through some pointer. I can make kva_layout_la48 and kva_layout_la57 const, and copy either one over real kva_layout, but I do not think it buys much.
sys/amd64/include/vmparam.h
189	VM_MIN/MAX_KERNEL_ADDRESS symbols are required by MI code, so I provide them. DMAP_MIN/MAX_ADDRESS are confined to the arch sources, I decided to remove them to make it more explicit the references to kva_layout (less magic).

Use vm_offset_t for kva_layout members

markj accepted this revision.Thu, Jul 3, 1:32 PM

markj added inline comments.

sys/amd64/amd64/pmap.c
497
506	It would be nice to update the comment in vmparam.h which describes the kernel memory map, to define the layout for LA57 kernels.
4390	It would be nice to initialize PTPs in a consistent style. I find loops easier to read there, as is done in `pmap_pinit_pml4_pti()`.
4949	IMO it is more clearly written as a loop, like all of the other PMLtop initializations below.
sys/amd64/include/vmparam.h
189	I still find the old names clearer, if only because they're still there on other platforms, even if they are only used in MD code. But it is just a preference.

This revision is now accepted and ready to land.Thu, Jul 3, 1:32 PM

kib marked 4 inline comments as done.Thu, Jul 3, 2:17 PM

kib added inline comments.

sys/amd64/include/vmparam.h
189	I tried to convince myself, but lets wait some time more for the decision.

Add KVA map.
Use loops instead of memcpy/memset.

This revision now requires review to proceed.Thu, Jul 3, 2:17 PM

markj accepted this revision.Thu, Jul 3, 3:04 PM

markj added inline comments.

sys/amd64/include/pmap.h
176
sys/amd64/include/vmparam.h
189	This line should be removed.

This revision is now accepted and ready to land.Thu, Jul 3, 3:04 PM

kib marked 3 inline comments as done.Thu, Jul 3, 6:14 PM

kib added inline comments.

sys/amd64/include/pmap.h
176	Also this sentence is not true anymore, the start of the direct map does not need to be aligned on its size, because new code uses '+' instead of '\|'.
sys/amd64/include/vmparam.h
189	Rather, I keep it with s/direct map/unused/.

Update comments.

This revision now requires review to proceed.Thu, Jul 3, 6:14 PM

alc added inline comments.Thu, Jul 3, 9:23 PM

sys/amd64/include/vmparam.h
189	I'm just curious as to why you switched from min and max to low and high. Admittedly, max has usually been a misnomer, because the ranges have not been inclusive of the max values.