Page MenuHomeFreeBSD

Create DMAP dynamically on ARM64 platforms
AbandonedPublic

Authored by wma on Aug 31 2015, 10:13 AM.
Tags
None
Referenced Files
Unknown Object (File)
Sun, Mar 24, 6:12 PM
Unknown Object (File)
Feb 25 2024, 8:21 PM
Unknown Object (File)
Feb 23 2024, 12:12 PM
Unknown Object (File)
Feb 13 2024, 11:30 PM
Unknown Object (File)
Jan 15 2024, 2:04 PM
Unknown Object (File)
Jan 10 2024, 1:44 AM
Unknown Object (File)
Jan 2 2024, 3:25 PM
Unknown Object (File)
Dec 25 2023, 4:03 AM

Details

Reviewers
wma_semihalf.com
Group Reviewers
arm64
Summary

FreeBSD kernel requires to have all DDR memory mapped into DMAP.
Previous approach did not work when the DDR space was not
continuous. Now, the physical description of memory is resolved
from EFI and appropriate physical chunks are mapped into
continuous virtual memory space.

Diff Detail

Repository
rS FreeBSD src repository - subversion
Lint
Lint Skipped
Unit
Tests Skipped

Event Timeline

wma_semihalf.com retitled this revision from to Create DMAP dynamically on ARM64 platforms.
wma_semihalf.com updated this object.
wma_semihalf.com edited the test plan for this revision. (Show Details)
wma_semihalf.com added a reviewer: arm64.
wma_semihalf.com set the repository for this revision to rS FreeBSD src repository - subversion.

Fix:

  • PA chunks are 1GB-aligned
  • fix minidumps

Why such complication is needed ? The arm64 platforms are positioned as the server-class machines, so it should have enough memory to always provide tables for direct map from 0 to max present physical address. Static DMAP_MAX_PHYSADDR is indeed somewhat undesirable, while having dynamic segments and, as I understand, avoiding holes is equally bad: translation between physical and direct-mapped address is supposed to be fast.

At least what I described is true for amd64, and I do not see why arm64 should be different there. Just provide contig mapping from 0 to max phys. This has additional advantage of serving PCI hole on amd64, probably the same trick could be applied on arm64.

The problem is, that on ARM64 we do not have physically continuous memory. If we take ThunderX as an example, there is:
0#64GB @ 0x00000000_00000000 - 0x0000000F_FFFFFFFF
1#64GB @ 0x00000100_00000000 - 0x0000010F_FFFFFFFF
EFI runs us from region 1#, so we must have this mapped, because there is also where L1 pagetable is located.

Direct mapping of everything from 0x0..00 to 0x0000010F_FFFFFFFF is also impossible, because we have only 39 bits of VA available.
If you have any better idea of providing DMAP in this case we can discuss it, but I'm afraid there is no good solution which can be simultaneously fast and generic.

The arm64 platforms are positioned as the server-class machines

We also intend to run on smaller arm64 hardware, but I imagine that there won't be any such systems that also have the physical memory hole issue.

The problem is, that on ARM64 we do not have physically continuous memory. If we take ThunderX as an example, there is:
0#64GB @ 0x00000000_00000000 - 0x0000000F_FFFFFFFF
1#64GB @ 0x00000100_00000000 - 0x0000010F_FFFFFFFF
EFI runs us from region 1#, so we must have this mapped, because there is also where L1 pagetable is located.

Direct mapping of everything from 0x0..00 to 0x0000010F_FFFFFFFF is also impossible, because we have only 39 bits of VA available.

This is the critical issue, it seems. Where does the 39 bits VA limit come from ? I cannot be considered even closer to a knowledgable person for AArch64 specification, but still I cannot find a mentioned possibility of handled VA bits being less than 48.

Is this some implementation limitation ? Could it happen that on some board, the whole 40-bit of the PA are filled with the DRAM ?

39 bits are the maximum address range the MMU can handle using 4KB pages and L1&L2&L3 pagetables. This corresponds to mapping of 512GB of VA.
Architecturally ARM64 can use up to VA 48-bits but that would require either using L0 pagetable as well (thus provide way more complex address translation and slow down the whole system, not only dmap pa/va resolving) or increase page size to 16KB or 64KB (that is also undesireable).

39 bits are the maximum address range the MMU can handle using 4KB pages and L1&L2&L3 pagetables. This corresponds to mapping of 512GB of VA.
Architecturally ARM64 can use up to VA 48-bits but that would require either using L0 pagetable as well (thus provide way more complex address translation and slow down the whole system, not only dmap pa/va resolving) or increase page size to 16KB or 64KB (that is also undesireable).

So the real issue is that right now the architecture uses 3-level page tables instead of 4-level ? I believe this is the reason and the place which should be fixed. Amd64 lives with 4-level page tables from the inception and it is fine.

Yep, that's also the way we can do this.

I'd also like to have this confirmed by arm64 guys.
Andrew, Ed, what do you think? Should we go to 4-level pagetables or use more complex DMAP instead?

Yep, that's also the way we can do this.

I'd also like to have this confirmed by arm64 guys.
Andrew, Ed, what do you think? Should we go to 4-level pagetables or use more complex DMAP instead?

In the end I think we're not going to have a choice -- our roadmap is going to have us supporting large-memory systems with 512GB, 1TB or more.

My preferred way to solve this is with D3885 + using 4 level page tables. I think we should use the correct macros in pmap.c, and have pointed them out (I think the code pre-dated the macros).

sys/arm64/arm64/pmap.c
864

Good

3128

Good

3146

Good

3166

Good

wma abandoned this revision.
wma added a reviewer: wma_semihalf.com.