Page MenuHomeFreeBSD

Create DMAP dynamically on ARM64 platforms
AbandonedPublic

Authored by wma on Aug 31 2015, 10:13 AM.

Details

Reviewers
wma_semihalf.com
Group Reviewers
arm64
Summary

FreeBSD kernel requires to have all DDR memory mapped into DMAP.
Previous approach did not work when the DDR space was not
continuous. Now, the physical description of memory is resolved
from EFI and appropriate physical chunks are mapped into
continuous virtual memory space.

Diff Detail

Repository
rS FreeBSD src repository
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

wma_semihalf.com retitled this revision from to Create DMAP dynamically on ARM64 platforms.
wma_semihalf.com updated this object.
wma_semihalf.com edited the test plan for this revision. (Show Details)
wma_semihalf.com added a reviewer: arm64.
wma_semihalf.com set the repository for this revision to rS FreeBSD src repository.

Fix:

  • PA chunks are 1GB-aligned
  • fix minidumps
kib added a subscriber: kib.Sep 2 2015, 1:00 PM

Why such complication is needed ? The arm64 platforms are positioned as the server-class machines, so it should have enough memory to always provide tables for direct map from 0 to max present physical address. Static DMAP_MAX_PHYSADDR is indeed somewhat undesirable, while having dynamic segments and, as I understand, avoiding holes is equally bad: translation between physical and direct-mapped address is supposed to be fast.

At least what I described is true for amd64, and I do not see why arm64 should be different there. Just provide contig mapping from 0 to max phys. This has additional advantage of serving PCI hole on amd64, probably the same trick could be applied on arm64.

The problem is, that on ARM64 we do not have physically continuous memory. If we take ThunderX as an example, there is:
0#64GB @ 0x00000000_00000000 - 0x0000000F_FFFFFFFF
1#64GB @ 0x00000100_00000000 - 0x0000010F_FFFFFFFF
EFI runs us from region 1#, so we must have this mapped, because there is also where L1 pagetable is located.

Direct mapping of everything from 0x0..00 to 0x0000010F_FFFFFFFF is also impossible, because we have only 39 bits of VA available.
If you have any better idea of providing DMAP in this case we can discuss it, but I'm afraid there is no good solution which can be simultaneously fast and generic.

emaste added a comment.Sep 2 2015, 1:12 PM

The arm64 platforms are positioned as the server-class machines

We also intend to run on smaller arm64 hardware, but I imagine that there won't be any such systems that also have the physical memory hole issue.

kib added a comment.Sep 2 2015, 1:46 PM

The problem is, that on ARM64 we do not have physically continuous memory. If we take ThunderX as an example, there is:
0#64GB @ 0x00000000_00000000 - 0x0000000F_FFFFFFFF
1#64GB @ 0x00000100_00000000 - 0x0000010F_FFFFFFFF
EFI runs us from region 1#, so we must have this mapped, because there is also where L1 pagetable is located.
Direct mapping of everything from 0x0..00 to 0x0000010F_FFFFFFFF is also impossible, because we have only 39 bits of VA available.

This is the critical issue, it seems. Where does the 39 bits VA limit come from ? I cannot be considered even closer to a knowledgable person for AArch64 specification, but still I cannot find a mentioned possibility of handled VA bits being less than 48.

Is this some implementation limitation ? Could it happen that on some board, the whole 40-bit of the PA are filled with the DRAM ?

39 bits are the maximum address range the MMU can handle using 4KB pages and L1&L2&L3 pagetables. This corresponds to mapping of 512GB of VA.
Architecturally ARM64 can use up to VA 48-bits but that would require either using L0 pagetable as well (thus provide way more complex address translation and slow down the whole system, not only dmap pa/va resolving) or increase page size to 16KB or 64KB (that is also undesireable).

kib added a comment.Sep 2 2015, 2:01 PM

39 bits are the maximum address range the MMU can handle using 4KB pages and L1&L2&L3 pagetables. This corresponds to mapping of 512GB of VA.
Architecturally ARM64 can use up to VA 48-bits but that would require either using L0 pagetable as well (thus provide way more complex address translation and slow down the whole system, not only dmap pa/va resolving) or increase page size to 16KB or 64KB (that is also undesireable).

So the real issue is that right now the architecture uses 3-level page tables instead of 4-level ? I believe this is the reason and the place which should be fixed. Amd64 lives with 4-level page tables from the inception and it is fine.

Yep, that's also the way we can do this.

I'd also like to have this confirmed by arm64 guys.
Andrew, Ed, what do you think? Should we go to 4-level pagetables or use more complex DMAP instead?

emaste added a comment.Sep 2 2015, 2:54 PM

Yep, that's also the way we can do this.
I'd also like to have this confirmed by arm64 guys.
Andrew, Ed, what do you think? Should we go to 4-level pagetables or use more complex DMAP instead?

In the end I think we're not going to have a choice -- our roadmap is going to have us supporting large-memory systems with 512GB, 1TB or more.

My preferred way to solve this is with D3885 + using 4 level page tables. I think we should use the correct macros in pmap.c, and have pointed them out (I think the code pre-dated the macros).

sys/arm64/arm64/pmap.c
869

Good

3133

Good

3151

Good

3171

Good

wma commandeered this revision.Feb 11 2016, 6:08 AM
wma abandoned this revision.
wma added a reviewer: wma_semihalf.com.