arm64: Set ATTR_CONTIGUOUS on DMAP's L2 blocks
ClosedPublic
Actions

Authored by alc on Thu, May 16, 5:58 PM.

Details

Reviewers

markj
andrew
manu

Commits

rG9fc5e3fb39ca: arm64: set ATTR_CONTIGUOUS on the DMAP's L2 blocks

Summary

On systems configured with 16K pages, this change creates 1G page mapping within the direct map where possible. Previously, the largest page size that was used to implement the direct map was 32M. On systems configured with 4K pages, this change creates 32M page mappings within the direct map where 1G is too large.

Test Plan

On a Microsoft DevKit configured with 4K pages, we see the following after this change:

vm.pmap.l2c.demotions: 2
vm.pmap.l1.demotions: 2

and

Direct map:
0xffffa00000600000-0xffffa00000700000 rw--sg     WB 0 0 0 16 0
0xffffa000008c0000-0xffffa000008f0000 rw--sg     WB 0 0 0 3 0
0xffffa00000c00000-0xffffa00002700000 rw--sg     WB 0 0 13 16 0
0xffffa0000e400000-0xffffa0001b26f000 rw--sg     WB 0 5 23 6 15
0xffffa0001c68b000-0xffffa0001f5d0000 rw--sg     WB 0 0 22 52 5
0xffffa0001f5f7000-0xffffa0002eb00000 rw--sg     WB 0 7 10 16 9
0xffffa00048600000-0xffffa000edf08000 rw--sg     WB 1 49 28 16 8
0xffffa000edf08000-0xffffa000edf0c000 rw--sg     UC 0 0 0 0 4
0xffffa000edf0c000-0xffffa0014fe3a000 rw--sg     WB 1 16 15 18 14
0xffffa0014fe3a000-0xffffa0014fe5c000 rw--sg     UC 0 0 0 0 34
0xffffa0014fe5c000-0xffffa0014fe5e000 rw--sg     WB 0 0 0 0 2
0xffffa0014fe5e000-0xffffa0014fe6e000 rw--sg     UC 0 0 0 0 16
0xffffa0014fe6e000-0xffffa0014fe74000 rw--sg     WB 0 0 0 0 6
0xffffa0014fe74000-0xffffa0014fe75000 rw--sg     UC 0 0 0 0 1
0xffffa0014fe75000-0xffffa0014fe77000 rw--sg     WB 0 0 0 0 2
0xffffa0014fe77000-0xffffa0014fe7b000 rw--sg     UC 0 0 0 0 4
0xffffa0014fe7b000-0xffffa0014fe83000 rw--sg     WB 0 0 0 0 8
0xffffa0014fe83000-0xffffa0014fe84000 rw--sg     UC 0 0 0 0 1
0xffffa0014fe84000-0xffffa0014fe86000 rw--sg     WB 0 0 0 0 2
0xffffa0014fe86000-0xffffa0014fe87000 rw--sg     UC 0 0 0 0 1
0xffffa0014fe87000-0xffffa0014fe89000 rw--sg     WB 0 0 0 0 2
0xffffa0014fe89000-0xffffa0014febf000 rw--sg     UC 0 0 0 0 54
0xffffa0014febf000-0xffffa0014fec1000 rw--sg     WB 0 0 0 0 2
0xffffa0014fec1000-0xffffa0014fec2000 rw--sg     UC 0 0 0 0 1
0xffffa0014fec2000-0xffffa0014fec4000 rw--sg     WB 0 0 0 0 2
0xffffa0014fec4000-0xffffa0014fec8000 rw--sg     UC 0 0 0 0 4
0xffffa0014fec8000-0xffffa0014fed0000 rw--sg     WB 0 0 0 0 8
0xffffa0014fed0000-0xffffa0014fed1000 rw--sg     UC 0 0 0 0 1
0xffffa0014fed1000-0xffffa0014fed3000 rw--sg     WB 0 0 0 0 2
0xffffa0014fed3000-0xffffa0014fed4000 rw--sg     UC 0 0 0 0 1
0xffffa0014fed4000-0xffffa0014fed6000 rw--sg     WB 0 0 0 0 2
0xffffa0014fed6000-0xffffa0014fed8000 rw--sg     UC 0 0 0 0 2
0xffffa0014fed8000-0xffffa0014feda000 rw--sg     WB 0 0 0 0 2
0xffffa0014feda000-0xffffa0014fedb000 rw--sg     UC 0 0 0 0 1
0xffffa0014fedb000-0xffffa0014fedd000 rw--sg     WB 0 0 0 0 2
0xffffa0014fedd000-0xffffa0014fedf000 rw--sg     UC 0 0 0 0 2
0xffffa0014fedf000-0xffffa0014fee1000 rw--sg     WB 0 0 0 0 2
0xffffa0014fee1000-0xffffa0014fee2000 rw--sg     UC 0 0 0 0 1
0xffffa0014fee2000-0xffffa0014fee4000 rw--sg     WB 0 0 0 0 2
0xffffa0014fee4000-0xffffa0014fee5000 rw--sg     UC 0 0 0 0 1
0xffffa0014fee5000-0xffffa0014feed000 rw--sg     WB 0 0 0 0 8
0xffffa0014feed000-0xffffa0014ff0f000 rw--sg     UC 0 0 0 0 34
0xffffa0014ff0f000-0xffffa0014ff11000 rw--sg     WB 0 0 0 0 2
0xffffa0014ff11000-0xffffa0014ff21000 rw--sg     UC 0 0 0 0 16
0xffffa0014ff21000-0xffffa0014ff27000 rw--sg     WB 0 0 0 0 6
0xffffa0014ff27000-0xffffa0014ff28000 rw--sg     UC 0 0 0 0 1
0xffffa0014ff28000-0xffffa0014ff2a000 rw--sg     WB 0 0 0 0 2
0xffffa0014ff2a000-0xffffa0014ff2f000 rw--sg     UC 0 0 0 0 5
0xffffa0014ff2f000-0xffffa0014ff31000 rw--sg     WB 0 0 0 0 2
0xffffa0014ff31000-0xffffa0014ff32000 rw--sg     UC 0 0 0 0 1
0xffffa0014ff32000-0xffffa0014ff34000 rw--sg     WB 0 0 0 0 2
0xffffa0014ff34000-0xffffa0014ff5a000 rw--sg     UC 0 0 0 0 38
0xffffa0014ff5a000-0xffffa0014ff5c000 rw--sg     WB 0 0 0 0 2
0xffffa0014ff5c000-0xffffa0014ff6c000 rw--sg     UC 0 0 0 0 16
0xffffa0014ff6c000-0xffffa0014ff72000 rw--sg     WB 0 0 0 0 6
0xffffa0014ff72000-0xffffa0014ff73000 rw--sg     UC 0 0 0 0 1
0xffffa0014ff73000-0xffffa0014ff75000 rw--sg     WB 0 0 0 0 2
0xffffa0014ff75000-0xffffa0014ff79000 rw--sg     UC 0 0 0 0 4
0xffffa0014ff79000-0xffffa0014ff7b000 rw--sg     WB 0 0 0 0 2
0xffffa0014ff7b000-0xffffa0014ff7f000 rw--sg     UC 0 0 0 0 4
0xffffa0014ff7f000-0xffffa0014ff81000 rw--sg     WB 0 0 0 0 2
0xffffa0014ff81000-0xffffa0014ff82000 rw--sg     UC 0 0 0 0 1
0xffffa0014ff82000-0xffffa0014ff84000 rw--sg     WB 0 0 0 0 2
0xffffa0014ff84000-0xffffa0014ff97000 rw--sg     UC 0 0 0 0 19
0xffffa0014ff97000-0xffffa0014ff99000 rw--sg     WB 0 0 0 0 2
0xffffa0014ff99000-0xffffa0014ff9d000 rw--sg     UC 0 0 0 0 4
0xffffa0014ff9d000-0xffffa0014ff9f000 rw--sg     WB 0 0 0 0 2
0xffffa0014ff9f000-0xffffa0014ffa0000 rw--sg     UC 0 0 0 0 1
0xffffa0014ffa0000-0xffffa0014ffa2000 rw--sg     WB 0 0 0 0 2
0xffffa0014ffa2000-0xffffa0014ffb5000 rw--sg     UC 0 0 0 0 19
0xffffa0014ffb5000-0xffffa00380000000 rw--sg     WB 8 24 0 4 11
0xffffa00780000000-0xffffa00c00000000 rw--sg     WB 18 0 0 0 0

Diff Detail

Repository

rG FreeBSD src repository

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

alc created this revision.Thu, May 16, 5:58 PM

Herald added a reviewer: andrew. · View Herald TranscriptThu, May 16, 5:58 PM

Herald added a reviewer: andrew. · View Herald Transcript

Herald added a reviewer: manu. · View Herald Transcript

Herald added a subscriber: emaste. · View Herald Transcript

alc requested review of this revision.Thu, May 16, 5:58 PM

@gallatin @markj Could you please test this patch? I've also tested this patch on EC2 VMs with both 4K and 16K base pages, but their device mappings don't trigger any L1 or L2C demotions in the direct map.

In D45224#1031599, @alc wrote:

@gallatin @markj Could you please test this patch? I've also tested this patch on EC2 VMs with both 4K and 16K base pages, but their device mappings don't trigger any L1 or L2C demotions in the direct map.

I tried this on my ampere altra test box. Our (netflix) tree is roughly 1 month old (based at 63b0165cdcbb178df63ac57dcd39c29cf77f346e).
I cherry-picked a803837cec6e17e04849d59afac7b6431c70cb93 c1ebd76c3f283b10afe6b64f29fe68c4d1be3f8c b5a1f0406b9d6bba28e57377dcfc8b83bce987ad and then applied the patch. Its possible that I missed something critical elsewhere.. I just looked at the history of sys/arm64/arm64/pmap.c and cherry-picked what seemed to have come in after our latest upstream sync.

In the new kernel, I see:

# sysctl vm.pmap
vm.pmap.l3c.promotions: 4284
vm.pmap.l3c.p_failures: 156
vm.pmap.l3c.mappings: 348
vm.pmap.l3c.demotions: 795
vm.pmap.l2.promotions: 276
vm.pmap.l2.p_failures: 0
vm.pmap.l2.mappings: 0
vm.pmap.l2.demotions: 120
vm.pmap.l2c.demotions: 0
vm.pmap.l1.demotions: 0
vm.pmap.superpages_enabled: 1
vm.pmap.vmid.epoch: 0
vm.pmap.vmid.next: 2
vm.pmap.vmid.bits: 16
vm.pmap.asid.epoch: 0
vm.pmap.asid.next: 12760
vm.pmap.asid.bits: 16
# sysctl vm.pmap.kernel_maps | tail -15
0xffff001d14fb8000-0xffff001d17b0c000 rw--sg     WB 0 0 0 21 85
0xffff007fffe24000-0xffff007fffffc000 rw--sg     WT 0 0 0 0 118
0xffff007fffffc000-0xffff008000000000 rw--sg    DEV 0 0 0 0 1

Direct map:
0xffffa00088300000-0xffffa00088400000 rw--sg     WB 0 0 0 0 64
0xffffa00090000000-0xffffa000ebf28000 rw--sg     WB 0 0 45 15 74
0xffffa000ec308000-0xffffa000ec30c000 rw--sg     WB 0 0 0 0 1
0xffffa000ec310000-0xffffa000ec530000 rw--sg     WB 0 0 0 0 136
0xffffa000ec540000-0xffffa000ee550000 rw--sg     WB 0 0 0 15 132
0xffffa000eeab8000-0xffffa000ffc8c000 rw--sg     WB 0 0 7 24 117
0xffffa000ffc90000-0xffffa00100000000 rw--sg     WB 0 0 0 1 92
0xffffa80000000000-0xffffa80080000000 rw--sg     WB 0 2 0 0 0
0xffffa80100000000-0xffffa82000000000 rw--sg     WB 0 124 0 0 0

In D45224#1031950, @gallatin wrote:
In D45224#1031599, @alc wrote:

@gallatin @markj Could you please test this patch? I've also tested this patch on EC2 VMs with both 4K and 16K base pages, but their device mappings don't trigger any L1 or L2C demotions in the direct map.

I tried this on my ampere altra test box. Our (netflix) tree is roughly 1 month old (based at 63b0165cdcbb178df63ac57dcd39c29cf77f346e).
I cherry-picked a803837cec6e17e04849d59afac7b6431c70cb93 c1ebd76c3f283b10afe6b64f29fe68c4d1be3f8c b5a1f0406b9d6bba28e57377dcfc8b83bce987ad and then applied the patch. Its possible that I missed something critical elsewhere.. I just looked at the history of sys/arm64/arm64/pmap.c and cherry-picked what seemed to have come in after our latest upstream sync.

In the new kernel, I see:
# sysctl vm.pmap
vm.pmap.l3c.promotions: 4284
vm.pmap.l3c.p_failures: 156
vm.pmap.l3c.mappings: 348
vm.pmap.l3c.demotions: 795
vm.pmap.l2.promotions: 276
vm.pmap.l2.p_failures: 0
vm.pmap.l2.mappings: 0
vm.pmap.l2.demotions: 120
vm.pmap.l2c.demotions: 0
vm.pmap.l1.demotions: 0
vm.pmap.superpages_enabled: 1
vm.pmap.vmid.epoch: 0
vm.pmap.vmid.next: 2
vm.pmap.vmid.bits: 16
vm.pmap.asid.epoch: 0
vm.pmap.asid.next: 12760
vm.pmap.asid.bits: 16
# sysctl vm.pmap.kernel_maps | tail -15
0xffff001d14fb8000-0xffff001d17b0c000 rw--sg     WB 0 0 0 21 85
0xffff007fffe24000-0xffff007fffffc000 rw--sg     WT 0 0 0 0 118
0xffff007fffffc000-0xffff008000000000 rw--sg    DEV 0 0 0 0 1

Direct map:
0xffffa00088300000-0xffffa00088400000 rw--sg     WB 0 0 0 0 64
0xffffa00090000000-0xffffa000ebf28000 rw--sg     WB 0 0 45 15 74
0xffffa000ec308000-0xffffa000ec30c000 rw--sg     WB 0 0 0 0 1
0xffffa000ec310000-0xffffa000ec530000 rw--sg     WB 0 0 0 0 136
0xffffa000ec540000-0xffffa000ee550000 rw--sg     WB 0 0 0 15 132
0xffffa000eeab8000-0xffffa000ffc8c000 rw--sg     WB 0 0 7 24 117
0xffffa000ffc90000-0xffffa00100000000 rw--sg     WB 0 0 0 1 92
0xffffa80000000000-0xffffa80080000000 rw--sg     WB 0 2 0 0 0
0xffffa80100000000-0xffffa82000000000 rw--sg     WB 0 124 0 0 0

Thank you. This looks okay. The last two entries show 2 and 124 1 GB page mappings being used instead of 32 MB page mappings.

I was just hoping for another machine besides the DevKit that would have to perform the modified L1 and new L2C demotion code.

P.S. You would also benefit from my recent change to locore.S.

Here are the vm.pmap counters and the direct map portion of vm.pmap.kernel_maps on a 4 GB Raspberry Pi 4:

vm.pmap.l3c.promotions: 9279
vm.pmap.l3c.p_failures: 450
vm.pmap.l3c.mappings: 498
vm.pmap.l3c.demotions: 216
vm.pmap.l2.promotions: 95
vm.pmap.l2.p_failures: 360
vm.pmap.l2.mappings: 0
vm.pmap.l2.demotions: 69
vm.pmap.l2c.demotions: 2
vm.pmap.l1.demotions: 0
vm.pmap.superpages_enabled: 1
vm.pmap.vmid.epoch: 0
vm.pmap.vmid.next: 2
vm.pmap.vmid.bits: 8
vm.pmap.asid.epoch: 0
vm.pmap.asid.next: 784
vm.pmap.asid.bits: 16

Direct map:
0xffffa00000002000-0xffffa000009bc000 rw--sg     WB 0 0 3 58 26
0xffffa000009bc000-0xffffa000009c3000 rw--sg     UC 0 0 0 0 7
0xffffa000009c3000-0xffffa00000d3d000 rw--sg     WB 0 0 1 22 26
0xffffa00000d3d000-0xffffa00000d3e000 rw--sg     UC 0 0 0 0 1
0xffffa00000d3e000-0xffffa00000d4e000 rw--sg     WB 0 0 0 0 16
0xffffa00000d4e000-0xffffa00000d63000 rw--sg     UC 0 0 0 0 21
0xffffa00000d63000-0xffffa00007386000 rw--sg     WB 0 2 18 33 19
0xffffa00007386000-0xffffa00007392000 rw--sg     UC 0 0 0 0 12
0xffffa00007392000-0xffffa00033adc000 rw--sg     WB 0 21 19 19 26
0xffffa00033adc000-0xffffa00033add000 rw--sg     UC 0 0 0 0 1
0xffffa00033add000-0xffffa00033adf000 rw--sg     WB 0 0 0 0 2
0xffffa00033adf000-0xffffa00033ae0000 rw--sg     UC 0 0 0 0 1
0xffffa00033ae0000-0xffffa00033ae4000 rw--sg     WB 0 0 0 0 4
0xffffa00033ae4000-0xffffa00033ae5000 rw--sg     UC 0 0 0 0 1
0xffffa00033ae5000-0xffffa00033ae9000 rw--sg     WB 0 0 0 0 4
0xffffa00033ae9000-0xffffa00033aea000 rw--sg     UC 0 0 0 0 1
0xffffa00033aea000-0xffffa00033aec000 rw--sg     WB 0 0 0 0 2
0xffffa00033aec000-0xffffa00033aed000 rw--sg     UC 0 0 0 0 1
0xffffa00033aed000-0xffffa00033af3000 rw--sg     WB 0 0 0 0 6
0xffffa00033af3000-0xffffa00033af4000 rw--sg     UC 0 0 0 0 1
0xffffa00033af4000-0xffffa00033b85000 rw--sg     WB 0 0 0 8 17
0xffffa00033b85000-0xffffa00033ba6000 rw--sg     UC 0 0 0 0 33
0xffffa00033ba6000-0xffffa00033ba8000 rw--sg     WB 0 0 0 0 2
0xffffa00033ba8000-0xffffa00033bb8000 rw--sg     UC 0 0 0 0 16
0xffffa00033bb8000-0xffffa00033e43000 rw--sg     WB 0 0 1 4 75
0xffffa00033e43000-0xffffa00033e44000 rw--sg     UC 0 0 0 0 1
0xffffa00033e44000-0xffffa00033e4e000 rw--sg     WB 0 0 0 0 10
0xffffa00033e4e000-0xffffa00033e4f000 rw--sg     UC 0 0 0 0 1
0xffffa00033e4f000-0xffffa00033e59000 rw--sg     WB 0 0 0 0 10
0xffffa00033e59000-0xffffa00033e5a000 rw--sg     UC 0 0 0 0 1
0xffffa00033e5a000-0xffffa00033e6b000 rw--sg     WB 0 0 0 0 17
0xffffa00033e6b000-0xffffa00033e6c000 rw--sg     UC 0 0 0 0 1
0xffffa00033e6c000-0xffffa00033e72000 rw--sg     WB 0 0 0 0 6
0xffffa00033e72000-0xffffa00033e73000 rw--sg     UC 0 0 0 0 1
0xffffa00033e73000-0xffffa0003b400000 rw--sg     WB 0 3 10 24 13
0xffffa00040000000-0xffffa000fc000000 rw--sg     WB 2 30 0 0 0

markj added inline comments.Mon, May 20, 4:03 PM

sys/arm64/arm64/pmap.c
8349	I wonder if it is worth introducing a macro for this expression? It arises fairly regularly and I have to stare at it for a while to make sure there are no misparenthesizations.
8397	It doesn't make a difference, but this looks like it should formally be `\|=`.

alc marked 2 inline comments as done.Mon, May 20, 6:10 PM

alc added inline comments.

sys/arm64/arm64/pmap.c
8349	I've spent a little time thinking about how to refactor the various demotion functions to reduce duplication, so I'm planning to do something along these lines.
8397	Yes. This applies to a few other functions too, so either Eliot or I will deal with this separately.
8410	I'm planning to change this later too. The L2 entry is invalid, so we really don't need to use fcmpset.