Here is a very simple example. Before the change:

97156     0x2eb08e600000     0x2eb08e64b000 rw-   37   37   1   0 ----- sw
97156     0x2eb08e800000     0x2eb08f021000 rw- 1031 29378   3   0 ----- sw
97156     0x2eb08f021000     0x2eb08f022000 rw-    1 29378   3   0 ----- sw
97156     0x2eb08f022000     0x2eb095f22000 rw- 28346 29378   3   0 --S-- sw
97156     0x2eb096000000     0x2eb09d000000 rw- 20411 20411   1   0 --S-- sw

After the change:

17423     0x258f55e00000     0x258f55e6c000 rw-   38   39   2   0 ----- sw
17423     0x258f55e6c000     0x258f55e6d000 rw-    1   39   2   0 ----- sw
17423     0x258f56000000     0x258f5d700000 rw- 29483 29483   1   0 --S-- sw
17423     0x258f5d800000     0x258f67000000 rw- 27646 27646   1   0 --S-- sw

With this patch in place, I see a small reduction in reservations allocated (e.g., 1.6% during buildworld), fewer partially populated reservations. a small increase in 64KB page promotions on arm64, and fewer map entries in the heap.

I'm going to suggest the following as the final resolution for this issue. It addresses the fragmentation that I described in my last message, and it and handles the scenario that you brought up wherein we run out of free space at the current anon_loc, perform an ASLR restart, and want to avoid repeating the ASLR restart on the next mapping.

diff --git a/sys/vm/vm_map.c b/sys/vm/vm_map.c
index 3c7afcb6642f..fa71bb8a01d6 100644
--- a/sys/vm/vm_map.c
+++ b/sys/vm/vm_map.c
@@ -2247,8 +2247,15 @@ vm_map_find(vm_map_t map, vm_object_t object, vm_ooffset_t offset,
                rv = vm_map_insert(map, object, offset, *addr, *addr + length,
                    prot, max, cow);
        }
-       if (rv == KERN_SUCCESS && update_anon)
-               map->anon_loc = *addr + length;
+
+       /*
+        * Update the starting address for clustered anonymous memory mappings
+        * if a starting address was not previously defined or an ASLR restart
+        * placed an anonymous memory mapping at a lower address.
+        */
+       if (update_anon && rv == KERN_SUCCESS && (map->anon_loc == 0 ||
+           *addr < map->anon_loc))
+               map->anon_loc = *addr;
 done:
        vm_map_unlock(map);
        return (rv);
@@ -4041,9 +4048,6 @@ vm_map_delete(vm_map_t map, vm_offset_t start, vm_offset_t end)
                    entry->object.vm_object != NULL)
                        pmap_map_delete(map->pmap, entry->start, entry->end);

I'm not convinced that we should be creating an ilog2.h header file. I would leave the definitions in the existing libkern.h header file.

By the way, you mentioned pmap_enter_l3c() only being called from one place. The next big chunk of @ehs3_rice.edu 's work will include a call to pmap_enter_l3c() from pmap_enter() with pagesizes[] updated to include the L3C page size as psind == 1.

Add KASSERT.

Remove two early calls to pmap_pte_bti() that are now redundant.

@kib Do you have any PKU test programs that could be run to check this?

Correct a couple errors in the previous version.

In principle, this is fine, but in practice. we mostly call vm_page_insert_after(). So, the impact will be limited. That said, I'm still excited by where I think this is headed.

For some reason, this revision did not automatically close upon commit.

Please coordinate this with @markj given his lazy init change.

@jhibbits I would encourage you to apply https://reviews.freebsd.org/D40478 and a few other superpage related follow on commits from amd64 to mmu_radix,c. This change would then apply to mmu_radix.c.

In D40403#1036869, @markj wrote:

@alc @dougm Do you have any more comments on this patch?

In D40403#1036869, @markj wrote:

@alc @dougm Do you have any more comments on this patch?

In D45431#1036870, @jhibbits wrote:

From a (very) quick check, it looks like the same change in amd64 should be made to mmu_radix for powerpc.

Set VM_PROT_NO_PROMOTE in vm_fault_prefault().

Revise comment.

@markj Do you have any other comments on this change?

In D45224#1031950, @gallatin wrote:

In D45224#1031599, @alc wrote:

@gallatin @markj Could you please test this patch? I've also tested this patch on EC2 VMs with both 4K and 16K base pages, but their device mappings don't trigger any L1 or L2C demotions in the direct map.

I tried this on my ampere altra test box. Our (netflix) tree is roughly 1 month old (based at 63b0165cdcbb178df63ac57dcd39c29cf77f346e).
I cherry-picked a803837cec6e17e04849d59afac7b6431c70cb93 c1ebd76c3f283b10afe6b64f29fe68c4d1be3f8c b5a1f0406b9d6bba28e57377dcfc8b83bce987ad and then applied the patch. Its possible that I missed something critical elsewhere.. I just looked at the history of sys/arm64/arm64/pmap.c and cherry-picked what seemed to have come in after our latest upstream sync.

In the new kernel, I see:

# sysctl vm.pmap
vm.pmap.l3c.promotions: 4284
vm.pmap.l3c.p_failures: 156
vm.pmap.l3c.mappings: 348
vm.pmap.l3c.demotions: 795
vm.pmap.l2.promotions: 276
vm.pmap.l2.p_failures: 0
vm.pmap.l2.mappings: 0
vm.pmap.l2.demotions: 120
vm.pmap.l2c.demotions: 0
vm.pmap.l1.demotions: 0
vm.pmap.superpages_enabled: 1
vm.pmap.vmid.epoch: 0
vm.pmap.vmid.next: 2
vm.pmap.vmid.bits: 16
vm.pmap.asid.epoch: 0
vm.pmap.asid.next: 12760
vm.pmap.asid.bits: 16
# sysctl vm.pmap.kernel_maps | tail -15
0xffff001d14fb8000-0xffff001d17b0c000 rw--sg     WB 0 0 0 21 85
0xffff007fffe24000-0xffff007fffffc000 rw--sg     WT 0 0 0 0 118
0xffff007fffffc000-0xffff008000000000 rw--sg    DEV 0 0 0 0 1

Direct map:
0xffffa00088300000-0xffffa00088400000 rw--sg     WB 0 0 0 0 64
0xffffa00090000000-0xffffa000ebf28000 rw--sg     WB 0 0 45 15 74
0xffffa000ec308000-0xffffa000ec30c000 rw--sg     WB 0 0 0 0 1
0xffffa000ec310000-0xffffa000ec530000 rw--sg     WB 0 0 0 0 136
0xffffa000ec540000-0xffffa000ee550000 rw--sg     WB 0 0 0 15 132
0xffffa000eeab8000-0xffffa000ffc8c000 rw--sg     WB 0 0 7 24 117
0xffffa000ffc90000-0xffffa00100000000 rw--sg     WB 0 0 0 1 92
0xffffa80000000000-0xffffa80080000000 rw--sg     WB 0 2 0 0 0
0xffffa80100000000-0xffffa82000000000 rw--sg     WB 0 124 0 0 0

@gallatin @markj Could you please test this patch? I've also tested this patch on EC2 VMs with both 4K and 16K base pages, but their device mappings don't trigger any L1 or L2C demotions in the direct map.

Add requirements comment.

In D45042#1028354, @gallatin wrote:

In D45042#1028058, @alc wrote:

In D45042#1027957, @markj wrote:

Do we have any idea what the downsides of the change are? If we make the default 64KB, then I'd expect memory usage to increase; do we have any idea what the looks like? It'd be nice to, for example, compare memory usage on a newly booted system with and without this change.

I had the same question. It will clearly impact a lot of page granularity counters, at the very least causing some confusion for people who look at those counters, e.g.,

./include/jemalloc/internal/arena_inlines_b.h-          arena_stats_add_u64(tsdn, &arena->stats,
./include/jemalloc/internal/arena_inlines_b.h-              &arena->decay_dirty.stats->nmadvise, 1);
./include/jemalloc/internal/arena_inlines_b.h-          arena_stats_add_u64(tsdn, &arena->stats,
./include/jemalloc/internal/arena_inlines_b.h:              &arena->decay_dirty.stats->purged, extent_size >> LG_PAGE);
./include/jemalloc/internal/arena_inlines_b.h-          arena_stats_sub_zu(tsdn, &arena->stats, &arena->stats.mapped,
./include/jemalloc/internal/arena_inlines_b.h-              extent_size);

However, it's not so obvious what the effect on the memory footprint will be. For example, the madvise(MADV_FREE) calls will have coarser granularity. If we set the page size to 64KB, then one in-use 4KB page within a 64KB region will be enough to block the application of madvise(MADV_FREE) to the other 15 pages. Quantifying the impact that this coarsening has will be hard.

This does, however, seem to be the intended workaround: https://github.com/jemalloc/jemalloc/issues/467

Buried in that issue is the claim that Firefox's builtin derivative version of jemalloc eliminated the statically compiled page size.

What direction does the kernel grow the vm map? They apparently reverted support for lg page size values larger than the runtime page size because it caused fragmentation when the kernel grows the vm map downwards..

In D45042#1027957, @markj wrote:

Do we have any idea what the downsides of the change are? If we make the default 64KB, then I'd expect memory usage to increase; do we have any idea what the looks like? It'd be nice to, for example, compare memory usage on a newly booted system with and without this change.

Eliminate unnecessary parentheses.

In D44677#1022012, @andrew wrote:

We could look at that as a follow up, however I'm unlikely to have time to make such a change and have it ready and well tested for 14.1 given it's due to be branched in just over 2 weeks.

In D44677#1021479, @alc wrote:

Why not deal with this issue in pmap_mapdev{,_attr}()? Specifically, if the given physical address falls within the DMAP region, don't call kva_alloc{,_aligned}(); instead, map the physical address at its corresponding DMAP virtual address. This is not all that different from amd64, where the DMAP (with appropriate attr settings) is used to access a good bit of device memory.

Why not deal with this issue in pmap_mapdev{,_attr}()? Specifically, if the given physical address falls within the DMAP region, don't call kva_alloc{,_aligned}(); instead, map the physical address at its corresponding DMAP virtual address. This is not all that different from amd64, where the DMAP (with appropriate attr settings) is used to access a good bit of device memory.

In D38852#1018877, @kib wrote:

In D38852#1018757, @alc wrote:

Just so we're all on the same page, I want to point out the following: While this patch achieves contiguity, it doesn't guarantee 2 MB alignment. Let 'F' represent a fully populated 2 MB reservation, 'E', represent a partially populated reservation, where the population begins in the middle and goes to the end, and 'B' is the complement of 'E', where the population begins at the start and ends in the middle. Typically, the physical memory allocation for one chunk of stacks on amd64 looks like 'EFFFB'. While it would be nice to achieve 'FFFF', this patch is already a great improvement over the current state of affairs.

But is it possible at all (perhaps a better word is it worth at all) since we do have the guard pages?

Just so we're all on the same page, I want to point out the following: While this patch achieves contiguity, it doesn't guarantee 2 MB alignment. Let 'F' represent a fully populated 2 MB reservation, 'E', represent a partially populated reservation, where the population begins in the middle and goes to the end, and 'B' is the complement of 'E', where the population begins at the start and ends in the middle. Typically, the physical memory allocation for one chunk of stacks on amd64 looks like 'EFFFB'. While it would be nice to achieve 'FFFF', this patch is already a great improvement over the current state of affairs.

Update to reflect committed change.

Eliminate an unnecessary variable.