In D44677#1021479, @alc wrote:

Why not deal with this issue in pmap_mapdev{,_attr}()? Specifically, if the given physical address falls within the DMAP region, don't call kva_alloc{,_aligned}(); instead, map the physical address at its corresponding DMAP virtual address. This is not all that different from amd64, where the DMAP (with appropriate attr settings) is used to access a good bit of device memory.

Why not deal with this issue in pmap_mapdev{,_attr}()? Specifically, if the given physical address falls within the DMAP region, don't call kva_alloc{,_aligned}(); instead, map the physical address at its corresponding DMAP virtual address. This is not all that different from amd64, where the DMAP (with appropriate attr settings) is used to access a good bit of device memory.

In D38852#1018877, @kib wrote:

In D38852#1018757, @alc wrote:

Just so we're all on the same page, I want to point out the following: While this patch achieves contiguity, it doesn't guarantee 2 MB alignment. Let 'F' represent a fully populated 2 MB reservation, 'E', represent a partially populated reservation, where the population begins in the middle and goes to the end, and 'B' is the complement of 'E', where the population begins at the start and ends in the middle. Typically, the physical memory allocation for one chunk of stacks on amd64 looks like 'EFFFB'. While it would be nice to achieve 'FFFF', this patch is already a great improvement over the current state of affairs.

But is it possible at all (perhaps a better word is it worth at all) since we do have the guard pages?

Just so we're all on the same page, I want to point out the following: While this patch achieves contiguity, it doesn't guarantee 2 MB alignment. Let 'F' represent a fully populated 2 MB reservation, 'E', represent a partially populated reservation, where the population begins in the middle and goes to the end, and 'B' is the complement of 'E', where the population begins at the start and ends in the middle. Typically, the physical memory allocation for one chunk of stacks on amd64 looks like 'EFFFB'. While it would be nice to achieve 'FFFF', this patch is already a great improvement over the current state of affairs.

Update to reflect committed change.

Eliminate an unnecessary variable.

In D44575#1017895, @markj wrote:

In D44575#1016703, @andrew wrote:

I do, however, want to point out that a good portion of the reduction in buildworld time is coming from performing a smaller number of icache flushes when creating executable mappings.

Have you looked at teaching the vm code to manage the icache? We currently call cpu_icache_sync_range more than we need to, e.g. if mapping the same physical address as twice we will call it twice.

We discussed this a while ago. To minimize icache syncing, I believe we need to identify all of the places where a kernel might modify a user-mapped page via the direct map. I think that hooking uiomove_* would get us most of the way there, but it's hard to be confident that that's sufficient.

Add KASSERT to vm_reserv_is_populated().

Rename pmap_enter_object()'s helpers to not have 64k or 2m in their names.

Reopen after reservation size fix was committed.

Correct VM_NFREEORDER for 16KB page size.

Teach sysctl vm.pmap.kernel_maps to correctly count ATTR_CONTIGUOUS superpages when the base page size is 16KB.

Add (void) casts. Refill a comment whose lines were too long.

I'd really like to see this committed.

Despite the long name, it's still two characters shorter than the original code. :-)

What happens if we increase VM_NFREEORDER, e.g., 262144, to support 1 GB allocations? I think that you might might want to have a different constant to cap the lazy init chunk size.

Eliot and I have addressed most of Mark's comments.

In D42737#975859, @markj wrote:

In D42737#975855, @alc wrote:

Thanks, Mark. We'd like to see the output from sysctl vm.pmap.kernel_maps too.

That's what I provided in the links - did you mean something else?

In both cases, this was the output of sysctl vm.pmap.kernel_maps immediately after booting. I can grab it from after the buildworld too, if that's useful.

In D42737#975855, @alc wrote:

Thanks, Mark. We'd like to see the output from sysctl vm.pmap.kernel_maps too.

Thanks, Mark. We'd like to see the output from sysctl vm.pmap.kernel_maps too.

There was one function where PTE_TO_PHYS/PHYS_TO_PTE conversion hadn't been done yet.

pmap_kremove_device: fix 2MB mapping removal; optimize TLB invalidation

I'm convinced that it's correct. That said, even the person who reported the problem is probably not exercising this while loop:

while (!vm_addr_ok(VM_PAGE_TO_PHYS(m_ret),
    size, alignment, boundary) &&
    ...

We may want to issue this as an errata for 14.0, so can we have a minimal version without style changes?

In D42512#970386, @markj wrote:

I'm kind of skeptical that this man page is useful at all. vm_page_advise() really exists just to implement the madvise() system call, it's not going to be called externally. And, it's just a piece of that system call. vm_map_madvise(), vm_object_madvise(), pmap_advise() also implement parts of madvise(); why are they not documented as well? IMHO it's somewhat more useful to focus on improving madvise.3, which is user-facing and describes the user-visible effect for each type of advice. With a high-level understanding of what MADV_FREE etc. are supposed to do, it becomes easier to understand why vm_page_advise() does what it does.

In D40772#966839, @eugen_grosbein.net wrote:

Note this became more important since we have ASLR turned on for 64 bit processes since 13.2-RELEASE. And ASLR adds great deal of fragmentation. It leads to significant performance degradation over long run due to superpages becoming unusable due memory fragmentation.

In D35709#961603, @mhorne wrote:

In D35709#961602, @alc wrote:

Before I forget, we also ran into a second bug: Specifying ",usr" on a counter no longer worked. Essentially, the ",usr" was just being ignored. We haven't checked if any of the recent changes might have addressed this issue.

I am not surprised, and I doubt anyone has fixed it (though it should be straightforward). This is on an AMD machine?

In D35709#961583, @mhorne wrote:

In D35709#950482, @alc wrote:

In D35709#950481, @mhorne wrote:

In D35709#950480, @alc wrote:

One of my graduate students found that this change had a seriously bad side effect. Specifically, on a Ryzen processor, instead of being able to collect data from 6 counters simultaneously, he could only configure 3 counters. So, we backed out this change locally.

Thanks Alan, I have found the same, and I have a fix for it. The problem is that we now allocate the requested event twice on CPU 0, thus reducing the total number of available counters by two.

I will put the fix up for review within the next week, and make sure it is present in 14.0.

I forgot to follow up here. The fix (c362fe939f6f) has landed in stable/14, and I will request to merge it to releng/14.0 on Thursday.

That's good to hear!

Out of curiosity, is anyone working on PEBS/IBS support?

Sort of. @br has developed a new "Hardware Tracing" framework, separate from pmc/hwpmc, which aims to enable these types of profiling features. The work currently focuses on supporting Coresight/ARM SPE, rather than x86, but this paves the way so that adding classes for e.g. Intel PT will be the "easy part".

See D40466, D40477, D40728.

@markj Are you overloaded? Should I commit this?

For the most part, these changes are setting the guarded bit on mappings that are already no-execute. Is there a reason to do that? To be clear, I don't object to doing so. It just seems redundant.

In D42080#960496, @markj wrote:

Can you set this flag on intermediate translation table entries as well?

In D39845#952794, @kib wrote:

In D39845#952791, @alc wrote:

@kib, it has never been clear to me why we update anon_loc. Is it meant solely as an optimization to free address space finding, or is it done for some other reason?

I probably never ever considered not updating anon_loc. When I asked myself why, the most obvious reason is the failure mode: if looking for free space after anon_loc failing, we need to move the clustering point somewhere, otherwise it would fail on the first try forever.

In D41635#957607, @markj wrote:

In D41635#954945, @alc wrote:

@markj I think that you should apply 34eeabff5a8636155bb02985c5928c1844fd3178 to i386 and riscv. Otherwise, I suspect that there will be assertion failures in places like pmap_remove_pde():

KASSERT(vm_page_all_valid(mpte),
    ("pmap_remove_pde: pte page not promoted"));

I was away for a few weeks and am only now getting caught up. I can spend some time on the riscv pmap if you aren't already working on it.

I've just committed 902ed64fecbe which eliminates the need for zeroing the PTP and the possibility of an assertion failure if you pass the same arguments to pmap_insert_pt_page() as you did on amd64. pmap_insert_pt_page() now has the needed extra argument.