Add support to demote and promote pages

[andrew]

Summary

We will promote a full set of level 3 pages to a single level 2 page
in pmap_enter.

We demote pages when changing attributes, or removing pages that have
been promoted. The former will happen if we are asked to map a page with
attributes different than what the DMAP page contains. In this case
we need to demote the DMAP page and set the new attribute. For the latter
we need to handle the reverse of the pmap_enter case to split the page
when needed.

Test Plan

Tested on a ThunderX while forcing all DMA mappings to be uncached as
this will call pmap_page_set_memattr and pmap_change_attr.

[mmel]

In pmap_demote_<foo>, you must temporary unmap relative large portion of DMAP.
How is ensured that this memory is not accessed by other threads on different CPUs.
You can umap vital kernel data (kernel stack, loaded modules code...) by this, no?

sys/arm64/arm64/pmap.c

3503

The given page can be mapped by multiple times, but attributes are changed only in DMAP. ARM doesn't allows multiple mapping with different page attributes.

3604

Why idcache ?

3609

Isn't tmpva just unmapped?

3667

New entries are not synced.

3691

This invalidate is superfluous, unmapped TLB entries cannot be cached.

[andrew]

sys/arm64/arm64/pmap.c

3503

How would this happen? My understanding is the VM code will only call this with unmapped pages.

3609

No, it is the virtual address we are currently changing. It will have been mapped in pmap_demote_l2.

[andrew]

Update to handle both promotion and demotion

[kib]

sys/arm64/arm64/pmap.c

868

On amd64/i386, the same knob is called vm.pmap.pg_ps_enabled. Not sure if it is required to have consistent name there, but at least this is something to consider.

3911

I remember that I asked the same question when reviewing the similar arm changes (break-before-make).

Could transient page fault occur recursively ? In other words, could you get the fault while the thread owns pmap lock ? E.g., might it happen that interrupt handler accesses the DMAP mapping which is being promoted or demoted by the same CPU and interrupted ?

[andrew]

sys/arm64/arm64/pmap.c

868

It looked like this was named after the flag in the pagetables. On armv6 is seems to be vm.pmap.sp_enabled. I don't have an opinion on the correct name.

3911

I can change the critical section to a spin lock to disable interrupts. I don't expect any other exception to be triggered within the break-before-make section, and any that do would indicate a bug in the break-before-make section.

[andrew]

Use spinlocks to disable interrupts in the break-before-make section

[kib]

spinlock_enter() is not a general-purpose KPI to disable interrupts. MD code uses machine-specific primitives to disable interrupts.

Did you measured whether the promotions and demotions with your patch occur at all ? As I understand, you do not operate on userspace mappings, de-facto limiting the pmap to kernel mappings. x86s have basic counters for promotions and demotions events available always.

[andrew]

Use intr_disable to disable interrupts.
Stop promoting unmanaged pages, until all needed demotions are ready

[andrew]

Demote when entering a new va with pmap_enter.
Add pmap_demote_l2_locked. It doesn't lock, but will in later revisions.

[andrew]

I have seen promotions & demotions, but as part of a larger patch that still needs testing. I'd like to get this in so we can allocate uncached memory, e.g. for DMA.