Address the COW problem in pmap_enter() on mips

[alc]

Here is an updated table.

                Uses            Has PV Alloc    Has COW         Needs r324665
                pv_chunk        Problem         Bug             and r325285
                -------------------------------------------------------------
amd64           Yes             Fixed           Fixed           Fixed
arm/pmap-v4.c   No              N/A             No[1]           N/A
arm/pmap-v6.c   Yes             No              No[2]           No[4]
arm64           Yes             Fixed           No[2]           Yes
i386            Yes             No              Fixed           No[4]
mips            Yes             No              Patch           No[4]
powerpc/booke   No              N/A             No[2]           N/A
powerpc/oea     No              N/A[3]          No[2]           N/A
powerpc/oea64   No              N/A             No[2]           N/A
powerpc/pseries[5]
riscv           Yes             Yes             Yes             No[4]
sparc64         No              N/A             No[2]           N/A

[1] SMP is not supported.
[2] Performs "break-before-make".
[3] The comments say that the PV entry is reused, but it is not.  That said,
    the old PV entry is freed before the new one is allocated.  I believe
    that reuse could be beneficial because it would eliminate two O(log n)
    Red-Black tree operations.
[4] Still has pvh_global_lock.
[5] Literally derived from powerpc/oea64.

[markj]

BTW, the patch is missing context so it's a bit hard to read.

mips/mips/pmap.c

2116 ↗

(On Diff #45078)

What's the purpose of setting the PTE this way?

[alc]

Provide additional context.

[alc]

mips/mips/pmap.c

2116 ↗

(On Diff #45078)

A single MIPS TLB entry actually maps two neighboring virtual pages, an even numbered virtual page followed by the subsequent odd numbered virtual page. Essentially, the tag associated with the TLB entry excludes the least significant bit of the virtual page number, and that excluded bit selects the even or odd mapping on an access. While it is okay for only one of the mappings in an entry to be valid, the setting of the G bit must always be the same in both mappings, or else the TLB hardware may misbehave.

[alc]

I found an old installation of gxemul on one of my machines, was able to boot a 64-bit kernel with this patch, and complete a kernel compilation. So, I think that this patch is okay to commit.

As an aside, this pmap still needs work. The "reference bit" emulation makes no sense to me and can't possibly lead to sensible results from pmap_ts_referenced(). Is anyone actually using mips these days?

[markj]

In D16199#344830, @alc wrote:

Is anyone actually using mips these days?

I have a tp-link wifi router with a mips core that runs FreeBSD. I haven't used it since last year but I believe there's a number of developers who do. When I did its memory usage would have been pretty static. I can resurrect it and help with testing if you're planning on making further changes to the mips pmap code, but it's a pretty constrained device and not at all suited to running builds.

[alc]

In D16199#344831, @markj wrote:

In D16199#344830, @alc wrote:

Is anyone actually using mips these days?

I have a tp-link wifi router with a mips core that runs FreeBSD. I haven't used it since last year but I believe there's a number of developers who do. When I did its memory usage would have been pretty static. I can resurrect it and help with testing if you're planning on making further changes to the mips pmap code, but it's a pretty constrained device and not at all suited to running builds.

I suspect that your router deployment is typical, and so the machines have no swap space. Consequently, the reference bit emulation doesn't matter.

Also, while the hardware doesn't dictate a specific page table format, it is really designed for a linear, VAX-style page table for user-space, i.e., where the page table is essentially a 1-dimensional array in kernel virtual memory. In particular, the hardware computes the expected position of the PTE so that the TLB miss handler has less to do in software. And, even though the page table is accessed through translated virtual addresses rather than the direct map, so that the entire array doesn't have to be backed by physical pages, a single kernel TLB entry gives you access to an entire page table page worth of user-space PTEs. Essentially, the TLB acts as its own x86-style page walk cache. In a nutshell, the cost of user-space TLB misses will be lower for programs that exhibit decent locality.

[kib]

In D16199#344830, @alc wrote:

As an aside, this pmap still needs work. The "reference bit" emulation makes no sense to me and can't possibly lead to sensible results from pmap_ts_referenced(). Is anyone actually using mips these days?

Ref bit is is minor problem for MIPS, comparing with the data corruption on virtual-indexed cache machines.

[jhb]

My usage of MIPS is confined to qemu for CHERI stuff. CheriBSD has various pmap changes for MIPS though none that fix the issue kib@ noted. In particular CheriBSD has a new map for mips64 that is derived from the amd64 pmap and supports superpages. It also supports using large pages for the kstack which is really needed for newer compilers which bloat kstack usage. (Though one thing that isn't clear to me is why we don't just contigalloc kstacks for mips and then use kseg0 addresses rather than relying on pinned TLB entries for the kstack?)

[alc]

In D16199#344896, @jhb wrote:

... It also supports using large pages for the kstack which is really needed for newer compilers which bloat kstack usage. (Though one thing that isn't clear to me is why we don't just contigalloc kstacks for mips and then use kseg0 addresses rather than relying on pinned TLB entries for the kstack?)

If you switch to using kseg0 addresses, then you are abandoning guards. That's the only downside that I see to it.