Revert r240317 to prevent leaking pmap entries

[cem]

Summary

Subsequent to r240317, kmem_free() was replaced with kva_free() (r254025).
kva_free() releases the KVA allocation for the mapped region, but no longer
clears the pmap (pagetable) entries.

An affected pmap_unmapdev operation would leave the still-pmap'd VA space
free for allocation by other KVA consumers. However, this bug easily
avoided notice for ~7 years because most devices (1) never call
pmap_unmapdev and (2) on amd64, mostly fit within the DMAP and do not need
KVA allocations. Other affected arch are less popular: i386, MIPS, and
PowerPC. Arm64, arm32, and riscv are not affected.

Reported by: Don Morris <dgmorris AT earthlink.net>
Submitted by: Don Morris (amd64)
Discussed with: markj

[markj]

sys/amd64/amd64/pmap.c

8283

size / PAGE_SIZE is more commonly spelled atop(size).

sys/mips/mips/pmap.c

3267

Why not use qremove? pmap_kremove_device() doesn't perform TLB invalidation, which I presume is why you're using qremove elsewhere. The subr_devmap.c implementation doesn't either, which seems like a bug.

[kib]

sys/mips/mips/pmap.c

3267

I agree. I think that the only problem that this issue introduces is the lack of invalidation on removal, which means that next mapping (which we typically do without invalidating the allocated VA) has undefined consequences. It is almost fine to leave stale PTEs, except for speculations, but not fine to leave TLB unflushed.

[cem]

sys/amd64/amd64/pmap.c

8283

will fix

sys/mips/mips/pmap.c

3267

will do.

[cem]

• pmap_qremove() over pmap_kremove_device() in mips
• atop() for "/ PAGE_SIZE"

[dgmorris_earthlink.net]

LGTM.