riscv/pmap.c: Add an Svinval-aware variant of pmap_invalidate_range

[bnovkov]

Summary

This change introduces a variant of pmap_invalidate_range that uses
the fine-grained TLB invalidation instructions introduced by the
Svinval extension. These instructions allow for more efficient TLB
flushing on certain implementations. Under this new scheme,
pmap_invalidate_range was converted to an ifunc that selects the
appropriate variant during boot.

Event: BSDCan 2026

[mhorne]

sys/riscv/riscv/pmap.c

1115–1116

[mhorne]

sys/riscv/riscv/pmap.c

1102

The relationship between this check and the definition above was a momentary confusion for me.

You might do away with the atop() and PAGE_SIZE division altogether.

[markj]

Why not use this for pmap_invalidate_page() as well?

[bnovkov]

Address @markj 's and @mhorne 's comments:

• add an svinval-aware pmap_invalidate_page variant
• remove awkward atop conversion

[markj]

sys/riscv/riscv/pmap.c

1129

Why is this done in the callback? Shouldn't we just fall back to a direct pmap_invalidate_all() call?

1152

I think you want to pass smp_no_rendezvous_barrier instead of NULL here, otherwise all CPUs will needlessly synchronize before invoking the callback, and again after. See smp_rendezvous_action().

[bnovkov]

Address @markj 's comments

[bnovkov]

sys/riscv/riscv/pmap.c

1129

No particular reason, it seemed like the most straightforward solution at the time.
I have no preferences here, I've worked your suggestion into the patch.

[markj]

sys/riscv/riscv/pmap.c

1151

Why exactly do we need this fence? Is it just copied from the SBI implementation?

1157

So above we invoked the callback on all harts except this one, waiting for them to finish, and now we're doing the invalidation on the current hart. Why not do them all in parallel?

[bnovkov]

sys/riscv/riscv/pmap.c

1151

The fence here follows the convention outlined in the comment above pmap_invalidate_page_sbi, which was introduced back in D17414.
I assume it's here because sfence_vmas that target an address only apply to the leaf PTEs, but I'm not sure.

1157

Right, that pattern only makes sense in the sbi_* variants where I assume we want to avoid calling into openSBI.
I'll include the original cpu in the mask as well.

[bnovkov]

Avoid clearing CPU mask for range invalidation

[markj]

Did you microbenchmark the new implementation? I'd be curious to know, say, how many cycles it takes to execute 10000 back-to-back 4KB page invalidations using the old and new implementations respectively, when the target CPU mask is just the current CPU and when it's the set of all CPUs.

[bnovkov]

In D57624#1336128, @markj wrote:

Did you microbenchmark the new implementation? I'd be curious to know, say, how many cycles it takes to execute 10000 back-to-back 4KB page invalidations using the old and new implementations respectively, when the target CPU mask is just the current CPU and when it's the set of all CPUs.

Unfortunately no, I haven't had the time to run a proper microbenchmark yet.

[markj]

There is an unfortunate bug in this change: the pm_active bitmask is indexed by hart IDs, not CPU IDs, and these are not the same in general, see e.g., the output from QEMU below:

FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs                                                                                                            
CPU 1  : Vendor=Unspecified Core=Unknown (Hart 14)                                                                                                             
CPU 2  : Vendor=Unspecified Core=Unknown (Hart 15)                                                                                                                                                                                                                                                                            
CPU 3  : Vendor=Unspecified Core=Unknown (Hart 0)                                                                                                              
CPU 4  : Vendor=Unspecified Core=Unknown (Hart 1)                                                                                                              
CPU 5  : Vendor=Unspecified Core=Unknown (Hart 2)                                                                                                                                                                                                                                                                             
CPU 6  : Vendor=Unspecified Core=Unknown (Hart 3)                                                                                                              
CPU 7  : Vendor=Unspecified Core=Unknown (Hart 4)                                                                                                                                                                                                                                                                             
CPU 8  : Vendor=Unspecified Core=Unknown (Hart 5)                                                                                                              
CPU 9  : Vendor=Unspecified Core=Unknown (Hart 6)                                                                                                              
CPU 10 : Vendor=Unspecified Core=Unknown (Hart 7)                                                                                                              
CPU 11 : Vendor=Unspecified Core=Unknown (Hart 8)                                                                                                              
CPU 12 : Vendor=Unspecified Core=Unknown (Hart 9)                                                                                                              
CPU 13 : Vendor=Unspecified Core=Unknown (Hart 10)                                                                                                             
CPU 14 : Vendor=Unspecified Core=Unknown (Hart 11)                                                                                                             
CPU 15 : Vendor=Unspecified Core=Unknown (Hart 12)

So, when we pass pm_active as the target cpuset to smp_rendezvous_cpus(), we end up signaling an unrelated set of CPUs.

[markj]

In D57624#1377340, @markj wrote:

There is an unfortunate bug in this change: the pm_active bitmask is indexed by hart IDs, not CPU IDs.

This also means that pmap_activate_cpus() is broken, but that has relatively limited fallout.