arm64 pmap: use range-based TLBI instructions

[alc]

Summary

Rewrite pmap_s1_invalidate_strided() to use range-based TLBI instructions when available. This change can significantly reduce the number of invalidation instructions issued. For example, because of the break-before-make requirement, a superpage promotion can issue as many invalidation instructions as there are pages being promoted. With this change, a single invalidation instruction suffices.

Assisted-by: Claude Code (Opus 5)

Test Plan

Tested on EC2 m8g.2xlarge.

[kib]

sys/arm64/arm64/pmap.c

2177

Why this macro cannot be a function?

[alc]

sys/arm64/arm64/pmap.c

2177

As a way to succinctly parameterize ("kvsu") the selection between pmap_s1_invalidate_range_{kernel,user} and pmap_s1_invalidate_{kernel,user}.

[markj]

sys/arm64/arm64/pmap.c

2177

It might be clearer to have this be an __always_inline function with a boolean flag which selects the invalidation instruction. I'd expect the compiler to eliminate the branch:

static __always_inline void                                                                                                                                                                                                                                                                                                 
pmap_s1_invalidate_loop(vm_offset_t sva, vm_offset_t eva, vm_offset_t stride,                                                                                                                                                                                                                                               
    int va_shift, vm_offset_t va_mask, uint64_t asid, const bool kernel,                                                                                                                                                                                                                                                       
    const bool final_only)                                                                                                                                                                                                                                                                                                        
{                                                                                                                                                                                                                                                                                                                           
      uint64_t units;                                                                                                                                                                                                                                                                                                                                                  
      vm_size_t pages;                                                                                                                                                                                                                                                                                                      
      int scale, unit_shift;                                                                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                                                            
      for (vm_offset_t va = sva; va < eva;) {                                                                                                                                                                                                                                                                                           
              if (pmap_tlbi_range_support && (va & va_mask) == 0) {                                                                                                                                                                                                                                                         
                      pages = atop(eva - va);                                                                                                                                                                                                                                                                               
                      if (pages >= TLBI_RANGE_UNIT(0)) {                                                                                                                                                                                                                                                                    
                              scale = TLBI_RANGE_SCALE(pages);                                                                                                                                                                                                                                                              
                              unit_shift = TLBI_RANGE_UNIT_SHIFT(scale);                                                                                                                                                                                                                                                    
                              units = ulmin(pages >> unit_shift,                                                                                                                                                                                                                                                            
                                  TLBI_RANGE_MAX_UNITS);                                                                                                                                                                                                                                                                    
                              if (kernel)
                                      pmap_s1_invalidate_range_kernel(asid |
                                          TLBI_RANGE_FIELDS(va, va_shift,
                                          units - 1, scale), final_only);    
                              else
                                      pmap_s1_invalidate_range_user(asid |   
                                          TLBI_RANGE_FIELDS(va, va_shift,
                                          units - 1, scale), final_only);
                              va += ptoa(units << unit_shift);
                              continue;
                      }                                                      
              }
              if (kernel)                                                 
                      pmap_s1_invalidate_kernel(asid | TLBI_VA(va),
                          final_only);
              else                          
                      pmap_s1_invalidate_user(asid | TLBI_VA(va),
                          final_only);
              va += stride;                                                  
      }
}

[alc]

Convert macro to __always_inline function.

[alc]

sys/arm64/arm64/pmap.c

2177

The difference was only a couple instructions.

[alc]

While the conversion from a macro to an inline pmap_s1_invalidate_loop() had no real effect, just a couple instructions changed, pmap_s1_invalidate_strided() is not being inlined in some places due to its size. In that case, we are not benefiting from final_only always being a constant, but I think that is something for a different patch to deal with.

[markj]

sys/arm64/arm64/pmap.c

484

You might write #define TLBI_RANGE_VA_SHIFT() (pmap_lpa_enabled ? 16 : PAGE_SHIFT) to make it more obvious that this isn't a compile-time constant.

2221

Should we assert somewhere that sva and eva are page-aligned?

[alc]

I ran a dozen -j8 buildworlds, each starting from an empty /usr/obj, with MALLOC_PRODUCTION enabled and LLVM assertions disabled on a GENERIC-NODEBUG kernel. Here is Claude's analysis of the data:

The change is a win.
		before		after		delta
sys		1,070.67	1,012.58	−58.1 s (−5.43%)
user		23,801.46	23,816.08	+14.6 s (+0.06%)
user+sys	24,872.13	24,828.66	−43.5 s (−0.17%)
wall		3,221.63 (53:41.63)	3,214.40 (53:34.40)	−7.2 s (−0.22%)

System time drops 5.4%, with the two ranges fully disjoint — t ≈ 38. That's the headline.

The user-time difference is noise: sd 45.7 and 42.1 s respectively, t = 0.82. Which is the right outcome, and it's the same argument as before pointed the other way — unchanged user time means no over-invalidation, so the range ops are tearing down exactly the entries the single-page path did. That corroborates the harness's exactly-once coverage result on real hardware.

The wall-clock number looks unimpressive only because of Amdahl: system time is 4.3% of total CPU in this workload, so cutting it 5.4% can only buy ~0.23% overall, and 0.22% is what you got. The two are consistent, not in tension. It also sits right at the edge of what 12 runs can resolve, so quote the sys number, not the wall number — the sys measurement is overwhelming and the wall measurement is marginal.

One more thing in the change's favor: run-to-run variance drops too, sd 8.13 s → 5.88 s. Fewer and more uniform TLBI operations is a plausible reason.

[alc]

As an aside, as the appearance of the word "harness" in the previous comment suggests, when I had Claude reviewing the changes, it actually built a user-space test harness containing the loop to check that every page in the range would be the target of an invalidation. And, it temporarily introduced a few bugs in the loop to check that those bugs were detected. Just thought this was interesting.

[alc]

sys/arm64/arm64/pmap.c

484

Done.

2221

Some do, e.g., pmap_kenter, but for the most part we rely on the MI layer to pass page-aligned addresses. I would argue that the right approach would be MI wrappers that performed the asserts at the interface, not here (duplicated in every pmap).

[markj]

In D58708#1347933, @alc wrote:

I ran a dozen -j8 buildworlds, each starting from an empty /usr/obj, with MALLOC_PRODUCTION enabled and LLVM assertions disabled on a GENERIC-NODEBUG kernel. Here is Claude's analysis of the data:

I wonder where these (ranged) invalidations are coming from, as I don't see very many when building on amd64. I guess they are mostly caused by break-before-make sequences triggered by L3C and L2 superpage promotion and demotion. I noticed that on amd64, munmap() will not trigger a ranged invalidation when unmapping a run of PTEs, we just call pmap_invalidate_all(). On arm64 this doesn't appear to be the case, it will use ranged invalidations. I wonder what fraction of ranged invalidations come from munmap() vs. promotion/demotion.

In D58708#1347935, @alc wrote:

As an aside, as the appearance of the word "harness" in the previous comment suggests, when I had Claude reviewing the changes, it actually built a user-space test harness containing the loop to check that every page in the range would be the target of an invalidation. And, it temporarily introduced a few bugs in the loop to check that those bugs were detected. Just thought this was interesting.

Yeah, it's surprisingly creative sometimes...

sys/arm64/arm64/pmap.c

2186

I forgot to ask before: does it make sense to have some upper bound on the size of the range, like amd64's pmap_invalidate_range() does?

[alc]

In D58708#1348047, @markj wrote:

In D58708#1347933, @alc wrote:

I ran a dozen -j8 buildworlds, each starting from an empty /usr/obj, with MALLOC_PRODUCTION enabled and LLVM assertions disabled on a GENERIC-NODEBUG kernel. Here is Claude's analysis of the data:

I wonder where these (ranged) invalidations are coming from, as I don't see very many when building on amd64. I guess they are mostly caused by break-before-make sequences triggered by L3C and L2 superpage promotion and demotion. I noticed that on amd64, munmap() will not trigger a ranged invalidation when unmapping a run of PTEs, we just call pmap_invalidate_all(). On arm64 this doesn't appear to be the case, it will use ranged invalidations. I wonder what fraction of ranged invalidations come from munmap() vs. promotion/demotion.

I suspect that arm64 overuses range-based invalidation and amd64 underutilizes it. Our use of invlpg was introduced to deal with PG_G mappings, and really hasn't changed much since. I believe Linux does range-based invalidation using invlpg until the range exceeds 32 pages.

sys/arm64/arm64/pmap.c

2186

I looked into this a while back. From memory, ... Linux has a bound, but supposedly not for performance. They were seeing lockups on some high processor count machines. (Old Ampere I believe, but may be wrong.) They set the bound at the number of PTEs in a page. If you look at pmap_{protect,remove}, we effectively implement the same bound due to PV list locking and simply the structure of the code. (A similar explanation applies to superpage promotions.) The exceptions, where an explicit bound might be called for, are less commonly used functions, e.g., pmap_kenter.

[markj]

sys/arm64/arm64/pmap.c

2186

My vague recollection is that commit 10386b56ad932 was motivated by some operation which unmaps a huge address range in the kernel, perhaps an nvdimm mapping, taking way too long.