In D60280#1383381, @kib wrote:

In D60280#1383380, @kib wrote:

We can compare the old and new uio_resid then.

For this to work, uiomove_phys() should also avoid crossing userspace page boundaries, not just phys page boundaries.

In D60280#1383380, @kib wrote:

We can compare the old and new uio_resid then.

Here is Claude's summary of a log that I collected on the DevKit:

I've been reviewing places where we already sync the icache or might need to.

In D60271#1383044, @markj wrote:

In D60271#1383042, @alc wrote:

Do any of you have access to a machine with IDC, but not DIC, e.g., an older Ampere I think? The results on a small Cortex-X1/A78 system are at best inclusive.

I think this is what you're looking for?

CPU  0: ARM Neoverse-N1 r3p1 affinity: 18  0  0
                   Cache Type = <IDC,64 byte CWG,64 byte ERG,64 byte D-cacheline,PIPT I-cache,64 byte I-cacheline>
 Instruction Set Attributes 0 = <DP,RDM,Atomic,CRC32,SHA2,SHA1,AES+PMULL>
 Instruction Set Attributes 1 = <RCPC-8.3,DCPoP>
 Instruction Set Attributes 2 = <>
         Processor Features 0 = <CSV3,CSV2,RAS,GIC,AdvSIMD+HP,FP+HP,EL3,EL2,EL1,EL0 32>
         Processor Features 1 = <MTE_frac,PSTATE.SSBS MSR>
         Processor Features 2 = <>
Trying to mount root from zfs:zroot/ROOT/bhyve []...
      Memory Model Features 0 = <TGran4,TGran64,TGran16,SNSMem,BigEnd,16bit ASID,256TB PA>
      Memory Model Features 1 = <XNX,PAN+ATS1E1,LO,HPD+TTPBHA,VH,16bit VMID,HAF+DS>
      Memory Model Features 2 = <EVT-8.2,32bit CCIDX,48bit VA,UAO,CnP>
      Memory Model Features 3 = <>
      Memory Model Features 4 = <>
             Debug Features 0 = <DoubleLock,SPE,2 CTX BKPTs,4 Watchpoints,6 Breakpoints,PMUv3p1,Debugv8p2>
             Debug Features 1 = <>
         Auxiliary Features 0 = <>
         Auxiliary Features 1 = <>

It's an Ampere Altra, not sure offhand which one. I can test this patch on it this weekend if you tell me what exactly you'd like to try.

Do any of you have access to a machine with IDC, but not DIC, e.g., an older Ampere I think? The results on a small Cortex-X1/A78 system are at best inclusive.

@andrew, Is this what you were asking for? A 16 processor AWS a1.4xlarge, which is Cortex A72-based, sees a small reduction in system time using arm64_pipt_icache_sync_range(). Specifically, the reduction is about 0.75%.

In D59865#1377062, @andrew wrote:

Do we have somewhere we can document the contract with userspace & when it needs to manage the cache? e.g. removing write & enabling execute on a page will cause the kernel to sync the cache, but having both together means userspace needs to perform a sync operation.

Here is a link to the paper I published a while back on PTE coalescing. At that time, the changes in Linux to exploit small folios were just beginning to get merged.

I've also tested the prior version of this patch on a 32-core machine (c6g.8xlarge), where icache coherence is maintained by the hardware and the sync operation is simply a dsb and an isb. I just wanted to make sure that the patch didn't negatively effect performance on such machines. I compared 18 buildworld runs on HEAD with 22 runs using this patch. The kernel was -NODEBUG, MALLOC PRODUCTION was enabled, and LLVM assertions were disabled. System time actually fell by 7.8 seconds, from 2,060 to 2,052 seconds, because we avoided a couple hundred million dsb and isb instructions at the cost of the flag maintenance. That is 0.4%, which is four times larger than the random error you would expect. User time fell by 21 seconds, or 0.05%, which is probably real but too small to assert a similar claim relative to the random error. Wall-clock time did not change measurably.

PGA_ICACHE_SYNCED and KASSERTs

Rename PGA_MACHDEP0004 to PGA_PMAP_PRIV1.

Tested. It works.

Yes, this will work. I will actually test it later today.

Introduce pmap_page_is_mapped_locked().

I recently updated my Microsoft Dev Kit, and because of this change I'm seeing:

bus_dmamem_alloc failed to align memory properly.

The warning stems from an allocation by the nvme driver:

KDB: stack backtrace:
db_trace_self() at db_trace_self
db_trace_self_wrapper() at db_trace_self_wrapper+0x44
bounce_bus_dmamem_alloc() at bounce_bus_dmamem_alloc+0x258
nvme_ctrlr_start() at nvme_ctrlr_start+0xbc8
nvme_ctrlr_start_config_hook() at nvme_ctrlr_start_config_hook+0x5ec
run_interrupt_driven_config_hooks() at run_interrupt_driven_config_hooks+0x94
boot_run_interrupt_driven_config_hooks() at boot_run_interrupt_driven_config_hooks+0x30
mi_startup() at mi_startup+0x1ec
virtdone() at virtdone+0x70

An added printf reports:

vaddr: 0xffffa000847d3f40, paddr: 1047d3f40, alignment: 1000

In this case, the misalignment appears to be harmless because page size alignment isn't actually needed.

I'm testing this patch on an EC2 a1.4xlarge (16x Cortex A72) machine to see the greatest impact. (This is the entire machine, so there isn't any variance due to other VMs on the machine.) I added a COUNTER_U64 to track icache flushes by the pmap. As expected, this patch results in an increased icache flush count, since we were not flushing on creating executable superpage mappings. During a -j16 buildworld on a -NODEBUG kernel, PRODUCTION malloc, and LLVM with assertions disabled, the number of flushes goes from ~190M to ~230M and wall clock time goes from ~2:09:10 to ~2:10:00. In particular, system time increased by ~6%.

I uncovered this while resurrecting an old patch for reducing the number of icache flushes. If all goes well, I will post that in a week or two.

I'm curious as to why this isn't also a problem for the uses of .arch_extension in arm64/vfp.c, arm64/mte.c, and include/atomic.h?

Revise a comment.

In D58917#1352088, @kib wrote:

Does arm64 use recursive mapping? If yes, could the function be used on recursive mapping?

Move td = curthread; out of the critical section.

In D58708#1348047, @markj wrote:

In D58708#1347933, @alc wrote:

I ran a dozen -j8 buildworlds, each starting from an empty /usr/obj, with MALLOC_PRODUCTION enabled and LLVM assertions disabled on a GENERIC-NODEBUG kernel. Here is Claude's analysis of the data:

I wonder where these (ranged) invalidations are coming from, as I don't see very many when building on amd64. I guess they are mostly caused by break-before-make sequences triggered by L3C and L2 superpage promotion and demotion. I noticed that on amd64, munmap() will not trigger a ranged invalidation when unmapping a run of PTEs, we just call pmap_invalidate_all(). On arm64 this doesn't appear to be the case, it will use ranged invalidations. I wonder what fraction of ranged invalidations come from munmap() vs. promotion/demotion.

I'm waiting to hear if @andrew has any comments or questions.

As an aside, as the appearance of the word "harness" in the previous comment suggests, when I had Claude reviewing the changes, it actually built a user-space test harness containing the loop to check that every page in the range would be the target of an invalidation. And, it temporarily introduced a few bugs in the loop to check that those bugs were detected. Just thought this was interesting.

I ran a dozen -j8 buildworlds, each starting from an empty /usr/obj, with MALLOC_PRODUCTION enabled and LLVM assertions disabled on a GENERIC-NODEBUG kernel. Here is Claude's analysis of the data:

The change is a win.
		before		after		delta
sys		1,070.67	1,012.58	−58.1 s (−5.43%)
user		23,801.46	23,816.08	+14.6 s (+0.06%)
user+sys	24,872.13	24,828.66	−43.5 s (−0.17%)
wall		3,221.63 (53:41.63)	3,214.40 (53:34.40)	−7.2 s (−0.22%)

In D58723#1347638, @markj wrote:

Possibly vfp_restore_state_common() should rely on its caller to enter a critical section (with an assert).

While the conversion from a macro to an inline pmap_s1_invalidate_loop() had no real effect, just a couple instructions changed, pmap_s1_invalidate_strided() is not being inlined in some places due to its size. In that case, we are not benefiting from final_only always being a constant, but I think that is something for a different patch to deal with.

Convert macro to __always_inline function.

I'm surprised that we didn't see this on arm64.

Just out of curiosity, where are the pmap changes?

In D56432#1294557, @markj wrote:

In D56432#1294556, @kib wrote:

Effectively, you agree that we can drop the support and disable passing VM_ALLOC_ZERO to vm_page_grab_unlocked?

After kern_kexec is fixed, yes. But now I am trying to understand what that code is doing, and failing.

In D56432#1292906, @kib wrote:

I thought what could be a use for such call, and I really do not see how it can be useful. Both vnode and swap objects might have the page content on the volume, so the invalid page we found on the queue cannot be safely zeroed without consulting pager first. In the end, the decision to zero such page belongs to the caller.

For other object types, like OBJT_PHYS in kern_kexec.c, it is still somewhat useful since we do not have a vm_page_alloc_pages(), so pages have to be allocated one-by-one. But yes, also note that vm_page_grab_valid() does not handle VM_ALLOC_ZERO.

In D56458#1293651, @kevans wrote:

I'm going to plan to commit this within the next day or so. I think linux's use of "stride" instead of size or shift for their naming of the bit is very telling, and their general range invalidation function seems to specifically just use PTE stride as well.

"... pmap locks, than witness ..." -> "... pmap locks, then witness ..."

@markj Do we know if there is actually a mix of clean and dirty pages being mapped?

Specifically, for now, I would commit the version in Diff 168641.

At this point, I would go back to the simpler, O(n) version, and commit that.