Rebase.

In vm_fault.c, correct the initial value of mpred in the case that src_object == dst_object.

Replace a vm_page_alloc with vm_page_alloc_after in vm_fault.c.

Undo a comment change.

Modify comments, in response to private @alc feedback.

Add comments regarding the choice of pred argument in several alloc_after() calls.

Elide stray comment fix.

Accept some suggestions from @markj.

Rebase. Testing shows this degrades performance, still, so dropping reviewers.

Rebase.

In D49033#1118416, @markj wrote:

In D49033#1118399, @dougm wrote:

Results of performing 11 buildworld tests on modified and unmodified sources, using a patch from @alc to instrument cycle counts for several page-related functions. Performance improvements are evident in the opr (vm_object_page_remove) and kmem_unback cases. Some cases that don't do much freeing may have gotten a little worse, for some reason.

Was this on amd64? The difference between the locked and unserialized variants of pctrie_node_store() are pretty minimal there, since x86 memory ordering constraints mean that plain stores already provide the desired semantics, ignoring compiler reordering of memory operations. So, it's not too surprising that the results are a bit inconclusive. On arm64 I'd hope to see a more consistent improvement.

Results of performing 11 buildworld tests on modified and unmodified sources, using a patch from @alc to instrument cycle counts for several page-related functions. Performance improvements are evident in the opr (vm_object_page_remove) and kmem_unback cases. Some cases that don't do much freeing may have gotten a little worse, for some reason.

Rebase after change to pctrie_iter_remove().

Update the comment header on pctrie_remove().

Rebase after addition of parent pointers.

Rebase.

@alc asks for another set of performance test results, after changes to this patch and updates to main. Here it is:

Fix overflow in lookup_le.

Remove an extra blank line.

Use smr to handle the parent pointers, as suggested by @alc.

The description of this patch includes:

Add comments.

.

How do these changes affect the various counts? Some of these counts change because iterators are used, and some change because lookup_le now uses parent pointers.

The same information as posted before, reordered. For most of the counters, the change to use parent pointers reduces the cycles/call in most of the builds.

With this patch {F108995447} from @alc, additional stats are maintained about several functions the process pages. A test script from @alc repeatedly dumps these stats and makes buildworld. A summary of the stats produced from 14 buildworld cycles, with and without this change in place, is here:

Clear out some obfuscation from vm_reserv_break().

Reposting the last version before review and commit. It passes the mmap41.sh stress test, but @alc expresses skepticism about what's happening in vm_reserv_break.

Crash report from pho:

Drop PCTRIE_LIMIT; no longer used.

In D48007#1105192, @markj wrote:

One other performance-related detail I'm wondering about is, we check the iterator limit twice when stepping: once before looking up a leaf, and once after finding one. I believe the latter check is sufficient for correctness. If so, are we making the right tradeoff in checking twice instead of once?

In D48007#1105192, @markj wrote:

In D48007#1103103, @dougm wrote:

In D48007#1102734, @markj wrote:

Are there still some concerns about the overhead of switching to iterators in some of the already-committed changes? I've lost track a bit, I'm sorry.

@alc has concerns. He's reported performance worsened a bit by some of the page management changes already made, and worsened a bit by some other changes that have been under review for a while.

Do we have a benchmark to experiment with, either a workload to test the kernel or a standalone userspace program embedding subr_pctrie.o? I wrote a patch which tries to add a fast path to pctrie_iter_step/next, wherein pctrie.h inlines iteration when the top-most node in the iterator contains a run of successive leaf values, and falls back to pctrie_iter_lookup_ge() to look up new nodes or handle holes. So far I've only tested it to check correctness, but I wonder if it would help performance in the case where a VM object is densely populated.

Undo the parts of the last change about moving the iterator initialization.

In vm_map.c, have page index jump ahead with a superpage jump-ahead. Initialize the cursor before acquiring a read lock.

Fix unused variable problem.

Rebase. Resolve a conflict.

Abandon VM_ALLOC_NOCREAT_LT.

Use VM_RADIX_FORALL in kcov_free.

Use VM_RADIX_FORALL_FROM one more place.

In D48007#1102734, @markj wrote:

Doug, how would you like to proceed with the patch? Since quite a few pieces of it are independent, I imagine they can be reviewed and peeled off one by one, especially in areas that aren't performance-critical.