Rebase.

Tweak the first loop in pctrie_iter_lookup_le to make it 2 whole bytes smaller. And, for the _ge version, one byte smaller.

Drop ..._iter_lookup_lt().

Jenkins says:
sys/fs/tmpfs/times_test:holey -> panic: Lock vmobject not exclusively locked @ /usr/src/sys/vm/swap_pager.c:2490

Do a bit of work around vm_page_grab.

Can't switch from _le to _lt in grab_pages.

Add and user lookup_iter_lt. Looking for the predecessor by starting by looking for the place where something will be inserted is a waste. Instead, start looking just before where something will be inserted; there's a change you'll find something there.

Drop an 'else' to fix a wraparound bug identified by @kib.

Restore requested KASSERT.

In D47150#1076221, @kib wrote:

In D47150#1075489, @dougm wrote:

Remove suspicious assertion.

What was the assert? I can see only one case where it might be not true, when vm_map_entry_delete() calls collapse for kernel object. But this is definitely not the case for the Peter' backtrace.

In D47200#1076225, @kib wrote:

What I proposed is close to the second option. But ok.

In D47200#1076214, @kib wrote:

May be, instead, allow to specify type of the trie in the TRIE macros?

A bigger patch that uses iterators for vm_page_alloc_after and the functions it calls. Changes to vm_page_alloc are outside the range of this patch.

Avoid an extra node_load in pctrie_insert_node by passing the node argument to it. Break pctrie_init_node out of pctrie_insert_node, in anticipation of batch insertion.

Restore passing mpred to vm_page_iter_insert.

Addresses this bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=282134

Remove unrelated change included by accident.

Remove suspicious assertion.

Rewrote some parts with perhaps a better understanding of what the busy/unbusy parts are intended to do.

Add an assertion.

Add a comment in grab_pages_unlocked.

Restore the name vm_page_acquire_unlocked and have it do the looping. Drop the enums.

Since the insert_lookup functions now never actually insert anything they don't have to have a uint64_t* argument that might get inserted, they only need a uint64_t argument for the lookup. Make that change.

Copy what bnoreuselist() does to handle the problem with LOOKUP_GE results, which is more robust than what I did before.

Restore the pctrie lookup. Add a check for negative block number. Passes mmap28.sh and truncate3.sh tests.

Where this change causes a couple of functions to only be invoked with parameter access == PCTRIE_UNSYNCHRONIZED, drop that parameter.

Pull the locked stores out of subr_pctrie.c and leave them for inline functions in pctrie.h to perform, with whatever synchronization type is appropriate.

Define functions to bundle prepare, insert, cleanup and finish.

In D46895#1072679, @markj wrote:

As a hack for x86, can we simply use an ifdef to merge the locked and unserialized cases in pctrie_node_store()?

Add a not-dead kassert.

Reorder deletion and tree removal in vm_object_split.

This is more complex than initializing and iterator and using it to insert consecutive elements. How have you measured its performance and cache-friendliness in comparison to that simpler implementation?

Set page fields before insertion, and clear them when insertion fails.

Don't use pctrie lookup to find the first block to scan; just use linear search. Using pctrie lookup leads to two crashes (truncate3.sh, mmap28.sh) on stress tests, and this version passes those tests. I guess there are items in the list not in the trie?

Update after vm_radix.h changes.

Add definition of vm_radix_iter_lookup_le. I thought I had added it to a vm_radix patch just committed, but apparently I had not.