vmm/vmm_mem.c: Use a custom 'pgo_populate' operation for sysmem segments

[bnovkov]

Summary

This change implements a 'pgo_populate' pager operation for objects
backing system memory segment, allowing the vmm module to take
advantage of the 'vm_fault_wire' fast path. Under this new scheme, each
system memory segment preallocates all pages before they are wired,
using 'vm_reserv_pindex_aligned' to allocate VM_LEVEL_0_ORDER-sized
pages where appropriate.

Test Plan

This change (and the whole patch stack) was tested by measuring how long it takes to wire
a 3GB system memory segment for a bhyve VM on an Intel i3-13100F machine.
I've used the following dtrace snippet to measure this: dtrace -n 'fbt::vm_mmap_memseg:entry { self->ts = timestamp; } fbt::vm_mmap_memseg:return /self->ts/ { printf("%d", (timestamp - self->ts));self->ts = 0 }'.

The results show a massive improvement when compared to the slow path:

x bhyve_wire_patch.txt
+ bhyve_wire_baseline.txt
    N           Min           Max        Median           Avg        Stddev
x  10      22315921      23504339      22437526      22536410      352224.4
+  10  4.789722e+08 4.8494576e+08 4.8042138e+08  4.811004e+08       1686655
Difference at 95.0% confidence
	4.58564e+08 +/- 1.14478e+06
	2034.77% +/- 22.7181%
	(Student's t, pooled s = 1.21837e+06)

bhyve VMs launched with this patch work as expected, both with and without -S flag.
The patch also survived an hour-long stress test with stress2's mlock and vm scripts.

[markj]

sys/dev/vmm/vmm_mem.c

205

Why can't we just call vm_page_alloc() a page at a time? Then the reservation layer will provide superpages automatically.

Note that this is probably too sneaky: VM_LEVEL_0_ORDER has a different definition inside of vm_reserv.c on arm64, see the Temporarily simulate two-level reservations block.

206

What if superpages are administratively disabled? pagesizes[1] will be zero in that case.

207

How do you know there isn't already a page resident at this index?

208

Surprisingly, this doesn't return zeroed memory. You have to zero it yourself.

264

sysmem isn't the right predicate. In fact, we don't know at this point whether the segment will be wired or not, that's only determined when the VM_MMAP_MEMSEG ioctl is invoked with the VM_MEMMAP_F_WIRED flag set. But if the segment isn't wired, we must not use an OBJT_PHYS object: pages belonging to such objects are not reclaimable, they are always wired, effectively.