Restore the reservation of boot pages for bucket zones.

[markj]

Summary

uma_startup2() sets booted = BOOT_BUCKETS after calling bucket_init(),
but before that assignment, startup_alloc() will use pages from the
reserved pool.

An alternative which also works is simply to set booted = BOOT_BUCKETS
before calling bucket_init(). I don't have any strong feelings as to
which is preferable as I don't know what bootstrapping problem is being
referred to in the commit message for r355707.

[rlibby]

The problem I was originally trying to solve was allowing embedded slabs on the bucket zones for D22768. This means the slab memory for the bucket zones must come from uma_small_alloc. It can't come from the startup pages because PHYS_TO_VM_PAGE doesn't work on the startup pages since they aren't in the page array (although, we could possibly change that, with some potential other cost, see the vm_page.c bit in D11784). So I added a check at keg creation time that only enables embedded slabs if the keg was created after BOOT_PAGEALLOC. That disabled embedding for bucket zones, which were being created before then.

But what we care about is the memory backing the slabs. The zones can be from startup_alloc, I don't think that matters. So, I think the fix here will still solve my problem, since the slabs still won't be allocated until later.

It might be nice to reduce startup pages, but that was not my goal. Since we are still single threaded here in uma_startup2(), I do think that setting booted = BOOT_BUCKETS one line earlier is viable with a good comment, if we care about the number of startup pages.

Aside, why is keg->uk_ppera > 1 for the zone zone? It seems if not for that then we would "switch to real allocator" ... ah I see. The zone size is a function of mp_maxid and vm_ndomains. I didn't hit this on my testing because I tested on a small machine. Sorry about that, and thanks for helping here.

[markj]

In D22797#499090, @rlibby wrote:

The problem I was originally trying to solve was allowing embedded slabs on the bucket zones for D22768. This means the slab memory for the bucket zones must come from uma_small_alloc. It can't come from the startup pages because PHYS_TO_VM_PAGE doesn't work on the startup pages since they aren't in the page array (although, we could possibly change that, with some potential other cost, see the vm_page.c bit in D11784). So I added a check at keg creation time that only enables embedded slabs if the keg was created after BOOT_PAGEALLOC. That disabled embedding for bucket zones, which were being created before then.

I see now, thanks.

I think it should be possible to allocate physical memory for startup pages following the end of the kernel and preloaded data (which are at the beginning of physical memory on amd64), instead of at the end of the largest phys_seg as we do currently. Then, in kmem_bootstrap_free() we have a mechanism for releasing such pages back to the generic physical memory allocator. That would allow us to drop the UMA_SLAB_BOOT check in keg_drain() too, I think.

But what we care about is the memory backing the slabs. The zones can be from startup_alloc, I don't think that matters. So, I think the fix here will still solve my problem, since the slabs still won't be allocated until later.

Agreed.

It might be nice to reduce startup pages, but that was not my goal. Since we are still single threaded here in uma_startup2(), I do think that setting booted = BOOT_BUCKETS one line earlier is viable with a good comment, if we care about the number of startup pages.

I don't think there's any harm in using startup pages for the bucket zones, so I'm slightly inclined towards the diff above for now.