Differential D45046
vm_reserv: add reservation-aware UMA small_alloc Authored by bnovkov on May 1 2024, 1:54 PM. Tags None Referenced Files
Details This patch adds a reservation-aware replacement for uma_small_alloc. The vm_reserv_uma_small_{alloc, free} routines use unmanaged reservations to allocate 0-order pages for UMA zones. The allocator falls back to vm_page_alloc_noobj_domain in case of memory pressure. All changes in this patch series were tested on amd64 using a bhyve vm. I can confirm that everything builds properly for all other architectures.
Diff Detail
Event TimelineComment Actions Update vm_reserv_uma_small_alloc to retry if someone else filled up a partial reservation. Comment Actions I spent some time evaluating this patch with four different metrics (thanks @markj for the first two):
The first two metrics track how scattered these pages types are in memory. I've gathered them by iterating through all UMA zones and their slab lists. I ran buildkernel four times in a row on a 10GB bhyve VM, and sampled the metrics before each run. Results without patches:
Results with patches:
The amount reservations "tainted" by NOFREE pages is drastically smaller with this patch. It also remains stable throughout the test, whereas the non-patched version grows steadily. Comment Actions Addendum - @netchild has been running -CURRENT with this patch series for some time with no issues encountered so far. I'd really love to get some feedback about the patch since it went through a couple of rounds of testing and appears to be both stable and effective at containing NOFREE and regular slabs. Comment Actions
Dual socket, 24 CPU (incl. HT), (old) Xeon server. 72 GB RAM. About 30 jails (not counting the poudriere jails). Various services, mysql, postgresql, redis, dns, ldap, various instances of php and nginx, various java services, ... I haven't updated yet to the most recent update of this patch. I've seen a lot of instabilities in other places (fixed based upon my reports), but the current BE I have is rock solid with this change (plus D45043 and D45045).
Comment Actions Reading this and the related patches, I am a bit uncertain as to why this code needs to live in the vm_reserv layer. From my reading, UMA's small page allocator is effectively using this code to maintain a list of 2MB allocations from which we draw slab allocations; a separate list is used to segregate slabs for UMA_ZONE_NOFREE zones. What problems does this vm_reserv extension solve? I can see that this will segregate NOFREE slabs, which is a good thing, but we don't need to modify the vm_reserv layer for that; UMA could easily maintain a linked list of 2MB chunks from which NOFREE slabs are allocated. Once a chunk is exhausted, try to allocate a new one and put it at the head of the list, otherwise fall back to regular 4KB slab allocations. The NOFREE case is simple and doesn't need a lot of machinery. The change also segregates the remaining UMA slab allocations, but this is also provided by another mechanism: VM_FREEPOOL_DIRECT. UMA is the main consumer of pages from that freepool; I'd guess that page table page allocations are a distant second. So, we already have a soft mechanism to ensure that UMA's 4KB slabs don't fragment physical memory too much. This change introduces a stronger segregation. However,
Apologies in advance if I missed anything while reading the diffs, but my feeling now is that we'd get most of the benefit of this patch series by just segregating NOFREE slabs as a special case, and handling that entirely within UMA. Comment Actions Placing these changes in vm_reserv was done mostly to reuse the existing code and abstractions. But after reading your comments I agree, in its current state this could be moved to UMA entirely.
We don't - thanks for catching this! I've overlooked that mechanism, but adding it in shouldn't be an issue.
This mechanism differs from VM_FREEPOOL_DIRECT in a few ways, but I think that the stronger segregation guarantees and the way pages are allocated are the most important differences. Please correct me if I'm wrong, but from what I understand "regular" page allocations (i.e. not vm_page_alloc_noobj) might "steal" pages from the DIRECT freepool if the default one is empty. This cannot happen with these changes making it less likely that the "regular" and noobj allocations mix. You'd have to explicitly break a noobj reservation and release it back to the freelists to mix the two page allocation types. This can be a good or bad thing depending on what we are aiming for, since it does provide stronger segregation for the two types but also makes repurposing the unused pages a bit more complicated. However, I think that the biggest difference here is that these changes will prioritize filling up partially populated reservations, which should pack things more tightly than VM_FREEPOOL_DIRECT. Allocating pages using VM_FREEPOOL_DIRECT will dequeue pages from the 0-order freelist, and there's no guarantee that the queued 0-order pages come from the same reservation. I also think that there's a minor but noteworthy performance difference with this approach - we reduce lock contention for the vm_phys freelists.
I did some light testing and posted the results in a comment here, apologies if you've seen it already. As expected there's a drastic change for NOFREE slabs, but the non-NOFREE slab numbers are in the same ballpark.
Right, the way I see it we have a couple of options:
Both VM_FREEPOOL_DIRECT and the proposed approach have their pros and cons, but I think applying the proposed approach to a few select cases could prove useful in the long run. Comment Actions A running system will over time need to reclaim unused memory from unmanaged, partially populated reservations. Otherwise, UMA's consumers can internal fragmentation within the 2MB chunks such that a significant amount of RAM may be "free" but unusable by the rest of the system. This has to be addressed somehow. We can fix that by capping the total amount of RAM used for unmanaged reservations, or by having a facility to reclaim unused memory from partially populated unmanaged reservations. The former is difficult to do in a general way since UMA's memory consumption is completely workload-dependent, and the latter means we lose the "strong" segregation that you're talking about above.
If a page is in a 0-order freelist, then its buddy is already allocated. (And that page has been in the freelist longer than all of the other free 0-order pages, so is not likely to see its buddy returned to the free lists in the near future.) vm_phys tries to import the largest possible chunk of contiguous memory into a pool when needed, so unless RAM is already very fragmented, the buddy will by allocated to another consumer of the same free pool, which in this case is likely to be UMA. It's not really obvious to me why this is objectively worse than the reservation-based scheme.
Hmm, I don't really follow why. What kind of workload is triggering sustained contention for vm_phys freelists? Keep in mind that:
My suggestion is to handle the easy, high-impact case first, i.e., try to fix NOFREE slab fragmentation within UMA. We already know that that alone benefits the system, and it can be done without modifying anything other than UMA. I would be quite cautious about making assumptions about memory access patterns of something as general as the ZFS ARC. It supports many diverse workloads. ZFS has much more information about the history and future accesses of a given ARC buffer than UMA (or the page allocator) does, so to make improvements there we should leverage that information more. I think you are specifically looking at the ABD UMA zone (larger buffers are allocated via the kmem_* interface, which already makes use of superpage reservations); the ABD allocator just allocates one 4KB page at a time from UMA (see abd_alloc_chunks) - are there opportunities for it to convey more information that might help us make smarter allocation choices? Comment Actions Thanks, your answers clear things up! I had an incomplete overview of the issue, I see now that its best too keep VM_FREEPOOL_DIRECT.
Agreed. I'm abandoning the unmanaged reservation revisions and adding a UMA-only NOFREE one. I'll look into the ABD allocator separately. Comment Actions @markj What you say about the 0-order freelist is true. However, consider a scenario in which we have three 2MB chunks, call them A, B, and C, that are mostly allocated. Let's assume that they all have a single 4KB, 8KB, and 16KB chunk free, and that A is the oldest of the allocations and C is the youngest. The "problem" with a buddy allocator is that its next three 4KB allocations will come from the three different chunks, A, B, and C. More often than not, I do think that we would better off if we sought to fill A first and give C more time for additional chunks to be deallocated. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||