This gives better efficiency for item sizes smaller than a page but
significantly larger than the slab header size. In particular, without
this patch, for every ZFS ABD chunk we allocate an 80 bytes slab header,
effectively a 2% tax on the ARC. With this patch we don't need to
allocate any extra memory.
One downside of the implementation is that this layout can't be used for
malloc(9) zones, which need to store extra metadata in plinks so that
free(9) can find the right zone. I think we can compress that metadata
somehow, e.g., instead of storing a pointer to the malloc zone, store
the corresponding kmemzones[] index in the oflags field, but I didn't
try to implement it.
With INVARIANTS disabled, we have 12 bytes available for slab bitmaps;
with INVARIANTS enabled, we have 4 bytes available. In a
GENERIC-NODEBUG kernel with 240 zones, 142 of them use the new layout.