Optimize keg_drain().

[markj]

Summary

Avoid holding the keg lock for longer than necessary. This helps ensure
that trimming the zone doesn't block allocations for longer than really
needed. If a zone's bucket list has recently been drained and we are
draining the keg, it is likely that allocations will block until
keg_drain() has emptied the free slab list, and that may take a long
time.

[rlibby]

sys/vm/uma_core.c

1280

You can delete this check now.

1287–1289

This is outside the lock, so we don't really care, but I think there's no particular reason to maintain the list here vs abandoning it as we iterate with LIST_FOREACH_SAFE. Additionally, only this list maintenance writes the slab header on destruction here.

3508–3516

It might be useful to leave a comment explaining that we know that slab was on the partial list, and NOT the free list (even if it is totally free, and so we aren't missing a ud_free_slabs-- here), and maybe to leave a similar comment about the slab returned by keg_fetch_slab.

[markj]

Apply Ryan's feedback:

• Remove an unneeded check in keg_drain().
• Avoid unnecessarily dirtying slab headers in keg_drain().
• Add some comments to explain how we know that no extra ud_free_slabs manipulation is needed.

[rlibby]

Fixups look good.

[jeff]

sys/vm/uma_core.c

4648–4651

Let's make this sequence an inline since it's now three statements that are repeated in multiple places.

sys/vm/uma_int.h

330

I'm I correct in there being 32bit left before we spill to a new line?

[markj]

sys/vm/uma_int.h

330

No, ud_lock is in its own cache line and struct slabhead is a single pointer, so there is still a lot of padding left over.