vm: fix swap reservation leak and clean up surrounding code

The code did not subtract from the global counter if per-uid reservation
failed.

Cleanup highlights:

• load overcommit once
• move per-uid manipulation to dedicated routines
• don't fetch wire count if requested size is below the limit
• convert return type from int to bool
• ifdef the routines with _KERNEL to keep vm.h compilable by userspace

Reviewed by: kib (previous version)
Differential Revision: https://reviews.freebsd.org/D25787