libutil: take a size_t in trimdomain()

INT_MAX is already larger than a reasonable hostname might be, but
size_t makes some of this easier to reason about as we do arithmetic
with it. This would maybe not be worth it if we had to bump the
soversion because of it, but libutil does symbol versioning now so we
can provide a compat shim.

While we're here, fix some inconsistencies in argument names in the
manpage.

Reviewed by: des
Obtained from: https://github.com/apple-oss-distributions/libutil
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D54622