sys/rpc: UNIX auth: Fix OOB reads on too short message

[olce]

Summary

In the inline version (_svcauth_unix()), fix multiple possible OOB reads
when the credentials part of a request is too short to contain mandatory
fields or with respect to the hostname length or number of groups it
advertises. The previously existing check was arriving too late and
relied on possibly wrong data coming from earlier OOB reads.

While here, use 'u_int' as the length/size type, as it is more than
enough and removes the need for conversions. While here, factor out
setting 'stat' to AUTH_BADCRED and then jumping to 'done' on error,
through the new 'badcred' label. While here, through comments, refer to
what the non-inline version is doing (xdr_authunix_parms() in
'authunix_prot.c') and the reasons.

[rmacklem]

Looks fine to me. Feel free to ignore the minor comments.

sys/rpc/svc_auth_unix.c

64

Since you are fixing the type, you might want
to use uint32_t, just in case u_int becomes 64bits
for some arch someday.

But, this is a minor nit. Use whichever type you'd like.

136

I end to use MIN() instead of min() so that
it works for all integer types, but using min()
here should be fine.

[olce]

sys/rpc/svc_auth_unix.c

136

Yes. See my more detailed answer in D52960.