linux: use sa_family_t for address family conversions

Express "conversion failed" with maximum possible value. This allows to
reduce number of size/signedness conversion in the code that utilizes the
functions.

PR: 274536
Reviewed by: melifaro
Differential Revision: https://reviews.freebsd.org/D44375