pfctl: Move AF-specific mask logic from callers into set_ipmask()

Instead of doing the same dance with every caller, check for user provided
mask or address familiy specific maximum inside the function itself.

Feedback and OK claudio

Obtained from: OpenBSD, kn <kn@openbsd.org>, c04427dd30
Sponsored by: Rubicon Communications, LLC ("Netgate")