Return ENOTSUP for mmap/mprotect if prot not subset of prot_max

From POSIX,

[ENOTSUP]

The implementation does not support the combination of accesses
requested in the prot argument.

This fits the case that prot contains permissions which are not a subset
of prot_max.

Reviewed by: brooks, cem
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D23843