[PowerPC64] Fix multiple issues in fpsetmask().

Building R exposed a problem in fpsetmask() whereby we were not properly
clamping the provided mask to the valid range.

R initilizes the mask by calling fpsetmask(~0) on FreeBSD. Since we
recently enabled precise exceptions, this was causing an immediate
SIGFPE because we were attempting to set invalid bits in the fpscr.

Properly limit the range of bits that can be set via fpsetmask().

While here, use the correct fp_except_t type instead of fp_rnd_t.

Reported by: pkubaj (in IRC)
Sponsored by: Tag1 Consulting, Inc.

(cherry picked from commit dd95b39235dd81c890aa3cce02a5bb7f91f23803)