[PowerPC64] Fix multiple issues in fpsetmask().

Building R exposed a problem in fpsetmask() whereby we were not properly
clamping the provided mask to the valid range.

R initilizes the mask by calling fpsetmask(~0) on FreeBSD. Since we
recently enabled precise exceptions, this was causing an immediate
SIGFPE because we were attempting to set invalid bits in the fpscr.

Properly limit the range of bits that can be set via fpsetmask().

While here, use the correct fp_except_t type instead of fp_rnd_t.

Reported by: pkubaj (in IRC)
MFC after: 1 week
Sponsored by: Tag1 Consulting, Inc.