[PowerPC] [PowerPCSPE] Fix multiple issues in fpsetmask().

Building R on powerpc64 exposed a problem in fpsetmask() whereby we
were not properly clamping the provided mask to the valid range.

This same issue affects powerpc and powerpcspe.

Properly limit the range of bits that can be set via fpsetmask().

While here, use the correct fp_except_t type instead of fp_rnd_t.

Reported by: pkubaj, jhibbits (in IRC)
Sponsored by: Tag1 Consulting, Inc.
MFC after: 1 week