[PowerPC] [PowerPCSPE] Fix multiple issues in fpsetmask().

Building R on powerpc64 exposed a problem in fpsetmask() whereby we
were not properly clamping the provided mask to the valid range.

This same issue affects powerpc and powerpcspe.

Properly limit the range of bits that can be set via fpsetmask().

While here, use the correct fp_except_t type instead of fp_rnd_t.

Reported by: pkubaj, jhibbits (in IRC)
Sponsored by: Tag1 Consulting, Inc.

(cherry picked from commit 384ee7cc6e9e4ddc91a6e9e623fcbbe5826bce38)