[PowerPC] [PowerPCSPE] Fix multiple issues in fpsetmask().

Building R on powerpc64 exposed a problem in fpsetmask() whereby we
were not properly clamping the provided mask to the valid range.

This same issue affects powerpc and powerpcspe.

Properly limit the range of bits that can be set via fpsetmask().

While here, use the correct fp_except_t type instead of fp_rnd_t.

Reported by: pkubaj, jhibbits (in IRC)
Sponsored by: Tag1 Consulting, Inc.

(cherry picked from commit 384ee7cc6e9e4ddc91a6e9e623fcbbe5826bce38)
(cherry picked from commit 8b96d6ac04e7e761ec6b9eff47c801a2b89fbd6d)