Fix the definition of RM_MAX_END.

Even though casting from signed to unsigned is well-defined in C, it's better to
first cast to the larger unsigned type, then negate.