r355120 | dim | 2019-11-26 23:01:09 +0100 (Tue, 26 Nov 2019) | 32 lines

The fdlibm hypot() implementations shouldn't potentially left-shift
negative numbers (invoking undefined behavior)

Summary:
Various paths through hypot(x, y) will multiply x and y by a power of
two, perform the calculation in a range where IEEE-754 provides greater
precision, then undo the multiplication to determine the true result.
Undoing that multiplication is implemented as t1*w, where t1=2**k.

2**k is often computed by taking the high word of 1.0, then adding k<<20
(for doubles or long doubles) or k<<23 (for floats) to it, then
overwriting that high word. But when k is negative this left-shifts a
negative value -- and that's undefined behavior in many editions of C
and C++.

This patch should fix all hypot implementations to compute 2**k without
triggering this particular bit of undefined behavior.

Test Plan: I've only very lightly tested out the hypot(double, double)
change, in SpiderMonkey's JavaScript engine, for consistency with prior
behavior. The other functions' changes have more or less only been
eyeballed. Careful examination appreciated! Do note, however, that an
error in any of these changes would most likely produce a value that is
incorrect by a factor of two, so any mistake would most likely be
glaring if invoked.

Submitted by: Jeff Walden <jwalden@mit.edu>
Obtained from: https://github.com/freebsd/freebsd/pull/414
Reviewed by: dim, lwhsu
Differential Revision: https://reviews.freebsd.org/D22354