HomeFreeBSD

Fix an off-by-two in the llquantize() action parameter validation.

Description

Fix an off-by-two in the llquantize() action parameter validation.

The aggregation created by llquantize() partitions values into buckets; the
lower bound of the bucket containing the largest values is b^{m+1}, where
b and m are the second and fourth parameters to the action, respectively.
Bucket bounds are stored in a 64-bit integer, and so the llquantize()
validation checks need to verify that b^{m+1} fits in 64 bits. However, it
was only verifying that b^{m-1} fits in 64 bits, so certain parameter
combinations could trigger assertion failures in libdtrace.

PR: 219451
MFC after: 1 week

Details

Provenance
markjAuthored on
Parents
rS322772: Enable hardfloat CPU instructions in the FP exception handler.
Branches
Unknown
Tags
Unknown