HomeFreeBSD

libc/qsort: Don't allow interposing recursive calls

Description

libc/qsort: Don't allow interposing recursive calls

This causes problems when using ASAN with a runtime older than 12.0 since
the intercept does not expect qsort() to call itself using an interposable
function call. This results in infinite recursion and stack exhaustion
when a binary compiled with -fsanitize=address calls qsort.
See also https://bugs.llvm.org/show_bug.cgi?id=46832 and
https://reviews.llvm.org/D84509 (ASAN runtime patch).

To prevent this problem, this patch uses a static helper function
for the actual qsort() implementation. This prevents interposition and
allows for direct calls. As a nice side-effect, we can also move the
qsort_s checks to the top-level function and out of the recursive calls.

Reviewed By: kib
Differential Revision: https://reviews.freebsd.org/D28133

(cherry picked from commit cbcfe28f9d5f975f97b7fb4a0d72bc9780eb0c46)

Details

Provenance
arichardsonAuthored on Feb 18 2021, 10:12 AM
Reviewer
kib
Differential Revision
D28133: libc/qsort: Don't allow interposing recursive calls
Parents
rGd37fb0e37814: AArch64: Don't set flush-subnormals-to-zero flag on startup
Branches
Unknown
Tags
Unknown