bsd.compat.mk: Allow finding non-internal libraries

[bdrewery]

Summary

Currently only libexec/rtld-elf32 uses internal LIBC_NOSSP_PIC during
the build but it gets it directly from the objdir rather than a sysroot.
For example, /usr/obj/usr/src/amd64.amd64/obj-lib32/lib/libc/libc_nossp_pic.a.
We don't stage lib32 libraries in WORLDTMP/usr/lib32 and doing so doesn't
buy much. If we want to use a staged lib32 library then we need to look in
LIBCOMPATTMP where they were staged. For example if LIBC_PIC were wanted then
look for /usr/obj/usr/src/amd64.amd64/obj-lib32/tmp/usr/lib32/libc_pic.a.

Note this problem is part of make everything and not make buildcompat*.

Reported by: rlibby
Sponsored by: Dell EMC

Test Plan

before:

# env OBJTOP=$(make -V OBJTOP) OBJTOPOBJROOT='${OBJTOP}/'  MAKEOBJDIRPREFIX= DESTDIR='${OBJTOP}/tmp' SYSROOT='${DESTDIR}' make -C libexec/rtld-elf32 -V LIBC_PIC -V LIBC_NOSSP_PIC -V LIBC
/scratch/obj/root/git/freebsd2/amd64.amd64/tmp/usr/lib32/libc_pic.a
/scratch/obj/root/git/freebsd2/amd64.amd64/obj-lib32/lib/libc/libc_nossp_pic.a
/scratch/obj/root/git/freebsd2/amd64.amd64/tmp/usr/lib32/libc.a
# ls -al /scratch/obj/root/git/freebsd2/amd64.amd64/tmp/usr/lib32/libc_pic.a /scratch/obj/root/git/freebsd2/amd64.amd64/obj-lib32/lib/libc/libc_nossp_pic.a /scratch/obj/root/git/freebsd2/amd64.amd64/tmp/usr/lib32/libc.a
ls: /scratch/obj/root/git/freebsd2/amd64.amd64/tmp/usr/lib32/libc.a: No such file or directory
ls: /scratch/obj/root/git/freebsd2/amd64.amd64/tmp/usr/lib32/libc_pic.a: No such file or directory
-rw-r--r--  1 root  wheel  12391432 Nov 25 10:09 /scratch/obj/root/git/freebsd2/amd64.amd64/obj-lib32/lib/libc/libc_nossp_pic.a

after:

# env OBJTOP=$(make -V OBJTOP) OBJTOPOBJROOT='${OBJTOP}/'  MAKEOBJDIRPREFIX= DESTDIR='${OBJTOP}/tmp' SYSROOT='${DESTDIR}' make -C libexec/rtld-elf32 -V LIBC_PIC -V LIBC_NOSSP_PIC -V LIBC
/scratch/obj/root/git/freebsd2/amd64.amd64/obj-lib32/tmp/usr/lib32/libc_pic.a
/scratch/obj/root/git/freebsd2/amd64.amd64/obj-lib32/lib/libc/libc_nossp_pic.a
/scratch/obj/root/git/freebsd2/amd64.amd64/obj-lib32/tmp/usr/lib32/libc.a
# ls -al /scratch/obj/root/git/freebsd2/amd64.amd64/obj-lib32/tmp/usr/lib32/libc_pic.a /scratch/obj/root/git/freebsd2/amd64.amd64/obj-lib32/lib/libc/libc_nossp_pic.a /scratch/obj/root/git/freebsd2/amd64.amd64/obj-lib32/tmp/usr/lib32/libc.a
-rw-r--r--  1 root  wheel  12391432 Nov 25 10:09 /scratch/obj/root/git/freebsd2/amd64.amd64/obj-lib32/lib/libc/libc_nossp_pic.a
-r--r--r--  1 root  wheel  12334382 Nov 20 12:28 /scratch/obj/root/git/freebsd2/amd64.amd64/obj-lib32/tmp/usr/lib32/libc.a
-r--r--r--  1 root  wheel  12535610 Nov 25 10:12 /scratch/obj/root/git/freebsd2/amd64.amd64/obj-lib32/tmp/usr/lib32/libc_pic.a

I have not yet done a tinderbox but plan to.

[rlibby]

I've tested this with buildworld on amd64. Before I was not able to refer to ${LIBC_PIC} from the rtld-elf32 compat Makefile and now I can.

[arichardson]

@brooks made some changes for CheriBSD to use a merged sysroot for compat and normal libraries. I wonder if we should be upstreaming that change instead?

[brooks]

In D27648#618541, @arichardson wrote:

@brooks made some changes for CheriBSD to use a merged sysroot for compat and normal libraries. I wonder if we should be upstreaming that change instead?

IIRC the change with @jhb's idea. The current changes have some issues on amd64 that would need to be solved first (the /usr/include/machine contents of amd64 aren't compatible with -m32 in some cases). I don't think it's an unsolvable problem, but will require some work.