bsd.lib.mk: Work around build system raciness

[arichardson]

Summary

We are seeing regular build failures due to libc.so being installed again and
another parallel make job tries to read the partially written libc.so at the
same time. When building with -j32 or higher this almost always happens on
the first clean build (subsequent incremental builds always work fine).
Using -S should "fix" the "section header table goes past the end of the
file: e_shoff = 0x..." errors that have started to plague our builds.

I originally thought this only affected CheriBSD, but I just got the same
error while building the latest upstream FreeBSD.

The real fix should be to not install libraries twice, but until then this
workaround is needed.

Obtained from: CheriBSD (https://github.com/CTSRD-CHERI/cheribsd/commit/49837edd3efd5d02a1b120c47f00cfc2d59a9a8e)
Patch by: Jessica Clarke <[e-mail]>

Test Plan

Fixed the build failures on CheriBSD and seems to work here.

[arichardson]

share/mk/bsd.lib.mk

416 ↗

(On Diff #79207)

Note: we could still skip the -S flag when doing installworld with -DNO_ROOT, it's just a problem during buildworld when installing to the sysroot.

[markj]

share/mk/bsd.lib.mk

416 ↗

(On Diff #79207)

Right. This could probably be more clear in the comment, e.g., by writing "... ends up getting installed to WORLDTMP a second time..."

[bdrewery]

This was just an optimization in rS322565 so there's no harm in losing it until we find a way to make it work. I found it was related to MK_PROFILE flipping back then.

Redundant building/installing for WORLDTMP is something I imagine addressing with META_MODE+filemon but I've been shy about pushing that upstream. At work it's been working well but has some subtle traps with files moving from objdir -> srcdir (or simply orphaned files in objdir). We have the same problem with buildworld without META_MODE now requiring Makefile.inc1's _cleanobj_fast_depend_hack.