Improve the coverage of debug symbols for MK_DEBUG_FILES.

[jhb]

Summary

• Include debug symbols in static libraries. This permits binaries to include debug symbols for functions obtained from static libraries.
• Permit the C/C++ compiler flags added for MK_DEBUG_FILES to be overridden by setting DEBUG_FILES_CFLAGS. Use this to limit the debug information for llvm libraries and binaries.

Test Plan

• compare installworld results to different DESTDIRs

[jhb]

The original discussion with Ed started here: https://github.com/bsdjhb/cheribsd/commit/a5f6a09647c5be6d7e45d390e778531f7ef22025

In summary, just adding -g for static libraries bumped the size of the amd64 installed world from 982M to 6.2G. All of this increase was in /usr/lib and /usr/lib32, the vast majority of which was in /usr/lib/debug for the debug symbols for ld.lld, lldb, and clang. I then tried using -gline-tables-only for the llvm libraries and binaries. That resulted in these overall sizes:

# du -hs /ufs/root[123]
982M    /ufs/root1
6.2G    /ufs/root2
1.8G    /ufs/root3

/usr/lib32 is the same increase (81M -> 238M). The libraries in /usr/lib is also the same increase (124M -> 349M). /usr/lib/debug for the three cases:

# du -hs /ufs/root[123]/usr/lib/debug
345M    /ufs/root1/usr/lib/debug
5.2G    /ufs/root2/usr/lib/debug
791M    /ufs/root3/usr/lib/debug

The biggest gains are still in clang and friends, but the other source of growth is also apparent which is that private libraries are static only and we did not include debug symbols for those (e.g. libraries used by svnlite binaries).

Existing top 10 debug info /usr/lib/debug/usr/bin:

-r--r--r--  1 root  wheel    1100992 Jul 28 20:39 svnlite.debug
-r--r--r--  1 root  wheel    1215296 Jul 28 20:38 as.debug
-r--r--r--  1 root  wheel    1910120 Jul 28 20:39 nvi.debug
-r--r--r--  1 root  wheel    4086800 Jul 28 20:39 dtc.debug
-r--r--r--  1 root  wheel    8568552 Jul 28 20:39 llvm-objdump.debug
-r--r--r--  1 root  wheel    9076176 Jul 28 20:39 lldb.debug
-r--r--r--  1 root  wheel    9366568 Jul 28 20:39 clang-tblgen.debug
-r--r--r--  1 root  wheel   15588200 Jul 28 20:39 clang.debug
-r--r--r--  1 root  wheel   45153488 Jul 28 20:39 llvm-tblgen.debug
-r--r--r--  1 root  wheel   48351040 Jul 28 20:39 ld.lld.debug

With the -gline-tables-only for clang and no CXXFLAGS in bsd.prog.mk:

-r--r--r--  1 root  wheel     6510480 Aug  2 05:06 svnlitesync.debug
-r--r--r--  1 root  wheel     7511904 Aug  2 05:05 llvm-tblgen.debug
-r--r--r--  1 root  wheel     8377928 Aug  2 05:06 svnlitemucc.debug
-r--r--r--  1 root  wheel     8424232 Aug  2 05:05 svnlitebench.debug
-r--r--r--  1 root  wheel     8469488 Aug  2 05:06 svnliterdump.debug
-r--r--r--  1 root  wheel    10443176 Aug  2 05:05 svnlite.debug
-r--r--r--  1 root  wheel    18807024 Aug  2 05:05 llvm-objdump.debug
-r--r--r--  1 root  wheel    96903512 Aug  2 05:05 ld.lld.debug
-r--r--r--  1 root  wheel   155313520 Aug  2 05:05 lldb.debug
-r--r--r--  1 root  wheel   190416928 Aug  2 05:05 clang.debug

[jhb]

On IRC, Ed wondered if old gcc supports -gline-tables-only. If it doesn't we can just use an empty flags for old GCC.

Also, my next test will be to enable the missing debug info for C++ binaries. This might cause a bit more explosion in clang and friends, but should otherwise have little effect (devd?)

[emaste]

% gcc -gline-tables-only hello.c
cc1: error: unrecognized debug output level 'line-tables-only'

[jhb]

gcc 5.x and 6.3 don't support it either. I'll have to make it depend on COMPILER_TYPE it seems.

[jhb]

share/mk/bsd.prog.mk

44 ↗

(On Diff #32057)

So changing this made no difference. I guess the CXXFLAGS ?= assignment in sys.mk makes this work otherwise. That is somewhat confusing compared to bsd.lib.mk, but ok.

[jhb]

• Use -g1 for clang when compiling with GCC.

[jhb]

For comparison, here are the results for mips64 built with external GCC (so builds and installs clang/llvm) using similar names. root1.mips is the existing tree, root2.mips is with -g for everything, and root3.mips is using -g1 for llvm libraries and binaries:

# du -hs /ufs/root[1234].mips
1.1G    /ufs/root1.mips
3.6G    /ufs/root2.mips
2.0G    /ufs/root3.mips

So in the fully patched case (1 -> 3) the MIPS world grows by about the same amount (0.9G) as amd64. lib32 grew on MIPS64 by about 200M:

# du -hs /ufs/root[1234].mips/usr/lib32
 99M    /ufs/root1.mips/usr/lib32
295M    /ufs/root2.mips/usr/lib32
295M    /ufs/root3.mips/usr/lib32

/usr/lib without /usr/lib/debug increased from 148M to 478M. The list of biggest binaries in /usr/lib/debug/usr/bin is similar as on amd64 (llvm, ld.bfd, as, svnlite*). Note that mips doesn't build ld.lld or lldb which is why the '2' case (full -g) is "only" an extra 1G or so. It would likely be inline with amd64 with lld and lldb.

[jhb]

So in summary, the current patchset nearly doubles the installed world size (adds 0.9G on both amd64 and mips64, a bit less due to non-lib32 on 32-bit platforms). Most of that gain is due to increased debug info in binaries (llvm and private libs like those used by svnlite).

[emaste]

I think this is acceptable, although perhaps consider having something like LIMITED_DEBUG_INFO=yes that usr.bin/clang/Makefile.inc could set rather than having -gline-tables-only and -g1 in usr.bin/clang/Makefile.inc.

[jhb]

In D12025#251276, @emaste wrote:

I think this is acceptable, although perhaps consider having something like LIMITED_DEBUG_INFO=yes that usr.bin/clang/Makefile.inc could set rather than having -gline-tables-only and -g1 in usr.bin/clang/Makefile.inc.

Part of me liked having DEBUG_FILES_CFLAGS be something that is settable to provide flexibility in the future, but I'm not super tied to that approach.

[emaste]

Ok, I just don't want to end up with the usr.bin/clang/Makefile.inc logic proliferating, but I think this is fine for now and we can revisit that question if/when we find additional places that might want it.