- Include debug symbols in static libraries. This permits binaries to include debug symbols for functions obtained from static libraries.
- Permit the C/C++ compiler flags added for MK_DEBUG_FILES to be overridden by setting DEBUG_FILES_CFLAGS. Use this to limit the debug information for llvm libraries and binaries.
- rS322824: Improve the coverage of debug symbols for MK_DEBUG_FILES.
- compare installworld results to different DESTDIRs
The original discussion with Ed started here: https://github.com/bsdjhb/cheribsd/commit/a5f6a09647c5be6d7e45d390e778531f7ef22025
In summary, just adding -g for static libraries bumped the size of the amd64 installed world from 982M to 6.2G. All of this increase was in /usr/lib and /usr/lib32, the vast majority of which was in /usr/lib/debug for the debug symbols for ld.lld, lldb, and clang. I then tried using -gline-tables-only for the llvm libraries and binaries. That resulted in these overall sizes:
# du -hs /ufs/root 982M /ufs/root1 6.2G /ufs/root2 1.8G /ufs/root3
/usr/lib32 is the same increase (81M -> 238M). The libraries in /usr/lib is also the same increase (124M -> 349M). /usr/lib/debug for the three cases:
# du -hs /ufs/root/usr/lib/debug 345M /ufs/root1/usr/lib/debug 5.2G /ufs/root2/usr/lib/debug 791M /ufs/root3/usr/lib/debug
The biggest gains are still in clang and friends, but the other source of growth is also apparent which is that private libraries are static only and we did not include debug symbols for those (e.g. libraries used by svnlite binaries).
Existing top 10 debug info /usr/lib/debug/usr/bin:
-r--r--r-- 1 root wheel 1100992 Jul 28 20:39 svnlite.debug -r--r--r-- 1 root wheel 1215296 Jul 28 20:38 as.debug -r--r--r-- 1 root wheel 1910120 Jul 28 20:39 nvi.debug -r--r--r-- 1 root wheel 4086800 Jul 28 20:39 dtc.debug -r--r--r-- 1 root wheel 8568552 Jul 28 20:39 llvm-objdump.debug -r--r--r-- 1 root wheel 9076176 Jul 28 20:39 lldb.debug -r--r--r-- 1 root wheel 9366568 Jul 28 20:39 clang-tblgen.debug -r--r--r-- 1 root wheel 15588200 Jul 28 20:39 clang.debug -r--r--r-- 1 root wheel 45153488 Jul 28 20:39 llvm-tblgen.debug -r--r--r-- 1 root wheel 48351040 Jul 28 20:39 ld.lld.debug
With the -gline-tables-only for clang and no CXXFLAGS in bsd.prog.mk:
-r--r--r-- 1 root wheel 6510480 Aug 2 05:06 svnlitesync.debug -r--r--r-- 1 root wheel 7511904 Aug 2 05:05 llvm-tblgen.debug -r--r--r-- 1 root wheel 8377928 Aug 2 05:06 svnlitemucc.debug -r--r--r-- 1 root wheel 8424232 Aug 2 05:05 svnlitebench.debug -r--r--r-- 1 root wheel 8469488 Aug 2 05:06 svnliterdump.debug -r--r--r-- 1 root wheel 10443176 Aug 2 05:05 svnlite.debug -r--r--r-- 1 root wheel 18807024 Aug 2 05:05 llvm-objdump.debug -r--r--r-- 1 root wheel 96903512 Aug 2 05:05 ld.lld.debug -r--r--r-- 1 root wheel 155313520 Aug 2 05:05 lldb.debug -r--r--r-- 1 root wheel 190416928 Aug 2 05:05 clang.debug
On IRC, Ed wondered if old gcc supports -gline-tables-only. If it doesn't we can just use an empty flags for old GCC.
Also, my next test will be to enable the missing debug info for C++ binaries. This might cause a bit more explosion in clang and friends, but should otherwise have little effect (devd?)
|44 ↗||(On Diff #32057)|
So changing this made no difference. I guess the CXXFLAGS ?= assignment in sys.mk makes this work otherwise. That is somewhat confusing compared to bsd.lib.mk, but ok.
For comparison, here are the results for mips64 built with external GCC (so builds and installs clang/llvm) using similar names. root1.mips is the existing tree, root2.mips is with -g for everything, and root3.mips is using -g1 for llvm libraries and binaries:
# du -hs /ufs/root.mips 1.1G /ufs/root1.mips 3.6G /ufs/root2.mips 2.0G /ufs/root3.mips
So in the fully patched case (1 -> 3) the MIPS world grows by about the same amount (0.9G) as amd64. lib32 grew on MIPS64 by about 200M:
# du -hs /ufs/root.mips/usr/lib32 99M /ufs/root1.mips/usr/lib32 295M /ufs/root2.mips/usr/lib32 295M /ufs/root3.mips/usr/lib32
/usr/lib without /usr/lib/debug increased from 148M to 478M. The list of biggest binaries in /usr/lib/debug/usr/bin is similar as on amd64 (llvm, ld.bfd, as, svnlite*). Note that mips doesn't build ld.lld or lldb which is why the '2' case (full -g) is "only" an extra 1G or so. It would likely be inline with amd64 with lld and lldb.
So in summary, the current patchset nearly doubles the installed world size (adds 0.9G on both amd64 and mips64, a bit less due to non-lib32 on 32-bit platforms). Most of that gain is due to increased debug info in binaries (llvm and private libs like those used by svnlite).
I think this is acceptable, although perhaps consider having something like LIMITED_DEBUG_INFO=yes that usr.bin/clang/Makefile.inc could set rather than having -gline-tables-only and -g1 in usr.bin/clang/Makefile.inc.
Ok, I just don't want to end up with the usr.bin/clang/Makefile.inc logic proliferating, but I think this is fine for now and we can revisit that question if/when we find additional places that might want it.