Install kernel debug data under /usr/lib/debug

[emaste]

Summary

In addition, change kernel debug file extension to .debug, to match userland debug files.

Posted for initial comment - I am aware of at least two issues to resolve:

• Need better integration with knobs. Setting WITH_DEBUG_FILES for userland automatically adds -g to DEBUG_FLAGS. Presumably kernel debug should work the same way (or perhaps WITH_KERNEL_DEBUG_FILES if more fine-grained control is desired).
• The /usr/lib/debug/boot/kernel/ directory will need to be renamed in installworld as we do with /boot/kernel/

[markj]

This looks ok to me.

For the src.conf knobs, maybe have WITH_{KERNEL,USERLAND}_DEBUG_FILES and have both of them be set if WITH_DEBUG_FILES is true. But that might be overkill - I think it's more likely for someone to want standalone debug files for the kernel and not userland than the other way around. That is, either all debug files should be standalone, or just the kernel's should be standalone.

[emaste]

Move /usr/lib/debug/boot/kernel/ to /usr/lib/debug/boot/kernel.old/

[brooks]

Looks good to me.

The one thing I wonder about is if it will trip up any mtree based IDS type scripts for users who build kernels under another name, but that seems like not much of a worry.

[emaste]

Discovered a failure case, with installkernel -DNO_ROOT and without WITH_DEBUG_FILES=YES in src.conf. In this case the BSD.debug.dist mtree file is not used, and so there is no /usr/lib/debug/ directory in the METALOG for the installation of /usr/lib/debug/boot/kernel/kernel.debug and such.

I'd say it's reasonable to create the whole /usr/lib/debug/ hierarchy if we are going to install anything under there (that is, if either WITH_DEBUG_FILES is set, or WITHOUT_KERNEL_SYMBOLS is not set).

[emaste]

Create debug hierarchy for for kernel debug too

If kernel debug data is being installed (i.e., WITHOUT_KERNEL_SYMBOLS is not set) then we need the /usr/lib/debug/boot/kernel directory to exist.

Note that this will create the full /usr/lib/debug hierarchy if either userland or kernel debug data will be installed, resulting in a few empty directories if only one of themis installed. This does not seem like an issue worth addressing.

Also this relies on the MK_KERNEL_SYMBOLS flag being available via the userland mk files, not just sys/conf/kern.opts.mk, so I have just duplicated the flag setting in src.conf.mk.

[brooks]

I agree that creation of a few extra directories under /usr/debug is fine.

[imp]

what's the motivation for moving the kernel stuff out of /boot?

[emaste]

what's the motivation for moving the kernel stuff out of /boot?

• consistency with debug data for userland
• the boot partition has to be relatively large to accommodate debug data
• it makes it easier to pull the debug data out into a separate package / tarball and install it later
• if you then store the debug data on an NFS server you could symlink / automount /usr/lib/debug on demand

[imp]

There's no, good, clean options here.

Userland and kernel can and do have different option settings. Generally, we try to avoid overlap between the two or dependencies between the two for this reason.

Option one: Treat the debug file directories the way we treat the info or man path directories. In this option, we always install them, so it wouldn't matter if some kernel user wants to opt in or out of the debug files later. This provides the least coupling, but some might think it lacks aesthetic appeal. While we don't create directories for sendmail or groff, those are individual programs. /usr/lib/debug is a global thing for many different programs, any one of which may wish to install debug symbols while not installing globally (see the cd /usr/src/bin/ls; make all install clean WITH_DEBUG_FILES=t use case).

Option two: Rename MK_KERNEL_SYMBOLS to MK_DEBUG_FILES and leverage off the name-space pollution in bsd.own.mk into resolving this issue. This sounds good at the top level, but does have some flaw. Userland and kernel defaults are now the same. Depending on your use case this could be good or bad. This also adds -g to the mix, but the kernel is already compiled that way, so it isn't a huge change. More troubling, though, is the coupling of userland and kernel options. Since they can be independent, doing install-time things based on this can lead to failure. For example, installworld with MK_DEBUG_FILES=no (so the debug directories aren't created) followed by a installkernel with MK_DEBUG_FILES=yes will fail due to directories not being present. This does make the defaults the same, but I suspect that users will often want them to be different. Plus MK_KERNEL_SYMBOLS has been the API for a while so you'd need a transition period (see the ill-named INSTALL_NODEBUG).

option three: Just put MK_KERNEL_SYMBOLS in src.opts.mk. Then we can test it and life will be good. Except it won't when MK_DEBUG_FILES is no and MK_KERNEL_SYMBOLS is no during the installworld phase, but MK_KERNEL_SYMBOLS is yes during the installkernel phase.

From my perspective, option one is the best: just always create the directories like we do for man pages even when we're not installing the man pages. This allows the individual file case to work (eg cd /usr/src/bin/ls) as well as it working globally. This allows the kernel to still install stuff while userland doesn't (though the size argument from days of yore isn't such a big deal since kernel debug files are ~300MB and userland are ~900MB both of which can be accommodated on the majority of /usr partitions in the wild today). Burning a few inodes seems to be the least-bad side-effect that we can have, and it makes all of the use-cases work and preserves the userland / kernel boundary best.

I'm agnostic on the directory move (though that will break some gdb scripts). I don't see any need at all to do that, but having the ability to install it elsewhere seems useful for small / scenarios (which we have plenty of in our user base: embedded, geli root, NFS sharing of most of the tree, etc). There are times, though, it is nice to have all the symbols next to the kernel in question (consider the kernel developer that wants to install three different kernels and run them under load. when the crash happens, it sure is convenient to have them in /bool/kernel.fred than to wonder if the ones in /usr/lib/debug are good (though with the KODIR appended, this is less of an issue).

Not sure I like the .symbols -> .debug move, since that will also break scripts (both for debugging and pruning in image building). Not sure I hate it, but I'd suggest some time to pause and think about it may be in order.

etc/Makefile

329 ↗

(On Diff #2140)

see below for why this is a bad idea.

etc/mtree/BSD.debug.dist

6

I'm rather of the opinion this while file should live in BSD.usr.dist for reasons discussed in my comment.

share/mk/src.opts.mk

101 ↗

(On Diff #2140)

Having kernel options listed in src.opts.mk is a really bad idea.

(1) if you think you want to test them in the install and/or build phase, then you break everybody that doesn't put them in src.conf or make.conf or uses some other alternate method. This makes the usefulness of testing them low since it is so brittle.
(2) The kernel can build without any other supporting files from /usr/src (well, mostly that's the goal, but we fall a smidge short of the goal when we include bsd.init.mk which we're pretty much forced into for compatibility reasons with old releases).
(3) If the defaults differ, then src.conf.5 building scripts fail in bizarre ways.

sys/conf/kmod.mk

287

This change (.symbols -> .debug) likely will break people's scripts even worse than moving the files.

[emaste]

Thanks for your detailed response Warner. I was on the fence before but I think the cd /usr/src/bin/ls && make all install clean with WITH_DEBUG_FILES=t use case is a compelling reason to always install the /usr/lib/debug hierarchy.

[emaste]

Not sure I like the .symbols -> .debug move, since that will also break scripts (both for debugging and pruning in image building). Not sure I hate it, but I'd suggest some time to pause and think about it may be in order.

If we're going to do the rename and the move then they have to happen at the same time so we only have one hurdle for users to address.

My argument for the ".debug" name is (in order of decreasing importance, I think):

• This is what everyone else calls it.
• It's not an accurate description of the file's content. Some symbol data also exists in the binary or library, and there is a lot more debugging information in the standalone debug file than just symbols.
• Userland and kernel debug data will have the same debug extension.

[emaste]

• create the /usr/lib/debug hierarchy unconditionally per @imp
• add test for empty ${DEBUGDIR} (before we would delete the just-created /boot/kernel.old when installing a new kernel with empty ${DEBUGDIR})
• add missing ${DEBUGDIR} to kernel-reinstall target

I left BSD.debug.dist for now, but can roll it into BSD.usr.dist.

[emaste]

etc/Makefile

327 ↗

(On Diff #2150)

Missed this one in the most recent diff; it is now in the MTREES list above after BSD.usr.dist.

[imp]

I'm cool. I do think the rename will cause some bumps for people, but there's already bumps...

[imp]

One thing that would make this better is to have
KERN_DEBUGDIR?=${DEBUGDIR}
early in kern.post.mk then use that consistently throughout the patch. This would allow one to set it to "" in one's src.conf file and have it restore the old behavior. This is assuming another patch would put DEBUGDIR globally to something like /usr/share/debug

[emaste]

Rebase, and implement KERN_DEBUGDIR suggestion from @imp

[bdrewery]

Sorry, I haven't read up on the discussion. I think it makes a lot of sense to keep debug files with the kernel files. It makes using kernel sets much simpler while using nextboot. If you move the files, how can I easily boot into a slightly different kernel and use the exact debug symbols from it? How can I boot into an older stable one if needed and still use the right symbols? There seems to be a lot of new room for confusion here by moving the symbols out of the kern.bootfile directory.

[emaste]

In D1006#33, @bdrewery wrote:

Sorry, I haven't read up on the discussion.

That's unfortunate, since this has been discussed in the past. If you use the documented way to install a kernel into a separate subdirectory and use nextboot -k to boot it, everything "just works."

If you have a habit of colouring outside of the lines (and I'm not saying that's a bad thing), you can set KERN_DEBUGDIR="" to keep the historical behaviour of installing the debug files alongside the kernel files.

[bdrewery]

I read up and discussed with Ed in private. This is fine with me.

It's just an awareness thing that moving /boot/kernel to /boot/kernel.whatever requires also moving the symbols.

[emaste]

That is a valid point, we need to make sure that users who would be affected are fully aware of this change so that they can either accommodate it in their processes or scripts, or add the src.conf knob to keep the current behaviour.

[imp]

An UPDATING entry would be useful.
Maybe it is time we have the externally visible build changes from 10 to 11, 11 to 12, etc in src.conf.5 sources somewhere? There have been a lot this time around.

[emaste]

In D1006#37, @imp wrote:

An UPDATING entry would be useful.

Yes, I will update the review w/ a proposed UPDATING entry. I am going to post a CFT to -CURRENT to get additional feedback too, before this would be committed.

Maybe it is time we have the externally visible build changes from 10 to 11, 11 to 12, etc in src.conf.5 sources somewhere? There have been a lot this time around.

I think so, but I'm not sure we should necessarily make reference to the changes themselves, rather just making sure that new or changed knobs / paths / options are documented. Right now it seems we only have WITH/WITHOUT knobs documented, not e.g. variables to set cross-compilation toolchain paths, etc.

[emaste]

Rebase and use KERN_DEBUGDIR in kmod.mk as well.

[emaste]

sys/conf/kern.post.mk

288

Accidentally lost this in the rebase. Will be ${INSTALL} -p -m 555 -o ${KMODOWN} -g ${KMODGRP} ${KERNEL_KO}.debug ${DESTDIR}${KERN_DEBUGDIR}${KODIR}

[emaste]

I plan to commit this later this week

[emaste]

Add UPDATING entry
Restore one ${KERN_DEBUGDIR} lost in rebase

[emaste]

Rebase and update UPDATING entry