depend-cleanup: Force a clean build when options change

[des]

Summary

Similar to the build epoch check, cache a list of source options in the
object tree, and force a clean build if the cached list does not match
the current list, after filtering out options which are known not to
affect the build (e.g. CLEAN, TESTS, WARNS).

This also adds a DEPEND_CLEANUP option (which defaults to yes unless
the CLEAN option is set) which can be used to skip depend-cleanup for
faster incremental builds.

[des]

define srcopts earlier

[kevans]

tools/build/depend-cleanup.sh

209

I tend to think it'd make more sense to go the other way around and enumerate options known to be broken when flipped, otherwise introducing a knob like GH_BC triggers needless rebuilds.

If not, we should certainly add more options to this list; there's no sense in rebuilding everything for knobs like BOOTPD/BOOTPARAMD that are just building or installing leaf components without needing to trigger, e.g., re-linking elsewhere.

[markj]

tools/build/depend-cleanup.sh

209

The problem is that list then needs to be manually maintained, which means that developers will forget to update it when they add new src.conf options, which means that we'll have unreliable builds. It's also pretty hard to know with certainty whether a given option is actually safe to toggle.

I'm a bit skeptical that users or developers are toggling options frequently enough for this to be a problem, and if they are, it's probably a small group of options that we can add to this list one by one. (For me, those options are WITHOUT_ZFS_TESTS=, which currently can't be toggled without breaking kyuafiles, and WITHOUT_{CLANG,LLD,LLDB}=, which are probably safe to toggle.)

We could maybe explicitly list both sets of options, and make it a fatal error for an option to be specified in src.opts.mk but not listed here. Then, when adding a new option, developers need to make sure they update this file.

297

I'd print this message unconditionally, it could be a useful breadcrumb if builds are taking longer than one expects.

[markj]

tools/build/depend-cleanup.sh

209

I guess kernel-specific options also don't need to be considered here, e.g., WITHOUT_KERNEL_SYMBOLS?

[kevans]

tools/build/depend-cleanup.sh

209

Right, after chatting with Dag-Erling a bit more about this offline, I think I'd mostly just prefer we add some options that are clearly harmless to toggle -- BOOTPD, BOOTPARAMD, CLANG, LLD, LLDB, for instance, aren't really doing anything invasive.

All of the _SUPPORT knobs will more or less change scattered bits around the tree, clearly those should never be added.

I have a list from auditing src knobs a week or two ago of knobs that would definitely break, I'd be happy to turn that into a list of knobs the other way that we could add here[0]. I'm inclined to avoid bugging developers to have to touch a third list (share/mk/src.opts.mk, tools/build/options, here) just to add a single option and would agree that it's better to assume that a newly-introduced one would break things until identified otherwise.

[0] Acknowledging that changes to the build could invalidate that list just as easily, but the odds of options like FREEBSD_UPDATE/FTP/FINGER growing new behaviors that would invalidate the assumption of safety are pretty slim.

[jhb]

tools/build/depend-cleanup.sh

209

WITHOUT_TOOLCHAIN is one that I use at times to do a quicker incremental build, and so it would be nice if that didn't force a clean build. That toggles various things and presumably you have to ignore all those options as well as TOOLCHAIN?

[des]

• Make “Cleaning ...” messages unconditional as requested by @markj.

• Add TOOLCHAIN and related options to IGNORED_OPTS as requested by @jhb. The rationale is that not only are they safe to toggle, they also have a massive impact on build time and are therefore commonly used temporarily to speed up a build. Many other options are equally safe to toggle, but their impact does not justify the cost of having to maintain the list, or the risk of getting it wrong.

[imp]

So I need to make sure there's not a blast radius on this one.

A common pattern is for there to be one build tree, with lots of options turned on, and then lots of installs from that tree with differing sets of options. This is done to speed the build. If suddenly world gets cleaned unconditionally, without my control, that could be very bad.

I get the epoch thing. It makes sense. That's "tree through time" domain. But to force it for build options without the ability to override with a "Yes, I know what I'm doing, don't be daft: I don't need to rebuild from clean when I toggle bluetooth or sendmail"

[imp]

Offline we talked. I'm totally cool with whatever so long as I can override the forced clean.

[des]

Add DEPEND_CLEANUP option

[des]

In D52011#1236230, @imp wrote:

A common pattern is for there to be one build tree, with lots of options turned on, and then lots of installs from that tree with differing sets of options. This is done to speed the build. If suddenly world gets cleaned unconditionally, without my control, that could be very bad.

This code only runs when building, not when installing.

[des]

share/man/man5/src.conf.5

458

This is untrue, but I'm not sure how to get makeman to DTRT here.