Page MenuHomeFreeBSD

Rewrite the Updating from Source section of the Handbook
ClosedPublic

Authored by wblock on Aug 26 2016, 7:23 PM.

Details

Summary

The section of the Handbook that documents building from source is byzantine and bizarre, disjointed and difficult to follow. Rip it out by the roots and start over. This rewrite shows etcupdate as preferred to mergemaster. It begins with a Quick Start to summarize the process. Later sections add more detail. Some sections need more details, like the ones lacking any detail at all.

Test Plan

Repeat until adequate to commit:

Rewrite document.
Show to people.
Hope they will help fix flaws.
Fix flaws.

Rendered preview version: http://www.wonkity.com/~wblock/tmp/updating/updating-src.html

Diff Detail

Repository
rD FreeBSD doc repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
wblock updated this revision to Diff 19733.Aug 26 2016, 7:23 PM
wblock set the repository for this revision to rD FreeBSD doc repository.
bcr added a subscriber: bcr.Aug 26 2016, 7:51 PM

Warren, thank you for working on this and including etcupdate. I've added a few comments of my own.

cutting-edge/chapter.xml
1099 ↗(On Diff #19733)

s/computer/FreeBSD system/

1173 ↗(On Diff #19733)

Maybe mention which step to jump to when not doing to the first time? Something like "If etcupdate extract has been performed before, continue with step X."

1185 ↗(On Diff #19733)

Ensure that svn is installed first or provide svnlite as an alternative?

1317 ↗(On Diff #19733)

Yes, we should describe the boring, default way of doing it here and not get lost in fancy optimizations. Things like MODULES_OVERRIDE and src.conf entries can easily be on their own (next) page. devel/ccache can also be mentioned there, but not here.

1348 ↗(On Diff #19733)

I'm thinking whether we should describe this here or not. Like I wrote above, it can distract from the standard, boring default way. People who want to it more advanced should refer to that section.

allanjude added inline comments.
cutting-edge/chapter.xml
1095 ↗(On Diff #19733)

step 1 should be making sure you have a src checkout. Since this is quick start, it might be worth duplicating the bit about how to check out the src from the appendix.

1118 ↗(On Diff #19733)

this should be buildkernel

1119 ↗(On Diff #19733)

kernel should be installed first, and the system rebooted onto the new kernel, before installing the newer world.

Otherwise things like top/ps might break.

1154 ↗(On Diff #19733)

you didn't actually install the kernel

wblock marked 2 inline comments as done.Aug 27 2016, 3:28 AM
wblock added inline comments.
cutting-edge/chapter.xml
1118 ↗(On Diff #19733)

The kernel target is buildkernel + installkernel.

1119 ↗(On Diff #19733)

top or ps would only be broken until the final/only reboot, the last step in the process. So that would be kind of self-inflicted by the reader not following the procedure.

Ideally, I'd like to make this quick start even shorter, preferably no more than three steps. Refinements can be explained in the detail sections that come later.

1154 ↗(On Diff #19733)

Did too! kernel is buildkernel + installkernel. The idea here is to boil the 384-step procedure into as few steps as possible.

1173 ↗(On Diff #19733)

If possible, this step should be removed entirely. My experience is that people will perform steps as shown, even if it says not to do it. So the question is: can etcupdate be like mergemaster, and figure out when it is being run for the first time?

1185 ↗(On Diff #19733)

All of the svnlite help pages refer to svn, because it really is svn. Rather than try to document all the other possible ways to do things, this guide tries to pick just the preferred method. So svn is preferred because it will be up to date, where svnlite is in base and it's impossible to tell how obsolete it might be.

1317 ↗(On Diff #19733)

Yes. Of course, the build system has big differences between 10.x and 11.x, so that is a challenge.

1348 ↗(On Diff #19733)

Mainly it is here because it is such a common concern. Putting it here keeps it out of the quick start.

woodsb02 edited edge metadata.Aug 29 2016, 2:21 PM

Hi Warren,
Nice work getting this draft done and up for review - apologies I haven't helped more to date.
I really like this simplification, so please don't take the number of my comments as meaning I am negative towards this work - I like it and just want to ensure it is both concise and accurate.

cutting-edge/chapter.xml
1071 ↗(On Diff #19733)

Why rip out the section that describes what FreeBSD-STABLE is? In my mind this should stay, as it immediately follows the description of FreeBSD-CURRENT, and is referred to in the first paragraph after the title "Tracking a Development Branch".

Refer here for how it looks today: https://www.freebsd.org/doc/handbook/current-stable.html

1095 ↗(On Diff #19733)

Just to clarify, checking /usr/src/UPDATING should only be done after the source has been updated.

Hence, checking /usr/src/UPDATING should instead be a step after "svn update" and before "make -j4 buildworld".

1095 ↗(On Diff #19733)

What the user can and should do at this early stage, is:

  • Backup, take filesystem snapshot, and/or create new boot environment
  • Review the recent freebsd-stable or freebsd-current mailing list entries, specifically looking for "HEADS UP" items which are bringing significant change or reports of broken functionality / unexpected behaviour.
1109 ↗(On Diff #19733)

This is true as long as the user does each and every update using etcupdate(8) to merge the configuration files. If a user does a binary update, or a source update using the alternative mergemaster(8), then the etcupdate(8) stored checkpoint will be out of date and will need to be re-bootstrapped by once again running "etcupdate extract" using a version of the source tree which matches the currently running system.

1115 ↗(On Diff #19733)

Might be cleaner to instruct the user to "cd /usr/src" and then "svn update", followed by other commands that need /usr/src to be the working directory.

1117 ↗(On Diff #19733)

Should we prompt users to save the output of the build by either first running script(1) or piping the output to tee(1)? This will allow them to review the output later in case of any errors, and post it to the mailing lists if seeking help. Perhaps this could be added to the detailed section below... but as you said, if you want people to do it, it should be in the quick start.

1141 ↗(On Diff #19733)

This should say "Compile and install the kernel."

1165 ↗(On Diff #19733)

Again, checking /usr/src/UPDATING should be done after "svn update".

1172 ↗(On Diff #19733)

During an extract, etcupdate is not making a record of the system settings, it is making a record of the default configuration files (as they would be installed from the source tree) in the pre-update state.

Since it is performing a 3-way merge, etcupdate needs 3 copies of the source to be able to determine how to perform the merge: the pre-update default version (from the source tree before "svn update"), the post-update default version (from the source tree after "svn update"), and the as-installed version (from /etc/).

"etcupdate extract" is creating a record of the pre-update default versions, since when the user runs the "etcupdate" command from /usr/src post-update, the pre-update copy will not otherwise be available on the system for comparison and 3-way merge.

1173 ↗(On Diff #19733)

The difficult thing for etcupdate to determine is, is the configuration file in /etc different from the one in /usr/src because the user edited it, or because the default version in the source tree has changed.

When you first install FreeBSD, it actually comes with a copy of the etcupdate checkpoint pre-populated with the as-installed source... so this "etcupdate extract" command is not technically required when performing the FIRST update.

The tricky part is when this is the first time the user has used etcupdate to perform post-update configuration file merging, but has performed previous binary updates or source updates using mergemaster(8), and therefore the pre-populated etcupdate checkpoint is out of date, and will lead to incorrect 3-way merges. This scenario is when the user really needs to re-bootstrap via the "etcupdate extract" command.

Each time etcupdate is used for merging, it saves a copy of the source tree configuration files used for comparison with the next time. Hence why it is said the "etcupdate extract" is only required the first time.

If the user wishes to check if the current etcupdate checkpoint is accurate, they should run "etcupdate diff" and confirm that the output correctly reflects each of their local /etc changes/customizations over the default version from the source tree, and no other changes.

1202 ↗(On Diff #19733)

Rather than repeating it, perhaps this should reference the existing section of the appendix which explains how to use subversion to checkout source?
https://www.freebsd.org/doc/handbook/svn.html

1332 ↗(On Diff #19733)

I would tend to use "rm -rf /usr/obj/*" to avoid removing the top directory of /usr/obj itself. This is because I tend to have that directory mounted as a separate zfs dataset. Without needing to describe the reasons for this, "rm -rf /usr/obj/*" may just cater for more use-cases.

1332 ↗(On Diff #19733)

Is it still the case that some files in the build directories may be set as immutable, and therefore "chflags -R noschg /usr/obj/*" may first be required? I have not needed this in a long time...

1341 ↗(On Diff #19733)

This section should explain how "make buildkernel" differs from "make kernel" (which was the basic step given above).

1372 ↗(On Diff #19733)

Perhaps also mention that instead of copying GENERIC (which means your custom config will not automatically receive any future changes to GENERIC), you can instead create a small custom kernel config that begins with the line "include GENERIC" and then has subsequent lines which overwrite the GENERIC kernel config.

1399 ↗(On Diff #19733)

Perhaps clarify that "installkernel" is only needed if "buildkernel" was used instead of "kernel"

1444 ↗(On Diff #19733)

Perhaps also mention you can use "make -DBATCH_DELETE_OLD_FILES delete-old" if you don't want to be prompted for each file that will be deleted.

1461 ↗(On Diff #19733)

Again, "make -DBATCH_DELETE_OLD_FILES delete-old-libs" can be used if you don't want to be prompted for each file that will be deleted.

cem added a subscriber: cem.Sep 1 2016, 9:40 PM

A much needed refresh for a very frequently referenced page. Good stuff!

cutting-edge/chapter.xml
1141 ↗(On Diff #19733)

It might be good to call out "make installkernel" separately as its useful for installing custom kernels even if it's included as part of installworld (Is it - I didn't know that). Its good feedback for the user to let them they they can do it. Doesn't affect the quick start approach.

1154 ↗(On Diff #19733)

+1 allanjude

wblock: your text is shorter but omitting installkernel seems cutting down on too much info. One extra line won't affect the quick start more importantly because its a crucial step.

1247 ↗(On Diff #19733)

The references to 10.3-release: is there a way to link to the current release automatically? New users might see 11 being the latest and greatest advertised everywhere and might think this doc is out of date. Just a suggestion.

1282 ↗(On Diff #19733)

Should be make buildkernel to match rest of text

1320 ↗(On Diff #19733)

I would suggest we move this above before starting build. Most users would want to remove the obj directory before starting a clean build.

1414 ↗(On Diff #19733)

A description of etcupdate and mergemaster would be useful, specifically noting that the user doesn't need to do both (if that's the case).

1424 ↗(On Diff #19733)

This should be wrapped in <screen></screen>?

1431 ↗(On Diff #19733)

This should be wrapped in <screen></screen>?

wblock marked 18 inline comments as done.Sep 30 2016, 7:02 PM
wblock added inline comments.
cutting-edge/chapter.xml
1071 ↗(On Diff #19733)

It is wordy and confusing. I'll add a third column to the table in the Updating the Source section that briefly describes each branch.

1095 ↗(On Diff #19733)

step 1 should be making sure you have a src checkout

This is meant to be a reminder of the standard steps rather than the exceptions. Maybe it should not be called a quick start, but an overview.
Details can go in the detail sections afterward. And the callout does link there.

What the user can and should do at this early stage

These details are also better in the later sections where they will not obscure the basic steps.

1099 ↗(On Diff #19733)

I don't understand what you are saying here. Is the distinction that we might be in a VM or a multi-boot system rather than on a FreeBSD-only computer?

1109 ↗(On Diff #19733)

This would be better mentioned in the detail sections below. In a section where we only show using etcupdate, it is a distraction to warn the user against using something we did not show.

1115 ↗(On Diff #19733)

It would save a little typing, but I feel that explicitly giving the directory in the update command is useful to illustrate which directory is being acted on.

1117 ↗(On Diff #19733)

Detail section would be better.

1141 ↗(On Diff #19733)

There is a bit about this in the detail section.

1154 ↗(On Diff #19733)

The kernel is installed. The "it's just a wafer-thin mint" approach ends up with a long, detailed list of instructions like the old version of this document, where it is too big to remember and has many exceptions that do not matter to the typical user. All of that details, or rather the part of it that is required, can go in the detail sections that come later in this chapter.

1172 ↗(On Diff #19733)

The challenge is to explain it to the reader without so much detail that it clouds the explanation.

1202 ↗(On Diff #19733)

I wrote some of that, but it is vague and not specific to the purposes here. Also, I have wanted for a while to split all the Subversion stuff out of at least four of our books into a single Subversion for FreeBSD book. Not that I've done it, but still.

1247 ↗(On Diff #19733)

We have entities for OS release version numbers, but not separate entities for major and minor version numbers, as far as I'm aware.

1320 ↗(On Diff #19733)

Moved to the start of the "additional options", but not before the "basic steps" section. Deleting /usr/obj is hardly ever necessary in any case.

1332 ↗(On Diff #19733)

The build scripts do it. At least it has been years since I've done it, so no, it is no longer required on 10, 11, or 12.

1341 ↗(On Diff #19733)

Added a note to the callouts in the quick start to cover this.

1372 ↗(On Diff #19733)

That can be added as one of these special options section. I considered showing the good method of creating the kernel config file in /root and linking it into /usr/src, so it does not go away when /usr/src is deleted. But that can also be a section added later.

1399 ↗(On Diff #19733)

Changed the long form instructions to use buildkernel, so this matches.

1414 ↗(On Diff #19733)

This needs to be expanded, yes. And I'm considering removing mergemaster entirely. But I still do not have a feel for etcupdate. It seems relatively "smart" and is probably the only thing to recommend. If only there were some easy way to explain three-way merges, or in fact, any kind of merges to the average reader. Getting away from horrible old-style non-unified diffs is a start.

1424 ↗(On Diff #19733)

It's just a placeholder screaming "expound here!"

1431 ↗(On Diff #19733)

As is/was this... but it's on borrowed time.

1444 ↗(On Diff #19733)

Or just yes | make delete-old if you don't care about how the screen looks afterward.

1454 ↗(On Diff #19733)

I would really, really like to know how to do this. Even the current docs tell the reader "Yes, delete these old libraries! Oh, but it might mean some of your old programs no longer work. Better yet, there is no way to tell which!"

Probably the way to address this is to say that if any old programs no longer work, they must just be recompiled.

1461 ↗(On Diff #19733)

The more modern notation would be make BATCH_DELETE_OLD_FILES=yes delete-old-libs, but I have not tried it. It makes the command much more intimidating. Maybe a single tip to mention this...

wblock updated this revision to Diff 20883.Sep 30 2016, 7:03 PM
wblock edited edge metadata.
wblock marked 9 inline comments as done.

Updated for feedback.

jhb edited edge metadata.Sep 30 2016, 9:19 PM

Perhaps not on this page, but somewhere it would be nice to mention 'etcupdate diff'.

cutting-edge/chapter.xml
1098 ↗(On Diff #20883)

FYI, if you have done a fresh install of 10.1 or later then this step is not necessary. Put another way, you should only have to do this for the first upgrade from 10.0 or earlier. (You don't need it for 9.3 either it turns out, but 10.0 and older is probably a simpler way of saying it).

1116 ↗(On Diff #20883)

You may need to include 'etcupdate -p' as a step between 'make buildworld' and 'make kernel'. It is rarely needed, but you can get installworld/installkernel failures if those need a newly added user or group.

1446 ↗(On Diff #20883)

I think it should be clear that you use either etcupdate or mergemaster, but not both. I'd hate to confuse users.

It is probably worth mentioning running 'etcupdate resolve' to resolve any conflicts reported by 'etcupdate'.

matthew added inline comments.Jan 10 2017, 8:57 PM
cutting-edge/chapter.xml
1454 ↗(On Diff #19733)

You can grep for the names of the old libraries in the output of procstat -af -- needs a bit of scripting to pull out the process name for each PID, but that's not difficult. Obviously this only picks out processes that are actually being run, but even so, it is pretty effective.

wblock updated this revision to Diff 28271.May 12 2017, 5:01 PM

Removed mention of etcupdate, leaving that for someone who actually uses it. Switch from deleting /usr/obj to using 'make cleanworld' on advice of bdrewery.

This revision was automatically updated to reflect the committed changes.
cem added a comment.May 12 2017, 6:27 PM

Removed mention of etcupdate, leaving that for someone who actually uses it.

Etcupdate was the main improvement (for me) in this change :-(.

I use it, but don't know if I'm doing it correctly or not. I skip mergemaster -p entirely. Here is what I use in place of mergemaster -iF:

etcupdate -F -s /path/to/my/src/tree

I don't recall if I had to do any initial setup for etcupdate, but I believe I may have.

bcr added a comment.May 12 2017, 7:34 PM

Here's what I did recently using etcupdate for the first time (worked flawlessly, will use it in the future like this):

  1. Bootstrap etcupdate (only needed once). Do this before upgrading your kernel sources (in /usr/src):
etcupdate extract
  1. Update the src tree in /usr/src, followed by (-j option omitted for simplicity)
make buildworld
make kernel
  1. In single user mode, install the world via
make installworld
  1. Now, simply run
etcupdate
  1. (optional)
make check-old && yes|make delete-old && yes|make delete-old-libs
  1. reboot into multi-user mode, done!

There is also an EXAMPLES section in etcupdate(8). Thanks for doing the work, it is definitely an improvement of the chapter!

jhb added a comment.May 12 2017, 9:47 PM

In reply to @bcr: For modern releases (10.0 and later I think), etcupdate is now bootstrapped during an install, so you can skip that first step on fresh installs. You can use 'etcupdate diff' to verify if you need to extract or not.

In D7665#221728, @cem wrote:

Removed mention of etcupdate, leaving that for someone who actually uses it.

Etcupdate was the main improvement (for me) in this change :-(.

Sorry about that. This rewrite has been held up way too long, and the situation was made worse recently by minor edits to the old version, subsequent merge conflicts, and Subversion "helping". The new layout is modular and instructions for etcupdate can be added by someone who uses it. I tried it, and think the merge functionality might be nice, but it did not seem to be a full replacement for mergemaster. For instance, it did not update MOTD. I might have been using it wrong, but don't expect to have more time to work on it any time soon.

cem added a comment.May 13 2017, 3:20 AM

Sorry about that. This rewrite has been held up way too long, and the situation was made worse recently by minor edits to the old version, subsequent merge conflicts, and Subversion "helping". The new layout is modular and instructions for etcupdate can be added by someone who uses it. I tried it, and think the merge functionality might be nice, but it did not seem to be a full replacement for mergemaster. For instance, it did not update MOTD. I might have been using it wrong, but don't expect to have more time to work on it any time soon.

Understood! I don't mean to complain too much. There's no need to fix everything at once; monotonic improvement is sufficient :-).