Index: head/en_US.ISO8859-1/books/handbook/cutting-edge/chapter.xml =================================================================== --- head/en_US.ISO8859-1/books/handbook/cutting-edge/chapter.xml (revision 43772) +++ head/en_US.ISO8859-1/books/handbook/cutting-edge/chapter.xml (revision 43773) @@ -1,3177 +1,3074 @@ Updating and Upgrading &os; Jim Mock Restructured, reorganized, and parts updated by Jordan Hubbard Original work by Poul-Henning Kamp John Polstra Nik Clayton Synopsis &os; is under constant development between releases. Some people prefer to use the officially released versions, while others prefer to keep in sync with the latest developments. However, even official releases are often updated with security and other critical fixes. Regardless of the version used, &os; provides all the necessary tools to keep the system updated, and allows for easy upgrades between versions. This chapter describes how to track the development system and the basic tools for keeping a &os; system up-to-date. After reading this chapter, you will know: - Which utilities are available to update the system and - the Ports Collection. - - - How to keep a &os; system up-to-date with freebsd-update, Subversion, or CTM. How to compare the state of an installed system against a known pristine copy. How to keep the installed documentation up-to-date with Subversion or documentation ports. The difference between the two development branches: &os.stable; and &os.current;. How to rebuild and reinstall the entire base system. Before reading this chapter, you should: Properly set up the network connection (). Know how to install additional third-party software (). Throughout this chapter, svn is used to obtain and update &os; sources. To use it, first install the devel/subversion port or package. &os; Update Tom Rhodes Written by Colin Percival Based on notes provided by Updating and Upgrading freebsd-update updating-upgrading Applying security patches in a timely manner and upgrading to a newer release of an operating system are important aspects of ongoing system administration. &os; includes a utility called freebsd-update which can be used to perform both these tasks. This utility supports binary security and errata updates to &os;, without the need to manually compile and install the patch or a new kernel. Binary updates are available for all architectures and releases currently supported by the security team. The list of supported releases and their estimated end-of-life dates are listed at http://www.FreeBSD.org/security/. This utility also supports operating system upgrades to minor point releases as well as upgrades to another release branch. Before upgrading to a new release, review its release announcement as it contains important information pertinent to the release. Release announcements are available from http://www.FreeBSD.org/releases/. If a crontab utilizing the features of &man.freebsd-update.8; exists, it must be disabled before upgrading the operating system. This section describes the configuration file used by freebsd-update, demonstrates how to apply a security patch and how to upgrade to a minor or major operating system release, and discusses some of the considerations when upgrading the operating system. The Configuration File The default configuration file for freebsd-update works as-is. Some users may wish to tweak the default configuration in /etc/freebsd-update.conf, allowing better control of the process. The comments in this file explain the available options, but the following may require a bit more explanation: # Components of the base system which should be kept updated. Components world kernel This parameter controls which parts of &os; will be kept up-to-date. The default is to update the entire base system and the kernel. Individual components can instead be specified, such as src/base or src/sys. However, the best option is to leave this at the default as changing it to include specific items requires every needed item to be listed. Over time, this could have disastrous consequences as source code and binaries may become out of sync. # Paths which start with anything matching an entry in an IgnorePaths # statement will be ignored. IgnorePaths /boot/kernel/linker.hints To leave specified directories, such as /bin or /sbin, untouched during the update process, add their paths to this statement. This option may be used to prevent freebsd-update from overwriting local modifications. # Paths which start with anything matching an entry in an UpdateIfUnmodified # statement will only be updated if the contents of the file have not been # modified by the user (unless changes are merged; see below). UpdateIfUnmodified /etc/ /var/ /root/ /.cshrc /.profile This option will only update unmodified configuration files in the specified directories. Any changes made by the user will prevent the automatic updating of these files. There is another option, KeepModifiedMetadata, which will instruct freebsd-update to save the changes during the merge. # When upgrading to a new &os; release, files which match MergeChanges # will have any local changes merged into the version from the new release. MergeChanges /etc/ /var/named/etc/ /boot/device.hints List of directories with configuration files that freebsd-update should attempt to merge. The file merge process is a series of &man.diff.1; patches similar to &man.mergemaster.8;, but with fewer options. Merges are either accepted, open an editor, or cause freebsd-update to abort. When in doubt, backup /etc and just accept the merges. See for more information about mergemaster. # Directory in which to store downloaded updates and temporary # files used by &os; Update. # WorkDir /var/db/freebsd-update This directory is where all patches and temporary files are placed. In cases where the user is doing a version upgrade, this location should have at least a gigabyte of disk space available. # When upgrading between releases, should the list of Components be # read strictly (StrictComponents yes) or merely as a list of components # which *might* be installed of which &os; Update should figure out # which actually are installed and upgrade those (StrictComponents no)? # StrictComponents no When this option is set to yes, freebsd-update will assume that the Components list is complete and will not attempt to make changes outside of the list. Effectively, freebsd-update will attempt to update every file which belongs to the Components list. Security Patches &os; security patches may be downloaded and installed using the following command: &prompt.root; freebsd-update fetch &prompt.root; freebsd-update install If the update applied any kernel patches, the system will need a reboot in order to boot into the patched kernel. Otherwise, the system should be patched and freebsd-update may be run as a nightly &man.cron.8; job by adding this entry to /etc/crontab: @daily root freebsd-update cron This entry states that freebsd-update will run once every day. When run with , freebsd-update will only check if updates exist. If patches exist, they will automatically be downloaded to the local disk but will not be applied. The root user will be sent an email so that they may be reviewed and manually installed. If anything goes wrong, freebsd-update has the ability to roll back the last set of changes with the following command: &prompt.root; freebsd-update rollback Once complete, the system should be restarted if the kernel or any kernel modules were modified. This will allow &os; to load the new binaries into memory. Only the GENERIC kernel can be automatically updated by freebsd-update. If a custom kernel is installed, it will have to be rebuilt and reinstalled after freebsd-update finishes installing the rest of the updates. However, freebsd-update will detect and update the GENERIC kernel if /boot/GENERIC exists, even if it is not the current running kernel of the system. It is a good idea to always keep a copy of the GENERIC kernel in /boot/GENERIC. It will be helpful in diagnosing a variety of problems, and in performing version upgrades using freebsd-update as described in . Unless the default configuration in /etc/freebsd-update.conf has been changed, freebsd-update will install the updated kernel sources along with the rest of the updates. Rebuilding and reinstalling a new custom kernel can then be performed in the usual way. The updates distributed by freebsd-update do not always involve the kernel. It is not necessary to rebuild a custom kernel if the kernel sources have not been modified by the execution of freebsd-update install. However, freebsd-update will always update /usr/src/sys/conf/newvers.sh. The current patch level, as indicated by the -p number reported by uname -r, is obtained from this file. Rebuilding a custom kernel, even if nothing else changed, allows &man.uname.1; to accurately report the current patch level of the system. This is particularly helpful when maintaining multiple systems, as it allows for a quick assessment of the updates installed in each one. Major and Minor Version Upgrades Upgrades from one minor version of &os; to another, like from &os; 9.0 to &os; 9.1, are called minor version upgrades. Generally, installed applications will continue to work without problems after minor version upgrades. Major version upgrades occur when &os; is upgraded from one major version to another, like from &os; 8.X to &os; 9.X. Major version upgrades remove old object files and libraries which will break most third party applications. It is recommended that all installed ports either be removed and re-installed or upgraded after a major version upgrade using a utility such as ports-mgmt/portmaster. A brute-force rebuild of all installed applications can be accomplished with this command: &prompt.root; portmaster -af This will ensure everything will be re-installed correctly. Note that setting the BATCH environment variable to yes will answer yes to any prompts during this process, removing the need for manual intervention during the build process. Dealing with Custom Kernels If a custom kernel is in use, the upgrade process is slightly more involved, and the procedure varies depending on the version of &os;. Custom Kernels with &os; 8.X A copy of the GENERIC kernel is needed, and should be placed in /boot/GENERIC. If the GENERIC kernel is not present in the system, it may be obtained using one of the following methods: If a custom kernel has only been built once, the kernel in /boot/kernel.old is actually GENERIC. Rename this directory to /boot/GENERIC. Assuming physical access to the machine is possible, a copy of the GENERIC kernel can be installed from the installation media using the following commands: &prompt.root; mount /cdrom &prompt.root; cd /cdrom/X.Y-RELEASE/kernels &prompt.root; ./install.sh GENERIC Replace X.Y-RELEASE with the actual version of the release being used. The GENERIC kernel will be installed in /boot/GENERIC by default. Failing all the above, the GENERIC kernel may be rebuilt and installed from source: &prompt.root; cd /usr/src &prompt.root; env DESTDIR=/boot/GENERIC make kernel __MAKE_CONF=/dev/null SRCCONF=/dev/null &prompt.root; mv /boot/GENERIC/boot/kernel/* /boot/GENERIC &prompt.root; rm -rf /boot/GENERIC/boot For this kernel to be picked up as GENERIC by freebsd-update, the GENERIC configuration file must not have been modified in any way. It is also suggested that it is built without any other special options. Rebooting to the GENERIC kernel is not required at this stage. Custom Kernels with &os; 9.X and Later If a custom kernel has only been built once, the kernel in /boot/kernel.old is actually the GENERIC kernel. Rename this directory to /boot/kernel. If physical access to the machine is available, a copy of the GENERIC kernel can be installed from the installation media using these commands: &prompt.root; mount /cdrom &prompt.root; cd /cdrom/usr/freebsd-dist &prompt.root; tar -C/ -xvf kernel.txz boot/kernel/kernel If the options above cannot be used, the GENERIC kernel may be rebuilt and installed from source: &prompt.root; cd /usr/src &prompt.root; make kernel __MAKE_CONF=/dev/null SRCCONF=/dev/null For this kernel to be identified as the GENERIC kernel by freebsd-update, the GENERIC configuration file must not have been modified in any way. It is also suggested that the kernel is built without any other special options. Rebooting to the GENERIC kernel is not required at this stage. Performing the Upgrade Major and minor version upgrades may be performed by providing freebsd-update with a release version target. The following command will update to &os; 9.1: &prompt.root; freebsd-update -r 9.1-RELEASE upgrade After the command has been received, freebsd-update will evaluate the configuration file and current system in an attempt to gather the information necessary to perform the upgrade. A screen listing will display which components have and have not been detected. For example: Looking up update.FreeBSD.org mirrors... 1 mirrors found. Fetching metadata signature for 9.0-RELEASE from update1.FreeBSD.org... done. Fetching metadata index... done. Inspecting system... done. The following components of FreeBSD seem to be installed: kernel/smp src/base src/bin src/contrib src/crypto src/etc src/games src/gnu src/include src/krb5 src/lib src/libexec src/release src/rescue src/sbin src/secure src/share src/sys src/tools src/ubin src/usbin world/base world/info world/lib32 world/manpages The following components of FreeBSD do not seem to be installed: kernel/generic world/catpages world/dict world/doc world/games world/proflibs Does this look reasonable (y/n)? y At this point, freebsd-update will attempt to download all files required for the upgrade. In some cases, the user may be prompted with questions regarding what to install or how to proceed. When using a custom kernel, the above step will produce a warning similar to the following: WARNING: This system is running a "MYKERNEL" kernel, which is not a kernel configuration distributed as part of FreeBSD 9.0-RELEASE. This kernel will not be updated: you MUST update the kernel manually before running "/usr/sbin/freebsd-update install" This warning may be safely ignored at this point. The updated GENERIC kernel will be used as an intermediate step in the upgrade process. Once all the patches have been downloaded to the local system, they will be applied. This process may take a while, depending on the speed and workload of the machine. Configuration files will then be merged. The merging process requires some user intervention as a file may be merged or an editor may appear on screen for a manual merge. The results of every successful merge will be shown to the user as the process continues. A failed or ignored merge will cause the process to abort. Users may wish to make a backup of /etc and manually merge important files, such as master.passwd or group at a later time. The system is not being altered yet as all patching and merging is happening in another directory. Once all patches have been applied successfully, all configuration files have been merged and it seems the process will go smoothly, the changes can be committed to disk by the user using the following command: &prompt.root; freebsd-update install The kernel and kernel modules will be patched first. At this point, the machine must be rebooted. If the system is running with a custom kernel, use &man.nextboot.8; to set the kernel for the next boot to the updated /boot/GENERIC: &prompt.root; nextboot -k GENERIC Before rebooting with the GENERIC kernel, make sure it contains all the drivers required for the system to boot properly and connect to the network, if the machine being updated is accessed remotely. In particular, if the running custom kernel contains built-in functionality usually provided by kernel modules, make sure to temporarily load these modules into the GENERIC kernel using the /boot/loader.conf facility. It is recommended to disable non-essential services as well as any disk and network mounts until the upgrade process is complete. The machine should now be restarted with the updated kernel: &prompt.root; shutdown -r now Once the system has come back online, restart freebsd-update using the following command. The state of the process has been saved and thus, freebsd-update will not start from the beginning, but will remove all old shared libraries and object files. &prompt.root; freebsd-update install Depending upon whether any library version numbers were bumped, there may only be two install phases instead of three. Rebuilding Ports After a Major Version Upgrade After a major version upgrade, all third party software needs to be rebuilt and re-installed. This is required as installed software may depend on libraries which have been removed during the upgrade process. This process can be automated using ports-mgmt/portmaster: &prompt.root; portmaster -af Once this has completed, finish the upgrade process with a final call to freebsd-update in order to tie up all the loose ends in the upgrade process: &prompt.root; freebsd-update install If the GENERIC kernel was temporarily used, this is the time to build and install a new custom kernel in the usual way. Reboot the machine into the new &os; version. The process is complete. System State Comparison freebsd-update can be used to test the state of the installed &os; version against a known good copy. This option evaluates the current version of system utilities, libraries, and configuration files. To begin the comparison, issue the following command: &prompt.root; freebsd-update IDS >> outfile.ids While the command name is IDS it is not a replacement for a real intrusion detection system such as security/snort. As freebsd-update stores data on disk, the possibility of tampering is evident. While this possibility may be reduced using kern.securelevel and by storing the freebsd-update data on a read only file system when not in use, a better solution would be to compare the system against a secure disk, such as a DVD or securely stored external USB disk device. The system will now be inspected, and a lengthy listing of files, along with the &man.sha256.1; hash values for both the known value in the release and the current installation, will be sent to the specified outfile.ids file. The entries in the listing are extremely long, but the output format may be easily parsed. For instance, to obtain a list of all files which differ from those in the release, issue the following command: &prompt.root; cat outfile.ids | awk '{ print $1 }' | more /etc/master.passwd /etc/motd /etc/passwd /etc/pf.conf This sample output has been truncated as many more files exist. Some files have natural modifications. For example, /etc/passwd has been modified because users have been added to the system. Other files, such as kernel modules, may differ as freebsd-update may have updated them. To exclude specific files or directories, add them to the IDSIgnorePaths option in /etc/freebsd-update.conf. This system may be used as part of an elaborate upgrade method, aside from the previously discussed version. - - - - - Portsnap: a Ports Collection Update Tool - - - - - Tom - Rhodes - - Written by - - - - - - - Colin - Percival - - Based on notes provided by - - - - - - Updating and Upgrading - - - Portsnap - Updating and Upgrading - - - The base system of &os; includes &man.portsnap.8; for - updating the Ports Collection. This utility connects to a - &os; site, verifies the secure key, and downloads a new copy of - the Ports Collection. The key is used to verify the integrity - of all downloaded files. To download the latest Ports - Collection files, issue the following command: - - &prompt.root; portsnap fetch -Looking up portsnap.FreeBSD.org mirrors... 9 mirrors found. -Fetching snapshot tag from geodns-1.portsnap.freebsd.org... done. -Fetching snapshot metadata... done. -Updating from Tue May 22 02:12:15 CEST 2012 to Wed May 23 16:28:31 CEST 2012. -Fetching 3 metadata patches.. done. -Applying metadata patches... done. -Fetching 3 metadata files... done. -Fetching 90 patches.....10....20....30....40....50....60....70....80....90. done. -Applying patches... done. -Fetching 133 new ports or files... done. - - What this example shows is that &man.portsnap.8; has found - and verified several patches to the current ports data. This - also indicates that the utility was run previously; if it was a - first time run, the collection would have simply been - downloaded. - - When &man.portsnap.8; successfully completes a - fetch operation, the Ports Collection and - subsequent patches which exist on the local system have passed - verification. The first time portsnap is - executed, use extract to install the - downloaded files: - - &prompt.root; portsnap extract -/usr/ports/.cvsignore -/usr/ports/CHANGES -/usr/ports/COPYRIGHT -/usr/ports/GIDs -/usr/ports/KNOBS -/usr/ports/LEGAL -/usr/ports/MOVED -/usr/ports/Makefile -/usr/ports/Mk/bsd.apache.mk -/usr/ports/Mk/bsd.autotools.mk -/usr/ports/Mk/bsd.cmake.mk -... - - To update an already installed Ports Collection, use - portsnap update: - - &prompt.root; portsnap update - - The process is now complete, and applications may be - installed or upgraded using the updated Ports Collection. - - When using fetch, the - extract or the update - operation may be run consecutively: - - &prompt.root; portsnap fetch update - - This command downloads the latest version of the Ports - Collection and updates the local version under - /usr/ports. Updating the Documentation Set Updating and Upgrading Documentation Updating and Upgrading Documentation is an integral part of the &os; operating system. While an up-to-date version of the &os; Documentation Set is always available on the &os; web site, some users might have slow or no permanent network connectivity. There are several ways to update the local copy of documentation with the latest &os; Documentation Set. Using <application>Subversion</application> to Update the Documentation The &os; documentation sources can be obtained with svn. This section describes how to: Install the documentation toolchain, the tools that are required to rebuild the &os; documentation from its source. Download a copy of the documentation source at /usr/doc, using svn. Rebuild the &os; documentation from its source, and install it under /usr/share/doc. Recognize some of the build options that are supported by the build system of the documentation, such as the options that build only some of the different language translations of the documentation or the options that select a specific output format. Installing <application>svn</application> and the Documentation Toolchain Rebuilding the &os; documentation from source requires a collection of tools which are not part of the &os; base system due to the amount of disk space these tools use. They are also not useful to all &os; users, only those users that are actively writing new documentation for &os; or are frequently updating their documentation from source. The required tools, including svn, are available in the textproc/docproj meta-port developed by the &os; Documentation Project. When no &postscript; or PDF documentation required, one might consider installing the textproc/docproj-nojadetex port instead. This version of the documentation toolchain includes everything except the teTeX typesetting engine. teTeX is a very large collection of tools, so it may be quite sensible to omit its installation if PDF output is not really necessary. Updating the Documentation Sources In this example, svn is used to fetch a clean copy of the documentation sources from the western US mirror using the HTTPS protocol: &prompt.root; svn checkout https://svn0.us-west.FreeBSD.org/doc/head /usr/doc Select the closest mirror from the available Subversion mirror sites. The initial download of the documentation sources may take a while. Let it run until it completes. Future updates of the documentation sources may be fetched by running: &prompt.root; svn update /usr/doc After checking out the sources, an alternative way of updating the documentation is supported by the /usr/doc/Makefile by running the following commands: &prompt.root; cd /usr/doc &prompt.root; make update Tunable Options of the Documentation Sources The updating and build system of the &os; documentation set supports a few options that ease the process of updating only parts of the documentation, or the build of specific translations. These options can be set either as system-wide options in /etc/make.conf, or as command-line options passed to &man.make.1;. The options include: DOC_LANG The list of languages and encodings to build and install, such as en_US.ISO8859-1 for English documentation. FORMATS A single format or a list of output formats to be built. Currently, html, html-split, txt, ps, pdf, and rtf are supported. DOCDIR Where to install the documentation. It defaults to /usr/share/doc. For more make variables supported as system-wide options in &os;, refer to &man.make.conf.5;. For more make variables supported by the build system of the &os; documentation, refer to the &os; Documentation Project Primer for New Contributors. Installing the &os; Documentation from Source Once an up-to-date snapshot of the documentation sources has been fetched to /usr/doc, everything is ready for an update of the installed documentation. A full update of all the languages defined in DOC_LANG may be performed by typing: &prompt.root; cd /usr/doc &prompt.root; make install clean If an update of only a specific language is desired, &man.make.1; can be invoked in a language specific subdirectory of /usr/doc: &prompt.root; cd /usr/doc/en_US.ISO8859-1 &prompt.root; make update install clean The output formats that will be installed may be specified by setting FORMATS: &prompt.root; cd /usr/doc &prompt.root; make FORMATS='html html-split' install clean For information on editing and submitting corrections to the documentation, refer to the &os; Documentation Project Primer for New Contributors. Using Documentation Ports Marc Fonvieille Based on the work of Updating and Upgrading documentation package Updating and Upgrading The previous section presented a method for updating the &os; documentation from sources. Source based updates may not be feasible or practical for all &os; systems as building the documentation sources requires the documentation toolchain, a certain level of familiarity with svn and source checkouts from a repository, and a few manual steps to build the checked out sources. This section describes an alternative method which uses the Ports Collection and makes it possible to: Download and install pre-built snapshots of the documentation, without having to locally build anything or install the documentation toolchain. Download the documentation sources and build them through the ports framework, making the checkout and build steps a bit easier. These two methods of updating the &os; documentation are supported by a set of documentation ports, updated by the &a.doceng; on a monthly basis. These are listed in the &os; Ports Collection, under the docs category. Building and Installing Documentation Ports The documentation ports use the ports building framework to make documentation builds easier. They automate the process of checking out the documentation source, running &man.make.1; with the appropriate environment settings and command-line options, and they make the installation or deinstallation of documentation as easy as the installation of any other &os; port or package. As an extra feature, when the documentation ports are built locally, they record a dependency to the documentation toolchain ports, so that they are also automatically installed. Organization of the documentation ports is as follows: The master port, misc/freebsd-doc-en, which installs all of the English documentation ports. The all in one port, misc/freebsd-doc-all, builds and installs all documentation in all available languages. There is a slave port for each translation, such as misc/freebsd-doc-hu for the Hungarian-language documents. For example, to build and install the English documentation in split HTML format, similar to the format used on http://www.FreeBSD.org, to /usr/local/share/doc/freebsd, install the following port &prompt.root; cd /usr/ports/misc/freebsd-doc-en &prompt.root; make install clean Common Knobs and Options There are many options for modifying the default behavior of the documentation ports, including: WITH_HTML Builds the HTML format with a single HTML file per document. The formatted documentation is saved to a file called article.html, or book.html, as appropriate, plus images. WITH_PDF Builds the &adobe; Portable Document Format (PDF). The formatted documentation is saved to a file called article.pdf or book.pdf, as appropriate. DOCBASE Specifies where to install the documentation. It defaults to /usr/local/share/doc/freebsd. The default target directory differs from the directory used svn. This is because ports are usually installed within /usr/local. This can be overridden by using PREFIX. This example uses variables to install the Hungarian documentation as a PDF: &prompt.root; cd /usr/ports/misc/freebsd-doc-hu &prompt.root; make -DWITH_PDF DOCBASE=share/doc/freebsd/hu install clean Using Documentation Packages Building the documentation ports from source, as described in the previous section, requires a local installation of the documentation toolchain and a bit of disk space for the build of the ports. When resources are not available to install the documentation toolchain, or because the build from sources would take too much disk space, it is still possible to install pre-built snapshots of the documentation ports. The &a.doceng; prepares monthly snapshots of the &os; documentation packages. These binary packages can be used with any of the bundled package tools, like &man.pkg.add.1;, &man.pkg.delete.1;, and so on. When binary packages are used, the &os; documentation will be installed in all available formats for the given language. For example, the following command will install the latest pre-built package of the Hungarian documentation: &prompt.root; pkg install hu-freebsd-doc Packages use a format that differs from the corresponding port's name: lang-freebsd-doc, where lang is the short format of the language code, such as hu for Hungarian, or zh_cn for Simplified Chinese. Updating Documentation Ports Documentation ports can be updated like any other port. For example, the following command updates the installed Hungarian documentation using ports-mgmt/portmaster by using packages only: &prompt.root; portmaster -PP hu-freebsd-doc Tracking a Development Branch -CURRENT -STABLE There are two development branches to &os;: &os.current; and &os.stable;. This section provides an explanation of each and describes how to keep a system up-to-date with each respective tree. &os.current; will be discussed first, then &os.stable;. Staying Current with &os; &os.current; is the bleeding edge of &os; development. &os.current; users are expected to have a high degree of technical skill and should be capable of solving difficult system problems on their own. If you are new to &os;, track &os.stable; instead. What Is &os.current;? snapshot &os.current; is the very latest source code for &os;. This includes work in progress, experimental changes, and transitional mechanisms that might or might not be present in the next official release of the software. While many &os; developers compile the &os.current; source code daily, there are periods of time when the sources are not buildable. These problems are resolved as quickly as possible, but whether or not &os.current; brings disaster or greatly desired functionality can be a matter of when the source code was synced. Who Needs &os.current;? &os.current; is made available for three primary interest groups: Members of the &os; community who are actively working on some part of the source tree and for whom keeping current is an absolute requirement. Members of the &os; community who are active testers, willing to spend time solving problems in order to ensure that &os.current; remains as sane as possible. These testers wish to make topical suggestions on changes and the general direction of &os;, and submit patches to implement them. Those who merely wish to keep an eye on things, or to use the current sources for reference purposes. These people also make the occasional comment or contribute code. What Is &os.current; <emphasis>Not</emphasis>? A fast-track to getting new features before the next release. Pre-release features are not yet fully tested and most likely contain bugs. A quick way of getting bug fixes. Any given commit is just as likely to introduce new bugs as to fix existing ones. In any way officially supported. Using &os.current; -CURRENT using Join the &a.current.name; and the &a.svn-src-head.name; lists. This is essential in order to see the comments that people are making about the current state of the system and to receive important bulletins which may be critical to the system's continued health. The &a.svn-src-head.name; list records the commit log entry for each change as it is made, along with any pertinent information on possible side-effects. To join these lists, go to &a.mailman.lists.link;, click on the list to subscribe to, and follow the instructions. In order to track changes to the whole source tree, subscribe to the &a.svn-src-all.name; list. Grab the sources from a &os; mirror site using one of the following methods: Use svn Subversion -CURRENT Syncing with Subversion to check out the desired development or release branch. This is the recommended method, providing access to &os; development as it occurs. Checkout the -CURRENT code from the head branch of one of the Subversion mirror sites. Due to the size of the repository, it is recommended that only desired subtrees be checked out. Use the CTM -CURRENT Syncing with CTM facility. If you have bad connectivity such as high price connections or only email access, CTM is an option, but it is not as reliable as Subversion. For this reason, Subversion is the recommended method for any system with Internet connectivity. If you plan to run, and not just look at the sources, download all of &os.current;, not just selected portions. Various parts of the source depend on updates elsewhere, and trying to compile just a subset is almost guaranteed to cause problems. Before compiling &os.current; -CURRENT compiling , read /usr/src/Makefile very carefully. Install a new kernel and rebuild the world the first time through as part of the upgrading process. Read the &a.current; and /usr/src/UPDATING to stay up-to-date on other bootstrapping procedures that sometimes become necessary on the road to the next release. Be active! &os.current; users are encouraged to submit their suggestions for enhancements or bug fixes. Suggestions with accompanying code are received most enthusiastically! Staying Stable with &os; What Is &os.stable;? -STABLE &os.stable; is the development branch from which major releases are made. Changes go into this branch at a different pace, and with the general assumption that they have first gone into &os.current; for testing. This is still a development branch, however, and this means that at any given time, the sources for &os.stable; may or may not be suitable for any particular purpose. It is simply another engineering development track, not a resource for end-users. Who Needs &os.stable;? Those interested in tracking or contributing to the FreeBSD development process, especially as it relates to the next point release of FreeBSD, should consider following &os.stable;. While security fixes go into the &os.stable; branch, one does not need to track &os.stable; to receive security fixes. Every security advisory for &os; explains how to fix the problem for the releases it affects which are not yet EOL. For a complete description of the current security policy for old releases of FreeBSD, refer to http://www.FreeBSD.org/security/.. While the &os.stable; branch should compile and run at all times, this cannot be guaranteed. While code is developed in &os.current; before including it in &os.stable;, more people run &os.stable; than &os.current;, so it is inevitable that bugs and corner cases will sometimes be found in &os.stable; that were not apparent in &os.current;. For these reasons, one should not blindly track &os.stable;. It is particularly important not to update any production servers to &os.stable; without first thoroughly testing the code in a development/testing environment. Except for those users who have the resources to perform testing, it is recommended that users instead run the most recent release of FreeBSD, and use the binary update mechanism to move from release to release. Using &os.stable; -STABLE using Join the &a.stable.name; list in order to stay informed of build-dependencies that may appear in &os.stable; or any other issues requiring special attention. Developers will also make announcements in this mailing list when they are contemplating some controversial fix or update, giving the users a chance to respond if they have any issues to raise concerning the proposed change. Join the relevant svn list for the branch being tracked. For example, users tracking the 9-STABLE branch should join the &a.svn-src-stable-9.name; list. This list records the commit log entry for each change as it is made, along with any pertinent information on possible side-effects. To join these lists, go to &a.mailman.lists.link;, click on the list to subscribe to, and follow the instructions. In order to track changes for the whole source tree, subscribe to &a.svn-src-all.name;. To install a new system running monthly snapshots built from &os.stable;, refer to Snapshots for more information. Alternatively, it is possible to install the most recent &os.stable; release from the mirror sites and follow the instructions below to upgrade the system to the most up-to-date &os.stable; source code. Several methods are available to upgrade from a &os; mirror site on a system already running a previous release of &os;: Use svn Subversion to check out the desired development or release branch. This is the recommended method, providing access to &os; development as it occurs. Branch names include head for the current development head, and branches identified in the release engineering page, such as stable/9 -STABLE syncing with Subversion or releng/9.2. URL prefixes for Subversion checkout of the base system are shown in Subversion mirror sites. Because of the size of the repository, it is recommended that only desired subtrees be checked out. Consider using CTM -STABLE syncing with CTM if you do not have a fast connection to the Internet. Before compiling &os.stable; -STABLE compiling , read /usr/src/Makefile carefully. Install a new kernel and rebuild the world the first time through as part of the upgrading process. Read &a.stable; and /usr/src/UPDATING to keep up-to-date on other bootstrapping procedures that sometimes become necessary on the road to the next release. Synchronizing Source There are various ways of using an Internet or email connection to stay up-to-date with any given area, or all areas, of the &os; project sources. The primary services are Subversion and CTM. While it is possible to update only parts of the source tree, the only supported update procedure is to update the entire tree and recompile all the programs that run in user space, such as those in /bin and /sbin, and kernel sources. Updating only part of the source tree, only the kernel, or only the userland programs will often result in problems ranging from compile errors to kernel panics or data corruption. Subversion Subversion uses the pull model of updating sources. The user, or a cron script, invokes the svn program, and it brings files up-to-date. Subversion is the preferred means of updating local source trees. The updates are up-to-the-minute and the user controls when they are downloaded. It is easy to restrict updates to specific files or directories and the requested updates are generated on the fly by the server. CTM CTM does not interactively compare the local sources with those on the master archive or otherwise pull them across. Instead, a script which identifies changes in files since its previous run is executed several times a day on the master CTM machine. Any detected changes are compressed, stamped with a sequence-number, and encoded for transmission over email in printable ASCII only. Once received, these CTM deltas can then be handed to the &man.ctm.rmail.1; utility which will automatically decode, verify, and apply the changes to the user's copy of the sources. This process is more efficient than Subversion and places less strain on server resources since it is a push rather than a pull model. There are other trade-offs. If a user inadvertently wipes out portions of the local archive, Subversion will detect and rebuild the damaged portions. CTM will not do this, and if a user deletes some portion of the source tree and does not have a backup, they will have to start from scratch from the most recent CTM base delta and rebuild it all with CTM. Rebuilding <quote>world</quote> Rebuilding world Once the local source tree is synchronized against a particular version of &os; such as &os.stable; or &os.current;, the source tree can be used to rebuild the system. Make a Backup It cannot be stressed enough how important it is to make a backup of the system before rebuilding the system. While rebuilding the world is an easy task, there will inevitably be times when mistakes in the source tree render the system unbootable. Create and verify a backup and have a bootable installation media at hand. You will probably never have to use it, but it is better to be safe than sorry! Subscribe to the Right Mailing List mailing list The &os.stable; and &os.current; branches are, by their nature, in development. People that contribute to &os; are human, and mistakes occasionally happen. Sometimes these mistakes can be quite harmless, just causing the system to print a new diagnostic warning. Or the change may be catastrophic, and render the system unbootable or destroy file systems. When problems occur, a heads up is posted to the appropriate mailing list, explaining the nature of the problem and which systems it affects. An all clear announcement is posted when the problem has been solved. Users who track &os.stable; or &os.current; and do not read &a.stable; or &a.current; respectively, are asking for trouble. Do Not Use <command>make world</command> Some older documentation recommends using make world. However, that command skips some important steps and should only be used by experts. For almost all circumstances make world is the wrong thing to do, and the procedure described here should be used instead. The Canonical Way to Update Your System Before updating the system, read /usr/src/UPDATING for any pre-buildworld steps necessary for that version of the sources. Then, use the procedure outlined here. These upgrade steps assume an upgrade from an older &os; version, consisting of an old compiler, old kernel, old world, and old configuration files. World includes the core system binaries, libraries, and programming files. The compiler is part of world, but has a few special concerns. These steps also assume that the sources to a newer version have already been obtained. If the sources are not up-to-date, refer to for detailed help about synchronizing to a newer version. Updating the system from source is a more subtle process than it might initially seem to be, and the &os; developers have found it necessary over the years to change the recommended approach fairly dramatically as new kinds of unavoidable dependencies come to light. The rest of this section describes the rationale behind the currently recommended upgrade sequence. Any successful update sequence must deal with the following issues: The old compiler might have a bug and not be able to compile the new kernel. So, the new kernel should be built with the new compiler, meaning that the new compiler must be built before the new kernel is built. This does not necessarily mean that the new compiler must be installed before building the new kernel. The new world might rely on new kernel features. So, the new kernel must be installed before the new world is installed. These first two issues are the basis for the core buildworld, buildkernel, installkernel, installworld sequence described in the following paragraphs. Other reasons for using these steps are listed below: The old world might not run correctly on the new kernel, so the new world must be installed immediately upon installing the new kernel. Some configuration changes must be made before the new world is installed, but others might break the old world. Hence, two different configuration upgrade steps are generally needed. For the most part, the update process only replaces or adds files and existing old files are not deleted. In a few cases, this can cause problems. As a result, the update procedure will sometimes specify certain files that should be manually deleted at certain steps. This may or may not be automated in the future. These concerns have led to the following recommended sequence. Note that the detailed sequence for particular updates may require additional steps, but this core process should remain unchanged for some time: make buildworld This first compiles the new compiler and a few related tools, then uses the new compiler to compile the rest of the new world. The result ends up in /usr/obj. make buildkernel This uses the new compiler residing in /usr/obj in order to protect against compiler-kernel mismatches. make installkernel Place the new kernel and kernel modules onto the disk, making it possible to boot with the newly updated kernel. Reboot into single user mode. Single user mode minimizes problems from updating software that is already running. It also minimizes any problems from running the old world on a new kernel. mergemaster -p This does some initial configuration file updates in preparation for the new world. For instance, it may add new user groups to the system, or new user names to the password database. This is often necessary when new groups or special system-user accounts have been added since the last update, so that the installworld step will be able to use the newly installed system user or system group names without problems. make installworld Copies the world from /usr/obj. The new kernel and new world are now installed on disk. mergemaster Repeated to update the remaining configuration files, now that the new world is on disk. make delete-old This target deletes old (obsolete) files. This is important because sometimes they cause problems if left on the disk, for example the presence of the old utmp.h causes problems in some ports when the new utmpx.h is installed. Reboot. A full machine reboot is needed now to load the new kernel and new world with new configuration files. make delete-old-libs Remove any obsolete libraries to avoid conflicts with newer ones. Make sure that all ports have been rebuilt before old libraries are removed. Upgrades from one release of the same &os; branch to a more recent release of the same branch, such as from 9.0 to 9.1, may not need this procedure since it is less likely to run into serious mismatches between compiler, kernel, userland, and configuration files. The approach of make world followed by building and installing a new kernel might work well enough for minor updates. When upgrading across major releases, people who do not follow this procedure should expect some problems. It is also worth noting that many upgrades may require specific additional steps such as renaming or deleting specific files prior to installworld. Read /usr/src/UPDATING carefully, especially at the end, where the currently recommended upgrade sequence is explicitly spelled out. This procedure has evolved over time as the developers have found it impossible to completely prevent certain kinds of mismatch problems. Hopefully, the current procedure will remain stable for a long time. To summarize, the currently recommended way of upgrading &os; from sources is: &prompt.root; cd /usr/src &prompt.root; make buildworld &prompt.root; make buildkernel &prompt.root; make installkernel &prompt.root; shutdown -r now There are a few rare cases when an extra run of mergemaster -p is needed before the buildworld step. These are described in UPDATING. In general, though, this step can safely be omitted when not updating across one or more major &os; versions. After installkernel finishes successfully, boot into single user mode using boot -s from the loader prompt. If using UFS, run: &prompt.root; mount -u / &prompt.root; mount -a -t ufs If using ZFS(assuming a zpool name of zroot), run: &prompt.root; zfs set readonly=off zroot &prompt.root; zfs mount -a Then run: &prompt.root; adjkerntz -i &prompt.root; mergemaster -p &prompt.root; cd /usr/src &prompt.root; make installworld &prompt.root; mergemaster &prompt.root; make delete-old &prompt.root; reboot &prompt.root; make delete-old-libs Read Further Explanations The following sections clearly describe each step, especially when using a custom kernel configuration. Read <filename>/usr/src/UPDATING</filename> Before updating, read /usr/src/UPDATING. This file contains important information about potential problems and may specify the order to run certain commands. If UPDATING contradicts the procedure in this section, UPDATING takes precedence. Reading UPDATING is not an acceptable substitute for subscribing to the correct mailing list. The two requirements are complementary, not exclusive. Check <filename>/etc/make.conf</filename> make.conf Available &man.make.1; options are shown in &man.make.conf.5; and /usr/share/examples/etc/make.conf. These settings can be added to /etc/make.conf to control the way &man.make.1; runs and how it builds programs. Changes to some settings can have far-reaching and potentially surprising effects. Read the comments in both locations and keep in mind that the defaults have been chosen for a combination of performance and safety. Options set in /etc/make.conf take effect every time &man.make.1; is used, including compiling applications from the Ports Collection or user-written C programs, or building the &os; operating system. Check <filename>/etc/src.conf</filename> src.conf /etc/src.conf controls the building of the operating system from source code. Unlike /etc/make.conf, the contents of /etc/src.conf only take effect when the &os; operating system itself is being built. Descriptions of the many options available for this file are shown in &man.src.conf.5;. Be cautious about disabling seemingly unneeded kernel modules and build options. Sometimes there are unexpected or subtle interactions. Update the Files in <filename>/etc</filename> /etc contains a large part of the system's configuration information, as well as scripts that are run at system startup. Some of these scripts change between &os; versions. Some of the configuration files are used in the day to day running of the system, such as /etc/group. There have been occasions when the installation part of make installworld expected certain usernames or groups to exist. When performing an upgrade, it is likely that these users or groups do not yet exist. In some cases make buildworld will check to see if these users or groups exist. The solution is to run &man.mergemaster.8; in pre-buildworld mode with . This compares only those files that are essential for the success of buildworld or installworld. To check which files are owned by the group being renamed or deleted: &prompt.root; find / -group GID -print This command will show all files owned by group GID, which can be either a group name or a numeric group ID. Drop to Single User Mode single-user mode Consider compiling the system in single user mode. Reinstalling the system touches a lot of important system files, all the standard system binaries, libraries, and include files. Changing these on a running system, particularly one with active users, is asking for trouble. multi-user mode Another method is to compile the system in multi-user mode, and then drop into single user mode for the installation. With this method, hold off on the following steps until the build has completed. Drop to single user mode in order to run installkernel or installworld. To enter single user mode from a running system: &prompt.root; shutdown now Alternatively, reboot the system, and at the boot prompt, select the single user option. Once at the single user mode shell prompt, run: &prompt.root; fsck -p &prompt.root; mount -u / &prompt.root; mount -a -t ufs &prompt.root; swapon -a This checks the file systems, remounts / read/write, mounts all the other UFS file systems referenced in /etc/fstab, and turns swapping on. If the CMOS clock is set to local time and not to GMT (this is true if the output of &man.date.1; does not show the correct time and zone), run the following command: &prompt.root; adjkerntz -i This ensures that the local time-zone settings get set up correctly. Remove <filename>/usr/obj</filename> As parts of the system are rebuilt, they are, by default, placed in subdirectories of /usr/obj. The directories shadow those under /usr/src. To speed up the make buildworld process, and possibly save some dependency headaches, remove this directory if it already exists. Some files below /usr/obj may have the immutable flag set which must be removed first using &man.chflags.1;. &prompt.root; cd /usr/obj &prompt.root; chflags -R noschg * &prompt.root; rm -rf * Recompile the Base System Saving the Output It is a good idea to save the output from running &man.make.1; to a file. If something goes wrong, a copy of the error message can be posted to one of the &os; mailing lists. The easiest way to do this is to use &man.script.1; with a parameter that specifies the name of the file to save all output to. Run this command immediately before rebuilding the world, and then type exit when the process has finished: &prompt.root; script /var/tmp/mw.out Script started, output file is /var/tmp/mw.out &prompt.root; make TARGET … compile, compile, compile … &prompt.root; exit Script done, … Do not save the output in /tmp as this directory may be cleared at next reboot. A better place to save the file is /var/tmp or in root's home directory. Compile the Base System While in /usr/src type: &prompt.root; cd /usr/src make To rebuild the world, use &man.make.1;. This command reads instructions from the Makefile, which describes how the programs that comprise &os; should be built and the order in which they should be built. The general format of the command is as follows: &prompt.root; make -x -DVARIABLE target In this example, is an option passed to &man.make.1;. Refer to &man.make.1; for an examples of available options. passes a variable to the Makefile. The behavior of the Makefile is controlled by these variables. These are the same variables as are set in /etc/make.conf, and this provides another way of setting them. For example: &prompt.root; make -DNO_PROFILE target is another way of specifying that profiled libraries should not be built, and corresponds with the NO_PROFILE= true # Avoid compiling profiled libraries line in /etc/make.conf. target tells &man.make.1; what to do. Each Makefile defines a number of different targets, and the choice of target determines what happens. Some targets listed in the Makefile are used by the build process to break out the steps necessary to rebuild the system into a number of sub-steps. Most of the time, no parameters need to be passed to &man.make.1; and the command looks like this: &prompt.root; make target Where target is one of many build options. The first target should always be buildworld. As the names imply, buildworld builds a complete new tree under /usr/obj and installworld installs this tree on the current machine. Having separate options is useful for two reasons. First, it allows for a self hosted build that does not affect any components of a running system. Because of this, buildworld can be run on a machine running in multi-user mode with no fear of ill-effects. It is still recommended that installworld be run in part in single user mode, though. Secondly, it allows NFS mounts to be used to upgrade multiple machines on a network. If order to upgrade three machines, A, B and C, run make buildworld and make installworld on A. B and C should then NFS mount /usr/src and /usr/obj from A, and run make installworld to install the results of the build on B and C. Although the world target still exists, users are strongly encouraged not to use it. Instead, run: &prompt.root; make buildworld It is possible to specify which will cause make to spawn several simultaneous processes. This is most useful on multi-CPU machines. However, since much of the compiling process is I/O bound rather than CPU bound, it is also useful on single CPU machines. On a typical single-CPU machine, run: &prompt.root; make -j4 buildworld &man.make.1; will then have up to 4 processes running at any one time. Empirical evidence posted to the mailing lists shows this generally gives the best performance benefit. On a multi-CPU machine using an SMP configured kernel, try values between 6 and 10 and see how they speed things up. Timings rebuilding world timings Many factors influence the build time, but fairly recent machines may only take a one or two hours to build the &os.stable; tree, with no tricks or shortcuts used during the process. A &os.current; tree will take somewhat longer. Compile and Install a New Kernel kernel compiling To take full advantage of the new system, recompile the kernel. This is practically a necessity, as certain memory structures may have changed, and programs like &man.ps.1; and &man.top.1; will fail to work until the kernel and source code versions are the same. The simplest, safest way to do this is to build and install a kernel based on GENERIC. While GENERIC may not have all the necessary devices for the system, it should contain everything necessary to boot the system back to single user mode. This is a good test that the new system works properly. After booting from GENERIC and verifying that the system works, a new kernel can be built based on a custom kernel configuration file. On &os; it is important to build world before building a new kernel. To build a custom kernel with an existing customized configuration file, use KERNCONF=MYKERNEL: &prompt.root; cd /usr/src &prompt.root; make buildkernel KERNCONF=MYKERNEL &prompt.root; make installkernel KERNCONF=MYKERNEL If kern.securelevel has been raised above 1 and noschg or similar flags have been set on the kernel binary, drop into single user mode to use installkernel. Otherwise, both these commands can be run from multi user mode without problems. See &man.init.8; for details about kern.securelevel and &man.chflags.1; for details about the various file flags. Reboot into Single User Mode single-user mode Reboot into single user mode to test that the new kernel works using the instructions in . Install the New System Binaries Next, use installworld to install the new system binaries: &prompt.root; cd /usr/src &prompt.root; make installworld If variables were specified to make buildworld, specify the same variables to make installworld. However, must never be used with installworld. For example, if you ran: &prompt.root; make -DNO_PROFILE buildworld install the results with: &prompt.root; make -DNO_PROFILE installworld otherwise, the command will try to install profiled libraries that were not built during the make buildworld phase. Update Files Not Updated by <command>make installworld</command> Remaking the world will not update certain directories, such as /etc, /var and /usr, with new or changed configuration files. The simplest way to update the files in these directories is to use &man.mergemaster.8;. Be sure to first make a backup of /etc in case anything goes wrong. <command>mergemaster</command> Tom Rhodes Contributed by mergemaster &man.mergemaster.8; is a Bourne script to aid in determining the differences between the configuration files in /etc, and the configuration files in the source tree /usr/src/etc. This is the recommended solution for keeping the system configuration files up to date with those located in the source tree. To begin, type mergemaster and it will build a temporary root environment, from / down, and populate it with various system configuration files. Those files are then compared to the ones currently installed in the system. Files that differ will be shown in &man.diff.1; format, with the sign representing added or modified lines, and representing lines that will be either removed completely, or replaced with a new file. Refer to &man.diff.1; for more information about the &man.diff.1; syntax and how file differences are shown. &man.mergemaster.8; will then display each file that differs, and present the options of either deleting the new file, referred to as the temporary file, installing the temporary file in its unmodified state, merging the temporary file with the currently installed file, or viewing the &man.diff.1; results again. Choosing to delete the temporary file will tell &man.mergemaster.8; to keep the current file unchanged and to delete the new version. This option is not recommended, unless there is no reason to change the current file. To get help at any time, type ? at the &man.mergemaster.8; prompt. If the user chooses to skip a file, it will be presented again after all other files have been dealt with. Choosing to install the unmodified temporary file will replace the current file with the new one. For most unmodified files, this is the best option. Choosing to merge the file will present a text editor, and the contents of both files. The files can be merged by reviewing both files side by side on the screen, and choosing parts from both to create a finished product. When the files are compared side by side, l selects the left contents and r selects contents from the right. The final output will be a file consisting of both parts, which can then be installed. This option is customarily used for files where settings have been modified by the user. Choosing to view the &man.diff.1; results again will display the file differences just like &man.mergemaster.8; did before prompting an option. After &man.mergemaster.8; is done with the system files, it will prompt for other options. &man.mergemaster.8; may prompt to rebuild the password file and will finish up with an option to remove left-over temporary files. Manual Update To perform the update manually instead, do not just copy over the files from /usr/src/etc to /etc and expect it to work. Some files must be installed first as /usr/src/etc is not a copy of what /etc should look like. In addition, some files that should be in /etc are not in /usr/src/etc. If you are using &man.mergemaster.8; (as recommended), you can skip forward to the next section. The simplest way to merge files by hand is to install the files into a new directory, and then work through them looking for differences. Backup Your Existing <filename>/etc</filename> It is recommended to first copy the existing /etc somewhere safe, like so: &prompt.root; cp -Rp /etc /etc.old where does a recursive copy and preserves times and the ownerships on files. Build a temporary set of directories into which the new /etc and other files can be installed: &prompt.root; mkdir /var/tmp/root &prompt.root; cd /usr/src/etc &prompt.root; make DESTDIR=/var/tmp/root distrib-dirs distribution This will build the necessary directory structure and install the files. A lot of the subdirectories that have been created under /var/tmp/root are empty and should be deleted. The simplest way to do this is to: &prompt.root; cd /var/tmp/root &prompt.root; find -d . -type d | xargs rmdir 2>/dev/null This will remove all empty directories while redirecting standard error to /dev/null to prevent the warnings about the directories that are not empty. /var/tmp/root now contains all the files that should be placed in appropriate locations below /. Go through each of these files, determining how they differ from the system's existing files. Some of the files installed into /var/tmp/root have a leading .. Make sure to use ls -a in order to catch them. The simplest way to compare files is to use &man.diff.1;: &prompt.root; diff /etc/shells /var/tmp/root/etc/shells This command will show the differences between the existing /etc/shells and the new /var/tmp/root/etc/shells. Review the differences to decide whether to merge in custom changes or to replace the existing file with the new one. Name the New Root Directory (<filename>/var/tmp/root</filename>) with a Time Stamp, so You Can Easily Compare Differences Between Versions Frequently rebuilding world entails frequently updating /etc as well, which can be a bit of a chore. To speed up this process, use the following procedure to keep a copy of the last set of changed files that were merged into /etc. Make the world as normal. When updating /etc and the other directories, give the target directory a name based on the current date: &prompt.root; mkdir /var/tmp/root-20130214 &prompt.root; cd /usr/src/etc &prompt.root; make DESTDIR=/var/tmp/root-20130214 \ distrib-dirs distribution Merge in the changes from this directory as outlined above. Do not remove the /var/tmp/root-20130214 directory when you have finished. After downloading the latest version of the source and remaking it, follow step 1. Create a new directory, which reflects the new date. This example uses /var/tmp/root-20130221. Use &man.diff.1; to see the differences that have been made in the intervening week by creating a recursive diff between the two directories: &prompt.root; cd /var/tmp &prompt.root; diff -r root-20130214 root-20130221 Typically, this will be a much smaller set of differences than those between /var/tmp/root-20130221/etc and /etc. Because the set of differences is smaller, it is easier to migrate those changes across into /etc. When finished, remove the older of the two /var/tmp/root-* directories: &prompt.root; rm -rf /var/tmp/root-20130214 Repeat this process whenever merging in changes to /etc. Use &man.date.1; to automate the generation of the directory names: &prompt.root; mkdir /var/tmp/root-`date "+%Y%m%d"` Deleting Obsolete Files and Directories Anton Shterenlikht Based on notes provided by Deleting obsolete files and directories As a part of the &os; development lifecycle, files and their contents occasionally become obsolete. This may be because functionality is implemented elsewhere, the version number of the library has changed, or it was removed from the system entirely. This includes old files, libraries, and directories, which should be removed when updating the system. The benefit is that the system is not cluttered with old files which take up unnecessary space on the storage and backup media. Additionally, if the old library has a security or stability issue, the system should be updated to the newer library to keep it safe and to prevent crashes caused by the old library. Files, directories, and libraries which are considered obsolete are listed in /usr/src/ObsoleteFiles.inc. The following instructions should be used to remove obsolete files during the system upgrade process. After the make installworld and the subsequent mergemaster have finished successfully, check for obsolete files and libraries as follows: &prompt.root; cd /usr/src &prompt.root; make check-old If any obsolete files are found, they can be deleted using the following command: &prompt.root; make delete-old Refer to /usr/src/Makefile for more targets of interest. A prompt is displayed before deleting each obsolete file. To skip the prompt and let the system remove these files automatically, use BATCH_DELETE_OLD_FILES: &prompt.root; make -DBATCH_DELETE_OLD_FILES delete-old The same goal can be achieved by piping these commands through yes: &prompt.root; yes|make delete-old Rebooting Verify that everything appears to be in the right place, then reboot the system using &man.shutdown.8;: &prompt.root; shutdown -r now Deleting obsolete libraries Warning Deleting obsolete files will break applications that still depend on those obsolete files. This is especially true for old libraries. In most cases, the programs, ports, or libraries that used the old library need to be recompiled before make delete-old-libs is executed. Utilities for checking shared library dependencies are available from the Ports Collection in sysutils/libchk or sysutils/bsdadminscripts. Obsolete shared libraries can conflict with newer libraries, causing messages like these: /usr/bin/ld: warning: libz.so.4, needed by /usr/local/lib/libtiff.so, may conflict with libz.so.5 /usr/bin/ld: warning: librpcsvc.so.4, needed by /usr/local/lib/libXext.so, may conflict with librpcsvc.so.5 To solve these problems, determine which port installed the library: &prompt.root; pkg which /usr/local/lib/libtiff.so /usr/local/lib/libtiff.so was installed by package tiff-3.9.4 &prompt.root; pkg which /usr/local/lib/libXext.so /usr/local/lib/libXext.so was installed by package libXext-1.1.1,1 Then deinstall, rebuild, and reinstall the port. ports-mgmt/portmaster can be used to automate this process. After all ports are rebuilt and no longer use the old libraries, delete the old libraries using the following command: &prompt.root; make delete-old-libs You should now have successfully upgraded the &os; system. Congratulations. If things went slightly wrong, it is easy to rebuild a particular piece of the system. For example, if /etc/magic was accidentally deleted as part of the upgrade or merge of /etc, &man.file.1; will stop working. To fix this, run: &prompt.root; cd /usr/src/usr.bin/file &prompt.root; make all install Questions Do I need to re-make the world for every change? There is no easy answer, as it depends on the nature of the change. For example, if running svn only shows the following files as being updated: src/games/cribbage/instr.c src/games/sail/pl_main.c src/release/sysinstall/config.c src/release/sysinstall/media.c src/share/mk/bsd.port.mk it probably is not worth rebuilding the entire world. Instead, go into the appropriate sub-directories and run make all install. But if something major changed, such as src/lib/libc/stdlib, either re-make world, or at least those parts of it that are statically linked. At the end of the day, it is your call. Some users re-make the world every fortnight and let changes accumulate over that fortnight. Others only re-make those things that have changed and are careful to spot all the dependencies. It all depends on how often a user wants to upgrade and whether they are tracking &os.stable; or &os.current;. My compile failed with lots of signal 11 signal 11 (or other signal number) errors. What happened? This normally indicates hardware problems. (Re)making world is an effective way to stress test hardware, and will frequently throw up memory problems which normally manifest themselves as the compiler mysteriously aborts. A sure indicator of this occurs when make is restarted and it dies at a different point in the process. To resolve this error, start swapping around the components in the machine to determine which one is failing. Can /usr/obj be removed when finished? The short answer is yes. /usr/obj contains all the object files that were produced during the compilation phase. Normally, one of the first steps in the make buildworld process is to remove this directory and start afresh. Keeping /usr/obj around when finished makes little sense, and its removal frees up a approximately 2 GB of disk space. Advances users can instruct make buildworld to skip this step. This speeds up subsequent builds, since most of the sources will not need to be recompiled. The flip side is that subtle dependency problems can creep in, causing the build to fail in odd ways. This frequently generates noise on the &os; mailing lists, when one person complains that their build has failed, not realizing that it is because they have tried to cut corners. Can interrupted builds be resumed? This depends on how far into the process the problem occurs. In general, make buildworld builds new copies of essential tools, such as &man.gcc.1; and &man.make.1;, and the system libraries. These tools and libraries are then installed, used to rebuild themselves, and are installed again. The entire system, including regular user programs such as &man.ls.1; or &man.grep.1;, is then rebuilt with the new system files. During the last stage, it is fairly safe to: … fix the problem … &prompt.root; cd /usr/src &prompt.root; make -DNO_CLEAN all This will not undo the work of the previous make buildworld. If you see the message: -------------------------------------------------------------- Building everything.. -------------------------------------------------------------- in the make buildworld output, it is probably fairly safe to do so. If that message is not displayed, or you are not sure, it is always better to be safe than sorry, and restart the build from scratch. How can I speed up making the world? Run it in single user mode. Put /usr/src and /usr/obj on separate file systems held on separate disks. If possible, put these disks on separate disk controllers. Alternately, put these file systems across multiple disks using &man.ccd.4;. Turn off profiling by setting NO_PROFILE=true in /etc/make.conf. Pass to &man.make.1; to run multiple processes in parallel. This usually helps on both single and multi processor machines. The file system holding /usr/src can be mounted or remounted with . This prevents the file system from recording the file access time which is probably not needed. &prompt.root; mount -u -o noatime /usr/src This example assumes /usr/src is on its own file system. If it is part of /usr, then use that file system mount point instead. The file system holding /usr/obj can be mounted or remounted with so that disk writes happen asynchronously. The write completes immediately, and the data is written to the disk a few seconds later. This allows writes to be clustered together, and can provide a dramatic performance boost. Keep in mind that this option makes the file system more fragile. With this option, there is an increased chance that, should power fail, the file system will be in an unrecoverable state when the machine restarts. If /usr/obj is the only directory on this file system, this is not a problem. If you have other, valuable data on the same file system, ensure that there are verified backups before enabling this option. &prompt.root; mount -u -o async /usr/obj If /usr/obj is not on its own file system, replace it in the example with the name of the appropriate mount point. What do I do if something goes wrong? Make absolutely sure that the environment has no extraneous cruft from earlier builds: &prompt.root; chflags -R noschg /usr/obj/usr &prompt.root; rm -rf /usr/obj/usr &prompt.root; cd /usr/src &prompt.root; make cleandir &prompt.root; make cleandir Yes, make cleandir really should be run twice. Then, restart the whole process, starting with make buildworld. If problems persist, send the error and the output of uname -a to &a.questions;. Be prepared to answer other questions about the setup! Tracking for Multiple Machines Mike Meyer Contributed by NFS installing multiple machines When multiple machines need to track the same source tree, it is a waste of disk space, network bandwidth, and CPU cycles to have each system download the sources and rebuild everything. The solution is to have one machine do most of the work, while the rest of the machines mount that work via NFS. This section outlines a method of doing so. Preliminaries First, identify a set of machines which will run the same set of binaries, known as a build set. Each machine can have a custom kernel, but will run the same userland binaries. From that set, choose a machine to be the build machine that the world and kernel are built on. Ideally, this is a fast machine that has sufficient spare CPU to run make buildworld and make buildkernel. Select a machine to be the test machine, which will test software updates before they are put into production. This must be a machine that can afford to be down for an extended period of time. It can be the build machine, but need not be. All the machines in this build set need to mount /usr/obj and /usr/src from the same machine, and at the same point. Ideally, those directories are on two different drives on the build machine, but they can be NFS mounted on that machine as well. For multiple build sets, /usr/src should be on one build machine, and NFS mounted on the rest. Finally, ensure that /etc/make.conf and /etc/src.conf on all the machines in the build set agree with the build machine. That means that the build machine must build all the parts of the base system that any machine in the build set is going to install. Also, each build machine should have its kernel name set with KERNCONF in /etc/make.conf, and the build machine should list them all in KERNCONF, listing its own kernel first. The build machine must have the kernel configuration files for each machine in /usr/src/sys/arch/conf if it is going to build their kernels. The Base System On the build machine, build the kernel and world as described in , but do not install anything. After the build has finished, go to the test machine, and install the built kernel. If this machine mounts /usr/src and /usr/obj via NFS, enable the network and mount these directories after rebooting to single user mode. The easiest way to do this is to boot to multi-user, then run shutdown now to go to single user mode. Once there, install the new kernel and world and run mergemaster as usual. When done, reboot to return to normal multi-user operations for this machine. After verifying that everything on the test machine is working properly, use the same procedure to install the new software on each of the other machines in the build set. Ports The same ideas can be used for the ports tree. The first critical step is to mount /usr/ports from the same machine to all the machines in the build set. Then, configure /etc/make.conf properly to share distfiles. Set DISTDIR to a common shared directory that is writable by whichever user root is mapped to by the NFS mounts. Each machine should set WRKDIRPREFIX to a local build directory. Finally, if the system is to build and distribute packages, set PACKAGES to a directory similar to DISTDIR.